Patent Publication Number: US-2022212100-A1

Title: Systems and methods for streaming interactive applications

Description:
BACKGROUND 
     Background and Relevant Art 
     Game servers based on retail commodity hardware have limited resources, most of which are dedicated to running the game application. Even general-purpose computing devices used to run game applications can dedicate most, if not all, available computing resources to running the game application. Comparatively little computing resources remain for video encoding and managing network protocol to communicate video and audio information to client devices over a network and to receive user inputs from the client devices over the network. 
     BRIEF SUMMARY 
     In some implementations, a distribution node for allowing interaction with interactive applications to a remote user includes a video connection device, a network connection device, a processor, and a hardware storage device. The video connection device is configured to connect to a game server and receive at least video information from the game server. The network connection device is configured to connect to a network. The hardware storage device has instructions stored that, when executed by the processor, cause the distribution node to encode the video information from the game server to encoded video information and distribute the encoded video information over the network to a client device. 
     In some implementations, a system for allowing interaction with game applications to a remote user includes a game server and a distribution node. The game server is configured to run a game application and generate at least video information. The distribution node is connected to the game server and configured to receive the video information and encode the video information for distribution via a network. 
     In some implementations, a method of providing interaction with a game instance of a game application by a remote user includes, at a distribution node, receiving video information from a game server, encoding the video information into encoded video information, distributing the encoded video information to a client device, receiving a user input from the client device, and transmitting the user input to the game server. 
     This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter. 
     Additional features and advantages will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the teachings herein. Features and advantages of the disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present disclosure will become more fully apparent from the following description and appended claims or may be learned by the practice of the disclosure as set forth hereinafter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       In order to describe the manner in which the above-recited and other features of the disclosure can be obtained, a more particular description will be rendered by reference to specific implementations thereof which are illustrated in the appended drawings. For better understanding, the like elements have been designated by like reference numbers throughout the various accompanying figures. While some of the drawings may be schematic or exaggerated representations of concepts, at least some of the drawings may be drawn to scale. Understanding that the drawings depict some example implementations, the implementations will be described and explained with additional specificity and detail through the use of the accompanying drawings in which: 
         FIG. 1  is a schematic representation of a streaming system for interactive game applications, according to at least some implementations of the present disclosure; 
         FIG. 2  is a schematic representation of another streaming system for interactive game applications, according to at least some implementations of the present disclosure; 
         FIG. 3  is a schematic representation of a streaming system for interactive game applications employing machine learning to improve performance, according to at least some implementations of the present disclosure; 
         FIG. 4  is a schematic representation of a streaming system for interactive game applications including modular post-processing, according to at least some implementations of the present disclosure; 
         FIG. 5  is a schematic representation of a streaming system for interactive game applications with spectating of a video stream, according to at least some implementations of the present disclosure; 
         FIG. 6  is a schematic representation of a streaming system for interactive game applications to a plurality of client device, according to at least some implementations of the present disclosure; and 
         FIG. 7  is a schematic representation of a particular implementation of a streaming system for interactive game applications. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure relates generally to systems and methods for allowing interaction with a game application. More specifically, the systems and methods described herein allow interaction with a game instance executed on a game server by a remote user viewing the game instance on a client device. In some implementations, receiving and transmitting information between the game server and the client device requires additional processing power that may compromise the performance of the game server actually running the game application. In a particular example, the game server is retail commodity hardware. 
     In some implementations, retail commodity hardware for a game server is a specialized computing device that is homologated for predictable development of, execution of, and communication between game applications and game instances. Examples of retail commodity hardware include video game consoles such as Microsoft Xbox Series X, Sony PlayStation 5, and Nintendo Switch. Retail commodity hardware may be used in contrast to general purpose computers, such as a personal computer that may include a wide variety of hardware computing components be used to play game applications at a variety of settings and performance levels. Game applications developed for retail commodity hardware may be developed to maximize the available computing resources of the retail commodity hardware, and some game applications may not leave much or any computing resources available for distribution of the video information, audio information, haptic information, etc. related to a game instance of the game application to a remote user on a client device connected via a network. 
     A distribution node is in data communication with the game server and a client device via a network. The distribution node is configured to receive at least video information from the game server and encode the video information for distribution to the client device over the network. In some implementations, offloading the encoding and distribution of the video information, as well as other communication functionality, to the distribution node allows the game server to execute the game application as the developers intended without a negative impact of the additional computing resources needed for remote play. 
     In some implementations, the game server is a general-purpose computer, and the distribution node allows modularity and/or additional functionality beyond the processing resources and/or available software on the game server. For example, the game server, even when a general-purpose computer, may be updated at different interaction cycles than the processing hardware of the distribution node. In at least one example, the distribution node is configured to execute at least one machine learning (ML) model to analyze and refine the ML model based upon the video information and/or user inputs received by the distribution node. The distribution node may be changed, upgraded, repaired, or otherwise altered independently of the game server, providing greater flexibility in game streaming and remote play compared to a direct connection between the game server and client device. 
     As illustrated in the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and advantages of the machine learning model. Additional detail is now provided regarding the meaning of such terms. For example, as used herein, a “machine learning model” refers to a computer algorithm or model (e.g., a classification model, a regression model, a language model, an object detection model) that can be tuned (e.g., trained) based on training input to approximate unknown functions. For example, a machine learning model may refer to a neural network or other machine learning algorithm or architecture that learns and approximates complex functions and generate outputs based on a plurality of inputs provided to the machine learning model. In some implementations, a machine learning system, model, or neural network described herein is an artificial neural network. In some implementations, a machine learning system, model, or neural network described herein is a convolutional neural network. In some implementations, a machine learning system, model, or neural network described herein is a recurrent neural network. In at least one implementation, a machine learning system, model, or neural network described herein is a Bayes classifier. As used herein, a “machine learning system” may refer to one or multiple machine learning models that cooperatively generate one or more outputs based on corresponding inputs. For example, a machine learning system may refer to any system architecture having multiple discrete machine learning components that consider different kinds of information or inputs. 
       FIG. 1  is a schematic representation of a system  100  for allowing a remote user at a client device  102  to play a game application  104  on a game server  106 . The game server  106  executes the game application  104  and transmits at least video information  108  to a distribution node  110 . In some implementations, the distribution node  110  encodes the video information  108  (to create encoded video information  112 ) from the game server  106  for distribution over a network  114  to the client device  102 . In some implementations, the network  114  is the World Wide Web. In some implementations, the network  114  is a local area network. In some implementations, the network  114  is a wide area network. In some implementations, the network  114  is the World Wide Web and the client device  102  may be anywhere in the world relative to the game server  106 . 
     The client device  102  received the encoded video information  112  from the distribution node  110  and displays to a user a video feed of a game instance of the game application  104  that is executed by the game server  106 . The user may then provide one or more user inputs  116  to the client device  102  to interact with the game instance of the game application  104 . The user inputs  116  are transmitted to and received by the game server  106 . The game server  106  interprets the user inputs  116  to allow the user to interact with the game application  104 . 
     In some implementations, the game server  106  is local to and directly connected to the distribution node  110 . For example, the game server  106  may be in the same datacenter as the distribution node  110 . In some examples, the game server  106  and distribution node  110  may be in the same server rack in a datacenter. In some implementations, the game server  106  is located remotely relative to the distribution node  110 . For example, the game server  106  may be located in a first datacenter and the distribution node  110  may be located in a second datacenter connected to the first datacenter via a network. 
     As described herein, the game server  106  may be a general-purpose computing device or a specialized computing device, such as a retail commodity hardware video game console. In either case, the game server  106  includes a processor(s), such as central processing unit, graphical processing unit, physics processing unit, or other processors; a hardware storage device; and a video output device. The hardware storage device has instructions stored thereon that, when executed by the processor(s) of the game server  106 , cause the game server  106  to execute a game application  104  and render a game instance of the game application  104 . 
     In some implementations, the hardware storage device is any non-transient computer readable medium that may store instructions thereon. The hardware storage device may be any type of solid-state memory; volatile memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM); or non-volatile memory, such as read-only memory (ROM) including programmable ROM (PROM), erasable PROM (ERPOM) or EEPROM; magnetic storage media, such as magnetic tape; platen-based storage device, such as hard disk drives; optical media, such as compact discs (CD), digital video discs (DVD), Blu-ray Discs, or other optical media; removable media such as USB drives; non-removable media such as internal SATA or non-volatile memory express (NVMe) style NAND flash memory, or any other non-transient storage media. In some implementations, the hardware storage device is local to and/or integrated with the computing device. In some implementations, the hardware storage device is accessed by the computing device through a network connection. 
     The video information  108  rendered by the game server  106  is provided, such as by the video output device, to the distribution node  110 . In some implementations, the distribution node  110  includes at least a processor(s), a hardware storage device, a video connection device, and a network connection device. The hardware storage device of the distribution node  110  has instructions stored thereon that, when executed by the processor(s) of the distribution node  110 , cause the distribution node  110  to execute any of the methods described herein. 
     The video connection device receives the video information  108  from the game server  106 . In some implementations, the video connection device is a physical piece of hardware, such as an HDMI-In port, or a logical software component, such as software to receive video information through a more generalized data connection, such as USB, Ethernet, wireless antenna, optical, etc. In some implementations, the game server  106  provides additional game instance information, such as audio information, haptic information, game state data, or other information, and the distribution node is configured to receive such additional information and distribute that information in addition to the video information. The processors may encode the video information  108  to create encoded video information  112 , and the network connection device of the distribution node  110  may distribute the encoded video information  112  to the client device  102  via the network  114 . 
     In some implementations, the video information  108  is encoded at the distribution node  110  for distribution to the client devices  102  as encoded video information  112 . In some implementations, the video information  108  is encoded at the game client  106  for transmission from the game client  106  to the distribution node  110  and re-encoded by the distribution node  110  for transmission to the client devices  102 . For example, the encoding used by game client  106  may be different from that used by the distribution node  110 , as the encoding used by the distribution node  110  may be selected based on the particular network connection established between the distribution node  110  and the client device  102 . In some implementations, the video information  108  may be encoded upon arrival at the distribution node, but the use of the term “encoded video information  112 ” indicates that the distribution node  110  performs an encoding process on the video data, whether that is a first encoding or a re-encoding of the video information  108  for distribution to the client device  102 . 
     The client device  102  is any electronic device with network connectivity to receive and transmit information to a remotely located game server  106  and/or distribution node  110 . In particular examples, client devices  102  include personal electronic devices, such as smartphones, tablet computers, desktop computers, laptop computers, hybrid computers, wearable electronic devices; television; automotive infotainment system; household appliance, or any other networked electronic device. In some implementations, the client device  102  views a game instance from a game server  106 , and the client device  102  transmits user inputs  116  to the game server  106  to interact with the game application  104 . In some implementations, the client device  102  is a general-purpose computing device with an internet browser-based connection to the distribution node  110  and/or game server  106 . In some implementations, the client device  102  executes a native application to connect to the distribution node  110  and/or game server  106 . 
     A user input, according to the present disclosure, should be understood to include any signal or input by any input mechanism that provides instructions to the game server to interact with and/or affect the game application. In some implementations, the user input  116  may be provided through a human interface device (HID) of the client device  102 , such as a keyboard, touchpad, or mouse, or through a dedicated gamepad controller or other peripheral primarily used for playing video games. 
     In some implementations, relaying user inputs  116  through the distribution node  110  introduces input lag to the system, compromising the responsiveness of the game application  104  to the user inputs  116  provided by the user at the client device  102 .  FIG. 2  illustrates another implementation of a system  200 , according to the present disclosure, which allows for a user inputs  216  to be routed directly to the game server  206 . Direct communication of the user inputs  216  between the client device  202  and the game server  206  may reduce latency in the communication between the user providing the user inputs  216  at the client device  202  and the user inputs  216  being received and processed by the game server  206 . 
     Additional processes may be used to further reduce perceived latency for the user through the use of ML models performed at the distribution node  210  and/or on an additional node, such as the ML node  318  illustrated in the system  300  of  FIG. 3 . In some implementations, a system  300  according to the present disclosure can include a ML model that receives game state data from the game server and uses the game state data to provide additional inputs to the game server to complement the user inputs. 
     Game state data includes any information that may allow a second electronic device to recreate a given game state. For example, the game state data of a first game instance running on a first electronic device may be provided to second copy of a game application running on a second electronic device, and the second electronic device may render a duplicate of the first game instance based on the game state data. In some implementations, game state data includes virtual object or avatar positions, movement, player character statistics or characteristics, player character inventory, player character status, ability cooldown status, non-player character status, or any other information about the game state of the first instance. In some implementations, the game state data is used to synchronize two instances of a game application to render the same game environment to two different players. In some implementations, the game state data is used to present the game state of the first instance asynchronously to a second instance to allow the same or different player to experience the game state of the first instance at a later time. 
     In some implementations, the game state data  320  is provided to a ML model, such as that used by the ML node  318 , and the ML model can identify the current status or situation of the game application. For example, the ML model may receive game state data  320  that includes data regarding the game application, the game instance, the player character, and the game environment, which allows the ML model to make predictions about what the user may intend to do next in the game. In a particular example, the ML model may identify that the game application is a driving game, and the user is racing against computer-controlled avatars. Due to latency in the network connection, the user inputs  316  may be delayed from the client device  302  or packets of video information may be delayed en route to the client device  302  that compromise the user&#39;s ability to react to the current game state. The ML model may identify the current game state from the game state data  320  and anticipate that the user will steer their car to follow the road. The ML node  318  may provide ML inputs  322  to supplement the user inputs  316  that help steer the user&#39;s car in the absence or delay of user inputs  316 . 
     In the implementation illustrated in  FIG. 3 , the distribution node  310  receives both the user input  316  from the client device  302  and the ML input from the ML node  318 . The distribution node  310 , in some implementations, merges the user inputs  316  with the ML inputs  322  and transmits both the user inputs  316  and the ML inputs  322  to the game server  306 . In some examples, the predicted ML input  322  and the actual user input  316  are the same, and merging the inputs has no noticeable effect. In other examples, the user input  316  is absent at the distribution node  310  due to latency and/or network packet drops, and the ML input  322  may fill in the gaps in the user inputs  316  with the predicted inputs. In a particular example, the user provides a continuous right directional input to steer the car to the right around a corner. The ML input  322  predicts a right input for the corner and predicts that a user input  316  will continue to be a right directional input throughout the corner in the game environment. If a network connection issue causes a break in the user input, the distribution node  310  may use the ML input to supplement the user input  316  to transmit a continuous right-direction input to the game server  306 . 
     In yet other examples, the distribution node  310  receives different user input  316  and a ML input  322 . In such examples, the distribution node  310  may prioritize the transmission of the user input  316  and ignore the ML input  322 , whereby the game server  306  receives the user input  316  as the user would expect. For example, while the ML model may predict that user steer to the right to follow the corner of the previous example, a user may instead continue in a straight line to pursue a shortcut or intentionally drive off the road. 
     In some implementations, the ML model may receive the user inputs from the client device and/or the distribution node to refine the ML model.  FIG. 4  is a schematic illustration of another implementation of a system  400  where the ML node  418  is located between the game server  406  and the distribution node  410  and may modify and/or supplement data transmitted between the game server  406  and the distribution node  410 . 
     In some examples, the ML node  418  may receive game state data  420 , as described herein, to recreate the game environment of the game instance running on the game server  406 . In some implementations, the ML node  418  also receives user inputs  416  from the client device  402 . In some examples, the user inputs  416  are received through the distribution node  410 , while in other examples, the user inputs  416  are received directly from the client device  402  (without traveling through the distribution node  410 ) to reduce latency. The ML node  418  can use a ML model to evaluate the game instance represented by the game state data  420  and compare a predicted ML input  422  to the received user input  416 . In some implementations, the ML node  418  transmits to the game server  406  the user input  416  and/or the ML input  422  based on the comparison of the user input  416  and the ML input  422 . 
     In some implementations, the ML node refines the ML model based on a comparison of the ML input  422  and the user input  416 . In the driving game example described herein, the ML input  422  supplemented the user input  416  to predict that the user would steer to remain on the road based on aggregated community statistics. In other words, most players that race on that track follow the road. In some implementations, the ML model may be refined using user-specific data to customize the ML inputs to predict what that particular user may input. For example, a particular user may be aggressive at cutting the inside curb of corners and allowing their car to partially leave the road. The user-specific data based on the received user input  416  may be compared to the predicted input from the generalized ML model and the generalized ML model may be refined to provide ML inputs  422  that more accurately predict the inputs of the particular user in the event of user inputs  416  being lost or delayed by network irregularities. 
     In some implementations, a ML node  418  provides benefits to the video information output by the game server  406 . For example, the game server  406  may output a first video information  408 - 1  that is received and processed by the ML node  418 . The ML node  418  may apply one or more post-processing operations to the first video information  408 - 1  to create a second video information  408 - 2  that is transmitted to the distribution node  410  for distribution to a client device  402 . In some implementations, the post-processing includes resolution up-scaling, such as upscaling from native 1080p resolutions to  4 K resolutions. In some implementations, the post-processing includes framerate interpolation, such as interpolating 60 frames per second to 120 frames per second. In some implementations, the post-processing includes improvement to the dynamic range of the video information  408 - 1 , such as changes from standard dynamic range (SDR) to high dynamic range (HDR). In some implementations, the post-processing includes additional shading and/or anti-aliasing of the video information  408 - 1 . 
     In some implementations, additional information can be added to and/or improved by the ML node  418 . For example, the ML node  418  may use the video information  408 - 1  and/or the game state data  420  to determine haptic information, such as rumble feedback or adaptive resistance in an input device, that may be transmitted to the client device for presentation to a user through an input device. In at least one example, a legacy or backward compatible game application may not support all types of haptic feedback available to the user on input device(s) of the client device  402 . The ML node  418  may analyze the information  408 - 1  and/or the game state data  420  to determine haptic information and add that haptic information to the second video information  408 - 2  or other information to be distributed to the client device  402 . 
     In some implementations, audio information may be improved or added for distribution to the client device  402 . For example, a game application may not support surround sound or 3D audio natively in the game application. The ML node  418  may analyze the game state data  420  and determine or add directionality of audio sources to simulate surround sound or 3D audio at the client device  402 . 
     While the post-processing operations (video, audio, haptic, etc.) and ML input calculations are described in relation to the ML node  418  being discrete from the distribution node  410 , in some implementations, the ML node  418  is integrated into the distribution node  410 . In at least one example, the ML model and the video encoding may both be performed at the distribution node  410  before the encoded video information  412  (and, optionally, audio information or haptic information) is distributed to the client device  402  and/or the inputs (user inputs  416  and/or ML inputs  422 ) are transmitted to the game server  406 . 
       FIG. 5  is a schematic representation of a system  500  according to the present disclosure that allows a viewing user at a spectator device  524  to view a video stream  526  (i.e., non-interactive viewing) of the game instance played by a playing user at a client device  502 . The game server  506  executes the game application to create a game instance, at least the video information  508  of which is provided to the distribution node  510  for encoding. The encoded video information  512  is encoded based on the network connection specifically with the client device  502 . Additionally, the video information is encoded at a plurality of bitrates in a bitrate ladder  528  and provided to a streaming node  530 . 
     Bitrate is a measure of bandwidth consumed by the video stream  526 . The bitrate is affected by the resolution, the frames per second, the color depth, the dynamic range, etc. By providing encoding the video stream  526  at a plurality of bitrates, an adaptive bitrate ladder can provide the best quality video stream  526  to a spectator device  524  based on the connection quality (i.e., speed) between the spectator device  524  and the streaming node  530 . While the streaming node  530  is illustrated as a discrete component of the system  500 , it should be understood that, in some implementations, the distribution node  510  is integrated with the streaming node  530 , and the distribution node  510  distributes both the encoded video information  512  to the client device  502  for interactive play with the game application, as well as the video stream  526  with an adaptive bitrate ladder to the spectator device  524 . 
     In some implementations, the streaming node  530  provides one or more social features to the spectator device  524  in addition to the video stream  526 . For example, while the spectating device  524  cannot interact with the game instance to which the client device  502  is interactively connected, the streaming node  530  may provide a text or voice chat function for a viewing user at the spectator device  524  to communicate with the playing user at the client device  502 . In some examples, a plurality of spectator devices  524  are viewing the video stream  526  from the streaming node  530 , and the streaming node  530  allows the spectator devices  524  to communicate with one another. 
     A system  600  according to the present disclosure allows for multiple client devices  602 - 1 ,  602 - 2  to connect to a single distribution node  610 , such as illustrated in the implementation of  FIG. 6 . While two client devices  602 - 1 ,  602 - 2  are illustrated in the implementation of  FIG. 6 , it should be understood that any number of client devices may be connected to the distribution node  610 . In some implementations, a first client device  602 - 1  and a second client device  602 - 2  are connected to the same distribution node to allow both client devices  602 - 1 ,  602 - 2  to receive video information from a first game server  606 - 1  and send user inputs  616 - 1 ,  616 - 2  to the first game server  606 - 1 . For example, the first client device  602 - 1  and second client device  602 - 2  can play a single instance of a game application together, either cooperatively or competitively, depending on the game application. In such an implementation, the first client device  602 - 1  and second client device  602 - 2  each receive the same encoded video information  612 - 1 ,  612 - 2  from the distribution node  610 , such that both players see the same frames from the first game server  606 - 1  as if the players were playing “couch co-op” and both of the first user input  616 - 1  and second user input  616 - 2  affect the shared game instance. A shared game instance is a single game instance of a game application executed by the game server, and a plurality of client devices are connected to the shared game instance. Each client device of the plurality of client devices is able to interact with the game instance on the game server, and each client device of the plurality of client devices receives the same video information related to the shared game instance. 
     In some implementations, the first client device  602 - 1  and second client device  602 - 2  receive the same video information (i.e., see the same frames and/or images provided from the game server) but the distribution node  610  encodes the video information differently. For example, the video information may be encoded at the distribution node  610  based on the different network connections of each of the client devices  602 - 1 ,  602 - 2 . While the video information  608  is the same, the distribution node  610  may distribute a first encoded video information  612 - 1  to the first client device  602 - 1  based on the particular network connection of the first client device  602 - 1  with the distribution node  610  and distribute a second encoded video information  612 - 2  to the second client device  602 - 2  based on the particular network connection of the second client device  602 - 2  with the distribution node  610 . 
     In some implementations, the first client device  602 - 1  and second client device  602 - 2  are connected to the same distribution node  610 , but the distribution node  610  is connected to both a first game server  606 - 1  and a second game server  606 - 2 . While two game servers  606 - 1 ,  606 - 2  are illustrated in the implementation of  FIG. 6 , it should be understood that any number of game servers may be connected to the distribution node  610 . In some examples, a distribution node may be connected to an equal number of game servers and client devices. In some examples, a distribution node may be connected to more client devices than game servers, such as two client devices connected to a first game server and a third client device connected to a second game server. 
     The distribution node provides a connection between the first game server  606 - 1  and the first client device  602 - 1  and a connection between the second game server  606 - 2  and the second client device  602 - 2 . In some implementations, the first user inputs  616 - 1  from the first client device  602 - 1  are transmitted from the distribution node  610  to the first game server  606 - 1  and the second user inputs  616 - 2  from the second client device  602 - 2  are transmitted from the distribution node  610  to the second game server  606 - 2 . A first video information  608 - 1  relating to a first game instance of the first game application is transmitted from the first game server  606 - 1  to the distribution node  610 , and second video information  608 - 2  relating to a second game instance of the first game application or a second game application is transmitted from the second game server  606 - 2  to the distribution node  610 . 
     In some implementations, the first video information  608 - 1  and second video information  608 - 2  are received by the distribution node  610  and encoded into a multiple game instance stream that is provided in both of the encoded video information  612 - 1 ,  612 - 2 . The multiple game instance stream may include video information from game servers  606 - 1 ,  606 - 2  that are running the same game application. For example, the multiple game instance stream can be a split-screen or picture-in-picture view including the first video information  608 - 1  of a first game instance of Minecraft running on the first game server  606 - 1  and the second video information  608 - 2  of a second game instance of Minecraft running on the second game server  606 - 2 . In another example, the multiple game instance stream can be a split-screen or picture-in-picture view including the first video information  608 - 1  of a first game instance of Forza Horizon running on the first game server  606 - 1  and the second video information  608 - 2  of a second game instance of Minecraft running on the second game server  606 - 2 . In each case, both client devices  602 - 1   602 - 2  display video content for both game instances, even though the instances are different. In at least one example, the streaming node described in relation to  FIG. 5  (e.g., streaming node  530 ) can stream the multiple game instance stream to spectator devices, allowing spectators to watch two players playing different game instances, such as during a speedrunning competition. 
     In some implementations, the encoded video information  612 - 1 ,  612 - 2  provided to the client devices  602 - 1 ,  602 - 2  may include video information from each of the first game server  606 - 1  and the second game server  606 - 2  with a different proportion of the display being occupied by the video information from the game server to which the respective client device is connected. For example, both client devices  602 - 1 ,  602 - 2  may receive encoded video information  612 - 1 ,  612 - 2  including video information from both game servers, but the first client device  602 - 1  displays to a first user a fullscreen display of the video information from the first game server  606 - 1  with the video information from the second game server  606 - 2  in a picture-in-picture frame in the corner of the display of the first client device  602 - 1 . Conversely, the second client device  602 - 2  displays to a second user a fullscreen display of the video information from the second game server  606 - 2  with the video information from the first game server  606 - 1  in a picture-in-picture frame in the corner of the display of the second client device  602 - 2 . 
       FIG. 7  is a schematic illustration of a specific implementation of a system  700  according to the present disclosure. The system  700  includes a plurality of client devices  702 - 1 ,  702 - 2  communicating with a single game server  706  via a distribution node  710 . The distribution node  710  also provides a bitrate ladder  728  to a streaming node  730 , with which a plurality of spectator devices  724  are connected to view the video stream  726  of the game instance being played by the users of the client devices  702 - 1 ,  702 - 2 . 
     In some implementations, the system  700  uses more than one communication protocol to communicate between devices. In some implementations, the data connection between the game server  706  and the distribution node  710  shares a protocol. For example, in the illustrated implementation, the game server  706  and distribution node  710  use a WebSocket (HTTPS) protocol to transmit user inputs  716 - 1 ,  716 - 2  to the game server  706  and video information  708  and audio information  732  to the distribution node  710 . 
     In some implementations, each of the data connections between a client device  702 - 1 ,  702 - 2  and the distribution node  710  shares a protocol. For example, in the illustrated implementation, the first client device  702 - 1  and the distribution node  710  use a first protocol to transmit user inputs  716 - 1  to the distribution node  710  and encoded video information  712 - 1  with audio information to the first client device  702 - 1 . In another example, in the illustrated implementation, the second client device  702 - 2  and the distribution node  710  use a second protocol to transmit user inputs  716 - 2  to the distribution node  710  and encoded video information  712 - 2  with audio information to the second client device  702 - 2 . 
     In some implementations, the client devices  702 - 1 ,  702 - 2  are the same. For example, both client devices may be a general-purpose computer with an internet browser for communicating with the distribution node  710 . In some implementations, the client devices  702 - 1 ,  702 - 2  are different from one another. In an example, a first client device  702 - 1  is a native application for communicating with the distribution node  710  running on a specialized computing device, such as an Android smartphone, and a second client device is a general-purpose computer with an internet browser for communicating with the distribution node  710 . The protocols used to communicate between the client devices  702 - 1 ,  702 - 2  and the distribution node  710  may be the same or different, irrespectively of the type of client devices. 
     In some implementations, a system according to the present disclosure allows a game server to allocate more computing resources to running a game application, while communication, encoding, ML models, post-processing, or combinations thereof are offloaded to a distribution node and/or ML node of the system. Dedicating nodes to the communication, encoding, ML models, post-processing, or combinations thereof allows for greater modularity, flexibility, and ease of repair. 
     INDUSTRIAL APPLICABILITY 
     The present disclosure relates generally to systems and methods for allowing a remote user at a client device to play a game application on a game server. The game server executes the game application and transmits at least video information to a distribution node. In some implementations, the distribution node encodes the video information (to create encoded video information) from the game server for distribution over a network to the client device. In some implementations, the network is the World Wide Web. In some implementations, the network is a local area network. In some implementations, the network is a wide area network. In some implementations, the network is the World Wide Web, and the client device may be anywhere in the world relative to the game server. 
     The client device received the encoded video information from the distribution node and displays to a user a video feed of a game instance of the game application that is executed by the game server. The user may then provide one or more user inputs to the client device to interact with the game instance of the game application. The user inputs are transmitted to and received by the game server. The game server interprets the user inputs to allow the user to interact with the game application. 
     In some implementations, the game server is local to and directly connected to the distribution node. For example, the game server may be in the same datacenter as the distribution node. In some examples, the game server and distribution node may be in the same server rack in a datacenter. In some implementations, the game server is located remotely relative to the distribution node. For example, the game server may be located in a first datacenter and the distribution node may be located in a second datacenter connected to the first datacenter via a network. 
     As described herein, the game server may be a general-purpose computing device or a specialized computing device, such as a retail commodity hardware video game console. In either case, the game server includes a processor(s), such as central processing unit, graphical processing unit, physics processing unit, or other processors; a hardware storage device; and a video output device. The hardware storage device has instructions stored thereon that, when executed by the processor(s) of the game server, cause the game server to execute a game application and render a game instance of the game application. 
     In some implementations, the hardware storage device is any non-transient computer readable medium that may store instructions thereon. The hardware storage device may be any type of solid-state memory; volatile memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM); or non-volatile memory, such as read-only memory (ROM) including programmable ROM (PROM), erasable PROM (ERPOM) or EEPROM; magnetic storage media, such as magnetic tape; platen-based storage device, such as hard disk drives; optical media, such as compact discs (CD), digital video discs (DVD), Blu-ray Discs, or other optical media; removable media such as USB drives; non-removable media such as internal SATA or non-volatile memory express (NVMe) style NAND flash memory, or any other non-transient storage media. In some implementations, the hardware storage device is local to and/or integrated with the computing device. In some implementations, the hardware storage device is accessed by the computing device through a network connection. 
     The video information rendered by the game server is provided, such as by the video output device, to the distribution node. In some implementations, the distribution node includes at least a processor(s), a hardware storage device, a video connection device, and a network connection device. The hardware storage device of the distribution node has instructions stored thereon that, when executed by the processor(s) of the distribution node, cause the distribution node to execute any of the methods described herein. 
     In some implementations, a method of providing allowing a remote user at a client device to play a game application on a game server is performed at the distribution node and includes receiving video information from a game server, encoding the video information into encoded video information, distributing the encoded video information to a client device, receiving a user input from the client device, and transmitting the user input to the game server. Other implementations and variations of methods are described herein in relation to the implementations of game streaming systems. 
     The video connection device receives the video information from the game server. In some implementations, the video connection device is a physical piece of hardware, such as an HDMI-In port, or a logical software component, such as software to receive video information through a more generalized data connection, such as USB, Ethernet, wireless antenna, optical, etc. In some implementations, the game server provides additional game instance information, such as audio information, haptic information, game state data, or other information, and the distribution node is configured to receive such additional information and distribute that information in addition to the video information. The processors may encode the video information to create encoded video information, and the network connection device of the distribution node may distribute the encoded video information to the client device via the network. 
     The client device is any electronic device with network connectivity to receive and transmit information to a remotely located game server and/or distribution node. In particular examples, client devices include personal electronic devices, such as smartphones, tablet computers, desktop computers, laptop computers, hybrid computers, wearable electronic devices; television; automotive infotainment system; household appliance, or any other networked electronic device. In some implementations, the client device displays video information relating to a game instance from a game server, and the client device transmits user inputs to the game server to interact with the game application. In some implementations, the client device is a general-purpose computing device with an internet browser-based connection to the distribution node and/or game server. In some implementations, the client device executes a native application to connect to the distribution node and/or game server. 
     A user input, according to the present disclosure, should be understood to include any signal or input by any input mechanism that provides instructions to the game server to interact with and/or affect the game application. In some implementations, the user input may be provided through a human interface device (HID) of the client device, such as a keyboard, touchpad, or mouse, or through a dedicated gamepad controller or other peripheral primarily used for playing video games. 
     In some implementations, relaying user inputs through the distribution node introduces input lag to the system, compromising the responsiveness of the game application to the user inputs provided by the user at the client device. Other implementations of a system according to the present disclosure allow for user inputs to be routed directly to the game server. Direct communication of the user inputs between the client device and the game server may reduce latency in the communication between the user providing the user inputs at the client device and the user inputs being received and processed by the game server. 
     Additional processes may be used to further reduce perceived latency for the user through the use of ML models performed at the distribution node and/or on an additional node. In some implementations, a system according to the present disclosure can include a ML model that receives game state data from the game server and uses the game state data to provide additional inputs to the game server to complement the user inputs. 
     Game state data includes any information that may allow a second electronic device to recreate a given game state. For example, the game state data of a first game instance running on a first electronic device may be provided to second copy of a game application running on a second electronic device, and the second electronic device may render a duplicate of the first game instance based on the game state data. In some implementations, game state data includes virtual object or avatar positions, movement, player character statistics or characteristics, player character inventory, player character status, ability cooldown status, non-player character status, or any other information about the game state of the first instance. In some implementations, the game state data is used to synchronize two instances of a game application to render the same game environment to two different players. In some implementations, the game state data is used to present the game state of the first instance asynchronously to a second instance to allow the same or different player to experience the game state of the first instance at a later time. 
     In some implementations, the game state data is provided to a ML model, such as that used by the ML node, and the ML model can identify the current status or situation of the game application. For example, the ML model may receive game state data that includes data regarding the game application, the game instance, the player character, and the game environment, which allows the ML model to make predictions about what the user may intend to do next in the game. In a particular example, the ML model may identify that the game application is a driving game, and the user is racing against computer-controlled avatars. Due to latency in the network connection, the user inputs  316  may be delayed from the client device or packets of video information may be delayed en route to the client device that compromise the user&#39;s ability to react to the current game state. The ML model may identify the current game state from the game state data and anticipate that the user will steer their car to follow the road. The ML node may provide ML inputs to supplement the user inputs that help steer the user&#39;s car in the absence or delay of user inputs. 
     The distribution node receives, in some implementations, both the user input from the client device and the ML input from the ML node. The distribution node, in some implementations, merges the user inputs with the ML inputs and transmits both the user inputs and the ML inputs to the game server. In some examples, the predicted ML input and the actual user input are the same, and merging the inputs has no noticeable effect. In other examples, the user input is absent at the distribution node due to latency and/or network packet drops, and the ML input may fill in the gaps in the user inputs with the predicted inputs. In a particular example, the user provides a continuous right directional input to steer the car to the right around a corner. The ML input predicts a right input for the corner and predicts that a user input will continue to be a right directional input throughout the corner in the game environment. If a network connection issue causes a break in the user input, the distribution node may use the ML input to supplement the user input to transmit a continuous right-direction input to the game server. 
     In yet other examples, the distribution node receives different user input and a ML input. In such examples, the distribution node may prioritize the transmission of the user input and ignore the ML input, whereby the game server receives the user input as the user would expect. For example, while the ML model may predict that user steer to the right to follow the corner of the previous example, a user may instead continue in a straight line to pursue a shortcut or intentionally drive off the road. 
     In some implementations, the ML model may receive the user inputs from the client device and/or the distribution node to refine the ML model. For example, a system where the ML node is located between the game server and the distribution node may modify and/or supplement data transmitted between the game server and the distribution node. 
     In some examples, the ML node may receive game state data, as described herein, to recreate the game environment of the game instance running on the game server. In some implementations, the ML node also receives user inputs from the client device. In some examples, the user inputs are received through the distribution node, while in other examples, the user inputs are received directly from the client device (without traveling through the distribution node) to reduce latency. The ML node can use a ML model to evaluate the game instance represented by the game state data and compare a predicted ML input to the received user input. In some implementations, the ML node transmits to the game server the user input and/or the ML input based on the comparison of the user input and the ML input. 
     In some implementations, the ML node refines the ML model based on a comparison of the ML input and the user input. In the driving game example described herein, the ML input supplemented the user input to predict that the user would steer to remain on the road based on aggregated community statistics. In other words, most players that race on that track follow the road. In some implementations, the ML model may be refined using user-specific data to customize the ML inputs to predict what that particular user may input. For example, a particular user may be aggressive at cutting the inside curb of corners and allowing their car to partially leave the road. The user-specific data based on the received user input may be compared to the predicted input from the generalized ML model and the generalized ML model may be refined to provide ML inputs that more accurately predict the inputs of the particular user in the event of user inputs being lost or delayed by network irregularities. 
     In some implementations, a ML node provides benefits to the video information output by the game server. For example, the game server may output a first video information that is received and processed by the ML node. The ML node may apply one or more post-processing operations to the first video information to create a second video information that is transmitted to the distribution node for distribution to a client device. In some implementations, the post-processing includes resolution up-scaling, such as upscaling from native 1080p resolutions to  4 K resolutions. In some implementations, the post-processing includes framerate interpolation, such as interpolating 60 frames per second to 120 frames per second. In some implementations, the post-processing includes improvement to the dynamic range of the video information, such as changes from standard dynamic range (SDR) to high dynamic range (HDR). In some implementations, the post-processing includes additional shading and/or anti-aliasing of the video information. 
     In some implementations, additional information can be added to and/or improved by the ML node. For example, the ML node may use the video information and/or the game state data to determine haptic information, such as rumble feedback or adaptive resistance in an input device, that may be transmitted to the client device for presentation to a user through an input device. In at least one example, a legacy or backward compatible game application may not support all types of haptic feedback available to the user on input device(s) of the client device. The ML node may analyze the information and/or the game state data to determine haptic information and add that haptic information to the second video information or other information to be distributed to the client device. 
     In some implementations, audio information may be improved or added for distribution to the client device. For example, a game application may not support surround sound or 3D audio natively in the game application. The ML node may analyze the game state data and determine or add directionality of audio sources to simulate surround sound or 3D audio at the client device. 
     While the post-processing operations (video, audio, haptic, etc.) and ML input calculations are described in relation to the ML node being discrete from the distribution node, in some implementations, the ML node is integrated into the distribution node. In at least one example, the ML model and the video encoding may both be performed at the distribution node before the encoded video information (and, optionally, audio information or haptic information) is distributed to the client device and/or the inputs (user inputs and/or ML inputs) are transmitted to the game server. 
     In at least some implementations, a system allows a viewing user at a spectator device to view a video stream (i.e., non-interactive viewing) of the game instance played by a playing user at a client device. The game server executes the game application to create a game instance, at least the video information of which is provided to the distribution node for encoding. The encoded video information is encoded based on the network connection specifically with the client device and is distributed to the client device. Additionally, the video information is encoded at a plurality of bitrates in a bitrate ladder and provided to a streaming node. 
     Bitrate is a measure of bandwidth consumed by the video stream. The bitrate is affected by the resolution, the frames per second, the color depth, the dynamic range, etc. By providing encoding the video stream at a plurality of bitrates, an adaptive bitrate ladder can provide the best quality video stream to a spectator device based on the connection quality (i.e., speed) between the spectator device and the streaming node. While the streaming node is illustrated as a discrete component of the system, it should be understood that, in some implementations, the distribution node is integrated with the streaming node, and the distribution node distributes both the encoded video information to the client device for interactive play with the game application, as well as the video stream with an adaptive bitrate ladder to the spectator device. 
     In some implementations, the streaming node provides one or more social features to the spectator device in addition to the video stream. For example, while the spectating device cannot interact with the game instance to which the client device is interactively connected, the streaming node may provide a text or voice chat function for a viewing user at the spectator device to communicate with the playing user at the client device. In some examples, a plurality of spectator devices views the video stream from the streaming node, and the streaming node allows the spectator devices to communicate with one another. 
     A system according to the present disclosure allows for multiple client devices to connect to a single distribution node. In some implementations, a first client device and a second client device are connected to the same distribution node to allow both client devices to receive video information from a first game server and send user inputs to the first game server. For example, the first client device and second client device can play a single instance of a game application together, either cooperatively or competitively, depending on the game application. In such an implementation, the first client device and second client device each receive the same encoded video information from the distribution node, such that both players see the same frames from the first game server as if the players were playing “couch co-op” and both of the first user input and second user input affect the shared game instance. A shared game instance is a single game instance of a game application executed by the game server, and a plurality of client devices are connected to the shared game instance. Each client device of the plurality of client devices is able to interact with the game instance on the game server, and each client device of the plurality of client devices receives the same video information related to the shared game instance. 
     In some implementations, the first client device and second client device receive the same video information (i.e., see the same frames and/or images provided from the game server) but the distribution node encodes the video information differently. For example, the video information may be encoded at the distribution node based on the different network connections of each of the client devices. While the video information is the same, the distribution node may distribute a first encoded video information to the first client device based on the particular network connection of the first client device with the distribution node and distribute a second encoded video information to the second client device based on the particular network connection of the second client device with the distribution node. 
     In some implementations, the first client device and second client device are connected to the same distribution node, but the distribution node is connected to both a first game server and a second game server. The distribution node provides a connection between the first game server and the first client device and a connection between the second game server and the second client device. In some implementations, the first user inputs from the first client device are transmitted from the distribution node to the first game server and the second user inputs from the second client device are transmitted from the distribution node to the second game server. A first video information relating to a first game instance of the first game application is transmitted from the first game server to the distribution node, and second video information relating to a second game instance of the first game application or a second game application is transmitted from the second game server to the distribution node. 
     In some implementations, the first video information and second video information are received by the distribution node and encoded into a multiple game instance stream that is provided in both of the encoded video information. The multiple game instance stream may include video information from game servers that are running the same game application. For example, the multiple game instance stream can be a split-screen or picture-in-picture view including the first video information of a first game instance of Minecraft running on the first game server and the second video information of a second game instance of Minecraft running on the second game server. In another example, the multiple game instance stream can be a split-screen or picture-in-picture view including the first video information of a first game instance of Forza Horizon running on the first game server and the second video information of a second game instance of Minecraft running on the second game server. In each case, both client devices display video content for both game instances, even though the instances are different. In at least one example, the streaming node can stream the multiple game instance stream to spectator devices, allowing spectators to watch two players playing different game instances, such as during a speedrunning competition. 
     In some implementations, the encoded video information provided to the client devices may include video information from each of the first game server and the second game server with a different proportion of the display being occupied by the video information from the game server to which the respective client device is connected. For example, both client devices may receive encoded video information including video information from both game servers, but the first client device displays to a first user a fullscreen display of the video information from the first game server with the video information from the second game server in a picture-in-picture frame in the corner of the display of the first client device. Conversely, the second client device displays to a second user a fullscreen display of the video information from the second game server with the video information from the first game server in a picture-in-picture frame in the corner of the display of the second client device. 
     The present disclosure relates to systems and methods for allowing remote play of a game application to a user on a client device according to at least the examples provided in the sections below: 
     (A1) In some implementations, a distribution node for allowing interaction with interactive applications to a remote user includes a video connection device, a network connection device, a processor, and a hardware storage device. The video connection device is configured to connect to a game server and receive at least video information from the game server. The network connection device is configured to connect to a network. The hardware storage device has instructions stored that, when executed by the processor, cause the distribution node to encode the video information from the game server to encoded video information and distribute the encoded video information over the network to a client device. 
     (A2) In some implementations, the distribution node of (A1) can receive user inputs from the client device and transmit those user inputs to the game server to allow a user at the client device to interact with a game application running on the game server. 
     (A3) In some implementations, the distribution node of (A1) can encode the video information to a video stream for distribution to a spectator device. The video stream may be encoded in a bitrate ladder to facilitate transmission at different bitrates depending on network conditions. 
     (A4) In some implementations, the distribution node of (A1) can perform one or more machine learning models based upon user inputs to provide ML inputs to the game server. 
     (A5) In some implementations, the distribution node of (A1) can perform one or more post-processing operations on the video information to improve the visual fidelity or appearance. 
     (A6) In some implementations, a system for allowing interaction with game applications to a remote user includes a game server and a distribution node. The game server is configured to run a game application and generate at least video information. The distribution node is connected to the game server and configured to receive the video information and encode the video information for distribution via a network. 
     (A7) In some implementations, the system of (A6) includes a first client device in data communication with the distribution node. (A8) The system of (A7) can include a second client device in data communication with the distribution node. (A9) The first client device can be in data communication with the game server and the second device can be in data communication with the game server such that a first user input of the first client device and a second user input of the second client device are transmitted to the game server to interact with a shared instance of the game application. The system of (A9) can, thereby, allow couch co-op for players on different client devices. 
     (A10) In some implementations, the system of (A8) includes a second game server in data communication with the distribution node so that the second client device is in data communication with the second game server through the distribution node. A first user input of the first client device is transmitted to the first game server and a second user input of the second client device is transmitted to the second game server. 
     (A11) In some implementations, the distribution node of (A8) distributes a multiple instance video stream to the first client device and the second client device that includes video information from both the first game server and the second game server. 
     (A12) In some implementations, a method of providing interaction with a game instance of a game application by a remote user includes, at a distribution node, receiving video information from a game server, encoding the video information into encoded video information, distributing the encoded video information to a client device, receiving a user input from the client device, and transmitting the user input to the game server. 
     (A13) The distribution node and the game server may share a communication protocol. 
     (A14) The distribution node and the client device may share a communication protocol. 
     (A15) The method of (A12) may include generating a video stream with a bitrate ladder and making the video stream available for streaming. 
     The articles “a,” “an,” and “the” are intended to mean that there are one or more of the elements in the preceding descriptions. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to “one implementation” or “an implementation” of the present disclosure are not intended to be interpreted as excluding the existence of additional implementations that also incorporate the recited features. For example, any element described in relation to an implementation herein may be combinable with any element of any other implementation described herein. Numbers, percentages, ratios, or other values stated herein are intended to include that value, and also other values that are “about” or “approximately” the stated value, as would be appreciated by one of ordinary skill in the art encompassed by implementations of the present disclosure. A stated value should therefore be interpreted broadly enough to encompass values that are at least close enough to the stated value to perform a desired function or achieve a desired result. The stated values include at least the variation to be expected in a suitable manufacturing or production process, and may include values that are within 5%, within 1%, within 0.1%, or within 0.01% of a stated value. 
     A person having ordinary skill in the art should realize in view of the present disclosure that equivalent constructions do not depart from the scope of the present disclosure, and that various changes, substitutions, and alterations may be made to implementations disclosed herein without departing from the scope of the present disclosure. Equivalent constructions, including functional “means-plus-function” clauses are intended to cover the structures described herein as performing the recited function, including both structural equivalents that operate in the same manner, and equivalent structures that provide the same function. It is the express intention of the applicant not to invoke means-plus-function or other functional claiming for any claim except for those in which the words ‘means for’ appear together with an associated function. Each addition, deletion, and modification to the implementations that falls within the meaning and scope of the claims is to be embraced by the claims. 
     It should be understood that any directions or reference frames in the preceding description are merely relative directions or movements. For example, any references to “front” and “back” or “top” and “bottom” or “left” and “right” are merely descriptive of the relative position or movement of the related elements. 
     The present disclosure may be embodied in other specific forms without departing from its characteristics. The described implementations are to be considered as illustrative and not restrictive. The scope of the disclosure is, therefore, indicated by the appended claims rather than by the foregoing description. Changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.