Minimizing network latency in interactive internet applications

A method and system that enhances a user's performance while interacting with an interactive internet application such as a Massively Multiplayer Online (MMO) game is provided. The network latency experienced by users participating in the MMO game is minimized by dynamically determining an optimal transmission action for a message generated by the MMO game. In one embodiment, determining the optimal transmission action for a message includes dynamically determining the optimal number of redundant Forward Error Correction (FEC) packets to add to a message prior to transmitting a message to a receiving device. The optimal number of FEC packets is determined based on a wide range of varying network conditions.

BACKGROUND OF THE INVENTION

Massively Multiplayer Online (MMO) games are multiplayer games that enable large numbers of players to participate in game play simultaneously in a real-time shared virtual world that they are connected to, via the Internet. To support a virtual world, MMO's typically utilize one or more servers, where players connect to the servers to participate in game play. Due to their interactive nature, MMO games typically have stringent latency requirements. However, a user's perceived performance while participating in a MMO game may typically be affected by latency, delay variation, and packet loss in the Internet. In addition, users in different geographical locations may experience different network latencies while participating in a MMO game. This may result in each user interacting with the MMO game at different points in time, thereby affecting the user's overall interactivity with the MMO game.

SUMMARY

Disclosed herein is a method and system that enhances a user's performance while interacting with a MMO game by minimizing network latency. In one embodiment, the disclosed technology minimizes the network latency experienced by users by determining an optimal transmission policy for a message prior to and during transmission of the message to a receiving device. The optimal transmission policy determines the optimal number of FEC packets to be added to the message prior to and during transmission of the message to the receiving device. The optimal policy is determined by dynamically adapting to a wide range of varying network conditions. The optimal number of FEC packets is determined based on a wide range of varying network conditions.

In one embodiment, the disclosed technology implements a message-oriented, connectionless User Datagram Protocol (UDP)-based transport protocol that provides the MMO game with a variety of transport layer services. The UDP-based transport protocol includes an intelligent adaptive decision making engine. The intelligent adaptive decision making engine dynamically determines the optimal number of FEC packets to be added to a message generated by the MMO game prior to and during transmission of the message to a receiving device, while minimizing a transmission cost and message delivery latency associated with transmitting the message to the receiving device.

In another embodiment, a method for balancing a transmission cost and network latency associated with transmitting a message generated by an interactive internet application is disclosed. The method includes obtaining one or more network performance parameters associated with a message generated by an interactive internet application. The method further includes determining an optimal transmission action for the message based on the network performance parameters and the number of information packets to be transmitted in the message. An optimal transmission action is determined that minimizes a transmission cost and network latency associated with transmitting the message to a receiving device, subject to a constraint that a probability of arrival of the message after some network latency tolerance is below a certain threshold value.

DETAILED DESCRIPTION

In order to minimize network latency experienced by users, an optimal transmission strategy is determined for a message generated by an interactive internet application, such as a MMO game. In one embodiment, determining the optimal transmission strategy comprises dynamically determining the optimal number of redundant Forward Error Correction (FEC) packets to add to a message prior to and during transmission of the message to a receiving device. The disclosed technology adapts to a wide range of varying network conditions such as the network latency, round trip time and packet loss probability to dynamically determine the optimal transmission action to perform for a message that is to be transmitted. In one embodiment, a Markov Decision Process (MDP) framework is utilized to determine the optimal transmission action for a message. In one approach, the optimal transmission action for a message is pre-computed offline and stored as a look-up table. The optimal transmission action for a message is dynamically chosen during online execution by utilizing the appropriate look-up table based on the current state of the network and the number of information packets in the message that need to be transmitted to the receiving device.

FIG. 1Aillustrates an exemplary environment for implementing the disclosed technology.FIG. 1Aillustrates two users18,19at client devices10A,10B participating in a massively multiplayer online (MMO) game with each other by connecting to a centralized data server11. Centralized data server11executes a sever portion of the MMO game8so as to provide the MMO game8to client devices10A,10B. It should be noted that althoughFIG. 1Ashows two client devices10A,10B interacting with one centralized data server11, the present technology may be utilized to support any number of client devices and any number of servers in other embodiments.

FIG. 1Adescribes an exemplary operation of an MMO game. In a typical operation, centralized data server11sends a multiple choice question to users18,19at a time, t0. However, due to network latency, latency variation and packet loss, users18,19may receive the question at different times. To ensure that the game is fair to all users, server11may reveal the question to users18,19at the same time, based on a single timestamp from a global virtual clock (for example, client devices10A and10B may be synchronized with the server11on the same virtual clock). Accordingly, a user closer to the centralized data server11will not see the question earlier and thus have an unfair advantage. Therefore, at the moment the question is created, centralized data server11may determine how far into the future the question is to be revealed to users18,19. Intuitively, the question-reveal delay, (t1−t0), shown inFIG. 1Ashould be large enough so that users18,19receive the question by the reveal time, t1. After the question is revealed, each of the users18,19selects an answer and submits it to the centralized data server11after a certain deadline, t2. Centralized data server11collects the answers from users18,19before it aggregates all the results and announces the winner at time t3. To keep the game exciting and engaging, it is important that winners be announced shortly after the answer deadline. At the same time, the submit-announce delay, (t3−t2), shown inFIG. 1Aneeds to be long enough so that each of the users18,19can get their results in. In both phases (revealing the question and announcing the result), if the delay is too small, due to network latency and packet loss, users18,19may not see the question on time, or will not be able to get their answers to the centralized data server11before the deadline, thus making the game unfair. On the other hand, if the network latency is too large, the progress of the MMO game6,8is slowed, thereby affecting the interactivity of the game.

The disclosed technology enhances a user's performance while interacting with the MMO game by minimizing the maximum latency experienced by the users. In one embodiment, the disclosed technology minimizes the number of original data packet re-transmissions while transmitting messages between the client devices and the centralized server by dynamically determining the optimal number of redundant Forward Error Correction (FEC) packets to add to a message prior to and during transmission of the message to a receiving device to minimize the latency experienced by all users participating in the game. The operations performed by the disclosed technology are discussed in detail below.

FIG. 1Billustrates a system for implementing the present technology. Client devices100A,100B . . .100X communicate with a centralized data server110over an underlying network50. The network50may comprise the Internet, although other networks such as a LAN or WAN are contemplated. Client devices100A,100B . . .100X may include a gaming and media console, a personal computer, or one or more mobile devices such as, for example, a cell phone, a Internet-enabled smart phone, a personal digital assistant, a palmtop computer, a laptop computer, tablet computing device, smart appliance, etc.

In one embodiment, client devices100A,100B . . .100X and the centralized data server110may operate within a layered network communications framework130. The framework130enables client devices100A,100B . . .100X and the centralized data server110to receive and transmit information between each other over the underlying network50and provides a collection of services that applications106,108running on client devices100A,100B . . .100X and the centralized data server110may be invoked and utilized. As illustrated, applications106,108may be hosted in an upper-most layer of the layered network communications framework130on each of the client devices100A,100B . . .100X and the centralized data server110. In one embodiment, applications106,108generate messages that include a sequence of one or more information packets which can either be communicated to another local application hosted on the same device, or can be communicated over the network50to a remote application108hosted on the centralized data server110. In one example, applications106,108may comprise an interactive internet application such as a massively multiplayer online (MMO) game.

In one embodiment, a message-oriented, connectionless User Datagram Protocol (UDP)-based transport protocol112is implemented in a layer immediately beneath applications106,108on each of the client devices100A,100B . . .100X and the centralized data server110, as shown inFIG. 1B. The UDP-based transport protocol112provides applications106,108with a variety of transport layer services which enable applications106,108to communicate with each other and as necessary over the network50. In one embodiment, the disclosed UDP-based transport protocol112comprises an intelligent adaptive decision making engine114.FIG. 7, discussed below, describes an exemplary architecture of the disclosed message-oriented, connectionless UDP-based protocol that comprises an intelligent adaptive decision making engine. In one embodiment, the intelligent adaptive decision making engine114minimizes the maximum latency experienced by users by determining an optimal transmission action for a message transmitted between the client devices100A,100B . . .100X and the centralized data server110. In one embodiment, determining an optimal transmission action comprises dynamically determining the optimal number of redundant Forward Error Correction (FEC) packets to add to the message prior to and during transmission of the message to a receiving device. In addition, the disclosed intelligent adaptive decision making engine114adapts to a wide range of varying network conditions to dynamically determine the optimal number of FEC packets to add to a message. The operations performed by the intelligent adaptive decision making engine114are described below in detail with respect toFIGS. 4-8.

A suite of internetworking protocols116operate in a layer immediately beneath the UDP-based transport protocol112and a suite of link protocols118operate in a layer immediately beneath the internetworking protocols on each of the client devices100A,100B . . .100X and the centralized data server110. The internetworking protocols116and link protocols118together provide a variety of lower-level network communication services which facilitate the actual transmission of information packets between the client devices100A,100B . . .100X and the centralized data server110.

FIG. 2illustrates an example of a computing device architecture104that may be used to implement the client devices100A,100B . . .100X and/or the centralized data server110shown inFIG. 1B. In one embodiment, the computing device104ofFIG. 2may be a multimedia console102, such as a gaming console. As shown inFIG. 2, the multimedia console102has a central processing unit (CPU)200, and a memory controller202that facilitates processor access to various types of memory, including a flash Read Only Memory (ROM)204, a Random Access Memory (RAM)206, a hard disk drive208, and portable media drive107. In one implementation, CPU200includes a level 1 cache210and a level 2 cache212, to temporarily store data and hence reduce the number of memory access cycles made to the hard drive208, thereby improving processing speed and throughput.

CPU200, memory controller202, and various memory devices are interconnected via one or more buses (not shown). The details of the bus that is used in this implementation are not particularly relevant to understanding the subject matter of interest being discussed herein. However, it will be understood that such a bus might include one or more of serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus, using any of a variety of bus architectures. By way of example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus also known as a Mezzanine bus.

In one implementation, CPU200, memory controller202, ROM204, and RAM206are integrated onto a common module214. In this implementation, ROM204is configured as a flash ROM that is connected to memory controller202via a PCI bus and a ROM bus (neither of which are shown). RAM206is configured as multiple Double Data Rate Synchronous Dynamic RAM (DDR SDRAM) modules that are independently controlled by memory controller202via separate buses (not shown). Hard disk drive208and portable media drive107are shown connected to the memory controller202via the PCI bus and an AT Attachment (ATA) bus216. However, in other implementations, dedicated data bus structures of different types can also be applied in the alternative.

A graphics processing unit220and a video encoder222form a video processing pipeline for high speed and high resolution (e.g., High Definition) graphics processing. Data are carried from graphics processing unit220to video encoder222via a digital video bus (not shown). An audio processing unit224and an audio codec (coder/decoder)226form a corresponding audio processing pipeline for multi-channel audio processing of various digital audio formats. Audio data are carried between audio processing unit224and audio codec226via a communication link (not shown). The video and audio processing pipelines output data to an A/V (audio/video) port228for transmission to a television or other display. In the illustrated implementation, video and audio processing components220-228are mounted on module214.

FIG. 2shows module214including a USB host controller230and a network interface232. USB host controller230is shown in communication with CPU200and memory controller202via a bus (e.g., PCI bus) and serves as host for peripheral controllers104(1)-104(4). Network interface232provides access to a network (e.g., Internet, home network, etc.) and may be any of a wide variety of various wire or wireless interface components including an Ethernet card, a modem, a wireless access card, a Bluetooth module, a cable modem, and the like.

In the implementation depicted inFIG. 2, console102includes a controller support subassembly240for supporting four controllers104(1)-104(4). The controller support subassembly240includes any hardware and software components needed to support wired and wireless operation with an external control device, such as for example, a media and game controller. A front panel I/O subassembly242supports the multiple functionalities of power button115, the eject button117, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of console102. Subassemblies240and242are in communication with module214via one or more cable assemblies244. In other implementations, console102can include additional controller subassemblies. The illustrated implementation also shows an optical I/O interface235that is configured to send and receive signals that can be communicated to module214.

MUs140(1) and140(2) are illustrated as being connectable to MU ports “A”130(1) and “B”130(2) respectively. Additional MUs (e.g., MUs140(3)-140(6)) are illustrated as being connectable to controllers104(1) and104(3), i.e., two MUs for each controller. Controllers104(2) and104(4) can also be configured to receive MUs (not shown). Each MU140offers additional storage on which games, game parameters, and other data may be stored. In some implementations, the other data can include any of a digital game component, an executable gaming application, an instruction set for expanding a gaming application, and a media file. When inserted into console102or a controller, MU140can be accessed by memory controller202. A system power supply module250provides power to the components of gaming system100. A fan252cools the circuitry within console102.

An application260comprising machine instructions is stored on hard disk drive208. When console102is powered on, various portions of application260are loaded into RAM206, and/or caches210and212, for execution on CPU200, wherein application260is one such example. Various applications can be stored on hard disk drive208for execution on CPU200.

Gaming and media system104may be operated as a standalone system by simply connecting the system to monitor150(FIG. 1), a television, a video projector, or other display device. In this standalone mode, gaming and media system104enables one or more players to play games, or enjoy digital media, e.g., by watching movies, or listening to music. However, with the integration of broadband connectivity made available through network interface232, gaming and media system104may further be operated as a participant in a larger network gaming community.

FIG. 3illustrates a general purpose computing device architecture which can be used to implement another embodiment of client devices100A,100B . . .100X and the centralized data server110shown inFIG. 1B. With reference toFIG. 3, an exemplary system for implementing embodiments of the disclosed technology includes a general purpose computing device in the form of a computer310. Components of computer310may include, but are not limited to, a processing unit320, a system memory330, and a system bus321that couples various system components including the system memory to the processing unit320. The system bus321may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

The system memory330includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)331and random access memory (RAM)332. A basic input/output system333(BIOS), containing the basic routines that help to transfer information between elements within computer310, such as during start-up, is typically stored in ROM331. RAM332typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit320. By way of example, and not limitation,FIG. 4illustrates operating system334, application programs335, other program modules336, and program data337.

The computer310may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 4illustrates a hard disk drive340that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive351that reads from or writes to a removable, nonvolatile magnetic disk352, and an optical disk drive355that reads from or writes to a removable, nonvolatile optical disk356such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive341is typically connected to the system bus321through a non-removable memory interface such as interface340, and magnetic disk drive351and optical disk drive355are typically connected to the system bus321by a removable memory interface, such as interface350.

The drives and their associated computer storage media discussed above and illustrated inFIG. 3, provide storage of computer readable instructions, data structures, program modules and other data for the computer310. InFIG. 3, for example, hard disk drive341is illustrated as storing operating system344, application programs345, other program modules346, and program data347. Note that these components can either be the same as or different from operating system334, application programs335, other program modules336, and program data337. Operating system344, application programs345, other program modules346, and program data347are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer20through input devices such as a keyboard362and pointing device361, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit320through a user input interface360that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor391or other type of display device is also connected to the system bus321via an interface, such as a video interface390. In addition to the monitor, computers may also include other peripheral output devices such as speakers397and printer396, which may be connected through an output peripheral interface390.

The computer310may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer380. The remote computer380may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer310, although only a memory storage device381has been illustrated inFIG. 3. The logical connections depicted inFIG. 3include a local area network (LAN)371and a wide area network (WAN)373, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer310is connected to the LAN371through a network interface or adapter370. When used in a WAN networking environment, the computer310typically includes a modem372or other means for establishing communications over the WAN373, such as the Internet. The modem372, which may be internal or external, may be connected to the system bus321via the user input interface360, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer310, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 3illustrates remote application programs385as residing on memory device381. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

The hardware devices ofFIGS. 1-4discussed above can be used to implement a system that determines an optimal transmission action for a message transmitted between one or more client devices100A,100B . . .100X and the centralized data server110. In one embodiment, and as discussed above, the optimal transmission action determines the optimal number of FEC packets to add to a message prior to and during transmission of the message between the client devices100A,100B . . .100X and the centralized data server110.

FIG. 4Ais a flowchart describing a process for determining an optimal transmission strategy for a message generated by an interactive internet application. In one embodiment, the steps ofFIG. 4Amay be performed by the intelligent adaptive decision engine114shown inFIG. 1B. In step400, an input message consisting of one or more information packets to be transmitted to a receiving device is received. In one example, the input message may be generated by a user participating in an interactive internet application such as the MMO game shown inFIG. 1A. In step402, an optimal transmission action for the input message is dynamically and automatically determined based on monitoring one or more network performance parameters in real-time and based on the number of information packets in the message. The optimal transmission action includes sending all the information packets in the original message and an optimal number of forward error correction (FEC) packets. In one embodiment, the optimal transmission action is defined as minimizing the transmission cost subject to a constraint on the network latency or network reliability. Other definitions of optimality may be defined, in other embodiments.FIG. 4Bdescribes the process of performing step402in detail. In step404, an optimal number of redundant forward error correction (FEC) packets to be added to the input message prior to and during transmission of the input message to the receiving device are determined, resulting in the optimal transmission action. The optimal transmission action is determined prior to transmitting the message, given the network characteristics. Given the optimal transmission action, the optimal number of FEC packets to be added to the message during the first transmission stage of the message, is a deterministic value. The optimal number of FEC packets to be added during later transmission stages, is however not deterministic and may be determined at the beginning of each stage based on the actual packet losses that occurred during previous transmission stages.

In step406, the FEC packets are generated. In step408, an output message consisting of the information packets in the original message and the optimal number of FEC packets is created. In step410, the output message is transmitted to the receiving device. In step412, a reply message is received from the receiving device. The reply message acknowledges the number of packets received. The actual number of packets received by the receiving device is a function of the network characteristics. Depending on how many packets from the message have been received, the sending device may have to resend some of the packets. Therefore, the sending device again determines an optimal transmission action which consists of sending at least the number of missing packets in the message and an optimal number of FEC packets. This process is repeated until the entire message is sent or until some network latency tolerance is reached.

FIG. 4Bis a flowchart describing a process for dynamically determining an optimal transmission action for a message transmitted between one or more client devices and a centralized data server.FIG. 4Bdescribes a process of performing step402inFIG. 4A.

In step430, one or more network performance parameters are obtained. In one example, the network performance parameters include the round trip time (RTT), the network latency and packet loss probability of a message transmitted between the client devices100A,100B . . .100X and the centralized data server110. Alternatively, these parameters may be determined dynamically during each session using prior data transmissions to learn them. In one example, the RTT, network latency and packet loss probability may be obtained by analyzing packet level traces from prior MMO game sessions executed in the client devices100A,100B . . .100X and the centralized data server110. As used herein, the network latency refers to the interval of time between when a message is transmitted from one or more of the client devices100A,100B . . .100X over the network50to when the message is successfully received over the network50by the centralized data server110, or vice-versa. The term round-trip time as used herein refers to the latency time plus the time it takes for the one or more client devices100A,100B . . .100X to receive an acknowledgement over the network50from the centralized data server110that it has successfully received a particular message, or vice versa.

In step432, the latency tolerance of the input message is determined. In one embodiment, the latency tolerance is defined as a threshold time T of arrival of a percentage of messages. In one embodiment, it is desirable that the percentage of messages arriving later than the threshold time T is less than a very small target value, denoted as ε(T). The threshold time T may be pre-determined by the client devices and the centralized data server, in one embodiment. In one example, it is desirable that 99.9% of the messages arrive prior to the threshold time T.

In step434, the latency tolerance T is divided into one or more transmission stages based on the RTT for the message. In one embodiment, the number of transmission stages of a message is determined as the ratio of the RTT and the latency tolerance. For example, the latency tolerance T of a message can be divided into 3 transmission stages for the message if the network latency tolerance T is set to be 750 ms and the round trip time is set to be 250 ms, in one embodiment.

In step436, an optimal transmission action for each of the possible states (i, q) of the input message are determined. Here, q denotes the number of information packets that need to be transmitted during a transmission stage i. For example, suppose a message originally consists of k=4 information packets and the number of transmission stages i determined for the message is 3, then there are q=4 information packets to be transmitted during transmission stage i=1, and the state of the message (i, q)=(1, 4). In one embodiment, the different states of a message may be identified and represented in a Markov Decision Process (MDP) framework. As will be appreciated, a MDP framework is a probabilistic model of a sequential decision problem, where at each time step, the process is in an initial state and a current state and action selected by a decision maker determines the probability distribution of a set of future states.

In one embodiment, the optimal transmission action corresponds to a path leaving the input message, at a state (i, q) of the message. As will be appreciated, the optimal transmission action at each state (i, q) is determined by a policy, π, in a MDP framework. A mapping from states to actions in a MDP framework is referred to as a policy, which may be denoted as π={π(i, q)}. For a given policy π, the transmission action at each state (i, q) is deterministic and defined by the policy itself. In one embodiment, the optimal transmission action for the input message at each identified state (i, q) of the input message is determined as described in steps (440-446) below.

In step440, the packet loss probability p of each path leaving each of the identified states (i, q) of the input message is calculated. In one example, the packet loss probability p of each path is calculated as the compound probability of all loss patterns along the path.FIG. 5illustrates an exemplary MDP framework that represents different possibilities of loss patterns that result in different states of a message. In the exemplary illustration shown inFIG. 5, the latency tolerance T of the input message is divided into 3 transmission stages. The message originally consists of 4 information packets and the initial state of the message is denoted by (i, q)=(1, 4). R:0, R:1, R:2, R:3 and R>=4 represent the different possibilities of loss patterns for the message at the initial state (1,4) at the end of the first transmission stage, i=1RTT. For example, R:0 denotes a loss pattern in which none of the 4 information packets of the message are received by the end of the first transmission stage. Similarly, R:2 denotes a loss pattern in which 2 of the 4 information packets of the message are received by the end of the first transmission stage, and so on. Similarly, and as illustrated inFIG. 5, each of the different loss patterns R:0, R:1, R:2, R:3 and R>=4 may result in one or more different states, (2, 4), (2, 3), (2, 2), (2, 1) and (2, 0) for the message, where each state may further lead to a different set of loss patterns for the message, and so on.

In one example, the compound probability of all loss patterns is computed by aggregating the loss patterns along the path that end in non-zero states in the final transmission stage ‘i’ of the input message. As used herein, a non-zero state for a message denotes that the message has not been delivered completely within the latency tolerance T. For example, the states (1) and (2) at the end of transmission stage, 3RTT in the exemplary MDP framework shown inFIG. 5illustrate non-zero states for the input message. The probability that the input message cannot be delivered within the latency tolerance, T is then obtained based on the aggregation. This probability is equivalent to a fraction of messages arriving after T, which may be denoted as επ.

In step442, the transmission cost of delivery of the input message along each path is calculated as the total number of packets transmitted along the path. In step444, the aggregated cost for transmission along each path is determined. Aggregating the costs on all the paths weighted by their probabilities, the average cost of delivery of the input message is obtained, which is denoted as ρπ.

In step446, the aggregated cost is compared across each path leaving the state (i, q) to determine the optimal transmission action for the state. In step446, the optimal transmission action for the message state (i, q) is determined by determining the optimal transmission policy π. In one embodiment, the optimal policy π minimizes the transmission cost and network latency associated with transmitting the input message, while ensuring that the probability of the message arriving later than the threshold T is below the threshold network latency. In particular, the optimal transmission action may be formulated as an optimization problem as shown in equation (1) below:

The optimal transmission action shown in equation (1) determines the optimal number of FEC packets to be added to the input message by minimizing the average number of FEC packets sent per message to each individual client device subject to the constraint that the fraction of messages that do not arrive within T seconds is less than the target value ε(T). As discussed above, the disclosed technology adapts to a wide range of varying network conditions by obtaining one or more network performance parameters such as the packet loss probability, network latency and RTT to dynamically determine the optimal transmission action for a message. In one embodiment, the disclosed technology may also dynamically adapt to the geographical location of the client devices to determine an optimal transmission action for a message. For example, the disclosed technology may revert to a pure timeout-based retransmission for client devices that are geographically close to the centralized data server to determine the optimal transmission action for a message. In one example, the optimal transmission action may include not sending any FEC packets along with the information packets in the message to a receiving device, when the client devices are geographically close to the server or for example, if the network performance parameters show that there are no packet losses. In other embodiments, the disclosed technology may also dynamically apply a timeout-based retransmission during earlier transmission stages of a message and determine an optimal transmission action for the message during later transmissions stages.

In one embodiment, the constrained optimization problem shown in equation (1) may be converted into an unconstrained optimization problem using a standard Lagrangian technique. A Lagrangian multiplier λ is introduced and a combined objective function is defined as a weighted sum of the failure probability and transmission cost, denoted as shown in equation (2) below:
Jπ(i,q)=επ(i,q)+λρπ(i,q)  (2)

Here, Jπ(i,q) denotes the combined cost when beginning in the sub-trellis rooted in state (i, q). The objective function over the entire trellis is Jπ(l, k). For a given λ, the modified optimization problem which minimizes Jπ(l, k) may be solved to determine the optimal policy π* as shown in equation (3) below:

For the optimal policy π* determined by a given λ, the message failure rate επ(l, k) is then evaluated, which may or may not satisfy the latency requirement constraint (επ≦επ(T)). Hence, λ is varied to find the closest value through bi-section search that just satisfies the constraint. As will be appreciated, this corresponds to finding a point on the convex-hull along the trade-off curve between the message failure rate and the transmission cost.

To solve the above modified optimization problem, the objective function of a particular trellis can be expressed in terms of its sub-trellises, as shown in equation (4) below:

where p(q′|q,π(i,q)) represents the transitional probability from state (i, q) to state (i+1, q′) by transmitting π(i, q) number of packets. Given a model of packet loss, the transition probability can be readily calculated. For example, assuming the packet loss rate is uniform and denoted by p, the transition probability is calculated as shown in equation (5) below:

Also, the cost at the edge is computed as Jπ(I+1,q≠0)=επ(I+1,q)+λρπ(I+1,q)=1, since the failure probability and the transmission cost after the final stage I are επ(I+1,q)=1 and ρπ(I+1,q)=0, respectively. In addition, Jπ(I+1,q=0)=0.

Let J* (i, q) and π*(i, q) define the minimum value of the objective function and the corresponding action, over the sub-trellis rooted at (i, q). Then

By induction, it can be readily shown that J*(i, q)≦Jπ(i, q) for all (i, q) and all π, with equality achieved when π=π*. Therefore, the problem of finding the optimal policy π* (shown in equation 3) can be solved efficiently using dynamic programming using the recursive equations 6 and 7. It is to be noted that the deterministic policy derived above is nearly optimal and sufficient for practical purposes.

FIG. 6shows an MDP framework that illustrates one or more possible transmission actions for a message. In the exemplary illustration shown inFIG. 6, S:6, S:5 and S:4 denote the possible transmission actions for a message at an initial state (1, 4). Similarly, S:4, S:3 and S:2 denote the set of possible transmission actions for a message at a state (2,2) at the end of the first transmission stage. In one embodiment, the transmission actions for a message denote the number of FEC packets to be added to the message, prior to and during transmission of the message to a receiving device. For example, an optimal transmission action, S:6 for the message at the initial state (1, 4) denotes that 2 FEC packets are added to the message prior to transmitting the message to a receiving device at the beginning of transmission stage i=1. In one embodiment, the optimal transmission action for the message at state (1, 4) is determined as discussed inFIG. 4B. The optimal transmission action is pre-computed and stored in a look-up table. Similarly, the optimal transmission actions for each of the different message states (i, q) for the message are pre-computed and stored in separate look-up tables. In another embodiment, the optimal transmission actions and the look-up tables that store the optimal transmission actions for different states of a message may also be determined and created at real-time. In one example, each look-up table stores an optimal transmission action for a message, for a given state (i, q) of the message, a packet loss probability of the message, a transmission stage of the message, remaining transmission stages of the message, the number of remaining transmission stages of the message, the RTT and latency tolerance of the message. During online execution, the optimal transmission action is obtained using a table lookup, given the current state of the message (that is, the number of information packets that remain to be transmitted), the current transmission stage, the current estimate of the packet loss rate associated with the message and the RTT and the latency tolerance of the message.

An exemplary online execution performed by the disclosed technology may be more clearly understood by referring to the exemplary MDP framework illustrated inFIG. 6. Suppose that the optimal transmission action for a message at the initial state (1, 4) is π(1, 4)=6, denoted by S:6. The optimal transmission action, π(1, 4)=6 indicates that 6 packets will be transmitted during transmission stage 1, including 4 information packets and 2 redundant FEC packets. When packet loss occurs, the receiving device may receive less than 6 packets. If the receiving device receives 4 or more packets, the receiving device can recover the original message. In such cases, the message is successfully delivered by the end of transmission stage 1 and there is no need to transmit more packets in the next transmission stage. This is marked inFIG. 6as a special ending state (2,0) at the end of stage 1. However, if the receiving device receives only 2 packets, marked as “R:2” inFIG. 6, then there are still 2 more information packets that need to be transmitted during the beginning of the second transmission stage. Therefore, the current state of the message (i, q) becomes (2, 2). The MDP framework is again utilized to choose the optimal transmission for the message at the new state (2, 2). In one embodiment, and as discussed above, the optimal transmission action for a current state of the message may be dynamically chosen using a table lookup, given the current stage of transmission, the current packet loss rate, the RTT and the latency tolerance. Depending on the loss events in the network, the optimal transmission action at state (2, 2) may lead to another state at the end of the second transmission stage, and so on. By the end of transmission stage 3, the message may still not be delivered completely, denoted by a non-zero state of the message at the end of transmission stage 3. In this case, the message fails to satisfy the latency tolerance T. It may be noted that since all the optimal transmission actions for different states of the message are pre-computed offline and stored as look-up tables, the disclosed technology eliminates the need of performing expensive optimization computations during online execution.

FIG. 7describes an exemplary architecture of a message-oriented, connectionless User Datagram Protocol (UDP)-based protocol comprising an intelligent adaptive decision making engine. The disclosed UDP-based transport protocol112implements all the major functions of a transport protocol, such as maintaining per-flow status for each communicating end-point, estimating the parameters of communication channels, such as round trip time, packet loss rate and timeout period and delivering packets using a combination of FEC and retransmission, as determined by the intelligent adaptive decision making engine114. The architecture of the protocol includes a set of APIs that support both synchronous and asynchronous message transfers.

The interface module700interacts with upper layer applications106,108(shown inFIG. 1B) and provides applications106,108with a variety of transport layer services which enable applications106,108to communicate with each other and as necessary over the network50(shown inFIG. 1B). Outbound messages enter into an outgoing message queue702and wait to be processed by the scheduler706. Inbound messages are assembled and placed into the incoming message queue702for applications106,108. The interface module700can be invoked via both synchronous and asynchronous APIs. The flow manager module708maintains per flow status. For each flow, identified by a set of parameters (IP address, port number), the flow manager module708creates a transmission control block (TCB) and keeps all the important information, such as the number of transmissions and losses, the message deadline, a sliding window, and other statistics. The sliding window controls the number of messages on the fly, that is, those still in transmission and not completely acknowledged. The number of transmissions and losses, together with the message deadline, are used as inputs to the intelligent adaptive decision making engine114. The output of the intelligent adaptive decision making engine114is the optimal transmission action for each transmission stage as discussed inFIGS. 4-6.

The scheduler706is the core engine of the UDP-based protocol. It processes messages from the outgoing message queue704, encodes them using the intelligent adaptive decision making engine114, and enters prepared packets into the outgoing message queue704. The scheduler704also decodes incoming packets from the incoming message queue702, and enters assembled messages into the incoming message queue702. The scheduler706also periodically triggers the flow manager module708to execute background tasks, such as retransmitting timeout packets, cleaning up obsolete TCBs, and so on.

The Input/Output (I/O) engine module714is in charge of sending and receiving individual packets. In one embodiment, the I/O engine module714is implemented using an Input/Output (I/O) completion port to support the throughput and scalability requirements of high concurrency game servers in the service data center.

In another embodiment of the disclosed technology, the intelligent adaptive decision making engine114also takes into consideration the congestion in the network prior to determining the optimal number of FEC packets to add to a message. As will be appreciated, in the event of network congestion, adding FEC packets to a message may further effect the network congestion especially when there are multiple messages to be transmitted simultaneously. In one embodiment, the intelligent adaptive decision making engine114obtains feedback about packet delay and loss to determine a fair share of the network bandwidth so that a low queuing delay can be maintained even during network congestion. In one embodiment, the intelligent adaptive decision making engine114determines the optimal transmission action for a group of messages transmitted between the one or more client devices100A,100B . . .100X and the centralized data server110as discussed inFIG. 8below.

FIG. 8is a flowchart describing a process for dynamically determining an optimal transmission action for a group of messages transmitted between one or more client devices and a centralized data server. In step800, a group of input messages to be transmitted to a receiving device, are received. In one example, the group of input messages may be generated by an interactive application such as the MMO game shown inFIG. 1A. In step801, one or more network performance parameters associated with the group of input messages is obtained. As discussed inFIG. 4, in one example, the network performance parameters may include the round trip time (RTT), the network latency and the packet loss probability of a message transmitted between the client devices100A,100B . . .100X and the centralized data server110.

In step802, a transmission budget Bifor the group of input messages during a transmission stage i associated with the group of input messages is determined. As used herein, the transmission budget refers to the number of messages that are transmitted during a transmission stage i. The number of transmission stages for the group of input messages may be determined as the ratio of the RTT and the latency tolerance as discussed inFIG. 4.

In step804, one or more states associated with each message in the group of input messages are identified. In step806, one or more possibilities of loss patterns for the one or more states associated with each message are determined. In one example, and as discussed inFIG. 4, the different possibilities of loss patterns of a message that result in different states of the message may be identified and represented in a Markov Decision Process (MDP) framework.

In step808, an optimal transmission action for the different states of each of the messages in the group of input messages, M, is determined based on the one or more network performance parameters, the one or more states and the one or more possibilities of loss patterns. In one embodiment, the optimal transmission action minimizes a transmission cost and network latency associated with transmitting each of the messages in the group of messages and may be represented as shown in equation (8) below:

where i, Biand m denote the transmission stage (1≦i≦I), the budget or bandwidth constraint at stage i and the message index (1≦m≦M), respectively. πm(i) is a simplified representation of πm(i,q1, q2, . . . , qm), which denotes the transmission action of a message m at transmission stage i, given the states of the group of M input messages.

As will be appreciated, solving the optimization problem shown in equation (8) involves representing all the combinations of the states from the individual messages. To simplify the optimization and obtain a low complexity solution, the following approximations may be made. First, an assumption is made that the Lagrangian multiplier λ* discovered without the budget or bandwidth constraint represents a satisfactory trade-off between the message delivery latency and the transmission cost. Hence, the objective function is modified to incorporate the latency constraint into the objective function, which in turn becomes a combination of the latency and the cost, weighted by λ*. Secondly, the optimization problem shown in FIG. (8) is solved for a current transmission stage and the budget or bandwidth constraint is not taken into consideration for future transmission stages. When a future transmission stage becomes current, the optimization at that transmission stage ensures that all the actions satisfy the budget or bandwidth constraint. Therefore, the modified optimization problem is denoted as shown in equation (9) below:

minπm⁢∑m=1M⁢(ɛπm+λm*⁢ρπm)⁢⁢s.t.⁢∑m=1M⁢πm⁡(1)≤B1(9)
where λm* corresponds to the optimal policy of message m without the budget or bandwidth constraint.

As will be appreciated, the equation shown in FIG. (9) may be viewed as a classic knapsack problem to determine how to allocate the transmission budget B1among the M messages so that a total cost can be minimized. Hence, an optimal solution can be readily derived using dynamic programming, when the total cost is well-defined given a specific allocation. If {bm} denotes an allocation, where bmpackets are transmitted by message m, the total cost can be represented as shown in equation (10) below:

where Jm*(2,q′) is the optimal cost of message m at stage 2 given state q′. Since the budget or bandwidth constraint is dropped for all the future stages (including stage 2), Jm*(2, q′) can be calculated independently for each message without the budget or bandwidth constraint and is the same as equation (6). As discussed before, the optimal transmission actions J*(i,q′)'s for different states of each message in the group of messages can be pre-computed offline and stored in look-up tables.

In step810, a group of output messages for transmission to the receiving device are created based on the optimal transmission action determined for each message in the group of input messages. In one embodiment, the group of output messages includes the optimal number of redundant FEC packets and the information packets in each message in the group of input messages. In step812, the group of output messages is transmitted to the receiving device. In step814, one or more reply messages are received from the receiving device.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. It is intended that the scope of the invention be defined by the claims appended hereto.