Abstract:
A quality of service (QoS) management server and a method of managing QoS. One embodiment of the QoS management server, includes: (1) a network interface controller (NIC) configured to receive QoS statistics indicative of conditions of a network over which rendered video is transmitted, the rendered video having a fidelity and a latency, and (2) a graphics processing unit (GPU) operable to employ said QoS statistics to tune said fidelity to affect said latency.

Description:
TECHNICAL FIELD 
       [0001]    This application is directed, in general, to cloud gaming and, more specifically, to quality of service (QoS) in the context of cloud gaming. 
       BACKGROUND 
       [0002]    The utility of personal computing was originally focused at an enterprise level, putting powerful tools on the desktops of researchers, engineers, analysts and typists. That utility has evolved from mere number-crunching and word processing to highly programmable, interactive workpieces capable of production level and real-time graphics rendering for incredibly detailed computer aided design, drafting and visualization. Personal computing has more recently evolved into a key role as a media and gaming outlet, fueled by the development of mobile computing. Personal computing is no longer resigned to the world&#39;s desktops, or even laptops. Robust networks and the miniaturization of computing power have enabled mobile devices, such as cellular phones and tablet computers, to carve large swaths out of the personal computing market. Desktop computers remain the highest performing personal computers available and are suitable for traditional businesses, individuals and gamers. However, as the utility of personal computing shifts from pure productivity to envelope media dissemination and gaming, and, more importantly, as media streaming and gaming form the leading edge of personal computing technology, a dichotomy develops between the processing demands for “everyday” computing and those for high-end gaming, or, more generally, for high-end graphics rendering. 
         [0003]    The processing demands for high-end graphics rendering drive development of specialized hardware, such as graphics processing units (GPUs) and graphics processing systems (graphics cards). For many users, high-end graphics hardware would constitute a gross under-utilization of processing power. The rendering bandwidth of high-end graphics hardware is simply lost on traditional productivity applications and media streaming. Cloud graphics processing is a centralization of graphics rendering resources aimed at overcoming the developing misallocation. 
         [0004]    In cloud architectures, similar to conventional media streaming, graphics content is stored, retrieved and rendered on a server where it is then encoded, packetized and transmitted over a network to a client as a video stream (often including audio). The client simply decodes the video stream and displays the content. High-end graphics hardware is thereby obviated on the client end, which requires only the ability to play video. Graphics processing servers centralize high-end graphics hardware, enabling the pooling of graphics rendering resources where they can be allocated appropriately upon demand. Furthermore, cloud architectures pool storage, security and maintenance resources, which provide users easier access to more up-to-date content than can be had on traditional personal computers. 
         [0005]    Perhaps the most compelling aspect of cloud architectures is the inherent cross-platform compatibility. The corollary to centralizing graphics processing is offloading large complex rendering tasks from client platforms. Graphics rendering is often carried out on specialized hardware executing proprietary procedures that are optimized for specific platforms running specific operating systems. Cloud architectures need only a thin-client application that can be easily portable to a variety of client platforms. This flexibility on the client side lends itself to content and service providers who can now reach the complete spectrum of personal computing consumers operating under a variety of hardware and network conditions. 
       SUMMARY 
       [0006]    One aspect provides a QoS management server, including: (1) a network interface controller (NIC) configured to receive QoS statistics indicative of conditions of a network over which rendered video is transmitted, the rendered video having a fidelity and a latency, and (2) a GPU operable to employ said QoS statistics to tune said fidelity to affect said latency. 
         [0007]    Another aspect provides a method of managing QoS with respect to a client, including: (1) receiving QoS statistics indicative of network conditions between a server and the client, (2) employing the QoS statistics on the server in determining a balance of fidelity and latency, and (3) transmitting toward the client a video stream prepared according to the balance. 
         [0008]    Yet another aspect provides a QoS enabled rendering server, including: (1) a central processing unit (CPU) operable to execute a real-time interactive application to define a video stream, (2) a NIC configured to: (2a) manage communication with a client over a network, and (2b) receive QoS statistics indicative of conditions of the network, and (3) a GPU operable to render, capture and encode the video stream for transmission to the client through the NIC according to a plurality of fidelity settings derived from the QoS statistics. 
     
    
     
       BRIEF DESCRIPTION 
         [0009]    Reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: 
           [0010]      FIG. 1  is a block diagram of a cloud gaming system; 
           [0011]      FIG. 2  is a block diagram of a server; 
           [0012]      FIG. 3  is a block diagram of one embodiment of a virtual machine; 
           [0013]      FIG. 4  is a block diagram of one embodiment of a virtual GPU; and 
           [0014]      FIG. 5  is a flow diagram of one embodiment of a method for managing QoS. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]    Major limitations of cloud gaming, and cloud graphics processing in general, are latency and the unpredictable network conditions that bring it about. Latency in cloud gaming can be devastating to game play experience. Latency in simple media streaming is less catastrophic because it is overcome pre-encoding the streaming media, buffering the stream on the receiving end, or both. By its nature, cloud gaming employs a significant real-time interactive component in which a user&#39;s input closes the loop among the server, client and the client&#39;s display. The lag between the user&#39;s input and visualizing the resulting effect is considered latency. It is realized herein that pre-encoding or buffering does nothing to address this latency. 
         [0016]    Latency is induced by a variety of network conditions, including: network bandwidth constraints and fluctuations, packet loss over the network, increases in packet delay and fluctuations in packet delay from the server to the client, which manifest on the client as jitter. While latency is an important aspect of the game play experience, the apparent fidelity of the video stream to the client is plagued by the same network conditions. Fidelity is a measure of the degree to which a displayed image or video stream corresponds to the ideal. An ideal image mimics reality; its resolution is extremely high, and it has no compression, rendering or transmission artifacts. An ideal video stream is a sequence of ideal images presented with no jitter and at a frame rate so high that it, too, mimics reality. Thus, a higher-resolution, higher-frame-rate, lower-artifacted, lower-jitter video stream has a higher fidelity than one that has lower resolution, a lower frame rate, contains more artifacts or is more jittered. 
         [0017]    Latency and fidelity are essentially the client&#39;s measures of the game play experience. However, from the perspective of the server or a cloud service provider, the combination of latency and fidelity are components of QoS (QoS). A QoS system, often a server, is tasked with managing QoS for its clients. The goal is to ensure an acceptable level of latency and fidelity, the game play experience, is maintained under whatever network conditions arise and for whatever client device subscribes to the service. 
         [0018]    The management task involves collecting network data and evaluating the network conditions between the server and client. Traditionally, the client performs that evaluation and dictates back to the server the changes to the video stream it desires. It is realized herein that a better approach is to collect the network data, or “QoS statistics,” on the client and transmit it to the server so the server can evaluate and determine how to improve QoS. Given that the server executes the application, renders, captures, encodes and transmits the video stream to the client, it is realized herein the server is better suited to perform QoS management. It is also realized herein the maintainability of the QoS system is simplified by shifting the task to the server because QoS software and algorithms are centrally located on the server, and the client need only remain compatible, which should include continuing to transmit QoS statistics to the server. 
         [0019]    The client is capable of collecting a variety of QoS statistics. One example is packets lost, or packet loss count. The server marks packets with increasing packet numbers. When the client receives packets, it checks the packet numbers and determines how many packets were lost. The packet loss count is accumulated until QoS statistics are ready to be sent to the server. A corollary to the packet loss count is the time interval over which the losses were observed. The time interval is sent with the QoS statistics, to the server, which can calculate a packet loss rate. Meanwhile, the client resets the count and begins accumulating again. 
         [0020]    Another example of a QoS statistic is a one-way-delay. When a packet is ready to transmit, the server writes the transmit timestamp in the packet header. When the packet is received by the client, the receipt timestamp is noted. The time difference is the one-way-delay. Since clocks on the server and client are not necessarily synchronized, the one-way-delay value is not the same as the packet transmit time. So, as the client accumulates one-way-delay values for consecutive packets and transmits them to the server, the server calculates one-way-delay deltas between consecutive packets. The deltas give the server an indication of changes in latency. 
         [0021]    Yet another example of a QoS statistic is a frame number. Frame numbers are embedded in each frame of video. When the client sends statistics to the server, it includes the frame number of the frame being processed by the client at that time. From this, the server can determine the speed at which the client is able to process the video stream, which is to say, the speed at which the client receives, unpacks, decodes and renders for display. 
         [0022]    QoS statistics are sent periodically to the server for use in QoS determinations. It is realized herein the frequency at which the client sends QoS statistics is itself an avenue of tuning QoS to that client. Another example of a QoS setting, realized herein, is controlling the streaming bit rate. The streaming bit rate is basically the rate at which data is transmitted to the client. Increasing the bit rate consumes more network bandwidth and increases the processing load on the client. Conversely, decreasing the bit rate relieves the network and the client, generally at the cost of fidelity. 
         [0023]    Also realized herein are other QoS settings that are often used along with adjusting the streaming bit rate: capture frame rate scaling and resolution scaling. Capture frame rate scaling, or simply frame rate scaling, allows the server to reduce the capture frame rate, or simply frame rate, to free up network bandwidth. At a given frame rate, a certain number of bits are allocated to each frame. Reducing the frame rate while holding the bit rate steady allows the allocation of more bits to each frame, yielding higher fidelity frames at a lower frame rate. Similarly, resolution scaling allows the server to render frames at a lower resolution to free up bandwidth. At a given resolution, a certain number of bits are allocated to each pixel. Reducing the resolution, that is to reduce the number of pixels in a frame, while holding the bit rate steady allows the allocation of more bits to each pixel. With resolution scaling if bits-per-pixel remains high, the perceived fidelity remains high, and without consuming more network bandwidth. 
         [0024]    Additionally, it is realized herein that a variety of avenues, or QoS settings, for tuning QoS are possible, including: minimum and maximum bit rates, minimum and maximum capture frame rates, the frequency of bit rate changes and hysteresis in buffering thresholds. 
         [0025]    Before describing various embodiments of the QoS system or method introduced herein, a cloud gaming environment within which the system or method may be embodied or carried out will be described. 
         [0026]      FIG. 1  is a block diagram of a cloud gaming system  100 . Cloud gaming system  100  includes a network  110  through which a server  120  and a client  140  communicate. Server  120  represents the central repository of gaming content, processing and rendering resources. Client  140  is a consumer of that content and those resources. Server  120  is freely scalable and has the capacity to provide that content and those services to many clients simultaneously by leveraging parallel and apportioned processing and rendering resources. The scalability of server  120  is limited by the capacity of network  110  in that above some threshold of number of clients, scarcity of network bandwidth requires that service to all clients degrade on average. 
         [0027]    Server  120  includes a network interface card (NIC)  122 , a central processing unit (CPU)  124  and a GPU  130 . Upon request from Client  140 , graphics content is recalled from memory via an application executing on CPU  124 . As is convention for graphics applications, games for instance, CPU  124  reserves itself for carrying out high-level operations, such as determining position, motion and collision of objects in a given scene. From these high level operations, CPU  124  generates rendering commands that, when combined with the scene data, can be carried out by GPU  130 . For example, rendering commands and data can define scene geometry, lighting, shading, texturing, motion, and camera parameters for a scene. 
         [0028]    GPU  130  includes a graphics renderer  132 , a frame capturer  134  and an encoder  136 . Graphics renderer  132  executes rendering procedures according to the rendering commands generated by CPU  124 , yielding a stream of frames of video for the scene. Those raw video frames are captured by frame capturer  134  and encoded by encoder  136 . Encoder  134  formats the raw video stream for transmission, possibly employing a video compression algorithm such as the H.264 standard arrived at by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) or the MPEG-4 Advanced Video Coding (AVC) standard from the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC). Alternatively, the video stream may be encoded into Windows Media Video® (WMV) format, VP8 format, or any other video encoding format. 
         [0029]    CPU  124  prepares the encoded video stream for transmission, which is passed along to NIC  122 . NIC  122  includes circuitry necessary for communicating over network  110  via a networking protocol such as Ethernet, Wi-Fi or Internet Protocol (IP). NIC  122  provides the physical layer and the basis for the software layer of server  120 &#39;s network interface. 
         [0030]    Client  140  receives the transmitted video stream for display. Client  140  can be a variety of personal computing devices, including: a desktop or laptop personal computer, a tablet, a smart phone or a television. Client  140  includes a NIC  142 , a decoder  144 , a video renderer  146 , a display  148  and an input device  150 . NIC  142 , similar to NIC  122 , includes circuitry necessary for communicating over network  110  and provides the physical layer and the basis for the software layer of client  140 &#39;s network interface. The transmitted video stream is received by client  140  through NIC  142 . Client  140  can employ NIC  142  to collect QoS statistics based on the received video stream, including packet loss and one-way-delay. 
         [0031]    The video stream is then decoded by decoder  144 . Decoder  144  should match encoder  136 , in that each should employ the same formatting or compression scheme. For instance, if encoder  136  employs the ITU-T H.264 standard, so should decoder  144 . Decoding may be carried out by either a client CPU or a client GPU, depending on the physical client device. Once decoded, all that remains in the video stream are the raw rendered frames. The rendered frames a processed by a basic video renderer  146 , as is done for any other streaming media. The rendered video can then be displayed on display  148 . 
         [0032]    An aspect of cloud gaming that is distinct from basic media streaming is that gaming requires real-time interactive streaming. Not only must graphics be rendered, captured and encoded on server  120  and routed over network  110  to client  140  for decoding and display, but user inputs to client  140  must also be relayed over network  110  back server  120  and processed within the graphics application executing on CPU  124 . This real-time interactive component of cloud gaming limits the capacity of cloud gaming systems to “hide” latency. 
         [0033]    Client  140  periodically sends QoS statistics back to Server  120 . When the QoS statistics are ready to be sent, Client  140  includes the frame number of the frame of video being rendered by video renderer  146 . The frame number is useful for server  120  to determine how well network  110  and client  140  are handling the video stream transmitted from server  120 . Server  120  can then use the QoS statistics to determine what actions in GPU  130  can be taken to improve QoS. Actions available to GPU  130  include: adjusting the resolution at which graphics renderer  132  renders, adjusting the capture frame rate at which frame capturer  134  operates and adjusting the bit rate at which encoder  136  encodes. 
         [0034]      FIG. 2  is a block diagram of server  120  of  FIG. 1 . This aspect of server  120  illustrates the capacity of server  120  to support multiple simultaneous clients. In  FIG. 2 , CPU  124  and GPU  130  of  FIG. 1  are shown. CPU  124  includes a hypervisor  202  and multiple virtual machines (VMs), VM  204 - 1  through VM  204 -N. Likewise, GPU  130  includes multiple virtual GPUs, virtual GPU  206 - 1  through virtual GPU  206 -N. In  FIG. 2 , server  120  illustrates how N clients are supported. The actual number of clients supported is a function of the number of users ascribing to the cloud gaming service at a particular time. Each of VM  204 - 1  through VM  204 -N is dedicated to a single client desiring to run a respective gaming application. Each of VM  204 - 1  through VM  204 -N executes the respective gaming application and generates rendering commands for GPU  130 . Hypervisor  202  manages the execution of the respective gaming application and the resources of GPU  130  such that the numerous users share GPU  130 . Each of VM  204 - 1  through VM  204 -N respectively correlates to virtual GPU  206 - 1  through virtual GPU  206 -N. Each of the virtual GPU  206 - 1  through virtual GPU  206 -N receives its respective rendering commands and renders a respective scene. Each of virtual GPU  206 - 1  through virtual GPU  206 -N then captures and encodes the raw video frames. The encoded video is then streamed to the respective clients for decoding and display. 
         [0035]    Having described a cloud gaming environment in which the QoS system and method introduced herein may be embodied or carried out, various embodiments of the system and method will be described. 
         [0036]      FIG. 3  is a block diagram of virtual machine (VM)  204  of  FIG. 2 . VM  204  includes a VM operating system (OS)  310  within which an application  312 , a virtual desktop infrastructure (VDI)  314 , a graphics driver  316  and a QoS manager  318  operate. VM OS  310  can be any operating system on which available games are hosted. Popular VM OS  310  options include: Windows®, iOS®, Android®, Linux and many others. Within VM OS  310 , application  312  executes as any traditional graphics application would on a simple personal computer. The distinction is that VM  204  is operating on a CPU in a server system (the cloud), such as server  120  of  FIG. 1  and  FIG. 2 . VDI  314  provides the foundation for separating the execution of application  312  from the physical client desiring to gain access. VDI  314  allows the client to establish a connection to the server hosting VM  204 . VDI  314  also allows inputs received by the client, including through a keyboard, mouse, joystick, hand-held controller, or touchscreens, to be routed to the server, and outputs, including video and audio, to be routed to the client. Graphics driver  316  is the interface through which application  312  can generate rendering commands that are ultimately carried out by a GPU, such as GPU  130  of  FIG. 1  and  FIG. 2  or virtual GPUs, virtual GPU  206 - 1  through virtual GPU  206 -N. 
         [0037]    QoS manager  318  collects QoS statistics transmitted from a particular client, such as client  140 , and determines how to configure various QoS settings for that client. The various QoS settings influence the perceived fidelity of the video stream and, consequently, the latency. The various QoS settings generally impact the streaming bit rate, capture frame rate and resolution; however, certain QoS settings are more peripheral, including: the frequency of QoS statistic transmissions, the frequency of bit rate changes and the degree of hysteresis in the various thresholds. Once determined, QoS manager  318  implements configuration changes by directing the GPU accordingly. Alternatively, the QoS manager tasks can be carried out on the GPU itself, such as GPU  130 . 
         [0038]      FIG. 4  is a block diagram of virtual GPU  206  of  FIG. 2 . Virtual GPU  206  includes a renderer  410 , a framer capturer  412 , an encoder  414  and a QoS manager  416 . Virtual GPU  206  is responsible for carrying out rendering commands for a single virtual machine, such as VM  204  of  FIG. 3 . Rendering is carried out by renderer  410  and yields raw video frames having a resolution. The raw frames are captured by frame capturer  412  at a capture frame rate and then encoded by encoder  414 . The encoding can be carried out at various bit rates and can employ a variety of formats, including H.264 or MPEG4 AVC. The inclusion of an encoder in the GPU, and, moreover, in each virtual GPU  206 , reduces the latency often introduced by dedicated video encoding hardware or CPU encoding processes. 
         [0039]    Similar to QoS manager  318  of  FIG. 3 , QoS manager  416  collects QoS statistics and determines how to configure various QoS settings for the client. Unlike the embodiment of  FIG. 3 , the inclusion of QoS manager  416  within virtual GPU  206  allows more direct control over the elements of each virtual GPU, including renderer  410 , frame capturer  412  and encoder  414 . These elements are largely responsible for implementing the various QoS settings arrived at by QoS manager  416 , or QoS manager  318  of the embodiment of  FIG. 3 . Certain other QoS settings originate at the client itself, such as the frequency of QoS statistics transmissions. 
         [0040]      FIG. 5  is a flow diagram of one embodiment of a method of managing QoS with respect to a client that receives a video stream transmitted by a server. In certain embodiments, the video stream is a product of executing a real-time interactive application, a game for instance, that yields a scene to be rendered and a set of rendering commands to be carried out by a GPU. The method in  FIG. 5  begins at a start step  510 . QoS statistics are received at a step  520  and are indicative of the network conditions between the server and the client. QoS statistics include a packet loss count over a particular period of time, one-way-delay times per packet and the frame number of the frame being processed by the client at the time of QoS statistic transmission. If the network conditions indicate the client is experiencing excessive latency, or a significant drop in perceived fidelity, then changes to various QoS settings will be considered. At a step  530 , the QoS statistics received by the server are employed to determine a new balance of fidelity and latency. 
         [0041]    Certain embodiments employ QoS statistics to adjust the capture frame rate, also referred to as frame rate scaling. A reduction in the capture frame rate frees up network bandwidth by reducing the number of video frames transmitted over a particular time interval. This also reduces the processing load on the client. Additionally, reducing the capture frame rate allows greater bits-per-frame allocations without consuming more network bandwidth. While this alternate would not necessarily reduce latency, it could manifest as improved perceived fidelity. Other embodiments employ QoS statistics to adjust the resolution of rendered frames, or resolution scaling. Similar to frame rate scaling, a reduction in resolution allows either greater bits-per-pixel allocations or frees up network bandwidth. Increasing the bits-per-pixel can also manifest as improved perceived fidelity. 
         [0042]    Continuing the embodiment of  FIG. 5 , once the new balance is determined and distilled down to the various QoS settings, those settings are implemented. Some QoS settings are implemented on the client, while others are implemented on the server in the GPU. The QoS settings implemented in the GPU include settings related to the streaming bit rate, capture frame rate and resolution. The implementations generally manifest in the rendering, frame capture and encoding stages of video stream preparation. At a step  540 , the video stream, prepared according to the new balance of fidelity and latency, is transmitted toward the client. 
         [0043]    In other embodiments, this procedure repeats: QoS statistics are constantly logged and transmitted to the server, employed to determine QoS settings that ultimately manifest in the video stream transmitted from the server to the client as fidelity and latency, otherwise referred to as QoS. The method then ends at a step  550 . 
         [0044]    Those skilled in the art to which this application relates will appreciate that other and further additions, deletions, substitutions and modifications may be made to the described embodiments.