Offloading the processing of signals

A server computer that is connected over a network to one or more remote client computers includes specialized hardware for handling computationally-intensive signal processing operations involving video, audio, and/or other signals. As a result, users of the client computers experience little or no delay when accessing over the network applications that are hosted on the server.

TECHNICAL FIELD

The invention relates to processing video and/or other signals faster within a server system in order to improve the experience of a user accessing the server system over a network with a remote client computer.

BACKGROUND INFORMATION

In the past, time-shared computers, also known as mainframe computers, allowed a large number of users to interact concurrently with a single computer through the use of remotely located user terminals. The next wave of computing saw individual computers, also known as personal computers move onto a user's desktop. Over time, personal computers became more powerful than the centralized mainframe computers. However, over the last five to ten years, the computing industry has seen the deconstruction of personal computers into individual components of storage, processors, and user-interfaces, where each of these components are combinable in different ways. More recently, a growing trend has been to shift back to a centralized computing model with a processor and storage located in a data center, and a user interface extended out to the desktop for the user. The benefits of centralization in this model include reduced cost, increased security, reduced operational expense, increased resilience, and increased performance.

Microsoft Corporation's Remote Desktop Protocol (RDP) is a protocol which allows a user's local computer to interact across a network with a server system by transferring graphics display information from the server(s) to the user's local display and also transporting input from the user to the server(s). RDP allows for specialized software processing and communication between components on the user's computer and components running on the server(s). For example, if a server computer processes a 1920×1200 video running at 30 frames per second (fps), which would otherwise result in about 1.7 Gigabits per second (Gbps) of video data being transferred to memory within the server, RDP will compress the data down to about 1 Mbps (megabits per second) to 30 Mbps and thereby reduce the amount of data that needs to be written into system memory within the server. Even with this compression performed in software by RDP, there still are significant performance costs (in, for example, bandwidth, power, and latency), particularly when many users are sharing the resources of the server(s) as is typically the case.

SUMMARY OF THE INVENTION

The invention relates to offloading, to one or more specialized electronic hardware components, the processing of video, audio, and/or USB (Universal Serial Bus) peripheral signals in order to optimize the experience of a user at a local client computer when connected over a network to a remote server system. Video in particular can be very expensive to transfer between client and server computers, in terms of at least bandwidth and latency. The overall experience of users at client computers can be enhanced when servers are able to perform better and faster by offloading certain signal processing tasks to one or more specialized signal processing hardware components. Specialized hardware according to the invention is designed to work with any server computer as well as software utilized by server computers such as Microsoft's RDP. The inventive hardware cooperates with the server's existing processor(s) and memory to offload from the server's processor(s) and memory computationally demanding and intensive tasks such as the compression of video signals. This allows the tasks to be done faster on the server which in turn improves the experience of a user accessing the server over a network with a remote client computer.

In one aspect, the invention generally relates to a method of offloading the processing of one or more signals such as video, audio, and/or other signals. The method is executed by one or more server computers connected via a communications network to one or more client computing devices. The method includes receiving, over the communications network, one or more requests associated with one or more signals, transmitting the one or more signals to specialized signal processing hardware associated with the server computer, receiving one or more tokens from the signal processing electronics in response to the transmitted one or more signals, storing the one or more tokens, and requesting the signal processing electronics to process the one or more requests by passing the one or more stored tokens to the signal processing electronics.

Embodiments according to this aspect of the invention can include a variety of features. For example, the one or more requests can involve processing of video, audio, and/or other signals. And the processing can be compression, encoding, decoding, and/or encryption, for example. The one or more server computers can execute a remote access protocol such as Microsoft Corporation's Remote Desktop Protocol (RDP).

In another aspect, the invention generally relates to a server computer that performs the above-described method.

These and other aspects, features, objects, and advantages of the invention will become apparent with reference to the following description, the accompanying drawings, and the claims. Furthermore, it is to be understood that the features and the various embodiments described herein are not mutually exclusive and can exist in various combinations and permutations whether or not expressly set forth.

DESCRIPTION

The invention generally relates to offloading the processing of video signals (and/or other signals such as audio signals and/or USB peripheral signals) in order to optimize the experience of a user at a client computer that is communicating with a server computer over a communications network, such as an intranet or the Internet. By offloading to specialized hardware some or all of the more compute-intensive signal processing tasks required of the server computer and that the server computer otherwise would perform in software using its main processor(s) and system memory, the server computer is able to provide the user at the client computer with reduced delays and an experience that is as close as possible to the user's experience when his or her client computer is running all applications locally on the client computer and not accessing them remotely over the network from the server computer.

Referring toFIG. 1, in one embodiment according to the invention, a system100includes a server102, two or more computing device terminals104a,104b, and a communications network106. Each of the computing device terminals104a,104bis wired or wirelessly connected to the network106. In addition, each of the computing device terminals104a,104bis configured to transmit and receive data over the network106. The server102is also wired or wirelessly connected to the network106. The system100generally includes software such as an operating system running on each of the computers (102,104a,104b, etc.), and each of the computers can have the same or a different operating system. Possible operating systems include any version of Microsoft's operating system, any version of the Linux operating system, and any version of Apple's operating system. Each operating system typically will support Microsoft Corporation's RDP or some other remote access protocol.

Referring toFIG. 2A, the system100is not limited to a minimum or maximum number of computing device terminals104a,104b. There could be one, two, three, etc. of these computing device terminals. The computing device terminals104a,104bcan be general-purpose client computers such as laptops, desktops, tablets, etc., and as such each of these terminals typically will include a processor108, storage110, a network interface112, and a display terminal114. The display terminal114can be a cathode ray tube (CRT) display device, a flat screen display device, or any other type of display device that can be viewed by a user of any of the client terminals104a,104b. The processor108is a central processing unit (CPU) that typically comprises a single microprocessor but can comprise two or more cooperating microprocessors, and in any event the processor108is operatively coupled to the storage110and the network interface112. The storage110includes instructions which, when executed by the processor108, cause the processor108at least to display a video signal on the display terminal114. Each of these computing device terminals104a,104btypically will include a variety of software programs including an operating system, an Internet browser, and other applications. Each of these computing device terminals also typically will include various peripheral devices such as a keyboard and a mouse or other pointing device.

Referring toFIGS. 2A and 2B, the server102can be a general-purpose computer like the terminals104a,104bdescribed herein but more typically will be one or more dedicated computers such as file servers, print servers, database servers, application servers, and/or web servers. As such, the server102typically will include a processor200, storage202, and one or more network interfaces204. In accordance with the invention, the server102also includes a bridge206, a bus208, specialized signal processing electronics210, and storage212associated specifically with the electronics210. The processor200of the server102is a CPU that can comprise a single microprocessor but more typically comprises a multiprocessor. In any event, the processor200, whatever its architecture, is operatively coupled to the storage202and, in one embodiment, the network interface204. The storage202includes instructions which, when executed by the processor200, cause the processor200to perform a variety of functions.

Each of the network interfaces112,204is operatively coupled to the network106via a connection. The connection can be an Ethernet cable. In other embodiments, the connection can be a coaxial or fiber optic cable. In yet other embodiments, the connection can be a wireless link or connection such as, for example, a cellular link or a Wi-Fi link. The network106can be, for example the Internet, a dedicated communications network, an intranet, a cellular network, or any other network capable of effecting communication between the server102and each of the computing device terminals104aand104b.

The processor200is in communication with the signal processing electronics210via the bridge206and the bus208. The signal processing electronics210can include an encoder/decoder component and other processing components. The bridge206can be, for example, a peripheral component interconnect express component (PCI Express or PCIe). The bus208can be, for example, a conventional peripheral component interconnect bus (Conventional PCI bus) or a peripheral component interconnect express bus (PCIe Bus). The signal processing electronics210is operatively coupled to the storage212, and in one embodiment also to the network interface204.

The storage110, storage202, and storage212is any physical computer-readable media such as one or more of magnetic storage media, optical storage media, magneto-optical storage media, read-only memory (“ROM”), random-access memory (“RAM”), and flash memory devices. Magnetic storage media may be hard disks, floppy disks, and magnetic tape. Optical storage media may be Compact Disc/Digital Video Discs (“CD/DVDs”), Compact Disc-Read Only Memories (“CD-ROMs”), and holographic devices.

A network protocol is in communication with the processor200via the network interface204, and/or it is in communication with the signal processing electronics210via the network interface204. In any event, the network protocol may be Microsoft Coporation's RDP or some other remote access protocol. The system100is configured to transmit video, audio, and other signals (such as USB-based peripheral traffic) across the network106to the computing device terminal104aand/or the computing device terminal104bvia the network interface204. In order to increase performance of the system100, the signal processing electronics210is configured as one or more specialized and/or dedicated hardware components for handling exclusively certain processes that are offloaded by the processor200and that otherwise would be executed more slowly in software by the processor200. The signal processing electronics210performs in hardware computationally-intensive tasks which would otherwise be performed in software by the processor200. The compute-intensive tasks can include, for example, compression of audio and video signals, encoding and decoding audio and video signals, bulk compression of audio, video, and other signals (such as signals from USB peripherals), and encryption of signals for transmission across the network106.

The signal processing electronics210thus can be considered a hardware accelerator, and it, together with at least its associated accelerator storage212, can be incorporated onto a physical card or circuit board that can be communicatively connected to the processor200of the server102(by, for example, the bus208). Alternatively, at least the signal processing electronics210and its storage212can be realized as one or more integrated circuits (ICs) that can be incorporated into the motherboard of the server102and thereby communicatively connected to the processor200. In any case, at least the signal processing electronics210and its storage212are, in accordance with a preferred embodiment of the invention, physical electronic hardware that are specialized and dedicated to processing very fast and efficiently various computationally-intensive signal processing type operations that otherwise would be handled more slowly and less efficiently in software by the existing general-purpose processor200of the server102.

As indicated by one of the communication arrows inFIG. 2Band as indicated elsewhere herein, the signal processing electronics210can communicate over the network106via the network interface204. In some embodiments, both the signal processing electronics210and the processor200communicate over the network106via the network interface204, or else just one of the signal processing electronics210and the processor200communicates over the network106via the network interface204. Also, while just a single box is shown as the network interface204inFIG. 2B, it is possible and within the scope of this description that each of the signal processing electronics210and the processor200has its own separate and distinct network interface204. In a preferred embodiment according to the invention, the maximum benefits of the token-based scheme described herein are achieved when the signal processing electronics210is able to communicate over the network106directly via either a shared or a dedicated network interface204.

Referring toFIG. 3, a method300of offloading the processing of one or more video (and/or other) signals, which is executed by a server computer, includes the following steps: receiving a request302, transmitting the request304, receiving a token306, storing the token308, returning the token310, and processing the request312. In operation, one of the terminals104a,104bof the system100sends a request to the server102via the network106for a video signal. The terminal104a,104bcan display the video signal on its display terminal114once the video signal is received over the network106from the server102. The request for the video signal is received at the server102(step302). And, in response to the received request, the processor200transmits the video signal to the signal processing electronics210for compression (and/or one or more other computationally-intensive signal processing operations) prior to sending the video signal across the network106(step304). The signal processing electronics210then stores in the storage212the video signal transmitted to it by the processor200. The signal processing electronics210creates a unique token for each separate video signal stored in the storage212. This token acts as a place-holder for the actual signal/data (whether video, audio, etc.) kept in the storage212. Each of these tokens may be, for example, a 4-byte data packet. The token is then transmitted from the signal processing electronics210to the processor200for storing in the storage202(step306). The token is stored in the storage202at step308. The signal processing electronics210maintains a mapping between each token transmitted and stored in the storage202and each video signal stored in the storage212. The processor200can subsequently request the signal processing electronics210to process the video signal by retrieving the token from the storage202and transmitting the token back to the signal processing electronics210(step310). In response to receiving the token, the signal processing electronics210associates the token with the stored video signal and performs the requested function (e.g., compression of the video signal for transmission across the network106) at step312.

In this manner, a high volume of requests for video signals can be received and processed by the server102without causing unacceptable latency between the server102and the computing device terminals104a,104b. This operation can take place several times during a particular session for users of the computing device terminals104a,104b.

This inventive token-based scheme that is used in connection with the specialized signal processing hardware described herein can result in data traffic within the server102being reduced by a factor of about sixty as compared to the conventional software-based handling of data according to RDP. After a token is established according to the inventive token-based scheme, all future steps of the RDP (or any other remote access protocol) involving the data represented by that token do not require the actual represented data to be transformed (for example, compressed) except by the highly efficient and specialized signal processing electronics210. If encoded video needs to be compressed or sent out over the network106, for example, only the one or more tokens are sent back to the signal processing electronics210by the processor200, and then the signal processing electronics210uses the token(s) to locate the actual data in the storage212and then process that data in hardware as required (for example, encoding it, compressing it, encrypting it, sending it out over the network106, etc.). As already indicated, the signal processing electronics210maintains a one-to-one mapping between the actual data held in the storage212and the related token that is passed back to the RDP (or other remote access protocol) stack being executed by the processor200, and thus there is a unique relationship between each token and its corresponding actual data.

Also, it is noted that the inventive token-based scheme described herein allows hardware offloading (to the signal processing electronics210) to occur for any given network protocol without that protocol being aware of, concerned with, or involved with the offloading to the specialized signal processing hardware. It is the use of tokens that allows any given network protocol to operate as usual and without any awareness of or involvement with the hardware offloading. Whatever network protocol is being used (such as Microsoft's RDP in the preferred embodiment according to the invention), that protocol is unaware of the existence of the signal processing electronics210and also is unaware of the mapping that is done by the signal processing electronics210to relate a particular token to its associated data held in the storage212. In the preferred embodiment, when RDP wants to send compressed video data across the network, RDP simply performs all of the same functions that it would if the offloading signal processing electronics210were not present. That is, the RDP stack is not aware of or involved with any of the swapping of tokens for actual data. This swapping function is a function that the signal processing electronics210handles when it receives a token from the processor200in conjunction with, for example, an operation where the RDP is sending compressed video data out across the network106via the network interface204.

Referring toFIG. 4, an implementation of a protocol stack400in the networking protocol suite of the system100can be configured in several manners. In one embodiment, the network protocol402utilizes the Remote Desktop Protocol (RDP) by Microsoft Corporation and is the top level of the protocol stack400. The network protocol402can be configured to be in communication with application programming interfaces (APIs) or other software drivers to allow the signal processing electronics210to perform computationally intensive tasks, such as encoding or decoding the video signals404or bulk compression of audio, video, and/or other signals406. Other layers of the protocol stack400include the application layer408(secure socket layers (SSL)), the transport layer410(transmission control protocol (TCP)), and the Internet layer412(Internet Protocol (IP)) for the transmission of data across the network106. This process takes advantage of optimizing across layers of the protocol stack400rather than treating each layer separately.

Referring toFIG. 5, a protocol sequence500to establish communication between the computing device terminals104a,104band the server102can include several steps. The protocol sequence500typically is not a computationally intensive process and as such it may run on the processor200of the server102and not be offloaded to the signal processing electronics210. The steps of the protocol sequence500between the computing device terminals104a,104band the server at step502can include: initiating a connection at step502, exchanging basic settings at step504, establishing a channel connection at step506, commencing security of the network protocol at step508, exchanging secure settings at step510, confirming appropriate licenses at step512, exchanging capabilities at step514, and finalizing the connection at step516. Once the connection has been established at step516, video, audio and USB peripheral traffic can be transferred between the computing device terminals104a,104band the server102.

Referring toFIGS. 1,2A,2B,3,4, and5, once the protocol sequence500has been established, a series of session identifications (ID) can be used to track individual user sessions on each of the computing device terminals104a,104b. In this manner, the signal processing electronics210can take over the operation of the protocol sequence500from the protocol stack400. If an error occurs once the protocol sequence500is established, the signal processing electronics210can hand back control of the protocol sequence500to the protocol stack400with session status information such that the protocol stack400can seamlessly take back control.

In the case of Virtual Desktop Infrastructure (VDI), the process of allowing the network protocol to offload the processing of one or more video signals can be shared by many computing devices by using standard hardware device virtualization techniques. In this case, the use of tokens rather than the transmission of the actual data to move through the protocol sequence500is increased due to the extra overhead virtualization places on the movement of data within the server102.

Referring toFIG. 6, in another embodiment according to this aspect of the invention, a system600includes a server102, an offload server602, a computing device terminal104a, and a communications network106. In contrast to the system100ofFIGS. 1,2,3,4, and5, the system600includes non-Microsoft operating protocols and systems, such as Linux® and Apple® based systems. In operation, the server102is a non-Microsoft server that transmits sequenced audio, video, and USB signals to the offload server602. The protocol between the server102and the offload server602can use a Remote Direct Memory Access (RDMA) like mechanism such that the data from the server102is transmitted without software intervention. This mechanism can reduce the protocol processing overhead which is beneficial on the server102. Alternatively, this can also be achieved by taking the raw audio, video, or USB outputs from the server102and bundling it for processing by the offload server602. The offload server602analyzes the sequenced signals and processes the signals for transmission as does the RDP network protocol ofFIGS. 4 and 5. The offload server602transmits the signals to the computing device terminal104avia the communications network106. In operation, the process of requesting and processing tokens operates as does the processes of the system100.

It will be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be construed as limiting, but merely as illustrative of some embodiments according to the invention.