Method, system, and computer-readable medium for the adaptive filtering and compression of video data

A method of filtering and encoding video data in a compressed format is provided. The method includes receiving pixels representing video data from an initial video display, receiving pixels representing video data from a subsequent video display, selecting a group of pixels from the subsequent video display as new video data, determining filtered pixel values from the new video data, and encoding the filtered values in a compressed format for communication to another computer accessible via a computer network.

TECHNICAL FIELD

The present invention is related to video compression in a computer network. More particularly, the present invention is related to the adaptive filtering and compression of video data.

BACKGROUND OF THE INVENTION

Computer networks generally include a plurality of interconnected computer systems. Some computer networks utilize a local computer for communicating data to one or more remote computers that are connected to the local computer through the network. From the remote computer, users may control or view activity on a local computer over the network utilizing a hardware interface device connected to the local computer. For instance, utilizing the interface device, a user may view screens of video data on the remote computer that were generated by the local computer. Each screen of video data may comprise thousands or millions of pixels, with each pixel representing a single point in a graphic image. Each point or pixel in a graphic image is represented by a predetermined number of bits based on the number of colors that are displayed on a graphics display. For example, 256 colors may be represented utilizing eight bits per pixel while a “true color” image may be generated utilizing 24 bits per pixel.

Since a single screen comprising a graphics image may comprise millions of bits of video data, the video data must be compressed before being communicated between computers in a network. Video compression enables the communication of video data in fewer bits (and in less time) than if the video data was communicated in its raw form. Typically, video data compression involves compressing the pixel values that make up the frames of video data utilizing a compression algorithm. Many current compression algorithms utilize an exclusive disjunction or “XOR” filtering process to compare the pixel values at various locations in two frames (or screens) of video data. In the XOR process, pixel values at the same locations in successive frames of video data are compared. The result of each comparison is then encoded for communication over the network.

However, current video data compression algorithms based on the XOR process, suffer from a number of drawbacks. One drawback is that as a result of the XOR filtering process, pixel values from previous screens of video data are often compressed along with pixel values from current screens of video data. That is, using the XOR process, unchanged pixel values from previous screens of video data are carried over with changed pixel values from newer screens of video data for compression. As a result, when the video data is compressed, the unchanged pixel values are compressed along with the changed pixel values resulting in less than ideal compression.

It is with respect to these considerations and others that the present invention has been made.

SUMMARY OF THE INVENTION

In accordance with the present invention, the above and other problems are solved by providing in a computer network, a method of increasing the efficiency of filtering and compressing screens of new or changed video data from screens of previously viewed video data. In accordance with other aspects, the present invention relates to a method of compressing video data in a computer network.

According to one aspect of the invention, a method is provided for compressing video data in a computer system. According to the method, a first plurality of pixels representing video data from a previous video display are received. A second plurality of pixels representing video data from a current video display are also received. A pixel is selected in the second plurality of pixels as a current pixel. A filtered value is generated for the current pixel by determining whether the values of pixels adjacent to the current pixel are within a threshold value of the values of other adjacent pixels within the first plurality of pixels. An encoded value is then generated for the filtered value by applying a first algorithm wherein lossless compression is desired or by applying a second algorithm wherein lossy compression is desired. Utilizing lossy compression, some information in the screen display may be discarded. Utilizing lossless compression, no information is discarded. The encoded value is then compressed for transmission to a remote computer system.

According to aspects of the method, generating a filtered value may comprise determining whether the value of a first pixel adjacent to the current pixel in the second plurality of pixels is equal to the value of a corresponding pixel in the first plurality of pixels. If the value of the first pixel adjacent to the current pixel in the second plurality of pixels is equal to the value of a corresponding pixel in the first plurality of pixels, then the filtered value of the current pixel is set equal to the value of a pixel corresponding to the current pixel in the first plurality of pixels.

If the value of the pixel adjacent to the current pixel in the second plurality of pixels is not equal to the value of a corresponding pixel in the first plurality of pixels, a first threshold check is performed to determine whether the value of the first pixel adjacent to the current pixel is within a threshold value of a second pixel adjacent to the first pixel. A second threshold check is also performed to determine whether the value of the second pixel adjacent to the current pixel is within a threshold value of a third pixel adjacent to the current pixel. The filtered value of the current pixel is set equal to the value of the third pixel adjacent to the current pixel if the first threshold check is true and the second threshold check is false. Otherwise, the filtered value of the current pixel is set equal to the value of the first pixel adjacent to the current pixel.

According to other aspects of the method, generating an encoded value for the filtered value may include determining whether lossless or lossy compression should be utilized. If lossless compression should be utilized, a determination is made as to whether the value of the current pixel is equal to the filtered value of the current pixel. If so, the encoded value for the current pixel is set equal to zero. If not, the encoded value is set to a code representing the color black if the value of the current pixel is zero. Otherwise, the encoded value for the current pixel is set equal to the actual value of the current pixel.

If lossy compression is to be utilized, a threshold check is performed to determine whether the value of the current pixel and the filtered value are within a threshold value of one another. If the value of the current pixel and the filtered value are within a threshold value of one another, the encoded value for the current pixel is set to zero. If the value of the current pixel and the filtered value are not within a threshold value of one another and the actual value of the current pixel is zero, the encoded value is set to a code representing the color black. Otherwise, the encoded value for the current pixel is set equal to the actual value of the current pixel.

According to other aspects of the invention, compressing the encoded value comprises determining whether the encoded value is zero. In response to determining that the encoded value is zero, incrementing a run length. In response to determining that the encoded value is not zero, flushing a pending run and outputting the encoded value. Flushing a pending run may include outputting a first code and the run length when the length of the run is greater than a predetermined length and outputting a second code and the run length when the length of the run is less than the predetermined length.

According to other aspects of the invention, the first pixel adjacent to the current pixel may comprise the pixel immediately to the left of the current pixel. The second pixel adjacent to the current pixel may comprise the pixel immediately to the left and above the current pixel. The third pixel adjacent to the current pixel may comprise the pixel immediately above the current pixel. Other configurations of adjacent pixels may also be utilized.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention provide methods, systems, apparatus, and computer-readable media for compressing screens of video data. In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments or examples. Referring now to the drawings, in which like numerals represent like elements through the several figures, aspects of the present invention and the exemplary operating environment will be described.

FIG. 1and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. While the invention will be described in the general context of program modules that execute in conjunction with an application program that runs on an operating system on a computer system, those skilled in the art will recognize that the invention may also be implemented in combination with other program modules.

Turning now toFIG. 1, an illustrative computer architecture for practicing the various embodiments of the invention will now be described. In particular, a computer100is provided that is operative to compress and transmit its video display to a remote computer. The compressed data may be decompressed and displayed at the remote computer. The computer100is also operative to receive input remotely from the remote computer in the form of keyboard or mouse input. In this manner, the computer100may be controlled remotely. In order to provide this functionality, the computer100includes a baseboard, or “motherboard”, which is a printed circuit board to which a multitude of components or devices may be connected by way of a system bus112or other electrical communication path. In one illustrative embodiment, these components include, without limitation, a redirection controller104, a central processing unit (“CPU”)108, a network adapter122, a system memory and an input/output module110. It is also contemplated that the computer100may include other components that are not explicitly shown inFIG. 1.

The system bus112utilized by the computer100provides a two-way communication path for all components connected to the system bus112. The component that initiates a communication is referred to as a “master” component and the component to which the initial communication is sent is referred to as a “slave” component. A master component therefore issues an initial command to or requests information from a slave component. Each slave component is addressed, and thus communicatively accessible to the master component, using a particular slave address. Both master components and slave components are operable to transmit and receive communications over the system bus112. Buses and the associated functionality of master-slave communications are well-known to those skilled in the art, and therefore not discussed in further detail herein.

The redirection controller104allows a user to control the keyboard and mouse functions of the local computer100from the remote computer200over the network18. The redirection controller104may also be utilized to provide the video display shown on the local computer100to the remote computer200. In particular, in accordance with illustrative embodiments of the invention, the redirection controller104communicates compressed video data generated on the local computer100to the remote computer200. To accomplish the above-noted and other functions, the redirection controller104is communicatively connected to one or more components either directly or by way of a management bus130. In particular, the redirection controller104is connected to video out port116of the graphics adapter113, as well as a keyboard input port and a mouse input port of the input/output module110, via the communications lines118and120, respectively. It will be appreciated that the keyboard port and mouse port may include universal serial bus (“USB”) ports and/or PS/2 ports. It should be appreciated that the redirection controller104may receive keyboard and mouse commands from the computer200via the network18. When received, the redirection controller104is operative to pass the commands through to the input/output controller110so that the commands appear to the computer100to have been made utilizing local keyboard and mouse devices.

The network adapter122is communicatively connected to the management bus130. The management bus130is used by the redirection controller104to communicate compressed video data to the remote computer200over the network adapter122. Like the system bus112, the component that initiates communication on a bus is referred to a master and the component to which the communication is sent is referred to a slave. As such, the redirection controller104functions as the master on the management bus130in most circumstances, but may also function as a slave in other circumstances. Each of the various components communicatively connected to the redirection controller104by way of the management bus is addressed using a slave address. In one embodiment, the management bus130may be an I2C® bus, which is manufactured by Phillips Semiconductors® and described in detail in the I2C® bus Specification, version 2.1 (January 2000). The redirection controller104also includes compression program code which may be an executable program module containing program code for filtering and compressing video data for communication over the network18to the remote computer200. The operations performed by the compression program code for compressing video data will be described in greater detail below with respect toFIGS. 4-7. It should be appreciated that the redirection controller104may be configured with its own network adapter for communicating with the remote computer200directly over the network18.

The system memory in the computer100may include including a random access memory (“RAM”)106and a read-only memory (“ROM”)107. The ROM107may store a basic input/output system that includes program code containing the basic routines that help to transfer information between elements within the computer100. The network adapter122may be capable of connecting the local computer100to the computer200via the network18. Connections which may be made by the network adapter122may include local area network (“LAN”) or wide area network (“WAN”) connections. LAN and WAN networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

The CPU108is a standard central processor that performs arithmetic and logical operations necessary for the operation of the remote computer system100. CPUs are well-known in the art, and therefore not described in further detail herein. The input/output module110is used as a communication medium between any number and type of peripheral devices and the system bus112. Communications destined for the CPU108, the redirection controller104or any other component coupled to the system bus112and issued by a peripheral device must therefore pass through the input/output module110to the system bus112and then to the necessary component.

As shown inFIG. 1, the input/output module110is connected a mass storage device14for storing an operating system16and application programs31. The operating system16comprises a set of programs that control operations of the local computer100and allocation of resources. The set of programs, inclusive of certain utility programs, also provide a graphical user interface to the user. An application program is software that runs on top of the operating system software and uses computer resources made available through the operating system to perform application specific tasks desired by the user.

The mass storage device14and its associated computer-readable media, provide non-volatile storage for the local computer100. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the local computer100. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.

A graphics adapter113is also utilized that enables the display of video data (i.e., text and/or graphics) on a display unit114. It will be appreciated that the video graphics adapter may process analog signals (i.e., VGA) or digital signals (i.e., DVI) for display on a compatible display unit. The video graphics adapter114includes a video buffer for temporarily storing one or more lines of video data to be displayed on the display unit114.

It will be appreciated that the computer200described briefly above with respect toFIG. 1may be a general purpose computer including some or all of the conventional computing components described above relative to the local computer100. In addition, the computer200may further include a hardware keyboard and mouse connected to an input/output module for controlling keyboard and mouse functions of the computer100utilizing the redirection controller104. It should also be appreciated that the computers100and200may also comprise other types of computing devices, including hand-held computers, embedded computer systems, personal digital assistants, and other types of computing devices known to those skilled in the art.

Turning now toFIG. 2, an illustrative hardware architecture of the redirection controller104will now be described. As briefly discussed above with respect toFIG. 1, the redirection controller104is communicatively connected to the graphics adapter113for receiving, filtering, and compressing video data which is then communicated to the remote computer200through the network adapter122. As shown inFIG. 2, the redirection controller104may be connected to the graphics adapter113through a digital video or analog video connection. Where an analog video connection is utilized, the redirection controller104may utilize an analog to digital converter117to convert the received analog display screens into a digital format.

In accordance with an illustrative embodiment of the invention, the components of the redirection controller104may be incorporated into a firmware card, such as a PCI card, which is “plugged-in” to the motherboard of the computer100. These components may include a field-programmable gate array (“FPGA”)150and a service processor160. The FPGA150communicates with the service processor160over parallel bus145. The service processor160is a microcontroller that instructs the FPGA150to capture screens of video data and compress changed video data from the video graphics adapter113in accordance with program instructions contained in the compression program code132. Once the changed video data has been compressed, the FPGA150generates and sends an interrupt signal to the service processor160. The service processor160then sends the compressed video data to the remote computer200via the network adapter122. It will be appreciated that the FPGA150and the service processor160may be application specific circuits (“ASICs”) designed for performing the aforementioned tasks. ASICs are well known to those skilled in the art. Those skilled in the art will further appreciate that the redirection controller104may also be incorporated as an external hardware device. The external device may include a video port for connection to a video graphics adapter, keyboard and mouse ports, and a network port (e.g., a network interface card) for connection to a computer network.

Turning now toFIG. 3, two screens300A and300B of pixels representing video data are shown. In particular, the screen300A represents several pixels302A-302D from a previous screen display. The screen300B represents several pixels302E-302H of a current screen display. It should be appreciated that the pixels in each of the screen displays correspond to one another. For instance, pixel302E corresponds to pixel302A, pixel302F corresponds to pixel302B, pixel302G corresponds to pixel302C, and pixel302H corresponds to pixels302D. It should be appreciated that each of the pixels has an associated value that describes the color and intensity of the pixel. For instance, in an embodiment of the invention, a 15 bit value is utilized to represent each pixel.

As will be described in greater detail below, the values of the various pixels302A-302H are utilized to determine a filtered value and an encoded value for a current pixel. The current pixel is the pixel302E and may be referred to herein as “X2.” The pixel corresponding to the current pixel in the previous display screen is the pixel302A and may be referred to herein as “X1.” The pixel immediately to the left of the current pixel is the pixel302F, which may be referred to herein as “A2.” The pixel302B in the previous screen display corresponds to A2. The pixel302B may be referred to herein as “A1.” The pixel to the left and above the current pixel is the pixel302G, which may be referred to herein as “C2.” The pixel302C in the previous screen display corresponds to C2. The pixel302C may be referred to herein as “C1.” The pixel directly above the current pixel is the pixel302H, which may be referred to herein as “B2.” The pixel320D corresponds to B2and may be referred to herein as “B1.” It should be appreciated that the alphanumeric label assigned to each pixel is for convenient reference only.

It should also be appreciated that each pixel in a current display screen is processed in the manner described herein. In this manner, an entire display screen is compressed and transmitted. Once processing for the current display screen has completed, the current display screen becomes the previous display screen and a new display screen becomes the current display screen. In this manner, all pixels of all display screens are processed. It should also be appreciated that the computer200performs a decompression algorithm to decompress the data encoded by the compression algorithm described herein. Decompression is performed in a reverse manner to the compression algorithm described herein.

Turning now toFIG. 4, an illustrative software architecture for the software components of the redirection controller104will be described. It should be appreciated that, according to the embodiments of the invention, the software described herein has been implemented as a FPGA. Alternatively, however, the software operations described herein may be performed by a dedicated hardware circuit, by program code executing on a general-purpose or specific-purpose microprocessor, or through some other combination of hardware and software.

The logical operations of the various embodiments of the present invention are implemented (1) as a sequence of computer implemented acts or program modules running on a computing system and/or (2) as interconnected machine logic circuits or circuit modules within the computing system. The implementation is a matter of choice dependent on the performance requirements of the computing system implementing the invention. Accordingly, the logical operations ofFIGS. 4-7making up the embodiments of the present invention described herein are referred to variously as operations, structural devices, acts or modules. It will be recognized by one skilled in the art that these operations, structural devices, acts and modules may be implemented in software, in firmware, in special purpose digital logic, and any combination thereof without deviating from the spirit and scope of the present invention as recited within the claims attached hereto.

As shown inFIG. 4, the redirection controller104includes a filter402that takes the previous screen display300A and the current screen display300B and generates a filtered value for a current pixel within the current screen display300B. The filtering process generates a predicted value for the current pixel based upon the values of pixels adjacent to the current pixel. An illustrative routine500will be described below with respect toFIG. 5for generating a filtered value for the current pixel.

The redirection controller104also includes an encoder404. The encoder404generates encoded values from predicted values. Depending upon whether lossless or lossy compression is desired, an encoded value is generated. In particular, the encoder404operates in a manner designed to allow the encoded values to be highly compressed. Additional details regarding the operation of the encoder404are provided below with respect toFIG. 6.

The redirection controller104also includes a compressor406. The compressor406receives the encoded values from the encoder404and generates compressed data as its output. Because of the manner in which the encoder404encodes the data stream, the compressor406can utilize a simplified compression algorithm. In particular, according to one embodiment of the invention, the compressor406utilizes a simplified run length encoded (“RLE”) algorithm. Additional details regarding the operation of the compressor406are provided below with respect toFIG. 7.

It should be understood that the process illustrated inFIG. 4is repeated for each pixel location in the current screen of pixel data. Accordingly, at operation408, a determination is made as to whether additional pixels remain to be filtered, encoded, and compressed. If additional pixels remain, the process begins again. Otherwise, the process continues to operation410, where a pending run length of encoded values is flushed by the compressor406. The process then continues to operation412, where the compression process is completed. It will be appreciated by those skilled in the art that by utilizing the above-described method of filtering pixel values, many of the values to be encoded by the encoder404will be runs of repeating zero values which are more easily compressed than non-repeating values.

Referring now toFIG. 5, additional details regarding the operation of the filter402will be described. In particular,FIG. 5illustrates a routine500performed by the filter402for filtering pixel values according to one embodiment of the invention. As described briefly above, the filter402receives a first plurality of pixel values representing video data from a previous video display. The filter402also receives a second plurality of pixel values representing video data from a current video display. The filter402then selects a pixel within the second group of pixels as a current pixel. As mentioned briefly above, the current pixel is referred to herein as “X2.” The filter also identifies several pixels surrounding the current pixel. Pixels corresponding to the identified pixels in the first plurality of pixel values are also identified. In particular, the pixels discussed previously with respect toFIG. 3as A2, B2, C2, X1, A1, B1, and C1are identified by the filter402.

The routine500begins at operation502, where the filter402determines whether the values for pixels A1and A2are equal. If these values are equal, the routine500branches to operation504, where a predicted value for the current pixel (referred to herein as “Xpred”) is set to the value of X1. The routine500then continues from operation504to operation514. If the values of A1and A2are not equal, the routine500continues to operation506.

At operation506, a threshold check is made to determine whether the values of pixels A2and C2are within are within a threshold value (“T”) of one another. In order to perform this analysis, a threshold check routine is provided that returns true if the values of two variables are within a threshold value of one another. Otherwise the routine returns a false value. At operation506, a threshold check is also made to determine whether the values of pixels B2and C2are within a threshold value of one another. As used herein, “BA” refers to the threshold check between A2and C2, and “BB” refers to the threshold check between B2and C2. Based on the values of these comparisons, a prediction can be made about the value of the current pixel.

From operation506, the routine500continues to operation508, where a determination is made as to whether BA is true and BB is false. If not, the routine500branches to operation510, where the value of Xpred is set to the value of A2. If so, the routine500continues to operation512, where the value of Xpred is set to the value of B2. From operations510and512, the routine500continues to operation514, where encoding of the predicted value for the current pixel occurs.

Turning now toFIG. 6, additional details regarding the operation of the encoder404will be described. As discussed briefly above, the encoder404receives the predicted value for the current pixel, Xpred, as input. If the compression for the current screen is to be lossless, the routine600begins at operation604, where an evaluation is made as to whether Xpred is equal to X2. If the compression for the current screen is to be lossy, the routine600begins at operation602, where an evaluation is made as to whether X2and Xpred are within a threshold value of one another. From operations604and602, the routine600continues to operation606.

At operation606, a determination is made as to whether the evaluation made at operations604or602were true. If the respective operation was evaluated as true, the routine600branches to operation608, where the encoded value for the current pixel (referred to herein as “Xenc”) is set to zero. It should be appreciated that zero represents the color black. It should also be appreciated that highly compressible runs of pixel values are created by encoding the value as black where the predicted and actual pixel values for the current pixel are the same or within a threshold value of one another.

If, at operation606, it is determined that the actual and predicted value of the current pixel are not equal or that the actual and predicted values of the current pixel are not within a threshold value of one another, the routine600continues to operation610. At operation610, a determination is made as to whether the actual pixel value for the current pixel is zero. If the actual value is zero, the routine600branches to operation612, where Xenc is set to a value specified to represent the actual color black. In this manner, encoded values actually representing black pixels can be distinguished from encoded values representing runs of identical or close pixel values. According to an embodiment, the value ‘0x8000’ is utilized to represent black. If, at operation610, it is determined that the actual pixel value is not zero, the routine600continues to operation614, where the value of Xenc is set to the value of the current pixel. From operation612, the routine600continues to operation612, where the value of Xenc is set to the value of X2. This operation is also performed at block609. From operations609,613, and614, the routine600continues to operation616, where compression is performed on the encoded value.

Referring now toFIG. 7, additional details regarding the operation of the compressor406will be provided. In particular, the routine700A illustrates a simplified RLE algorithm utilized to compress the encoded values output by the encoder404. The routine700A begins at operation702, where the encoded value, Xenc, is received from the encoder404. The routine700A then continues to operation704, where a determination is made as to whether the encoded value is equal to zero. If the encoded value is equal to zero, the routine700A continues to operation706, where the current run length is incremented. The routine700A then continues to operation712.

If, at operation704, the compressor406determines that the encoded value is not zero, the routine700A continues to operation708, where the pending run is flushed. An illustrative routine700B is described below for flushing the pending run. The routine700A then continues to operation710, where the encoded value, Xenc, is output into the output data stream. The routine700A then continues to operation712, where filtering operations are continued as described inFIG. 5, above.

The routine700B illustrates a process performed by the compressor406for flushing a pending run of data. In particular, the routine700B begins at operation750, where a determination is made as to whether the current run length exceeds a predetermined number of bytes. If the current run length exceeds the predetermined number of bytes, the routine700B continues to operation752, where a first code is output plus the run length. If the current run length does not exceed the predetermined number of bytes, the routine700B branches to operation754, where a second code is output plus the run length. From operations752and754, the routine700B continues to operation756, where it returns to operation710, described above.

It will be appreciated that embodiments of the present invention provide methods, systems, apparatus, and computer-readable medium for filtering, encoding, and compressing video data. Although the invention has been described in language specific to computer structural features, methodological acts and by computer readable media, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific structures, acts or media described. Therefore, the specific structural features, acts and mediums are disclosed as exemplary embodiments implementing the claimed invention.