Optimizing encrypted wide area network traffic

Optimization of encrypted traffic flowing over a WAN is provided by an arrangement in which WAN compression is distributed between endpoints (i.e., client machines or servers) in a subnet of a hub and branch network and a WAN compression server in the subnet. A client portion of the WAN compression running on each of one or more endpoints interfaces with a disposable local cache of data seen by endpoints in the subnet that is used for compressing and decompressing traffic using dictionary-based compression techniques. The local WAN compression server in a subnet stores a shared central database of all the WAN traffic in the subnet which is used to populate local disposable caches in the endpoints.

BACKGROUND

Information technology (“IT”) management in organizations that operate branch offices must accommodate the often-conflicting needs of local-like application performance and manageability versus deployment costs. To reduce total cost of ownership (“TCO”), there is a trend where branch office servers are consolidated, and services and applications are pushed from the LAN (local area network) to being hosted across a WAN (wide area network) from a hub that is commonly located at an enterprise's headquarters location. While such branch and hub architectures can provide substantial cost benefits, the reliance on WAN resources can often lead to depleted bandwidth and increased end-user wait time. This typically results in a reduction in the quality of the user experience at a branch office compared to that at the main office, and an overall loss of productivity in the branch.

One solution to the problem has been to add more wide area bandwidth, and historically data services commonly consume a large portion of enterprise IT budgets. However, incremental increases in bandwidth can carry a disproportionate price increase and limiting factors such as network latency and application behavior can restrict both performance and the return on bandwidth investment.

WAN acceleration solutions have emerged that seek to enable the cost advantages provided by centralized servers without compromising performance by maximizing WAN utilization which can often delay or eliminate the need to purchase additional WAN bandwidth. While WAN acceleration solutions can provide significant benefits and typically represent a good return on investment, current WAN acceleration solutions are incompatible with end-to-end data integrity protocols such as IPsec (Internet Protocol Security) and SMB (Server Message Block) signing that enable secure communications between the branch clients and servers at the hub. While some current solutions are using SSL (Secure Socket Layer) encryption to provide end-to-end security, these solutions relay on deploying a private key in an intermediate device which can increase the vulnerability of a network to what are known as the “man in the middle” attacks.

SUMMARY

Optimization of encrypted traffic flowing over a WAN is provided by an arrangement in which WAN compression is distributed between endpoints (i.e., client machines or servers) in a subnet of a hub and branch network and a WAN compression server in the subnet. A client portion of the WAN compression running on each of one or more endpoints interfaces with a disposable local cache of data seen by endpoints in the subnet that is used for compressing and decompressing traffic using dictionary-based compression techniques. The local WAN compression server in a subnet stores a shared central database of all the WAN traffic seen in the subnet which is used to populate the disposable local data caches in the endpoints.

In an illustrative example, an endpoint intercepts outbound traffic prior to being encrypted. WAN optimization is performed using dictionary-based compression that relies on dictionaries which are locally cached at the endpoints in the subnet, or by using dictionaries that are downloaded from the central database stored on the local WAN compression server. Once optimized, the traffic is passed down the TCP/IP (Transmission Control Protocol/Internet Protocol) stack and is encrypted using IPsec prior to being sent over the WAN link to the remote subnet of the hub and branch network. An endpoint at the remote subnet decrypts, and then decompresses the traffic using locally cached dictionaries, or by using dictionaries downloaded from the central WAN compression server on the remote subnet.

Advantageously, the present arrangement for optimizing encrypted WAN traffic increases WAN utilization to significantly improve the quality of the user experience at the branch subnet while maintaining end-to-end security through IPsec encryption and lowering costs. Furthermore, such performance, security, and cost reduction is achieved without using additional intermediate devices and private keys so as to avoid the man in the middle vulnerability.

Like reference numerals indicates like elements in the drawings.

DETAILED DESCRIPTION

FIG. 1shows an illustrative hub and branch network100in which a branch105is coupled to a hub112over a WAN link116. The term “branch” is used to describe a remote location of any-sized organization that connects to a collection of resources provided by a “hub” located, for example, as part of a main or headquarters operation. Branch105includes a number of client machines1181, 2 . . . Nthat are coupled to a router1201, which places traffic onto the WAN link116and takes traffic from the link in both directions between the branch105and hub112. Client machines118commonly run business and productivity applications such as word processing, email, spreadsheets and the like.

A number of servers1241, 2 . . . Nare configured at the hub112to provide services to the client machines118in the branch105. Such services commonly include those provided by a file server1241, mail server1242and web server124N. However, it is emphasized that these servers are merely illustrative and the actual number and configuration of servers may vary from that shown and will generally be dependent on the requirements of a particular branch-hub deployment. The consolidation of server infrastructure into the hub112typically enables all maintenance, troubleshooting, security policy enforcement, backups and auditing to be performed centrally which can significantly lower TOC for most enterprises.

WAN link116may operate over portions of private networks and/or public networks such as the Internet. WAN116is representative of many current WANs that are commonly utilized to support remote branch operations. Typical WAN issues include high network latency, constraints on bandwidth, and packet loss. Such limitations can constrain branch productivity. In addition, many business or productivity applications operating in the network100were developed for LAN environments and are not particularly WAN-optimized. Consequently, it is recognized that optimizing the utilization of the limited available WAN bandwidth can significantly contribute to better user experience in the branch105. Optimizing WAN traffic provides users with the perception of a quick and responsive network and an overall experience in the branch that is more transparent, seamless, and LAN-like. In addition, many enterprises will benefit from lowered operating costs which result from a decrease in the traffic crossing the WAN link116.

WAN compression servers1261and1262are located in respective subnets (i.e., the branch105and hub112) of the network100in a symmetrical configuration. WAN compression servers126are located in the direct traffic paths at opposite ends of the WAN link116, and are coupled to routers120.

In this illustrative example, WAN compression servers126function to overcome some of the limitations in the WAN link116by optimizing traffic flowing over the link. Such optimization is typically implemented using various techniques, such as stateless and stateful data compression, caching, protocol specific optimizations, data pre-fetching, policy-based routing, quality of service (“QoS”) techniques, and the like.

Data compression algorithms typically identify relatively short byte sequences that are repeated frequently over time. These sequences get replaced with shorter segments of code to reduce the size of the data that gets transmitted over the WAN link. Data compression can be implemented using various methodologies or algorithms including stateless compression such as the well known LZW (Lempel-Ziv-Welch) technique, and stateful compression such dictionary-based compression. Dictionary compression relies on storing all the data passing a compression engine in an external dictionary. In addition to storing the data, the compression engine identifies the data already seen and replaces it with a much smaller reference to an index in the dictionary, thereby enabling subsequent decompression of the data.

Caching entails the WAN compression server126simulating an application server by watching all requests and saving copies of the responses. If another request is made from a client machine118for the same file, the WAN compression server126functions as a proxy and, after validating with the server that the file has not been altered, may serve the file from its cache.

Policy-based routing is commonly used to implement quality of service techniques that classify and prioritize traffic by application, by user, or in accordance with characteristics of the traffic (e.g., source and/or destination addresses). In combination with queuing, policy-based routing can allocate available WAN bandwidth to ensure that traffic associated with some applications does not disrupt enterprise-critical traffic. Prioritization may be implemented, for example, using policy-based QoS to mark outbound traffic with a specific Differentiated Services Code Point (“DSCP”) value. DSCP-capable routers read the DSCP value and place traffic being forwarded into a specific queue (e.g., a high-priority queue, best effort, lower than best effort, etc.) that are serviced based on priority.

The particular techniques utilized can vary by deployment, but most types of compression servers commonly utilize data compression in one form or another. Data that is encrypted, however, is generally perceived as random data by compression algorithms, which makes it virtually impossible to compress.

As encrypted traffic is not suited for compression, the hub and branch arrangement shown inFIG. 1could be arranged to do without acceleration of encrypted traffic which will typically result in a significant performance penalty. Alternatively, traffic could be accelerated but then not encrypted which can present a significant security vulnerability.

Another alternative is to utilize intermediate devices or servers at both the branch and hub which terminate SSL (Secure Socket Layer) traffic and then decrypt, store segments of the data for future reference, and re-encrypt it. Later traffic through the devices is compared with these segments. When data being sent matches a segment, the devices send a compact reference rather than the longer complete segment, thereby reducing the amount of traffic that has to cross the WAN link. In some cases, devices use the private key of the server to decrypt the session key that is used over the WAN link.

While use of SSL can provide desirable end-to-end security for traffic between the branch and hub, the intermediate devices suffer from several drawbacks. The stored segments are typically stored in unencrypted form which can present some security vulnerability. In addition, by putting the private key on the intermediate device, there is increased risk that security holes may be opened and accessed through the device in a man in the middle attack.

FIG. 2shows an illustrative hub and branch network200in which WAN compression is implemented using software2061, 2 . . . Nthat is located at each of the client machines218and servers224(termed “endpoints”) in the network. In this illustrative arrangement, the WAN compression servers are removed from the traffic paths over WAN link216. Instead, traffic at each endpoint is compressed before being handed off to a TCP/IP stack, encrypted (e.g., using IPsec), and placed onto the WAN link216by routers220. At the receiving endpoint, the traffic is decrypted, decompressed and delivered to the appropriate process or application. Therefore, IPsec-secured traffic traverses the WAN link216between endpoints as indicated by reference numeral230(while secure traffic is illustratively shown between endpoints218Nand224N, it is noted that such traffic may typically flow between any endpoint in the branch205and any endpoint in the hub212).

While moving WAN compression to the endpoints provides end-to-end security for traffic, one of the principal advantages provided by the WAN compression servers shown inFIG. 1is lost. Specifically, data reduction and caching is unable to be performed on a subnet-basis and instead is limited to traffic that is seen at a particular endpoint. In addition, the size of the cached data, even when scoped to a single machine may present a substantial impact to the available resources particularly as the client machines218may be limited in terms of processor power, memory, and storage.

The limitations of the WAN compression servers and WAN compression performed at the endpoints shown respectively inFIGS. 1 and 2and described in the accompanying text are overcome by the present arrangement for optimizing encrypted WAN traffic shown inFIGS. 3-8and described in the accompanying text.

FIG. 3shows an illustrative hub and branch network300in which WAN compression is implemented using a distributed architecture. WAN compression servers3261and3262are respectively used in the branch305and hub312. As shown inFIG. 4, each WAN compression server326is arranged to store a shared central database405of previously seen traffic in the subnet as dictionaries that are indexed to support dictionary-based compression. It is emphasized that while dictionary-based compression is used in this example, the present arrangement for optimizing encrypted WAN traffic is not limited to dictionary-based compression, and other compression techniques may be used as may be required to meet the needs of a particular implementation. The WAN compression servers326are also arranged to perform conventional WAN optimization, as indicated by reference numeral416, for non-encrypted traffic traversing the WAN link316via routers3201and3202(FIG. 3).

Returning again toFIG. 3, a client portion (as indicated by reference numeral3061, 2 . . . N) of the WAN compression is also included on one or more of the endpoints (i.e., client machines3181, 2 . . . Nand servers3241, 2 . . . N) in the network300so that traffic is intercepted and compressed prior to being encrypted and sent over the WAN link316. In this illustrative example, the traffic is encrypted using IPsec as indicated by reference numeral330.

FIG. 5shows a set of illustrative components that are implemented at an endpoint in the network. The endpoint shown inFIG. 5is a client machine318, however it is emphasized that similar components are typically implemented in a symmetric manner at endpoints in the opposite subnets of the network300. The client portion316, interfaces with a database cache505and a traffic interception functionality513, as shown.

Traffic seen by the endpoint is stored in a disposable local cache505in accordance with a caching policy that is formulated to limit the possibility that the client machine's resources will not be overused. Cache505is used by an endpoint when performing dictionary-based compression and decompression, and may be supplemented or updated with dictionaries from the central database405(FIG. 4) when necessary, as described below.

The traffic interception functionality513is utilized to intercept traffic traversing the WAN link316to and from the hub312so that the client portion3061can compress outbound traffic and decompress inbound traffic to the endpoint. Traffic interception functionality513is accordingly arranged to interface with the TCP/IP stack520on the endpoint. TCP/IP stack520, in turn, interfaces with an IPsec driver526that sends and receives IPsec-protected IP packets532over the WAN link316via the routers320.

The traffic interception functionality513may be implemented differently depending upon the operating system (“OS”) that is utilized by a particular endpoint. For example, as shown inFIG. 6, for endpoints using Microsoft® Windows XP® and Windows Server® 2003 on the respective client machines318and servers324utilize a Transport Driver Interface605(“TDI”) which is a common interface used to communicate with the network transport protocols in the TCP/IP stack520. TCP/IP stack520includes a Transport Layer612that contains implementations of TCP and UDP (User Datagram Protocol) and a mechanism to send raw IP packets that do not need a TCP or UDP header. TCP/IP stack520also includes a Network Layer616that contains implementations of Internet Protocols, such as IPv4 or IPv6. A Framing Layer619contains modules that frame IPv4 or IPv6 packets.

FIG. 7shows an alternative illustrative implementation of the traffic interception functionality that is utilized, for example by endpoints using Microsoft Windows Vista™ or Microsoft Windows Server® 2008 (formerly known by its codename “Longhorn”) OSs. Here, the TCP/IP stack520is arranged to expose a Windows Filtering Platform (“WFP”) Callout API710(application programming interface) that enables the client portion306of the WAN compression to access packet processing paths at both the Transport Layer612and Network Layer616. It is noted, however, that TDI is also supported by Windows Vista and Windows Server 2008 to maintain backwards compatibility for TDI-based clients and processes.

FIG. 8is a flowchart of an illustrative method800for implementing the present distributed WAN compression in the hub and branch network300shown inFIG. 3. The method starts at block805. At block810, the client portion306of the WAN compression on an endpoint (i.e., either a client machine318or server324) is initiated. The client portion306performs discovery of the local WAN compression server326in the subnet (i.e., either the branch305or hub312) by either broadcasting within the subnet, or by having the name of the WAN compression server pre-configured in the endpoint.

At block815, once a connection is successfully established with the local WAN compression server326, the endpoint downloads a list of known peers (i.e., other endpoints in the network that are also running an instance of the client portion306of the WAN compression). The endpoint also downloads signatures of the most recently seen data at each peer.

The endpoint then determines if traffic originating from the endpoint is going to be encrypted at block820. In this illustrative example, IPsec is utilized. Thus, the determination may be accomplished by checking the IPsec policy. Alternatively, the network data may be examined and analyzed. In both cases, the goal of the determination step is to identify one or more encrypted streams that are going from the local subnet over the WAN link316.

At block825, if a stream is destined for an unknown remote endpoint, an auto-discovery routine is initiated, for example, using a reserved TCP/UDP port, or through use of other methods for facilitating endpoint discovery. If the remote endpoint is discovered to include a client portion306of the WAN compression, then the address of the remote endpoint is reported back to the local WAN compression server326. In a symmetrical manner, the address of the endpoint initiating the auto-discovery routine is reported to the remote WAN compression server326.

At block830, compression is applied to the traffic using one of various alternative techniques. Such techniques include, for example, LZW compression applied to a packet or group of packets, or dictionary-based compression. Compression may further be applied to packets across the same TCP stream using, for example, a conventional proxy approach or using a collective operation such as gather/scatter.

As indicated in blocks830A-F, the compression step includes substeps which take a number of considerations into account. At block830A, the endpoint is configured to remember data that is currently seen by it and other peers in the subnet. The data is cached locally in the traffic database cache505(FIG. 5) and periodically uploaded to the central traffic database (e.g., database405inFIG. 4) at the local WAN compression server326, along with signatures described in the text accompanying block815.

At block830B, existing locally cached dictionaries (e.g., those in cache505inFIG. 5at each of the endpoint peers) are used to compress traffic being sent from the endpoint. Current signatures downloaded from the local WAN compression server326in the subnet are used to determine the best set of dictionaries to perform the compression. Dictionaries that are cached locally will be immediately used to perform compression. If there are insufficient locally cached dictionaries in the subnet, then the endpoint can download an appropriate portion of the central dictionary (e.g., from the central database405inFIG. 4). Each endpoint may further be configured to dispose of those dictionaries from its cache, for example, that are not used for a period of time equaling some predetermined threshold or using some other disposal criteria or methodologies.

At block830C, the existing cached dictionaries in the endpoint peers in the subnet are used to decompress traffic received from remote endpoints. Dictionaries which are used by a remote endpoint to compress traffic which are not available in the local caches505(FIG. 5) in the receiving endpoint are downloaded from the WAN compression server326. These downloaded dictionaries are cached in the endpoint dictionary caches505for possible future use. The endpoint performing the compression may be further configured to provide hints to the endpoint performing the decompression to enable that endpoint to pre-fetch the appropriate dictionaries that will be needed to perform the decompression.

At block830D, the one or more of the endpoint in the subnet will periodically refresh the current list of signatures and known peers from the local WAN compression server326. This is typically accomplished by downloading changes in the form of deltas between the stored signature and peer list and the refreshed list. At block830E, one or more of the endpoints will periodically upload their dictionaries from the local cache505(FIG. 5) to the central database405(FIG. 4) in the local WAN compression server326.

At block830F, the WAN compression servers326in each subnet communicate with each other over the WAN link316to synchronize the purging of data from their respective central traffic databases as it becomes obsolete. In addition, as noted above, the WAN compression servers326may perform WAN compression and optimization of non-encrypted traffic using, for example, conventional IP tunneling techniques. The illustrative method800ends at block835.