Inter-cache communication using HTTP resource

Servicing resource requests. A method includes at a first caching node, receiving a request for a first resource. The method further includes at the first caching node, determining at least one of health or load information of the first caching node. The method further includes generating a response to the request for the first resource, including generating a header to the response. The header is a custom header. The custom header includes at least one of health or load information. The method further includes sending the response to the request including sending the custom header with the at least one of health or load information.

BACKGROUND

Background and Relevant Art

Further, computing system functionality can be enhanced by a computing systems ability to be interconnected to other computing systems via network connections. Network connections may include, but are not limited to, connections via wired or wireless Ethernet, cellular connections, or even computer to computer connections through serial, parallel, USB, or other connections. The connections allow a computing system to access services at other computing systems and to quickly and efficiently receive application data from other computing systems.

To access a service on the internet, a user at a client machine will typically type a domain name into an address input of a user interface such as a browser. This domain name can be converted to an IP address at a domain name service (DNS), which can then be used to access the service on the internet.

Often, users may attempt to access a service that is physically located a great distance from the use's client machine. For example, a user in Japan may attempt to access a service in the United States. Generally, this presents no real difficulties so long as the amount of data being provided by the service is minimal, such as a minimal amount of text. Only a small amount of data must travel the long distance from the United States to Japan. However, when larger amounts of data, such as large text files, pictures, videos, audio recordings, and the like are requested, this can contribute to worldwide network congestion. In particular, a single set of data may occupy network bandwidth on multiple networks between the client and the service, where the greater the distance between the client and the service, the more network resources are likely to be required. Additionally, even for smaller amounts of data there are network latency problems. Simply the fact that data travels a large distance across a network results in higher latency of data transmission.

To mitigate this congestion and latency, services will often implement a distributed caching system. In particular a service may have a cache located in closer geographical proximity to a client than the actual service. For example, a client in Osaka, Japan may access data from a cache in Tokyo, Japan for a service in the United States. Other data caches for the same service in the United States may be located at other geographical locations to allow clients to obtain cached data from the service from a cache in closer proximity to the client than the service.

Caching systems may also be organized in an hierarchical fashion. For example, caches may have an hierarchy where caches higher in the hierarchy cover larger geographical regions or more networks than caches lower in the hierarchy. For example, at a first tier is the service or origin located in Redmond, Wash. which provides the service for the entire world. At a second tier is a cache in Singapore that caches data for clients and lower level caches in Asia. At a third tier is a cache in Tokyo, Japan which caches data for clients and lower level caches in all of Japan. At a fourth tier is a cache in Osaka, Japan that caches data for clients in Osaka itself.

Cache hierarchy is designed with an assumption that only a subset of requests are routed to the next tier, meaning each tier is being served as a “filter”. For example, consider a 3 tier topology, with a child cache server tier, a parent cache server tier and an origin server. In this example, the child cache server tier will receive 100% of the end-user traffic, while the parent cache server tier will receive something less than 100% (say for example 30%, meaning there was 70% cache hit or 70% of the requested resources have already been cached and will therefore be served directly from the child cache server tier), and the origin server will receive even less, say for example 10%. Based on this assumption, the scale and the capacity planning of a cache hierarchy is done accordingly.

However, this assumption can be wrong, and in particular can be affected by acute conditions. For example, consider the case when news of celebrity death first begins to spread. This results in many users requesting the same information from a news server at the same time. This results in many fresh content (i.e. not cached) requests by a large number of clients. In such event, the content has not been cached by the child servers or parent servers. In the example above, the parent may receive 90% of the traffic (not 30%) and the origin server may receive 70% of the traffic (not 10%). In this case, there is a cascading effect of cache-miss requests that may eventually overload the cache hierarchy.

Another example could involve an unexpected degradation in service, such as network and/or hardware failure where the level of service (as indicated with the health of the system) is sub-optimal. Thus, as opposed to the increase in demand, there is a decrease in performance/availability/reliability that negatively impacts the overall “health”. Still other scenarios, though not illustrated specifically here, may occur.

BRIEF SUMMARY

One embodiment described herein is directed to a method practiced in a computing environment including a distributed caching topology. The method includes acts for servicing resource requests. The method includes at a first caching node, receiving a request for a first resource. The method further includes at the first caching node, determining at least one of health or load information of the first caching node. The method further includes generating a response to the request for the first resource, including generating a header to the response. The header is a custom header. The custom header includes at least one of health or load information. The method further includes sending the response to the request including sending the custom header with the at least one of health or load information.

Another embodiment described herein is directed to a method practiced in a computing environment including a distributed caching topology. The method includes acts for servicing resource requests. The method includes at a first caching node, receiving a request from a user for a first resource. The method further includes checking a local cache at the first caching node for the resource and determining that the resource is not available at the local cache of the first caching node. The method further includes as a result, checking information about the health of a second caching node and based on the health information of the second caching node determining whether to request the resource from the second caching node or a different caching node.

DETAILED DESCRIPTION

Some embodiments illustrated herein may be able to prevent a flood of cache-miss requests that overwhelms a system by leveraging response headers between caches or caches and origin servers to include health and performance information so that the “previous” tier servers can make alternate decisions rather than simply forwarding the cache-miss requests to the “next” tier servers.

Embodiments may include the ability to inject health and performance information in custom headers from “next” (i.e. parent) tier cache servers to “previous” (i.e. child) tier cache servers. Embodiments may include the ability for the “previous” tier servers to make alternate/intelligent decisions based on the information included in the custom headers. Embodiments may include the ability for the “previous” tier servers to remove the custom headers before sending the responses to a client.

Referring now toFIG. 1A, an example cache hierarchy100is illustrated. In the example illustrated inFIG. 1A, the hierarchy100is based on geographical location and serves to establish caches at various geographic locations. In the example illustrated, an origin server102is illustrated in Redmond Wash. In a distributed caching system a cache server with content in the origin server102is located in closer proximity to a client than the actual origin server102. For example, a client in Osaka, Japan may access data from a cache in Tokyo, Japan for a service in the United States. Other data caches for the same origin server102in the United States may be located at other geographical locations to allow clients to obtain cached data from the origin server102from a cache in closer proximity to the client than the service.

Below the origin server are a number of second tier cache servers,104-1,104-2,104-3,104-4,104-5,104-6, and104-7(second tier servers may be referred to herein generically as104) located at various geographical locations throughout the world. Further still, a number of third tier cache servers,106-1,106-2,106-3,106-4,106-5,106-6,106-7,106-8,106-9, and106-10(third tier servers may be referred to herein generically as106) below the second tier cache servers104are located at still other geographical locations. The third tier cache servers106may subdivide a geographical region. For example, cache server104-7may be a cache server located in Singapore, but that is intended to serve all or large parts of Asia, while cache servers106-9and106-10each serve smaller portions of Asia.

However, it should be noted, that for load balancing purposes, different cache servers may serve areas outside of their principle responsibilities. Further, it should be noted that cache servers may be organized in a fashion other than one that is strictly, or even primarily geographic. A cache server hierarchy may be based on network traffic, network connection types, network speed, or any of various other factors.

Further still, it should be noted that in a cache server hierarchy, each lower level (or child) cache server is not necessarily dependant on a particular higher level (or parent) cache server. Thus, for example, while cache server106-10might typically contact cache server104-7when a cache miss occurs at cache server106-10, it can contact other cache servers or the origin server102. For convenience,FIG. 1Billustrates a more traditional hierarchical view of the hierarchy100which illustrates some interconnections allowing lower level cache servers to contact various higher level cache servers.

Embodiments may be implemented where cache servers are proxy based. In particular, a cache server will act as a proxy for a system below it. For example, cache server106-10will act as a proxy for a client machine to the cache server104-7, which will act as a proxy for the cache server106-10and the client to the origin server102. As such, both the requests for content or services and responses to the requests flow through the proxy. Given this traffic pattern, when a parent cache server introduces a custom header that includes server health and performance information about the parent cache server, the custom header can be used by a child cache server to make alternate, more intelligent, routing decisions to avoid overwhelming the parent cache server.

For example, a client may request content or services from the cache server106-10. The request may be a request that requests content or services not available at the cache server106-10, but which are available from the origin server102. Thus, the request to the cache server106-10results in a cache miss. The cache server106-10may then act as a proxy for the client to the next level cache server, such as cache server104-7.

The cache server104-7may have health and/or load issues which mean that the cache server104-7may not be an appropriate cache server choice. Similarly, the cache server104-7may have health and/or load issues that, in fact, make it a very appropriate cache server for higher level requests. In replying to the cache server106-10, the cache server104-7may include a custom header that indicates various conditions about the cache server104-7to the cache server106-10. The cache server106-10can then use this information to make intelligent decisions as to what higher level cache server to forward future requests.

In some embodiments, the cache server104-7may have health or load issues that prevent it from responding to a request from the cache server106-10. The reply from the cache server104-7to the cache server106-10may reject a request, where the rejection includes an indication of load or health information in the response headers of the rejection.

Cache servers may also include the ability to rewrite response headers. For example, a child cache server (e.g. cache server106-10) can remove the custom header that the parent cache server (e.g. cache server104-7) has introduced before forwarding the response to the client. This can be used, for example, to eliminate any end-user impact caused by the custom header information. This can be used to eliminate traces that such communication is taking place between the child and the parent cache servers. This may be desirable to prevent knowledge about the topology of the network to be discovered, for hackers or other attackers to be able to identify weak links in a cache server network fabric. In some embodiments, health and load information may be encrypted in the header such that nefarious entities are not able to obtain the information.

However, it should be noted, that in some embodiments it may be useful to allow the response header, including the health and/or load information to be forwarded to a client machine. The client machine may include functionality for determining appropriate cache servers in the first instance.

Illustrating now further details, in content distribution and caching, the performance, scalability and responsiveness are often affected by several factors. Such factors may include one or more of memory utilization on the cache server; CPU utilization on the cache server; network utilization on the cache server; disk i/o queue length on the cache server; HTTP queue length on the cache server; number of files on disk on the cache server; number of file handles open in memory on the cache server, etc.

Information about these factors, which include health and load factors, may be available by using performance counters on servers. By including information about health and/or load, as well as a possible recommendations in a custom header, the parent cache server can inform a child cache server how to make better routing decisions so that it does not overwhelm an entire cache hierarchy100environment.

The following illustrates a custom header including health and load information. In the present example, for ease of illustration, the health and load information is underlined for convenience of the reader.

With this information received from the parent cache server, the child cache server may decide that the disk queue length of 30 on the parent cache server is too high. And therefore, for the next 10 requests, it will route to another cache server in the cache hierarchy100topology rather than sending additional requests to the same parent cache server, which may overload the parent cache server, and eventually the entire cache hierarchy100environment. Thus, in this example a predetermined number of requests may be routed to a different cache server.

In addition, the child cache server has the capability to remove this custom header before sending it back to the client, so that the client is unaware of any inter-cache communications that may take place between the cache servers in the cache hierarchy topology.

Details are now further illustrated with reference toFIG. 2.FIG. 2illustrates a higher level cache server204and a lower level cache server206in a cache hierarchy. The lower level cache server206is connected, such as through a network or other appropriate connection, to a client208.

The client208sends a resource request210to the lower level cache server206requesting some resource, such as a file, web page, style sheet, image, etc. The lower level cache server206experiences a cache miss, meaning that it does not have in local storage, one or more of the resources requested in the resource request210. As a result, the lower level cache server206sends its own resource request212to the higher level cache server204. The resource request212in some embodiments may be very similar or nearly identical to the resource request210except with some changes to routing information in the header.

The higher level cache server204may have the resources originally requested by the client208and therefore returns a response214. The response214may include the requested resources, an error message, or other appropriate response. The response214further includes a custom header215. The custom header215may include various header information as is illustrated in the sample header above. Further, as illustrated above, the custom header215includes at least one of health or load information about the higher level cache server204. Notably, if the higher level cache server204has health or load information for one or more other cache servers, the custom header may include, alternatively or additionally, at least one of health or load information for one or more of the other cache servers.

In another example, a response from a cache server from the higher level cache server204may include all or parts of the custom header, which are then simply forwarded with no or little modification to the lower level cache server. For example, the higher level cache server may itself experience a cache miss, and thus may request the originally requested resource from an even higher level cache server, a sibling cache server, or in some embodiments, even a lower level cache server. A cache server responding to a request from the higher level cache server204may send a response where the response includes a custom header including at least one of health or load information of the responding cache server (and/or as can be imagined iteratively following the present example, at least one of health or load information of yet another caching server).

However, returning once again to the example explicitly illustrated inFIG. 2, the lower level caching server206may send a response216to the client208. The response may include resources, error messages, and the like that were received in the response214. In fact, the response216may be identical or nearly identical to the response214except for changes in routing information or other contextual information. In the illustrated example inFIG. 2, the response216differs from the response214in at least that it does not include the custom header115including the health or load information. While the response216may include a header, the header may not include the same information as the custom header215, for security or other reasons.

In a subsequent resource request scenario, the health or load information may be used by the lower level cache server. For example, the client208may send a resource request218. While in this example, the client208is illustrated as sending both the resource request210and218, it should be appreciated that different clients could send the different resource requests with similar outcomes. In particular a first client could send the resource request210and a different second client could send the resource request218. The resource request218may request one or more resources such as a web page, image, style sheet, script code, etc. from the lower level cache server206. The lower level cache server206may experience a cache miss in that it does not have a resource requested in the resource request218. Thus, the lower level cache server206may need to request resources, using a resource request220, from another cache server. The lower level cache server206can use the health or load information obtained in the custom header215to determine if the resource request220should be sent to the higher level cache server204or to another different cache server. For example, if the health or load information from the custom header215indicates that the higher level cache server204is healthy and/or not overloaded, the resource request may be sent to the higher level cache server. Otherwise, the resource request may be sent to a different cache server.

Referring now toFIG. 3, a method300is illustrated. The method300may be practiced in a computing environment including a distributed caching topology. The method300includes acts for servicing resource requests. The method300includes at a first caching node, receiving a request for a first resource (act302). For example, a client or other caching server may send a request for one or more resources, such as a web page, image, style sheet, script code, etc.

The method300further includes at the first caching node, determining at least one of health or load information of the first caching node (act304). Such information may include, for example, one or more of memory utilization for the first caching node; CPU utilization for the first caching node; network utilization for the first caching node; disk i/o queue length for the first caching node; HTTP queue length for the first caching node; number of files on disk at the first caching node; or number of file handles open in memory for the first caching node.

The method300further includes generating a response to the request for the first resource, including generating a header to the response. The header is a custom header. The custom header includes the at least one of health or load information (act306). An example custom header is illustrated above.

The method300further includes sending the response to the request including sending the custom header with the at least one of health or load information (act308). For example, a response may be sent to a client or another caching node where the response includes the custom header with health or load information.

The method300may be practiced where receiving a request for a first resource includes receiving a request from a child caching node. For example, as illustrated inFIG. 1B, the caching node104-4may receive a request from a child caching node106-2.

In an alternative embodiment, the method300may be practiced where receiving a request for a first resource comprises receiving a request from a sibling caching node. For example, the caching node104-4may receive a request from the caching node104-6. Or, the caching node106-2may receive a request from the caching node106-3.

Referring now toFIG. 4, a method400is illustrated. The method400may be practiced in a computing environment including a distributed caching topology. The method400includes acts for servicing resource requests. The method includes at a first caching node, receiving a request from a user for a first resource (act402). For example, a caching node may receive a request from a caching node lower in a cache hierarchy or directly from a user at a client machine.

The method400further includes checking a local cache at the first caching node for the resource and determining that the resource is not available at the local cache of the first caching node (act404).

The method400further includes as a result, checking information about at least one of health or load of a second caching node and based on the health or load information of the second caching node determining whether to request the resource from the second caching node or a different caching node (act406). For example, when the caching node106-2determines that it does not have a requested resource available at the caching node106-2, the caching node106-2may check information that it has stored about caching node104-4to determine the health or load at caching node104-4. If the caching node104-4appears to be in an unhealthy or overloaded state, then the caching node106-2may determine to request a resource from a different caching node than the caching node104-4. The resource request may be sent to a caching node that is higher in the hierarchy100(such as caching node104-5) or a caching node that is at the same level in the hierarchy, such as a sibling caching node (such as caching node106-1) or any other appropriate caching node in the hierarchy100.

The method400may be practiced where the second caching node is a parent node of the first caching node. In the example illustrated above, the first caching node may be node106-2, and the second caching node may be caching node104-4. Embodiments of the method400may be practiced where the second caching node is a sibling to the first caching node. In the example illustrated above, the first caching node may be node106-2, and the second caching node may be caching node106-3.

The method400may be practiced where determining whether to request the resource from the second caching node or a different caching node includes checking the at least one of the health or load information of a plurality of siblings of the first caching node. For example, the caching node106-2may check the health of nodes106-1and106-3.

The method400may further include receiving at least one of health or load information from the second caching node. For example, the health or load information that is being checked may be received directly from the node to which it applies. For example, node104-4may send health or load information to the node106-2. In some embodiments, receiving at least one of health or load information from the second caching node includes receiving the at least one of health or load information in a custom response header. Still further, in some embodiments, the at least one of health or load information is received as a result of a request for a different resource than the first resource. For example, as illustrated above, a client or caching node, for example node106-2, may request a resource from another caching the node, for example node104-4, as a result of a cache miss. The other caching node104-4may send back the resource with the health or load information in a custom header. When the caching node106-2experiences another cache miss for a different resource, the caching node106-2may consult the health or load information previously received to determine if a request for the different resource should be sent to the caching node104-4or some other caching node.

Embodiments may further include removing the at least one of health or load information from the custom response header prior to forwarding a response to the user. For example, as illustrated inFIG. 2, load or health information in the custom header215may be stripped out by the lower level cache server206before the response214is forwarded on to the client208as the response216.

The method400may be further practiced to include storing the at least one of health or load information locally for use in subsequent dynamic routing of resource requests. For example, as illustrated inFIG. 2, the lower level cache server206may store the health or load information at local storage or in local memory for use in determining how subsequent requests (such as request218) are handled.

The method400may be practiced where at least one of the health or load information includes acute health information indicating an immediate or more short term condition. For example, the health information may indicate the immediate or imminent failure of a hardware component at the caching node.

Alternatively, the method400may be practiced where at least one of the health or load information includes chronic health information. For example, the health or load information may indicate a trend of increasing requests, that does not create an immediate problem, but which could gradually create future problems.

Alternatively, the method400may be practiced where at least one of the health or load information includes future health information. For example, the health information may indicate that a caching node will be shut down for maintenance at a given time. Health information could identify planned tasks that could impact the health such as anti-virus and/or anti-spam scanning operation, disk de-fragmentation tasks, etc.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission computer readable media to physical computer readable storage media (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer readable physical storage media at a computer system. Thus, computer readable physical storage media can be included in computer system components that also (or even primarily) utilize transmission media.