Patent Publication Number: US-9898477-B1

Title: Writing to a site cache in a distributed file system

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation-in-part of and claims priority to co-pending U.S. patent application Ser. No. 14/673,071, filed Mar. 30, 2015, which claims benefit of U.S. Provisional Patent Application 62/088,427, filed Dec. 5, 2014. All of the foregoing applications are incorporated by reference in their entirety. 
    
    
     FIELD OF THE INVENTION 
     This invention relates generally to distributed file systems, and more particularly to systems and methods for writing data objects to a site cache associated with the distributed file system. 
     BACKGROUND OF THE INVENTION 
     Distributed file systems manage files and folders spread across multiple computers. They may serve a similar function as traditional file systems, but are designed to provide file/folder storage and controlled access over local and wide area networks. Some individuals and/or enterprises may rely on distributed file systems to manage their personal and/or organizational data. 
     There is a need, therefore, for an improved method, article of manufacture, and apparatus for accessing a distributed file system. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which: 
         FIG. 1  depicts a system architecture for a distributed file system in a cloud environment consistent with an embodiment of the present disclosure 
         FIG. 2  depicts a system architecture for a distributed file system using a site cache consistent with an embodiment of the present disclosure. 
         FIG. 3  depicts a detailed site cache architecture consistent with an embodiment of the present disclosure. 
         FIG. 4  depicts a process for reading from a site cache consistent with an embodiment of the present disclosure. 
         FIG. 5  depicts a process for writing to a site cache consistent with an embodiment of the present disclosure. 
         FIG. 6  depicts a varying site cache volume consistent with an embodiment of the present disclosure. 
         FIG. 7  depicts a process for managing a site cache&#39;s state consistent with an embodiment of the present disclosure. 
         FIG. 8  depicts an eviction process consistent with an embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. While the invention is described in conjunction with such embodiment(s), it should be understood that the invention is not limited to any one embodiment. On the contrary, the scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications, and equivalents. For the purpose of example, numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. These details are provided for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the present invention is not unnecessarily obscured. 
     It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein computer program instructions are sent over optical or electronic communication links. Applications may take the form of software executing on a general purpose computer or be hardwired or hard coded in hardware. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. 
     An embodiment of the invention will be described with reference to a data storage system in the form of a storage system configured to store files, but it should be understood that the principles of the invention are not limited to this configuration. Rather, they are applicable to any system capable of storing and handling various types of objects, in analog, digital, or other form. Although terms such as document, file, object, etc. may be used by way of example, the principles of the invention are not limited to any particular form of representing and storing data or other information; rather, they are equally applicable to any object capable of representing information. 
     Embodiments of the present disclosure provide a site cache for processing IO requests on a distributed file system. Distributed files systems may be used to manage files, folders, and other data spread across multiple computing systems. They may be presented to users, applications, or other clients as traditional file systems, but may actually provide access to data over local and wide area networks. For example, the data could be stored in a cloud based object stores, such as Amazon S3, Microsoft Azure, Google Drive, a private object store, and/or a hybrid object store. Access to the data on these object stores may be managed by a metadata server, which could be a local or remote server from the client. 
     While cloud based object stores may create the appearance of a single object store, the data may actually be physically stored across multiple datacenters that are geographically diverse. For example, portions of data may be stored at data centers in both California and Arizona, while still being part of the same logical object store. Clients that are not physically located near an object store may experience a negative performance impact when issuing IO requests to that store. For example, it may take longer for a client in New York to write data to an object store located in California than it would if the object store were located at the client&#39;s site. The present disclosure provides a site cache to compensate for this performance degradation. 
     An embodiment of the present disclosure provides a site cache at client sites. For example, if clients are spread across multiple branch offices, each branch may have its own site cache. Access to the site cache may be provided to each client over a high bandwidth, low latency connection. This may be contrasted with a lower bandwidth, higher latency connection to an authoritative object store, such as a public cloud. Clients may issue IO requests to the site cache via a metadata server, and the site cache may service those requests if it is able. This allows the clients to realize the performance benefits of a local object store, while still being able to access a geographically diverse distributed file system. 
       FIG. 1  depicts a system implementing a distributed file system in a cloud environment. The system of  FIG. 1  may include client  100 , metadata server (“MDS”)  102 , and object store  104 . While only one client  100  is shown, the system may include multiple clients accessing the distributed file system. Similarly, the system may include multiple object stores  104  and/or multiple MDS  102 . 
     Client  100  may be any general purpose computing device. For example, client  100  may be a personal computer, workstation, handheld computer, smart phone, and/or tablet computer. Additionally or alternatively, client  100  may be a software module or application running on a general purpose computing device. Client  100  may be in communication with a MDS  102  and object store  104  over a network connection, such as a local area network (“LAN”) or wide area network (“WAN”), or via any other form of communication. Client computer  100  may interact with the distributed file system as it would with a traditional file system, such as by writing data to and reading data from the distributed file system. 
     MDS  102  may be a general purpose computing device managing distributed file system metadata. This metadata could include, for example, the location of data stored in the distributed file system. MDS  102  may be a physical or a virtual machine, and may operate in an environment local to or remote from client  100 . For example, MDS  102  may be a virtual machine operating in the same datacenter as client  100 . Additionally or alternatively, MDS  102  may operate in a third party cloud environment, such as Amazon Web Services (“AWS”). In some embodiments, MDS  102  may operate in the same third party cloud environment as object store  104 . 
     Object store  104  may comprise a storage location for storing data in the distributed file system. Object store  104  may be a private, public, or hybrid cloud environment capable of storing data. A private cloud may be an object store only available to clients belonging to a particular enterprise. For example, a private cloud may be an OpenStack Swift instance operating in a datacenter completely under the control of an enterprise. The install, including the associated data and services, may not be accessible to anyone outside of the enterprise. A public cloud may be any object store accessible to the public that requires authentication to access certain data. For example, Amazon S3 is available to members of the public but data stored in the object store is only accessible by authorized clients. A hybrid cloud may be a combination of a private and public cloud, such that some data is stored in the private cloud and other data is stored in the public cloud. 
     In some embodiments, client  100  may transmit communications to and receive responses from MDS  102 . Similarly, client  100  may transmit communications to and receive responses from object store  104 . Typically these communications will be IO requests and responses, such as read/write communications, though any other type of communication is consistent with the present disclosure. 
     For example, client  100  may decide to read data from the distributed file system. Client  100  may first mount the distributed file system by transmitting a mount request and/or intent to MDS  102 . Similarly, if the distributed file system has already been mounted, client  100  may transmit a change location/directory request to MDS  102 . In response, MDS  102  may consult a metadata table to determine data objects located at the root of the mount or in the new location, and transmit information related to the data back to client  100 . This data could be, for example, a list of files and/or directories located at the root or new location. The data may also include a unique identifier for each data object, such as a hash and/or path of the object. 
     Once client  100  has a list of files and/or directories, client  100  may select a data object to read. Client  100  may transmit a read request identifying the desired data object back to MDS  102 . In some embodiments, this read request may include a path or hash identifier for the data object the client desires. Once MDS  102  receives the request, it may attempt to locate the data object on the distributed file system. 
     In an embodiment, MDS  102  maintains location data for all of the data objects in the distributed file system. This location data may be maintained with other data object metadata in a database on MDS  102 . For example, the database may comprise a table mapping a data object to one or more object store locations. These object store locations could reside, for example, on object store  104 . 
     In response to the read request received from client  100 , MDS  102  may consult the database table to determine the object location. MDS  102  may then return the object location back to client  100 . In an embodiment, the object location returned might be a URL the client may use to access all or part of the data object. For example, the URL may comprise “http://&lt;object store domain&gt;/&lt;container identifier&gt;/&lt;object identifier&gt;”, where &lt;object store domain&gt; is the domain of the object store, &lt;container identifier&gt; is an identifier for the distributed file system and/or a container on the object store used by the distributed file system, and &lt;object identifier&gt; identifies the object to be read. In an embodiment, the object identifier is a hash of the object and/or a hash of a version of the object. 
     Client  100  may attempt to access the data object once it receives the data object location from MDS  102 . If the data object location is a URL, the client may issue an HTTP GET to the URL. For example, the client may issue a GET to object store  104  and/or the cloud service provider holding the data object. In response, object store  104  may return the requested data object to client  100 . 
     The present system may also be used to write data objects to the distributed file system. This process may be similar to reading data objects, as discussed above. Once the distributed file system is mounted and client  100  has identified the file system location where it wishes to write the data, client  100  may transmit a write intent to MDS  102 . This write intent may include the identified file system location and an object identifier for the data object client  100  intends to write. In some embodiments, this object identifier may be a hash of the data object or a hash of a version of the data object. 
     Upon receiving the intent, MDS  102  may consult a database table to determine if the data object has already been placed in an object store, such as object store  104 . If the data object already exists, there is no need to write it to the object store a second time. MDS  102  may perform this check by comparing the provided object identifier to all of the object identifiers in the table. If there is a match, the data object exists. If there is not a match, the data object does not exist. 
     If the data object already exists in object store  104 , client  100  may not need to transmit the data object to the store a second time. Instead, MDS  102  may create a new entry in the table comprising the object identifier and the location client  100  wishes to write the data. MDS  102  may then transmit a write complete notification to client  100 , and the write process may terminate. Should client  100  issue a subsequent read for the object, MDS  102  may provide a URL to the data object on object  104  as discussed above. This process provides an inherent form of data deduplication by ensuring a data object is not written to the same object store multiple times. 
     If MDS  102  determines object store  104  does not have a copy of the data object (i.e. the object identifier is not found in the table), it may create a new entry for the object as discussed above. MDS  102  may additionally provide an object location back to client  100 , and associate this object location with the new table entry. In some embodiments the object location is a URL constructed in the same manner as the URL generated during the read process. 
     Once client  100  receives the object location it may write the data object to that location. If the object location is a URL identifying an object store, such as object store  104 , client  100  may write the data to that location using an HTTP POST and/or PUT. The POST and/or PUT request may include the data object client  100  wishes to store on object store  104 . In some embodiments client  100  may conclude that the write was successful after the post was sent and terminate the write process. Alternatively, client  100  may wait for a confirmation from object store  104  before determining the write was successful. 
     While the above examples discuss reading and writing data objects as individuals, other configurations may exist. For example, individual data objects may be broken into a set of data chunks. Each of these data chunks may be stored and accessed on the object store in the same manner as the individual data objects discussed above. When a client wishes to read a data object, the client may submit identifiers for all the data object&#39;s constituent chunks to the MDS and receive a URL for each. Similarly, for writes the client may submit identifiers for all the data object&#39;s constituent chunks to the MDS. In response, the MDS may only provide write URLs for the chunks that do not already exist on the object store. If the chunks already exist the MDS may simply update the metadata table; there is no need to write the chunks a second time. 
     Turning now to  FIG. 2 , a system providing a distributed file system using a site cache is discussed.  FIG. 2  may comprise client  200 , MDS  202 , and object store  204 . In some embodiments, client  200  may be similar to client  100 , MDS  202  may be similar to MDS  102 , and authoritative object store  204  may be similar to object store  104 .  FIG. 2  may also include site cache  206 , which may be physically located at or near client  200  as showed by the dashed box. While not shown, the system of  FIG. 2  may include a plurality of locations, and some locations may have a site cache while other locations do not. While MDS  202  is shown at a separate location in  FIG. 2 , an embodiment of the disclosure may have MDS physically located at or near client  200  and/or site cache&#39;s  206  location. 
     Site cache  206  may comprise a transient object store capable of storing data object and/or data segments. In an embodiment, the transient object store is similar to the authoritative object store. For example, the transient object store may be a local cloud storage install, such as a local OpenStack Swift install, while the authoritative object store may be a public cloud storage, such as a hosted Microsoft Azure instance. The transient object store may be capable of storing data and responding to read and/or write requests in the same manner as object store  104  discussed in reference to  FIG. 1 . 
     In some embodiments, client  200  may be in communication with site cache  206  over a high bandwidth, low latency connection. While not shown, client  200  may also be in communication with authoritative object store  204  over a lower bandwidth, higher latency connection. When possible, client  200  may prefer to have IO requests processed by site cache  206  to take advantage of the superior connectivity. 
     Site cache  206  may improve the distribute file system&#39;s performance by processing read and/or write requests when able. In some embodiments, the site cache may by physically located near a client, such as in the client&#39;s geographical branch office. When the client issues a read request for a data object, the site cache may first process the request. If the cache contains the object, it may be returned to the client via the site cache&#39;s superior connectivity. If the site cache does not contain the data object, the read request may be forwarded to authoritative object store  204 . Similarly, a client may prefer to issue write requests to the site cache. The data objects associated with the write request may be stored in the site cache&#39;s transient object store and periodically evicted and/or flushed to authoritative object store  204 . 
       FIG. 3  depicts a site cache architecture consistent with an embodiment of the present disclosure. In an embodiment, all read and/or write requests made by a client may be processed by site cache  306 . For example, MDS  302  may return two URLs in response to a client&#39;s  300  read and/or write request. One URL may be for site cache  306  and the other URL may be fore authoritative object store  304 . The client may then communicate the request to site cache  306  which may either process the request locally or forward it to authoritative object store  304 . 
     The system shown in  FIG. 3  includes client  300 , MDS  302 , and authoritative object store  304 . These components may behave similar to like named elements discussed above.  FIG. 3  also shows site cache  306 , which may process read and/or write requests from client  300 . In some embodiments, site cache  306  may be substantially similar to site cache  206 . 
     Site cache  306  may include traffic processing engine  308 . In some embodiments, traffic processing engine  308  may be the gateway into site cache  306 . When client  300  issues a read and/or a write request, traffic processing engine  308  may receive the request and determine an appropriate response. For example, traffic processing engine  308  may communicate with other site cache components to service the request. Additionally or alternatively, traffic processing engine  308  may forward the request to authoritative object store  304 . These scenarios are discussed in further detail below. In some embodiments, traffic processing engine  308  is an HTTP proxy, such as NGINX. 
     Site cache  306  may also include transient object store service  310 . Transient object store service  310  may comprise a software service, application, module, or plugin containing logic for managing site cache  306 . For example, transient object store service  310  may monitor the capacity of transient object store  312 , manage evictions and/or flushing policies, and/or maintain the site cache state. Transient object store service may also be in communication with metadata store  314 , and may be used to determine whether site cache  306  contains a copy of a requested data object. 
     Site cache  306  may further include transient object store  312 , which may be substantially similar to authoritative object store  304 . In an embodiment, transient object store  312  may be used to store data objects. Storing these data objects in transient object store  312  may improve the distributed file system&#39;s performance as it allows the site cache to quickly service read and/or write requests from client  300  over a high bandwidth, low latency communication. In some embodiments, transient object store  312  comprises an OpenStack Swift instance. 
     Finally, site cache  306  may contain metadata store  314 . Metadata store  314  may include metadata information about the data objects stored in transient object store  312 . In an embodiment, metadata store  314  may be substantially similar to MDS  302  and contain similar information. Additionally or alternatively, metadata store may include more limited data, such as the data object identifier and data object size of the data objects stored in the transient object store. In an embodiment, the transient object store service may be used to determine whether a data object exists in the site cache. In some embodiments, the metadata store  314  is a relational database, such as MySQL. 
     Turning now to  FIG. 4 , a process for handling a read request for a data object in a distributed file system using a site cache is discussed. The process of  FIG. 4  contemplates that a client, such as client  300 , has already issued a read request to a metadata server and received a response. The response may include two URLs: one for the site cache and one for an authoritative object store. In some embodiments, the site cache may always attempt to service the read request before transmitting it to the authoritative object store. 
     At block  400 , a transient object store may receive a read request from a client. The read request may include an identifier associated with a data object the client is attempting to read. In some embodiments, the read request may also include a location reference for an authoritative object store. This location reference could be, for example, a URL associated with the authoritative object store. In some embodiments, the request may be received by a traffic processing engine in the site cache, such as traffic processing engine  308 . 
     At  402 , a check is made to determine whether the site cache contains a copy of the requested data object. The data object could be stored, for example, in a transient object store such as transient object store  312 . The check may be performed in at least two ways. First, the traffic processing engine may transmit the data object identifier to a transient object store service, such as transient object store service  310 . The transient object store service may then query a metadata store, such as metadata store  314 , to determine whether the transient object store contains a copy of the data object. If the transient object store contains the data object, the metadata store may contain the data object&#39;s identifier. Second, the traffic processing engine may contain logic for querying the metadata store directly. This may make the transient object store service unnecessary. 
     If the check at  402  is positive, and the transient object store does contain the requested data object, the process may continue to block  404 . At block  404  the data object may be returned to the client, either directly or via the traffic processing engine, and the process may terminate. If the check at  402  is negative, and the transient object store does not contain the data object, the process may continue to block  406 . 
     At block  406 , the site cache and/or its associated transient object store may not contain the data object. As a result, the object may need to be retrieved from the authoritative object store to service the client&#39;s request. The traffic processing engine may forward the client&#39;s request to the authoritative object store using the location reference. For example, the traffic processing engine may issue and HTTP GET to the location reference to retrieve the data object. 
     At block  408 , the data object may be received from the authoritative object store in response to the second request. The object could be received, for example, by the site cache&#39;s traffic processing engine. 
     At block  410 , the site cache may attempt to store the received data object in the transient object store. This may be beneficial if future requests are for the same data object. In such an embodiment, the data object may be served to the requesting client from the site cache rather than from the authoritative object store. Storing the data object to the transient object store may be substantially similar to the write process discussed in reference to  FIG. 5 . If the transient object store is in a READ_ONLY state or otherwise incapable of processing the write, the process may continue straight to block  414 . 
     At block  412 , the metadata store may be updated in response to storing the data object in the transient object store. This may include storing the data object&#39;s identifier, size, and/or other metadata in the metadata store. In some embodiments, this update may be performed by the transient object store service. 
     Finally, at block  414  the data object may be returned to the client. In some embodiments, the data object may be returned to the client before or in parallel with storing it to the transient object store. The data object could be returned, for example, through the traffic processing engine. 
     Turning now to  FIG. 5 , a process for writing a data object to a distribute file system using a site cache is discussed. At block  500 , a write request may be received from a client. The write request may include a data object to write, an object identifier, and a location reference, such as a URL, for an authoritative object store. In an embodiment, the write request is received at a traffic processing engine, such as traffic processing engine  308 . Additionally or alternatively, the data object may be received from the authoritative object store during a read request, as discussed in reference to  FIG. 4 . In such an embodiment the request may not include the location reference and/or object identifier, but may only include the data object itself. 
     At block  502 , a check may determine whether the transient object store is in a READ_ONLY state. This may occur, for example, if the volume of a transient object store has exceeded a defined threshold, as discussed in reference to  FIG. 6 . If the transient object store is in a READ_ONLY state the site cache will be unable to store the data object. In an embodiment, this check is performed by a transient object store service, such as transient object store service  310 . 
     If the transient object store is in a READ_ONLY state, the process may continue to block  504 . At block  504  the data object may be transmitted to an authoritative object store. In some embodiments, the data object is communicated using an HTTP PUT and/or POST request. For example, a traffic processing engine may transmit the data object to the authoritative object store in response to receiving an indication from the transient object store service that the transient object store is in a READ_ONLY state. After the data object is transmitted to the authoritative object store, the traffic processing engine may receive a response indicating the write was successful. The traffic processing engine may thereafter transmit the response to the client and/or a MDS to update the data object&#39;s location information. 
     If the transient object store is not in a READ_ONLY state, or is in a READ_WRITE state, the process may continue to block  506 . At block  506 , a write intent may be communicated to a metadata store. This write intent may be sent by the transient object store service. Additionally or alternatively, the intent may be communicated directly to the metadata store from the traffic processing engine. In some embodiments, the write intent includes the data object identifier, size, and/or other data object metadata. 
     At block  508 , the metadata store may be updated to include at least the data object identifier. This may enable the site cache to quickly determine whether it contains a data object during a read operation. 
     Finally, at block  510 , the data object may be written to the transient object store. This may include, for example, issuing an HTTP PUT or POST request containing the data object from the traffic processing engine to the transient object store. Once the data object is written to the transient object store, a success transmission may be communicated from the traffic processing engine to the client and/or the metadata server. 
       FIG. 6  depicts the site cache&#39;s volume as it varies based on an eviction policy and/or process. Transient object stores, such as transient object store  206 , may have a limited capacity. For example, the transient object store may have limited disk and/or solid-state storage space. As the data volume in the transient object store reaches the store&#39;s capacity, it may be beneficial to evict data objects to the authoritative object store. Evicting a data object could mean it is stored on the authoritative object store and removed from the site cache. This may enable, for example, less used or accessed data objects in the transient object store to be replaced by newer and/or higher value objects. This eviction process may be managed by a transient object store service, which could change the state of the site cache and/or transient object store. In an embodiment, the transient object store service and/or an administrator may set or dynamically generate policies for the eviction process in addition to managing the data objects in the transient object store. 
     The example depicted in  FIG. 6  shows the volume of data stored in the transient object store as it varies based on eviction policies. The y-axis of the graph depicts the transient object store&#39;s capacity, and the x-axis represents time. Curve  600  represents the volume of stored data on a site cache&#39;s transient object store over time as data objects are being ingested or evicted by the site cache. For example, at mark  602  the volume of data in the transient object store is consuming ˜40% of the store&#39;s total capacity. 
       FIG. 6  further depicts both a high capacity threshold and low capacity threshold. These thresholds may be defined using the transient object store service, and may be used by the eviction process to assign and/or determine different transient object store states. For example, if the volume of data in the transient object store exceeds the high capacity threshold, it may be beneficial to prevent the ingestion of new data objects by setting the transient object store state to READ_ONLY. Similarly, while the volume is below the low capacity threshold, the transient object store state may be READ_WRITE thereby enabling clients to realize the site cache benefits for both data object reads and data object writes. These states and thresholds are described in further detail with reference to positions  602 ,  604 ,  606 ,  608 ,  610 , and  612 . 
     Position  602  on curve  600  represents when the volume of stored data in the site cache is below the low capacity threshold. This may occur, for example, when the site cache is first brought online. While below the low capacity threshold, the transient object store may have sufficient capacity to store new data objects received from a client and/or authoritative object store, and the transient object store&#39;s state may therefore be READ_WRITE. The READ_WRITE state allows the site cache to serve either read or write requests from a client. 
     Moving along curve  600 , position  604  represents a later point in time when the low capacity threshold is reached and/or exceeded. At this point the transient object store may have sufficient remaining capacity to service new write requests, but the transient object store service may nevertheless initiate an eviction process. Initiating the eviction process upon reaching a low capacity threshold prevent the volume of stored data from reaching a point where the site cache can no longer service write requests. For example, if the volume of data evicted is larger than the volume ingested, the curve may move in a downward direction. This is unlike the example of  FIG. 6 , where the volume of data continues to rise despite the fact that objects are being evicted. 
     In some embodiments, the eviction process may be based on the transient object stores service&#39;s cache eviction policy. This policy may be determined by an administrator or dynamically generated by the transient object store service. For example, the cache eviction policy could be First in First out (“FIFO”), meaning the first data object stored in the site cache is the first transmitted to the authoritative object store, Least Recently Used/Accessed (“LRU”), meaning the last data object used or accessed in the transient object store is the first evicted to the authoritative object store, Largest out First (“LOF”) policy, meaning the largest data objects are evicted first, and/or any combination of the above. In some embodiments, metadata about the data objects is stored in a metadata store, such as metadata store  314 , and may be queried by the transient object store service to enforce the eviction policy. For example, the metadata store may include a timestamp and/or data object size for each data object in the transient object store. 
     Continuing along curve  600  from position  604 , the transient object store continues to store new data objects. At position  606  the volume of stored data has exceeded the low capacity threshold, but still has not reached the high capacity threshold. The transient object store service will continue to evict stored data objects while permitting the ingestion of new data objects. 
     Position  608  on curve  600  represents a point at which the high capacity threshold is reached. At this point the volume of data objects being ingested by the site cache may be larger than the volume being evicted. As a result, the transient object store may be in danger of reaching maximum capacity and being unable to service new data objects. At this point, the transient object store service may set the site cache to a READ_ONLY state. While in the READ_ONLY state the transient object store service will continue to evict stored data objects, but will not permit new data objects to be written. If the new objects are received from a client, they will be passed through to the authoritative object store. If the new objects are received from an authoritative object store, they will be passed through to the client without caching a local copy. While the READ_ONLY state may prevent new data objects from being written, the site cache is still able to service read requests. As a result, a client may still realize the benefits of the site cache when attempting to read data objects (assuming the transient object store contains a copy of the data object). 
     Position  610  on curve  600  represents a peak in the volume. At this point, the eviction process starts decreasing the volume of stored data on the site cache. This peak could be reached, for example, if a large data object was written immediately before crossing the high capacity threshold. At this stage the site cache may still be in a READ_WRITE state servicing write requests. Once the data object is written and/or the threshold is exceeded the state will switch to READ_ONLY. The site cache will remain in READ_ONLY state until the volume of stored data falls below the low capacity threshold. 
     Moving further along curve  600 , position  612  represents a point where the eviction process has brought the volume of stored data below the low capacity threshold. At this point the transient object store may have sufficient capacity to again begin servicing write requests. The transient object store service will set the site cache back to a READ_WRITE state and allow ingestion of new data objects. 
     Turning now to  FIG. 7 , a process for managing a site cache&#39;s state based on the volume of stored data on the transient object store, a high capacity threshold, and a low capacity threshold is discussed. 
     At block  700 , a data object is received at the site cache. The data object could be, for example, received from a traffic processing engine. In an embodiment, the data object is received from a client attempting to write the data object to a distributed file system. Additionally or alternatively, the data object may be received from an authoritative object store as part of a client&#39;s read request. These and other embodiments are discussed in further detail above. 
     At block  702 , a transient object store service determines if a site cache is in READ_ONLY or READ_WRITE state. As noted in reference  FIG. 6 , the state of the site cache may be based on the volume of data in relation to a high and/or low threshold. If the volume of stored data previously exceeded the high capacity threshold, and has not yet dropped below the low capacity threshold, then the site cache is in a READ_ONLY state. Otherwise the site cache is in a READ_WRITE state. In some embodiments the site cache&#39;s current state is stored in a metadata store. A transient object store service may query that store at block  702  to determine the state. 
     At block  703 , the transient object store service, having determined in block  702  that the site cache is in READ_ONLY state, processes the data object by transmitting directly to an authoritative object store. This could occur, for example, between positions  610  and  612  of  FIG. 6 . Once the data object is stored the site cache may transmit a confirmation to a client and/or the process may terminate. 
     If the check at  702  determines the site cache is in a READ_WRITE state, the process may continue to block  704 . At block  704  the data object may be written to the transient object store of the site cache. This could occur, for example, at position  602  and/or  606  of  FIG. 6 . 
     At block  706  the transient object store service may determine whether the volume of stored data on a site cache exceeds the low capacity threshold. This volume change may be the result of writing the data object at block  704 . If the volume of stored data exceeds the low capacity threshold, such as if the volume after the write is at position  606  of  FIG. 6 , the process may continue to block  708 . If the volume of stored data is below the low capacity threshold then the transient object store service does not change the state of the site cache and the process may terminate. 
     At block  708 , after the transient object store service has determined that the volume of stored data exceeds the low capacity threshold, the site cache begins cache eviction based on the transient object store service&#39;s cache eviction policy. The cache eviction policy could be a First in First out (“FIFO”), a Least Recently Used (“LRU”), or a Largest out First (“LOF”) policy, as noted above. 
     At block  710 , the transient object store service may determine if the volume of stored data exceeds the high capacity threshold. This could occur, for example, at position  610  of  FIG. 6 . If the volume of stored data exceeds the high capacity threshold then the process may continue to block  712 . Alternatively, if the volume of stored data is below the high capacity threshold then the process may terminate. 
     Finally, at block  712 , the transient object store service sets the site cache to a READ_ONLY state in response to determining that the volume of stored data exceeds the high capacity threshold. 
     Turning now to  FIG. 8 , an eviction process is discussed. This eviction process may move data objects from the transient object store and/or site cache to an authoritative object store. In some embodiments, the site cache state changes after eviction. This process could occur, for example, between positions  604  and  612  of  FIG. 6 . Additionally, it may occur in parallel with the process of  FIG. 7 . 
     At block  800 , stored data objects are transmitted to the authoritative object store. For example, the data object may be transmitted using an HTTP PUT or POST. In some embodiments, transmitting the data object from the site cache to the authoritative object store may be substantially similar to writing the data object directly from a client to the authoritative object store. Data objects may be transmitted continuously to the authoritative object store, and in an embodiment are based on available egress bandwidth, a defined policy (e.g. FIFO, LRU, LOF, etc), and/or a defined schedule. 
     At block  802 , a stored data object may be evicted from the transient object store. In some embodiments, evicting a data object comprises deleting the instance of the data object from the transient object store, removing its reference from the site cache metadata store, and/or updating the distributed file system&#39;s MDS to identify the objects new location. This eviction process could occur, for example, in parallel with block  800 . 
     At block  804 , the transient object store service determines if the volume of stored data is below the low capacity threshold. For example, the check may determine if the volume is at position  612  of  FIG. 6 . In some embodiments, this check may only occur if the site cache state is READ_ONLY. For example, the check will not occur if the volume is between positions  604  and  610 , but will occur if the volume is between  610  and  612 . If the volume is below the low capacity threshold and the state is READ_ONLY, the process may continue to block  806 . Otherwise the eviction process may terminate. 
     At block  806 , the transient object store service sets the site cache to a READ_WRITE state. The volume has returned to a place where the transient object store may service new data objects, which may be represented by the remainder of curve  600  after position  612 . 
     For the sake of clarity, the processes and methods herein have been illustrated with a specific flow, but it should be understood that other sequences may be possible and that some may be performed in parallel, without departing from the spirit of the invention. Additionally, steps may be subdivided or combined. As disclosed herein, software written in accordance with the present invention may be stored in some form of computer-readable medium, such as memory or CD-ROM, or transmitted over a network, and executed by a processor. 
     All references cited herein are intended to be incorporated by reference. Although the present invention has been described above in terms of specific embodiments, it is anticipated that alterations and modifications to this invention will no doubt become apparent to those skilled in the art and may be practiced within the scope and equivalents of the appended claims. More than one computer may be used, such as by using multiple computers in a parallel or load-sharing arrangement or distributing tasks across multiple computers such that, as a whole, they perform the functions of the components identified herein; i.e. they take the place of a single computer. Various functions described above may be performed by a single process or groups of processes, on a single computer or distributed over several computers. Processes may invoke other processes to handle certain tasks. A single storage device may be used, or several may be used to take the place of a single storage device. The disclosed embodiments are illustrative and not restrictive, and the invention is not to be limited to the details given herein. There are many alternative ways of implementing the invention. It is therefore intended that the disclosure and following claims be interpreted as covering all such alterations and modifications as fall within the true spirit and scope of the invention.