Disconnected file operations in a scalable multi-node file system cache for a remote cluster file system

Facilitating access to data in a network, is provided. One implementation involves maintaining a scalable multi-node file system cache in a local cluster file system, and caching file data in a cache by fetching file data on demand from a remote cluster file system into the local cached file system over the network. The local file data corresponds to file data in the remote cluster file system. Upon disconnection from the remote cluster file system, all file operations are supported at the cache. Then, upon reconnection to the remote cluster file system over the network, the remote cluster file system is synchronized with the information cached in the cache during the disconnection even in the presence of failures.

BACKGROUND

1. Field of the Invention

The present invention relates generally to data storage. In particular, the present invention relates to disconnected file operations in cluster file systems.

Information technology (IT) systems require sharing of large amounts of file data in a consistent, efficient, and reliable manner across a wide-area network (WAN). WAN data storage systems, including cluster file systems, need to scale in capacity and bandwidth to support a large number of client nodes. A Cluster is a group of interconnected independent nodes working together as a single system. A cluster file systems manages data stored within a cluster and provides client nodes with access to all files located on storage devices in the file system.

BRIEF SUMMARY

Disconnected file operations in a scalable multi-node file system cache for a remote cluster file system, is provided. One embodiment comprises maintaining a scalable multi-node file system cache in a local cluster file system, and caching file data in a cache by fetching file data on demand from a remote cluster file system into the local cached file system over the network. The local file data corresponds to file data in the remote cluster file system. Upon disconnection from the remote cluster file system, all file operations are supported at the cache. Then, upon reconnection to the remote cluster file system over the network, the remote cluster file system is synchronized with the information cached in the cache during the disconnection even in the presence of failures.

Further, a system for disconnected file operations in a scalable multi-node file system cache for a remote cluster file system, is provided. One embodiment comprises a cache subsystem including a cache for maintaining data for a local cluster file system including multiple computing nodes. A caching layer function is configured for caching file data in the cache including fetching file data on demand from a remote cluster file system into the cache over the network, wherein the local file data corresponds to file data in the remote cluster file system. The caching layer function further is configured such that upon disconnection from the remote cluster file system, the caching layer function supports file data and metadata update and access operations at the cache, and upon reconnection to the remote cluster file system the caching layer function synchronizes the remote cluster file system with the information cached in the cache during the disconnection.

Further, a computer program product for facilitating access to data is provided, the computer program product comprises a computer readable storage medium having computer usable program code embodied therewith. The computer usable program code is configured to maintain a cache subsystem maintaining data in a cache for a local cluster file system, and provide a caching layer function for caching file data in the cache by fetching file data on demand from a remote cluster file system into the cache over the network. The local file data corresponds to the file data in the remote cluster file system. Upon disconnection from the remote cluster file system, the computer supports file data and metadata update and access operations at the cache. Upon reconnection to the remote cluster file system over the network, the computer synchronizes the remote cluster file system with the information cached in the cache during the disconnection.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating the general principles of the invention and is not meant to limit the inventive concepts claimed herein. Further, particular features described herein can be used in combination with other described features in each of the various possible combinations and permutations. Unless otherwise specifically defined herein, all terms are to be given their broadest possible interpretation including meanings implied from the specification as well as meanings understood by those skilled in the art and/or as defined in dictionaries, treatises, etc. The description may disclose several preferred embodiments for caching of remote file data in an information technology (IT) computing environment, including multiple computing resources, as well as operation and/or component parts thereof. While the following description will be described in terms of caching of remote file data for clarity and placing the invention in context, it should be kept in mind that the teachings herein may have broad application to all types of systems, devices and applications.

A cluster file system that implements caching of remote file data in a cluster file system is provided. The system caches data on demand while guaranteeing well defined file system consistency semantics. A preferred embodiment provides a scalable cache architecture for a cache in a file system to cache remote file system data while providing the consistency semantics of a distributed file system. The scalable caching architecture enables the file system to cache remote file system data wherein the cache can scale in capacity and bandwidth similar to a clustered file system. Further, such a cache can support a remote server file system from different vendors. The cached data can be exported and accessed transparently by a file system client for both read and write access. The cache utilizes open, standard protocols for over-the-wire file access. Further the cache can significantly mask network latency and continue to function with network outages.

The scalable, writable, caching system caches remote file data in the wide area network (WAN) when connectivity may be intermittent. The cache supports data and the metadata writes to the cache even during temporary or long term network or remote cluster file system outages. The cache updates are synchronized with the remote server without explicitly logging each operation or replaying an event log. The update synchronization operates for the multi-node cache wherein any of the nodes may be updating the data or metadata. The updates, when made in a disconnected mode, store minimal information about cache state, occupying less storage space than a complete event log. Upon reconnection, the entire state of the cache may be recreated at the remote cluster file system by efficiently using the cache state to determine updates to the cached data during a disconnect period.

In an example implementation described below, the scalable caching architecture is integrated with a General Parallel File System (GPFS) clustered file system. The remote data is accessed over a network file system (NFS) so that any remote server exporting data over NFS can be the caching target. To get better performance, the cache can switch to a parallel NFS (pNFS) for data transfer if the remote system exports the data using pNFS. The cache is visible to any file system client as a Portable Operating System Interface (POSIX) compliant file system, thus any file system client can browse the cache and access the data as if it was in a local file system. The cached data can be further exported via NFS or Common Internet file system (CIFS) to a remote NFS or CIFS client.

Referring toFIG. 1, a GPFS parallel cluster file system10comprising a local cached file system cluster15that stores cached data, wherein pNFS is used to move the data between the cluster15and a remote cluster file system18. GPFS is used for both file system clusters15and18, to provide concurrent high-speed file access across multiple file system computing nodes of a cluster. The cached cluster includes the file system15, computing nodes11(e.g., processors) that support the GPFS and the applications16that use it. The nodes11are connected to storage media12, such as shared disks or disk subsystems, over a switching fabric13. A cache manager14maintains cached data in the storage media12. All nodes11in the cluster have equal access to all disk12. Files are striped across all disks12in the file system where the number of disks12can range from tens to several thousand disks. In addition to balancing the load on the disks, striping achieves the full throughput that a disk subsystem is capable of by reading and writing the blocks in parallel.

The switching fabric13that connects file system nodes11to the shared disks12may comprise a storage area network (SAN) such as fibre channel or iSCSI. Alternatively, individual disks12may be attached to some number of input/output (I/O) server nodes that allow access from file system nodes11through a software layer running over a general-purpose communication network, such as IBM Virtual Shared Disk (VSD). Regardless of how shared disks12are implemented, the GPFS only assumes a conventional block I/O interface with no particular intelligence at the disks13. Parallel read-write disk accesses from multiple nodes11in the cluster15are synchronized to prevent corruption of both user data and file system metadata. The cluster15uses distributed locking to synchronize access to shared disks12. Wherein distributed locking protocols ensure file system consistency regardless of the number of nodes11that simultaneously read from and write to a file system on the disks12on behalf of an application16, while at the same time allowing data migration parallelism to achieve maximum throughput.

For remote accesses over a WAN, pNFS clients access storage devices in a remote cluster file system in parallel. This is achieved by separating the data and metadata paths, and by moving the metadata server out of the data path. As a result, each pNFS client can leverage the full aggregate bandwidth of the cluster file system. Using pNFS, clients can query the metadata server to determine the layout of how files are distributed across data servers. Layouts are maintained internally by the metadata server. Based on the file layout, the client is able to directly access the data servers in parallel. A pNFS client communicates with the data servers using a variety of storage access protocols, including NFSv4 and iSCSI/Fibre Channel. The pNFS specification allows for the addition of new layout distributions and storage access protocols, in addition to flexibility for implementation of the back-end storage system.

A more detailed description of a cached file system is described below.

FIG. 2shows a function block diagram of an IT system20. The IT system20includes a local cache cluster21and a remote cluster file system22. Every computing node23in the cache cluster21has the same access to data cached by a local cache file system24of the local cache cluster21. However, only some of the nodes23(i.e., I/O nodes23A) may have the hardware and software support for remote network connectivity. The I/O nodes23A act as pNFS clients to fetch the data from the remote cluster21. The remote cluster21may store the data in any POSIX-compliant cluster file system that is exported via pNFS (NFS may also be used but with a performance penalty). The I/O nodes23A are responsible for reading the data from the remote cluster21and writing it to the local cache file system24and vice-versa. The other nodes of the cluster (i.e., application nodes23B) service file access requests of applications16, by reading and writing the cached data in the local cache file system (i.e., cache)24. The remote file system22similarly includes I/O nodes25A and application nodes25B.

The application nodes23B of the local cache file system21are also responsible for forwarding access requests by requesting applications16to the I/O nodes23A (i.e., writes to be synchronized with a remote server node25A of the remote file system21, and reads to be fetched from the remote server25A on a local cache miss).

The split between I/O and application nodes23A,23B in the local system21is conceptual and any node23in the local cluster21can function both as an I/O node or an application node based on its configuration. The I/O nodes23A can be viewed as the edge of the cluster cloud that can communicate with the remote cluster22while the application nodes23B interface with the applications.

To access the remote data consistently, the system20associates a cache state with every object in the local cache file system24, wherein the cache state includes the NFS file handle and inode (e.g., data structure) attributes of the corresponding object in the remote file system26. As multiple nodes23in the local system21can be accessing the cached data in the local cached file system24, the accesses may be serialized by a standard GPFS distributed lock management with one of the nodes23being the token manager and issuing read and write tokens. The data can be concurrently written at the remote file system26of the remote cluster22, and at the local cache file system24of the local cache cluster21. Between the remote cluster22and the local cache cluster21, the system20supports the well known close-to-open consistency guarantees provided by NFS. To reduce the frequent checking of cached attributes with the remote file system26, the I/O nodes23A leverage the read and write delegation support of NFSv4. With delegations, the pNFS server25A of the remote cluster22can transfer the ownership of a file to the local cache cluster21, so that the local cache cluster21can safely assume that the data is valid and service local requests.

A cache manager27integrated into local cache file system24intercepts the application file access requests, wherein the applications simply experience the local cache file system24as a traditional GPFS file system. The cache manager27of the local cluster21mimics the same namespace as the remote cluster22. Thus browsing through the cache cluster21will show the same listing of directories and files as the remote cluster22. The caching function can be further exported via NFS to enable access by NFS clients. Example file system operations are now described, including Open/Close operations, Data Read operations, Data Write operations and Directory Traversal operations.

FIG. 3illustrates an example open/close operation process30. File operations that modify in-memory file state (e.g., open, close, lock, and unlock), are performed by the cache manager27(FIG. 2) locally in the local cached file system24without consulting the remote server22(block31). The cache manager27functions as a file system, with the remote cluster22being primarily used as a source of data that is fetched on demand into the local cached file system24from the remote cluster22over WAN (block32). When a locally opened file needs to be read from, or written to, the remote server25A of the remote cluster22, an I/O node23A opens the file remotely prior to, and closes it after; performing the read/write using the same name used when the file was opened locally (block33).

The files are typically read and written in their entirety using whole file caching and write coalescing, respectively. In a typical usage scenario, intersite conflicts are expected to be minimal and continued operation is required in the face of WAN outages (similar to an NFSv4 client handling of its file state in the presence of delegations).

The system allows disconnected operations, wherein user processes (applications) supported by the local cluster21, may continue to function in the absence of network connectivity to the remote cluster22.

Data Read Operations

FIG. 4illustrates an example data read operation process40. The cache file system24(FIG. 2) is initially created in the local cluster21, and contains no data (block41). When mounted, the cache file system24is associated with the root of the remote cluster exported space using a mount operation (block42). When an object such as a file is first accessed by an application node23B of the local cluster21, a result of a user-invoked opened on a file or a directory (i.e., via a GPFS lookup request), the cache manager27performs a VFS (Virtual File System) lookup or read directory (i.e., readdir) operation in the local cache24(block43). VFS is a generic file system layer to enable an operating system to interface with any file system. If the object is not found in the local cache file system (as the case would be on an initialized cache file system), the application node23B requests a selected I/O node23A to service the request from the remote cluster22(block44). The selection of I/O node23A is based on a hashing function that ensures that requests for an object are always sent to the same I/O node23A.

The selected I/O node23A converts a GPFS lookup request to an NFS LOOKUP request and forwards it to the remote cluster22to obtain the file information (block45). On success in obtaining file information from the remote cluster22, the I/O node23A creates the object in the local file system cache24via the cache manager27, associates a mapping between the local GPFS inode and the remote cluster (or home cluster) state (the cache is local but it contains information of the remote object (object modification times, unique identifier, etc.)). The I/O node23A provides the obtained file handle and attributes of the object to the application node23B and returns success status back to the application node23B (block46). In effect, once the lookup operation completes successfully, the object would have been created in the local cache file system24but would not contain any data. The state associated with a cached object indicates if the object is incomplete or empty.

On an application read request, in block47the application node23B first checks with the cache manager27to determine if the object exists in the local cache file system24. If the object exists but is empty or incomplete (i.e., a cache miss), the application node23B requests the designated I/O node23A to fetch the data from the remote cluster22. The I/O node23A, based on a prefetch policy, fetches/retrieves the entire file or the requested bytes from the remote cluster22(via pNFS over WAN) and writes the fetched information in the local cache file system via the cache manager27. If only a portion of the file (object) was retrieved from the remote cluster22, then the rest of the file may be prefetched asynchronously after the application request is completed.

The system20supports both whole file and partial file caching (segments including a set of contiguous blocks). The application node23B, when notified of completion, reads the requested bytes from the local cache24via the cache manager27and returns it to a requesting application16as if the requested data file (i.e., object) was present in the local cluster21all along. It should be noted that the I/O and application nodes23A,23B only exchange request and response messages while the actual data is accessed locally by the cache manager27via the shared disks12. Thereafter, if said previously requested file (object) is read again, the application node23B checks via the cache manager27if the complete valid object exists in the local cache24. On a cache hit, the application node23B can itself service the file read request from the local cache24via the cache manager27. The system20uses file and directory attribute checking, performed by an NFS client at the I/O node23A to guarantee close-to-open consistency of the data in the local cache24of the local cluster21, with the file system26of the remote cluster22. All the “read class” of requests which include lookup, get attribute (getattr) and read, follow a similar data flow. These requests can be considered synchronous on a cache miss, because the application is blocked waiting for the response back from the I/O node23A.

Asynchronous Operations

The WAN latencies are substantially masked by ensuring applications experience the cache cluster performance on all updates. In contrast to synchronous operations, asynchronous requests do not need to be performed at the remote cluster before the request returns a success to a requesting application. Such requests can be simply queued at an I/O node (i.e., gateway node) for a delayed execution at the remote cluster.

If the data at the remote cluster is unchanging (a read-only system), the difference between the remote and the local cache file systems is enumerated by the requests queued at the gateway nodes. Performance is improved since updates and writes execute at local speeds.

Typical asynchronous operations include operations that encapsulate modifications to the cached file system. These include relatively simple modify requests that involve a single file or directory (e.g., write, truncate) and modification of attributes (e.g., ownership, times), and more complex requests that involve changes to the name space through updates of one or more directories (e.g., creation, deletion or renaming of a file and directory or symbolic links).

For dependent metadata operations, each gateway node maintains a queue of asynchronous request messages that were sent by the application nodes. Each message contains the unique tuple <fileId: inode_num, gen_num, fsid>of one or more data objects being operated upon. To maintain correctness, a potential dependency between two requests is detected based on the intersection (overlap) of the set of objects in a set of write requests: if the set of objects do not overlap (i.e., not interdependent) then the requests can be asynchronously serviced in any order (e.g., create file A, create file B are not dependent and can execute in parallel). If there is a dependency, the requests are asynchronously serviced in the order in which they were executed on the application nodes (e.g., create A and remove A). Different objects can be serviced at different gateway nodes, but operations on a particular object are serviced by a specific gateway node. Certain operations are based on the object name (e.g., create) and are hashed on the parent directory file Id. Certain operations (e.g., write) are hashed on the object file Id. The result is that two dependent operations can be queued on two different gateway nodes. One implementation ensures that the file create is pushed before the file write is executed at the remote cluster.

To maintain the distributed ordering among dependent operations across multiple gateway node queues, in one embodiment the GPFS distributed token management infrastructure is utilized. As such, when an operation is enqueued, it acquires a shared token on objects that it depends on. When an operation is ready to be executed, it upgrades the token to be exclusive, which in turn forces a token revoke on the shared tokens that were acquired by the dependent operations on other nodes. This results in a chain reaction of token revokes. As a side effect, the operations are pushed out of the distributed queues in the order in which they acquired the token which matches the order in which they occurred. Table 1 below shows a subset of the dependencies between the most common types of update operations.

TABLE 1Dependency of Update Operations: The listed dependentearlier operations need to be executed beforethe given operation is to be executed.OperationDependent Prior Ops.Writecreate, setattrCreatemkdir (parent), renameRemovesetattr, write, renameMkdirmkdir (parent), setattrRmdirmkdir, remove (children), setattr

Observe that the create operation of a file depends on the parent directory being created before it, which in turn depends on its parent and so on. The remove operations (rmdir) follow the reverse order where the rmdir depends on the directory being empty so that the remove operations for the children (subdirectory) need to execute earlier.

For data write operations, on a write request, the application node first writes the data locally to the cache cluster and then sends a message to the designated gateway node to perform the write operation at the remote cluster. At a later time, the gateway node reads the data from the cache cluster and completes the remote write over pNFS.

Parallel writes across multiple gateway nodes may be performed. In one example, it may be assumed both that the cache has sufficient storage to delay flushing dirty data and that a single pNFS client data transfer bandwidth is sufficient.

The delayed nature of the queued write requests allow optimizations that would not otherwise be possible if the requests had been synchronously serviced. One such optimization is write coalescing, which groups the write request to match the optimal NFS buffer size (e.g., 1 MB). The queue is also evaluated before requests are serviced to eliminate transient data updates, for example, the creation and deletion of temporary files. All such “canceling” operations are purged without affecting the behavior of the remote cluster.

The queue of asynchronous requests is stored in dynamically allocated memory, and thus is limited to the available free memory. There are several options if memory cannot be allocated. One option is to force the application node to block until the previously queued operations are serviced.

FIG. 5illustrates an example of asynchronous data write operation process50. On a write request, the application node23B first writes the data to the local cache24via the cache manager27(block51), and then sends a message to the designated I/O node23A to perform the write operation at the remote cluster22(block52). The I/O node23A queues the request (block53) and returns acknowledgement immediately, allowing the requesting application16(e.g., user process) to complete (block54). At a later time, the I/O node23A reads the data from the local cache24and completes the remote write asynchronously (block55). This is performed for all “write class” of requests such as; create, make directory (mkdir), write and unlink. Since data modifying operations are performed asynchronously, optimizations such a write coalescing and elimination of transient creates may be performed.

As such, an implementation of a preferred embodiment of the system according to the invention comprises a remote file data caching module integrated with the GPFS cluster file system, providing a scalable, multi-node, consistent cache of data exported by a remote file system cluster. The example system uses the pNFS protocol to move data in parallel from the remote file cluster. Furthermore, the system provides a POSIX compliant file system interface, making the cache completely transparent to applications. The system can mask the fluctuating wide-area-network (WAN) latencies and outages by supporting asynchronous and disconnected-mode operations. The system allows concurrent updates to be made at the cache and at the remote cluster and synchronizes them by using conflict detection techniques to flag and handle conflicts. The system may rely on open standards for high-performance file serving and does not require any proprietary hardware or software to be deployed at a remote cluster.

The cache manager27, gateway nodes23A and application nodes23B collectively provide a caching layer function integrated into the local GPFS cluster file system21that can persistently and consistently store data and metadata exported by the remote cluster22across a wide-area network27. Since every node23has direct access to cached data and metadata in the file system24, once data is cached, applications16running on the cached cluster21achieve the same performance as if they were running directly on the remote cluster22. Furthermore, NFS clients can access the cache24in cached cluster21and see the same view of the data (as defined by NFS consistency semantics) as NFS clients directly access the data from the remote cluster22. In essence, both in terms of consistency and performance, applications16can function as if there was no cache24and WAN27in between the applications16and the remote cluster22. More importantly, the caching layer27can function as a standalone file system cache. Thus applications16can run on the cache cluster21using POSIX semantics and access, update, and traverse the directory tree even when the remote cluster22is offline.

The caching layer27can operate on a multi-node cluster (henceforth called the cache cluster) where all nodes need not be identical in terms of hardware, operating system (OS), or support for remote network connectivity. The nodes23B of the cache cluster21see a shared storage24, either by connecting to SAN attached storage or relying on a Network Shared Disk layer that enables all nodes in a GPFS cluster to “access” direct attached storage on another node in the cluster, as if it were local. Only a set of designated I/O nodes23A (Gateway nodes) need to have the hardware and software support for remote access to the remote cluster22. The nodes23A internally act as NFS/pNFS clients to fetch the data in parallel from the remote cluster22. Parallel NFS can be used if the remote cluster file system22provides support, otherwise NFSv4 can be used. As noted, the remaining nodes23B of the local cached cluster21called (Application nodes) service the data requests of applications16from the local caches cluster21.

The I/O nodes23A communicate with each other via internal remote procedure call (RPC) requests. As the application nodes23B service data requests by the requesting applications16, whenever an application request cannot be satisfied by the cache24(due to a cache miss or when the cached data is invalid), an application node23B sends a read request to one of the I/O nodes23A which accesses the data from the remote cluster22on behalf of the application node23B.

Different mechanisms can be implemented for the I/O nodes23A to share the data with the application nodes23B. One option is for the I/O nodes to write the remote data to the shared storage12, which application nodes can then access and return the data to the applications16. Another option is for the I/O nodes to transfer the data directly to the application nodes using the cluster interconnect. In the first option, data sharing occurs through the storage subsystem12, which can provide higher performance than a typical network link. All updates to the cache24are also made by the application nodes23B via the cache manager27and a command message (again no data) is sent to the I/O node23A and queued.

FIG. 6shows an example process70illustrating build up of queues71at the I/O nodes23A for asynchronous requests sent by an application node23B (i.e., create, write requests made at one of the application nodes23B). These requests are queued at the designated I/O node23A before being sent to remote cluster22, wherein inFIG. 6Ci, Ri and Wi indicate create, read and write for file i respectively. At a later time, the I/O node(s)23A read the data in parallel from the storage subsystem12and push it to the remote cluster22over pNFS27. The selection of an I/O node23A to service a request ensures that dependent requests are executed in the intended order. To provide node affinity, as a first step, an application node23B selects an I/O node23A using a hash function based on a unique identifier of the object on which a file system operation is requested. Coordination for operations that are based on a name (e.g., lookup, create, remove etc.) and operations that affect multiple objects (e.g., rename, link), are provided. Each I/O node23A maintains an in-memory queue of operations that need to be sent to the remote cluster22. All the file system operations, from the point of view of the cache24, fall into two classes: synchronous (i.e., those that require the request to block until the remote operation completes and returns, e.g., read, lookup), and asynchronous (i.e., those that can proceed without the remote operation completing, e.g., create, write). Each I/O node can delay asynchronous operations for a configurable duration of time.

Data consistency can be controlled across various dimensions and can be defined relative to the cache cluster21, the remote cluster22and the network connectivity. The cached data in the cache24is considered locally consistent if a read from a node of the cache cluster21returns the last write from any node of the cache cluster21. A validity lag is defined as the time delay between a read at the cache cluster21reflecting the last write at the remote cluster22. A synchronization lag is defined as the time delay between a read at the remote cluster22reflecting the last write at the cache cluster21.

Using GPFS distributed locking mechanism, the data cache is locally consistent for the updates made at the cache cluster21. The accesses are serialized by electing one of the nodes23to be the token manager and issuing read and write tokens. Local consistency within the cache cluster21translates to the traditional definition of strong consistency. For cross-cluster consistency across the WAN27, the local cluster21allows both the validity lag and the synchronization (or synch) lag to be tunable based on the workload requirements. Basic NFS close-to-open consistency can be achieved by setting the validity lag to zero on a file open (i.e., the data is always validated with the remote cluster22on an open command) and setting the synch lag to zero on a file close (i.e., cache writes are flushed to the remote cluster22on a close). NFS uses an attribute timeout value (typically 30 seconds) to recheck with the server if the file attributes have changed. The validity lag is bounded by this attribute timeout value or set explicitly as a parameter.

The synch lag can also be set to NFS semantics or set explicitly as a parameter. However, NFS consistency semantics can also be strengthened via the 0 DIRECT parameter (which disables NFS client caching) or by disabling attribute caching (effectively setting the attribute timeout value to 0). NFSv4 file delegations can reduce the overhead of consistency management by having the remote cluster22NFS/pNFS server transfer ownership of a file to the cache cluster21so that the cache24can safely assume that the data is valid and service local requests.

When the synch lag is greater than zero, all updates made to the cache24are asynchronously committed at the remote cluster22. The semantics will no longer be close-to-open as data writes regardless of the file close time delay. When the network is disconnected both the validation lag and synch lag become indeterminate. When connectivity is restored, the cache and remote clusters are synchronized, with conflicts being detected and resolved.

Disconnected Operations

With the remote cluster across the WAN27(FIG. 2), when network connectivity experiences frequent intermittent outages and occasional long term disruptions, in one embodiment the invention allows file operations during such disconnected network situations.FIG. 7shows an example process80in the local cluster system21(implemented by the local cache layer function), for facilitating access to data in a network during disconnection from, and reconnection to, the remote cluster file system22. The process80includes:Block81: Maintaining a scalable multi-node file system cache in a local cluster file system including multiple computing nodes.Block82: Caching file data in the cache including: supporting data and metadata writes to the local cache and fetching file data on demand from the remote cluster file system into the local cached file system over the network, wherein the local file data corresponds to remote file data in the remote cluster file system.Block83: Upon disconnection from the remote cluster file system, supporting file data operations at the local cache.Block84: Maintaining a cache state representing cache updates during a disconnection, wherein the cache state includes information for recreating at the remote cluster file system upon reconnection, the entire state of the local data cached during disconnection.Block85: Upon reconnection to the remote cluster file system over the network, synchronizing the remote cluster file system with the information cached in the local caches file system during the disconnection.

For synchronous operations, a revalidation lag becomes indeterminate. The revalidation lag is the time delay between a write operation done at the remote cluster getting reflected in a read operation being done at the cache cluster. If data is updated at the remote cluster an application reading from the cache cluster will see the changes after the revalidation lag time interval. By policy, either the local application16requests are blocked to wait until connectivity over the WAN is restored, or all synchronous operations are handled locally by the caching layer function and no request is sent to the gateway nodes23A for execution at the remote cluster22.

In contrast, all asynchronous operations can still update the cache24and return successfully to the requesting application16. The requests simply remain queued at the gateway nodes23A pending execution at the remote cluster22. The updates will experience an indeterminate synchronization lag, so the consistency semantics will be affected. The synchronization lag is tunable. Thus, an update in the cache is reflected at the remote cluster based on this time value. The consistency is dependent on the synchronization lag and the revalidation lag.

Although the in-memory queue of pending updates at the gateway nodes23A is necessary to mask network latencies and short-term network outages, long term network disconnections, memory pressures and gateway node failures may require additional processing. Long term disconnections are handled by re-synchronization after disconnection.

In one implementation a persistent data state is stored at the cache cluster21to recreate the changes made to the cache24that have to be synchronized (synched) with the remote cluster22upon reconnection. This involves persistently logging all updates in log records and maintaining the ordering of requests. One example involves providing an rsync program which compares the content of the local and remote files and directories, and performs and update (synchronization) by transferring changes from the local cache24to the remote cluster22. The rsync operation is unidirectional.

When updates have been made both at the remote cluster and the cache cluster, conflicts must be detected. Further, inode values are preserved when performing an update to prevent confusion of the NFS in treating an update as a file delete.

Another implementation involves dumping the local in-memory queue on a persistent store and replaying it as a recovery log. On-disk logging of all changes may affect performance for cache operations.

Another example involves maintaining sufficient information to capture the state of the cache24from the point-in-time before the network was disconnected, to the current time. Snapshots of the file system including cache cluster fileset (or a portion of thereof, such as a file set) is obtained at a time instant just before disconnection and then immediately after reconnection. This is termed a state transition. Differences between the before and after snapshots are applied to the remote cluster file system. The ordering of the operations is immaterial so long as the current cache cluster state is achieved after application of said differences to the remote cluster system.

In a preferred embodiment, the caching function layer of the local cache cluster21maintains a cache state as a part of a persistent state, wherein the cache state includes type information (e.g., minimal “bit vector”) per data object inode in the cache24to denote the type of updates performed on each data object in the cache24during a disconnection. For example, when a file is written in the cache cluster, the application node23B marks a bit in the file inode to denote that it is “dirty,” meaning it needs to be transferred to (updated at) the remoter cluster22by a gateway node23A. The gateway node23A then resets the bit once the file data has been synchronized with the remote cluster after reconnection. Multiple bits may be used to denote if the entire file is dirty or segments of the file are dirty.

An example process for supporting disconnected operations according to the invention includes:

During normal operations:1. For each asynchronous operation (e.g., create, rename, set attributes, write, link):Mark the special bits in the inode to reflect the type of update done,Store attributes in the inode to capture the metadata of the operation.2. For delete operations, log a delete record in a special log file.3. Proceed to queue the operation at the I/O node.4. When the I/O node performs the operation at the remote cluster the special bits and additional metadata stored in the inode is reset.

During recovery on failure:1. Read the deleted records from the special log file and recreate the delete operations and queue them at the I/O nodes as during normal operation.2. Scan the inode file to filter out all inodes which have any of the special bits set.3. Recreate the operation based on the set bit and from the additional attributes stored, and queue them at the I/O node as during normal operation.4. Special cases:i. CASE I: Handle subtree creation (e.g., create the entire path dir1/dir2/dir3/dir4/file1) wherein the directories are created before the file create is queued. Check if the parent directory is also not created and queue that directory-create before the file-create in a recursive fashion.ii. Case II: Handle sub tree deletion. Check if the object to be deleted was a directory and queue all the objects in the directory for deletion and then queue the directory delete operation.iii. CASE III: Handle a chain of renames (e.g., a to b, b to c). Queue a rename operation e.g. from a to c. If a file was renamed to an existing file, queue a delete of the file.

In one operation scenario, upon reconnection, the caching layer function (i.e., a gateway node23A) first performs a fast inode scan to find all inodes in the cache24with a “dirty” bit set, and pushes the corresponding data to the remote cluster22over pNFS using the same procedure as if an application16requested a write operation. Inode scans are very efficient as they may be performed in parallel across multiple nodes.

In case of changes to the namespace (e.g., in case of create, remove, rename, mkdir operations) during a disconnection, additional processing is performed. The caching function layer stores additional file attributes in extended attributes associated with an inode. In case of namespace changes, such extended attributes are used to determine deleted inodes for deleted files. The inode data is provided with the name of an object and parent directory information. Upon reconnection, when replaying a file create operation over NFS certain of the attributes, such as name and parent filehandle, are used.

Ordering of operations is immaterial for writes or simple file creates. There are, however, certain operation combinations where the ordering affects the correctness of the data. Consider the following sequence of seven example steps in order at the cache cluster during a disconnection:1. lookup fileB; remote fileB gets created in cache2. lookup fileA; remote fileA gets created in cache3. Network disconnected4. rename fileA to fileA.bak5. create fileA; new fileA gets created in cache6. delete fileB; delete old fileB in cache7. create fileB; new fileB is created in cache

Upon reconnection, if step 5 (file create) is performed remotely before step 4 (rename), then a new file (fileA) is renamed and is stored as fileA.bak and the original data for fileA is lost. Similarly, if step 6 and 7 are reversed in order, then fileB will be deleted from the remote cluster and not recreated.

To handle such a scenario correctly, the caching function layer maintains additional information on operations based on name, and executes the remove (delete) operations before the other updates seen by the inode scan.

As such, the caching function layer supports updates in a disconnected mode and synchronizes the caching cluster state with the remote cluster without an explicit event log. In one example, the cache state (ondisk state) includes two bits in the inode and a list of directory inodes that have been updated when the network connectivity was impaired. This information is used for updating the remote cluster as described.

Asynchronous updates may result in non-serializable executions and conflicting updates. For example, the same file may be created or updated by both the cache cluster21and the remote cluster22. The caching function layer detects such conflicts and resolves them based on policies. For example, one policy may dictate that the cache cluster21always overrides any conflict, while another policy may dictate to move a copy of the conflicting file to a special directory for manual inspection and intervention (similar to a lost and found directory generated on a file system check (fsck) scan). Further, certain types of conflicts may be merged without intervention. For example, a directory with two new files, one created by the cache cluster21and another by the remote cluster22can be merged to form a directory containing both files.

FIG. 8shows a block diagram of an example architecture of an embodiment of a system100for implementing an embodiment of the invention. The system100includes one or more client devices101connected to one or more server computing systems130. A server130includes a bus102or other communication mechanisms for communicating information, and a processor (CPU)104coupled with the bus102for processing information. The server130also includes a main memory106, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus102for storing information and instructions to be executed by the processor104. The main memory106also may be used for storing temporary variables or other intermediate information during execution or instructions to be executed by the processor104. The server computer system130further includes a read only memory (ROM)108or other static storage device coupled to the bus102for storing static information and instructions for the processor104. A storage device110, such as a magnetic disk or optical disk, is provided and coupled to the bus102for storing information and instructions. The bus102may contain, for example, thirty-two address lines for addressing video memory or main memory106. The bus102can also include, for example, a 32-bit data bus for transferring data between and among the components, such as the CPU104, the main memory106, video memory and the storage110. Alternatively, multiplex data/address lines may be used instead of separate data and address lines.

The server130may be coupled via the bus102to a display112for displaying information to a computer user. An input device114, including alphanumeric and other keys, is coupled to the bus102for communicating information and command selections to the processor104. Another type of user input device comprises cursor control116, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor104and for controlling cursor movement on the display112.

The functions of the system10(FIG. 1) are performed by the server130in response to the processor104executing one or more sequences of one or more instructions contained in the main memory106. Such instructions may be read into the main memory106from another computer-readable medium, such as the storage device110. Execution of the sequences of instructions contained in the main memory106causes the processor104to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in the main memory106. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and software.

Generally, the term “computer-readable medium” as used herein refers to any medium that participated in providing instructions to the processor104for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as the storage device110. Volatile media includes dynamic memory, such as the main memory106. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus102. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to the processor104for execution. For example, the instructions may initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the server130can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector coupled to the bus102can receive the data carried in the infrared signal and place the data on the bus102. The bus102carries the data to the main memory106, from which the processor104retrieves and executes the instructions. The instructions received from the main memory106may optionally be stored on the storage device110either before or after execution by the processor104.

The server130also includes a communication interface118coupled to the bus102. The communication interface118provides a two-way data communication coupling to a network link120that is connected to the world wide packet data communication network now commonly referred to as the Internet128. The Internet128uses electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link120and through the communication interface118, which carry the digital data to and from the server130, are exemplary forms or carrier waves transporting the information.

In another embodiment of the server130, interface118is connected to a network122via a communication link120. For example, the communication interface118may be an integrated services digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line, which can comprise part of the network link120. As another example, the communication interface118may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface118sends and receives electrical electromagnetic or optical signals that carry digital data streams representing various types of information.

The network link120typically provides data communication through one or more networks to other data devices. For example, the network link120may provide a connection through the local network122to a host computer124or to data equipment operated by an Internet Service Provider (ISP)126. The ISP126in turn provides data communication services through the Internet128. The local network122and the Internet128both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link120and through the communication interface118, which carry the digital data to and from the server130, are exemplary forms or carrier waves transporting the information.

The server130can send/receive messages and data, including e-mail, program code, through the network, the network link120and the communication interface118. Further, the communication interface118can comprise of a USB/Tuner and the network link120may be an antenna or cable for connecting the server130to a cable provider, satellite provider or other terrestrial transmission system for receiving messages, data and program code from another source.

The example versions of the invention described herein are implemented as logical operations in a distributed processing system such as the system100including the servers130. The logical operations of the present invention can be implemented as a sequence of steps executing in the server130, and as interconnected machine modules within the system100. The implementation is a matter of choice and can depend on performance of the system100implementing the invention. As such, the logical operations constituting said example versions of the invention are referred to for e.g. as operations, steps or modules.

Similar to a server130described above, a client device101can include a processor, memory, storage device, display, input device and communication interface (e.g., e-mail interface) for connecting the client device to the Internet128, the ISP126, or LAN122, for communication with the severs130.

The system100can further include computers (e.g., personal computers, computing nodes)105operating the same manner as client devices101, wherein a user can utilize one or more computers105to manage data in the server130.