Independent fileset generations in a clustered redirect-on-write filesystem

Maintaining a generation value for each fileset that is distinct from a corresponding fileset manager preserves the independence of nodes while also allowing distributed fileset management. A fileset manager can maintain a value that reflects consistency snapshots for that node (“node generation”) separately from a value that reflects consistency snapshots for a particular fileset (“fileset generation”).

BACKGROUND

Embodiments of the inventive subject matter generally relate to the field of filesystems, and, more particularly, to clustered filesystems with redirect-on-write snapshotting.

SUMMARY

Embodiments include a method comprising tracking generations of a first node of a plurality of nodes in a cluster that are managing a plurality of filesets. The generations of the first node represent progression of consistency snapshots by the first node. Generations for each of the plurality of filesets in a distributed redirect-on-write clustered filesystem are independently tracked. Management of the plurality of filesets is distributed across the plurality of nodes in the cluster that hosts the distributed redirect-on-write clustered filesystem. The generations for each of the plurality of filesets represent progression of consistency snapshots of the plurality of filesets.

Embodiments include a method comprising maintaining a first fileset generation value for a first fileset of a plurality of filesets in memory of a first node of a plurality of nodes of a cluster. A second fileset generation value is maintained for a second fileset of the plurality of filesets independently of the first fileset generation value in the memory of the first node. Management of the first fileset and the second fileset has been delegated to the first node and the plurality of filesets are of a clustered redirect-on-write filesystem. A node generation value is maintained for the first node in the memory of the first node. The node generation value indicates a progression of consistency snapshots by the first node. The first fileset generation value represents a progression of consistency snapshots that have included the first fileset. The second fileset generation value represents a progression of consistency snapshots that have included the second fileset. The first fileset generation value and the second fileset generation value are stored into persistent cluster storage incident with publishing first metadata of the first fileset and second metadata of the second fileset.

DESCRIPTION OF EMBODIMENT(S)

The description that follows includes exemplary systems, methods, techniques, instruction sequences and computer program products that embody techniques of the present inventive subject matter. However, it is understood that the described embodiments may be practiced without these specific details. In other instances, well-known instruction instances, protocols, structures and techniques have not been shown in detail in order not to obfuscate the description.

A cluster is formed from multiple computer systems or nodes and resources, including persistent storage resources. A clustered file system is implemented across the storage resources of a cluster. The cluster storage resources are coupled to allow direct access by the nodes of the cluster. The storage resources can be directly cabled to the nodes and/or accessible via a network (e.g., storage area network).

When a cluster is established, an administrator configures one of the nodes of the cluster to operate as a cluster leader. Embodiments can also program a cluster to a automatically choose the leader. The cluster leader maintains cluster role data that indicates whether a node is operating as a client, as a server, or as both a client and a server. A node operating as a server manages a fileset in the clustered filesystem. A node operating as a server in a cluster is also referred to herein as a fileset manager. In addition to the indications of which nodes are operating as servers within the cluster, the cluster leader can also maintain an indication of which filesets are managed by which servers. The cluster leader also maintains an indication of which node operates as a clustered file system manager. A node within a cluster can be configured to operate as the cluster leader and the clustered file system manager. Whether a node operates as a cluster leader, server, client, etc., can be transparent to users of the cluster. A user will perceive a same behavior whether a node operates as both the client and server, or the client is on a remote node.

The clustered file system manager manages metadata for the clustered file system. The clustered file system manager maintains the clustered file system metadata (“metadata”) in a hierarchical data structure. Elements of the hierarchical structure can comprise inodes. The clustered filesystem metadata comprises metadata for filesets and files. A fileset comprises a set of files. A fileset can also comprise one or more filesets. In the hierarchical data structure, a fileset can be nested within a fileset to reflect a fileset comprising another fileset.

The clustered file system manager maintains a root for the clustered filesystem metadata at a known location (e.g., predefined location) in the persistent cluster storage resources (“cluster storage”). In a cluster that implements redirect-on-write consistency snapshots, multiple locations in the cluster storage are reserved or defined for storing roots of consistency snapshots along with root metadata of the corresponding consistency snapshots. The root metadata helps to identify the consistency snapshots and to ensure integrity of the consistent snapshots. Embodiments can use a time-based identifier of consistency snapshots (e.g., generation value) and root checksums. Embodiments can write a first root checksum (“header checksum”) when a node begins to write the root and a second root checksum (“trailer checksum”) after the root has successfully been written to persistent cluster storage. Embodiments can use the header checksum and trailer checksum to ensure writing of the root of a consistency snapshot was not interrupted. To recover from a failure, the node examines each of the locations and selects the location with the most recent generation value to allow recovery to begin with that consistency snapshot. Furthermore, progression of consistency snapshots (“generations”) can be tracked with generation values that allow for snapshots of different times for a filesystem object (e.g., fileset) to be distinguishable. A cluster can be configured to preserve a certain number of consistency snapshots.

An efficient cluster allows clients to perform writes to filesets of the clustered filesystem. In a redirect-on-write clustered filesystem, the client writes are to cluster storage. Allowing clients to write to the cluster storage provides independence among the nodes within the cluster, and avoids congestion at a server. A clustered filesystem can distribute fileset management for further operational efficiency in the cluster. In this case, the clustered file system manager delegates management of filesets within the clustered file system to other nodes of the cluster. When a node operating solely as a client is delegated management of a fileset, the delegation is referred to herein as promotion from client to server. The clustered filesystem manager can delegate management of a fileset from itself to another node, and can re-delegate fileset management among the other nodes.

In a redirect-on-write clustered filesystem that distributes fileset management, consistency snapshots could be synchronized across fileset managers. Synchronizing consistency snapshots across the fileset managers, however, exposes the cluster to individual node failures. Moreover, synchronizing consistency snapshots across fileset managers impedes the snapshot process with the slowest performing fileset manager. Maintaining a generation value for each fileset that is distinct from a corresponding fileset manager preserves the independence of nodes while also allowing distributed fileset management. A fileset manager can maintain a value that reflects consistency snapshots for that node (“node generation”) separately from a value that reflects consistency snapshots for a particular fileset (“fileset generation”).

FIG. 1depicts a conceptual diagram of an example redirect-on-write clustered filesystem tracking fileset generations and node generations. The example cluster comprises node103, node105, and node107. The nodes can be any of a variety of computers or computing systems. The nodes are coupled with a pool101of directly accessible storage devices. The storage devices that constitute pool101can be accessible via a network (e.g., storage area network) and/or via cables directly coupling the storage devices to nodes103,105,107. Node105is depicted as hosting clustered filesystem manager109. InFIG. 1, clustered filesystem manager109represents an executing instance of a computer program or part of a computer program. Node107is depicted as hosting fileset manager121, and node103is depicted as hosting fileset manager115. Fileset managers115,121represent executing instances of computer programs or parts of computer programs. Embodiments, however, can implement the functionality of a fileset manager or a clustered filesystem manager partially or wholly within hardware.

In this example illustration, nodes103,107have already been promoted to server or fileset manager by node105. Clustered filesystem manager109has already delegated management of a fileset FS1to node103. Clustered filesystem manager109has also already delegated management of filesets FS2and FS7to node107. Clustered filesystem manager109remains responsible for filesets FS3-FSN as reflected by hierarchical clustered filesystem metadata113. Hierarchical clustered filesystem metadata113is depicted with a root referencing an array of pointers to roots of extent trees inFIG. 1. The extent trees are depicted as triangles. An extent tree refers to a collection of data structures that correspond to a fileset and a fileset(s) and/or file(s) within the fileset. For example, an extent tree can represent a fileset inode, and other inodes referenced by the fileset inode. The other inodes can be other fileset inodes and file inodes. The term “tree” is used for naming simplicity and should not be used to limit embodiments and/or scope of the claims. A variety of data structures can be used to store metadata for files and/or filesets. Reference is made toFIG. 2to illustrate example operations that occur when delegating fileset management.

FIG. 2depicts an example conceptual diagram of metadata updates corresponding to fileset management delegation and consistency snapshot publication. Delegating management of filesets FS1, FS2, and FS7comprises clustered filesystem manager109updating hierarchical clustered filesystem metadata203to reflect the delegation. Prior to the delegation, hierarchical clustered filesystem metadata203comprised metadata for filesets FS1-FSN, as well as metadata for filesets and files within those filesets (assuming no previous delegation). InFIG. 2, hierarchical clustered filesystem metadata203is depicted with a root referencing an array of pointers to roots of extent trees for the filesets FS1, FS2, FS3, FS4, and FSN. The extent tree corresponding to fileset FS3references another extent tree corresponding to fileset FS7. The naming of the filesets have no significance and should not be interpreted as anything more than distinguishing filesets. In addition, the fileset FS7is depicted as nested within fileset FS2merely to depict a nested fileset. Filesets can be nested within the other filesets and no special relationship is intended by the depiction of nested FS7other than depicting one of many variations in structure of hierarchical clustered filesystem metadata.

At a stage A, clustered filesystem manager109updates hierarchical clustered filesystem metadata203in accordance with delegation of filesets FS1, FS2, and FS7. When delegating the filesets, clustered filesystem manager109writes the extent tree for each of the delegated filesets to new locations in cluster storage. InFIG. 2, clustered filesystem manager109writes roots of those extent trees (“fileset metadata roots”) into cluster storage of pool101at stage B. After each of the extent trees have been written to new locations, clustered filesystem manager109writes location of the fileset metadata roots to reserved or predefined locations in cluster storage. Embodiments are not limited to using cluster storage locations reserved or predefined for fileset metadata roots. Embodiments can write locations of fileset metadata roots to a database, directory, etc., which persists. After writing the extent trees to new locations and writing the fileset metadata roots, clustered filesystem manager109updates hierarchical clustered filesystem metadata203. The update results in hierarchical clustered filesystem metadata205, which corresponds to hierarchical clustered filesystem metadata113ofFIG. 1.

To update hierarchical clustered filesystem metadata203, in this illustration, clustered filesystem manager109removes the extent trees for the filesets FS1, FS2and FS7. Clustered filesystem manager109replaces the extent trees with pointers in cluster storage where the fileset metadata roots will be maintained by corresponding fileset managers. In this illustration, clustered filesystem manager109replaces the extent trees with two pointers. Clustered filesystem manager109uses two pointers to allow a fileset manager to alternate writing its fileset metadata root(s). With at least two pointers, a consistency snapshot survives with a consistent view of the fileset or filesets managed by a fileset manager that fails while publishing a consistency snapshot to cluster storage. The extent tree for fileset FS1has been replaced with a pointer “51” and a pointer “52.” These pointers identify locations or logical blocks in cluster storage. After a fileset manager failure, a succeeding fileset manager or clustered filesystem manager109reads both locations and uses the fileset metadata root with a most recent generation value. The extent tree for fileset FS2has been replaced with pointers “77” and “78.” The extent tree for fileset FS7has been replaced with pointers “95” and “96.” Again, the block numbers selected are merely for illustration. Embodiments are not limited to using adjacent blocks for alternate fileset metadata roots, and are not limited to separating fileset metadata roots of different fileset managers. After the metadata update has completed, clustered filesystem manager109can publish hierarchical clustered filesystem metadata205to cluster storage or wait until a consistency snapshot interval is reached. Clustered filesystem manager109informs the fileset managers of the location of their fileset metadata roots.

InFIG. 2, each of the locations in pool101are annotated to indicate the two fileset metadata roots for a fileset manager. Locations “51” and “52” for FS1are respectively annotated in blocks of pool101as “(FS1)” and “(FS1)′”. Locations “77” and “78” for FS2are respectively annotated in blocks of pool101as “(FS2)” and “(FS2)′”. Locations “95” and “96” for FS7are respectively annotated in blocks of pool101as “(FS7)” and “(FS7)′”.

Returning toFIG. 1, operations are depicted with various stages at each of nodes103,105,107. The operations at node105are labeled as stages A1-C1. The operations at node103are labeled as stages A2-C2. The operations at node107are labeled as stages A3-C3. The stage labels are used to illustrate parallelism that is not necessarily concurrent. Each of nodes103,105,107perform similar operations for publishing a consistency snapshot that involves transactional bookkeeping and snapshot bookkeeping. The transactional bookkeeping includes maintaining a node generation value relevant to the particular node. The snapshot bookkeeping includes maintaining a fileset generation value relevant to the particular fileset. Although each of nodes103,105,107is subject to a snapshot interval defined for the cluster, differences among nodes103,105,107can lead to variations when the snapshot interval is reached.

At stage A1, clustered filesystem manager109begins a publication transaction and updates a node generation value from 20 to 21 in a transactional data structure111. The node generation value can be used to distinguish between operations of a transaction that occur in different consistency snapshot intervals. To preserve consistency, the cluster employs transactional barriers or transactions to ensure atomicity of operations that constitute a transaction. For instance, if publication is interrupted, then the publication transaction is not complete. If the publication transaction is incomplete, then the operations that have been performed for the incomplete transaction are considered as not having been done. A log can be written to persistent storage after each operation of a transaction to allow a succeeding server (fileset manager or clustered filesystem manager) to at least partially recover operations of an incomplete transaction. The node generation can be employed to distinguish between data or metadata of a current generation and of a preceding generation that is in the process of being published to cluster storage. For example, a fileset manager can be responsible for different filesets that are in different generations. The node generation can be employed to distinguish between a preceding generation and a current generation instead of maintaining several different generations for several different filesets.

At stage B1, clustered filesystem manager109updates fileset generations for filesets FS3-FSN in hierarchical clustered filesystem metadata113. Clustered filesystem manager109will increment the fileset generations within the metadata of each of the filesets if a modification occurs to the filesets. For example, the generation value for fileset FS3will be incremented to N+1 if a write occurs to fileset FS3after node105begins publishing generation N of fileset FS3. Clustered filesystem manager109will publish hierarchical clustered filesystem metadata113in a bottom-up order. For instance, clustered filesystem manager109will start writing metadata for files and end with metadata of the topmost filesets. After the publishing has successfully completed, clustered filesystem manager109ends the publication transaction at stage C1.

Fileset manager115performs similar operations when it arrives at the snapshot consistency interval. At stage A2, fileset manager115begins a publication transaction and updates a node generation value from 2 to 3 in a transactional data structure117. At stage B2, fileset manager115increments the fileset generation for fileset FS1from 9 to 10 in hierarchical fileset manager metadata119in memory, assuming a write occurs to FS1after node103begins publishing the snapshot for generation9of fileset FS1. Hierarchical fileset manager metadata119comprises metadata for fileset FS1and the extent tree for fileset FS1. The metadata for fileset FS1comprises the generation value. After the publishing has successfully completed, fileset manager115ends the publication transaction at stage C2.

Referring toFIG. 2, fileset manager115writes the generation value of 9 into fileset metadata211for fileset FS1at location “51” in pool101incident with publishing the consistency snapshot of hierarchical fileset manager metadata119. Since the clustered filesystem implements redirect-on-write, fileset metadata is always written to a different location. But the fileset metadata root is bound to one of two locations in this illustrated example. Hence, the “new” location will alternate between the two locations specified for the fileset metadata root.FIG. 2depicts location “51” as comprising fileset metadata211that includes the new fileset generation value. Fileset metadata211is also depicted with an extent tree for contextual reference. The fileset metadata211,213,215,217,219,221are not intended to depict a block as hosting fileset metadata and an extent tree. Fileset manager115selects location “51” for the new root of the generation9consistency snapshot because location “51” references fileset metadata with an older generation value than location the generation value of the fileset metadata referenced by location “52.” Thus, the generation value of 9 will persist for fileset FS1. The location “52” is depicted as referencing fileset metadata213that includes generation value of 8, which is now a past snapshot.

Returning toFIG. 1, fileset manager121performs similar operations when it arrives at the snapshot consistency interval. But fileset manager121manages two filesets FS2and FS7. At stage A3, fileset manager121begins a publication transaction and updates a node generation value from 3 to 4 in a transactional data structure123. Nodes can have different node generation values for a variety of reasons (e.g., nodes can be promoted at different times, nodes may be taking over for a failed node, etc.). At stage B3, fileset manager121increments the fileset generation value for fileset FS2from 7 to 8 responsive to detecting a write to FS2after fileset manager121begins publishing the snapshot for generation7of FS1. And the fileset manager121increments the fileset generation value for fileset FS7from 9 to 10 in in-memory hierarchical fileset manager metadata125responsive to detecting a write to FS7after fileset manager121begins publishing the snapshot for generation9of FS7. Hierarchical fileset manager metadata125comprises metadata for fileset FS2and the extent tree for fileset FS1and for fileset FS7and the extent tree for FS7. The metadata for fileset FS2and the metadata for fileset FS7comprise the respective generation values. After the publishing has successfully completed, fileset manager121ends the publication transaction at stage C3.

Referring again toFIG. 2, fileset manager121writes the new generation value of 7 into fileset metadata217for fileset FS2at location “78” in pool101incident with publishing the consistency snapshot of hierarchical fileset manager metadata125. Fileset manager121also writes the new generation value of 9 into fileset metadata219for fileset FS7at location “95” in pool101incident with publishing the consistency snapshot of hierarchical fileset manager metadata125. As with fileset FS1, the generation values of 9 and 7 will persist for fileset FS7and FS2, respectively. The location “77” is depicted as referencing fileset metadata215that includes generation value of 6, which is now a past snapshot. The location “96” is depicted as referencing fileset metadata221that includes generation value of 8, which is now a past snapshot. As with the block numbers, no special significance should be ascribed to the layout of the fileset metadata roots and to depiction of (FS1)′, (FS2)′, and (FS7)′ referencing past snapshots.

Different generation values can arise for different filesets for various reasons. Filesets can be created at different times. Filesets can be more active. For instance, a generation can be skipped for a fileset that does not have any changes during a snapshot interval. In addition, a generation can be skipped if publication of a preceding snapshot did not complete within a snapshot interval. A node can take over management of a fileset because of a node failure or because of reassignment or re-delegation by the clustered filesystem manager (e.g., from load balancing, from a change in accessing patterns, etc.). Since a persistent generation value is maintained for the fileset, consistency of snapshots can be preserved without sacrificing independence of nodes. In addition, persistent fileset generation values allows publication of a fileset snapshot independent of other filesets managed at a same node. For instance, fileset manager121can successfully publish the metadata for fileset FS7, and suffer a failure before completing publication of the metadata for fileset FS2. Even though the generation of the fileset manager was interrupted, the generation of the fileset FS7can persist. Clustered filesystem manager109can delegate management of fileset FS2to node103. In that case, fileset manager115will determine that location “78” has a most recent generation value since stage D ofFIG. 2did not occur. And fileset manager115will load the fileset metadata217, and corresponding extent tree. Regardless of the failure at node107, fileset manager115proceeds with generation7for FS2.

FIG. 3depicts a flowchart of example operations for maintaining a fileset generation value and a node generation value. At some point, a node reaches a consistency snapshot interval (301). Incident or responsive to reaching the consistency snapshot interval, a node generation value is incremented (305). For each fileset managed by the node (307), the node publishes the fileset metadata.

The node begins a consistency snapshot transaction on a per fileset basis (308). Embodiment can also begin a transaction when the interval is reached, in addition to or instead of the fileset based transaction. Embodiments can enforce atomicity of all operations across filesets managed by a node for a consistency snapshots. Embodiments can enforce atomicity for each fileset publication, and separately enforce atomicity of snapshot operations that do not directly relate to the filesets (e.g., publishing free block tracking data). For example, a fileset manager can also write to persistent cluster storage any of data used to track allocation of free blocks to clients, data used to track a transaction at the fileset manager, etc.

The node then determines whether fileset metadata has changed since the last generation (309). The node can use the node generation value to determine whether fileset metadata has changed. If the fileset metadata has not changed, then the consistency snapshot may be skipped, and the consistency snapshot transaction for that fileset ends (315).

If the fileset metadata has changed, then the node increments the in-memory fileset generation value for the fileset (311). The node then proceeds with operations to publish the fileset metadata from memory of the node to persistent cluster storage (313). After all of the fileset metadata has been written out to persistent cluster storage, the fileset metadata root is updated to reference the new location of the fileset metadata. After the fileset metadata root is updated, the consistency snapshot transaction ends (315). If the node manages another fileset, then the node processes the next fileset (317).

The depicted flowcharts are examples intended to aid in understanding the inventive subject matter, and should not be used to limit embodiments and/or the scope of the claims. Embodiments can perform the operations depicted in the flowcharts in a different order, can perform the depicted operations in parallel, can perform additional operations, can perform fewer operations, etc. Referring toFIG. 3, additional operations can be performed for logging operations as a transaction progresses.

FIG. 4depicts an example computer system. A computer system includes a processor unit401(possibly including multiple processors, multiple cores, multiple nodes, and/or implementing multi-threading, etc.). The computer system includes memory407. The memory407may be system memory (e.g., one or more of cache, SRAM, DRAM, zero capacitor RAM, Twin Transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM, etc.) or any one or more of the above already described possible realizations of machine-readable media. The computer system also includes a bus403(e.g., PCI, ISA, PCI-Express, HyperTransport®, InfiniBand®, NuBus, etc.), a network interface405(e.g., an ATM interface, an Ethernet interface, a Frame Relay interface, SONET interface, wireless interface, etc.), and a storage device(s)409(e.g., optical storage, magnetic storage, etc.). Distributed cluster generation tracker425is also coupled with the bus403. Distributed cluster generation tracker425maintains fileset generation values for each fileset managed by the computer system, and separately maintains a node generation value for the computer system. Any one of these functionalities may be partially (or entirely) implemented in hardware and/or on the processing unit401. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processing unit401, in a co-processor on a peripheral device or card, etc. Further, realizations may include fewer or additional components not illustrated inFIG. 4(e.g., video cards, audio cards, additional network interfaces, peripheral devices, etc.). The processor unit401, the storage device(s)409, and the network interface405are coupled to the bus403. Although illustrated as being coupled to the bus403, the memory407may be coupled to the processor unit401.