Using Versioning to Simplify the Implementation of a Highly-Available File System

One of the most urgent challenges facing systems research is to improve the reliability, availability, and maintainability of system software. The importance of these issues is being driven by new applications and more sophisticated users that increasingly expect key system software to work all of the time, even in the presence of failures or increasing load. To meet these goals, modern systems must minimize the human administration required for configuration, maintenance, and error recovery and must scale gracefully to handle loads of unprecedented size. A modern file system must thus guarantee the availability and integrity of valuable data in the face of the widest possible range of human, software, and hardware failures. High-availability and fault tolerance are, of course, hardly new issues for file system research. A large number of systems have focused on these issues both in the context of file systems [1, 2, 4, 6] and lower-level storage [8, 5, 12]. We believe, however, that it is now possible and necessary to deliver these goals in a simpler and more automatic way. This new approach is made possible by advances in disk storage technology. Commodity inexpensive disks currently have capacities near 100-GB and disk capacity continues to grow. As a result, modern workstation and PC nodes will have tremendous disk storage available. Enough, we believe, to substantially change how file systems use this storage to achieve key goals. In particular, abundant disk storage can be used to greatly simplify the replication of valuable data to ensure its availability and to protect it from failures. This paper describes the design and implementation of Mammoth, a novel distributed file system that we are building. The key feature of Mammoth is that it stores files and directories as histories of immutable versions that are created each time a file or directory is modified. All Mammoth meta-data is stored in append-only fashion and all file data is read-only once it is created. As a result, Mammoth can replicate data on multiple loosely-connected nodes with minimal concern for consistency. The only overhead for creating and maintaining a replica is the storage it occupies. Mammoth replicates file data and meta-data as necessary to satisfy availability policies specified by users. Users assign these policies to individual files or groups of files and the system uses these policies to determine which file versions it replicates and on which nodes. This approach allows users to determine how their files are replicated, a task that is typically left to a system administrator in a traditional system. Mammoth handles node and network failures by allowing users to operate on whatever versions of files are currently accessible, even if more recent versions exist on other, currently unavailable nodes. This approach is similar to disconnected operation in Coda [4], with a key difference. Mammoth’s versioning approach greatly simplifies consistency maintenance and it provides a natural way to represent update conflicts in the file system. If disconnected nodes modify the same version of a file in Mammoth, the file system stores both new versions and records a branch in the file’s history. In summary, our goal is to build a simpler, more robust, and more scalable distributed file system. We believe that the fact that our approach is loosely coupled and requires minimal global coordination indicates that it has promise to substantially improve on previous systems such as those that implement RAID-like striping across multiple nodes [1, 2], implement virtual-disk level RAID or mirroring [5, 11, 12], or perform synchronous file updates to multiple nodes [4, 6].