Scale and Concurrency of GIGA+: File System Directories with Millions of Files

Over the last two decades, research in large file systems was driven by application workloads that emphasized access to very large files. Most cluster file systems provide scalable file I/O bandwidth by enabling parallel access using techniques such as data striping, object-based architectures and distributed locking. [16–18, 30, 41, 47]. Few file systems scale metadata performance by using a coarse-grained distribution of metadata over multiple servers [12, 36, 41, 49]. But most file systems cannot scale access to a large number of files [14, 52]. In particular, they lack scalable support for ingesting millions to billions of small files in a single directory a growing use case for data-intensive applications [14, 34, 39]. One such motivating application is the most important I/O workload in today’s supercomputers: checkpointrestart, where many parallel applications running on, for instance, ORNL’s CrayXT5 cluster (with 18,688 nodes of twelve processors each) periodically write application state into a file per process, all stored in one directory [7]. Applications that do this per-process checkpointing are sensitive to long file creation delays because of the generally slow file creation rate, especially in one directory, in today’s file systems [7]. Today’s requirement for 40,000 file creates per second in a single directory [34] will become much bigger in the impending Exascale-era, when applications may run on clusters with up to billions of CPU cores [23]. Supercomputing checkpoint-restart, although important, might not be a sufficient reason for overhauling the current file system directory implementations. Yet there are diverse applications, such as gene sequencing, image processing [48], phone logs for accounting and billing, and photo storage [6], that essentially want to store an unbounded number of files that are logically part of one directory. Although these applications are often using the file system as a fast, lightweight “key-value store”, replacing the underlying file system with a database is an oft-rejected option because it is undesirable to port existing code to use a new API (like SQL) and because traditional databases do not provide the scalability of cluster file systems running on thousands of nodes [3, 5, 42, 46]. We present a file system directory service, GIGA+, that uses highly concurrent and decentralized hash-based indexing, and that scales to store at least millions of files in a single POSIX-compliant directory and sustain hundreds of thousands of create insertions per second. The key feature of the GIGA+ approach is to enable higher concurrency for index mutations (particularly creates) by eliminating system-wide serialization and synchronization. GIGA+ realizes this principle by aggressively distributing large, mutating directories over a cluster of server nodes, by disabling directory entry caching in clients, and by allowing each node to migrate, without notification or synchronization, portions of the directory for load balancing. Like traditional hash-based distributed indices [13, 27, 41], GIGA+ incrementally hashes a directory into a growing number of partitions. However, GIGA+ tries harder to eliminate synchronization and prohibits migration if load balancing is unlikely to be improved. Clients do not cache directory entries; they cache only the directory index. This cached index can have stale pointers to servers that no longer manage specific ranges in the space of the hashed directory entries (filenames). Clients using stale index values to target an incorrect server have their cached index corrected by the incorrectly targeted server. Stale client indices are aggressively improved by transmitting the history of splits of all partitions known to a server. Even the addition of new servers is supported with minimal migration of directory entries and delayed notification to clients. In addition, because 99.99% of the directories have less than 8,000 entries [4, 10], GIGA+ represents small directories in one partition so most directories will be essentially like traditional directories. We have built a skeleton cluster file system with GIGA+ directories that layers on existing lower layer file systems using FUSE [15]. It uses the traditional UNIX VFS interface and provides POSIX-like semantics to support unmodified applications. Our evaluation demonstrates that GIGA+ directories scale linearly on a cluster of 32 servers and deliver a throughput of more than 98,000 file creates per second – outscaling the Ceph file system [49] and the HBase distributed key-value store [19], and exceeding peta-scale scalability requirements [34]. GIGA+ indexing also achieves effective load balancing with one to two orders of magnitude less re-partitioning than if it was based on consistent hashing [22, 45].