# Presentation: 035220

## Ceph: A Scalable, High-Performance Distributed File System

- Sage Weil
- Scott Brandt
- Ethan Miller
- Darrell Long
- Carlos Maltzahn
*University of California, Santa Cruz*

## Project Goal

**Reliable**, **high-performance** distributed file system with excellent **scalability**
- Petabytes to exabytes, multi-terabyte files, billions of files
- Tens or hundreds of thousands of clients simultaneously accessing same files or directories
- POSIX interface
- Storage systems have long promised scalability, but have failed to deliver
- Continued reliance on traditional file systems principles
- Inode tables
- Block (or object) list allocation metadata
- Passive storage devices

## Ceph—Key Design Principles

- Maximal separation of data and metadata
- Object-based storage
- Independent metadata management
- CRUSH – data distribution function
- Intelligent disks
- Reliable Autonomic Distributed Object Store
- Dynamic metadata management
- Adaptive and scalable

## Outline

**Outline**

- Maximal separation of data and metadata
- Object-based storage
- Independent metadata management
- CRUSH – data distribution function
- Intelligent disks
- Reliable Autonomic Distributed Object Store
- Dynamic metadata management
- Adaptive and scalable

## Object-based StorageParadigm

- File System

- Storage component

- Applications

- Applications

- Logical Block Interface

- File System

- _Traditional Storage_

- _Object-based Storage_

- Hard Drive

- Object-based Storage Device (OSD)

**Object-based Storage****Paradigm**

## Ceph—Decoupled Data and Metadata

**Ceph—****Decoupled Data and Metadata**

- File System

- Applications

- Metadata Manager

- Client

## CRUSH—Simplifying Metadata

- Conventionally
- Directory contents (filenames)
- File inodes
- Ownership, permissions
- File size
- Block list
- CRUSH
- Small “map” completely specifies data distribution
- Function calculable everywhere used to locate objects
- Eliminates allocation lists
- Inodes “collapse” back into small, almost fixed-sized structures
- Embed inodes into directories that contain them
- No more large, cumbersome inode tables

## Outline

**Outline**

- Maximal separation of data and metadata
- Object-based storage
- Independent metadata management
- CRUSH – data distribution function
- Intelligent disks
- Reliable Autonomic Distributed Object Store
- Dynamic metadata management
- Adaptive and scalable

## RADOS—Reliable AutonomicDistributed Object Store

- RADOS

- EBOFS

- RADOS

- EBOFS

**RADOS—Reliable Autonomic****Distributed Object Store**

- Ceph OSDs are *intelligent*
- Conventional drives only respond to commands
- OSDs communicate and collaborate with their peers
- CRUSH allows us to delegate
- data replication
- failure detection
- failure recovery
- data migration
- OSDs collectively form a single logical object store
- Reliable
- Self-managing (autonomic)
- Distributed
- RADOS manages peer and client interaction
- EBOFS manages local object storage

## RADOS –Scalability

- Failure detection and recovery are distributed
- Centralized monitors used *only* to update map
- Maps updates are propagated by OSDs themselves
- No monitor broadcast necessary
- Identical “recovery” procedure used to respond to all map updates
- OSD failure
- Cluster expansion
- OSDs always collaborate to realize the newly specified data distribution

## EBOFS—Low-level object storage

**E**xtent and **B**-tree-based **O**bject **F**ile **S**ystem
- Non-standard interface and semantics
- Asynchronous notification of commits to disk
- Atomic compound data+metadata updates
- Extensive use of copy-on-write
- Revert to consistent state after failure
- User-space implementation
- We define our own interface—not limited by ill-suited kernel file system interface
- Avoid Linux VFS, page cache—designed under different usage assumptions

- RADOS

- EBOFS

## OSD Performance—EBOFS vs ext3, ReiserFSv3, XFS

- EBOFS writes saturate disk for request sizes over 32k
- Reads perform significantly better for large write sizes

## Outline

**Outline**

- Maximal separation of data and metadata
- Object-based storage
- Independent metadata management
- CRUSH – data distribution function
- Intelligent disks
- Reliable Autonomic Distributed Object Store
- Dynamic metadata management
- Adaptive and scalable

## Metadata—Traditional Partitioning

**Metadata—****Traditional Partitioning**

- Coarse distribution (static subtree partitioning)
- hierarchical partition preserves locality
- high management overhead: distribution becomes imbalanced as file system, workload change
- Finer distribution (hash-based partitioning)
- probabilistically less vulnerable to “hot spots,” workload change
- destroys locality (ignores underlying hierarchical structure)

- _Directory Hashing_
- Hash on directory portion of path only

**Coarse partition**

**Fine partition**

- _Static Subtree Partitioning_
- Portions of file hierarchy are statically assigned to MDS nodes
- (NFS, AFS, etc.)

- _File Hashing_
- Metadata distributed based on hash of full path (or inode #)

## Dynamic Subtree Partitioning

- Root

- Busy directory hashed across many MDS’s

- MDS 0

- MDS 1

- MDS 2

- MDS 3

- MDS 4

- Scalability
- Arbitrarily partitioned metadata
- Adaptability
- Cope with workload changes over time, and hot spots

## Metadata Scalability

- Up to 128 MDS nodes, and 250,000 metadata ops/second
- I/O rates of potentially many terabytes/second
- Filesystems containing many petabytes (or exabytes?) of data

## Conclusions

- Decoupled metadata improves scalability
- Eliminating allocation lists makes metadata simple
- MDS stays out of I/O path
- Intelligent OSDs
- Manage replication, failure detection, and recovery
- CRUSH distribution function makes it possible
- Global knowledge of complete data distribution
- Data locations *calculated *when needed
- Dynamic metadata management
- Preserve locality, improve performance
- Adapt to varying workloads, hot spots
- Scale
- High-performance and reliability with excellent scalability!

## Ongoing and Future Work

- Completion of prototype
- MDS failure recovery
- Scalable security architecture [Leung, StorageSS ’06]
- Quality of service
- Time travel (snapshots)
- RADOS improvements
- Dynamic replication of objects based on workload
- Reliability mechanisms: scrubbing, etc.

## Thanks!

- http://ceph.sourceforge.net/
- _Support from_
- Lawrence Livermore, Los Alamos, and Sandia
- National Laboratories