bcachefs: Principles of Operation
Kent Overstreet

Contents
1 Introduction and overview
1.1 Performance overview . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Bucket based allocation . . . . . . . . . . . . . . . . . . . . . . .

22
3

2 Feature overview
2.1 IO path options . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.1 Checksumming . . . . . . . . . . . . . . . . . . . . . . . .
2.1.2 Encryption . . . . . . . . . . . . . . . . . . . . . . . . . .
2.1.3 Compression . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Multiple devices . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 Replication . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Erasure coding . . . . . . . . . . . . . . . . . . . . . . . .
2.2.3 Device labels and targets . . . . . . . . . . . . . . . . . .
2.2.4 Caching . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.5 Durability . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Reflink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Inline data extents . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Subvolumes and snapshots . . . . . . . . . . . . . . . . . . . . . .
2.6 Quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4
4
4
4
5
5
6
6
6
7
7
8
8
8
9

3 Management
3.1 Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Mounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Fsck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Status of data . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Device management . . . . . . . . . . . . . . . . . . . . . . . . .
3.6.1 Filesystem resize . . . . . . . . . . . . . . . . . . . . . . .
3.6.2 Device add/removal . . . . . . . . . . . . . . . . . . . . .
3.7 Data management . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7.1 Data rereplicate . . . . . . . . . . . . . . . . . . . . . . .
3.7.2 Rebalance . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7.3 Scrub . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9
9
9
10
10
10
11
11
11
12
12
12
12

1

4 Options
12
4.1 File and directory options . . . . . . . . . . . . . . . . . . . . . . 12
4.2 Full option list . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.3 Error actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.4 Checksum types . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.5 Compression types . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.6 String hash types . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5 Debugging tools
16
5.1 Sysfs interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.1.1 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.1.2 Time stats . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.1.3 Internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.1.4 Unit and performance tests . . . . . . . . . . . . . . . . . 19
5.2 Debugfs interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3 Listing and dumping filesystem metadata . . . . . . . . . . . . . 20
5.3.1 bcachefs show-super . . . . . . . . . . . . . . . . . . . . . 20
5.3.2 bcachefs list . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3.3 bcachefs list journal . . . . . . . . . . . . . . . . . . . . . 20
5.3.4 bcachefs dump . . . . . . . . . . . . . . . . . . . . . . . . 20
6 ioctl interface

21

7 On disk format
22
7.1 Superblock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
7.2 Journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
7.3 Btrees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
7.4 Btree keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

1

Introduction and overview

Bcachefs is a modern, general purpose, copy on write filesystem descended frombcache, a block layer cache.
The internal architecture is very different from most existing filesystems
where the inode is central and many data structures hang off of the inode.
Instead, bcachefs is architected more like a filesystem on top of a relational
database, with tables for the different filesystem data types - extents, inodes,
dirents, xattrs, et cetera.
bcachefs supports almost all of the same features as other modern COW
filesystems, such as ZFS and btrfs, but in general with a cleaner, simpler, higher
performance design.

1.1

Performance overview

The core of the architecture is a very high performance and very low latency b+tree, which also is not a conventional b+ tree but more of hybrid, taking concepts
2

from compacting data structures: btree nodes are very large, log structured, andcompacted (resorted) as necessary in memory. This means our b+ trees are very
shallow compared to other filesystems.
What this means for the end user is that since we require very few seeks or
disk reads, filesystem latency is extremely good - especially cache cold filesystem
latency, which does not show up in most benchmarks but has a huge impact
on real world performance, as well as how fast the system ”feels” in normalinteractive usage. Latency has been a major focus throughout the codebase notably, we have assertions that we never hold b+ tree locks while doing IO,
and the btree transaction layer makes it easily to aggressively drop and retake
locks as needed - one major goal of bcachefs is to be the first general purpose
soft realtime filesystem.
Additionally, unlike other COW btrees, btree updates are journalled. This
greatly improves our write efficiency on random update workloads, as it means
btree writes are only done when we have a large block of updates, or whenrequired by memory reclaim or journal reclaim.

1.2

Bucket based allocation

As mentioned bcachefs is descended from bcache, where the ability to efficiently
invalidate cached data and reuse disk space was a core design requirement. To
make this possible the allocator divides the disk up into buckets, typically 512k
to 2M but possibly larger or smaller. Buckets and data pointers have generation
numbers: we can reuse a bucket with cached data in it without finding anddeleting all the data pointers by incrementing the generation number.
In keeping with the copy-on-write theme of avoiding update in place wherever possible, we never rewrite or overwrite data within a bucket - when we
allocate a bucket, we write to it sequentially and then we don’t write to it again
until the bucket has been invalidated and the generation number incremented.
This means we require a copying garbage collector to deal with internalfragmentation, when patterns of random writes leave us with many buckets
that are partially empty (because the data they contained was overwritten) copy GC evacuates buckets that are mostly empty by writing the data they
contain to new buckets. This also means that we need to reserve space on thedevice for the copy GC reserve when formatting - typically 8% or 12%.
There are some advantages to structuring the allocator this way, besides
being able to support cached data: By maintaining multiple write points that are writing to different buckets,
we’re able to easily and naturally segregate unrelated IO from different
processes, which helps greatly with fragmentation.
 The fast path of the allocator is essentially a simple bump allocator - thedisk space allocation is extremely fast
 Fragmentation is generally a non issue unless copygc has to kick in, and it
usually doesn’t under typical usage patterns. The allocator and copygc are3doing essentially the same things as the flash translation layer in SSDs, but
within the filesystem we have much greater visibility into where writes arecoming from and how to segregate them, as well as which data is actually
live - performance is generally more predictable than with SSDs undersimilar usage patterns.
 The same algorithms will in the future be used for managing SMR hard
drives directly, avoiding the translation layer in the hard drive - doing
this work within the filesystem should give much better performance andmuch more predictable latency.

2

Feature overview

2.1

IO path options

Most options that control the IO path can be set at either the filesystem level
or on individual inodes (files and directories). When set on a directory via thebcachefs attr command, they will be automatically applied recursively.
2.1.1

Checksumming

bcachefs supports both metadata and data checksumming - crc32c by default,
but stronger checksums are available as well. Enabling data checksumming
incurs some performance overhead - besides the checksum calculation, writes
have to be bounced for checksum stability (Linux generally cannot guaranteethat the buffer being written is not modified in flight), but reads generally do
not have to be bounced.
Checksum granularity in bcachefs is at the level of individual extents, which
results in smaller metadata but means we have to read entire extents in orderto verify the checksum. By default, checksummed and compressed extents arecapped at 64k. For most applications and usage scenarios this is an ideal tradeoff, but small random O DIRECT reads will incur significant overhead. In the
future, checksum granularity will be a per-inode option.
2.1.2

Encryption

bcachefs supports authenticated (AEAD style) encryption - ChaCha20/Poly1305.
When encryption is enabled, the poly1305 MAC replaces the normal data andmetadata checksums. This style of encryption is superior to typical block layeror filesystem level encryption (usually AES-XTS), which only operates on blocks
and doesn’t have a way to store nonces or MACs. In contrast, we store a nonce
and cryptographic MAC alongside data pointers - meaning we have a chain oftrust up to the superblock (or journal, in the case of unclean shutdowns) and
can definitely tell if metadata has been modified, dropped, or replaced with anearlier version - replay attacks are not possible.

4Encryption can only be specified for the entire filesystem, not per file ordirectory - this is because metadata blocks do not belong to a particular file.
All metadata except for the superblock is encrypted.
In the future we’ll probably add AES-GCM for platforms that have hardware acceleration for AES, but in the meantime software implementations of
ChaCha20 are also quite fast on most platforms.
scrypt is used for the key derivation function - for converting the user
supplied passphrase to an encryption key.
To format a filesystem with encryption, usebcachefs format --encrypted /dev/sda1
You will be prompted for a passphrase. Then, to use an encrypted filesystem
use the commandbcachefs unlock /dev/sda1
You will be prompted for the passphrase and the encryption key will be
added to your in-kernel keyring; mount, fsck and other commands will thenwork as usual.
The passphrase on an existing encrypted filesystem can be changed with thebcachefs set-passphrase command. To permanently unlock an encrypted
filesystem, use the bcachefs remove-passphrase command - this can be usefulwhen dumping filesystem metadata for debugging by the developers.
There is a wide macs option which controls the size of the cryptographic
MACs stored on disk. By default, only 80 bits are stored, which should be
sufficient security for most applications. With the wide macs option enabled westore the full 128 bit MAC, at the cost of making extents 8 bytes bigger.
2.1.3

Compression

bcachefs supports gzip, lz4 and zstd compression. As with data checksumming,
we compress entire extents, not individual disk blocks - this gives us better
compression ratios than other filesystems, at the cost of reduced small random
read performance.
Data can also be compressed or recompressed with a different algorithm inthe background by the rebalance thread, if the background compression option
is set.

2.2

Multiple devices

bcachefs is a multi-device filesystem. Devices need not be the same size: by
default, the allocator will stripe across all available devices but biasing in favorof the devices with more free space, so that all devices in the filesystem fill up at
the same rate. Devices need not have the same performance characteristics: wetrack device IO latency and direct reads to the device that is currently fastest.

5

2.2.1

Replicationbcachefs supports standard RAID1/10 style redundancy with the data replicas
and metadata replicas options. Layout is not fixed as with RAID10: a given
extent can be replicated across any set of devices; the bcachefs fs usage command shows how data is replicated within a filesystem.
2.2.2

Erasure coding

bcachefs also supports Reed-Solomon erasure coding - the same algorithm usedby most RAID5/6 implementations) When enabled with the ec option, thedesired redundancy is taken from the data replicas option - erasure coding of
metadata is not supported.
Erasure coding works significantly differently from both conventional RAID
implementations and other filesystems with similar features. In conventional
RAID, the ”write hole” is a significant problem - doing a small write within a
stripe requires the P and Q (recovery) blocks to be updated as well, and sincethose writes cannot be done atomically there is a window where the P and Q
blocks are inconsistent - meaning that if the system crashes and recovers witha drive missing, reconstruct reads for unrelated data within that stripe will becorrupted.
ZFS avoids this by fragmenting individual writes so that every write becomes a new stripe - this works, but the fragmentation has a negative effect on
performance: metadata becomes bigger, and both read and write requests areexcessively fragmented. Btrfs’s erasure coding implementation is more conventional, and still subject to the write hole problem.
bcachefs’s erasure coding takes advantage of our copy on write nature since updating stripes in place is a problem, we simply don’t do that. And since
excessively small stripes is a problem for fragmentation, we don’t erasure code
individual extents, we erasure code entire buckets - taking advantage of bucket
based allocation and copying garbage collection.
When erasure coding is enabled, writes are initially replicated, but one of
the replicas is allocated from a bucket that is queued up to be part of a new
stripe. When we finish filling up the new stripe, we write out the P and Q
buckets and then drop the extra replicas for all the data within that stripe - theeffect is similar to full data journalling, and it means that after erasure coding
is done the layout of our data on disk is ideal.
Since disks have write caches that are only flushed when we issue a cache
flush command - which we only do on journal commit - if we can tweak the
allocator so that the buckets used for the extra replicas are reused (and thenoverwritten again) immediately, this full data journalling should have negligible
overhead - this optimization is not implemented yet, however.
2.2.3

Device labels and targets

By default, writes are striped across all devices in a filesystem, but they may bedirected to a specific device or set of devices with the various target options. The
6allocator only prefers to allocate from devices matching the specified target; if
those devices are full, it will fall back to allocating from any device in thefilesystem.
Target options may refer to a device directly, e.g. foreground target=/dev/sda1,
or they may refer to a device label. A device label is a path delimited by periods- e.g. ssd.ssd1 (and labels need not be unique). This gives us ways of referring
to multiple devices in target options: If we specify ssd in a target option, thatwill refer to all devices with the label ssd or labels that start with ssd. (e.g.
ssd.ssd1, ssd.ssd2).
Four target options exist. These options all may be set at the filesystemlevel (at format time, at mount time, or at runtime via sysfs), or on a particular
file or directory:
foreground target: normal foreground data writes, and metadata if
metadata target is not set
metadata target: btree writes
background target: If set, user data (not metadata) will be moved to thistarget in the backgroundpromote target: If set, a cached copy will be added to this target on read, if
none exists
2.2.4

Caching

When an extent has multiple copies on different devices, some of those copies
may be marked as cached. Buckets containing only cached data are discarded
as needed by the allocator in LRU order.
When data is moved from one device to another according to thebackground target option, the original copy is left in place but marked as
cached. With the promote target option, the original copy is left unchanged
and the new copy on the promote target device is marked as cached.
To do writeback caching, set foreground target and promote target tothe cache device, and background target to the backing device. To do writearoundcaching, set foreground target to the backing device and promote target tothe cache device.
2.2.5

Durability

Some devices may be considered to be more reliable than others. For example,
we might have a filesystem composed of a hardware RAID array and several
NVME flash devices, to be used as cache. We can set replicas=2 so that losing
any of the NVME flash devices will not cause us to lose data, and then additionally we can set durability=2 for the hardware RAID device to tell bcachefs
that we don’t need extra replicas for data on that device - data on that device
will count as two replicas, not just one.

7

The durability option can also be used for writethrough caching: by setting
durability=0 for a device, it can be used as a cache and only as a cache - bcachefs
won’t consider copies on that device to count towards the number of replicas
we’re supposed to keep.

2.3

Reflinkbcachefs supports reflink, similarly to other filesystems with the same feature.
cp –reflink will create a copy that shares the underlying storage. Reading fromthat file will become slightly slower - the extent pointing to that data is moved
to the reflink btree (with a refcount added) and in the extents btree we leave a
key that points to the indirect extent in the reflink btree, meaning that we now
have to do two btree lookups to read from that data instead of just one.

2.4

Inline data extents

bcachefs supports inline data extents, controlled by the inline data option (onby default). When the end of a file is being written and is smaller than half ofthe filesystem blocksize, it will be written as an inline data extent. Inline data
extents can also be reflinked (moved to the reflink btree with a refcount added):
as a todo item we also intend to support compressed inline data extents.

2.5

Subvolumes and snapshotsbcachefs supports subvolumes and snapshots with a similar userspace interface
as btrfs. A new subvolume may be created empty, or it may be created as asnapshot of another subvolume. Snapshots are writeable and may be snapshotted again, creating a tree of snapshots.
Snapshots are very cheap to create: they’re not based on cloning of COW
btrees as with btrfs, but instead are based on versioning of individual keys inthe btrees. Many thousands or millions of snapshots can be created, with the
only limitation being disk space.
The following subcommands exist for managing subvolumes and snapshots: bcachefs subvolume create: Create a new, empty subvolume
 bcachefs subvolume destroy: Delete an existing subvolume or snapshot
 bcachefs subvolume snapshot: Create a snapshot of an existing subvolume

A subvolume can also be deleting with a normal rmdir after deleting all
the contents, as with rm -rf. Still to be implemented: read-only snapshots,recursive snapshot creation, and a method for recursively listing subvolumes.

8

2.6

Quotasbcachefs supports conventional user/group/project quotas. Quotas do not currently apply to snapshot subvolumes, because if a file changes ownership in thesnapshot it would be ambiguous as to what quota data within that file should
be charged to.
When a directory has a project ID set it is inherited automatically by descendants on creation and rename. When renaming a directory would cause theproject ID to change we return -EXDEV so that the move is done file by file,so that the project ID is propagated correctly to descendants - thus, project
quotas can be used as subdirectory quotas.

3Management

3.1

Formatting

To format a new bcachefs filesystem use the subcommand bcachefs format,
or mkfs.bcachefs. All persistent filesystem-wide options can be specified atformat time. For an example of a multi device filesystem with compression,
encryption, replication and writeback caching:bcachefs format --compression=lz4--encrypted--replicas=2--label=ssd.ssd1 /dev/sda--label=ssd.ssd2 /dev/sdb--label=hdd.hdd1 /dev/sdc--label=hdd.hdd2 /dev/sdd--label=hdd.hdd3 /dev/sde--label=hdd.hdd4 /dev/sdf--foreground_target=ssd--promote_target=ssd--background_target=hdd

3.2

\
\
\
\
\
\
\
\
\
\
\

Mounting

To mount a multi device filesystem, there are two options. You can specify allcomponent devices, separated by hyphens, e.g.
mount -t bcachefs /dev/sda:/dev/sdb:/dev/sdc /mnt
Or, use the mount.bcachefs tool to mount by filesystem UUID. Still todo: improve the mount.bcachefs tool to support mounting by filesystem label.
No special handling is needed for recovering from unclean shutdown. Journal
replay happens automatically, and diagnostic messages in the dmesg log will
indicate whether recovery was from clean or unclean shutdown.

9The -o degraded option will allow a filesystem to be mounted without all
the the devices, but will fail if data would be missing. The -o very degraded
can be used to attempt mounting when data would be missing.
Also relevant is the -o nochanges option. It disallows any and all writes tothe underlying devices, pinning dirty data in memory as necessary if for example
journal replay was necessary - think of it as a ”super read-only” mode. It canbe used for data recovery, and for testing version upgrades.
The -o verbose enables additional log output during the mount process.

3.3

Fsck

It is possible to run fsck either in userspace with the bcachefs fsck subcommand (also available as fsck.bcachefs, or in the kernel while mounting byspecifying the -o fsck mount option. In either case the exact same fsck implementation is being run, only the environment is different. Running fsck in the
kernel at mount time has the advantage of somewhat better performance, whilerunning in userspace has the ability to be stopped with ctrl-c and can prompt
the user for fixing errors. To fix errors while running fsck in the kernel, use the-o fix errors option.
The -n option passed to fsck implies the -o nochanges option; bcachefs
fsck -ny can be used to test filesystem repair in dry-run mode.

3.4

Status of data

The bcachefs fs usage may be used to display filesystem usage broken out
in various ways. Data usage is broken out by type: superblock, journal, btree,
data, cached data, and parity, and by which sets of devices extents are replicatedacross. We also give per-device usage which includes fragmentation due topartially used buckets.

3.5

Journal

The journal has a number of tunables that affect filesystem performance. Journal commits are fairly expensive operations as they require issuing FLUSH andFUA operations to the underlying devices. By default, we issue a journal flush
one second after a filesystem update has been done; this is controlled with thejournal flush delay option, which takes a parameter in milliseconds.
Filesystem sync and fsync operations issue journal flushes; this can be disabled with the journal flush disabled option - the journal flush delay
option will still apply, and in the event of a system crash we will never lose
more than (by default) one second of work. This option may be useful on a
personal workstation or laptop, and perhaps less appropriate on a server.
The journal reclaim thread runs in the background, kicking off btree node
writes and btree key cache flushes to free up space in the journal. Even in theabsence of space pressure it will run slowly in the background: this is controlled
by the journal reclaim delay parameter, with a default of 100 milliseconds.

10

The journal should be sized sufficiently that bursts of activity do not fill up
the journal too quickly; also, a larger journal mean that we can queue up larger
btree writes. The bcachefs device resize-journal can be used for resizing
the journal on disk on a particular device - it can be used on a mounted or
unmounted filesystem.
In the future, we should implement a method to see how much space iscurrently utilized in the journal.

3.6

Device management

3.6.1

Filesystem resize

A filesystem can be resized on a particular device with the bcachefs device
resize subcommand. Currently only growing is supported, not shrinking.
3.6.2

Device add/removal

The following subcommands exist for adding and removing devices from amounted filesystem: bcachefs device add: Formats and adds a new device to an existing
filesystem.
 bcachefs device remove: Permenantly removes a device from an existing filesystem.
 bcachefs device online: Connects a device to a running filesystem thatwas mounted without it (i.e. in degraded mode) bcachefs device offline: Disconnects a device from a mounted filesystem without removing it.
 bcachefs device evacuate: Migrates data off of a particular device toprepare for removal, setting it read-only if necessary.
 bcachefs device set-state: Changes the state of a member device:
one of rw (readwrite), ro (readonly), failed, or spare.

A failed device is considered to have 0 durability, and replicas on thatdevice won’t be counted towards the number of replicas an extent shouldhave by rereplicate - however, bcachefs will still attempt to read fromdevices marked as failed.
The bcachefs device remove, bcachefs device offline and bcachefs
device set-state commands take force options for when they would leave thefilesystem degraded or with data missing. Todo: regularize and improve those
options.

11

3.7

Data management

3.7.1

Data rereplicate

The bcachefs data rereplicate command may be used to scan for extents
that have insufficient replicas and write additional replicas, e.g. after a device
has been removed from a filesystem or after replication has been enabled orincreased.
3.7.2

Rebalance

To be implemented: a command for moving data between devices to equalize
usage on each device. Not normally required because the allocator attempts
to equalize usage across devices as it stripes, but can be necessary in certain
scenarios - i.e. when a two-device filesystem with replication enabled that isvery full has a third device added.
3.7.3

Scrub

To be implemented: a command for reading all data within a filesystem andensuring that checksums are valid, fixing bitrot when a valid copy can be found.

4

Options

Most bcachefs options can be set filesystem wide, and a significant subset canalso be set on inodes (files and directories), overriding the global defaults.
Filesystem wide options may be set when formatting, when mounting, or at
runtime via /sys/fs/bcachefs/<uuid>/options/. When set at runtime via
sysfs the persistent options in the superblock are updated as well; when options
are passed as mount parameters the persistent options are unmodified.

4.1

File and directory options

Options set on inodes (files and directories) are automatically inherited by theirdescendants, and inodes also record whether a given option was explicitly set or
inherited from their parent. When renaming a directory would cause inherited
attributes to change we fail the rename with -EXDEV, causing userspace to do
the rename file by file so that inherited attributes stay consistent.
Inode options are available as extended attributes. The options that havebeen explicitly set are available under the bcachefs namespace, and the effective options (explicitly set and inherited options) are available under thebcachefs effective namespace. Examples of listing options with the getfattr
command:$ getfattr -d -m ’^bcachefs\.’ filename$ getfattr -d -m ’^bcachefs_effective\.’ filename

12

Options may be set via the extended attribute interface, but it is preferable
to use the bcachefs setattr command as it will correctly propagate options
recursively.

4.2

Full option list

block size
Filesystem block size (default 4k)

formatbtree node size
Btree node size, default 256k

formatformat,mount,rutime

errorsAction to take on filesystem error

metadata replicas
format,mount,runtime
Number of replicas for metadata (journal and btree)
format,mount,runtime,inodedata replicas
Number of replicas for user data

replicas
Alias for both metadata replicas and data replicas
metadata checksum
Checksum type for metadata writes

formatformat,mount,runtime

data checksum
Checksum type for data writes

format,mount,runtime,inodecompression
Compression typeformat,mount,runtime,inodebackground compression
Background compression typeformat,mount,runtime,inodeformat,mount,runtime,inodestr hashHash function for string hash tables (directories and xattrs)
metadata target
Preferred target for metadata writes

format,mount,runtime,inodeforeground target
Preferred target for foreground writes

format,mount,runtime,inodebackground target

format,mount,runtime,inode13

Target for data to be moved to in the backgroundpromote target
Target for data to be copied to on read

format,mount,runtime,inodeerasure code
Enable erasure coding

format,mount,runtime,inode

inodes 32bit
Restrict new inode numbers to 32 bits

format,mount,runtime

shard inode numbers
format,mount,runtime
Use CPU id for high bits of new inode numbers.
wide macs
format,mount,runtime
Store full 128 bit cryptographic MACs (default 80)
inline data
Enable inline data extents (default on)

format,mount,runtime

format,mount,runtime
journal flush delay
Delay in milliseconds before automatic journal commit (default 1000)
journal flush disabledformat,mount,runtime
Disables journal flush on sync/fsync. journal flush delay remains ineffect, thus with the default setting not more than 1 second of work will
be lost.
journal reclaim delay
format,mount,runtime
Delay in milliseconds before automatic journal reclaim
format,mountacl
Enable POSIX ACLs
usrquota
Enable user quotas

format,mount

grpquota
Enable group quotas

format,mountprjquota
Enable project quotas

format,mountdegraded
Allow mounting with data degraded

14

mount

very degraded
Allow mounting with data missing

mountverboseExtra debugging info during mount/recovery

mount

fsck
Run fsck during mount

mount

fix errors
Fix errors without asking during fsck

mountratelimit errors
Ratelimit error messages during fsck

mountread only
Mount in read only mode

mountnochanges
Issue no writes, even for journal replay

mountnorecovery
Don’t replay the journal (not recommended)

mountnoexclDon’t open devices in exclusive mode

mountversion upgradeUpgrade on disk format to latest version

mountdiscard
Enable discard/TRIM support

device4.3

Error actions

The errors option is used for inconsistencies that indicate some sort of a bug.
Valid error actions are:continue Log the error but continue normal operation
ro Emergency read only, immediately halting any changes to the filesystem on
disk
panic Immediately halt the entire machine, printing a backtrace on the system
console15

4.4

Checksum types

Valid checksum types are:none
crc32c (default)
crc64

4.5

Compression types

Valid compression types are:none (default)
lz4
gzip
zstd

4.6

String hash types

Valid hash types for string hash tables are:crc32ccrc64
siphash (default)

5

Debugging tools

5.1

Sysfs interface

Mounted filesystems are available in sysfs at /sys/fs/bcachefs/<uuid>/ withvarious options, performance counters and internal debugging aids.
5.1.1

Options

Filesystem options may be viewed and changed via/sys/fs/bcachefs/<uuid>/options/, and settings changed via sysfs will bepersistently changed in the superblock as well.

16

5.1.2Time stats

bcachefs tracks the latency and frequency of various operations and events, withquantiles for latency/duration in the /sys/fs/bcachefs/<uuid>/time stats/directory.
blocked allocate
Tracks when allocating a bucket must wait because none are immediately
available, meaning the copygc thread is not keeping up with evacuating
mostly empty buckets or the allocator thread is not keeping up with invalidating and discarding buckets.
blocked allocate open bucket
Tracks when allocating a bucket must wait because all of our handles for
pinning open buckets are in use (we statically allocate 1024).
blocked journal
Tracks when getting a journal reservation must wait, either because journal reclaim isn’t keeping up with reclaiming space in the journal, or because journal writes are taking too long to complete and we already havetoo many in flight.
btree gc
Tracks when the btree gc code must walk the btree at runtime - for recalculating the oldest outstanding generation number of every bucket in thebtree.
btree lock contended readbtree lock contended intent
btree lock contended write
Track when taking a read, intent or write lock on a btree node must block.
btree node mem alloc
Tracks the total time to allocate memory in the btree node cache for a
new btree node.
btree node split
Tracks btree node splits - when a btree node becomes full and is split into
two new nodes
btree node compact
Tracks btree node compactions - when a btree node becomes full and needs
to be compacted on disk.
btree node merge
Tracks when two adjacent btree nodes are merged.

17

btree node sort
Tracks sorting and resorting entire btree nodes in memory, either after
reading them in from disk or for compacting prior to creating a new sorted
array of keys.
btree node read
Tracks reading in btree nodes from disk.
btree interior update foreground
Tracks foreground time for btree updates that change btree topology - i.e.
btree node splits, compactions and merges; the duration measured roughly
corresponds to lock held time.
btree interior update total
Tracks time to completion for topology changing btree updates; first they
have a foreground part that updates btree nodes in memory, then after
the new nodes are written there is a transaction phase that records an
update to an interior node or a new btree root as well as changes to thealloc btree.
data read
Tracks the core read path - looking up a request in the extents (and possibly also reflink) btree, allocating bounce buffers if necessary, issuing reads,
checksumming, decompressing, decrypting, and delivering completions.
data write
Tracks the core write path - allocating space on disk for a new write, allocating bounce buffers if necessary, compressing, encrypting, checksumming, issuing writes, and updating the extents btree to point to the new
data.
data promote
Tracks promote operations, which happen when a read operation writes
an additional cached copy of an extent to promote target. This is done
asynchronously from the original read.
journal flush write
Tracks writing of flush journal entries to disk, which first issue cache flush
operations to the underlying devices then issue the journal writes as FUA
writes. Time is tracked starting from after all journal reservations havereleased their references or the completion of the previous journal write.
journal noflush write
Tracks writing of non-flush journal entries to disk, which do not issuecache flushes or FUA writes.
journal flush seq
Tracks time to flush a journal sequence number to disk by filesystem sync
and fsync operations, as well as the allocator prior to reusing buckets whennone that do not need flushing are available.
185.1.3

Internals

btree cachedirty btree nodes
btree key cache
btree transactions
btree updates
new stripes
stripes heap
open buckets
journal debug
journal pins
io timers readio timers write
trigger journal flush
trigger gc
prune cacheread realloc races
extent migrate done
extent migrate raced
5.1.4

Unit and performance tests

Echoing into /sys/fs/bcachefs/<uuid>/perf test runs various low level btree
tests, some intended as unit tests and others as performance tests. The syntax
isecho <test_name> <nr_iterations> <nr_threads> > perf_test
When complete, the elapsed time will be printed in the dmesg log. The fulllist of tests that can be run can be found near the bottom of fs/bcachefs/tests.c.

19

5.2

Debugfs interface

The contents of every btree, as well as various internal per-btree-node information, are available under /sys/kernel/debug/bcachefs/<uuid>/.
For every btree, we have the following files:
btree name
Entire btree contents, one key per line
btree name-formats
Information about each btree node: the size of the packed bkey format,
how full each btree node is, number of packed and unpacked keys, andnumber of nodes and failed nodes in the in-memory search trees.
btree name-bfloat-failed
For each sorted set of keys in a btree node, we construct a binary searchtree in eytzinger layout with compressed keys. Sometimes we aren’t able toconstruct a correct compressed search key, which results in slower lookups;
this file lists the keys that resulted in these failed nodes.

5.3

Listing and dumping filesystem metadata

5.3.1

bcachefs show-superThis subcommand is used for examining and printing bcachefs superblocks. It
takes two optional parameters:-l: Print superblock layout, which records the amount of space reserved forthe superblock and the locations of the backup superblocks.
-f, --fields=(fields): List of superblock sections to print, all to print all
sections.
5.3.2

bcachefs list

This subcommand gives access to the same functionality as the debugfs interface,listing btree nodes and contents, but for offline filesystems.
5.3.3

bcachefs list journalThis subcommand lists the contents of the journal, which primarily records
btree updates ordered by when they occured.
5.3.4

bcachefs dump

This subcommand can dump all metadata in a filesystem (including multi device filesystems) as qcow2 images: when encountering issues that fsck can notrecover from and need attention from the developers, this makes it possible tosend the developers only the required metadata. Encrypted filesystems mustfirst be unlocked with bcachefs remove-passphrase.
20

6

ioctl interface

This section documents bcachefs-specific ioctls:
BCH IOCTL QUERY UUID
Returs the UUID of the filesystem: used to find the sysfs directory givena path to a mounted filesystem.
BCH IOCTL FS USAGE
Queries filesystem usage, returning global counters and a list of countersby bch replicas entry.
BCH IOCTL DEV USAGE
Queries usage for a particular device, as bucket and sector counts broken
out by data type.
BCH IOCTL READ SUPER
Returns the filesystem superblock, and optionally the superblock for aparticular device given that device’s index.
BCH IOCTL DISK ADD
Given a path to a device, adds it to a mounted and running filesystem. The
device must already have a bcachefs superblock; options and parameters
are read from the new device’s superblock and added to the member info
section of the existing filesystem’s superblock.
BCH IOCTL DISK REMOVE
Given a path to a device or a device index, attempts to remove it froma mounted and running filesystem. This operation requires walking thebtree to remove all references to this device, and may fail if data would
become degraded or lost, unless appropriate force flags are set.
BCH IOCTL DISK ONLINE
Given a path to a device that is a member of a running filesystem (indegraded mode), brings it back online.
BCH IOCTL DISK OFFLINE
Given a path or device index of a device in a multi device filesystem,
attempts to close it without removing it, so that the device may be readded later and the contents will still be available.
BCH IOCTL DISK SET STATEBCH IOCTL DISK GET IDX
BCH IOCTL DISK RESIZE

21

BCH IOCTL DISK RESIZE JOURNAL
BCH IOCTL DATABCH IOCTL SUBVOLUME CREATEBCH IOCTL SUBVOLUME DESTROYBCHFS IOC REINHERIT ATTRS

7

On disk format

7.1

Superblock

The superblock is the first thing to be read when accessing a bcachefs filesystem.
It is located 4kb from the start of the device, with redundant copies elsewhere- typically one immediately after the first superblock, and one at the end of thedevice.
The bch sb layout records the amount of space reserved for the superblock
as well as the locations of all the superblocks. It is included with every superblock, and additionally written 3584 bytes from the start of the device (512bytes before the first superblock).
Most of the superblock is identical across each device. The exceptions are
the dev idx field, and the journal section which gives the location of the journal.
The main section of the superblock contains UUIDs, version numbers, number of devices within the filesystem and device index, block size, filesystemcreation time, and various options and settings. The superblock also has anumber of variable length sections:BCH SB FIELD journal
List of buckets used for the journal on this device.
BCH SB FIELD members
List of member devices, as well as per-device options and settings, including bucket size, number of buckets and time when last mounted.
BCH SB FIELD crypt
Contains the main chacha20 encryption key, encrypted by the user’s passphrase,
as well as key derivation function settings.
BCH SB FIELD replicas
Contains a list of replica entries, which are lists of devices that have extents
replicated across them.
22

BCH SB FIELD quota
Contains timelimit and warnlimit fields for each quota type (user, groupand project) and counter (space, inodes).
BCH SB FIELD disk groups
Formerly referred to as disk groups (and still is throughout the code); thissection contains device label strings and records the tree structure of label
paths, allowing a label once parsed to be referred to by integer ID by thetarget options.
BCH SB FIELD clean
When the filesystem is clean, this section contains a list of journal entries
that are normally written with each journal write (struct jset): btree
roots, as well as filesystem usage and read/write counters (total amountof data read/written to this filesystem). This allows reading the journal
to be skipped after clean shutdowns.

7.2

JournalEvery journal write (struct jset) contains a list of entries: struct jset entry.
Below are listed the various journal entry types.
BCH JSET ENTRY btree key
This entry type is used to record every btree update that happens. It
contains one or more btree keys (struct bkey), and the btree id andlevel fields of jset entry record the btree ID and level the key belongs
to.
BCH JSET ENTRY btree root
This entry type is used for pointers btree roots. In the current implementation, every journal write still records every btree root, although that is
subject to change. A btree root is a bkey of type KEY TYPE btree ptr v2,
and the btree id and level fields of jset entry record the btree ID and
depth.
BCH JSET ENTRY clock
Records IO time, not wall clock time - i.e. the amount of reads and writes,in 512 byte sectors since the filesystem was created.
BCH JSET ENTRY usage
Used for certain persistent counters: number of inodes, current maximum
key version, and sectors of persistent reservations.
BCH JSET ENTRY data usage
Stores replica entries with a usage counter, in sectors.
BCH JSET ENTRY dev usage
Stores usage counters for each device: sectors used and buckets used,broken out by each data type.
23

7.3

Btrees7.4

Btree keys

KEY TYPE deleted
KEY TYPE whiteout
KEY TYPE error
KEY TYPE cookie
KEY TYPE hash whiteout
KEY TYPE btree ptr
KEY TYPE extent
KEY TYPE reservation
KEY TYPE inodeKEY TYPE inode generation
KEY TYPE dirent
KEY TYPE xattr
KEY TYPE alloc
KEY TYPE quota
KEY TYPE stripe
KEY TYPE reflink p
KEY TYPE reflink v
KEY TYPE inline data
KEY TYPE btree ptr v2
KEY TYPE indirect inline data
KEY TYPE alloc v2
KEY TYPE subvolume
KEY TYPE snapshot
KEY TYPE inode v2
KEY TYPE alloc v3

24