Kafka on ZFS: Better Living Through Filesystems -- Kafka Summit San Francisco 2018
Channel: Hugh O'Brien
Date: 2019-02-14

Slides and higher-res version available here:
https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs
https://kafka-summit.org/sessions/kafka-zfs-better-living-filesystems/
https://docs.google.com/presentation/d/16Rj_OvJYueKAHeEs6XNqLcGWWxwsfR5id0Nr5Do2qao/edit?usp=sharing
----

You’re doing disk IO wrong, let ZFS show you the way. ZFS on Linux is now stable. Say goodbye to JBOD, to directories in your reassignment plans, to unevenly used disks. Instead, have 8K Cloud IOPS for $25, SSD speed reads on spinning disks, in-kernel LZ4 compression and the smartest page cache on the planet. (Fear compactions no more!)

Learn how Jet’s Kafka clusters squeeze every drop of disk performance out of Azure, all completely transparent to Kafka.

. Striping cheap disks to maximize instance IOPS
. Block compression to reduce disk usage by ~80% (JSON data)
. Instance SSD as the secondary read cache (storing compressed data), eliminating 99% of disk reads and safe across host redeployments
. Upcoming features: Compressed blocks in memory, potentially quadrupling your page cache (RAM) for free

We’ll cover:

. Basic Principles
. Adapting ZFS for cloud instances (gotchas)
. Performance tuning for Kafka
. Benchmarks