When eBPF Meets FUSE
Improving the performance of user file systems
Ashish Bijlani, PhD Student, Georgia Tech@ashishbijlani

1

In-Kernel

vs“People who think thatuserspace filesystems arerealistic for anything but
toys are just misguided.”- Linus Torvalds

User File Systems“A lot of people once
thought Linux and themachines it ran on were
toys…
Apparently I’m misguided.”- Jeff Darcy

2

In-Kernel

vs• Examples– Ext4, OverlayFS, etc.
• Pros– Native performance• Cons– Poor security/reliability– Not easy to develop/
debug/maintainUser File Systems• Examples– EncFS, Gluster, etc.
• Pros– Improved security/reliability– Easy to develop/debug/
maintain• ConsPoor performance!
––Poorperformance!
3File Systems in User Space (FUSE)• State-of-the-art framework– All file system handlers
implemented in user space• Over 100+ FUSE file systems– Stackable: Android SDCardFS,EncFS, etc.
– Network: GlusterFS, Ceph,
Amazon S3FS, etc.

struct fuse_lowlevel_ops ops {.lookup = handle_lookup,.access = NULL,.getattr = handle_getattr,.setattr = handle_setattr,.open = handle_open,.read= handle_read,.readdir = handle_readdir,.write = handle_write,// more handlers …
.getxattr = handle_getxattr,.rename = handle_rename,.symlink = handle_symlink,.flush= NULL,}
4

FUSE Architecture
4’Over thenetwork

FUSE Daemon

Application

LIB FUSE
4

Stackable

User
Kernel1VFS
25FUSE DriverLower FS (e.g., EXT4)

3

QUEUE

65

FUSE Performance• “tar xf linux-4.17.tar.xz”– Intel i5-3350 quad core, Ubuntu 16.04.4 LTS– Linux 4.11.0, LibFUSE commit # 386b1bTime (sec)15

NativeFUSE

Overhead
HDD: 78.8%
SSD: 81.1%

118

More noticeable
on faster media!

40HDD

SSD

6

FUSE Performance• “cd linux-4.17; make tinyconfig; make -j4”
– Intel i5-3350 quad core, SSD, Ubuntu 16.04.4 LTS– Linux 4.11.0, LibFUSE commit # 386b1bTime (sec)4030

17.54%
overhead

20
100
NativeFUSE

7

FUSE Architectureopen(“/mnt/foo/bar”)

FUSE Daemon

Application

LIB FUSE

UserKernel

14VFS
25lookup(“foo”)

FUSE DriverLower FS (e.g., EXT4)

63

QUEUE

CONTEXTSWITCH

lookup()getattr()setattr()open()
read()
readdir()
write()…
rename()symlink()close()
getxattr()
setxattr()

8

300K

200K

100K
# Requests

Requests received by FUSE daemon• “cd linux-4.17; make tinyconfig; make -j4”
400K

0K

e
Writ

nkd
Rea
ir
ased
Rele
ddir
Rea
ndir
Ope
Unliir

ser
xatt

Mkd
Ge ta
RelenOpe
tettr

Crea
Setameattr

aRenGetkup
Loo9

FUSE Optimizations• Big 128K writes– “-o max_write=131072”• Zero data copying for data I/O– “-o splice_read, splice_write, splice_move”• Leveraging VFS caches– Page cache for data I/O• “-o writeback_cache”– Dentry and Inode caches for lookup() and getattr()• “entry_timeout”, “attr_timeout”
10

FUSE Performance• “cd linux-4.17; make tinyconfig; make -j4”
• Intel i5-3350 quad core, Ubuntu 16.04.4 LTS• Linux 4.11.0, LibFUSE commit # 386b1bTime (sec)

40
30
20
10

Opts do not help much!

0NativeRegular

Opts Enabled-o max_write=128K-o splice_read-o splice_write-o splice_move
entry_timeout > 0
attr_timeout > 0

Optimized
11Requests received by FUSE daemon• “cd linux-4.17; make tinyconfig; make -j4”
# Requests

400K

Regular

Optimized

300K
200K

4x fewer
lookup()s100K
0K

e
Writ

nkd
Rea
ir
ased
Rele
ddir
Rea
ndir
Ope

Unliir

ser
xatt

Mkd

Ge ta
RelenOpe

tettr

Crea

Setameattr

aRen

Getkup
Loo12

Requests received by FUSE daemon• “cd linux-4.17; make tinyconfig; make -j4”
# Requests

400K
300K

Regular13

Optimized

24200K

atime changes duringread() invalidate
cached attributes

100K0KeWritnkd
Rea
ir
ased
Releddir
Rea
ndir
Ope

Unliir

ser
xatt

Mkd

Ge ta
RelenOpetettr

Crea

Seta

me

attr

aRenGet

kup
Loo

13

Requests received by FUSE daemon• “cd linux-4.17; make tinyconfig; make -j4”
# Requests

400K

Regular

300K

Optimized

VFS issues getxattr()
for each write() forreading security labels

200K
100K

2’0K1’eWritnkd
Rea
ir
asedRele
ddir
Rea
ndir
Ope

Unliirserxatt

MkdGe taRelenOpetettr

CreaSeta

meattraRenGet

kup
Loo

14

eBPF
• Berkeley Packet Filter (BPF)– Pseudo machine architecture for packet filtering• eBPF enhances BPF– Evolved as a generic kernel extension framework– Used by tracing, perf, and network subsystems

15

eBPF Overview
Clang/LLVM• Extensions written in C
bytecode• Compiled into BPF codesyscall()• Code is verified andloaded into kernelVerifier• Execution under virtualmachine runtime• Shared BPF maps with bpf virtual machine
sandboxuser space
Kernel functions

BPF C program
userkernel

BPF Mapkey-value
data struct
16

eBPF Simplified Example
struct bpf_map_def map = {.type = BPF_MAP_TYPE_ARRAY,.key_size = sizeof(u32),.value_size = sizeof(u64),.max_entries = 1, // single element};// tracepoint/syscalls/sys_enter_open
int count_open(struct syscall *args) {u32 key = 0;
u64 *val = bpf_map_lookup_elem(map, &key);if (val) __sync_fetch_and_add(val, 1);}

17

ExtFUSE: eBPF meets FUSE
• Extension framework for File systems in User space– Register “thin” extensions - handle requests in kernel• Avoid user space context switch!
– Share data between FUSE daemon and extensions
using BPF maps
• Cache metadata in the kernel
18

FUSE Architecture
BPF Handlers

FUSE Daemon

LIB ExtFUSE

Application

LIB FUSE
470’CacheMetadata

LoadBPF
CodeVFS
Deliver req toextension
3’BPF VM
BPF MapUserKernel

14’Serve fromcache

2FUSE Driver

5Lower FS (e.g., EXT4)3QUEUE

619

ExtFUSE Simplified Example
struct bpf_map_def map = {.type = BPF_MAP_TYPE_HASH,.key_size = sizeof(u64), // ino (param 0).value_size = sizeof(struct fuse_attr_out),.max_entries = MAX_NUM_ATTRS, // 2 << 16};// getattr() kernel extension - cache attrs
int getattr(struct extfuse_args *args) {u32 key = bpf_extfuse_read(args, PARAM0);u64 *val = bpf_map_lookup_elem(map, &key);if (val) bpf_extfuse_write(args, PARAM0, val);}
20

ExtFUSE Simplified Example• Invalidate cached attrs from kernel extensions. E.g.,// setattr() kernel extension - invalidate attrs
int setattr(struct extfuse_args *args) {u32 key = bpf_extfuse_read(args, PARAM0);if (val) bpf_map_delete_elem(map, &key);}• Cache attrs from FUSE daemon– Insert into map on atime change• Similarly, cache lookup()s and xattr()s21

ExtFUSE PerformanceTime (sec)• “cd linux-4.17; make tinyconfig; make -j4”
• Intel i5-3350 quad core, SSD, Ubuntu
16.04.4 LTS
Overhead
Regular Latency:• Linux 4.11.0, LibFUSE commit # 386b1b17.54%
40
ExtFUSE Latency:
5.71%
30
ExtFUSE Memory:
20
50MB (worst case)10
Cached0
lookup, attr, xattr
NativeRegular

Optimized ExtFUSE

22

Requests received by FUSE daemon• “cd linux-4.17; make tinyconfig; make -j4”
# Requests

400K

Regular

Optimized

ExtFUSE

300K

Very few
getattr()s200K

Very few
getxattr()s100K0KeWritnkd
Rea
ir
ased
Rele
ddir
Rea
ndir
Ope

Unliirserxatt

MkdGe taRelenOpe

tettr

CreaSetameattr

aRenGetkup
Loo23

ExtFUSE Applications• BPF code to cache/invalidate meta-data in kernel– Applies potentially to all FUSE file systems– e.g., Gluster readdir ahead results could be cached• BPF code to perform custom filtering or perm checks– e.g., Android SDCardFS uid checks in lookup(), open()• BPF code to forward I/O requests to lower FS in kernel– e.g., install/remove target file descriptor in BPF map
24

ExtFUSE Status and References• Work in progress at Georgia Tech
– Applying to Gluster, Ceph, EncFS, Android
SDCardFS, etc.
– Project page: https://extfuse.github.io• References– FUSE performance study by FSL, Stony Brook– IOVisor eBPF Project– BPF Compiler Collection (BCC) Toolchain
25Thank You!
ashish.bijlani@gatech.edu
www.linkedin.com/in/ashishbijlani2627