zfsonlinux/zfs — Issue #458: Slow unlink
Author: behlendorf
Created: 2011-11-21 23:30:24+00:00
State: open
Labels: Performance

From Prakash Surya

Currently, I dd'd /dev/urandom to a 1M file, and then duplicated that in another file up to the size of 1T. I then rm'd the 1T file, and things didn't seem too bad, except for the load on the system during and shortly after the remove. The load averages jumped way up, but now as I am writing this, they are dropping down to levels that I would think are normal (0.15, 6.22, 15.02).

Some output from top taken shortly after the rm completed:

top - 10:46:18 up 1 day, 14:15, 3 users, load average: 97.69, 66.12, 30.48
Tasks: 592 total, 92 running, 500 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 99.7%sy, 0.0%ni, 0.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 24599424k total, 11533504k used, 13065920k free, 1312k buffers
Swap: 0k total, 0k used, 0k free, 143948k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10153 root 0 -20 0 0 0 R 99.7 0.0 14:11.85 txg_sync
9980 root 39 19 0 0 0 R 18.2 0.0 0:43.47 z_fr_iss/26
10000 root 39 19 0 0 0 R 18.2 0.0 1:18.53 z_fr_iss/46
10041 root 39 19 0 0 0 R 18.2 0.0 1:19.05 z_fr_iss/87
9997 root 39 19 0 0 0 R 17.8 0.0 1:18.76 z_fr_iss/43
10011 root 39 19 0 0 0 R 17.8 0.0 1:08.75 z_fr_iss/57

There were many more z_fs_iss processes active, I would imagine those processes are what caused the load to skyrocket like it did (possibly processes waiting on disks?).

# zeno1 /mnt > time rm bigdata

real 0m27.420s
user 0m0.000s
sys 0m3.586s

# zeno1 /mnt > zpool status
pool: tank
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
sdm ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
sdr ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
sdy ONLINE 0 0 0
sdz ONLINE 0 0 0

errors: No known data errors

# zeno1 /mnt > zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 1.00T 42.6T 1 186 95.7K 22.7M
sda 32.3G 27.2G 0 6 3.68K 776K
sdb 40.1G 1.77T 0 7 3.59K 905K
sdc 40.1G 1.77T 0 7 3.63K 904K
sdd 40.0G 1.77T 0 7 3.68K 904K
sde 40.1G 1.77T 0 7 3.72K 904K
sdf 40.1G 1.77T 0 7 3.68K 904K
sdg 40.1G 1.77T 0 7 3.66K 904K
sdh 40.1G 1.77T 0 7 3.76K 904K
sdi 40.0G 1.77T 0 7 3.72K 904K
sdj 40.1G 1.77T 0 7 3.72K 904K
sdk 40.0G 1.77T 0 7 3.71K 904K
sdl 40.1G 1.77T 0 7 3.76K 904K
sdm 40.0G 1.77T 0 7 3.77K 904K
sdn 40.0G 1.77T 0 7 3.69K 904K
sdo 40.0G 1.77T 0 7 3.66K 903K
sdp 32.3G 27.2G 0 6 3.60K 776K
sdq 40.1G 1.77T 0 7 3.66K 905K
sdr 40.1G 1.77T 0 7 3.71K 905K
sds 40.0G 1.77T 0 7 3.70K 904K
sdt 40.1G 1.77T 0 7 3.71K 904K
sdu 40.1G 1.77T 0 7 3.70K 904K
sdv 40.0G 1.77T 0 7 3.63K 904K
sdw 40.1G 1.77T 0 7 3.65K 905K
sdx 40.1G 1.77T 0 7 3.64K 904K
sdy 40.0G 1.77T 0 7 3.62K 904K
sdz 40.0G 1.77T 0 7 3.67K 903K
---------- ----- ----- ----- ----- ----- -----