zfsonlinux/zfs — Issue #583: kernel panic during large files removal
Author: mikhmv
Created: 2012-02-24 02:02:53+00:00
State: open

Hi, I have a blocking bug!!!

When I remove small files (40GB as an example) command rm is working well but on removing large files (184 GB) I have regularly kernel panic.
------------------------------------------------------------------------------------------------------------------------------
File:
-rwxr-x--- 1 oneadmin cloud 184G 2012-02-05 10:27 bioUML-DB-10003-disk-1.raw.bz2
------------------------------------------------------------------------------------------------------
Command:
$date; rm -r bioUML-DB-10003-disk-1.raw.bz2; date
Thu Feb 23 20:32:32 EST 2012
Thu Feb 23 20:42:01 EST 2012
After executing this command I got messages in dmesg.
-----------------------------------------------------------------------------------------------------------

$ uname -a
Linux s0 3.0.0-16-server #28-Ubuntu SMP Fri Jan 27 18:03:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
------------------------------------------------------------------------------------------------------------------
$ sudo zpool status
pool: tank
state: ONLINE
scan: scrub repaired 0 in 7h41m with 0 errors on Wed Feb 15 17:42:42 2012
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
d1 ONLINE 0 0 0
d2 ONLINE 0 0 0
d3 ONLINE 0 0 0
d4 ONLINE 0 0 0
d5 ONLINE 0 0 0

errors: No known data errors
---------------------------------------------------------------------------------------------------------
max@s0:~$ dpkg -s ubuntu-zfs
Package: ubuntu-zfs
Status: install ok installed
Priority: optional
Section: metapackages
Installed-Size: 32
Maintainer: Darik Horn
Architecture: amd64
Version: 4~oneiric
Depends: spl, spl-dkms, zfs-dkms, zfsutils
Description: Native ZFS filesystem metapackage for Ubuntu.
max@s0:~$ dpkg -s zfsutils
Package: zfsutils
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 876
Maintainer: Darik Horn
Architecture: amd64
Source: zfs-linux
Version: 0.6.0.34-0ubuntu1~oneiric1
Replaces: zfs
Depends: libc6 (>= 2.8), libefi1, libnvpair1, libspl1, libuuid1 (>= 2.16), libuutil1, libzavl1, libzfs1, libzpool1
Recommends: zfs-dkms
Suggests: nfs-kernel-server, zfs-initramfs
Conflicts: zfs, zfs-fuse
Conffiles:
/etc/default/zfs df1821c2e84e8f19ebc805f3a656cf1c
/etc/init.d/zfs d549d0455e2299b4318cd9038742ec4a
/etc/zfs/zdev.conf b006284e64b215ca619aeb56d2df9bf5
Description: Native ZFS management utilities for Linux
This package provides the zpool and zfs commands that are used to
manage ZFS filesystems.

-----------------------------------------------------------------------------------------------------------------------------------
dmesg:

[687848.632630] INFO: task oned:8231 blocked for more than 120 seconds.
[687848.651278] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.688812] oned D 0000000000000038 0 8231 1 0x00000000
[687848.688826] ffff880aed60db88 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.688837] ffff880aed60dfd8 ffff880aed60dfd8 ffff880aed60dfd8 0000000000012a40
[687848.688845] ffff8827b1611720 ffff880e94312e40 ffff880aed60db98 ffff8827f7dd3b30
[687848.688852] Call Trace:
[687848.688870] [] schedule+0x3f/0x60
[687848.688902] [] cv_wait_common+0x77/0xd0 [spl]
[687848.688911] [] ? add_wait_queue+0x60/0x60
[687848.688922] [] __cv_wait+0x13/0x20 [spl]
[687848.688982] [] txg_wait_open+0x73/0xa0 [zfs]
[687848.689009] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.689046] [] zfs_write+0x377/0xc50 [zfs]
[687848.689056] [] ? __dentry_open+0x2bb/0x320
[687848.689063] [] ? vma_adjust+0x464/0x490
[687848.689096] [] ? dmu_object_size_from_db+0x69/0xa0 [zfs]
[687848.689132] [] zpl_write_common+0x52/0x80 [zfs]
[687848.689165] [] zpl_write+0x68/0xa0 [zfs]
[687848.689182] [] vfs_write+0xb3/0x180
[687848.689201] [] sys_write+0x4a/0x90
[687848.689223] [] system_call_fastpath+0x16/0x1b
[687848.689245] INFO: task kvm:8147 blocked for more than 120 seconds.
[687848.708632] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.748404] kvm D 0000000000000012 0 8147 1 0x00000000
[687848.748418] ffff88396b3e3a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.748428] ffff88396b3e3fd8 ffff88396b3e3fd8 ffff88396b3e3fd8 0000000000012a40
[687848.748439] ffff8827b1ac5c80 ffff883c93af9720 ffff88396b3e3a88 ffff8827f7dd3b30
[687848.748450] Call Trace:
[687848.748465] [] schedule+0x3f/0x60
[687848.748480] [] cv_wait_common+0x77/0xd0 [spl]
[687848.748488] [] ? add_wait_queue+0x60/0x60
[687848.748503] [] __cv_wait+0x13/0x20 [spl]
[687848.748547] [] txg_wait_open+0x73/0xa0 [zfs]
[687848.748578] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.748611] [] zfs_write+0x377/0xc50 [zfs]
[687848.748621] [] ? __perf_event_task_sched_out+0x30/0x60
[687848.748630] [] ? rb_erase+0xce/0x150
[687848.748638] [] ? __remove_hrtimer+0x60/0xc0
[687848.748671] [] zpl_write_common+0x52/0x80 [zfs]
[687848.748703] [] zpl_write+0x68/0xa0 [zfs]
[687848.748740] [] ? zpl_write_common+0x80/0x80 [zfs]
[687848.748749] [] do_loop_readv_writev+0x59/0x90
[687848.748758] [] do_readv_writev+0x1ae/0x1d0
[687848.748769] [] ? do_futex+0xa8/0x1d0
[687848.748777] [] vfs_writev+0x3c/0x50
[687848.748784] [] sys_pwritev+0xba/0xc0
[687848.748795] [] system_call_fastpath+0x16/0x1b
[687848.748800] INFO: task kvm:8242 blocked for more than 120 seconds.
[687848.769694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.811872] kvm D 0000000000000038 0 8242 1 0x00000000
[687848.811886] ffff880ab95e1b88 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.811897] ffff880ab95e1fd8 ffff880ab95e1fd8 ffff880ab95e1fd8 0000000000012a40
[687848.811906] ffff8827b1611720 ffff880de3092e40 ffff880ab95e1b98 ffff8827f7dd3b30
[687848.811915] Call Trace:
[687848.811924] [] schedule+0x3f/0x60
[687848.811938] [] cv_wait_common+0x77/0xd0 [spl]
[687848.811947] [] ? add_wait_queue+0x60/0x60
[687848.811960] [] __cv_wait+0x13/0x20 [spl]
[687848.811994] [] txg_wait_open+0x73/0xa0 [zfs]
[687848.812036] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.812071] [] zfs_write+0x377/0xc50 [zfs]
[687848.812105] [] zpl_write_common+0x52/0x80 [zfs]
[687848.812138] [] zpl_write+0x68/0xa0 [zfs]
[687848.812148] [] vfs_write+0xb3/0x180
[687848.812154] [] sys_pwrite64+0xa2/0xb0
[687848.812163] [] system_call_fastpath+0x16/0x1b
[687848.812169] INFO: task kvm:8243 blocked for more than 120 seconds.
[687848.834606] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.879674] kvm D 0000000000000002 0 8243 1 0x00000000
[687848.879683] ffff880b9dc97b88 0000000000000086 ffff880b9dc97b28 ffffffffa01052b6
[687848.879694] ffff880b9dc97fd8 ffff880b9dc97fd8 ffff880b9dc97fd8 0000000000012a40
[687848.879705] ffff8827b28dc560 ffff880de3091720 ffff880b9dc97b98 ffff8827f7dd3b30
[687848.879713] Call Trace:
[687848.879727] [] ? kmem_free_debug+0x16/0x20 [spl]
[687848.879736] [] schedule+0x3f/0x60
[687848.879748] [] cv_wait_common+0x77/0xd0 [spl]
[687848.879763] [] ? add_wait_queue+0x60/0x60
[687848.879789] [] __cv_wait+0x13/0x20 [spl]
[687848.879842] [] txg_wait_open+0x73/0xa0 [zfs]
[687848.879885] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.879929] [] zfs_write+0x377/0xc50 [zfs]
[687848.879972] [] zpl_write_common+0x52/0x80 [zfs]
[687848.880018] [] zpl_write+0x68/0xa0 [zfs]
[687848.880058] [] vfs_write+0xb3/0x180
[687848.880074] [] sys_pwrite64+0xa2/0xb0
[687848.880096] [] system_call_fastpath+0x16/0x1b
[687848.880113] INFO: task kvm:11994 blocked for more than 120 seconds.
[687848.903888] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687848.951632] kvm D 0000000000000038 0 11994 1 0x00000000
[687848.951640] ffff883fda8a1a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687848.951650] ffff883fda8a1fd8 ffff883fda8a1fd8 ffff883fda8a1fd8 0000000000012a40
[687848.951660] ffff883fdca02e40 ffff883fda82dc80 ffff883fda8a1a88 ffff8827f7dd3b30
[687848.951671] Call Trace:
[687848.951678] [] schedule+0x3f/0x60
[687848.951692] [] cv_wait_common+0x77/0xd0 [spl]
[687848.951698] [] ? add_wait_queue+0x60/0x60
[687848.951711] [] __cv_wait+0x13/0x20 [spl]
[687848.951746] [] txg_wait_open+0x73/0xa0 [zfs]
[687848.951773] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687848.951806] [] zfs_write+0x377/0xc50 [zfs]
[687848.951835] [] ? kvm_fast_pio_out+0x64/0x80 [kvm]
[687848.951846] [] ? handle_exit+0x163/0x380 [kvm_amd]
[687848.951880] [] zpl_write_common+0x52/0x80 [zfs]
[687848.951912] [] zpl_write+0x68/0xa0 [zfs]
[687848.951930] [] ? svm_get_msr+0x155/0x320 [kvm_amd]
[687848.951973] [] ? zpl_write_common+0x80/0x80 [zfs]
[687848.951988] [] do_loop_readv_writev+0x59/0x90
[687848.952029] [] do_readv_writev+0x1ae/0x1d0
[687848.952056] [] vfs_writev+0x3c/0x50
[687848.952078] [] sys_pwritev+0xba/0xc0
[687848.952094] [] system_call_fastpath+0x16/0x1b
[687848.952108] INFO: task kvm:8183 blocked for more than 120 seconds.
[687848.976763] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.026272] kvm D 0000000000000026 0 8183 1 0x00000000
[687849.026280] ffff8806b2143a78 0000000000000086 0000000000000001 ffff8827f7dd3ae0
[687849.026290] ffff8806b2143fd8 ffff8806b2143fd8 ffff8806b2143fd8 0000000000012a40
[687849.026299] ffff883ff157dc80 ffff8807f55d0000 ffff8806b2143a88 ffff8827f7dd3b30
[687849.026308] Call Trace:
[687849.026315] [] schedule+0x3f/0x60
[687849.026327] [] cv_wait_common+0x77/0xd0 [spl]
[687849.026334] [] ? add_wait_queue+0x60/0x60
[687849.026347] [] __cv_wait+0x13/0x20 [spl]
[687849.026381] [] txg_wait_open+0x73/0xa0 [zfs]
[687849.026409] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.026446] [] zfs_write+0x377/0xc50 [zfs]
[687849.026454] [] ? __perf_event_task_sched_out+0x30/0x60
[687849.026489] [] zpl_write_common+0x52/0x80 [zfs]
[687849.026522] [] zpl_write+0x68/0xa0 [zfs]
[687849.026529] [] ? rw_copy_check_uvector+0xf5/0x130
[687849.026564] [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.026571] [] do_loop_readv_writev+0x59/0x90
[687849.026577] [] do_readv_writev+0x1ae/0x1d0
[687849.026584] [] ? futex_wake+0x113/0x130
[687849.026592] [] ? do_futex+0xa8/0x1d0
[687849.026597] [] vfs_writev+0x3c/0x50
[687849.026605] [] sys_pwritev+0xba/0xc0
[687849.026615] [] system_call_fastpath+0x16/0x1b
[687849.026630] INFO: task kvm:8240 blocked for more than 120 seconds.
[687849.052708] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.105106] kvm D 0000000000000038 0 8240 1 0x00000000
[687849.105118] ffff883459339a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.105127] ffff883459339fd8 ffff883459339fd8 ffff883459339fd8 0000000000012a40
[687849.105137] ffff8827b1611720 ffff8837f7134560 ffff883459339a88 ffff8827f7dd3b30
[687849.105147] Call Trace:
[687849.105154] [] schedule+0x3f/0x60
[687849.105168] [] cv_wait_common+0x77/0xd0 [spl]
[687849.105175] [] ? add_wait_queue+0x60/0x60
[687849.105191] [] __cv_wait+0x13/0x20 [spl]
[687849.105228] [] txg_wait_open+0x73/0xa0 [zfs]
[687849.105260] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.105294] [] zfs_write+0x377/0xc50 [zfs]
[687849.105333] [] zpl_write_common+0x52/0x80 [zfs]
[687849.105366] [] zpl_write+0x68/0xa0 [zfs]
[687849.105400] [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.105408] [] do_loop_readv_writev+0x59/0x90
[687849.105416] [] do_readv_writev+0x1ae/0x1d0
[687849.105423] [] vfs_writev+0x3c/0x50
[687849.105427] [] sys_pwritev+0xba/0xc0
[687849.105433] [] system_call_fastpath+0x16/0x1b
[687849.105438] INFO: task kvm:8244 blocked for more than 120 seconds.
[687849.131994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.184830] kvm D 000000000000003b 0 8244 1 0x00000000
[687849.184840] ffff8833a1c2db88 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.184850] ffff8833a1c2dfd8 ffff8833a1c2dfd8 ffff8833a1c2dfd8 0000000000012a40
[687849.184861] ffff8827b1615c80 ffff8837f7135c80 ffff8833a1c2db98 ffff8827f7dd3b30
[687849.184870] Call Trace:
[687849.184878] [] schedule+0x3f/0x60
[687849.184890] [] cv_wait_common+0x77/0xd0 [spl]
[687849.184899] [] ? add_wait_queue+0x60/0x60
[687849.184912] [] __cv_wait+0x13/0x20 [spl]
[687849.184947] [] txg_wait_open+0x73/0xa0 [zfs]
[687849.184975] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.185007] [] zfs_write+0x377/0xc50 [zfs]
[687849.185016] [] ? _raw_spin_lock+0xe/0x20
[687849.185025] [] ? fsnotify_clear_marks_by_inode+0x94/0xf0
[687849.185036] [] ? __perf_event_task_sched_out+0x30/0x60
[687849.185045] [] ? hrtick_update+0x38/0x40
[687849.185090] [] zpl_write_common+0x52/0x80 [zfs]
[687849.185131] [] zpl_write+0x68/0xa0 [zfs]
[687849.185144] [] vfs_write+0xb3/0x180
[687849.185159] [] sys_pwrite64+0xa2/0xb0
[687849.185176] [] system_call_fastpath+0x16/0x1b
[687849.185186] INFO: task kvm:8246 blocked for more than 120 seconds.
[687849.212468] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.267657] kvm D 000000000000003b 0 8246 1 0x00000000
[687849.267665] ffff883799351a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.267676] ffff883799351fd8 ffff883799351fd8 ffff883799351fd8 0000000000012a40
[687849.267686] ffff8827b1615c80 ffff88362dd2dc80 ffff883799351a88 ffff8827f7dd3b30
[687849.267693] Call Trace:
[687849.267699] [] schedule+0x3f/0x60
[687849.267710] [] cv_wait_common+0x77/0xd0 [spl]
[687849.267716] [] ? add_wait_queue+0x60/0x60
[687849.267727] [] __cv_wait+0x13/0x20 [spl]
[687849.267764] [] txg_wait_open+0x73/0xa0 [zfs]
[687849.267791] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.267823] [] zfs_write+0x377/0xc50 [zfs]
[687849.267857] [] zpl_write_common+0x52/0x80 [zfs]
[687849.267900] [] zpl_write+0x68/0xa0 [zfs]
[687849.267939] [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.267953] [] do_loop_readv_writev+0x59/0x90
[687849.267968] [] do_readv_writev+0x1ae/0x1d0
[687849.267981] [] vfs_writev+0x3c/0x50
[687849.267996] [] sys_pwritev+0xba/0xc0
[687849.268011] [] system_call_fastpath+0x16/0x1b
[687849.268039] INFO: task kvm:8241 blocked for more than 120 seconds.
[687849.296215] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[687849.353700] kvm D 0000000000000038 0 8241 1 0x00000000
[687849.353709] ffff8832fe875a78 0000000000000082 0000000000000001 ffff8827f7dd3ae0
[687849.353731] ffff8832fe875fd8 ffff8832fe875fd8 ffff8832fe875fd8 0000000000012a40
[687849.353771] ffff8827b1611720 ffff883653405c80 ffff8832fe875a88 ffff8827f7dd3b30
[687849.353795] Call Trace:
[687849.353809] [] schedule+0x3f/0x60
[687849.353841] [] cv_wait_common+0x77/0xd0 [spl]
[687849.353859] [] ? add_wait_queue+0x60/0x60
[687849.353886] [] __cv_wait+0x13/0x20 [spl]
[687849.353926] [] txg_wait_open+0x73/0xa0 [zfs]
[687849.353973] [] dmu_tx_wait+0xed/0xf0 [zfs]
[687849.354027] [] zfs_write+0x377/0xc50 [zfs]
[687849.354047] [] ? _raw_spin_lock+0xe/0x20
[687849.354092] [] zpl_write_common+0x52/0x80 [zfs]
[687849.354133] [] zpl_write+0x68/0xa0 [zfs]
[687849.354146] [] ? perf_event_task_sched_out+0x2e/0xa0
[687849.354184] [] ? zpl_write_common+0x80/0x80 [zfs]
[687849.354202] [] do_loop_readv_writev+0x59/0x90
[687849.354217] [] do_readv_writev+0x1ae/0x1d0
[687849.354229] [] vfs_writev+0x3c/0x50
[687849.354240] [] sys_pwritev+0xba/0xc0
[687849.354252] [] system_call_fastpath+0x16/0x1b