zfsonlinux/zfs — Issue #568: Hang of zfs on heavy delete
Author: fox-pluto
Created: 2012-02-09 10:07:37+00:00
State: open

HI,

I found a problem deleting a lot of file from a very full volume. My system is intel x86_64 ubuntu 3.0.0-12-server #20-Ubuntu SMP with 4 GB of ram.

The linux system is live but all the operations of reading and writing on the zfs volumes stop.

My zfs version is 0.6RC6

Here is the log I found in messages:

Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.636307] INFO: task arc_reclaim:2461 blocked for more than 120 seconds.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.636690] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637078] arc_reclaim D 0000000000000001 0 2461 2 0x00000000
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637083] ffff88010af23a40 0000000000000046 ffff88010af239e0 ffffffff810329a9
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637085] ffff88010af23fd8 ffff88010af23fd8 ffff88010af23fd8 0000000000012a40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637088] ffff8800d1925c80 ffff88010ebcdc80 ffff88010af23a60 ffff880103ede330
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637091] Call Trace:
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637100] [] ? default_spin_lock_flags+0x9/0x10
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637115] [] cv_wait_common+0x77/0xd0 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637118] [] ? add_wait_queue+0x60/0x60
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637123] [] __cv_wait+0x13/0x20 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637156] [] txg_wait_open+0x73/0xa0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637171] [] dmu_tx_wait+0xed/0xf0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637185] [] dmu_tx_assign+0x66/0x420 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637196] [] dmu_free_long_range_impl+0x184/0x270 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637208] [] dmu_free_long_range+0x4b/0x70 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637224] [] ? sa_handle_destroy+0x8f/0xa0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637241] [] zfs_rmnode+0x60/0x340 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637260] [] zfs_zinactive+0x89/0xd0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637278] [] zfs_inactive+0x5e/0x1c0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637282] [] ? truncate_pagecache+0x59/0x70
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637300] [] zpl_evict_inode+0x28/0x30 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637313] [] evict+0x91/0x170
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637316] [] dispose_list+0x3e/0x50
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637318] [] prune_icache+0x16f/0x320
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637321] [] shrink_icache_memory+0x21/0x50
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637331] [] arc_kmem_reap_now+0x82/0x110 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637342] [] arc_reclaim_thread+0x11a/0x150 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637352] [] ? arc_shrinker_func+0xd0/0xd0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637357] [] thread_generic_wrapper+0x78/0x90 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637362] [] ? __thread_create+0x160/0x160 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637364] [] kthread+0x8c/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637371] [] kernel_thread_helper+0x4/0x10
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637373] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637375] [] ? gs_change+0x13/0x13
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637472] INFO: task zfs_iput_taskq/:12268 blocked for more than 120 seconds.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.637892] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638326] zfs_iput_taskq/ D 0000000000000000 0 12268 2 0x00000000
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638330] ffff880104789ad0 0000000000000046 ffff880104789a70 ffffffff810329a9
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638332] ffff880104789fd8 ffff880104789fd8 ffff880104789fd8 0000000000012a40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638335] ffff8800810ddc80 ffff8800c701ae40 0000000000000000 ffffffff81c02380
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638337] Call Trace:
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638342] [] ? default_spin_lock_flags+0x9/0x10
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638345] [] __wait_on_freeing_inode+0xa8/0xd0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638349] [] ? autoremove_wake_function+0x40/0x40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638351] [] find_inode_fast+0x6c/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638353] [] ilookup+0x76/0xd0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638380] [] zfs_zget+0xba/0x210 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638393] [] ? dnode_rele+0x54/0x90 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638410] [] zfs_unlinked_drain+0x94/0x120 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638416] [] ? __switch_to+0xca/0x310
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638420] [] ? _raw_spin_lock+0xe/0x20
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638424] [] ? add_partial+0x58/0x90
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638441] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638444] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638449] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.638452] [] kthread+0x8c/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639466] [] ? dnode_rele+0x54/0x90 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639484] [] zfs_unlinked_drain+0x94/0x120 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639487] [] ? __switch_to+0xca/0x310
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639490] [] ? _raw_spin_lock+0xe/0x20
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639493] [] ? add_partial+0x58/0x90
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639501] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639512] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639517] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639520] [] kthread+0x8c/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639523] [] kernel_thread_helper+0x4/0x10
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639526] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639528] [] ? gs_change+0x13/0x13
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.639552] INFO: task rm:6210 blocked for more than 120 seconds.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640560] rm D 0000000000000001 0 6210 27181 0x00000000
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640563] ffff880052791ad8 0000000000000086 ffff880052791a78 ffffffff810329a9
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640566] ffff880052791fd8 ffff880052791fd8 ffff880052791fd8 0000000000012a40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640569] ffff8800a0712e40 ffff88010e9b2e40 ffff880052791af8 ffff880103ede330
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640572] Call Trace:
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640576] [] ? default_spin_lock_flags+0x9/0x10
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640586] [] cv_wait_common+0x77/0xd0 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640589] [] ? add_wait_queue+0x60/0x60
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640593] [] __cv_wait+0x13/0x20 [spl]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640617] [] txg_wait_open+0x73/0xa0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640629] [] dmu_tx_wait+0xed/0xf0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640642] [] dmu_tx_assign+0x66/0x420 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640663] [] dmu_free_long_range_impl+0x184/0x270 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640674] [] dmu_free_long_range+0x4b/0x70 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640692] [] zfs_rmnode+0x60/0x340 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640710] [] zfs_zinactive+0x89/0xd0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640728] [] zfs_inactive+0x5e/0x1c0 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640745] [] ? truncate_pagecache+0x59/0x70
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640762] [] zpl_evict_inode+0x28/0x30 [zfs]
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640765] [] evict+0x91/0x170
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640767] [] iput_final+0xd2/0x1a0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640769] [] iput+0x38/0x50
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640771] [] do_unlinkat+0x148/0x1c0
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640774] [] ? sys_newfstatat+0x2a/0x40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640776] [] sys_unlinkat+0x22/0x40
Feb 9 10:23:43 ubuntu-brick kernel: [ 2157.640780] [] system_call_fastpath+0x16/0x1b
Feb 9 10:24:44 ubuntu-brick kernel: [ 2217.814731] INFO: rcu_sched_state detected stall on CPU 0 (t=15000 jiffies)
Feb 9 10:25:01 ubuntu-brick CRON[18556]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.458556] INFO: task arc_reclaim:2461 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459761] arc_reclaim D 0000000000000001 0 2461 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459765] ffff88010af23a40 0000000000000046 ffff88010af239e0 ffffffff810329a9
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459768] ffff88010af23fd8 ffff88010af23fd8 ffff88010af23fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459771] ffff8800d1925c80 ffff88010ebcdc80 ffff88010af23a60 ffff880103ede330
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459774] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459782] [] ? default_spin_lock_flags+0x9/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459795] [] cv_wait_common+0x77/0xd0 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459800] [] ? add_wait_queue+0x60/0x60
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459805] [] __cv_wait+0x13/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459830] [] txg_wait_open+0x73/0xa0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459844] [] dmu_tx_wait+0xed/0xf0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459858] [] dmu_tx_assign+0x66/0x420 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459871] [] dmu_free_long_range_impl+0x184/0x270 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459884] [] dmu_free_long_range+0x4b/0x70 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459902] [] ? sa_handle_destroy+0x8f/0xa0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459919] [] zfs_rmnode+0x60/0x340 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459939] [] zfs_zinactive+0x89/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459957] [] zfs_inactive+0x5e/0x1c0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459961] [] ? truncate_pagecache+0x59/0x70
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459979] [] zpl_evict_inode+0x28/0x30 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459983] [] evict+0x91/0x170
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459985] [] dispose_list+0x3e/0x50
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459987] [] prune_icache+0x16f/0x320
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.459989] [] shrink_icache_memory+0x21/0x50
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460000] [] arc_kmem_reap_now+0x82/0x110 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460010] [] arc_reclaim_thread+0x11a/0x150 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460021] [] ? arc_shrinker_func+0xd0/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460026] [] thread_generic_wrapper+0x78/0x90 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460031] [] ? __thread_create+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460034] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460038] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460041] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460043] [] ? gs_change+0x13/0x13
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460094] INFO: task z_wr_int/5:11812 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.460702] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461338] z_wr_int/5 D 0000000000000001 0 11812 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461341] ffff8800816d1d10 0000000000000046 ffff880110f87488 ffffffffa01812b6
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461344] ffff8800816d1fd8 ffff8800816d1fd8 ffff8800816d1fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461346] ffff880098e78000 ffff8800816cae40 ffffffffffffff10 ffff880082a12720
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461349] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461355] [] ? kmem_free_debug+0x16/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461358] [] __mutex_lock_slowpath+0xd7/0x150
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461377] [] ? zio_destroy+0xa6/0xe0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461379] [] mutex_lock+0x22/0x40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461397] [] vdev_queue_io_done+0x30/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461415] [] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461433] [] zio_execute+0x9f/0xf0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461438] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461442] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461446] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461448] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461451] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461453] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461455] [] ? gs_change+0x13/0x13
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.461457] INFO: task z_wr_int/6:11813 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462876] z_wr_int/6 D 0000000000000002 0 11813 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462891] ffff8800816d3d10 0000000000000046 000000002cd05616 ffffffffa01812b6
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462898] ffff8800816d3fd8 ffff8800816d3fd8 ffff8800816d3fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462904] ffff88008164c560 ffff8800816cc560 ffffffffffffff10 ffff880082a12720
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462910] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462917] [] ? kmem_free_debug+0x16/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462921] [] __mutex_lock_slowpath+0xd7/0x150
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462942] [] ? vdev_cache_write+0x154/0x170 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462946] [] mutex_lock+0x22/0x40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462965] [] vdev_queue_io_done+0x30/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.462985] [] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463004] [] zio_execute+0x9f/0xf0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463011] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463014] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463021] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463025] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463029] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463033] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463037] [] ? gs_change+0x13/0x13
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463040] INFO: task z_wr_int/9:11816 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.463750] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464480] z_wr_int/9 D 0000000000000001 0 11816 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464484] ffff8800816e1d10 0000000000000046 ffffffffa01812b6 ffff8800816d9720
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464487] ffff8800816e1fd8 ffff8800816e1fd8 ffff8800816e1fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464490] ffff88010da18000 ffff8800816d9720 ffff8800816e1d40 ffff880082a12720
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464493] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464500] [] ? kmem_free_debug+0x16/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464503] [] __mutex_lock_slowpath+0xd7/0x150
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464524] [] ? vdev_cache_write+0x154/0x170 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464527] [] mutex_lock+0x22/0x40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464546] [] vdev_queue_io_done+0x30/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464564] [] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464581] [] zio_execute+0x9f/0xf0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464586] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464589] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464594] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464596] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464599] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464602] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464604] [] ? gs_change+0x13/0x13
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.464606] INFO: task z_wr_int/13:11820 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.465332] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466093] z_wr_int/13 D 0000000000000001 0 11820 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466096] ffff8800816f9d10 0000000000000046 ffffffffa01812b6 ffff8800816f0000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466099] ffff8800816f9fd8 ffff8800816f9fd8 ffff8800816f9fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466101] ffff8800986adc80 ffff8800816f0000 ffff8800816f9d40 ffff880082a12720
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466104] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466109] [] ? kmem_free_debug+0x16/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466112] [] __mutex_lock_slowpath+0xd7/0x150
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466130] [] ? vdev_cache_write+0x154/0x170 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466133] [] mutex_lock+0x22/0x40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466150] [] vdev_queue_io_done+0x30/0xd0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466168] [] zio_vdev_io_done+0x88/0x190 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466186] [] zio_execute+0x9f/0xf0 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466191] [] taskq_thread+0x1b5/0x390 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466193] [] ? try_to_wake_up+0x200/0x200
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466198] [] ? task_alloc+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466200] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466203] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466205] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466207] [] ? gs_change+0x13/0x13
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.466220] INFO: task txg_sync:11946 blocked for more than 120 seconds.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467064] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467890] txg_sync D 0000000000000002 0 11946 2 0x00000000
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467893] ffff880100ad5c00 0000000000000046 ffff880100ad5ba0 ffffffff810329a9
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467896] ffff880100ad5fd8 ffff880100ad5fd8 ffff880100ad5fd8 0000000000012a40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467899] ffff8800816dae40 ffff880082a02e40 ffff880082821640 ffffc9008622d790
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467901] Call Trace:
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467905] [] ? default_spin_lock_flags+0x9/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467912] [] cv_wait_common+0x77/0xd0 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467915] [] ? add_wait_queue+0x60/0x60
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467920] [] __cv_wait+0x13/0x20 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467938] [] zio_wait+0xfb/0x170 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467954] [] dsl_pool_sync+0xca/0x450 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467971] [] spa_sync+0x38e/0xa00 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467973] [] ? default_wake_function+0x12/0x20
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467976] [] ? autoremove_wake_function+0x16/0x40
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467979] [] ? __wake_up+0x53/0x70
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.467996] [] txg_sync_thread+0x216/0x390 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468012] [] ? txg_init+0x260/0x260 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468029] [] ? txg_init+0x260/0x260 [zfs]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468034] [] thread_generic_wrapper+0x78/0x90 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468039] [] ? __thread_create+0x160/0x160 [spl]
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468041] [] kthread+0x8c/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468044] [] kernel_thread_helper+0x4/0x10
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468047] [] ? flush_kthread_worker+0xa0/0xa0
Feb 9 10:25:43 ubuntu-brick kernel: [ 2277.468049] [] ? gs_change+0x13/0x13
Feb 9 10:27:44 ubuntu-brick kernel: [ 2397.660752] INFO: rcu_sched_state detected stall on CPU 0 (t=60031 jiffies)

Do you think the but is already solved in the head??

thanks for your help, let me know if you need more info.
Stefano