zfsonlinux/zfs — Issue #521: deadlock on multithreaded I/O
Author: akorn
Created: 2012-01-08 15:25:06+00:00
State: open

Hi,

I've experienced a deadlock similar to #496; however, the stack traces are sufficiently different for me to think it's not the same issue.

My pool configuration is:

pool: tank
state: ONLINE
scan: scrub canceled on Sun Jan 8 01:21:49 2012
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sas1 ONLINE 0 0 0
sas2 ONLINE 0 0 0
sas3 ONLINE 0 0 0
sas4 ONLINE 0 0 0
sas5 ONLINE 0 0 0
logs
zfs_zil ONLINE 0 0 0
cache
zfs_l2arc ONLINE 0 0 0

errors: No known data errors

At the time of the deadlock, I was rsyncing (at a pathologically low speed) from a remote box, and several local processes were also accessing the pool. I/O load was not particularly high.

After the deadlock, it was impossible to log in (SSH would connect, but the shell would never start).

I'm running zfs 0.6.0.43-0ubuntu1~oneiric1 with https://github.com/zfsonlinux/zfs/commit/a576bc07951aac3b207559d792750dd34ba950d6 merged and ZFS_LINK_MAX raised to ((0x1 << 31) - 1U) as suggested in #514.

Kernel log:

```
Jan 8 07:18:01 jayhawk kernel: INFO: rcu_sched_state detected stall on CPU 0 (t=6000 jiffies)
Jan 8 07:18:01 jayhawk kernel: sending NMI to all CPUs:
Jan 8 07:18:01 jayhawk kernel: NMI backtrace for cpu 0
Jan 8 07:18:01 jayhawk kernel: CPU 0
Jan 8 07:18:01 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:01 jayhawk kernel:
Jan 8 07:18:01 jayhawk kernel: Pid: 31058, comm: zfs_iput_taskq/ Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:01 jayhawk kernel: RIP: 0010:[] [] find_next_bit+0x1e/0xc0
Jan 8 07:18:01 jayhawk kernel: RSP: 0018:ffff88041fc03d68 EFLAGS: 00000046
Jan 8 07:18:01 jayhawk kernel: RAX: 000000000000000c RBX: 0000000000000002 RCX: 0000000000000000
Jan 8 07:18:01 jayhawk kernel: RDX: 0000000000000002 RSI: 000000000000000c RDI: ffffffff81aaae80
Jan 8 07:18:01 jayhawk kernel: RBP: ffff88041fc03d68 R08: ffffffff81aaae80 R09: 0000000000000000
Jan 8 07:18:01 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000d442
Jan 8 07:18:03 jayhawk kernel: R13: ffffffff81aaae80 R14: 0000000000000400 R15: 0000000000000096
Jan 8 07:18:03 jayhawk kernel: FS: 00007fdae3d17700(0000) GS:ffff88041fc00000(0000) knlGS:000000005611b960
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 0000000055f8f394 CR3: 00000003f93a8000 CR4: 00000000000006f0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process zfs_iput_taskq/ (pid: 31058, threadinfo ffff8803d5876000, task ffff8802fcc7c470)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff88041fc03db8 ffffffff81067dc2 ffff880400000001 0000000000000001
Jan 8 07:18:03 jayhawk kernel: ffff88041fc03dd8 0000000000002710 ffffffff81a29280 ffffffff81a29280
Jan 8 07:18:03 jayhawk kernel: ffff88041fc0df40 7fffffffffffffff ffff88041fc03dc8 ffffffff8106bd52
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 07:18:03 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 07:18:03 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 07:18:03 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 07:18:03 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 07:18:03 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 07:18:03 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 07:18:03 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 07:18:03 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 07:18:03 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 07:18:03 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 07:18:03 jayhawk kernel: [] ? call_softirq+0x1c/0x30
Jan 8 07:18:03 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 07:18:03 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 07:18:03 jayhawk kernel: [] ? dbuf_rele_and_unlock+0x1a/0x1f0 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_buf_rele+0x25/0x30 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dbuf_prefetch+0x1cf/0x280 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? dmu_zfetch+0x291/0xd40 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 07:18:03 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 07:18:03 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? __schedule+0x3bc/0x890
Jan 8 07:18:03 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 07:18:03 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 07:18:03 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 07:18:03 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 07:18:03 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 07:18:03 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 07:18:03 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 07:18:03 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 07:18:03 jayhawk kernel: Code: e2 06 48 0f bd f6 48 8d 04 32 5d c3 90 55 48 39 f2 48 89 f0 48 89 e5 73 79 48 89 d1 49 89 d1 48 c1 e9 06 49 83 e1 c0 4c 8d 04 cf
Jan 8 07:18:03 jayhawk kernel: <48> 89 f7 4c 29 cf 83 e2 3f 75 5f 48 f7 c7 c0 ff ff ff 74 2b 49
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 07:18:03 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 07:18:03 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 07:18:03 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 07:18:03 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 07:18:03 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 07:18:03 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 07:18:03 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 07:18:03 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 07:18:03 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 07:18:03 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 07:18:03 jayhawk kernel: [] ? call_softirq+0x1c/0x30
Jan 8 07:18:03 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 07:18:03 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 07:18:03 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 07:18:03 jayhawk kernel: [] ? dbuf_rele_and_unlock+0x1a/0x1f0 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_buf_rele+0x25/0x30 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dbuf_prefetch+0x1cf/0x280 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? dmu_zfetch+0x291/0xd40 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 07:18:03 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 07:18:03 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 07:18:03 jayhawk kernel: [] ? __schedule+0x3bc/0x890
Jan 8 07:18:03 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 07:18:03 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 07:18:03 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 07:18:03 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 07:18:03 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 07:18:03 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 07:18:03 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 07:18:03 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 07:18:03 jayhawk kernel: NMI backtrace for cpu 1
Jan 8 07:18:03 jayhawk kernel: CPU 1
Jan 8 07:18:03 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: Pid: 0, comm: kworker/0:0 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:03 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 07:18:03 jayhawk kernel: RSP: 0000:ffff8804054cbeb8 EFLAGS: 00000246
Jan 8 07:18:03 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054cbfd8 RCX: 00000000ffffffff
Jan 8 07:18:03 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 07:18:03 jayhawk kernel: RBP: ffff8804054cbeb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: FS: 00002b78693086f0(0000) GS:ffff88041fc20000(0000) knlGS:000000005611b960
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 0000000055bf2000 CR3: 00000001eaa8c000 CR4: 00000000000006e0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process kworker/0:0 (pid: 0, threadinfo ffff8804054ca000, task ffff8804054d0000)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff8804054cbee8 ffffffff81054410 ffff8804054cbfd8 ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054cbf18 ffffffff8104aad6
Jan 8 07:18:03 jayhawk kernel: ffff8804054cbf08 03f4fe87f591794a 0000000000000001 0000000000000000
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: NMI backtrace for cpu 2
Jan 8 07:18:03 jayhawk kernel: CPU 2
Jan 8 07:18:03 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:03 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 07:18:03 jayhawk kernel: RSP: 0018:ffff8804054e3eb8 EFLAGS: 00000246
Jan 8 07:18:03 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054e3fd8 RCX: 00000000ffffffff
Jan 8 07:18:03 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 07:18:03 jayhawk kernel: RBP: ffff8804054e3eb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: FS: 00007f10feb20700(0000) GS:ffff88041fc40000(0000) knlGS:000000005611b960
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 000000000805eb50 CR3: 0000000400c60000 CR4: 00000000000006e0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff8804054e2000, task ffff8804054d5b40)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff8804054e3ee8 ffffffff81054410 ffff8804054e3fd8 ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054e3f18 ffffffff8104aad6
Jan 8 07:18:03 jayhawk kernel: ffff8804054e3f08 0cf32962194b79f0 0000000000000002 0000000000000000
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: NMI backtrace for cpu 3
Jan 8 07:18:03 jayhawk kernel: CPU 3
Jan 8 07:18:03 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:03 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 07:18:03 jayhawk kernel: RSP: 0000:ffff8804054ffeb8 EFLAGS: 00000246
Jan 8 07:18:03 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054fffd8 RCX: 00000000ffffffff
Jan 8 07:18:03 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 07:18:03 jayhawk kernel: RBP: ffff8804054ffeb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: FS: 00007f266407c700(0000) GS:ffff88041fc60000(0000) knlGS:000000005611b960
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 00000000f7664a09 CR3: 0000000131795000 CR4: 00000000000006e0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff8804054fe000, task ffff8804054ec470)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff8804054ffee8 ffffffff81054410 ffff8804054fffd8 ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054fff18 ffffffff8104aad6
Jan 8 07:18:03 jayhawk kernel: ffff8804054fff08 b7b1c45da28780f9 0000000000000003 0000000000000000
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: NMI backtrace for cpu 4
Jan 8 07:18:03 jayhawk kernel: CPU 4
Jan 8 07:18:03 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:03 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 07:18:03 jayhawk kernel: RSP: 0018:ffff880405513eb8 EFLAGS: 00000246
Jan 8 07:18:03 jayhawk kernel: RAX: 0000000000000000 RBX: ffff880405513fd8 RCX: 00000000ffffffff
Jan 8 07:18:03 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 07:18:03 jayhawk kernel: RBP: ffff880405513eb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: FS: 00007f5cb68a67a0(0000) GS:ffff88041fc80000(0000) knlGS:0000000055849e70
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 00007fff0f8dedb8 CR3: 000000038454b000 CR4: 00000000000006e0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff880405512000, task ffff88040550ada0)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff880405513ee8 ffffffff81054410 ffff880405513fd8 ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: 0000000000000000 0000000000000000 ffff880405513f18 ffffffff8104aad6
Jan 8 07:18:03 jayhawk kernel: ffff880405513f08 2ebceaa58ffe4b79 0000000000000004 0000000000000000
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: NMI backtrace for cpu 5
Jan 8 07:18:03 jayhawk kernel: CPU 5
Jan 8 07:18:03 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 07:18:03 jayhawk kernel:
Jan 8 07:18:03 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 07:18:03 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 07:18:03 jayhawk kernel: RSP: 0000:ffff88040552beb8 EFLAGS: 00000246
Jan 8 07:18:03 jayhawk kernel: RAX: 0000000000000000 RBX: ffff88040552bfd8 RCX: 00000000ffffffff
Jan 8 07:18:03 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 07:18:03 jayhawk kernel: RBP: ffff88040552beb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: FS: 00002aeec5586790(0000) GS:ffff88041fca0000(0000) knlGS:000000005b7fab70
Jan 8 07:18:03 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 07:18:03 jayhawk kernel: CR2: 00000000f75c9a09 CR3: 0000000316c7a000 CR4: 00000000000006e0
Jan 8 07:18:03 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 07:18:03 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 07:18:03 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff88040552a000, task ffff8804055216d0)
Jan 8 07:18:03 jayhawk kernel: Stack:
Jan 8 07:18:03 jayhawk kernel: ffff88040552bee8 ffffffff81054410 ffff88040552bfd8 ffffffff81aaae80
Jan 8 07:18:03 jayhawk kernel: 0000000000000000 0000000000000000 ffff88040552bf18 ffffffff8104aad6
Jan 8 07:18:03 jayhawk kernel: ffff88040552bf08 5e3fcc1570a6a5d4 0000000000000005 0000000000000000
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 07:18:03 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 07:18:03 jayhawk kernel: Call Trace:
Jan 8 07:18:03 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 07:18:03 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 07:18:03 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
```
I don't know if the box actually recovered, but a similar message was logged a few hours later:

```
Jan 8 09:03:56 jayhawk kernel: INFO: rcu_sched_state detected stall on CPU 0 (t=6000 jiffies)
Jan 8 09:03:56 jayhawk kernel: sending NMI to all CPUs:
Jan 8 09:03:56 jayhawk kernel: NMI backtrace for cpu 0
Jan 8 09:03:56 jayhawk kernel: INFO: rcu_sched_state detected stalls on CPUs/tasks: { 0} (detected by 3, t=6002 jiffies)
Jan 8 09:03:56 jayhawk kernel: CPU 0
Jan 8 09:03:56 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:56 jayhawk kernel:
Jan 8 09:03:56 jayhawk kernel: Pid: 31058, comm: zfs_iput_taskq/ Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:56 jayhawk kernel: RIP: 0010:[] [] find_next_bit+0x1e/0xc0
Jan 8 09:03:56 jayhawk kernel: RSP: 0018:ffff88041fc03d68 EFLAGS: 00000046
Jan 8 09:03:56 jayhawk kernel: RAX: 000000000000000c RBX: 0000000000000002 RCX: 0000000000000000
Jan 8 09:03:56 jayhawk kernel: RDX: 0000000000000002 RSI: 000000000000000c RDI: ffffffff81aaae80
Jan 8 09:03:56 jayhawk kernel: RBP: ffff88041fc03d68 R08: ffffffff81aaae80 R09: 0000000000000000
Jan 8 09:03:56 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000d442
Jan 8 09:03:56 jayhawk kernel: R13: ffffffff81aaae80 R14: 0000000000000400 R15: 0000000000000096
Jan 8 09:03:56 jayhawk kernel: FS: 00007fdae3d17700(0000) GS:ffff88041fc00000(0000) knlGS:00000000556fa6c0
Jan 8 09:03:56 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:56 jayhawk kernel: CR2: 00000000ffee4e10 CR3: 0000000001a05000 CR4: 00000000000006f0
Jan 8 09:03:56 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:56 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:56 jayhawk kernel: Process zfs_iput_taskq/ (pid: 31058, threadinfo ffff8803d5876000, task ffff8802fcc7c470)
Jan 8 09:03:56 jayhawk kernel: Stack:
Jan 8 09:03:56 jayhawk kernel: ffff88041fc03db8 ffffffff81067dc2 ffff880400000001 0000000000000001
Jan 8 09:03:56 jayhawk kernel: ffff88041fc03dd8 0000000000002710 ffffffff81a29280 ffffffff81a29280
Jan 8 09:03:56 jayhawk kernel: ffff88041fc0df40 7fffffffffffffff ffff88041fc03dc8 ffffffff8106bd52
Jan 8 09:03:56 jayhawk kernel: Call Trace:
Jan 8 09:03:56 jayhawk kernel:
Jan 8 09:03:56 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 09:03:56 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 09:03:56 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 09:03:56 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 09:03:56 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 09:03:56 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 09:03:57 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 09:03:57 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 09:03:57 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 09:03:57 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 09:03:57 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 09:03:57 jayhawk kernel: [] ? call_softirq+0x1c/0x30
Jan 8 09:03:57 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 09:03:57 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: [] ? kmem_alloc_debug+0x65/0x120 [spl]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_hold_impl+0x52/0xb0 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_prefetch+0xe2/0x280 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? dmu_zfetch+0x291/0xd40 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 09:03:57 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:03:57 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? finish_task_switch+0x44/0xe0
Jan 8 09:03:57 jayhawk kernel: [] ? __schedule+0x3bc/0x890
Jan 8 09:03:57 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:03:57 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 09:03:57 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 09:03:57 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 09:03:57 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 09:03:57 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 09:03:57 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 09:03:57 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 09:03:57 jayhawk kernel: Code: e2 06 48 0f bd f6 48 8d 04 32 5d c3 90 55 48 39 f2 48 89 f0 48 89 e5 73 79 48 89 d1 49 89 d1 48 c1 e9 06 49 83 e1 c0 4c 8d 04 cf
Jan 8 09:03:57 jayhawk kernel: <48> 89 f7 4c 29 cf 83 e2 3f 75 5f 48 f7 c7 c0 ff ff ff 74 2b 49
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 09:03:57 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 09:03:57 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 09:03:57 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 09:03:57 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 09:03:57 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 09:03:57 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 09:03:57 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 09:03:57 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 09:03:57 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 09:03:57 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 09:03:57 jayhawk kernel: [] ? call_softirq+0x1c/0x30
Jan 8 09:03:57 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 09:03:57 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 09:03:57 jayhawk kernel: [] ? kmem_alloc_debug+0x65/0x120 [spl]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_hold_impl+0x52/0xb0 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_prefetch+0xe2/0x280 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? dmu_zfetch+0x291/0xd40 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 09:03:57 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:03:57 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 09:03:57 jayhawk kernel: [] ? finish_task_switch+0x44/0xe0
Jan 8 09:03:57 jayhawk kernel: [] ? __schedule+0x3bc/0x890
Jan 8 09:03:57 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:03:57 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 09:03:57 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 09:03:57 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 09:03:57 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 09:03:57 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 09:03:57 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 09:03:57 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 09:03:57 jayhawk kernel: NMI backtrace for cpu 1
Jan 8 09:03:57 jayhawk kernel: CPU 1
Jan 8 09:03:57 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: Pid: 0, comm: kworker/0:0 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:57 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:03:57 jayhawk kernel: RSP: 0018:ffff8804054cbeb8 EFLAGS: 00000246
Jan 8 09:03:57 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054cbfd8 RCX: 00000000ffffffff
Jan 8 09:03:57 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:03:57 jayhawk kernel: RBP: ffff8804054cbeb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: FS: 00002b78693086f0(0000) GS:ffff88041fc20000(0000) knlGS:000000005611b960
Jan 8 09:03:57 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:57 jayhawk kernel: CR2: 0000000000619000 CR3: 000000011e2fd000 CR4: 00000000000006e0
Jan 8 09:03:57 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:57 jayhawk kernel: Process kworker/0:0 (pid: 0, threadinfo ffff8804054ca000, task ffff8804054d0000)
Jan 8 09:03:57 jayhawk kernel: Stack:
Jan 8 09:03:57 jayhawk kernel: ffff8804054cbee8 ffffffff81054410 ffff8804054cbfd8 ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054cbf18 ffffffff8104aad6
Jan 8 09:03:57 jayhawk kernel: ffff8804054cbf08 03f4fe87f591794a 0000000000000001 0000000000000000
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: NMI backtrace for cpu 2
Jan 8 09:03:57 jayhawk kernel: CPU 2
Jan 8 09:03:57 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:57 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:03:57 jayhawk kernel: RSP: 0018:ffff8804054e3eb8 EFLAGS: 00000246
Jan 8 09:03:57 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054e3fd8 RCX: 00000000ffffffff
Jan 8 09:03:57 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:03:57 jayhawk kernel: RBP: ffff8804054e3eb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: FS: 00002b653981a6f0(0000) GS:ffff88041fc40000(0000) knlGS:0000000055849e70
Jan 8 09:03:57 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:57 jayhawk kernel: CR2: 00002b6539566850 CR3: 000000038454b000 CR4: 00000000000006e0
Jan 8 09:03:57 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:57 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff8804054e2000, task ffff8804054d5b40)
Jan 8 09:03:57 jayhawk kernel: Stack:
Jan 8 09:03:57 jayhawk kernel: ffff8804054e3ee8 ffffffff81054410 ffff8804054e3fd8 ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054e3f18 ffffffff8104aad6
Jan 8 09:03:57 jayhawk kernel: ffff8804054e3f08 0cf32962194b79f0 0000000000000002 0000000000000000
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: NMI backtrace for cpu 3
Jan 8 09:03:57 jayhawk kernel: CPU 3
Jan 8 09:03:57 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:57 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:03:57 jayhawk kernel: RSP: 0000:ffff8804054ffeb8 EFLAGS: 00000246
Jan 8 09:03:57 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054fffd8 RCX: 00000000ffffffff
Jan 8 09:03:57 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:03:57 jayhawk kernel: RBP: ffff8804054ffeb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: FS: 00002b78693086f0(0000) GS:ffff88041fc60000(0000) knlGS:000000005611b960
Jan 8 09:03:57 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:57 jayhawk kernel: CR2: 00000000012bc38c CR3: 00000001eaa8c000 CR4: 00000000000006e0
Jan 8 09:03:57 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:57 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff8804054fe000, task ffff8804054ec470)
Jan 8 09:03:57 jayhawk kernel: Stack:
Jan 8 09:03:57 jayhawk kernel: ffff8804054ffee8 ffffffff81054410 ffff8804054fffd8 ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054fff18 ffffffff8104aad6
Jan 8 09:03:57 jayhawk kernel: ffff8804054fff08 b7b1c45da28780f9 0000000000000003 0000000000000000
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: NMI backtrace for cpu 4
Jan 8 09:03:57 jayhawk kernel: CPU 4
Jan 8 09:03:57 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:57 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:03:57 jayhawk kernel: RSP: 0000:ffff880405513eb8 EFLAGS: 00000246
Jan 8 09:03:57 jayhawk kernel: RAX: 0000000000000000 RBX: ffff880405513fd8 RCX: 00000000ffffffff
Jan 8 09:03:57 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:03:57 jayhawk kernel: RBP: ffff880405513eb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: FS: 00007f266407c700(0000) GS:ffff88041fc80000(0000) knlGS:00000000f75c98d0
Jan 8 09:03:57 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:57 jayhawk kernel: CR2: 00000000f7686c6a CR3: 0000000001a05000 CR4: 00000000000006e0
Jan 8 09:03:57 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:57 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff880405512000, task ffff88040550ada0)
Jan 8 09:03:57 jayhawk kernel: Stack:
Jan 8 09:03:57 jayhawk kernel: ffff880405513ee8 ffffffff81054410 ffff880405513fd8 ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: 0000000000000000 0000000000000000 ffff880405513f18 ffffffff8104aad6
Jan 8 09:03:57 jayhawk kernel: ffff880405513f08 2ebceaa58ffe4b79 0000000000000004 0000000000000000
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: NMI backtrace for cpu 5
Jan 8 09:03:57 jayhawk kernel: CPU 5
Jan 8 09:03:57 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:03:57 jayhawk kernel:
Jan 8 09:03:57 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:03:57 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:03:57 jayhawk kernel: RSP: 0018:ffff88040552beb8 EFLAGS: 00000246
Jan 8 09:03:57 jayhawk kernel: RAX: 0000000000000000 RBX: ffff88040552bfd8 RCX: 00000000ffffffff
Jan 8 09:03:57 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:03:57 jayhawk kernel: RBP: ffff88040552beb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: FS: 00007f10feb20700(0000) GS:ffff88041fca0000(0000) knlGS:00000000f76516c0
Jan 8 09:03:57 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:03:57 jayhawk kernel: CR2: 0000000009033208 CR3: 0000000400c60000 CR4: 00000000000006e0
Jan 8 09:03:57 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:03:57 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:03:57 jayhawk kernel: Process kworker/0:1 (pid: 0, threadinfo ffff88040552a000, task ffff8804055216d0)
Jan 8 09:03:57 jayhawk kernel: Stack:
Jan 8 09:03:57 jayhawk kernel: ffff88040552bee8 ffffffff81054410 ffff88040552bfd8 ffffffff81aaae80
Jan 8 09:03:57 jayhawk kernel: 0000000000000000 0000000000000000 ffff88040552bf18 ffffffff8104aad6
Jan 8 09:03:57 jayhawk kernel: ffff88040552bf08 5e3fcc1570a6a5d4 0000000000000005 0000000000000000
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:03:57 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:03:57 jayhawk kernel: Call Trace:
Jan 8 09:03:57 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:03:57 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:03:57 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
```
And then again:
```
Jan 8 09:11:54 jayhawk kernel: INFO: rcu_sched_state detected stall on CPU 0 (t=6000 jiffies)
Jan 8 09:11:54 jayhawk kernel: sending NMI to all CPUs:
Jan 8 09:11:54 jayhawk kernel: NMI backtrace for cpu 0
Jan 8 09:11:54 jayhawk kernel: CPU 0
Jan 8 09:11:54 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:11:54 jayhawk kernel:
Jan 8 09:11:54 jayhawk kernel: Pid: 31058, comm: zfs_iput_taskq/ Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:11:54 jayhawk kernel: RIP: 0010:[] [] find_next_bit+0x1e/0xc0
Jan 8 09:11:54 jayhawk kernel: RSP: 0018:ffff88041fc03d68 EFLAGS: 00000046
Jan 8 09:11:54 jayhawk kernel: RAX: 000000000000000c RBX: 0000000000000002 RCX: 0000000000000000
Jan 8 09:11:54 jayhawk kernel: RDX: 0000000000000002 RSI: 000000000000000c RDI: ffffffff81aaae80
Jan 8 09:11:55 jayhawk kernel: RBP: ffff88041fc03d68 R08: ffffffff81aaae80 R09: 0000000000000000
Jan 8 09:11:55 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000d442
Jan 8 09:11:55 jayhawk kernel: R13: ffffffff81aaae80 R14: 0000000000000400 R15: 0000000000000096
Jan 8 09:11:55 jayhawk kernel: FS: 00007ffe3b8257c0(0000) GS:ffff88041fc00000(0000) knlGS:00000000f76798d0
Jan 8 09:11:55 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:11:55 jayhawk kernel: CR2: 00002ae456fd48f0 CR3: 0000000001a05000 CR4: 00000000000006f0
Jan 8 09:11:55 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:11:55 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:11:55 jayhawk kernel: Process zfs_iput_taskq/ (pid: 31058, threadinfo ffff8803d5876000, task ffff8802fcc7c470)
Jan 8 09:11:55 jayhawk kernel: Stack:
Jan 8 09:11:55 jayhawk kernel: ffff88041fc03db8 ffffffff81067dc2 ffff880400000001 0000000000000001
Jan 8 09:11:55 jayhawk kernel: ffff88041fc03dd8 0000000000002710 ffffffff81a29280 ffffffff81a29280
Jan 8 09:11:55 jayhawk kernel: ffff88041fc0df40 7fffffffffffffff ffff88041fc03dc8 ffffffff8106bd52
Jan 8 09:11:55 jayhawk kernel: Call Trace:
Jan 8 09:11:55 jayhawk kernel:
Jan 8 09:11:55 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 09:11:55 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 09:11:55 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 09:11:55 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 09:11:55 jayhawk kernel: [] ? kfree+0x36/0x130
Jan 8 09:11:55 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 09:11:55 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 09:11:55 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 09:11:55 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 09:11:55 jayhawk kernel: [] ? __do_softirq+0xf4/0x200
Jan 8 09:11:55 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 09:11:55 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 09:11:55 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 09:11:55 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 09:11:55 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 09:11:55 jayhawk kernel:
Jan 8 09:11:55 jayhawk kernel: [] ? kmem_free_debug+0xc/0x20 [spl]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_hold_impl+0x93/0xb0 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_prefetch+0xe2/0x280 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? dmu_zfetch+0x8fc/0xd40 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 09:11:55 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:11:55 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:11:55 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 09:11:55 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 09:11:55 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 09:11:55 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 09:11:55 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 09:11:55 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 09:11:55 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 09:11:55 jayhawk kernel: Code: e2 06 48 0f bd f6 48 8d 04 32 5d c3 90 55 48 39 f2 48 89 f0 48 89 e5 73 79 48 89 d1 49 89 d1 48 c1 e9 06 49 83 e1 c0 4c 8d 04 cf
Jan 8 09:11:55 jayhawk kernel: <48> 89 f7 4c 29 cf 83 e2 3f 75 5f 48 f7 c7 c0 ff ff ff 74 2b 49
Jan 8 09:11:55 jayhawk kernel: Call Trace:
Jan 8 09:11:55 jayhawk kernel: [] default_send_IPI_mask_sequence_phys+0x52/0xe0
Jan 8 09:11:55 jayhawk kernel: [] physflat_send_IPI_all+0x12/0x20
Jan 8 09:11:55 jayhawk kernel: [] arch_trigger_all_cpu_backtrace+0x55/0x80
Jan 8 09:11:55 jayhawk kernel: [] __rcu_pending+0x1b6/0x380
Jan 8 09:11:55 jayhawk kernel: [] ? kfree+0x36/0x130
Jan 8 09:11:55 jayhawk kernel: [] rcu_check_callbacks+0x65/0x120
Jan 8 09:11:55 jayhawk kernel: [] update_process_times+0x43/0x80
Jan 8 09:11:55 jayhawk kernel: [] tick_sched_timer+0x5f/0xb0
Jan 8 09:11:55 jayhawk kernel: [] __run_hrtimer+0x63/0x1b0
Jan 8 09:11:55 jayhawk kernel: [] ? __do_softirq+0xf4/0x200
Jan 8 09:11:55 jayhawk kernel: [] ? tick_nohz_handler+0xe0/0xe0
Jan 8 09:11:55 jayhawk kernel: [] ? read_tsc+0x9/0x20
Jan 8 09:11:55 jayhawk kernel: [] hrtimer_interrupt+0xe3/0x1f0
Jan 8 09:11:55 jayhawk kernel: [] smp_apic_timer_interrupt+0x64/0xa0
Jan 8 09:11:55 jayhawk kernel: [] apic_timer_interrupt+0x6e/0x80
Jan 8 09:11:55 jayhawk kernel: [] ? kmem_free_debug+0xc/0x20 [spl]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_hold_impl+0x93/0xb0 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_prefetch+0xe2/0x280 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? dmu_zfetch+0x8fc/0xd40 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_zfetch+0x6a9/0xd40 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dbuf_read+0x5e1/0x740 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? kmem_free_debug+0x11/0x20 [spl]
Jan 8 09:11:55 jayhawk kernel: [] dnode_hold_impl+0x13b/0x550 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:11:55 jayhawk kernel: [] dnode_hold+0x14/0x20 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] dmu_object_info+0x23/0x60 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] zfs_unlinked_drain+0x7d/0x120 [zfs]
Jan 8 09:11:55 jayhawk kernel: [] ? default_spin_lock_flags+0x9/0x10
Jan 8 09:11:55 jayhawk kernel: [] taskq_thread+0x1e2/0x410 [spl]
Jan 8 09:11:55 jayhawk kernel: [] ? try_to_wake_up+0x290/0x290
Jan 8 09:11:55 jayhawk kernel: [] ? task_alloc+0x160/0x160 [spl]
Jan 8 09:11:55 jayhawk kernel: [] kthread+0x87/0x90
Jan 8 09:11:55 jayhawk kernel: [] kernel_thread_helper+0x4/0x10
Jan 8 09:11:55 jayhawk kernel: [] ? flush_kthread_worker+0xa0/0xa0
Jan 8 09:11:55 jayhawk kernel: [] ? gs_change+0x13/0x13
Jan 8 09:11:55 jayhawk kernel: NMI backtrace for cpu 1
Jan 8 09:11:55 jayhawk kernel: CPU 1
Jan 8 09:11:55 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:11:55 jayhawk kernel:
Jan 8 09:11:55 jayhawk kernel: Pid: 0, comm: kworker/0:0 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:11:55 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:11:55 jayhawk kernel: RSP: 0000:ffff8804054cbeb8 EFLAGS: 00000246
Jan 8 09:11:55 jayhawk kernel: RAX: 0000000000000000 RBX: ffff8804054cbfd8 RCX: 00000000ffffffff
Jan 8 09:11:55 jayhawk kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffff81b57fa8
Jan 8 09:11:55 jayhawk kernel: RBP: ffff8804054cbeb8 R08: 0000000000000000 R09: 0000000000000000
Jan 8 09:11:55 jayhawk kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81aaae80
Jan 8 09:11:55 jayhawk kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Jan 8 09:11:55 jayhawk kernel: FS: 00002ad32c5ea6f0(0000) GS:ffff88041fc20000(0000) knlGS:000000005b8fab70
Jan 8 09:11:55 jayhawk kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 8 09:11:55 jayhawk kernel: CR2: 0000000055576000 CR3: 0000000316c7a000 CR4: 00000000000006e0
Jan 8 09:11:55 jayhawk kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 8 09:11:55 jayhawk kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 8 09:11:55 jayhawk kernel: Process kworker/0:0 (pid: 0, threadinfo ffff8804054ca000, task ffff8804054d0000)
Jan 8 09:11:55 jayhawk kernel: Stack:
Jan 8 09:11:55 jayhawk kernel: ffff8804054cbee8 ffffffff81054410 ffff8804054cbfd8 ffffffff81aaae80
Jan 8 09:11:55 jayhawk kernel: 0000000000000000 0000000000000000 ffff8804054cbf18 ffffffff8104aad6
Jan 8 09:11:55 jayhawk kernel: ffff8804054cbf08 03f4fe87f591794a 0000000000000001 0000000000000000
Jan 8 09:11:55 jayhawk kernel: Call Trace:
Jan 8 09:11:55 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:11:55 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:11:55 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:11:55 jayhawk kernel: Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Jan 8 09:11:55 jayhawk kernel: Call Trace:
Jan 8 09:11:55 jayhawk kernel: [] default_idle+0x40/0x100
Jan 8 09:11:55 jayhawk kernel: [] cpu_idle+0xc6/0x110
Jan 8 09:11:55 jayhawk kernel: [] start_secondary+0x1f7/0x1fe
Jan 8 09:11:55 jayhawk kernel: NMI backtrace for cpu 2
Jan 8 09:11:55 jayhawk kernel: INFO: rcu_sched_state detected stalls on CPUs/tasks: { 0} (detected by 1, t=6073 jiffies)
Jan 8 09:11:55 jayhawk kernel: CPU 2
Jan 8 09:11:55 jayhawk kernel: Modules linked in: zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl loop ipt_LOG xt_limit xt_tcpudp xt_state iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables nfsd lockd nfs_acl auth_rpcgss sunrpc dummy fuse zlib_deflate sp5100_tco i2c_piix4 k10temp amd64_edac_mod hwmon edac_core i2c_core button ata_generic pata_acpi mpt2sas pata_atiixp scsi_transport_sas raid_class e1000e [last unloaded: spl]
Jan 8 09:11:55 jayhawk kernel:
Jan 8 09:11:55 jayhawk kernel: Pid: 0, comm: kworker/0:1 Tainted: P 3.1.6-vs2.3.2.5-newjayhawk #4 Supermicro H8DCL/H8DCL
Jan 8 09:11:55 jayhawk kernel: RIP: 0010:[] [] native_safe_halt+0x6/0x10
Jan 8 09:11:55 jayhawk kernel: RSP: 0000:ffff8804054e3eb8 EFL