zfsonlinux/spl — Issue #81:  BUG: soft lockup - CPU stuck for 67s
Author: dipeit
Created: 2012-01-25 05:51:52+00:00
State: open

This happens when I copy large files via nfs to this system. (Kernel 2.6.37, zfs 0.6.0-rc6). Copy happens with ca 60-70 MB/s

Anyone knows what could be the issue

on console:

Message from syslogd@orca1 at Jan 24 18:22:51 ...
kernel:[1066884.745025] Code: d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 07 f3 90 0f b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28

syslog:

an 24 17:04:01 orca1 smartd[3718]: Device: /dev/sdc [SAT], Read Summary SMART Error Log failed
Jan 24 17:04:02 orca1 smartd[3718]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Jan 24 17:04:12 orca1 smartd[3718]: Device: /dev/sdl [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Jan 24 17:10:01 orca1 /USR/SBIN/CRON[31147]: (root) CMD (/usr/local/sbin/syslog_check.sh >>/var/tmp/syslog_check 2>&1)
Jan 24 17:18:54 orca1 kernel: [1062972.284011] BUG: soft lockup - CPU#2 stuck for 67s! [z_wr_iss/2:5852]
Jan 24 17:18:54 orca1 kernel: [1062972.284015] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.284067] CPU 2
Jan 24 17:18:54 orca1 kernel: [1062972.284068] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.284095]
Jan 24 17:18:54 orca1 kernel: [1062972.284098] Pid: 5852, comm: z_wr_iss/2 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062972.284102] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062972.284111] RSP: 0018:ffff880f917e9968 EFLAGS: 00000283
Jan 24 17:18:54 orca1 kernel: [1062972.284112] RAX: 0000000000003a62 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062972.284114] RDX: 0000000000003a5d RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062972.284115] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062972.284117] R10: ffffc90350c1e020 R11: 0000000000000023 R12: 0000000200000041
Jan 24 17:18:54 orca1 kernel: [1062972.284119] R13: ffff880ffffece00 R14: 00000000000612d0 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062972.284120] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284122] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062972.284124] CR2: 00007ff925765000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062972.284126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062972.284129] Process z_wr_iss/2 (pid: 5852, threadinfo ffff880f917e8000, task ffff880f91bea140)
Jan 24 17:18:54 orca1 kernel: [1062972.284131] Stack:
Jan 24 17:18:54 orca1 kernel: [1062972.284140] ffffffff81122ba4 ffff880f91bea140 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062972.284144] ffffc90000005000 ffffc90000004fff ffff880273a462a0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062972.284147] ffff880760b748c0 0000000000000000 ffff880760b748c0 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284150] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.284165] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.284170] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.284173] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.284187] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284204] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284263] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284390] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284508] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284595] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284714] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284799] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284902] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284912] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.284917] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.284920] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062972.284945] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.284947] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.284951] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.284954] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.284959] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284969] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284993] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285108] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285225] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285311] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285428] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285511] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285611] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.285619] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.285622] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.652010] BUG: soft lockup - CPU#6 stuck for 67s! [z_wr_iss/6:5856]
Jan 24 17:18:54 orca1 kernel: [1062972.652014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.652066] CPU 6
Jan 24 17:18:54 orca1 kernel: [1062972.652067] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.652094]
Jan 24 17:18:54 orca1 kernel: [1062972.652096] Pid: 5856, comm: z_wr_iss/6 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062972.652100] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062972.652109] RSP: 0018:ffff880f913b98f8 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1062972.652110] RAX: 0000000000003a66 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062972.652112] RDX: 0000000000003a64 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062972.652114] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062972.652115] R10: ffffc90350c36020 R11: 0000000000000031 R12: 0000000200000000
Jan 24 17:18:54 orca1 kernel: [1062972.652117] R13: ffff880ffffece00 R14: 0000000000000002 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062972.652118] FS: 0000000000000000(0000) GS:ffff8800cfd80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062972.652122] CR2: 00007ff925766000 CR3: 00000007662f8000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062972.652124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062972.652127] Process z_wr_iss/6 (pid: 5856, threadinfo ffff880f913b8000, task ffff880f815ee340)
Jan 24 17:18:54 orca1 kernel: [1062972.652129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062972.652137] ffffffff81122ba4 ffff880ffffece68 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062972.652141] ffffc90000005000 ffffc90000004fff ffff8802e2e1a6c0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062972.652144] ffff880274018940 0000000000000000 ffff880274018940 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652147] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.652162] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.652167] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.652170] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.652184] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.652200] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.652257] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652382] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652501] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652587] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652705] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652789] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652908] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653011] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653021] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.653027] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.653029] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062972.653054] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.653057] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.653059] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.653062] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.653068] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653077] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653101] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653216] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653332] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653417] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653534] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653617] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653734] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653834] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653842] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.653844] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.100012] BUG: soft lockup - CPU#0 stuck for 67s! [z_wr_iss/0:5850]
Jan 24 17:18:54 orca1 kernel: [1062984.100016] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.100066] CPU 0
Jan 24 17:18:54 orca1 kernel: [1062984.100066] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.100094]
Jan 24 17:18:54 orca1 kernel: [1062984.100096] Pid: 5850, comm: z_wr_iss/0 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.100100] RIP: 0010:[] [] _raw_spin_lock+0x18/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.100109] RSP: 0018:ffff880fb4d978f8 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1062984.100110] RAX: 0000000000003b44 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.100112] RDX: 0000000000003b42 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.100113] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.100115] R10: ffffc90355184020 R11: 0000000000000061 R12: 000000027daba0c0
Jan 24 17:18:54 orca1 kernel: [1062984.100117] R13: ffff880ffffece00 R14: ffffe8ffffffffff R15: ffffffffb34f0f58
Jan 24 17:18:54 orca1 kernel: [1062984.100118] FS: 0000000000000000(0000) GS:ffff8800cfc00000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.100122] CR2: 00007ff92576a000 CR3: 0000000001a03000 CR4: 00000000000006f0
Jan 24 17:18:54 orca1 kernel: [1062984.100123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.100127] Process z_wr_iss/0 (pid: 5850, threadinfo ffff880fb4d96000, task ffff880f7daba0c0)
Jan 24 17:18:54 orca1 kernel: [1062984.100129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.100137] ffffffff81122ba4 0000000000000246 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.100141] ffffc90000005000 ffffc90000004fff ffff880b257dd620 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.100143] ffff8805f1e92b40 0000000000000000 ffff8805f1e92b40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100146] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.100160] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.100165] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.100169] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.100182] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.100198] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.100253] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100377] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100495] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100581] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100700] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100783] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100903] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101006] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101016] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.101021] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.101024] Code: 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c 89 64 24
Jan 24 17:18:54 orca1 kernel: [1062984.101048] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.101051] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.101054] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.101057] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.101063] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101072] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101096] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101210] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101327] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101412] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101529] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101612] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101729] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101829] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101836] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.101839] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.192012] BUG: soft lockup - CPU#1 stuck for 67s! [z_wr_iss/1:5851]
Jan 24 17:18:54 orca1 kernel: [1062984.192016] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.192067] CPU 1
Jan 24 17:18:54 orca1 kernel: [1062984.192068] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.192095]
Jan 24 17:18:54 orca1 kernel: [1062984.192098] Pid: 5851, comm: z_wr_iss/1 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.192101] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.192110] RSP: 0018:ffff880fb33ab8f8 EFLAGS: 00000297
Jan 24 17:18:54 orca1 kernel: [1062984.192112] RAX: 0000000000003b49 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.192113] RDX: 0000000000003b43 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.192115] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.192116] R10: ffff880fff805160 R11: ffff880a1301d030 R12: 0000000200000010
Jan 24 17:18:54 orca1 kernel: [1062984.192118] R13: ffff880ffffece00 R14: 000000000000001f R15: ffffffff810fa855
Jan 24 17:18:54 orca1 kernel: [1062984.192120] FS: 0000000000000000(0000) GS:ffff8800cfc40000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.192122] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.192123] CR2: 00007ffdb4824000 CR3: 000000046a01a000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.192125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.192127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.192129] Process z_wr_iss/1 (pid: 5851, threadinfo ffff880fb33aa000, task ffff880f815ec100)
Jan 24 17:18:54 orca1 kernel: [1062984.192130] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.192139] ffffffff81122ba4 ffff880ffffece68 ffffffff81138231 000000000000003c
Jan 24 17:18:54 orca1 kernel: [1062984.192142] ffffc90000005000 ffffc90000004fff 0000000000000000 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.192145] ffff880b0a201e40 0000000000000000 ffff880b0a201e40 0000000000000001
Jan 24 17:18:54 orca1 kernel: [1062984.192148] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.192163] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.192168] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.192172] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.192185] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.192201] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.192257] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192383] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192501] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192587] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192706] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192790] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192909] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193011] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193022] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.193027] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.193030] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.193054] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.193056] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.193059] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.193062] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.193068] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193077] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193101] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193215] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193332] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193417] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193534] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193617] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193734] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193834] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193841] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.193844] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.376010] BUG: soft lockup - CPU#3 stuck for 67s! [z_wr_iss/3:5853]
Jan 24 17:18:54 orca1 kernel: [1062984.376012] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.376051] CPU 3
Jan 24 17:18:54 orca1 kernel: [1062984.376052] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.376079]
Jan 24 17:18:54 orca1 kernel: [1062984.376081] Pid: 5853, comm: z_wr_iss/3 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.376084] RIP: 0010:[] [] rb_next+0x27/0x50
Jan 24 17:18:54 orca1 kernel: [1062984.376092] RSP: 0018:ffff880fa7dcf9d8 EFLAGS: 00000286
Jan 24 17:18:54 orca1 kernel: [1062984.376093] RAX: ffff880c5739a5d8 RBX: ffffffffffffffff RCX: ffff8807842ce558
Jan 24 17:18:54 orca1 kernel: [1062984.376095] RDX: ffff8809b1968e58 RSI: 0000000000000000 RDI: ffff88015516a658
Jan 24 17:18:54 orca1 kernel: [1062984.376096] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.376098] R10: ffffc9035502e020 R11: 000000000000002c R12: ffff880f8f7f8180
Jan 24 17:18:54 orca1 kernel: [1062984.376099] R13: ffffc90355196000 R14: ffffe8ffffffffff R15: ffff8802e75c3958
Jan 24 17:18:54 orca1 kernel: [1062984.376101] FS: 0000000000000000(0000) GS:ffff8800cfcc0000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.376103] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.376104] CR2: 00007f6295d7b000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.376106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.376108] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.376110] Process z_wr_iss/3 (pid: 5853, threadinfo ffff880fa7dce000, task ffff880f8f7f8180)
Jan 24 17:18:54 orca1 kernel: [1062984.376111] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.376117] ffffffff81122c88 ffff880fa7dcfbd0 0000000000000001 0000000081a3ae64
Jan 24 17:18:54 orca1 kernel: [1062984.376121] ffffc90000009000 ffffc90000008fff ffffc90000000000 0000000000009000
Jan 24 17:18:54 orca1 kernel: [1062984.376124] ffff8802e75c3d40 0000000000000000 ffff8802e75c3d40 0000000000000212
Jan 24 17:18:54 orca1 kernel: [1062984.376126] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.376137] [] alloc_vmap_area+0x1b8/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.376141] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.376145] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.376155] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376167] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376204] [] vdev_raidz_io_start+0x2ad/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376293] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376411] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376494] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376614] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376715] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376724] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.376728] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.376730] Code: f3 c3 f3 c3 48 8b 17 31 c0 48 89 d1 48 83 e1 fc 48 39 cf 74 34 48 8b 47 08 48 85 c0 75 09 eb 1a 0f 1f 40 00 48 89 d0 48 8b 50 10 <48> 85 d2 75 f4 f3 c3 66 90 48 8b 10 48 89 c7 48 89 d0 48 83 e0
Jan 24 17:18:54 orca1 kernel: [1062984.376754] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.376756] [] alloc_vmap_area+0x1b8/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.376759] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.376762] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.376768] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376777] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376802] [] vdev_raidz_io_start+0x2ad/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376886] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377002] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377085] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377202] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377302] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.377309] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.377312] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.468011] BUG: soft lockup - CPU#4 stuck for 67s! [z_wr_iss/4:5854]
Jan 24 17:18:54 orca1 kernel: [1062984.468015] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.468064] CPU 4
Jan 24 17:18:54 orca1 kernel: [1062984.468065] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.468091]
Jan 24 17:18:54 orca1 kernel: [1062984.468093] Pid: 5854, comm: z_wr_iss/4 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.468097] RIP: 0010:[] [] _raw_spin_lock+0x18/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.468106] RSP: 0018:ffff880f947c9968 EFLAGS: 00000297
Jan 24 17:18:54 orca1 kernel: [1062984.468108] RAX: 0000000000003b4d RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.468109] RDX: 0000000000003b49 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.468111] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.468112] R10: ffffc903551ba020 R11: 000000000000007b R12: ffff880ffffece00
Jan 24 17:18:54 orca1 kernel: [1062984.468114] R13: ffffffff810fb3e6 R14: 0000000000001270 R15: ffff880ffffedc00
Jan 24 17:18:54 orca1 kernel: [1062984.468116] FS: 0000000000000000(0000) GS:ffff8800cfd00000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468117] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.468119] CR2: 00007ff925767000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.468121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.468124] Process z_wr_iss/4 (pid: 5854, threadinfo ffff880f947c8000, task ffff880f903c61c0)
Jan 24 17:18:54 orca1 kernel: [1062984.468126] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.468134] ffffffff81122ba4 ffff880f903c61c0 ffffffff551bdfff 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468138] ffffc90000005000 ffffc90000004fff ffff880ffffece00 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.468141] ffff880a54931bc0 0000000000000000 ffff880a54931bc0 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468144] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.468158] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.468163] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.468166] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.468180] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468196] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468249] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468372] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468490] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468576] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468694] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468778] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468880] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468890] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.468895] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.468897] Code: 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c 89 64 24
Jan 24 17:18:54 orca1 kernel: [1062984.468922] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.468924] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.468927] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.468930] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.468936] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468944] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468968] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469083] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469199] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469284] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469400] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469483] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469582] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.469590] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.469592] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.560010] BUG: soft lockup - CPU#5 stuck for 67s! [z_wr_iss/5:5855]
Jan 24 17:18:54 orca1 kernel: [1062984.560013] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.560066] CPU 5
Jan 24 17:18:54 orca1 kernel: [1062984.560066] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.560093]
Jan 24 17:18:54 orca1 kernel: [1062984.560096] Pid: 5855, comm: z_wr_iss/5 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.560100] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.560109] RSP: 0018:ffff880ffefb78f8 EFLAGS: 00000287
Jan 24 17:18:54 orca1 kernel: [1062984.560110] RAX: 0000000000003b50 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.560112] RDX: 0000000000003b4a RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.560113] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.560115] R10: ffffc903551d0020 R11: 000000000000002a R12: 0000000200000246
Jan 24 17:18:54 orca1 kernel: [1062984.560117] R13: ffff880ffffece00 R14: ffffffff8110db74 R15: fffffffffffece00
Jan 24 17:18:54 orca1 kernel: [1062984.560118] FS: 0000000000000000(0000) GS:ffff8800cfd40000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.560122] CR2: 00007ff925760000 CR3: 0000000f8c5cc000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.560124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.560127] Process z_wr_iss/5 (pid: 5855, threadinfo ffff880ffefb6000, task ffff880ffadf2600)
Jan 24 17:18:54 orca1 kernel: [1062984.560129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.560140] ffffffff81122ba4 ffff880ffadf2600 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.560144] ffffc90000005000 ffffc90000004fff ffff88047c197b80 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.560147] ffff88083b6832c0 0000000000000000 ffff88083b6832c0 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560150] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.560165] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.560170] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.560173] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.560187] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.560203] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.560259] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560384] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560502] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560589] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560707] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560791] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560911] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561013] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561024] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.561029] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.561032] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.561057] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.561059] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.561062] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.561065] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.561071] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561080] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561104] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561219] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561335] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561420] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561537] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561620] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561738] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561838] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561845] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.561848] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.744011] BUG: soft lockup - CPU#7 stuck for 67s! [z_wr_iss/7:5857]
Jan 24 17:18:54 orca1 kernel: [1062984.744014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.744062] CPU 7
Jan 24 17:18:54 orca1 kernel: [1062984.744063] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.744090]
Jan 24 17:18:54 orca1 kernel: [1062984.744092] Pid: 5857, comm: z_wr_iss/7 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.744096] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.744104] RSP: 0018:ffff880f901b9968 EFLAGS: 00000283
Jan 24 17:18:54 orca1 kernel: [1062984.744105] RAX: 0000000000003b53 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.744107] RDX: 0000000000003b4e RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.744109] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.744110] R10: ffffc903551e2020 R11: 000000000000006a R12: 0000000200000041
Jan 24 17:18:54 orca1 kernel: [1062984.744112] R13: ffff880ffffece00 R14: 00000000000612d0 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062984.744114] FS: 0000000000000000(0000) GS:ffff8800cfdc0000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744115] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.744117] CR2: 00007ff925768000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.744119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.744122] Process z_wr_iss/7 (pid: 5857, threadinfo ffff880f901b8000, task ffff880f89bb2200)
Jan 24 17:18:54 orca1 kernel: [1062984.744123] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.744131] ffffffff81122ba4 ffff880f89bb2200 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.744135] ffffc90000005000 ffffc90000004fff ffff8806a6e7e900 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.744138] ffff880ac9df4e40 0000000000000000 ffff880ac9df4e40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744141] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.744154] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.744160] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.744163] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.744175] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744191] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744242] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744362] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744480] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744566] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744684] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744768] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744870] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744879] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.744884] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.744887] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.744911] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.744914] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.744917] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.744919] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.744925] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744934] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744958] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745072] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745188] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745273] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745389] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745472] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745571] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.745579] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.745582] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1063056.284010] BUG: soft lockup - CPU#2 stuck for 67s! [z_wr_iss/2:5852]
Jan 24 17:18:54 orca1 kernel: [1063056.284014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1063056.284063] CPU 2
Jan 24 17:18:54 orca1 kernel: [1063056.284064] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1063056.284091]
Jan 24 17:18:54 orca1 kernel: [1063056.284093] Pid: 5852, comm: z_wr_iss/2 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1063056.284097] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1063056.284105] RSP: 0018:ffff880f917e9968 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1063056.284107] RAX: 00000000000040a2 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1063056.284109] RDX: 00000000000040a0 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1063056.284110] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1063056.284112] R10: ffffc90374dd0020 R11: 000000000000006f R12: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284113] R13: ffff880ffffece00 R14: 0000000000000002 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1063056.284115] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284117] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1063056.284119] CR2: 00007ff925765000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1063056.284120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1063056.284124] Process z_wr_iss/2 (pid: 5852, threadinfo ffff880f917e8000, task ffff880f91bea140)
Jan 24 17:18:54 orca1 kernel: [1063056.284125] Stack:
Jan 24 17:18:54 orca1 kernel: [1063056.284133] ffffffff81122ba4 0000000000001272 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1063056.284137] ffffc90000005000 ffffc90000004fff ffff880675f8fbc0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1063056.284140] ffff880ab32d6a40 0000000000000000 ffff880ab32d6a40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284143] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1063056.284158] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1063056.284162] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1063056.284166] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1063056.284179] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284195] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284249] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284374] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284492] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284578] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284696] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284780] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.284882] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284892] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1063056.284897] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1063056.284900] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1063056.284924] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1063056.284927] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1063056.284930] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1063056.284933] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1063056.284938] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284947] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284971] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285085] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285201] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285286] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285402] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285485] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1063056.285585] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.285592] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1063056.285595] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1063056.652012] BUG: soft lockup - CPU#6 stuck for 67s! [z_wr_iss/6:5856]
Jan 24 17:18:54 orca1 kernel: [1063056.652015] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1063056.652064] CPU 6
Jan 24 17:18:54 orca1 kernel: [1063056.652064] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma