zfsonlinux/zfs — Issue #540: z_wr_iss/0: page allocation failure
Author: bziller
Created: 2012-01-20 06:01:29+00:00
State: open

Sorry, but after looking around, I still can't tell if this a follow up or a real new issue...

This occurred under RHEL 6.2, 2.6.32-220.2.1.el6.x86_64, 48GB RAM, 2x raidz2 with 6+2 2TB disks.
zfs/spl compiled from master zfsonlinux-zfs-b9c59ec and zfsonlinux-spl-e05bec8 (Jan 4th).

```
Jan 20 01:57:58 anor kernel: _thread+0x1d2/0x330 [spl]
Jan 20 01:57:58 anor kernel: [] ? default_wake_function+0x0/0x20
Jan 20 01:57:58 anor kernel: [] ? taskq_thread+0x0/0x330 [spl]
Jan 20 01:57:58 anor kernel: [] ? kthread+0x96/0xa0
Jan 20 01:57:58 anor kernel: [] ? child_rip+0xa/0x20
Jan 20 01:57:58 anor kernel: [] ? kthread+0x0/0xa0
Jan 20 01:57:58 anor kernel: [] ? child_rip+0x0/0x20
Jan 20 01:57:58 anor kernel: z_wr_iss/0: page allocation failure. order:0, mode:0x20
Jan 20 01:57:58 anor kernel: Pid: 1663, comm: z_wr_iss/0 Tainted: P W ---------------- 2.6.32-220.2.1.el6.x86_64 #1
Jan 20 01:57:58 anor kernel: Call Trace:
Jan 20 01:57:58 anor kernel: [] ? __alloc_pages_nodemask+0x77f/0x940
Jan 20 01:57:58 anor kernel: [] ? kmem_getpages+0x62/0x170
Jan 20 01:57:58 anor kernel: [] ? fallback_alloc+0x1ba/0x270
Jan 20 01:57:58 anor kernel: [] ? cache_grow+0x2cf/0x320
Jan 20 01:57:58 anor kernel: [] ? ____cache_alloc_node+0x99/0x160
Jan 20 01:57:58 anor kernel: [] ? __alloc_skb+0x7a/0x180
Jan 20 01:57:58 anor kernel: [] ? kmem_cache_alloc_node_notrace+0x6f/0x130
Jan 20 01:57:58 anor kernel: [] ? __kmalloc_node+0x7b/0x100
Jan 20 01:57:58 anor kernel: [] ? netif_receive_skb+0x58/0x60
Jan 20 01:57:58 anor kernel: [] ? __alloc_skb+0x7a/0x180
Jan 20 01:57:58 anor kernel: [] ? __netdev_alloc_skb+0x36/0x60
Jan 20 01:57:58 anor kernel: [] ? e1000_clean_rx_irq+0x392/0x530 [e1000e]
Jan 20 01:57:58 anor kernel: [] ? handle_irq+0x49/0xa0
Jan 20 01:57:58 anor kernel: [] ? e1000_clean+0xb0/0x2b0 [e1000e]
Jan 20 01:57:58 anor kernel: [] ? net_rx_action+0x103/0x2f0
Jan 20 01:57:58 anor kernel: [] ? __do_softirq+0xc1/0x1d0
Jan 20 01:57:58 anor kernel: [] ? handle_IRQ_event+0x60/0x170
Jan 20 01:57:58 anor kernel: [] ? __do_softirq+0x11a/0x1d0
Jan 20 01:57:58 anor kernel: [] ? call_softirq+0x1c/0x30
Jan 20 01:57:58 anor kernel: [] ? do_softirq+0x65/0xa0
Jan 20 01:57:58 anor kernel: [] ? irq_exit+0x85/0x90
Jan 20 01:57:58 anor kernel: [] ? do_IRQ+0x75/0xf0
Jan 20 01:57:58 anor kernel: [] ? ret_from_intr+0x0/0x11
Jan 20 01:57:58 anor kernel: [] ? rb_next+0x20/0x50
Jan 20 01:57:58 anor kernel: [] ? alloc_vmap_area+0x1c0/0x390
Jan 20 01:57:58 anor kernel: [] ? kmem_cache_alloc_node_notrace+0x104/0x130
Jan 20 01:57:58 anor kernel: [] ? __get_vm_area_node+0xf2/0x230
Jan 20 01:57:58 anor kernel: [] ? kv_alloc+0x7c/0xc0 [spl]
Jan 20 01:57:58 anor kernel: [] ? __vmalloc_node+0x89/0xb0
Jan 20 01:57:58 anor kernel: [] ? kv_alloc+0x7c/0xc0 [spl]
Jan 20 01:57:58 anor kernel: [] ? __vmalloc+0x22/0x30
Jan 20 01:57:58 anor kernel: [] ? kv_alloc+0x7c/0xc0 [spl]
Jan 20 01:57:58 anor kernel: [] ? spl_kmem_cache_alloc+0x339/0x650 [spl]
Jan 20 01:57:58 anor kernel: [] ? zio_buf_alloc+0x23/0x30 [zfs]
Jan 20 01:57:58 anor kernel: [] ? vdev_raidz_io_start+0x29e/0x6b0 [zfs]
Jan 20 01:57:58 anor kernel: [] ? zio_vdev_io_start+0xa7/0x2e0 [zfs]
Jan 20 01:57:58 anor kernel: [] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Jan 20 01:57:58 anor kernel: [] ? zio_nowait+0xa9/0x120 [zfs]
Jan 20 01:57:58 anor kernel: [] ? vdev_mirror_io_start+0x17e/0x3c0 [zfs]
Jan 20 01:57:58 anor kernel: [] ? vdev_mirror_child_done+0x0/0x30 [zfs]
Jan 20 01:57:58 anor kernel: [] ? zio_vdev_io_start+0x21f/0x2e0 [zfs]
Jan 20 01:57:58 anor kernel: [] ? zio_execute+0x99/0xf0 [zfs]
Jan 20 01:57:58 anor kernel: [] ? taskq_thread+0x1d2/0x330 [spl]
Jan 20 01:57:58 anor kernel: [] ? default_wake_function+0x0/0x20
Jan 20 01:57:58 anor kernel: [] ? taskq_thread+0x0/0x330 [spl]
Jan 20 01:57:58 anor kernel: [] ? kthread+0x96/0xa0
Jan 20 01:57:58 anor kernel: [] ? child_rip+0xa/0x20
Jan 20 01:57:58 anor kernel: [] ? kthread+0x0/0xa0
Jan 20 01:57:58 anor kernel: [] ? child_rip+0x0/0x20
```

This repeats several times followed later by

```
Jan 20 01:58:12 anor kernel: z_wr_iss/0: page allocation failure. order:0, mode:0x20
Jan 20 01:58:12 anor kernel: Pid: 1663, comm: z_wr_iss/0 Tainted: P W ---------------- 2.6.32-220.2.1.el6.x86_64 #1
Jan 20 01:58:12 anor kernel: Call Trace:
Jan 20 01:58:12 anor kernel: [] ? __alloc_pages_nodemask+0x77f/0x940
Jan 20 01:58:12 anor kernel: [] ? kmem_getpages+0x62/0x170
Jan 20 01:58:12 anor kernel: [] ? fallback_alloc+0x1ba/0x270
Jan 20 01:58:12 anor kernel: [] ? cache_grow+0x2cf/0x320
Jan 20 01:58:12 anor kernel: [] ? ____cache_alloc_node+0x99/0x160
Jan 20 01:58:12 anor kernel: [] ? kmem_cache_alloc+0x11b/0x190
Jan 20 01:58:12 anor kernel: [] ? dst_alloc+0x30/0x80
Jan 20 01:58:12 anor kernel: [] ? ip_route_input_slow+0x32c/0xb00
Jan 20 01:58:12 anor kernel: [] ? ip_route_input+0x66/0x5b0
Jan 20 01:58:12 anor kernel: [] ? __kfree_skb+0x47/0xa0
Jan 20 01:58:12 anor kernel: [] ? arp_process+0x3ab/0x730
Jan 20 01:58:12 anor kernel: [] ? ret_from_intr+0x0/0x11
Jan 20 01:58:12 anor kernel: [] ? kmem_cache_alloc_node_notrace+0x77/0x130
Jan 20 01:58:12 anor kernel: [] ? arp_rcv+0x111/0x140
Jan 20 01:58:12 anor kernel: [] ? __netif_receive_skb+0x49b/0x6e0
Jan 20 01:58:12 anor kernel: [] ? netif_receive_skb+0x58/0x60
Jan 20 01:58:12 anor kernel: [] ? napi_skb_finish+0x50/0x70
Jan 20 01:58:12 anor kernel: [] ? napi_gro_receive+0x39/0x50
Jan 20 01:58:12 anor kernel: [] ? e1000_receive_skb+0x5b/0x90 [e1000e]
Jan 20 01:58:12 anor kernel: [] ? e1000_clean_rx_irq+0x2b0/0x530 [e1000e]
Jan 20 01:58:12 anor kernel: [] ? e1000_clean+0xb0/0x2b0 [e1000e]
Jan 20 01:58:12 anor kernel: [] ? net_rx_action+0x103/0x2f0
Jan 20 01:58:12 anor kernel: [] ? __do_softirq+0xc1/0x1d0
Jan 20 01:58:12 anor kernel: [] ? handle_IRQ_event+0x60/0x170
Jan 20 01:58:12 anor kernel: [] ? __do_softirq+0x11a/0x1d0
Jan 20 01:58:12 anor kernel: [] ? call_softirq+0x1c/0x30
Jan 20 01:58:12 anor kernel: [] ? do_softirq+0x65/0xa0
Jan 20 01:58:12 anor kernel: [] ? irq_exit+0x85/0x90
Jan 20 01:58:12 anor kernel: [] ? do_IRQ+0x75/0xf0
```
and more...

The full log (826kB) can be found at http://wwwlehre.dhbw-stuttgart.de/~bziller/zfs-messages.txt