Allocate spl_kmem_alloc_t objects from mempool

Allocating new slabs requires first allocating a spl_kmem_alloc_t
object before the actual allocation routine is done. This allocation can
fail when the system is memory constrained. This is particularly bad
because swap operations on ZFS can depend on the success of slab
allocation. We attempt to ensure that spl_kmem_alloc_t always succeed by
switching to the Linux slab allocator and pre-reserving memory via a
Linux mempool. This should prevent spl_kmem_alloc_t allocations from
failing.

Signed-off-by: Richard Yao <ryao@gentoo.org>
kmem_alloc/vmem_alloc: Yield CPU before retrying memory allocations

Solaris and Linux both have kswapd for paging out memory. However, the
Linux kernel also has a concept of direct reclaim, where an allocation
that is low on memory will attempt to free memory. Unfortunately, we can
have hundreds of threads doing this when the system is under heavy load.
This can cause discernible lags and will leave the system in a state
where it appears to be deadlocked.

We set __GFP_NORETRY to prevent the kernel's own routines from retrying
allocations repeatedly. We then try the allocation, yield the CPU if it
fails and then try again. This should allow us to perform more
gracefully when hundreds of threads block on memory allocations during
heavy system load.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Change spl_kmem_cache_expire default setting to 2

This makes us better emulate the Solaris behavior.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Remove unused flags variable from __cv_init()

GCC 4.8.1 complained about an unused flags variable when building
against Linux 2.6.26.8:

/var/tmp/portage/sys-kernel/spl-9999/work/spl-9999/module/spl/../../module/spl/spl-condvar.c:
In function ‘__cv_init’:
/var/tmp/portage/sys-kernel/spl-9999/work/spl-9999/module/spl/../../module/spl/spl-condvar.c:39:6:
error: variable ‘flags’ set but not used
[-Werror=unused-but-set-variable]
  int flags = KM_SLEEP;
        ^
	cc1: all warnings being treated as errors

Signed-off-by: Richard Yao <ryao@gentoo.org>
Implement kpreempt()

This is required by "25068 arc_evict_ghost() should not hold arcs_mtx
for a long time" from Delphix OS.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Replace current_kernel_time() with getnstimeofday()

current_kernel_time() is used by the SPLAT, but it is not meant for
performance measurement. We modify the SPLAT to use getnstimeofday(),
which is equivalent to the gethrestime() function on Solaris.
Additionally, we update gethrestime() to invoke getnstimeofday().

Signed-off-by: Richard Yao <ryao@gentoo.org>
Implement sgbufs

sgbufs are scatter-gather buffers that consist of pages that can
be manipulated directly or mapped itno the page tables on an as-needed
basis. This a new data structure that will be proposed for inclusion
into Illumos. Their main purpose is a replacement for slab allocation in
ZIO buffers.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Force emergency allocations to trigger under low memory conditions

A concept of an emergency allocation was introduced last year to permit
allocations that took excessive periods of time to switch to physical
memory to make forward progress. Emergency allocations currently occur
by setting a flag, telling kmem_cache_alloc() to try again and then
doing the allocation. This is useful, but it might not always trigger
because we do not permit allocations in the spl_cache_grow() to
retry. This permits allocation failures to fail before the timer for
emergency allocations can trigger. This will result in a return to
kmem_cache_alloc(), which will then repeat the process. The result is
that we can loop infinitely without any forward progress under low
memory conditions because the emergency allocations will not trigger.

We change the code to force the worker to retry indefinitely. This
guarentees that the emergency allocations will trigger under low memory
conditions. In addition, we remove the KMC_BIT_DEADLOCKED bit in favor
of immediate use of the emergency allocation.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Use pages for emergency allocations when possible and necessary

A concept of an emergency allocation was introduced last year to permit
allocations that took excessive periods of time to switch to physical
memory to make forward progress.

Unfortunately, the Linux kernel's physical memory allocator does SLAB
allocation for all allocations less than or equal to 8192. This permits memory
fragmentation to cause allocations to hang whenever an allocation is made from
a cache that uses slabs that span multiple pages. In specific, whenever the
allocation is greater than 256-bytes.

Whenever an emergency object is small enough to fit into a single page and is
larger than 256-bytes, we allocate a page for the emergency allocation instead
of going through kmalloc(). Larger object sizes are still at risk, but there is
little we can do for them.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Permit KM_PUSHPAGE to trigger OOM killer

KM_PUSHPAGE is used in situations where we need to dip into emergency
memory reserves to ensure forward progress in operations that can
involve pageout. If the reserves are depleted, then it is acceptable to
invoke the OOM killer, but it is not appropriate to invoke another swap
operation. Currently, KM_PUSHPAGE is the equivalent to ( __GFP_WAIT |
__GFP_HIGH). The OOM killer will only trigger when (gfp_mask & __GFP_FS)
&& !(gfp_mask & __GFP_NORETRY) is true. Therefore, we redefine
KM_PUSHPAGE to be (__GFP_WAIT | __GFP_HIGH | __GFP_FS). The absence of
___GFP_IO prevents swap operations from being triggered while the
presence of __GFP_FS permits the OOM killer to trigger.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Map kmem_alloc() invocations to kmem_cache_alloc()

Historically, the SPL has mapped kmem_alloc() to Linux's kmalloc().
Linux's kmalloc() was mapped to its kmem_cache_alloc(). This proved
problematic whenever we had a situation where Linux's kmem_cache_alloc()
had to allocate another slab for a few reasons:

1. The allocation routine could invoke direct reclaim, which would then
block on swap operations. If kmem_alloc() was called in ZFS' sync thread
or another context that could block swap operations, ZFS would deadlock.

2. Allocations that were greater than 256-bytes in size are made from
slabs that span multiple pages, which made us vulnerable to internal
memory fragmentation.

This patch maps kmem_alloc() allocations of 4096 bytes or less to the
SPL's kmem_cache_alloc(). This enables us to take advantage of the work
that was done to ensure that forward progress is maintained under memory
pressure in allocations from the kmem_cache_alloc().

Signed-off-by: Richard Yao <ryao@gentoo.org>