Do not block on IO or book-keeping when attempting to grow caches

Growing cache can block on swap operations, which will force all
allocations to be emergency allocations until the swap operations have
finished. Each emergency allocation will wait a maximum of 1 second,
which means that we will spend a fairly substantial amount of time
waiting when growing blocks on swap operations and swap happens to be a
zvol. We disallow ___GFP_IO to prevent this.

Additionally, __GFP_WAIT appears to tell the kernel that is okay to do
book keeping. That could potentially make swap operations that depend on
spl_kmem_cache_alloc() to wait longer, so we disallow that as well.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Use Linux SLAB allocator for spl_kmem_alloc_t objects

The Linux kernel's virtual memory subsystem has a single global lock.
Contention for this lock suffers from fairness issues where allocations
that depend on it could be starved for an indefinite period of time.
The SPL SLAB allocator is implemented on top of the Linux kernel's
virtual memory allocator. It attempts to mitigate such problems by doing
allocations in bulk.  However, other things will use the same interfaces
for virtual memory allocations, which can be a source of contention.

When an attempt at an allocation is done and a cache is full, the SPL
SLAB allocator will allocate a temporary spl_kmem_alloc_t object before
doing any actual allocations. This increases contention on the global
lock and slows down allocations. This also adds a potential source of
internal fragmentation in the virtual memory address space immediately
prior to an allocation, which is problematic on 32-bit systems.

We switch to the kernel's physical memory slab allocator. The objects
themselves have the property that they are small and are used
infrequently, which should permit the kernel's slab allocator to satisfy
all allocations from a single page. This provides us with O(1)
allocations, reduces contention and also eliminates a source of internal
fragmentation of the virtual memory address space.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Illumos #3006

3006 VERIFY[S,U,P] and ASSERT[S,U,P] frequently check if first argument is zero
Reviewed by Matt Ahrens <matthew.ahrens@delphix.com>
Reviewed by George Wilson <george.wilson@delphix.com>
Approved by Eric Schrock <eric.schrock@delphix.com>

References:
  illumos/illumos-gate@fb09f5aad449c97fe309678f3f604982b563a96f
  https://www.illumos.org/issues/3006

This is a fairly cosmetic change for ZoL, however, it will help in merging
subsequent upstream commits.

For ZoL, this issue requres a commit to both the spl and the zfs
repositories.

Ported-by: Tim Chase <tim@chase2k.com>