Perform KM_SLEEP|KM_PUSHPAGE allocations immediately

If a SLAB cache is full and two allocations occur from the same SLAB
cache nearly simultaneously where one is KM_SLEEP|KM_PUSHPAGE and another
is KM_NOSLEEP.  Then the one that occurs first will dictate the KM_FLAGS
used for SLAB growth for both of them. This is a race condition which
can hurt performance.

We address this by modifying `spl_cache_grow()` to only provide the
emergency allocation semantics to KM_NOSLEEP and SLABs backed by
vmalloc()'d memory.  KM_SLEEP|KM_PUSHPAGE allocations are coalesced
KM_NOSLEEP allocations will failing immediately.

The previous memory management changes in this patch stack largely make
the emergency allocation code unnessisary.  However, for the moment it
must be kept for kernels older then 3.9 which do not support the
PF_MEMALLOC_NOIO flag.
Make slab reclaim more aggressive

Many people have noticed that the kmem cache implementation is slow
to release its memory.  This patch makes the reclaim behavior more
aggressive by immediately freeing a slab once it is emptry.  Unused
objects which are cached in the magazines will still prevent a slab
from being freed.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Update spl-module-parameters(5) man page

The spl-module-parameters(5) was not kept up to date.  Refresh
the man page so that it lists all the possible module options,
describes what the do, and justify why the default values are
set they way the are.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Refine slab cache sizing

This change is designed to improve the memory utilization of
slabs by more carefully setting their size.  The way the code
currently works is problematic for slabs which contain large
objects (>1MB).  This is due to slabs being unconditionally
rounded up to a power of two which may result in unused space
at the end of the slab.

The reason the existing code rounds up every slab is because it
assumes it will backed by the buddy allocator.  Since the buddy
allocator can only performs power of two allocations this is
desirable because it avoids wasting any space.  However, this
logic breaks down if slab is backed by vmalloc() which operates
at a page level granularity.  In this case, the optimal thing to
do is calculate the minimum required slab size given certain
constraints (object size, alignment, objects/slab, etc).

Therefore, this patch reworks the spl_slab_size() function so
that it sizes KMC_KMEM slabs differently than KMC_VMEM slabs.
KMC_KMEM slabs are rounded up to the nearest power of two, and
KMC_VMEM slabs are allowed to be the minimum required size.

This change also reduces the default number of objects per slab.
This reduces how much memory a single cache object can pin, which
can result in significant memory saving for highly fragmented
caches.  But depending on the workload it may result in slabs
being allocated and freed more frequently.  In practice, this
has been shown to be a better default for most workloads.

Also the maximum slab size has been reduced to 4MB on 32-bit
systems.  Due to the limited virtual address space it's critical
the we be as frugal as possible.  A limit of 4M still let's us
reasonably comfortably allocate a limited number of 1MB objects.

Finally, the kmem:slab_small and kmem:slab_large SPLAT tests
were extended to provide better test coverage of various object
sizes and alignments.  Caches are created with random parameters
and their basic functionality is verified by allocating several
slabs worth of objects.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>