Fix --enable-debug-kmem-tracking option

Re-order initialization in spl_kmem_init to allow for kmem tracing
to work.  The spl_kmem_init function calls taskq_create prior to
initializing the tracking (calling spl_kmem_init_tracking).  Since
taskq_create uses kmem_alloc, NULL dereferences occur because the
global kmem_list hasn't had its next & prev pointers initialized yet.

This commit moves the calls to spl_kmem_init_tracking earlier in the
spl_kmem_init function in order that the subsequent kmem_alloc calls
(by taskq_create) work properly.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #243
Add ASSERT0 and VERIFY0 macros

The Illumos code introduced the ASSERT0 and VERIFY0 macros which
are to be used instead of ASSERT3S(x, ==, 0) and VERIFY3S(x, ==, 0).

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Madhav Suresh <madhav.suresh@delphix.com>
Closes #246
Simplify hostid logic

There is plenty of compatibility code for a hw_hostid
that isn't used by anything. At the same time, there are apparently
issues with the current hostid logic. coredumb in #zfsonlinux on
freenode reported that Fedora 17 changes its hostid on every boot, which
required force importing his pool. A suggestion by wca was to adopt
FreeBSD's behavior, where it treats hostid as zero if /etc/hostid does
not exist

Adopting FreeBSD's behavior permits us to eliminate plenty of code,
including a userland helper that invokes the system's hostid as a
fallback.

Signed-off-by: Richard Yao <ryao@cs.stonybrook.edu>
Return success from spl_cache_grow when spl_cache_grow_work finishes quickly

Consumers of the SPL invoke spl_kmem_cache_alloc to allocate memory from
caches. When a cache is out of memory, spl_kmem_cache_alloc() will
invoke spl_cache_refill(), which will invoke spl_cache_grow().

spl_cache_grow() will give the job of doing allocations to a task queue.
It will then call wait_event_timeout() to wait for completion. If
completion is too slow, it will time out. This gives us the option of
either telling spl_cache_refill() that we succeeded and there should be
memory or we did not succeed. If we do not succeed, spl_cache_refill() it will report that to
spl_cache_refill() which will try again. The return value for
spl_cache_grow(), which means that this retry will always happen. This
wastes CPU time unnecessarily, so we instead return ENOMEM when we
timeout and 0 otherwise.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Do emergency allocation instead of spinning.

A concept of an emergency allocation was introduced last year to permit
allocations that took excessive periods of time to switch to physical
memory to make forward progress. Unfortunately, emergency allocations
were implemented such that kmem_cache_alloc() would be given ENOMEM and
try again, which would be a waste of CPU time. We restructure the code
to do the emergency allocation immediately to correct that.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Honor __GFP_NORETRY in spl_kmem_cache_alloc()

The Open Solaris man page states that KM_NOSLEEP will "Return NULL
immediately if memory is not available". However, we currently do not
honor this behavior. We map KM_NOSLEEP to a Linux equivalent that
contains __GFP_NORETRY, which is meant to prevent this behavior. We
current this by checking for __GFP_NORETRY in spl_kmem_cache_alloc() and
spl_cache_refill() If it is set, exit the busy loop and return with
whatever we have done.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Do not block on IO or book-keeping when attempting to grow caches

Growing cache can block on swap operations, which will force all
allocations to be emergency allocations until the swap operations have
finished. Each emergency allocation will wait a maximum of 1 second,
which means that we will spend a fairly substantial amount of time
waiting when growing blocks on swap operations and swap happens to be a
zvol. We disallow ___GFP_IO to prevent this.

Additionally, __GFP_WAIT appears to tell the kernel that is okay to do
book keeping. That could potentially make swap operations that depend on
spl_kmem_cache_alloc() to wait longer, so we disallow that as well.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Use Linux SLAB allocator for spl_kmem_alloc_t objects

The Linux kernel's virtual memory subsystem has a single global lock.
Contention for this lock suffers from fairness issues where allocations
that depend on it could be starved for an indefinite period of time.
The SPL SLAB allocator is implemented on top of the Linux kernel's
virtual memory allocator. It attempts to mitigate such problems by doing
allocations in bulk.  However, other things will use the same interfaces
for virtual memory allocations, which can be a source of contention.

When an attempt at an allocation is done and a cache is full, the SPL
SLAB allocator will allocate a temporary spl_kmem_alloc_t object before
doing any actual allocations. This increases contention on the global
lock and slows down allocations. This also adds a potential source of
internal fragmentation in the virtual memory address space immediately
prior to an allocation, which is problematic on 32-bit systems.

We switch to the kernel's physical memory slab allocator. The objects
themselves have the property that they are small and are used
infrequently, which should permit the kernel's slab allocator to satisfy
all allocations from a single page. This provides us with O(1)
allocations, reduces contention and also eliminates a source of internal
fragmentation of the virtual memory address space.

Signed-off-by: Richard Yao <ryao@gentoo.org>
kmem_alloc/vmem_alloc: Yield CPU before retrying memory allocations

Solaris and Linux both have kswapd for paging out memory. However, the
Linux kernel also has a concept of direct reclaim, where an allocation
that is low on memory will attempt to free memory. Unfortunately, we can
have hundreds of threads doing this when the system is under heavy load.
This can cause discernible lags and will leave the system in a state
where it appears to be deadlocked.

We set __GFP_NORETRY to prevent the kernel's own routines from retrying
allocations repeatedly. We then try the allocation, yield the CPU if it
fails and then try again. This should allow us to perform more
gracefully when hundreds of threads block on memory allocations during
heavy system load.

Signed-off-by: Richard Yao <ryao@gentoo.org>