Rate limit debugging stack traces

There have been issues in the past where excessive debug logging
to the console has resulted in significant performance impacts.
In the vast majority of these cases only a few stack traces are
required to diagnose the issue.  Therefore, stack traces dumped to
the console will now we limited to 5 every 60s.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Closes #374
Add functions and macros as used upstream.

Added highbit64() and howmany() which are used in recent upstream
code.  Both highbit() and highbit64() should at some point be
re-factored to use the optimized fls() and fls64() functions.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Tim Chase <tim@chase2k.com>
Closes #363
Add atomic_swap_32() and atomic_swap_64()

The atomic_swap_32() function maps to atomic_xchg(), and
the atomic_swap_64() function maps to atomic64_xchg().

Signed-off-by: Tim Chase <tim@chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #377
Set spl_kmem_cache_reclaim=0 to default

Reinstate the correct default behavior of returning the number of objects
in the cache for reclaim.  This behavior was disabled in recent releases
to do occasional reports of spinning in shrink_slabs().  Those issues have
been resolved and can no longer can be reproduced.  See commit 376dc35.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: DHE <git@dehacked.net>
Issue #358
Closes #379
Set spl_kmem_cache_slab_limit=16384 to default

For small objects the Linux slab allocator should be used to make the most
efficient use of the memory.  However, large objects are not supported by
the Linux slab and therefore the SPL implementation is preferred.  A cutoff
of 16K was determined to be optimal for architectures using 4K pages.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: DHE <git@dehacked.net>
Issue #356
Closes #379
Linux 3.17 compat: remove wait_on_bit action function

Linux kernel 3.17 removes the action function argument from
wait_on_bit().  Add autoconf test and compatibility macro to support
the new interface.

The former "wait_on_bit" interface required an 'action' function to
be provided which does the actual waiting. There were over 20 such
functions in the kernel, many of them identical, though most cases
can be satisfied by one of just two functions: one which uses
io_schedule() and one which just uses schedule().  This API change
was made to consolidate all of those redundant wait functions.

References: torvalds/linux@7431620

Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #378
Cleanup vn_rename() and vn_remove()

zfsonlinux/spl#bcb15891ab394e11615eee08bba1fd85ac32e158 implemented
Linux 3.6+ support by adding duplicate vn_rename and vn_remove
functions. The new ones were cleaner, but the duplicate functions made
the codebase less maintainable. This adds some compatibility shims that
allow us to retire the older vn_rename and vn_remove in favor of the new
ones on old kernels. The result is a net 143 line reduction in lines of
code and a cleaner codebase.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #370
Avoid PAGESIZE redefinition

Add #ifndef PAGESIZE to avoid redefinition warning on platforms
where this value is already provided.

Signed-off-by: stf <s@ctrlc.hu>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #382
Linux 3.16 compat: smp_mb__after_clear_bit()

The smp_mb__{before,after}_clear_bit functions have been renamed
smp_mb__{before,after}_atomic.  Rather than adding a compatibility
function to handle this the code has been updated to use smp_wmb().

This has the advantage of being a stable functionally equivalent
interface.  On many architectures smp_mb__after_clear_bit() expands
to smp_wmb().  Others might be able to do something slightly more
efficient but this will be safe and correct on all of them.

Signed-off-by: Turbo Fredriksson <turbo@bayour.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #386
Fix bug in SPLAT taskq:front

While running SPLAT on a kernel with CONFIG_DEBUG_ATOMIC_SLEEP
enabled the taskq:front was flagged as a test which might sleep
which in an unsafe context.  Specifically, the splat_vprint()
function which internally takes a mutex was being called under
a spin lock.  Moving the log function outside the spin lock
cleanly solves this issue.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Use Linux kzalloc to allocate kmem_cache_t objects

wake_up_bit() is called on a word inside kmem_cache_t objects.  This
calls virt_to_page() on the address of the memory. That is incompatible
with virtual memory, so we must switch to Linux's memory allocator.

Signed-off-by: Richard Yao <ryao@gentoo.org>
spl-kmem should not use taskq interface

This is a prerequisite for the Illumos taskq import.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Revert "Add PF_NOFS debugging flag"

This reverts commit eb0f407a2b9089113ef6f2402ebd887511315b43.
Refactor generic memory allocation interfaces

This patch achieves the following goals:

1. It replaces the preprocessor kmem flag to gfp flag mapping with
proper translation logic. This eliminates the potential for surprises
that were previously possible where kmem flags were mapped to gfp flags.

2. It maps `kmem_{,z}alloc()` KM_SLEEP allocations
that were previously mapped to `kmalloc()` to `vmalloc()` to reduce
internal memory fragmentation.

3. It discards the distinction between vmem_* and kmem_* that was
previously made by mapping them to vmalloc() and kmalloc() respectively.
This achieve better compatibility because kmem_* allocates are done from
slabs allocated from vmem_*. Both are therefore virtual memory
allocators and it makes no sense to implement them differently than one
another.

The detailed reasons for each are as follows:

1. Solaris derivatives have different allocation flag semantics than Linux.
This was originally handled by trying to map Solaris flags to Linux
flags, but the semantics are different enough that this approach does
not correctly handle all cases. For example, 0 is KM_SLEEP on Solaris
derivatives while 0 is illegal on Linux. This means that things like
assertions that make assumptions about the flag semantics are no longer
portable because reasonable assertions such as `ASSERT0(flags)` on
Solaris derivatives are illegal on Linux. In addition, a trivial mapping
allows us to freely mix and match flags.  This is bad for portability
and it can lead to unexpected consequences when clashes between
semantics means that one expects one system's semantics, and receives
another.

3. The SPL originally mapped kmem_alloc() to kmalloc() and
vmem_alloc() to vmalloc(). One would be inclined to think this is
correct by applying the reasonable expectation that things with similar
names on each platform are similar things. However, this is not the case
here. On Solaris, vmem_* is a general purpose arena allocator that does
kernel virtual memory allocations. The Solaris SLAB allocator
`kmem_cache_alloc()` operates by allocating slabs from vmem and returns
objects. Allocations from kmem_alloc() work by performing HOARD-style
allocations on pre-existing power of 2 SLAB caches. When mapping uses of
these allocators to Linux equivalents, we must consider 4 allocators on
Linux and how they interact:

1. The buddy allocator
2. The slab allocator
3. The vmap allocator
4. The kernel virtual memory allocator

The buddy allocator is used for allocating both pages and slabs for
Linux's kmem_slab_alloc. These are then used to provide generic power of
2 caches to which kmalloc() is mapped. Allocations that are larger than
the largest power of 2 are sent directly to the buddy allocator.  This
is analogous to kmem_cache_alloc() and kmem_alloc() on Solaris. The
third allocator is the vmap allocator, which concerns itself with
allocating address space. The four allocator is the kernel virtual
memory allocator and is invoked via `vmalloc()`. This uses pages from
the buddy allocator and address space from the vmap allocator to perform
virtual memory allocations.

3. Switching the KM_SLEEP allocations to `vmalloc()` provides some
protection from deadlocks caused by internal memory fragmentation. It
would have been ideal to make all allocations virtual like they are on
Illumos. However, virtual memory allocations allocations that require
KM_PUSHPAGE or KM_NOSLEEP semantics will receive KM_SLEEP semantics on
Linux whenever a page directory table entry must be allocated, which is
unsafe. We are therefore forced to use physical memory for KM_PUSHPAGE
and KM_NOSLEEP allocations. That is suboptimal from the perspective of
reducing internal memory fragmentation, but we still partially benefit
by mapping KM_SLEEP allocations to `vmalloc()`.

A caveat that aries from replacing `kmalloc()` with `vmalloc()` is that
code using Linux's wake_up_bit should use the native Linux allocators.
This has no equivalent on Solaris.  While it might seem fragile at first
glance, that is not the case for three reasons:

1. Linux's locking structures use atomic instructions and churn in them
is rare. When churn does occur, there is an incredible amount of
pressure to maintain both the size of the structure and remain backward
compatible precisely because changes to locking structures can cause
unanticipated issues that are hard to debug.

2. The incompatibility arises because `wait_on_bit()` does a hashtable
lookup that assumes that `wait_on_bit()` is never called on virtual
memory. This hashtable lookup should be more expensive than conventional
locking because it involves more memory accesses than an atomic
instruction for taking a mutex, so it will never be used inside one of
the locking primitives that we map to the Solaris ones.

3. The kernel will print a backtrace to dmesg whenever wait_on_bit() is
used inside memory from vmalloc(), so any problems that arise would
appear very quickly in buildbots.

Consequently, it is reasonable to expect allocations intended for
structures that use `wake_up_bit()` to be done using the Linux
allocator. At present, the only allocations for such structures are done
inside the SPL SLAB allocator and for super blocks. No other code uses
it or is likely to use it.

These changes appear to create the most semantically equivalent mapping
possible on Linux. The result is the elimination of concerns regarding
proper use of generic interfaces when writing portable code, which posed
problems for the development of things like sgbuf.

Signed-off-by: Richard Yao <ryao@gentoo.org>

Remove kmem_alloc_node interface

This interface has no external consumers and does not exist on Solaris.
Lets remove it.

Signed-off-by: Richard Yao <ryao@gentoo.org>
KM_SLEEP kmem_cache allocations should not block high priority allocations

If we have a cache that has no free objects, two concurrent allocations
occur from a cache where one is KM_SLEEP and another is KM_PUSHPAGE in a
the sync task context (for example), the KM_SLEEP begins growing the
cache, the KM_PUSHPAGE allocation is silently converted to a KM_SLEEP
allocation. We have been changing allocations from KM_SLEEP to
KM_PUSHPAGE to try to avoid deadlocks. If there is merit to that, then
we should not allow races to randomly convert KM_PUSHPAGE allocations
into KM_SLEEP allocations.

Taking advantage of PF_FSTRANS will allow us to switch all allocations
that we changed to KM_PUSHPAGE back to KM_SLEEP because KM_SLEEP
allocations in key callpaths will automatically become KM_PUSHPAGE
allocations. That will make avoiding the potential for a KM_PUSHPAGE
allocation to be silently converted into a KM_SLEEP allocation even more
important because there will be increased opportunity for it. It might
seem better to let everything use KM_PUSHPAGE, but that is not the case
because allowing us to block in all allocations that are not necessary
for forward progress will reduce our burden under memory pressure, which
will ultimately improve system throughput.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Add hooks for disabling direct reclaim inside DMU transactions

The port of XFS to Linux introduced a thread-specific PF_FSTRANS bit
that is used to mark transactions to that the translation of the IRIX
kmem flags into Linux gfp flags for allocations inside of transactions
will dip into kernel memory reserves to avoid deadlocks during
writeback. Linux 3.9 provided the additional PF_MEMALLOC_NOIO for
disabling __GFP_IO in page allocations, which XFS began using in 3.15.

This patch implements hooks for marking transactions via PF_FSTRANS.
When an allocation is performed with in the context of PF_FSTRANS, any
KM_SLEEP allocation is transparently converted into a KM_PUSHPAGE
allocation. It will also set PF_MEMALLOC_NOIO to prevent direct reclaim
from entering `pageout()` on any KM_PUSHPAGE or KM_NOSLEEP allocation on
Linux 3.9 or later.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Revert "Linux 3.16 compat: smp_mb__after_clear_bit()"

This reverts commit e3020723dc43af2bc22af0d68571a61daf9b44d0.
spl-kmem: Enforce architecture-specific barriers around clear_bit()

The comment above the Linux 3.17 kernel's clear_bit() states:

/**
 * clear_bit - Clears a bit in memory
 * @nr: Bit to clear
 * @addr: Address to start counting from
 *
 * clear_bit() is atomic and may not be reordered.  However, it does
 * not contain a memory barrier, so if it is used for locking purposes,
 * you should call smp_mb__before_atomic() and/or smp_mb__after_atomic()
 * in order to ensure changes are visible on other processors.
 */

This comment does not make sense in the context of x86 because x86 maps these
operations to barrier(), which is a compiler barrier. However, it does make
sense to me when I consider architectures that reorder around atomic
instructions. In such situations, a processor is allowed to execute the
wake_up_bit() before clear_bit() and we have a race. There are a few
architectures that suffer from this issue:

http://lxr.free-electrons.com/source/arch/arm/include/asm/barrier.h?v=3.16#L83
http://lxr.free-electrons.com/source/arch/arm64/include/asm/barrier.h?v=3.16#L102
http://lxr.free-electrons.com/source/arch/mips/include/asm/barrier.h?v=3.16#L199
http://lxr.free-electrons.com/source/arch/powerpc/include/asm/barrier.h?v=3.16#L88
http://lxr.free-electrons.com/source/arch/s390/include/asm/barrier.h?v=3.16#L32
http://lxr.free-electrons.com/source/arch/tile/include/asm/barrier.h?v=3.16#L83
https://en.wikipedia.org/wiki/Memory_ordering

In such situations, the other processor would wake-up, see the bit is still
taken and go to sleep, while the one responsible for waking it up will assume
that it did its job and continue.

https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/bitops.h#L100

It is important to note that
smp_mb__{before,after}_{atomic,clear}_{dec,inc,bit}(), were replaced by
smp_mb__{before,after}_atomic() in recent kernels:

https://github.com/torvalds/linux/commit/febdbfe8a91ce0d11939d4940b592eb0dba8d663

Some compatibility code was added to replace it in the time being, although it
does not interact well with -Werror:

http://www.spinics.net/lists/backports/msg02669.html
http://lxr.free-electrons.com/source/include/linux/bitops.h?v=3.16#L48

In addition, the kernel's code paths are using clear_bit_unlock() in situations
where clear_bit is used for unlocking. This adds smp_mb__before_atomic(), which
I assume is for Alpha.

This patch implements a wrapper that maps smp_mb__{before,after}_atomic() to
smp_mb__{before,after}_clear_bit() on older kernels and changes our code to
leverage it in a manner consistent with the mainine kernel.

Signed-off-by: Richard Yao <ryao@gentoo.org>
kmem_cache: Call constructor/destructor on each alloc/free

This has a few benefits. First, it fixes a regression that "Rework
generic memory allocation interfaces" appears to have triggered in
splat's slab_reap and slab_age tests. Second, it makes porting code from
Illumos to ZFSOnLinux easier. Third, it has the side effect of making
reclaim from slab caches that specify reclaim functions an order of
magnitude faster. The splat slab_reap test usually took 30 to 40
seconds. With this change, it takes 3 to 4.

Signed-off-by: Richard Yao <ryao@gentoo.org>
Implement KM_USER for CONFIG_PAX_USERCOPY_SLABS support

Signed-off-by: Richard Yao <ryao@gentoo.org>