Set zfs_arc_min to 4MB

Decrease the mimimum ARC size from 1/32 of total system memory
(or 64MB) to a much smaller 4MB.

1) Large systems with over a 1TB of memory are being deployed
   and reserving 1/32 of this memory (32GB) as the mimimum
   requirement is overkill.

2) Tiny systems like the raspberry pi may only have 256MB of
   memory in which case 64MB is far too large.

The ARC should be reclaimable if the VFS determines it needs
the memory for some other purpose.  If you want to ensure the
ARC is never completely reclaimed due to memory pressure you
may still set a larger value with zfs_arc_min.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Enable l2arc reclaim

Currently all l2arc headers count against the meta limit and
the only way to release them is to cleanup an l2arc device.
This means that given a large enough l2arc device the headers
can accumulate until they push out all other meta data.

While we absolutely don't want to commonly discard these headers,
because it invalidates any blocks written to the l2arc device,
it should be possible to reclaim them.  If memory is desperately
needed by an application it's preferable to reclaim this memory
and pay the performance penalty.

To do this a l2arc_evict_headers() function has been added.  It
will attempt to reclaim N bytes of l2arc headers by repeatedly
advancing the write hand and evicting blocks from the l2arc.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Allow arc_evict_ghost() to only evict meta data

The default behavior of arc_evict_ghost() is to start by evicting
data buffers.  Then only if the requested number of bytes to evict
cannot be satisfied by data buffers move on to meta data buffers.

This is ideal for honoring arc_c since it's preferable to keep the
meta data cached.  However, if we're trying to free memory from the
arc to honor the meta limit it's a problem because we will need to
discard all the data to get to the meta data.

To avoid this issue the arc_evict_ghost() is now passed a fourth
argumented describing which buffer type to start with.  The
arc_evict() function already behaves exactly like this for a
same reason so this is consistent with the existing code.

All existing callers have been updated to pass ARC_BUFC_DATA so
this patch introduces no functional change.  New callers may
pass ARC_BUFC_METADATA to skip immediately to evicting meta
data leaving the normal data untouched.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Evict meta data from ghost lists + l2arc headers

When the meta limit is exceeded the ARC evicts some meta data
buffers from the mfu+mru lists.  Unfortunately, for meta data
heavy workloads it's possible for these buffers to accumulate
on the ghost lists if arc_c doesn't exceed arc_size.

To handle this case arc_adjust_meta() has been entended to
explicitly evict meta data buffers from the ghost lists in
proportion to what was evicted from the mfu+mru lists.

If this is insufficient we request that the VFS release
some inodes and dentries.  As a last resort if we're still
more than 12.5% over the arc_meta_limit evict some l2arc
headers.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Fix l2arc_write_done/l2arc_evict race

A GPF may occur if an l2arc buffer is evicted before the write
completes for the l2arc buffer.  The l2arc_write_done() function
unconditionally starts walking the list at the write head so we
must verify that l2arc_evict() has not removed it from the list.

This long standing issue was exposed by allowing arc_adjust() to
evict entries from the l2arc due to memory pressue.  Prior to
this change buffers would only be evicted immediately in front
of the write hand or during l2arc cleanup.  In neither case could
we wrap around remove the write head as part of l2arc_evict().

PID: 3182   TASK: ffff88020df28080  CPU: 0   COMMAND: "z_null_int/0"
  [exception RIP: l2arc_write_done+128]
 6 [ffff880210e97c58] zio_done at ffffffffa0597e0c [zfs]
 7 [ffff880210e97cd8] zio_done at ffffffffa05982a7 [zfs]
 8 [ffff880210e97d58] zio_done at ffffffffa05982a7 [zfs]
 9 [ffff880210e97dd8] zio_execute at ffffffffa05948a3 [zfs]
10 [ffff880210e97e18] taskq_thread at ffffffffa041d8d8 [spl]
11 [ffff880210e97ee8] kthread at ffffffff81090d26
12 [ffff880210e97f48] kernel_thread at ffffffff8100c14a

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Write dirty inodes on close

When the property atime=on is set operations which only access
and inode do cause an atime update.  However, it turns out that
dirty inodes with updated atimes are only written to disk when
the inodes get evicted from the cache.  Somewhat surprisingly
the source suggests that this isn't a ZoL specific issue.

This behavior may in part explain why zfs's reclaim logic has
been observed to be slow.  When reclaiming inodes its likely
that they have a dirty atime which will force a write to disk.

Obviously we don't want to force a write to disk for every
atime update, these needs to be batched.  The right way to
do this is to fully implement the .dirty_inode and .write_inode
callbacks.  However, to do that right requires proper unification
of some fields in the znode/inode.  Then we could just mark the
inode dirty and leave it to the VFS to call .write_inode
periodically.

Until that work gets done we have to settle for some middle
ground.  The simplest and safest thing we can do for now is
to write the dirty inode on last close.  This should prevent
the majority of inodes in the cache from having dirty atimes
and not drastically increase the number of writes.

Some rudimentally testing to show how long it takes to drop
500,000 inodes from the cache shows promising results.  This
is as expected because we're no longer do lots of IO as part
of the eviction, it was done earlier during the close.

w/out patch: ~30s to drop 500,000 inodes with drop_caches.
with patch:  ~3s to drop 500,000 inodes with drop_caches.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>