Adjust arc_p based on "bytes" in arc_shrink

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Limit L2ARC header footprint

Currently there is not mechanism in place to reclaim or limit the number
of L2ARC headers contained in RAM. This means that given a large enough
L2ARC device, the headers can accumulate until they push out all other
data, and can potentially cause an out-of-memory event on the system.

While we'd rather not discard these headers, because it invalidates any
blocks written to the L2ARC device, we also don't want these headers to
push out all other (potentially more useful) data on the system and/or
cause the system to hit a out-of-memory event.

As a fist attempt to fix this issue, a new "l2arc_hdr_limit" module
parameter has been introduced. This option allows an administrator to
set a limit on the cumulative size that L2ARC headers can grow to on the
system. By default this limit is 1/8th of the total ARC size.

To do this a l2arc_evict_headers() function has been added. It will
attempt to reclaim N bytes of l2arc headers by repeatedly advancing the
write hand and evicting blocks from the l2arc.

In addition, since we're not pruning from the L2ARC devices based on
"arc_meta_limit" or even "arc_c_max", so these headers were completely
taken out of the calculation of "arc_meta_used" and "arc_size".

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Redefine the arc_meta_used and arc_size values

The meaning of the arc_meta_used and arc_size values are ambiguous at
best. Prior to this change, the best definition I could come up with are
the following:

    * "arc_meta_used" is the sum of bonus buffers, dmu_buf_impl_t's,
      dnode_t's, arc headers, and arc buffers of type ARC_BUFC_METADATA

    * "arc_size" is the sum of arc_meta_used and arc buffers of type
      ARC_BUFC_DATA

With these two definitions in mind, their limits and associated logic for
eviction and arc tuning don't make a whole lot of sense; and here's why:

    1. First off, in the arc_adapt thread, we call arc_adjust_meta when
       arc_meta_used is greater than arc_meta_limit. The problem is that
       we evict buffer's of type ARC_BUFC_METADATA from the arc's mru
       and mfu before we every try to evict from the upper layers (via
       the prune callbacks). This can potentially lead us to evicting
       "hot" buffers from the mru and/or mfu while leaving "cold"
       dnode_t's, dmu_buf_impl_t's, etc. in the cache (consuming
       precious arc_meta_used space).

    2. Even though arc headers consume space within arc_meta_used, the
       ghost lists are never pruned based on arc_meta_limit. This can
       lead to a *very* unfortunate situation where the arc_met_used
       space is consumed entirely by "useless" arc headers (any new arc
       buffers of type ARC_BUFC_METADATA would almost immediately be
       evicted and placed on it's respective ghost list).

    3. The arc code appears to be littered these following assumptions:

           a) the size of the mfu is capped by arc_c - arc_p
           b) the sum of the mru and mfu sizes is capped by arc_c
           c) and finally, the sum of the mru and mfu sizes is arc_size

       The problem is, (c) is blatantly untrue (see previous definition
       above), which then invalidates the other assumptions as well.
       Using the proper definition of arc_size, we can arrive at the
       following truths:

           d) the arc_size is capped by arc_c
           e) the mru size is capped by arc_p
           f) the sum of arcstat_[data,other,hdr]_size is arc_size
           g) the sum of the mru and mfu sizes is arcstat_data_size
           h) which leads to, the sum of the mru and mfu sizes being
              capped by arc_c - arcstat_other_size - arcstat_hdr_size

       The unveiling of (h) means that, although unlikely, for certain
       values of arc_c, arcstat_other_size, and arcstat_hdr_size, one
       can completely starve the arc lists.

       The more likely scenario is not that *both* lists are starved
       simultaneously, it's that *one* list is starved for a certain
       time interval. For example, I've seen cases where the mru list
       grows to arc_p, and the mfu is squashed to a negligible size
       because it's space (the incorrectly assumed arc_c - arc_p space)
       is actually being consumed by arc header buffers, dnodes, etc.

As an attempt to remedy the situation, the meaning of the arc_meta_used
and arc_size values have been redefined as the following:

    * "arc_meta_used" is the sum of bonus buffers, and dmu_buf_impl_t's
      (i.e. arc_meta_used == arcstat_other_size)

    * "arc_size" is the sum of all arc buffers and arc headers
      (i.e. arc_size == arcstat_data_size + arcstat_hdr_size)

Signed-off-by: Prakash Surya <surya1@llnl.gov>