Reports pools which have a removed l2cache disk under -x as this is what happens when a cache device is dropped for any reason.

Reviewed by:	pjd
Approved by:	pjd (mentor)
MFC after:	2 weeks
Enhance the ZFS vdev layer to maintain both a logical and a physical
minimum allocation size for devices.  Use this information to
automatically increase ZFS's minimum allocation size for new top-level
vdevs to a value that more closely matches the optimum device
allocation size.

Use GEOM's stripesize attribute, if set, as the physical sector
size of the GEOM.

Calculate the minimum blocksize of each metaslab class.  Use the
calculated value instead of SPA_MINBLOCKSIZE (512b) when determining
the likelyhood of compression yeilding a reduction in physical space
usage.

Report devices with sub-optimal block size configuration in "zpool
status".  Also properly fail attempts to attach devices with a
logical block size greater than 8kB, since this will cause corruption
to ZFS's label area.

Sponsored by:       Spectra Logic Corporaion
MFC after:  2 weeks

Background
==========
Many modern devices use physical allocation units that are much
larger than the minimum logical allocation size accessible by
external commands.  Two prevalent examples of this are 512e disk
drives (512b logical sector, 4K physical sector) and flash devices
(512b logical sector, 4K or larger allocation block size, and 128k
or larger erase block size).  Operations that modify less than the
physical sector size result in a costly read-modify-write or garbage
collection sequence on these devices.

Simply exporting the true physical sector of the device to ZFS would
yield optimal performance, but has two serious drawbacks:

1) Existing pools created with devices that have different logical
   and physical block sizes, but were configured to use the logical
   block size (e.g. because the OS version used for pool construction
   reported the logical block size instead of the physical block
   size) will suddenly find that the vdev allocation size has
   increased.  This can be easily tolerated for active members of
   the array, but ZFS would prevent replacement of a vdev with
   another identical device because it now appears that the smaller
   allocation size required by the pool is not supported by the new
   device.

2) The device's physical block size may be too large to be supported
   by ZFS.  The optimal allocation size for the vdev may be quite
   large.  For example, a RAID controller may export a vdev that
   requires read-modify-write cycles unless accessed using 64k
   aligned/sized requests.  ZFS currently has an 8k minimum block
   size limit.

Reporting both the logical and physical allocation sizes for vdevs
solves these problems.  A device may be used so long as the logical
block size is compatible with the configuration.  By comparing the
logical and physical block sizes, new configurations can be optimized
and administrators can be notified of any existing pools that are
sub-optimal.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h:
    Add the SPA_ASHIFT constant.  ZFS currently has a hard upper
    limit of 13 (8k) for ashift and this constant is used to
    both document and enforce this limit.

sys/cddl/contrib/opensolaris/uts/common/sys/fs/zfs.h:
    Add the VDEV_AUX_ASHIFT_TOO_BIG error code.

    Add fields for exporting the configured, logical, and
    physical ashift to the vdev_stat_t structure.

    Add VDEV_STAT_VALID() macro which can be used to verify the
    presence of required vdev_stat_t fields in nvlist data.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
    Provide a SYSCTL_PROC handler for "max_auto_ashift".  Since
    the limit is only referenced long after boot when a create
    operation occurs, there's no compelling need for it to be
    a boot time configurable tunable.  This also allows the
    validation code for the max_auto_ashift value to be contained
    within the sysctl handler.

    Populate the new fields in the vdev_stat_t structure.

    Fail vdev opens if the vdev reports an ashift larger than
    SPA_MAXASHIFT.

    Propogate vdev_logical_ashift and vdev_physical_ashift between
    child and parent vdevs as is done for vdev_ashift.

    In vdev_open(), restore code that fails opens for devices
    where vdev_ashift grows.  This can only happen now if the
    device's logical ashift grows, which means it really isn't
    safe to use the device.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c:
    Update the vdev_open() API so that both logical (what was
    just ashift before) and physical ashift are reported.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h:
    Add two new fields, vdev_physical_ashift and vdev_logical_ashift,
    to vdev_t.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_config.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:
    Add vdev_ashift_optimize().  Call it anytime a new top-level
    vdev is allocated.

cddl/contrib/opensolaris/cmd/zpool/zpool_main.c:
    Add text for the VDEV_AUX_ASHIFT_TOO_BIG error.

    For each sub-optimally configured leaf vdev, report configured
    and native block sizes.

cddl/contrib/opensolaris/cmd/zpool/zpool_main.c:
cddl/contrib/opensolaris/lib/libzfs/common/libzfs.h:
cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c:
    Introduce a new zpool status: ZPOOL_STATUS_NON_NATIVE_ASHIFT.
    This status is reported on healthy pools containing vdevs
    configured to use a block size smaller than their reported
    physical block size.

cddl/contrib/opensolaris/lib/libzfs/common/libzfs_status.c:
    Update find_vdev_problem() and supporting functions to
    provide the full vdev_stat_t structure to problem checking
    routines, and to allow decent into replacing vdevs.

    Add a vdev_non_native_ashift() validator which is used on
    the full vdev tree to check for ZPOOL_STATUS_NON_NATIVE_ASHIFT.

cddl/contrib/opensolaris/lib/libzpool/common/kernel.c:
cddl/contrib/opensolaris/lib/libzpool/common/sys/zfs_context.h:
    Enhance sysctl userland stubs now that a SYSCTL_PROC handler
    is used in vdev.c.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab_impl.h:
    When the group membership of a metaslab class changes (i.e.
    when a vdev is added or removed from a pool), walk the group
    list to determine the smallest block size currently available
    and record this in the metaslab class.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/metaslab.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:
    Add the metaslab_class_get_minblocksize() accessor.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio_compress.h:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c:
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
    In zio_compress_data(), take the minimum blocksize as an
    input parameter instead of assuming SPA_MINBLOCKSIZE.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:
    In l2arc_compress_buf(), pass SPA_MINBLOCKSIZE as the minimum
    blocksize of the device.  The l2arc code performs has it's own
    code for deciding if compression is worth while, so this
    effectively disables zio_compress_data() from second guessing
    the original decision.

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:
    In zio_write_bp_init(), use the minimum blocksize of the
    normal metaslab class when compressing data.