Allow three IOCTLs to be used on suspended pool, restoring state that
existed before IOCTL code refactoring merged change 4445fffb from illumos
at r248571.

This change allows `zpool clear` to be used again to recover suspended pool.
It seems the only was supposed by the code to restore pool operation after
reconnecting lost disks that were required for data completeness.  There
are still cases where `zpool clear` command can just safely stuck due to
deadlocks inside ZFS kernel part, but probably that is better then having
no chances to recover at all.

(cherry picked from commit c034c95e67abdb7eef95a34c9dc188f227a085cf)
Make ZFS to use separate thread to handle SPA_ASYNC_REMOVE async events.
Existing async thread is running only on successfull spa_sync() completion,
that is impossible in case of pool loosing required (last) disk(s).  That
indefinite delay of SPA_ASYNC_REMOVE processing made ZFS to not close the
lost disks, preventing GEOM/CAM from destroying devices and reusing names
on later disk reattach.

In earlier version of the patch I've tried to just run existing thread
immediately, unrelated to spa_sync() completion, but that exposed number
of situations where it could stuck due to locks held by stuck spa_sync(),
that are required for other kinds of async events.

Experiments with OpenIndiana snapshot confirmed that they also have this
issue with lost disks reattach.

(cherry picked from commit 070e8f39ecaa8deb6d80ad9f485ffb7d62225c9a)
Make `zpool clear` to reopen also reconnected cache and spare devices.
Since `zpool status` reports about such kinds of errors, it is strange
that they are not cleared by `zpool clear`.

(cherry picked from commit 3cc4899abadb075e1a1c87b9fe540fec982c89e5)
Block reporting of ZFS features for suspended pools.

Before executing any subcommand, zpool tool fetches pools configuration from
the kernel.  Before features support was added, kernel was regenerating that
configuration based on data always present in memory.  Unfortunately, pool
features list and activity counters are not such. They are stored in ZAP,
that normally resides in ARC, but under heavy memory pressure may be swapped
out.  If pool is suspended at this point, there is no way to recover it back
since any zpool command will stuck.

This change has one predictable flaw: `zpool upgrade` always wish to upgrade
suspended pools, but fortunately it can't do it due to the suspension.

(cherry picked from commit 2a26325c0f59f3a7e507ea2b1c10158ffe46590b)
Detect unsupported PREVENT ALLOW MEDIUM REMOVAL and SYNCHRONIZE CACHE(10)
to not spam devices with useless commands and logs with errors.

(cherry picked from commit 3be5fcbcfd0d79067617f4be3e084cf64188a218)
Oops, revert unwanted part of r253721.

(cherry picked from commit 1e4488ade806c781c177abec051ee54bfc5143a6)
Improve r253721 by reporting detected lack of BIO_FLUSH support to GEOM.
That prevents more of such requests from coming and errors from logging.

(cherry picked from commit 329753c36e1e9d64afce0bc82c9a4f02a64b7934)
Synchronize device cache on close only if there were some write operations.
While these operations are not really needed otherwise, at least for SCSI
they may cause extra errors if some other initiator holds write exclusive
reservation on the LUN (SYNCHRONIZE CACHE handled as "write" operation).

Conflicts:
	sys/cam/ata/ata_da.c

(cherry picked from commit 6761e997d0a0fa65f172398e3801639bb4de5ce1)