MFC: r253120

- As it turns out, not only MSI-X is broken for devices passed through by
  VMware up to at least ESXi 5.1. Actually, using INTx in that case instead
  may still result in interrupt storms, with MSI being the only working
  option in some configurations. So introduce a PCI_QUIRK_DISABLE_MSIX quirk
  which only blacklists MSI-X but not also MSI and use it for the VMware
  PCI-PCI-bridges. Note that, currently, we still assume that if MSI doesn't
  work, MSI-X won't work either - but that's part of the internal logic and
  not guaranteed as part of the API contract. While at it, add and employ
  a pci_has_quirk() helper.
  Reported and tested by: Paul Bucher
- Use NULL instead of 0 for pointers.

Submitted by:	jhb (mostly)
Approved by:	re (hrs), jhb
MFC r252852:

Update hpt27xx(4) driver to address a problem reported by FreeNAS
user, where when more than one hpt27xx adapters are being used,
the "unit number" stays at 0.

Many thanks to HighPoint for providing this driver update.

(This changeset have not included change found in 249468)

(cherry picked from commit b889e129e6f2e5075c932985a8de61f69aa1a4f9)
MFC/backport core kernel and userspace parts of r237263 (TCP_OFFLOAD
rework).  MFC r237563, r239511, r243603, r245915, r245916, r245919,
r245921, r245922, r245924, r245925, r245932, r245934 too.

Build tested with make universe.

r237263:
- Updated TOE support in the kernel.
...

r237563:
Fix clang warning when compiling iw_cxgb.

r239511:
Correctly handle the case where an inp has already been dropped by the time
the TOE driver reports that an active open failed.  toe_connect_failed is
supposed to handle this but it should be provided the inpcb instead of the
tcpcb which may no longer be around.

r243603:
Make sure that tcp_timer_activate() correctly sees TCP_OFFLOAD (or not).

r245915:
Heed SO_NO_OFFLOAD.

r245916:
Teach toe_4tuple_check() to deal with IPv6 4-tuples too.

r245919:
Add TCP_OFFLOAD hook in syncache_respond for IPv6 too, just like the one
that exists for IPv4.

r245921:
There is no need to call into the TOE driver twice in pru_rcvd (tod_rcvd
and then tod_output right after that).

r245922:
Avoid NULL dereference in nd6_storelladdr when no mbuf is provided.  It
is called this way from a couple of places in the OFED code.  (toecore
calls it too but that's going to change shortly).

r245924:
Move lle_event to if_llatbl.h

lle_event replaced arp_update_event after the ARP rewrite and ended up
in if_ether.h simply because arp_update_event used to be there too.
IPv6 neighbor discovery is going to grow lle_event support and this is a
good time to move it to if_llatbl.h.

The two in-tree consumers of this event - OFED and toecore - are not
affected.

r245925:
Generate lle_event in the IPv6 neighbor discovery code too.

r245932:
Teach toe_l2_resolve to resolve IPv6 destinations too.

r245934:
Add checks for SO_NO_OFFLOAD in a couple of places that I missed earlier
in r245915.

(cherry picked from commit 14ce029690f60a751344eeb985f594e19de0534a)
MFC r252867:

Import HighPoint DC Series Data Center HBA (DC7280 and R750) driver.
This driver works for FreeBSD/i386 and FreeBSD/amd64 platforms.

Many thanks to HighPoint for providing this driver.

(This changeset have not included changes found in 249468 and 246713)

(cherry picked from commit bbe575d85898a4942c9aeee7a8e18b6549702a4a)
Make it possible to import pool if ashift increased.

(cherry-picked from TrueNAS 3b8a0f9f7963caee66433c5ac4eb435e9e8ae6f6)
Fix typo; while I'm there, also correct the description for the sysctl variable.
Increase default ashift to 4K.
Change txg_quiesce back to non-static

This change (originally eb021586e4) has been mistakenly reverted in d01b55b1c3.
It is required to make zilstat work.

Submitted by:	Davide Italiano
MFC r252380:
Unbreak "zfs jail" and "zfs unjail" (broken in stable/9 since r249643)

I missed to register zfs_ioc_jail and zfs_ioc_unjail as legacy ioctl's
with the new zfs_ioctl_register_legacy() function.

These operations do not modify pools or datasets so there is no need to
log them to pool history.

Reported by:	Alexander Leidinger <netchild@FreeBSD.org> on current@
(cherry picked from commit e9345dc8e4b904827c63d74657a9e6cb30e3c849)
MFC r252056:
Fix destroyed ZFS pools failing to import

MFC r252059:
Fix ZFS zpool freeze (debug command) not processing due to invalid ioctl call syntax.

MFC r252060:
Fix intermittent ZFS lock panic

MFC r252061:
Switch ZFS mutex_owner macro to use sx_xholder as its now exported via sx.h

(cherry picked from commit b2b7537d513594904c16ddba20ff401854d2ed19)
MFC r252390:
Remove invalid ASSERT

(cherry picked from commit dabb037967e4a7519429defe038a956acc9b0b62)
Merge r252337 from head:

  Don't try to re-insert an already present but invalid page.

  This could happen if a thread doing a page-in loses a ZFS range lock
  race to a thread writing to the same range

  This fixes "panic: vm_page_alloc: pindex already allocated" in
  http://docs.FreeBSD.org/cgi/mid.cgi?1372165971.96049.42.camel

Submitted by:	avg

(cherry picked from commit 3d1c860017700196460cc094d4718fc9052c79e5)
Allow three IOCTLs to be used on suspended pool, restoring state that
existed before IOCTL code refactoring merged change 4445fffb from illumos
at r248571.

This change allows `zpool clear` to be used again to recover suspended pool.
It seems the only was supposed by the code to restore pool operation after
reconnecting lost disks that were required for data completeness.  There
are still cases where `zpool clear` command can just safely stuck due to
deadlocks inside ZFS kernel part, but probably that is better then having
no chances to recover at all.

(cherry picked from commit c034c95e67abdb7eef95a34c9dc188f227a085cf)
Make ZFS to use separate thread to handle SPA_ASYNC_REMOVE async events.
Existing async thread is running only on successfull spa_sync() completion,
that is impossible in case of pool loosing required (last) disk(s).  That
indefinite delay of SPA_ASYNC_REMOVE processing made ZFS to not close the
lost disks, preventing GEOM/CAM from destroying devices and reusing names
on later disk reattach.

In earlier version of the patch I've tried to just run existing thread
immediately, unrelated to spa_sync() completion, but that exposed number
of situations where it could stuck due to locks held by stuck spa_sync(),
that are required for other kinds of async events.

Experiments with OpenIndiana snapshot confirmed that they also have this
issue with lost disks reattach.

(cherry picked from commit 070e8f39ecaa8deb6d80ad9f485ffb7d62225c9a)
Make `zpool clear` to reopen also reconnected cache and spare devices.
Since `zpool status` reports about such kinds of errors, it is strange
that they are not cleared by `zpool clear`.

(cherry picked from commit 3cc4899abadb075e1a1c87b9fe540fec982c89e5)
Block reporting of ZFS features for suspended pools.

Before executing any subcommand, zpool tool fetches pools configuration from
the kernel.  Before features support was added, kernel was regenerating that
configuration based on data always present in memory.  Unfortunately, pool
features list and activity counters are not such. They are stored in ZAP,
that normally resides in ARC, but under heavy memory pressure may be swapped
out.  If pool is suspended at this point, there is no way to recover it back
since any zpool command will stuck.

This change has one predictable flaw: `zpool upgrade` always wish to upgrade
suspended pools, but fortunately it can't do it due to the suspension.

(cherry picked from commit 2a26325c0f59f3a7e507ea2b1c10158ffe46590b)
Detect unsupported PREVENT ALLOW MEDIUM REMOVAL and SYNCHRONIZE CACHE(10)
to not spam devices with useless commands and logs with errors.

(cherry picked from commit 3be5fcbcfd0d79067617f4be3e084cf64188a218)
Oops, revert unwanted part of r253721.

(cherry picked from commit 1e4488ade806c781c177abec051ee54bfc5143a6)
Improve r253721 by reporting detected lack of BIO_FLUSH support to GEOM.
That prevents more of such requests from coming and errors from logging.

(cherry picked from commit 329753c36e1e9d64afce0bc82c9a4f02a64b7934)
Synchronize device cache on close only if there were some write operations.
While these operations are not really needed otherwise, at least for SCSI
they may cause extra errors if some other initiator holds write exclusive
reservation on the LUN (SYNCHRONIZE CACHE handled as "write" operation).

Conflicts:
	sys/cam/ata/ata_da.c

(cherry picked from commit 6761e997d0a0fa65f172398e3801639bb4de5ce1)