Migrate the multicast queue assembly code to not use the axq_link pointer
and instead use the HAL method to set the link pointer.

Tested:

* AR9280, hostap mode, CABQ frames being queued and transmitted


git-svn-id: svn://svn.freebsd.org/base/head@248713 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Convert the EDMA multicast queue code over to use the HAL method to set
the descriptor link pointer, rather than directly.

This is needed on AR9380 and later (ie, EDMA) NICs so the multicast queue
has a chance in hell of being put together right.

Tested:

* AR9380, AR9580 in hostap mode, CABQ traffic (but with other patches..)


git-svn-id: svn://svn.freebsd.org/base/head@248714 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Convert the CABQ queue code over to use the HAL link pointer method
instead of axq_link.

This (among a bunch of uncommitted work) is required for EDMA chips
to correctly transmit frames on the CABQ.

Tested:

* AR9280, hostap mode
* AR9380/AR9580, hostap mode (staggered beacons)

TODO:

* This code only really gets called when burst beacons are used;
  it glues multiple CABQ queues together when sending to the hardware.
* More thorough bursted beacon testing! (first requires some work with
  the beacon queue code for bursted beacons, as that currently uses the
  link pointer and will fail on EDMA chips.)


git-svn-id: svn://svn.freebsd.org/base/head@248715 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Remove this dead code - it's no longer relevant (as yes, we do actually
support TX on EDMA chips.)


git-svn-id: svn://svn.freebsd.org/base/head@248716 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Remove the mcast path calls to ath_hal_gettxdesclinkptr() for axq_link -
they're no longer needed for the legacy path and they're not wanted
for the EDMA path.

Tested:

* AR9280, hostap + CABQ
* AR9380/AR9580, hostap + CABQ


git-svn-id: svn://svn.freebsd.org/base/head@248717 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Remove extra bio_data and bio_length copying to child request after calling
g_clone_bio(), that already copied them.


git-svn-id: svn://svn.freebsd.org/base/head@248720 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
GEOM NOP does not touch the data, so pass G_PF_ACCEPT_UNMAPPED flag through.


git-svn-id: svn://svn.freebsd.org/base/head@248721 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
geom_slice.c and its consumers like GEOM_LABEL are not touching the data
unless hotspots are used.  Pass G_PF_ACCEPT_UNMAPPED flag through except
such rare cases (obsolete GEOM_SUNLABEL and GEOM_BSD).


git-svn-id: svn://svn.freebsd.org/base/head@248722 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Cleanup: wrap long lines, cleanup comments, etc.


git-svn-id: svn://svn.freebsd.org/base/head@248724 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Return ENOMEM if malloc() fails.


git-svn-id: svn://svn.freebsd.org/base/head@248725 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Introduce vm_radix_isleaf() and use it in a couple places.  As compared to
using vm_radix_node_page() == NULL, the compiler is able to generate one
less conditional branch when vm_radix_isleaf() is used.  More use cases
involving the inner loops of vm_radix_insert(), vm_radix_lookup{,_ge,_le}(),
and vm_radix_remove() will follow.

Reviewed by:	attilio
Sponsored by:	EMC / Isilon Storage Division


git-svn-id: svn://svn.freebsd.org/base/head@248728 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Do not look at the namespace's thin provisioning field to determine if DSM
command is supported.  The two are not related.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248729 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Make the DSM range count 0-based.  Previously we were deallocating one more
LBA than we should have been.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248730 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Add an internal _nvme_qpair_submit_request function, which performs
the submit action assuming the qpair lock has already been acquired.

Also change nvme_qpair_submit_request to just lock/unlock the mutex
around a call to this new function.

This fixes a recursive mutex acquisition in the retry path.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248731 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Add support for ABORT commands, including issuing these commands when
an I/O times out.

Also ensure that we retry commands that are aborted due to a timeout.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248732 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Break out the code for completing an nvme_tracker object into a separate
function.

This allows for completions outside the normal completion path, for example
when an ABORT command fails due to the controller reporting the targeted
command does not exist.  This is mainly for protection against a faulty
controller, but we need to clean up our internal request nonetheless.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248733 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Explicitly abort a timed out command, if the ABORT command sent to the
controller indicates the command was not found.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248734 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Specify command timeout interval on a per-command type basis.

This is primarily driven by the need to disable timeouts for asynchronous
event requests, which by nature should not be timed out.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248735 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Move controller destruction code from nvme_detach() to new nvme_ctrlr_destruct()
function.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248736 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
Enable asynchronous event requests on non-Chatham devices.

Also add logic to clean up all outstanding asynchronous event requests
when resetting or shutting down the controller, since these requests
will not be explicitly completed by the controller itself.

Sponsored by:	Intel


git-svn-id: svn://svn.freebsd.org/base/head@248737 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f