In addition to r259632 completely block receive upcalls if we have more
data than we need.  This reduces lock pressure from xprt_active() side.

(cherry picked from commit 6ab680bb65a144b9f95b93ecb6acc9e1ae617204)
Introduce xprt_inactive_self()  -- variant for use when sure that port
is assigned to thread.  For example, withing receive handlers.  In that
case the function reduces to single assignment and can avoid locking.

(cherry picked from commit 082b09553bf99d782de5d8c7d673661d4dd9b442)
Move most of NFS file handle affinity code out of the heavily congested
global RPC thread pool lock and protect it with own set of locks.

On synthetic benchmarks this improves peak NFS request rate by 40%.

(cherry picked from commit 16b0bb1da43bd50fbfc9131167e0d1103aa37de9)
Fix the build.
Revert "The NFS drc code uses TCP seq# in order to decide if an RPC"

This reverts commit 67497ccaf074d3fefadfa9bb768981884452b8d2.
Fix several performance related issues in the new NFS server's
DRC for NFS over TCP.
- Increase the size of the hash tables.
- Create a separate mutex for each hash list of the TCP hash table.
- Single thread the code that deletes stale cache entries.
- Add a tunable called vfs.nfsd.tcphighwater, which can be increased
  to allow the cache to grow larger, avoiding the overhead of frequent
  scans to delete stale cache entries.
  (The default value will result in frequent scans to delete stale cache
   entries, analagous to what the pre-patched code does.)
- Add a tunable called vfs.nfsd.cachetcp that can be used to disable
  DRC caching for NFS over TCP, since the old NFS server didn't DRC cache TCP.
It also adjusts the size of nfsrc_floodlevel dynamically, so that it is
always greater than vfs.nfsd.tcphighwater.

For UDP the algorithm remains the same as the pre-patched code, but the
tunable vfs.nfsd.udphighwater can be used to allow the cache to grow
larger and reduce the overhead caused by frequent scans for stale entries.
UDP also uses a larger hash table size than the pre-patched code.

Reported by:	wollman
Tested by:	wollman (earlier version of patch)
Submitted by:	ivoras (earlier patch)
Reviewed by:	jhb (earlier version of patch)
MFC after:	1 month

Conflicts:
	sys/fs/nfsserver/nfs_nfsdcache.c
Slightly simplify expiration logic introduced in r254337.

 - Do not update the histogram for items we are any way deleting from cache.
 - Do not update the histogram if nfsrc_tcphighwater is not set.
 - Remove some extra math operations.
Rework NFS Duplicate Request Cache cleanup logic.

 - Introduce additional hash to group requests by hash of sockref.  This
allows to process TCP acknowledgements without looping though all the cache,
and as result allows to do it every time.
 - Indroduce additional callbacks to notify application layer about sockets
disconnection.  Without this last few requests processed just before socket
disconnection never processed their ACKs and stuck in cache for many hours.
 - Implement transport-specific method for tracking reply acknowledgements.
New implementation does not cross multiple stack layers to get the data and
does not have race conditions that previously made some requests stuck
in cache.  This could be done more efficiently at sockbuf layer, but that
would broke some KBIs, while I don't know other consumers for it aside NFS.
 - Instead of traversing all DRC twice per request, run cleaning only once
per request, and except in some conditions traverse only single hash slot
at a time.

Together this limits NFS DRC growth only to situations of real connectivity
problems.  If network is working well, and so all replies are acknowledged,
cache remains almost empty even after hours of heavy load.  Without this
change on the same test cache was growing to many thousand requests even
with perfectly working local network.

As another result this reduces CPU time spent on the DRC handling during
SPEC NFS benchmark from about 10% to 0.5%.

Sponsored by:	iXsystems, Inc.

Conflicts:
	sys/rpc/svc_vc.c
Replace locks added in r260229 to protect sequence counters with atomics.

New algorithm does not create additional lock congestion, while some races
it includes should not be a problem.  Those races may keep requests in DRC
cache for some more time by returning ACK position smaller then actual,
but it still should be able to drop thems when proper ACK finally read.

Races of the original algorithm based on TCP seq number were worse because
they happened when reply sequence number were recorded. After that even
correctly read ACKs could not clean DRC sometimes.

(cherry picked from commit 048eda83eb56efd0ddfd66e87d6885d60c22183a)