Create pnet, parallel network stack, project branch.  This branch will
hold a variety of bits and pieces I have in flight, including netisr2
(multi-CPU netisr model), IP subset (multiple sockets with the same
binding for load-balancing purposes), and other WIPs.
Add IP SUBSET patch to pnet branch: the IP_SUBSET socket option allows
identically bound UDP sockets to balance load between them using various
strategies, including random assignment, flow-based assignment,
CPU-based assignment, and kernel thread ID-based assignment.

UDP applications, such as BIND, memcached, etc, can create multiple
sockets, each with SO_REUSEPORT set, followed by specifying their index
among a set of matching sockets all servicing the same port number.
Drop netisr2 prototype from a few years ago into svn; previously it was
in Perforce.  This prototype has per-CPU netisr threads, and allows
protocols to provide handlers to analyze packets to decide where they
should be processed.  The current implementation is fairly dated, and is
not aware of per-mbuf flow IDs, for example, but may be a useful
starting point for future work.
Force 64-byte alignment of workstreams so that sequential arrays, indexed
by CPU ID, are less likely to cause cache line thrashing.
Merge head/sys r186685-191380 to projects/pnet/sys.
implify and improve netisr2 significantly:

- Allow protocols to map mbufs to flow IDs, and flow IDs to CPUs
- Always use the flowid field in the packet header to carry flow IDs with
  packets
- Always use write-locking for protocol registration, even if we've
  compiled out read-locking for the fast path
- Don't lock around per-workflow statistics
- Use a simpler default flow->CPU mapping for protocols without specific
  affinity requirements
Add NETISR protocol constant for ethernet input, in order to allow the
ethernet input path to be scheduled in a deferred context.
Revise netisr2 interface some more:

- Add public APIs to query available workstream counts and CPU IDs for
  them (netisr2_get_cpucount(), netisr2_get_cpuid()).

- Provide default flow2cpu inmplementation as netisr2_default_flow2cpu,
  and install it if the protocol doesn't register one.

- Add uintptr_t source argument to netisr2_queue() and netisr2_dispatch(),
  allowing to caller to provide additional information about a source that
  might be used to improve parallelism by constraining ordering.  Should
  be set to 0 if none is identified (i.e., maximum ordering).

- If the protocol doesn't provide a flow ID conversion function and there
  isn't one on the packet, use the source to order.
Use SWIs for netisr2 rather than kthreads -- SWIs have a slightly
more efficient wakeup routine (since they don't need to look up
sleep queues).

Suggested by:	jhb
Create at most one thread by default -- we want this to be a drop-in
replacement for netisr1 in 8.0, so don't want to deal with full
performance evaluation of the MP case yet.

Add a tunable/sysctl to disable binding threads to CPUs, and set that
by default, as we'll be using just one netisr by default.
Add netisr2_dispatch_if() and netisr2_queue_if(), which accept an explicit
ifnet argument.  For now we just use it to contribute to an overall flow
identifier, but in the future we might want to use it as a source of work
placement policy.

Implement netisr_dispatch() and netisr_queue() for netisr2, which allows
existing netisr dispatch points to be used unchanged.  These are simple
wrappers around netisr2_*_if(), and pass in the mbuf's receieve interface
pointer, which will cause netisr2 to distribute work to threads even
without protocol- or driver-generated flow IDs.

When there's only one netisr2 worker, don't bother with flow IDs, CPU
lookups, etc, just use the one thread, avoiding overhead.

Rename netisr2_deregister() to netisr2_unregister() to match existing
netisr naming convention.
Allow netisr2 to be used wherever netisr is used, subject to options
NETISR2.  In most cases, rely on existing ifnet ordering, or driver-
provided flow IDs, but for IP divert sockets use the sending socket,
and IPSEC use the security association for ordering.  No custom
protocol handlers are registered.
Binding interrupts to a CPU consists of two parts: setting up CPU
affinity for the interrupt thread, and requesting that underlying
hardware direct interrupts to the CPU.  For software interrupt
threads, implement a no-op interrupt event binder that returns
success, so that the interrupt management code will just set the
ithread's affinity and succeed.
Further refinement of netisr2:

- Add netisr2_getqlimit() and netisr2_setqlimit() interfaces to allow
  protocols to query and manipulate per-workstream queue depth limits.
  This is required for routing socket and IPv4 netisrs which currently
  offer this functionality.

- Add netisr2_getqdrops() and netisr2_clearqdrops() interfaces to allow
  protocols to query drops across CPUs, as well as clear drop
  statistics.  This is required for IPv4.

- Use u_int64_t rather than u_int for stats.

- Rather than passing in each parameter individually for
  netisr2_register(), netisr2_unregister(), define a public struct
  netisr_handler, with padding, to describe protocols.

- Explicitly enumerate policies supported by netisr2, rather than
  deriving them from implemented function pointers; this allows multiple
  policies to depend on the same function pointers if desired.  We
  implement three policies now: NETISR_POLICY_SOURCE, NETISR_POLICY_FLOW,
  NETISR_POLICY_CPU.

- Now that we use swi's, we can acquire the netisr lock around processing
  runs, since the wakeup can be waited for without holding the workstream
  lock.

- Garbage collect NWS_SWI_BOUND and manual binding with sched_bind(),
  use intr_event_bind() now that this is supported for software
  interrupt threads.
Update netisr2 consumers for new netisr2 interfaces -- queue limits,
stats, and struct netisr_handler registration.
Fix bug in netisr_setqlimit().
Implement nh_m2flow for IPv4 -- if no flow ID is available on an mbuf
destined for NETISR_IP, calculate a flow ID quickly using the source
and destination IP addresses.  It's easy to imagine something more
complicated here, such as using the same RSS algorithm and parameters
programmed into cxgb and other supporting devices.
Use rmlocks for netisr2 and enable read-locking in the dispatch
path.  Despite deeply upsetting WITNESS, this should actually be
safe as the rmlock is always acquired read-locked in any
processing path that could otherwise lead to a lock order
reversal, so we'll want to supress these warnings as with pfil
lock warnings.
Use NETISR2 in GENERIC in this branch.
Minor style tweak for netisr2 sysctls.

Submitted by:	bde