msgport: Add message put done receipt support

The receipt callback is called immediately after the message is put on to
the target msgport.  Complex operation is NOT allowed in this callback,
since it could be called w/ spinlock for spin msgports.

It will be used to fix the race between the upper half (socket code)
and the so_port switching.
tcp: Fix upper half (socket code) and so_port switching race.

Use message put done receipt to change this socket's so_port, i.e. _after_
this message was put onto the target netisr's msgport but _before_ the
message could be pulled from the target netisr's msgport, so that:
- The upper half (socket code) will not see the new msgport before this
  message reaches the new msgport and messages for this socket will be
  ordered.
- This message will see the new msgport, when its handler is called in the
  target netisr.

Add comment about it and explain why the so_port switching should be done
in this commit's fashion.

Reported-by: dillon@
tools/netrate: Use FIONBIO to make the socket nonblocking