Add special support for IPAddress.None for servers

Fixes #420