Fix assertion.

This would cause crashes in BasicServer (or other servers
that set a non-default host that was identical on client
and worker listeners).
Merge pull request #185 from vibraphone/fix-assert

Fix assertion.