Merge updated patch found in #664, which fixes #664 by removing the optimization of not using 'poll' when we are listening on only 1 socket. This will slow us down when listening to one socket only,
but hopefully only slightly. It does simplify the code.