Correct ignoring socket when inappropriate

When fixing the performance by declining to select() on a socket
until a recvfrom() explictly occurs to clear the socket, a problem
was injected where a socket could be made ignored by particular
timing of incoming traffic on the socket.  Correct this by having
the _poller function forcefully return True if any sockets
are ignored (which also implies they are ready since they
should be discarded on read).

Change-Id: I6be39d39e4d2ed3b05af9a4c954fb64c993ffb50