Fix race condition between adding to poller and trying to send.