Remove unnecessary mask
Convert to using a gen_server
Remove the fd on timeout

If a poll timeout was requested and the poll times out, remove the fd to
prevent future events from being sent.