Ensure that the underlying file descriptor is removed from the poll-set.