Correctly handle remove from epoll and EPOLLRDHUP. Also fix a bug with removing EPOLLIN and EPOLLOUT