Remove fd from epoll when close.