Handle the case when the fd is removed from the list.