Handle file descriptors and poll fds separately.