Switch on errno, not fd, if there's an error on accept()