Close the correct file descriptor