Fix assert error failures.

Only remove the file descriptors using remove_reader() or remove_writer() depending on how
the file descriptor was previously used.