When using Streaming and AsyncCallbackHandler, sometimes it is not possible to obtain the complete token concatenation string.
Labels: name: auto:bug, name: area: models

### System Info

I encountered a very strange problem when using AsyncCallbackHandler. Most of the time, it works normally, but occasionally strange problems occur. Therefore, I logged the entire call process in each function, and finally learned from the logs that under normal circumstances, the logs are on_ llm_ start, then the API call log for calling openai, and finally on_llm_ end, but occasionally this is not the case, and in exceptional cases, the log order is on_ llm_ start, on_ llm_ end, followed by the API call log of OpenAI, which occasionally prevents me from obtaining the complete token concatenated string because I need to record the complete answer of AI each time. Therefore, I checked the source code and temporarily did not find how the while loop broke out. While not self. queue. empty() or not self. done.is_ Set () This needs to satisfy both that the queue is not empty and that done has been set. Where are the specific issues and I hope to receive answers

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [X] Async

### Reproduction

class AsyncIteratorCallbackHandler(AsyncCallbackHandler):
    """Callback handler that returns an async iterator."""

    queue: asyncio.Queue[str]

    done: asyncio.Event

    @property
    def always_verbose(self) -> bool:
        return True

    def __init__(self) -> None:
        self.queue = asyncio.Queue()
        self.done = asyncio.Event()

    async def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> None:
        # If two calls are made in a row, this resets the state
        self.done.clear()

    async def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        if token is not None and token != "":
            self.queue.put_nowait(token)

    async def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
        self.done.set()

    async def on_llm_error(
        self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any
    ) -> None:
        self.done.set()

    # TODO implement the other methods

    async def aiter(self) -> AsyncIterator[str]:
        while not self.queue.empty() or not self.done.is_set():
            # Wait for the next token in the queue,
            # but stop waiting if the done event is set
            done, other = await asyncio.wait(
                [
                    # NOTE: If you add other tasks here, update the code below,
                    # which assumes each set has exactly one task each
                    asyncio.ensure_future(self.queue.get()),
                    asyncio.ensure_future(self.done.wait()),
                ],
                return_when=asyncio.FIRST_COMPLETED,
            )

            # Cancel the other task
            if other:
                other.pop().cancel()

            # Extract the value of the first completed task
            token_or_done = cast(Union[str, Literal[True]], done.pop().result())

            # If the extracted value is the boolean True, the done event was set
            if token_or_done is True:
                break

            # Otherwise, the extracted value is a token, which we yield
            yield token_or_done

### Expected behavior

AsyncCallbackHandler can execute strictly in the order of on_llm_start, openai call(on_llm_new_token), and on_llm_end each time.