【BUG】ConversationalRetrievalChain.from_llm and pass in chat_history, there is a problem with the callback.
Labels: name: auto:bug, name: area: models

### System Info

python3.10
langchain 0.0.333

### Who can help?

@hwchase17 @agola11  @agola11 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [X] Callbacks/Tracing
- [x] Async

### Reproduction

1. I tried to use the asynchronous call chain=ConversationalRetrievalChain.from_llm combined with the local knowledge base to find the answer.
2. When chat_history is not passed in chain.acall({"question": query, "chat_history":[]}), it can correctly return the result of the streaming output.
3. When I pass in chat_history, the returned result is new_question. new_question is a processed question and is not the answer I want.

code:


`        db = FAISS.load_local(COMIXGPT_VECTOR, embeddings)

        retriever = db.as_retriever(search_type="similarity_score_threshold",
                                    search_kwargs={"score_threshold": score_threshold,
                                                   "k": VECTOR_SEARCH_TOP_K})
        prompt = PromptTemplate(
            input_variables=["chat_history", "context", "question"],
            template=prompt_template
        )
  chain = ConversationalRetrievalChain.from_llm(
            llm=model,
            chain_type="stuff",
            retriever=retriever,
            #memory=memory,
            return_source_documents=True,
            return_generated_question=True,
            combine_docs_chain_kwargs={'prompt': prompt},
            condense_question_llm=model,
            verbose=True
        )

  task = asyncio.create_task(wrap_done(
            chain.acall({"question": query, "chat_history":chat_history}),
            callback.done),
        )


  if stream:
            async for token in callback.aiter():
                # Use server-sent-events to stream the response
                yield json.dumps({"answer": token}, ensure_ascii=False)
            yield json.dumps({"docs": source_documents}, ensure_ascii=False)
        else:
            answer = ""
            async for token in callback.aiter():
                answer += token
            yield json.dumps({"answer": answer,
                              "docs": source_documents},
                             ensure_ascii=False)
        await task

  return StreamingResponse(knowledge_base_chat_iterator(query=query,
                                                          top_k=top_k,
                                                          history=history,
                                                          chat_history=chat_history,
                                                          model_name=model_name,
                                                          prompt_name=prompt_name),
                             media_type="text/event-stream")`



### Expected behavior

I don't know why it called LLM twice, and then it returned the updated Question, which was not the Assistant I wanted. The call log is as follows:

log:
> Entering new LLMChain chain...
Prompt after formatting:
Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.
Chat History:
Human: hello
Assistant: Hi there! Is there anything I can help you with? Youre welcome, just tell me~
Human: hello hello make friends
Assistant: ok
Follow Up Input: hello!
Standalone question:


> Entering new StuffDocumentsChain chain...


> Entering new LLMChain chain...
Prompt after formatting:
 Chat History:         
Human: hello
Assistant: Hi there! Is there anything I can help you with? Youre welcome, just tell me~
Human: hello hello make friends
Assistant: ok
 Question: How can I make friends?  
Helpful Answer:
2023-12-26 17:27:43,269 - _client.py[line:1758] - INFO: HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
> Finished chain.
> Finished chain.