Getting "ValueError: Unsupported chat history format:" while using ConversationalRetrievalChain with memory type VectorStoreRetrieverMemory
Labels: name: bug

### System Info

LangChain-0.0.207, Windows, Python-3.9.16

Memory (VectorStoreRetrieverMemory) Settings:

dimension = 768
index = faiss.IndexFlatL2(dimension)

embeddings = HuggingFaceEmbeddings()

vectorstore = FAISS(embeddings.embed_query, index, InMemoryDocstore({}), {})
retriever = vectorstore.as_retriever(search_kwargs=dict(k=3))
memory = VectorStoreRetrieverMemory(
    memory_key="chat_history",
    return_docs=True,
    retriever=retriever,
    return_messages=True,
)

ConversationalRetrievalChain Settings:

_template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\
Make sure to avoid using any unclear pronouns.

Chat History:
{chat_history}
(You do not need to use these pieces of information if not relevant)
Follow Up Input: {question}
Standalone question:"""
CONDENSE_QUESTION_PROMPT = PromptTemplate(
    input_variables=["chat_history", "question"], template=_template
)

question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_with_sources_chain(
    llm, chain_type="refine"
)  # stuff  map_reduce  refine

qa = ConversationalRetrievalChain(
    retriever=chroma.as_retriever(search_kwargs=dict(k=5)),
    memory=memory,
    combine_docs_chain=doc_chain,
    question_generator=question_generator,
    return_source_documents=True,
    # verbose=True,
)

responses = qa({"question": user_input})

ISSUE: At first cal, it is giving results. BUT in second call, it is throwing an error as follows:

ValueError: Unsupported chat history format: <class 'langchain.schema.Document'>. Full chat history:

What am I doing wrong here?

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. Memory (VectorStoreRetrieverMemory) Settings:

dimension = 768
index = faiss.IndexFlatL2(dimension)

embeddings = HuggingFaceEmbeddings()

vectorstore = FAISS(embeddings.embed_query, index, InMemoryDocstore({}), {})
retriever = vectorstore.as_retriever(search_kwargs=dict(k=3))
memory = VectorStoreRetrieverMemory(
    memory_key="chat_history",
    return_docs=True,
    retriever=retriever,
    return_messages=True,
)

2. ConversationalRetrievalChain Settings:

_template = """Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\
Make sure to avoid using any unclear pronouns.

Chat History:
{chat_history}
(You do not need to use these pieces of information if not relevant)
Follow Up Input: {question}
Standalone question:"""
CONDENSE_QUESTION_PROMPT = PromptTemplate(
    input_variables=["chat_history", "question"], template=_template
)

question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_with_sources_chain(
    llm, chain_type="refine"
)  # stuff  map_reduce  refine

qa = ConversationalRetrievalChain(
    retriever=chroma.as_retriever(search_kwargs=dict(k=5)),
    memory=memory,
    combine_docs_chain=doc_chain,
    question_generator=question_generator,
    return_source_documents=True,
    # verbose=True,
)

responses = qa({"question": user_input})

ISSUE: At first cal, it is giving results. BUT in second call, it is throwing an error as follows:

ValueError: Unsupported chat history format: <class 'langchain.schema.Document'>. Full chat history:

What am I doing wrong here?

### Expected behavior

Chat history should be injected in chain