Issue: Not getting desired output while implementing memory
Labels: name: area: memory, name: auto:bug, name: integration: chroma

### Issue you'd like to raise.

`below is my code for generating custom prompt which takes context and user query and we pass it into model:
def generate_custom_prompt(new_project_qa,query,name,not_uuid):  
    check = query.lower() 
    result = new_project_qa(query)
    relevant_document = result['source_documents']
    context_text="\n\n---\n\n".join([doc.page_content for doc in relevant_document])
    # print(context_text,"context_text")
    user_experience_inst = UserExperience.objects.get(not_uuid=not_uuid)
    greetings = ['hi', 'hello', 'hey', 'hui', 'hiiii', 'hii', 'hiii', 'heyyy']
    if check in greetings:
        custom_prompt_template = f"""
        Just simply reply with "Hello {name}! How can I assist you today?"
        """
    elif check not in greetings and user_experience_inst.custom_prompt:
        custom_prompt_template = f"""Answer the question based only on following context: ```{context_text} ```
        You are a chatbot designed to provide answers to User's Questions:```{check}```, delimited by triple backticks.
        Generate your answer to match the user's requirements: {user_experience_inst.custom_prompt}

        If you encounter a question for which you don't know the answer, please respond with 'I don't know' and refrain from making up an answer.
        - Before saying 'I don't know,' please re-verify your vector store to ensure the answer is not present in the database.

        Remember, your goal is to assist the user in the best way possible. If the question is unclear or ambiguous, feel free to ask for clarification.

        User's Question: ```{check}```
        AI Answer:"""
    else:
        # Create the custom prompt template
        custom_prompt_template = f"""Generate your response exclusively from the provided context: {{context_text}}. You function as a chatbot specializing in delivering detailed answers to the User's Question: ```{{check}} ```, enclosed within triple backticks.
            Generate your answer in points in the following format:
            1. Point no 1
                1.1 Its subpoint in details
                1.2 More information if needed.
            2. Point no 2
                2.1 Its subpoint in details
                2.2 More information if needed.
            …
            N. Another main point.

            If you encounter a question for which you don't know the answer based on the predefined points, please respond with 'I don't know' and refrain from making up an answer.
            However, if the answer is not present in the predefined points, then Provide comprehensive information related to the user's query.
            Remember, your goal is to assist the user in the best way possible. If the question is unclear or ambiguous, you can ask for clarification.

            User's Question: ```{{check}} ```  
            AI Answer:"""


    # Create the PromptTemplate
    custom_prompt = ChatPromptTemplate.from_template(custom_prompt_template)
    formatted_prompt = custom_prompt.format(context_text=context_text, check=check)
    return formatted_prompt
below is my conversation chain where i am inplementing memory
def retreival_qa_chain(chroma_db_path):
    embedding = OpenAIEmbeddings()
    vectordb = Chroma(persist_directory=chroma_db_path, embedding_function=embedding)
    llm = ChatOpenAI(temperature=0.1)
    memory = ConversationBufferMemory(llm=llm,output_key='answer',memory_key='chat_history',return_messages=True)
    retriever = vectordb.as_retriever(search_kwargs={"k": 2})
    qa = ConversationalRetrievalChain.from_llm(llm=llm,memory=memory,chain_type="stuff",retriever=retriever,return_source_documents=True,get_chat_history=lambda h : h,verbose=True)
    # qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever,return_source_documents=True)
    return qa
    but I am not getting desired output as expected   `

### Suggestion:

_No response_