System message not working with ConversationalRetrievalChain
Labels: name: area: memory, name: auto:bug, name: stale, name: area: models

### System Info

    CHAT_PROMPT = ChatPromptTemplate(
        messages=[
            SystemMessagePromptTemplate.from_template(general_system_template),
            # The `variable_name` here is what must align with memory
            MessagesPlaceholder(variable_name="chat_history"),
            HumanMessagePromptTemplate.from_template("{question}"),
        ]
    )

    memory = ConversationBufferMemory(
        llm=llm, 
        max_token_limit=MAX_TOKENS,
        memory_key="chat_history", # used for the qa which has a "chat_history" key
        # prompt=CHAT_PROMPT, # To give system instructions thorugh the custom prompt
        return_messages=True # otherwise just a str and won't work for chat_history
    )

    # memory.chat_memory.add_message({"role": "system", "content": "You are a surfer. end all setences with 'dude'"})
    memory.chat_memory.add_message(SystemMessage(content="You are a surfer. end all setences with 'dude'"))
    memory.chat_memory.add_user_message("I think you're great")
    memory.chat_memory.add_ai_message("Thank you as I am a surfer")

    qa = ConversationalRetrievalChain.from_llm(
        llm=llm, 
        retriever=retriever, 
        memory=memory,
        # combine_docs_chain_kwargs={
        #     "prompt": CHAT_PROMPT
        # }
    )
    res = qa({"question": msg, "chat_history": memory.chat_memory})

    print("QA", res)
    
    await ctx.send(res['answer'])

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

    CHAT_PROMPT = ChatPromptTemplate(
        messages=[
            SystemMessagePromptTemplate.from_template(general_system_template),
            # The `variable_name` here is what must align with memory
            MessagesPlaceholder(variable_name="chat_history"),
            HumanMessagePromptTemplate.from_template("{question}"),
        ]
    )

    memory = ConversationBufferMemory(
        llm=llm, 
        max_token_limit=MAX_TOKENS,
        memory_key="chat_history", # used for the qa which has a "chat_history" key
        # prompt=CHAT_PROMPT, # To give system instructions thorugh the custom prompt
        return_messages=True # otherwise just a str and won't work for chat_history
    )

    # memory.chat_memory.add_message({"role": "system", "content": "You are a surfer. end all setences with 'dude'"})
    memory.chat_memory.add_message(SystemMessage(content="You are a surfer. end all setences with 'dude'"))
    memory.chat_memory.add_user_message("I think you're great")
    memory.chat_memory.add_ai_message("Thank you as I am a surfer")

    qa = ConversationalRetrievalChain.from_llm(
        llm=llm, 
        retriever=retriever, 
        memory=memory,
        # combine_docs_chain_kwargs={
        #     "prompt": CHAT_PROMPT
        # }
    )
    res = qa({"question": msg, "chat_history": memory.chat_memory})

    print("QA", res)
    
    
    Running this and interacting with this will lead to no acknowledgement of the system message by the LLM    

### Expected behavior

The system message is listened to