GPTIndexChatMemory doesn't work as expected with langchain and the agent doesn't use the chat history
Labels: name: auto:bug

### System Info

langchain version: '0.0.230'
llama-index version: '0.7.4'
python: 3.10.11

### Who can help?

@hwchase17 
@agola11 
@eyurtsev 

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [X] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

**When I'm trying to use GPTIndexChatMemory to embed my conversation and store the whole of it, this feature (memory) doesn't work anymore, and this is my code, also I want to save the memory in a folder with its embeddings and I can't**

```
from llama_index.langchain_helpers.memory_wrapper import GPTIndexChatMemory, GPTIndexMemory
from langchain.chat_models import ChatOpenAI
from langchain.agents import AgentType
from llama_index import  ServiceContext
from llama_index import GPTListIndex
from langchain.embeddings import OpenAIEmbeddings

llm = ChatOpenAI(temperature=0)
embed_model = LangchainEmbedding(OpenAIEmbeddings())
service_context = ServiceContext.from_defaults(embed_model=embed_model)
index = GPTListIndex([], service_context=service_context)
memory = GPTIndexChatMemory(
    index=index,
    memory_key="chat_history",
    query_kwargs={"response_mode": "compact", "service_context":service_context},
    input_key="input",
    return_messages=True,
    return_source= True
)

agent_executor = initialize_agent(
    [], llm, verbose = True,agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, memory=memory, handle_parsing_errors="Check your output and make sure it conforms!"
)

agent_executor.agent.llm_chain.prompt.template = """Assistant is a large language model trained by OpenAI.

Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.

Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.

Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.


please use the following format:


AI: [your response here]


Begin!

Previous conversation history:
{chat_history}

New input: {input}
{agent_scratchpad}"""


print(agent_executor.run("my name is zeyad"))
```

this would be the output for the first print statement.

**AI: Hello Zeyad! How can I assist you today?**

`print("Do you know my name?")`

**this would be the output for the first print statement (unexcepted output), one week ago it was working fine without any problems.**

**AI: As an AI language model, I don't have access to personal information unless you provide it to me. Therefore, I don't know your name unless you tell me. Is there anything specific you would like assistance with?**
 



### Expected behavior

the expected output for the second statement must be:

AI: Yes, you told me before that your name is Zeyad.


**I really appreciate any help you can provide.**