AgentTokenBufferMemory does not remove old messages, leading to the "context_length_exceeded" error from OpenAI.
Labels: name: area: memory, name: auto:bug, name: integration: redis

### System Info

Hello Langchain team,

I have encountered an error while using `AgentTokenBufferMemory` and `RedisChatMessageHistory`. The problem occurs because the buffer is not removing old messages when new ones are added. This causes an issue with OpenAI as the context window exceeds the token limit. Upon investigation, I found that the issue is in the `save_context()` method of the `AgentTokenBufferMemory` class.

```python
def save_context(self, inputs: Dict[str, Any], outputs: Dict[str, Any]) -> None:
      """Save context from this conversation to buffer. Pruned."""
      input_str, output_str = self._get_input_output(inputs, outputs)
      self.chat_memory.add_user_message(input_str)
      steps = format_to_openai_function_messages(outputs[self.intermediate_steps_key])
      for msg in steps:
          self.chat_memory.add_message(msg)
      self.chat_memory.add_ai_message(output_str)
      # Prune buffer if it exceeds max token limit
      buffer = self.chat_memory.messages
      curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)
      if curr_buffer_length > self.max_token_limit:
          while curr_buffer_length > self.max_token_limit:
              buffer.pop(0)
              curr_buffer_length = self.llm.get_num_tokens_from_messages(buffer)
```
In the `save_context()` method, the input and output messages are retrieved and added to the chat memory. The intermediate steps are then converted to OpenAI function messages and added to the chat memory as well. If the maximum token limit is exceeded, the buffer is pruned by removing the oldest messages until the current buffer length is below the limit.

However, the problem arises from the line `buffer.pop(0)`, which removes a message from a local variable rather than removing it from the Redis list. Since the list is not pruned, the agent becomes stuck and fails permanently. 

Here is the error I am receiving:
```json
{'message': "This model's maximum context length is 16385 tokens. However, your messages resulted in 16602 tokens (16529 in the messages, 73 in the functions). Please reduce the length of the messages or functions.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}
```

Here is the information from LangSmith -> Metadata -> Runtime:
<img width="422" alt="image" src="https://github.com/langchain-ai/langchain/assets/101429097/eb3105f2-fd18-4527-b4b2-f89e2e87bdb7">







### Who can help?

@baskarya

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```python
import uuid

from langchain.memory.chat_message_histories import RedisChatMessageHistory
from langchain.agents.openai_functions_agent.agent_token_buffer_memory import AgentTokenBufferMemory
from langchain.schema import SystemMessage
from langchain.agents import OpenAIFunctionsAgent, AgentExecutor
from langchain.prompts import MessagesPlaceholder

prompt: SystemMessage = None # my prompt
chat_message_history = RedisChatMessageHistory(
    session_id=uuid.uuid4(),
    url="redis://localhost:6379",
    key_prefix="my_feature_",
    ttl=3600,
)
memory = AgentTokenBufferMemory(
        chat_memory=chat_message_history,
        memory_key="chat_history",
        return_messages=True,
        llm=llm,
        max_token_limit=10000,
 )
prompt = OpenAIFunctionsAgent.create_prompt(
        system_message=prompt, extra_prompt_messages=[MessagesPlaceholder(variable_name="chat_history")]
 )
agent = OpenAIFunctionsAgent(llm=llm, tools=tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True, return_intermediate_steps=True)
agent_response = agent_executor.invoke({"input": user_message})
```

### Expected behavior

Once the current messages have been saved, if the tokens exceed the maximum token limit, the `AgentTokenBufferMemory` should remove old messages from the Redis list (RPOP) until they are below the maximum token limit.