VectorStoreRetrieverMemory doesn't work with AgentExecutor
Labels: name: area: memory, name: area: agent, name: auto:bug

### System Info

aiohttp==3.8.4
aiosignal==1.3.1
altair==5.0.1
annotated-types==0.6.0
anyio==3.7.1
asgiref==3.7.2
async-timeout==4.0.2
attrs==23.1.0
backoff==2.2.1
blinker==1.6.2
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
click==8.1.5
clickhouse-connect==0.6.6
coloredlogs==15.0.1
cryptography==41.0.2
dataclasses-json==0.5.9
decorator==5.1.1
deprecation==2.1.0
dnspython==2.4.0
duckdb==0.8.1
ecdsa==0.18.0
et-xmlfile==1.1.0
fastapi==0.104.1
fastapi-pagination==0.12.12
filetype==1.2.0
flatbuffers==23.5.26
frozenlist==1.4.0
gitdb==4.0.10
GitPython==3.1.32
greenlet==2.0.2
h11==0.14.0
hnswlib==0.7.0
httpcore==0.17.3
httptools==0.6.0
humanfriendly==10.0
idna==3.4
importlib-metadata==6.8.0
install==1.3.5
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.3.1
jsonpatch==1.33
jsonpointer==2.4
jsonschema==4.18.3
jsonschema-specifications==2023.6.1
langchain==0.0.335
langsmith==0.0.64
loguru==0.7.0
lxml==4.9.3
lz4==4.3.2
Markdown==3.4.3
markdown-it-py==3.0.0
MarkupSafe==2.1.3
marshmallow==3.19.0
marshmallow-enum==1.5.1
mdurl==0.1.2
monotonic==1.6
mpmath==1.3.0
msg-parser==1.2.0
multidict==6.0.4
mypy-extensions==1.0.0
nltk==3.8.1
numexpr==2.8.4
numpy==1.25.1
olefile==0.46
onnxruntime==1.15.1
openai==0.27.8
openapi-schema-pydantic==1.2.4
openpyxl==3.1.2
overrides==7.3.1
packaging==23.1
pandas==2.0.3
pdf2image==1.16.3
pdfminer.six==20221105
Pillow==9.5.0
pinecone-client==2.2.4
posthog==3.0.1
protobuf==4.23.4
psycopg2-binary==2.9.7
pulsar-client==3.2.0
py-automapper==1.2.3
pyarrow==12.0.1
pyasn1==0.5.0
pycparser==2.21
pycryptodome==3.18.0
pydantic==2.5.0
pydantic-settings==2.1.0
pydantic_core==2.14.1
pydeck==0.8.1b0
Pygments==2.15.1
pymongo==4.6.0
Pympler==1.0.1
pypandoc==1.11
python-dateutil==2.8.2
python-docx==0.8.11
python-dotenv==1.0.0
python-jose==3.3.0
python-keycloak==2.16.6
python-magic==0.4.27
python-pptx==0.6.21
pytz==2023.3
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0
referencing==0.29.1
regex==2023.6.3
requests==2.31.0
requests-toolbelt==1.0.0
rich==13.4.2
rpds-py==0.8.11
rsa==4.9
six==1.16.0
smmap==5.0.0
sniffio==1.3.0
SQLAlchemy==2.0.19
starlette==0.27.0
streamlit==1.24.1
sympy==1.12
tabulate==0.9.0
tenacity==8.2.2
tiktoken==0.4.0
tokenizers==0.13.3
toml==0.10.2
toolz==0.12.0
tornado==6.3.2
tqdm==4.65.0
typing-inspect==0.9.0
typing_extensions==4.8.0
tzdata==2023.3
tzlocal==4.3.1
unstructured==0.8.1
urllib3==2.0.3
uvicorn==0.23.0
uvloop==0.17.0
validators==0.20.0
watchdog==3.0.0
watchfiles==0.19.0
websockets==11.0.3
xlrd==2.0.1
XlsxWriter==3.1.2
yarl==1.9.2
zipp==3.16.2
zstandard==0.21.0


### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
// Omitted LLM and store retriever code

memory = VectorStoreRetrieverMemory(
    retriever=retriever,
    return_messages=True,
)

tool = create_retriever_tool(
      retriever,
      "search_egypt_mythology",
      "Searches and returns documents about egypt mythology",
  )
  tools = [tool]
  
  system_message = SystemMessage(
      content=(
          "Do your best to answer the questions. "
          "Feel free to use any tools available to look up "
          "relevant information, only if necessary"
      )
  )
  
  prompt = OpenAIFunctionsAgent.create_prompt(
      system_message=system_message,
      extra_prompt_messages=[
          MessagesPlaceholder(variable_name="history")
      ],
  )
  
  agent = OpenAIFunctionsAgent(llm=__llm, tools=tools, prompt=prompt)

  chat = AgentExecutor(
      agent=agent,
      tools=tools,
      memory=memory,
      verbose=True,
      return_intermediate_steps=True,
  )

result = chat({"input":"my question"})
answer = result['output']
```

**The error is:
ValueError: variable history should be a list of base messages, got**


The agent works with the mongodb as chat history though, so it should have worked with the vector memory retriever:

```
mongo_history = MongoDBChatMessageHistory(
        connection_string=settings.mongo_connection_str,
        session_id=__get_chat_id(user_uuid),
        database_name = settings.mongo_db_name,
        collection_name = 'chat_history',
    )
    memory = ConversationBufferMemory(
        chat_memory=mongo_history,
        memory_key='history',
        input_key="input",
        output_key="output",
        return_messages=True,
    )
```

### Expected behavior

Vector retrieval memory should have worked like the MongoDBChatMessageHistory memory.
There is some omission about this issue in the documentation.