Vertex ChatVertexAI()  doesn't support initialize_agent() as OutputParserException error

### System Info

google-cloud-aiplatform==1.25.0
langchain==0.0.180
python 3.11

### Who can help?

@dev2049
@Jflick58
@hwchase17 

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [X] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

  question1 = "I am axa, I'm a 2 months old baby.."
  question2 = "I like eating 🍌 🍉 🫐 but dislike 🥑"
  question3 = "what is my name?"
  question4 = "Do i disklike 🍌?"

  agent_chain = initialize_agent(
    agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION,
    tools=[],
    llm=llm,
    verbose=True,
    max_iterations=3,
    memory=ConversationBufferMemory(
      memory_key="chat_history", return_messages=True),
  )

  agent_chain.run(input=question1)
  agent_chain.run(input=question2)
  agent_chain.run(input=question3)
  agent_chain.run(input=question4)


  File "/Users/axa/workspace/h/default/genai_learning/post/api/app/routes/v1/quiz_chat.py", line 271, in ask
    agent_chain.run(input=question1)
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/chains/base.py", line 239, in run
    return self(kwargs, callbacks=callbacks)[self.output_keys[0]]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/chains/base.py", line 140, in __call__
    raise e
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/chains/base.py", line 134, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 951, in _call
    next_step_output = self._take_next_step(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 773, in _take_next_step
    raise e
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 762, in _take_next_step
    output = self.agent.plan(
             ^^^^^^^^^^^^^^^^
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 444, in plan
    return self.output_parser.parse(full_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/axa/workspace/h/default/genai_learning/post/.venv/lib/python3.11/site-packages/langchain/agents/conversational/output_parser.py", line 23, in parse
    raise OutputParserException(f"Could not parse LLM output: `{text}`")
langchain.schema.OutputParserException: **Could not parse LLM output: `Hi Axa, it's nice to meet you! I'm Bard, a large language model, also known as a conversational AI or chatbot trained to be informative and comprehensive. I am trained on a massive amount of text data, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions. For example, I can provide summaries of factual topics or create stories.**


### Expected behavior

When I used same code but ChatOpenAI() it worked perfectly.


> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? No

AI: Hello Axa! As an AI language model, I'm not able to see or interact with you physically, but I'm here to assist you with any questions or topics you might have. How can I assist you today?

> Finished chain.


> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? No

AI: It's great to hear that you enjoy eating bananas, watermelons, and blueberries! However, it's understandable that you might not like avocados. Everyone has their own preferences when it comes to food. Is there anything else you would like to discuss or ask about?

> Finished chain.


> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? No
AI: Your name is Axa, as you mentioned earlier.

> Finished chain.


> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? No
AI: You did not mention that you dislike bananas, so I cannot say for sure. However, based on your previous message, it seems that you enjoy eating bananas.

> Finished chain.
INFO:     127.0.0.1:57044 - "POST /api/v1/quiz/ask HTTP/1.1" 200 OK