500 Internal Error when using google genai langchain with agents
Labels: name: auto:bug, name: area: models

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

```python
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain.agents import load_tools, initialize_agent, AgentType, Agent
from dotenv import dotenv_values

config = dotenv_values(".env")
GOOGLE_API_KEY = config["GOOGLE_API_KEY"]

model = ChatGoogleGenerativeAI(
    model="gemini-pro", google_api_key=GOOGLE_API_KEY, temperature=0.8, max_tokens=32000, top_p=0.9, top_k=1)

tools = load_tools(["serpapi", "wikipedia"], llm=model)

tool_names = [tool.name for tool in tools]

FORMAT_INSTRUCTIONS="""
    
    The context includes relevant information you can use to generate a an email response:
    
    {context}
    
    Here is the students email message:
    
    {message}
    
    To use a tools, please use the following format:
    '''
    Thought: Do I need to use a tool? Yes
    Action: the action to take.
    Action Input: the input to the action
    Observation: the result of the action
    Final Answer: the final answer
    '''
    
    Write a professional reply to {sender_name} answering his questions. Do not include anything that is not relevant to Email sent by {sender_name}.
        
    Always represent ... and address the student by his name.
    
    If the context does not contain relevant information, try using the tools to search for answers to write the response.
    
    Don't try searching anything in the context. Only use it as source of information to write the response.
    
    If you don't find anything relevent in your search or in the context, just write the response with your best guess.

    """
    
PREFIX = '''You are an email assistant with more than 10 years of experience working at ....'''
    
SUFFIX='''
    Begin!
    
    Instructions: {input}
    {agent_scratchpad}
    '''
    
prompt = PromptTemplate(
        template=FORMAT_INSTRUCTIONS,
        input_variables=["context", "sender_name", "message"],
    )
    
agent = initialize_agent(
        tools,
        llm=model,
        agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
        verbose=True,
        handle_parsing_errors=True,
        agent_kwargs={
            "suffix": SUFFIX,
            "prefix": PREFIX
        }
    )
    
result = agent(prompt.format(context=retriever, sender_name=sender_name, message=message))
    
print(result["output"])
```




### Description

I am trying to use the langchain-google-genai library to build an agent to answer emails

I expert to see a successful agent execution which I saw from the start

However, I am getting an error: 

```
Retrying langchain_google_genai.chat_models._chat_with_retry.<locals>._chat_with_retry in 2.0 seconds as it raised InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting.
Retrying langchain_google_genai.chat_models._chat_with_retry.<locals>._chat_with_retry in 4.0 seconds as it raised InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting.
Retrying langchain_google_genai.chat_models._chat_with_retry.<locals>._chat_with_retry in 8.0 seconds as it raised InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting.
Retrying langchain_google_genai.chat_models._chat_with_retry.<locals>._chat_with_retry in 16.0 seconds as it raised InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting.
```

### System Info

```
langchain==0.1.0
langchain-community==0.0.12  
langchain-core==0.1.10       
langchain-google-genai==0.0.6
```

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async