ConversationalRetrievalChain with ChatGoogleGenerativeAI performs poorly
Labels: name: auto:bug, name: area: models

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

I have omitted as much as possible irrelevant portions:

```python

def get_text(docs):
    if not docs:
        return None
    all_text = ""
    for doc in docs:
        reader = PdfReader(doc)
        for page in reader.pages:
            all_text += page.extract_text()
    return all_text

def get_chunks(text):
    splitter = CharacterTextSplitter(
        separator='\n',
        chunk_size=1000,
        chunk_overlap=200,
    )
    chunks = splitter.split_text(text)
    return chunks

def get_vectorstore(chunks, model='models/embedding-001'):
    client = qdrant_client.QdrantClient(
        os.environ['QDRANT_HOST'],
        api_key=os.environ["QDRANT_API_KEY"],
    )
    vectors_config = qdrant_client.http.models.VectorParams(
        size=768,
        distance=qdrant_client.http.models.Distance.COSINE
    )
    client.recreate_collection(
        collection_name=os.environ["QDRANT_COLLECTION"],
        vectors_config=vectors_config
    )
    embeddings = GoogleGenerativeAIEmbeddings(model=model)
    vectorstore = Qdrant(
        client=client, collection_name=os.environ["QDRANT_COLLECTION"],
        embeddings=embeddings,
    )
    vectorstore.add_texts(chunks)
    return vectorstore

def get_conversation_chain(vectorstore):
    llm = ChatGoogleGenerativeAI(model="gemini-pro", convert_system_message_to_human=True, temperature=0)
    memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
    chain = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=vectorstore.as_retriever(),
        memory=memory,
    )
    return chain

docs = st.file_uploader("Upload your documents:", accept_multiple_files=True, type=["pdf", "txt", "csv"])
user_input = "Some question"
if st.button("Proess"):
    with st.spinner("Processing..."):
        text = get_text(docs)
        chunks = get_chunks(text)
        vectorstore = get_vectorstore(chunks)
        chain = get_conversation_chain(vectorstore)
	    response = chain({'question': user_input})
```

### Description

I'm trying to use ``ConversationalRetrievalChain`` with the ``ChatGoogleGenerativeAI`` integration. This seems to perform rather poorly in several scenarios involving PDF documents, with 2 issues frequently arising:
1. The model seems to default quite often to "I don't know" or "I cannot find the information in the context provided", despite the fact that the context seems to have the requested information within (so it's unlikely to be a problem with the vector DB).
2. Some of the LangChain-built system prompts (converted to human messages via the recently-added hyperparameter) which ask the model to translate something to the "original language" completely confuse it, making it output its response in a random language. I have tried using the prompt created by LangChain with the Google API directly, and the response is also bad. Example prompt created by LangChain from my conversation: ``"Human: Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question, in its original language.\n\nChat History:\n\nHuman: what does the third amendment say?\nAssistant: I do not have access to the context necessary to answer that question.\nFollow Up Input: what does the second amendment say?\nStandalone question:"``. This outputs a message in a random language on both the Google Gemini API and LangChain.

Example document: <a href="https://www.archives.gov/files/legislative/resources/education/bill-of-rights/images/handout-3.pdf"> Bill of rights</a>. Example questions: (see example prompt above).

### System Info

langchain==0.0.351
langchain-community==0.0.4
langchain-core==0.1.1
langchain-experimental==0.0.47
langchain-google-genai==0.0.5

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async