LLMs start replying in other languages
Labels: name: auto:bug, name: area: models, name: integration: pinecone

### System Info

Currently, I am using OpenAI LLM and Gemini Pro all being used my LangChain. I am also using Google's embedding-001 model and Cohere base model (tested each embedding and both either reply back in english first then another language or straight to another language).

Here is my prompt template:
<code>
def doc_question_prompt_template():
        template = """
        You are a helpful assistant that has the ability to answer all users questions to the best of your ability.
        Your answers should come from the context you are provided. Provide an answer with detail and not short answers.
        Your only response should be in the English langeuage.
        Context:
        {context}
        User: {question}
        """
        return PromptTemplate(
            input_variables=["question"], 
            template=template
        )


       def doc_question_command(body, conversation_contexts):
        llmlibrary = LLMLibrary()        
        channel_id = body['channel_id']
        user_id = body['user_id']
        context_key = f"{channel_id}-{user_id}"
        prompt = ChatPromptTemplate.doc_question_prompt_template()
        if context_key not in conversation_contexts:
            conversation_contexts[context_key] = {
                "memory": ConversationBufferMemory(memory_key="chat_history", output_key="answer", return_messages=True, max_token_limit=1024),
                "history": "",
            }
        user_memory = conversation_contexts[context_key]["memory"]
        question = body['text']
        conversation = llmlibrary.doc_question(user_memory, prompt, question)
        #print(f"Conversation: {conversation}")
        return question, conversation

    def doc_question(self, user_memory, prompt, question)

        llm = ChatGoogleGenerativeAI(model="gemini-pro",temperature=0.0, convert_system_message_to_human=True)
        vectordb = self.vectorstore.get_vectordb()
        print(f"Vector DB: {vectordb}\n")
        retriever = vectordb.as_retriever(
                    search_type="similarity_score_threshold",
                    search_kwargs={'score_threshold': 0.8}
            )
        docs = retriever.get_relevant_documents(question)
        print(f"Docs: {docs}\n")
        print(f"Initiating chat conversation memory\n")
        #print(f"Conversation Memory: {memory}\n")
        conversation_chain= ConversationalRetrievalChain.from_llm(
              llm,
              retriever=retriever,
              memory=user_memory,
              combine_docs_chain_kwargs={'prompt': prompt},
              return_source_documents=True,
              verbose=False,
        )
        #print(f"Conversation chain: {conversation_chain}\n")
        return conversation_chain


@app.command("/doc_question")
def handle_doc_question_command(ack, body, say):
    # Acknowledge the command request
    ack()
    print(body)
    say(f"🤨 {body['text']}")
    question, conversation  = ChatHandler.doc_question_command(body, conversation_contexts)
    response = conversation({'question': question})
    print(f"(INFO) Doc Question Response: {response} {time.time()}")
    print(f"(INFO) Doc Question Response answer: {response['answer']} {time.time()}")
    say(f"🤖 {response['answer']}")
</code>

Logs:
[output.txt](https://github.com/langchain-ai/langchain/files/13732990/output.txt)


![Screenshot 2023-12-20 at 2 00 14 PM](https://github.com/langchain-ai/langchain/assets/37451503/e12169a0-4e9b-4d31-9c4f-40e650c2aee4)


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [X] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

1. user sends a message through Slack 
2. The message is received by @app.command("/doc_question)
3. ChatHandler.doc_question_command gets called passing the body and conversation_contexts
4. doc_question_command gets information about the message that was sent and gets the doc_question_prompt_template from ChatPromptTemplate module
5. conversation_contexts gets a context key of memory and history
6. llmlibrary.doc question is then called passing user_member, prompt, question
7. the doc_question function uses the ChatGoogleGenerativeAI module and gets the vectordb which is Pinecone
8. uses the ConversationRetrievalChain.from_llm and passes it back to the handler and the handler passes question and conversation back to @app.command("/doc_question")
9. The question is then submitted to the LLM and the response is spat out within Slack (sometimes english, Spanish, or other)

### Expected behavior

Only reply in english