multi_retrieval_qa.py hardwires ChatOpenAI() at the _default_chain
Labels: name: auto:bug, name: area: models

### System Info

System:
langchain   0.0.321
Python 3.10

Hi,

I get an error related to a missing OpenAI token when trying to reuse the code to [dynamically select from multiple retrievers](https://python.langchain.com/docs/use_cases/question_answering/multi_retrieval_qa_router) at the MultiRetrievalQAChain initialization step,

```
multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm, # Llama2 served by vLLM via VLLMOpenAI
    retriever_infos=retrievers_info,
    verbose=True)
```

I get an error because I'm not using OpenAI LLMs and the LangChain code (multi_retrieval_qa.py) hard wires ChatOpenAI() as the LLM  for the _default_chain. 

```
_default_chain = ConversationChain(
                llm=ChatOpenAI(), prompt=prompt, input_key="query", output_key="result"
            )
```

I think you need to assign the llm variable to the llm provided when initializing the class.





### Who can help?

_No response_

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Including the Python code and the PDF files that get loaded by the retrievers.
```
#%% 
import torch
from langchain.llms import VLLMOpenAI
from langchain.document_loaders import PyPDFLoader

# Import for retrieval-augmented generation RAG
from langchain import hub
from langchain.chains import RetrievalQA, MultiRetrievalQAChain
from langchain.vectorstores import Chroma
from langchain.text_splitter import SentenceTransformersTokenTextSplitter
from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
#%% 
# URL for the vLLM service
INFERENCE_SRV_URL = "http://localhost:8000/v1"

def setup_chat_llm(vllm_url, max_tokens=512, temperature=0.2):
    """ 
    Intializes the vLLM service object. 
     
    :param vllm_url: vLLM service URL 
    :param max_tokens: Max number of tokens to get generated by the LLM 
    :param temperature: Temperature of the generation process 
    :return: The vLLM service object 
    """
    chat = VLLMOpenAI(
        openai_api_key="EMPTY",
        openai_api_base=vllm_url,
        model_name="meta-llama/Llama-2-7b-chat-hf",
        temperature=temperature,
        max_tokens=max_tokens,
    )
    return chat
#%% 
# Initialize LLM service
llm = setup_chat_llm(vllm_url=INFERENCE_SRV_URL)
#%% 
%%time
# Set up the embedding encoder (Sentence Transformers) and vector store 
model_name = "all-mpnet-base-v2"
model_kwargs = {'device': 'cuda' if torch.cuda.is_available() else 'cpu'}
encode_kwargs = {'normalize_embeddings': False}
embeddings = SentenceTransformerEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)

# Set up the document splitter 
text_splitter = SentenceTransformersTokenTextSplitter(chunk_size=500, chunk_overlap=0)

# Load PDF documents
loader = PyPDFLoader(file_path="../data/AI_RMF_Playbook.pdf")
rmf_doc = loader.load()
rmf_splits = text_splitter.split_documents(rmf_doc)
rmf_retriever = Chroma.from_documents(documents=rmf_splits, embedding=embeddings)

loader = PyPDFLoader(file_path="../data/OWASP-Top-10-for-LLM-Applications-v101.pdf")
owasp_doc = loader.load()
owasp_splits = text_splitter.split_documents(owasp_doc)
owasp_retriever = Chroma.from_documents(documents=owasp_splits, embedding=embeddings)


loader = PyPDFLoader(file_path="../data/Threat Modeling LLM Applications - AI Village.pdf")
ai_village_doc = loader.load()
ai_village_splits = text_splitter.split_documents(ai_village_doc)
ai_village_retriever = Chroma.from_documents(documents=ai_village_splits, embedding=embeddings)
#%% 
retrievers_info = [
    {
        "name": "NIST AI Risk Management Framework",
        "description": "Guidelines for organizations and people to manage risks associated with the use of AI ",
        "retriever": rmf_retriever.as_retriever()
    },
    {
        "name": "OWASP Top 10 for LLM Applications",
        "description": "Provides practical, actionable, and concise security guidance to navigate the complex and evolving terrain of LLM security",
        "retriever": owasp_retriever.as_retriever()
    },
    {
        "name": "Threat Modeling LLM Applications",
        "description": "A high-level example from Gavin Klondike on how to build a threat model for LLM applications",
        "retriever": ai_village_retriever.as_retriever()
    }
]
#%% 
# Import default LLama RAG prompt
prompt = hub.pull("rlm/rag-prompt-llama")
print(prompt.dict()['messages'][0]['prompt']['template'])
#%% 
multi_retriever_chain = MultiRetrievalQAChain.from_retrievers(
    llm=llm,
    retriever_infos=retrievers_info,
    #default_retriever=owasp_retriever.as_retriever(),
    #default_prompt=prompt,
    #chain_type_kwargs={"prompt": prompt},
    verbose=True)
#%% 
question = "What is prompt injection?"
result = multi_retriever
```



[AI_RMF_Playbook.pdf](https://github.com/langchain-ai/langchain/files/13110652/AI_RMF_Playbook.pdf)
[OWASP-Top-10-for-LLM-Applications-v101.pdf](https://github.com/langchain-ai/langchain/files/13110653/OWASP-Top-10-for-LLM-Applications-v101.pdf)
[Threat Modeling LLM Applications - AI Village.pdf](https://github.com/langchain-ai/langchain/files/13110654/Threat.Modeling.LLM.Applications.-.AI.Village.pdf)


### Expected behavior

I was expecting to get results equivalent to what is shown at LangChain's documentation  [dynamically select from multiple retrievers](https://python.langchain.com/docs/use_cases/question_answering/multi_retrieval_qa_router)