create_conversational_retrieval_agent does not work with fallbacks
Labels: name: area: agent, name: auto:bug, name: stale, name: area: models

### System Info

Langhchain version: 0.0.296
Python version: 3.9.7
Platform: Windows 10



### Who can help?

I need to implement a fallback for my RAG application based on Langchain conversational retrievers because my application  needs to use GPT 4 only for RAG and GPT 3.5 for generic questions (eg.: ChatGPT).  A way to implement this behaviour is by means of fallbacks, because infortunately create_conversational_retrieval_agent() enforces the same LLM for all the tools.  So, I wrote the following code:

```
short_llm = ChatOpenAI(
    deployment_id=DEPLOYMENT_NAME_GPT_3_5_4K,
    engine=DEPLOYMENT_NAME_GPT_3_5_4K,
    model_name=model_gpt_3_5_4k,
    temperature=0,
    openai_api_base=BASE_URL,
    openai_api_key=API_KEY,
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()]
)

long_llm = ChatOpenAI(
    deployment_id=DEPLOYMENT_NAME_GPT_4_8K,
    engine=DEPLOYMENT_NAME_GPT_4_8K,
    model_name=model_gpt_4_8k,
    temperature=0,
    openai_api_base=BASE_URL,
    openai_api_key=API_KEY,
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()]
)

chat_llm = short_llm.with_fallbacks([long_llm])
```

And to create the agent:

`agent_executor = create_conversational_retrieval_agent(chat_llm, tools, verbose=True, sytem_message=system_prompt, remember_intermediate_steps=False)`

But the following error occurs:

```
C:\ProgramData\Anaconda3\lib\site-packages\langchain\agents\agent_toolkits\conversational_retrieval\openai_functions.py in create_conversational_retrieval_agent(llm, tools, remember_intermediate_steps, memory_key, system_message, verbose, max_token_limit, **kwargs)
     58 
     59     if not isinstance(llm, ChatOpenAI):
---> 60         raise ValueError("Only supported with ChatOpenAI models.")
     61     if remember_intermediate_steps:
     62         memory: BaseMemory = AgentTokenBufferMemory(

ValueError: Only supported with ChatOpenAI models.
```

In fact, chat_llm has the following type: langchain.schema.runnable.base.RunnableWithFallbacks.  It apparently does not inherit ChatOpenAI .

Is there any workaround?  My requirement is to use long LLM for RAG/retriever tools and a short one for normal conversation with ChatGPT 3.5


### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

```
short_llm = ChatOpenAI(
    deployment_id=DEPLOYMENT_NAME_GPT_3_5_4K,
    engine=DEPLOYMENT_NAME_GPT_3_5_4K,
    model_name=model_gpt_3_5_4k,
    temperature=0,
    openai_api_base=BASE_URL,
    openai_api_key=API_KEY,
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()]
)

long_llm = ChatOpenAI(
    deployment_id=DEPLOYMENT_NAME_GPT_4_8K,
    engine=DEPLOYMENT_NAME_GPT_4_8K,
    model_name=model_gpt_4_8k,
    temperature=0,
    openai_api_base=BASE_URL,
    openai_api_key=API_KEY,
    streaming=True,
    callbacks=[StreamingStdOutCallbackHandler()]
)

chat_llm = short_llm.with_fallbacks([long_llm])
```

And to create the agent:

`agent_executor = create_conversational_retrieval_agent(chat_llm, tools, verbose=True, sytem_message=system_prompt, remember_intermediate_steps=False)`

### Expected behavior

Conversational retrieval agents should be able to work with fallbacks.