Error "expected string or buffer" when trying to use RetrievalQAWithSourcesChain as Tool for Agent
Labels: name: area: agent, name: area: vector store, name: auto:bug, name: area: models

### System Info

I am developing a chatbot (surprise!) for our company, and I have previously been able to execute the following code used by Agent:

```python
chain = RetrievalQAWithSourcesChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    max_tokens_limit=3000,
    reduce_k_below_max_tokens=True
)
tools.append(Tool.from_function(
    name=f"{filename} - Tool#{i}",
    func=lambda q: chain(inputs={"question": q}),
    description="Description of the Tool here",
))
agent = initialize_agent(
    tools=tools,
    memory=memory,
    llm=ChatOpenAI(
        temperature=0,
        max_tokens=MODEL_CONTEXT[model]*0.1,
        max_retries=2,
        model=model
    ),
    agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)
...
agent.run(query)
```

Despite having had this logic work in the past, when I tried to execute the Agent the other day it just didn't work...
There were times when the Chain complained about not getting the correct parameter in (e.g. "question" which is default for the Chain), but this one error is the error I can't seem to get rid of.

What am I doing wrong?

Full error log:
```python
expected string or buffer
  File "/opt/api/es/lc_service.py", line 686, in answer_agent
    results = agent.run(query)
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 487, in run
    return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 292, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 286, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1122, in _call
    next_step_output = self._take_next_step(
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 977, in _take_next_step
    observation = tool.run(
  File "/usr/local/lib/python3.10/site-packages/langchain/tools/base.py", line 360, in run
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain/tools/base.py", line 332, in run
    self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain/tools/base.py", line 511, in _run
    else self.func(*args, **kwargs)
  File "/opt/api/es/lc_service.py", line 1089, in <lambda>
    func=lambda q: chain(inputs={"question": q}),
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 292, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 286, in __call__
    self._call(inputs, run_manager=run_manager)
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/qa_with_sources/base.py", line 151, in _call
    docs = self._get_docs(inputs, run_manager=_run_manager)
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/qa_with_sources/retrieval.py", line 50, in _get_docs
    docs = self.retriever.get_relevant_documents(
  File "/usr/local/lib/python3.10/site-packages/langchain/schema/retriever.py", line 212, in get_relevant_documents
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain/schema/retriever.py", line 205, in get_relevant_documents
    result = self._get_relevant_documents(
  File "/usr/local/lib/python3.10/site-packages/langchain/vectorstores/base.py", line 562, in _get_relevant_documents
    docs = self.vectorstore.similarity_search(query, **self.search_kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain/vectorstores/elasticsearch.py", line 592, in similarity_search
    results = self._search(query=query, k=k, filter=filter, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/langchain/vectorstores/elasticsearch.py", line 663, in _search
    query_vector = self.embedding.embed_query(query)
  File "/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py", line 518, in embed_query
    return self.embed_documents([text])[0]
  File "/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py", line 490, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=self.deployment)
  File "/usr/local/lib/python3.10/site-packages/langchain/embeddings/openai.py", line 351, in _get_len_safe_embeddings
    token = encoding.encode(
  File "/usr/local/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
```

### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [X] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Steps to reproduce the behaviour:

Follow the rough steps above:
1. Create Agent from LLM
2. Use RetrievalQAWithSourcesChain as Tool
3. Try to get answer back from Agent

### Expected behavior

I expect to have an answer back... any is good at this point 😭