Document Comparison, OPENAI_FUNCTIONS does not always look into the tools
Labels: name: area: agent, name: auto:bug

### System Info

Having basicaly this demo code and the issue is that it's not working reliably!
Sometimes OPENAI_FUNCTIONS just doesn't look into tools and telling that he can't find the documents.

Seems like this happens because OpenAI decides on it's own if he rather use tools or replies on his own knowledge.

I also don't want to fill the memory with all the data every time I am starting a new conversation, this does not work with huge data files.

Any other way to solve this?



### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [X] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

      from pydantic import BaseModel, Field
      
      from langchain.chat_models import ChatOpenAI
      from langchain.agents import Tool
      from langchain.embeddings.openai import OpenAIEmbeddings
      from langchain.text_splitter import CharacterTextSplitter
      from langchain.vectorstores import FAISS
      from langchain.document_loaders import PyPDFLoader
      from langchain.chains import RetrievalQA
      
      class DocumentInput(BaseModel):
          question: str = Field()
      
      
      llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
      
      tools = []
      files = [
          # https://abc.xyz/investor/static/pdf/2023Q1_alphabet_earnings_release.pdf
          {
              "name": "alphabet-earnings",
              "path": "/Users/harrisonchase/Downloads/2023Q1_alphabet_earnings_release.pdf",
          },
          # https://digitalassets.tesla.com/tesla-contents/image/upload/IR/TSLA-Q1-2023-Update
          {
              "name": "tesla-earnings",
              "path": "/Users/harrisonchase/Downloads/TSLA-Q1-2023-Update.pdf",
          },
      ]
      
      for file in files:
          loader = PyPDFLoader(file["path"])
          pages = loader.load_and_split()
          text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
          docs = text_splitter.split_documents(pages)
          embeddings = OpenAIEmbeddings()
          retriever = FAISS.from_documents(docs, embeddings).as_retriever()
      
          # Wrap retrievers in a Tool
          tools.append(
              Tool(
                  args_schema=DocumentInput,
                  name=file["name"],
                  description=f"useful when you want to answer questions about {file['name']}",
                  func=RetrievalQA.from_chain_type(llm=llm, retriever=retriever),
              )
          )
          
          llm = ChatOpenAI(
          temperature=0,
          model="gpt-3.5-turbo-0613",
      )
      
      agent = initialize_agent(
          agent=AgentType.OPENAI_FUNCTIONS,
          tools=tools,
          llm=llm,
          verbose=True,
      )
      
      agent({"input": "did alphabet or tesla have more revenue?"})

### Expected behavior

Always checking the tools before answering a question.