Langchain bug: Responding to out of context questions when using GPT4 with Vector Database where as when the context is passed directly to the Azure OpenAI completion endpoint, we consistently get the correct response
Labels: name: area: vector store, name: auto:bug, name: area: models

### System Info

LangChain version used: 0.0.261 Python (this has been observed in older versions of LangChain too)
This context (context attached) is passed from the search results retrieved from Azure vector search.
Question: what are the legal services in Australia?
 
Response: "Some legal services in Australia include:\n\n1. Legal Aid: Provides legal assistance for low-income people, offering legal representation for eligible clients who appear in court without a lawyer, legal aid applications and information over the phone, 
legal resources, and referrals to other social assistance agencies.\n\n2. Pro Bono Ontario: Offers up to 30 minutes of free legal advice and assistance for 
those who can't afford a lawyer through their Free Legal Advice Hotline.\n\n3. Law Society of Upper Canada – Lawyer Referral Service: An online referral service 
that provides names of either a lawyer or licensed paralegal, offering a free consultation of up to 30 minutes.\n\n4. JusticeNet: A not-for-profit service promoting increased access to justice for low- and moderate-income individuals, connecting members of the public with qualified lawyers, mediators, and paralegals who are registered in the program.\n\nPlease note that these 
resources are specific to Ontario, Canada, and not Australia. The information provided is based on the available documents and may not cover all legal services in Australia."
[context.txt](https://github.com/langchain-ai/langchain/files/12328554/context.txt)

Code snippet used:
 
vector_store_address: str = https://guidecognitivesearch.search.windows.net/
            vector_store_password: str = "xxxxxx"
            model: str = "poc-embeddings"
            embeddings: OpenAIEmbeddings = OpenAIEmbeddings(deployment=model, chunk_size=1)
            index_name: str = "fss-vector-index"
            vector_store: AzureSearch = AzureSearch(
                azure_search_endpoint=vector_store_address,
                azure_search_key=vector_store_password,
                index_name=index_name,
                semantic_configuration_name="fss-semantic-config",
                embedding_function=embeddings.embed_query,
            )
       
            results = vector_store.semantic_hybrid_search(
                query=query,
                k=5
            )
 
            custom_QA_prompt = get_prompt_template()
            llm = get_llm()
            qa_chain = load_qa_with_sources_chain(llm, chain_type="stuff",
                                                prompt=custom_QA_prompt)
         
            openai_results = qa_chain({"input_documents":results,"question": query})
            return {"results":openai_results['output_text'], "metadata":{}}

When the RetrievalQAWithSourcesChain is used combined with load_qa_with_sources_chain – we do see correct response sometimes (say 1 out of 5 time, but this is not consistent every time)
qa_chain = load_qa_with_sources_chain(llm, chain_type="stuff",
                                                prompt=custom_QA_prompt)
            chain = RetrievalQAWithSourcesChain(combine_documents_chain=qa_chain, retriever=vector_store.as_retriever(search_kwargs={'search_type':searchType,'k': 5}),
                                        return_source_documents = True, reduce_k_below_max_tokens=True, max_tokens_limit=4000)
 
 
Sometimes the response returned is accurate. For example, below:
I am unable to provide information on legal services in Australia based on the provided documents. The documents provided information on legal services in Ontario, Canada.

When the LangChain wrapper is not used and the context is passed directly to the Azure OpenAI completion endpoint, we consistently get the correct response, as mentioned below:
Sorry, I cannot provide information about legal services in Australia as the provided documents only pertain to legal
services in Ontario, Canada. You might need to search other sources to address your question.
 


### Who can help?

_No response_

### Information

- [ ] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

Code snippet used:
 
vector_store_address: str = https://guidecognitivesearch.search.windows.net/
            vector_store_password: str = "xxxxxx"
            model: str = "poc-embeddings"
            embeddings: OpenAIEmbeddings = OpenAIEmbeddings(deployment=model, chunk_size=1)
            index_name: str = "fss-vector-index"
            vector_store: AzureSearch = AzureSearch(
                azure_search_endpoint=vector_store_address,
                azure_search_key=vector_store_password,
                index_name=index_name,
                semantic_configuration_name="fss-semantic-config",
                embedding_function=embeddings.embed_query,
            )
       
            results = vector_store.semantic_hybrid_search(
                query=query,
                k=5
            )
 
            custom_QA_prompt = get_prompt_template()
            llm = get_llm()
            qa_chain = load_qa_with_sources_chain(llm, chain_type="stuff",
                                                prompt=custom_QA_prompt)
         
            openai_results = qa_chain({"input_documents":results,"question": query})
            return {"results":openai_results['output_text'], "metadata":{}}
 
 
The issue is observed when searching against location Australia. Australia is not found in any of our ingested documents. If we try Italy or Cancun  (which are also not in the context), we get the expected response:
Image
 
When the RetrievalQAWithSourcesChain is used combined with load_qa_with_sources_chain – we do see correct response sometimes (say 1 out of 5 time, but this is not consistent every time)
qa_chain = load_qa_with_sources_chain(llm, chain_type="stuff",
                                                prompt=custom_QA_prompt)
            chain = RetrievalQAWithSourcesChain(combine_documents_chain=qa_chain, retriever=vector_store.as_retriever(search_kwargs={'search_type':searchType,'k': 5}),
                                        return_source_documents = True, reduce_k_below_max_tokens=True, max_tokens_limit=4000)
 
 
Sometimes the response returned is accurate. For example, below:
I am unable to provide information on legal services in Australia based on the provided documents. The documents provided information on legal services in Ontario, Canada.
 
But most of the time, the response is returned citing sources (which is incorrect)
"results": "Some legal services in Australia include Legal Aid, Pro Bono services, and community legal clinics. However, the provided documents primarily focus on legal services in Ontario, Canada. These services include the Law Society of Ontario, Justice Ontario, JusticeNet, the Law Society of Upper Canada-Lawyer Referral Service, and Legal Aid Ontario. Please note that these services are specific to Ontario, Canada, and not Australia. For more information on legal services in Australia, I would recommend conducting further research.\n\nSource: 9610820-Requested Resources.docx, 9169833-Requested Resources.docx, 8877484-Requested Resources.docx"

### Expected behavior

I am unable to provide information on legal services in Australia based on the provided documents. The documents provided information on legal services in Ontario, Canada.