RetrievalQAWithSourcesChain result sources is not deterministic
Labels: name: area: vector store, name: auto:bug

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.

### Example Code

I implemented a custom retriever like this:
```

class CustomUserRetriever(BaseRetriever):

    vectorstore: VectorStore
    username: str  # New attribute to store the filename

    def clean_metadata(self, doc):
        metadata = doc.metadata
        return {    # Add a button to clear chat history
            "Title": metadata["title"],
            "source": metadata["title"],            
            "chunk_id": metadata["chunk_id"]
        }

    def get_relevant_documents(self, query):
        docs = []
        for doc in self.vectorstore.similarity_search(query, search_type="hybrid",   k=5, filters=f"search.ismatch('{self.username.split('@')[0]}', 'usernames_result') "):
            cleaned_metadata = self.clean_metadata(doc)
            docs.append(Document(
                page_content=doc.page_content,
                metadata=cleaned_metadata))

        return docs

```
And then I created a custom functionality which should return the answer, and the source of the answer. (document name with extension)

  ```
 retriever = CustomUserRetriever(vectorstore=vector_store, username=user_name)

     chain = RetrievalQAWithSourcesChain.from_chain_type(llm, 
                                                            chain_type="stuff", 
                                                            memory=memory, 
                                                            retriever=retriever,
                                                            return_source_documents=True
                                                            )


        #result = chain({"question": user_question})
        result = chain({"question":user_question}, return_only_outputs=True)
```

THen I ask a question which should get a response from 2 document chunks, and the answer is generated correctly.
However asking the same question multiple times, sometimes I get an empty source and sometimes I get a list of the source documents,

![image](https://github.com/langchain-ai/langchain/assets/6962857/dfcfca8c-86c8-4c63-b681-c828e5ea529a)

![image](https://github.com/langchain-ai/langchain/assets/6962857/08bb64a6-d856-4907-979d-19bc0f37cedf)


### Description

I want to get answers from internal documents, but the document retriever returns by similarity chunks that are not relevant to the generated answer, so those ones should be removed, sometimes it works! I asked a question and the chunks are returned, sometimes I get an empty source list.

### System Info

python 3.11
langchain                 0.0.353
openai                    1.6.1

### Related Components

- [X] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async