MultiRetrievalQAChain - unable to return source document when I use multiple documents
Labels: name: area: agent, name: area: doc loader, name: area: vector store, name: auto:bug

### System Info

Python == 3.9.18
Langchain == 0.0.327


### Who can help?

MultiRetrievalQAChain

### Information

- [ ] The official example notebooks/scripts
- [X] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [X] Document Loaders
- [X] Vector Stores / Retrievers
- [ ] Memory
- [X] Agents / Agent Executors
- [ ] Tools / Toolkits
- [X] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

@hwchase17 @agol

Steps to reproduce behaviour:
1. Modified MultiRetrievalQAChain with source documents returned (added `return_source_documents=True`)
```
class MultiRetrievalQAChainAddSource(MultiRetrievalQAChain):
    @classmethod
    def from_retrievers(
        cls,
        llm: BaseLanguageModel,
        retriever_infos: List[Dict[str, Any]],
        default_retriever: Optional[BaseRetriever] = None,
        default_prompt: Optional[PromptTemplate] = None,
        default_chain: Optional[Chain] = None,
        **kwargs: Any,
    ):
        if default_prompt and not default_retriever:
            raise ValueError(
                "`default_retriever` must be specified if `default_prompt` is "
                "provided. Received only `default_prompt`."
            )
        destinations = [f"{r['name']}: {r['description']}" for r in retriever_infos]
        destinations_str = "\n".join(destinations)
        router_template = MULTI_RETRIEVAL_ROUTER_TEMPLATE.format(destinations=destinations_str)
        router_prompt = PromptTemplate(
            template=router_template,
            input_variables=["input"],
            output_parser=RouterOutputParser(next_inputs_inner_key="query"),
        )
        router_chain = LLMRouterChain.from_llm(llm, router_prompt, return_source_documents=True)
        destination_chains = {}
        print('RETRIEVER_INFOSSSS', retriever_infos)

        for r_info in retriever_infos:
            prompt = r_info.get("prompt")
            retriever = r_info["retriever"]
            chain = RetrievalQA.from_llm(
                llm, prompt=prompt, retriever=retriever, return_source_documents=True
            )

            name = r_info["name"]
            destination_chains[name] = chain
            print('USING DESTINATION CHAIN', destination_chains)
        if default_chain:
            _default_chain = default_chain
        elif default_retriever:
            _default_chain = RetrievalQA.from_llm(
                llm, prompt=default_prompt, retriever=default_retriever, return_source_documents=True
            )
        else:
            prompt_template = DEFAULT_TEMPLATE.replace("input", "query")
            prompt = PromptTemplate(template=prompt_template, input_variables=["history", "query"])
            _default_chain = ConversationChain(
                llm=ChatOpenAI(), prompt=prompt, input_key="query", output_key="result"  # type: ignore
            )
        return cls(
            router_chain=router_chain,
            destination_chains=destination_chains,
            default_chain=_default_chain,
            **kwargs,
        )
```
2. create vectorstore (db = Weaviate(client,index_name,'text',attributes=['source'])
3. create retrievers (retriever = db.as_retriever())
4. retriever_infos for multiple retrievers:
```
chain = MultiRetrievalQAChainAddSource.from_retrievers(
    llm=llm, 
    retriever_infos=retriever_infos, 
    verbose=True
    )
```
5. using meditations.pdf and bushido.pdf as source documents
6. query for one retriever will return source docs (in this example - meditations.pdf):
``` "text": "json\n{\n    \"destination\": \"meditations.pdf\",\n    \"next_inputs\": \"What does Marcus Aurelius emphasize the most in Meditations?\"\n}\n", ```

```
'result': 'Marcus Aurelius emphasizes the importance of..., 'source_documents': [Document(page_content=\"Marcus Aurelius' Meditations - tr. Casaubon  v. 8.16 , uploaded to www.philaletheians.co.uk , 14 July 2013  \\nPage 128 of 128 A parting thought\", metadata={'page': 127, 'source': 'meditations.pdf'}), Document(page_content=\"For it is not lawful , \\nthat anything  that is of another  and inferior  kind and nature , be it what it will, as \\neither  popular  applause , or honour , or riches , or pleasures ; should  be suffered  to \\nconfront  and contest  as it were, with that which  is rational , and operatively  good. For \\nall these  things , if once though  but for a while , they begin  to please , they presently  \\nprevail , and pervert  a man’s mind , or turn a man from the right way. Do thou there-\\nfore I say absolutely  and freely  make  choice  of that which  is best, and stick unto it. \\nNow, that they say is best, which  is most profitable . If they mean  profitable  to man \\nas he is a rational  man, stand  thou to it, and maintain  it; but if they mean  profitable , \\nas he is a creature , only reject  it; and from this thy tenet  and conclusion  keep off \\ncarefully  all plausible  shows  and colours  of external  appearance , that thou mayest  \\nbe able to discern  things  rightly .\", metadata={'page': 24, 'source': 'meditations.pdf'})]}
```

6. query something for multiple retrievers
``` "text": "json\n{\n    \"destination\": \"DEFAULT\",\n    \"next_inputs\": \"marcus meditations and bushido similarities in life long learning and self improvement\"\n}\n ```

```
"result": "Both Marcus Meditations and Bushido emphasize the importance of life-long learning and self-improvement. In Marcus Aurelius' \"Meditations,\" he emphasizes the need for constant self-reflection and introspection as a means to improve oneself. He encourages individuals to examine their thoughts, actions, and values in order to develop a virtuous character.\n\nSimilarly, Bushido, which is the code of conduct followed by samurais in feudal Japan, also highlights the significance of continuous self-improvement. Bushido emphasizes the values of loyalty, honor, and discipline, and it encourages individuals to constantly strive for self-mastery in various aspects of life.\n\nBoth philosophies recognize that personal growth and development are ongoing processes that require self-discipline, reflection, and a commitment to learning. They emphasize the importance of cultivating virtues and values that contribute to a meaningful and fulfilling life.\n\nIt's fascinating to see how different cultures and historical periods have explored similar concepts of self-improvement and personal growth. Do you have any other questions or topics you'd like to discuss?
```



### Expected behavior

For single retriever, there is an output of the source documents. But for multiple retrievers, there is no output of the source documents (potentially due to ` \"destination\": \"DEFAULT\"` - just a hunch here)

For multiple retrievers:
```
"result": "Both Marcus Meditations and Bushido emphasize the importance of life-long learning and self-improvement. In Marcus Aurelius' \"Meditations,\" he emphasizes the need for constant self-reflection and introspection as a means to improve oneself. He encourages individuals to examine their thoughts, actions, and values in order to develop a virtuous character.\n\nSimilarly, Bushido, which is the code of conduct followed by samurais in feudal Japan, also highlights the significance of continuous self-improvement. Bushido emphasizes the values of loyalty, honor, and discipline, and it encourages individuals to constantly strive for self-mastery in various aspects of life.\n\nBoth philosophies recognize that personal growth and development are ongoing processes that require self-discipline, reflection, and a commitment to learning. They emphasize the importance of cultivating virtues and values that contribute to a meaningful and fulfilling life.\n\nIt's fascinating to see how different cultures and historical periods have explored similar concepts of self-improvement and personal growth. Do you have any other questions or topics you'd like to discuss?", 'source_documents': [Document(page_content=....", metadata={'page': 127, 'source': 'meditations.pdf'}), Document(page_content=...", metadata={'page': 24, 'source': 'meditations.pdf'}),Document(page_content=....", metadata={'page': ###, 'source': 'bushido.pdf'}), Document(page_content=...", metadata={'page': ###, 'source': 'bushido.pdf'})]}
```