# A Hybrid Approach to Literature-Based Discovery: Combining Traditional Methods with LLMs
## 4 Discussion
### Identification of Background Knowledge
#### 4.2.2 Timeslicing with RAG  
To determine the impact of RAG on LLM-based background knowledge detection, Langchain's ([[https://www.langchain.com/](https://www.langchain.com/)]([https://www.langchain.com/](https://www.langchain.com/)), accessed 1 November 2024) retrieval chains were used to augment the prompt. The KB was queried for matches and close matches of the two terms within a CHKP and the additional information was passed to the LLM as context. The model was instructed to use this information in addition to its own knowledge to answer the question (the prompt template is shown in Appendix B.2 with the query remaining identical to direct background detection).  
Table 8 presents the results of augmented queries on the 1000 CHKP sample from Section 3.4.3, providing a breakdown of the LLM's classifications. The quantity of background knowledge remains lower for FSL-based LBD than when this is SemRep relationbased. RAG augmentation resulted in a slight increase (1%) in background knowledge detection with the FSL approach, but a small decrease (5%) in SemRep. The purpose of RAG is to provide up-to-date or highly specific information that the base LLM might not have. If our KB contained a high density of facts that were not present in the LLM's training data, we would expect to see a more significant change in the classification results. The fact that the changes are marginal suggests that most of the information in our 400k-edge KB is already \"known\" to the LLM (which was expected as it was created from information published before the LLM's knowledge cut-off). The effect of RAG was therefore to confirm, rather than to augment, the model's existing knowledge. However, since both approaches showed an increase in the number of 'Unclear' labels, these were manually explored.  
Aligning the output with [91]'s division of error types, specifically **factuality** (whether the response is factually correct), **refusal** (whether the response is a refusal to answer) and **contradiction** (whether the response is inconsistent with itself), the majority of the new 'Unclear' labels are due to contradictions within the response while the majority of 'Unclear' labels arising from the unaugmented LLM response correspond to refusals. A frequent source of contradictions was the confirmation of the context at the beginning of the response (whether or not the context contained the exact relation) with the answer and the explanation to the question following (with the connection not subsequently found).  
Exploring the CHKPs whose classification changed from 'known' when RAG was added shows that the LLM's response sometimes claimed the existence of a direct relationship when only an indirect one could be supported with evidence. This did not appear to be a hallucination, rather a potential shortcoming of the prompt design (note that various versions of the prompt were tried to emphasize the need for a direct connection, but it did not appear to be possible to eliminate this category of error completely while retaining the option to use synonyms). For example, when querying the existence of a direct connection between _anticonvulsant medications_ and _diabetic retinopathy_, the unaugmented LLM responds with this being a known connection but also adds the following:  
Given the potential metabolic impacts of anticonvulsants and considering the broader context of managing chronic diseases like diabetes, there appears to be a rationale supporting further investigation into any possible connections between anticonvulsants and diabetic retinopathy.  
This suggests that while the LLM accurately identifies potential relevance, its interpretation of a 'direct connection' can sometimes be broader than intended for strict LBD filtering.