BioLlama: Improving Language Models for Biomedicine by Retrieving from Millions of Tokens
Authors: Neel Rajani, Jake Lever


Reviewer paper summaries: The paper introduces BioLlama, a RETRO-fitted version of Llama-2 that retrieves from biomedical literature to improve language model performance on biomedical tasks. The RETRO approach augments a language model by adding chunked cross-attention parameters that allow it to attend to relevant retrieved texts during inference. The authors RETRO-fit the middle layers of Llama-2 and fine-tune the added parameters on biomedical datasets like RCT200k. They evaluate BioLlama on several biomedical question answering and multiple choice benchmarks, finding that the 13B version generally outperforms the original Llama-2. The retrieval augmentation allows BioLlama to incorporate relevant scientific information into its outputs, mitigating hallucinations and improving reliability on biomedical tasks.
Summary of strengths: 1. Novel Application: The work investigates RETRO-fitting the open-source decoder-only language model Llama-2 for biomedical applications, which is an important area that could benefit from more reliable and interpretable language models.

2. Scalability: By focusing on a decoder-only architecture like Llama-2, the approach allows scaling up to much larger model sizes compared to encoder-decoder models used in prior biomedical retrieval augmentation works.

3. Empirical Gains: The results show that the larger BioLlama 13B model is able to outperform the baseline Llama-2 13B on 4 out of 5 benchmarks showing promise of the approach, but the evaluation still needs to be more comprehensive. But newer versions of Llama and mistral should also be included in the study.

4. Efficient Fine-Tuning: Only the smaller number of added chunked cross-attention parameters need to be fine-tuned, making the approach relatively computationally efficient compared to full model fine-tuning.
Summary of weaknesses: 1. Motivation and problem statement: The paper could provide a clearer explanation of the specific issues faced by language models in the biomedical domain (e.g., hallucinations, lack of interpretability) and how retrieval augmentation approaches like RETRO aim to address these issues. Elaboration and more evaluation datapoints on why RETRO is a better technique as compared to traditional RAG-based techniques (using open-sourced LLMs) would clarify this point.

2. RETRO implementation details: The paper mentions that they developed a novel implementation of RETRO that can be adapted to any HuggingFace Transformers model. However, the details of this implementation are not clearly explained, and it would be helpful to have more information on how it works and how it differs from the original RETRO implementation.

3. Retrieval corpus details: While the paper mentions that they use the RCT200k dataset as the retrieval corpus, more details could be provided on the preprocessing steps, such as how the chunking and tokenization were performed, and any potential issues or challenges faced during this process.

4. More Elaborate evaluation and comparison with other RAG-based methods can be provided. RAG-based techniques have a lot lesser compute overhead and comparison with such agents would provide more confidence in the current RETRO-based finetuning method. Ablation with other retrievers would also provided a more comprehensive comparison.

5. Experiments could be done on newer models like llama 3, Mistral 7B etc.
Comments, suggestions and typos: please check weaknesses
Overall assessment scores: 2.5