JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability
Authors: Junda Wang, Zhichao Yang, Zonghai Yao, hong yu


Reviewer paper summaries: This paper introduces a novel approach called Joint Medical LLM and Retrieval Training (JMLR) that synchronizes the training of a Large Language Model (LLM) and a retriever for enhanced medical question answering and reasoning. The authors evaluate JMLR on several medical question answering datasets, demonstrating superior performance compared to state-of-the-art open-source models like Meditron-70B and RAG-based approaches, while requiring significantly less training time.

Unlike previous methods in RAG where the retrieval model was trained separately from the LLM, the paper introduces JMLR (for Jointly trains LLM and information Retrieval (IR)) during the fine-tuning phase. The synchronized training mechanism enhances model ability to retrieve relevant medical information to guide QA. The paper introduces a novel learning objective jointly train the retrieval and LLM. It evaluated on the medical QA datasets and showed that the model outperforms SOTA using pre-training and fine-tuning and conventional separate RAG + LLM framework.
Summary of strengths: JMLR presents an innovative approach to integrating information retrieval and LLM training, resulting in improved performance with less training time on medical question answering tasks compared to traditional domain pretraining methods and RAG-based approaches.

The paper includes a detailed ablation study and case study analysis, providing valuable insights into the effectiveness of the proposed method and its potential for real-world applications.

The paper proposes a joint training process of RAG + LLM in domain specific task and it demonstrated the effectiveness over the pretrained + fine-tuning model and separate retrieval + LLM. The method could be useful when it aims to solve domain-specific task and requires additional annotation task. The novelty of the paper comes from a rank loss to link LLM and retrieval.

The paper outlined the experimentation details comparing other traditional methods and demonstrated superiority.

The computational resource is less than pretraining + finetuning.
Summary of weaknesses: Though the paper provide comprehensive ablation study, it is encouraged to include more models for comparisons. For example in Table2, the authors compare their model(13B) with GPT4, ChatGPT and Meditron70B, which are quite different architecture and sizes. It would be beneficial to compare with same architecture for different size, or same size with different architectures, or more fair comparisons with general/medical domains.

The paper have limited impact since its more suitable to specific domain while LLM may lose generalization capability.

The paper lack some analysis over the effective of adding rank loss. Joint training could already bring performance gain and wonder how much ranked loss will bring.

Conducting some error analysis could be beneficial to understand the underlying issues with the paper. If the errors are due to limitations in the retrieval due to similarity-based, then fine-tuning those components may be necessary. Alternatively, if the errors stem from limitations in the reasoning capabilities of the language model, then exploring a different, potentially more capable language model may be warranted.
Comments, suggestions and typos: It is encouraged to provide some detail about the prompts design for the task.

Any interpretation about the models performance on different datasets? eg. why in Table 4 JMLR-7B and JLMR-13B performs quite different on Amboss?

As author mentioned, the method needs expert efforts to annotate domain-specific data.
Overall assessment scores: 3

3