RagMedge: Dense Retrieval Augmented Medical Dialogue Generation Towards Personalised Medical Knowledge Grounded Diagnosis
Authors: Chen Tang, Bo Lv, Chenghao Xiao, Bohao Yang, Kun Zhao, Xuehui Yu, Chenghua Lin


Reviewer paper summaries: The paper introduces RagMedge, a novel framework designed to enhance medical dialogue generation by integrating personalized medical knowledge into language models (LLMs). The framework addresses the limitations of existing methods that implicitly integrate medical knowledge, which often results in uncontrollable and unreliable outputs. RagMedge constructs a personalized knowledge base by extracting discrete knowledge points from real doctor-patient dialogues and representing them as semantically condensed vectors. During inference, the model retrieves relevant knowledge points from this personalized knowledge base to generate more informed and accurate responses.
Summary of strengths: - The paper proposes a new framework, RagMedge, and introduces a novel task called "Knowledge Retrieval Assisted Medical Dialogue Generation.  Key is the creation and utilization of a personalized knowledge base that captures patient-specific details, unlike standardized medical resources. This addresses the limitation of existing approaches that rely on generic knowledge, leading to more relevant and tailored responses.
- The paper presents extensive experimental results and human evaluations demonstrating that RagMedge significantly improves the quality of medical dialogue generation.
- The authors introduce a new dataset specifically designed for Knowledge Retrieval Assisted Medical Dialogue Generation. The dataset comprises medical dialogues paired with corresponding knowledge points generated by GPT-4 and Mixtral. The authors commit to releasing a new dataset specifically designed for the "Knowledge Retrieval Assisted Medical Dialogue Generation" task.
Summary of weaknesses: - The paper acknowledges limitations in the quality and diversity of the dialogue data used. The reliance on online sources introduces potential biases and inaccuracies that may not fully represent real-world medical scenarios.
- Computational cost of constructing and maintaining the personalized knowledge base is a concern. As the number of dialogues and medical domains expand, the scalability of the approach requires detailed investigation. While the paper mentions validating retrieved knowledge points, the overall explainability and transparency of the system require further exploration. Understanding how specific knowledge points influence the generated responses and ensuring their interpretability for medical professionals is crucial
-  While the human evaluation provides valuable insights, the reliance on internal recruitment and a limited number of evaluators (three) might introduce bias. Engaging with medical professionals for evaluation would strengthen the findings and is highly recommended.
Comments, suggestions and typos: - Providing more details about the prompt design and validation process for both knowledge base construction and dialogue generation would enhance the reproducibility and clarity of the methodology.
- The paper briefly mentions data privacy in the Ethics Statement but could benefit from a more in-depth discussion of potential ethical implications and risks associated with using personalized medical data.
- The paper uses the term "Mixtral" to refer to both the model used for knowledge base construction (Mixtral 8X7) and a smaller variant (Mistral) used as a baseline. Clearly distinguishing these models throughout the paper would improve clarity.
Overall assessment scores: 2.5