OLAPH: Improving Factuality in Biomedical Long-form Question Answering

Authors: Minbyul Jeong, Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, Jaewoo Kang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: medical question answering, automatic evaluation, factuality, hallucination

Abstract:
In the medical domain, numerous scenarios necessitate the long-form generation ability of large language models (LLMs). Specifically, when addressing patients' questions, it is essential that the model's response conveys factual claims, highlighting the need for an automated method to evaluate those claims. Thus, we introduce MedLFQA, a benchmark dataset reconstructed using long-form question-answering datasets related to the biomedical domain. We use MedLFQA to facilitate a cost-effective automatic evaluations of factuality. We also propose OLAPH, a simple and novel framework that utilizes cost-effective and multifaceted automatic evaluation to construct a synthetic preference set and answers questions in our preferred manner. Our framework leads us to train LLMs step-by-step to reduce hallucinations and include crucial medical claims. We highlight that, even on evaluation metrics not used during training, LLMs trained with our OLAPH framework demonstrate significant performance improvement in factuality. Our findings reveal that a 7B LLM trained with our OLAPH framework can provide long answers comparable to the medical experts' answers in terms of factuality. We believe that our work could shed light on gauging the long-text generation ability of LLMs in the medical domain. Our code and datasets are available.

Decision: reject
Reviewer scores: 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730333782389

thread_type:
official_review

<thread_object>
creator: reviewer_dezx

<content>
summary:
This research paper introduces OLAPH, a framework designed to improve the factual accuracy of long-form question answering in the medical domain. The authors address the challenge of hallucinations in large language models (LLMs) by utilizing a cost-effective and multifaceted automatic evaluation system to generate synthetic preference sets. These sets guide the LLMs in prioritizing factuality, semantic similarity, and word composition. 

To facilitate automatic evaluation, the authors introduce MedLFQA, a reconstructed benchmark dataset that includes long-form answers and crucial statements derived from existing biomedical question-answering datasets. Through experiments, the authors demonstrate that even 7B LLMs, trained with OLAPH, can generate answers comparable to GPT-4 and medical experts in terms of factuality.

strengths:
1. the data set collection follows good practice with human verification (high agreement).
2. method is clearly explained.
3. experiments are thorough (though I have questions and suggestions below).
4. most part of the paper is well written and easy to follow.

weaknesses:
1. The method itself (OLAPH) is not really "novel" as the authors claim - the general frameworks has been used in many post-training of LLMs such as Llama and Claude models. I suggest softer contribution claims here.
2. Some claims are not very rigorous - see my questions below.
3. The final analysis can be made clearer.

Recommended related work. 
RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering.

questions:
1. For the analysis of or RQ 3, what's exactly experimental setup? I am not getting why you will need to supply domain knowledge here, and since you train multiple models in RQ 1 and 2, which models did you end up using for this?
2. In Figure 5, if using SFT leads to initial drop of performances, have you try to remove it? And what would the impact be?
3. Line 235 - 236, this may simply suggest that answers in K-QA is terrible, not necessarily that GPT-4 answers are good. Do you have more analysis here?
4. Line 248 - 252, I am not 100% convinced by this claim. Could it be that annotators are biased to believe whatever the model generates?
5. Line 282 + 283, do you use statements as SFT targets too? If so, what's the data format?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730396814722

thread_type:
official_review

<thread_object>
creator: reviewer_oqav

<content>
summary:
the paper introduces MedLFQA dataset to evaluate long-form answers in biomedical contexts. The authors generated this dataset by combining multiple existing datasets to enable comprehensive assessment. This fills a gap in the automated evaluation of LFQA in a domain where factual accuracy is critical. Also, the authors propose & evaluate OLAPH framework on word composition, semantic similarity, factuality- using benchmarks and metrics like FACTSCORE. This framework shows improvements (in factual accuracy) for 7B small parameter models. This study is a novel attempt to reduce hallucinations in medical responses.

strengths:
The paper shows an innovative approach to enhancing the factual accuracy of long-form biomedical question answering. This work also creatively combines current techniques in preference-based learning and factual consistency checks to improve medical domain answers, addressing key limitations in factuality and response quality in prior research. This work is good in quality of methodological design. They have detailed each step in OLAPH’s alignment process making it easy for others to build on top of it. The evaluation approach is also comprehensive as they used well known metrics. The paper is clearly written and answers major questions readers (at least I) may have. The significance of OLAPH lies in its potential to impact the application of LLM in healthcare. Medlfqa dataset can help other researchers in developing and evaluating factually accurate medical response systems.

weaknesses:
The framework instroduced in this study is using GPT-4 to generate must-have and nice-to-have statements in medlfqa. My concern is that it may introduce biases or inaccuracies into the dataset. Although the researchers show that GPT-4-generated responses are close to human-curated answers, i did not find a critical analysis of where synthetic statements might diverge from medical experts.

Also, i think the study framework is a good step towards medical LLMs, but it does not cover the framework's behavior for a domain-specific case like disease diagnosis or recommendation of treatment.

questions:
1. Were there specific medical domains or types of questions where synthetic responses diverged from expert standards?
I think it would be helpful if you could discuss any plans to validate the quality of synthetic data using a diverse set of generative models to assess robustness.

2. As we know, medical knowledge advances over time. So, I am curious to understand how you plan to deal with the challenges of maintaining factual relevance if your framework/dataset is deployed over a longer period?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721289844

thread_type:
official_review

<thread_object>
creator: reviewer_au2s

<content>
summary:
This paper aims to improve factual accuracy of large language models (LLMs) in generating long-form answers within the biomedical domain through two main contributions.
The authors first introduce MedLFQA, a benchmark dataset designed to facilitate the automatic evaluation of factual claims in LLM responses.
Next, the paper proposes OLAPH (Optimizing Large language models' Answers with Preferences of mitigating Hallucination) framework, which iteratively trains LLMs to reduce hallucinations and incorporate essential medical information by leveraging synthetic preference sets derived from cost-effective, multifaceted automatic evaluations.
Experimental results demonstrate that a 7B parameter LLMs trained with OLAPH can generate responses with improved factuality comparable to those of medical experts.

strengths:
+  The paper addresses an important problem in the medical domain, where factuality is crucial for patient safety and trust in medical AI systems.
+ The paper is clearly written with well-structured presentation, clear visualizations, and illustrative examples.
+  The introduction of MedLFQA as a unified benchmark for evaluating factuality in biomedical LFQA is a valuable contribution to the field.
+ The effectiveness of OLAPH is comprehensively validated with thorough analyses, comparisons with proprietary models, and evaluation using metrics independent of the training process,

weaknesses:
+ The novelty of the proposed OLAPH framework is limited, as it mostly follows the standard preference optimization process (SFT and DPO)

questions:
+ The paper should address the convergence properties of OLAPH: Is convergence guaranteed? How do the number and types of evaluation metrics influence convergence rates?
+ Can the evaluation criteria for the qualification of generated data in section 3.2 be used as preference criteria for preference optimization process in section 4.2?
+ Why does the paper follow cross-validation approach for experiments instead of the traditional train/test split approach?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780908166

thread_type:
official_review

<thread_object>
creator: reviewer_ubp6

<content>
summary:
This paper proposed a dataset MedLFQA for automatic evaluation of factuality long-form question-answering in the biomedical domain. The authors also proposed a training framework and claimed that this framework can optimize LLMs to reduce hallucination step-by-step. 
The study topic in this paper is important as the hallucination problem is critical when applying LLMs in the health domain. The manual evaluation and the dataset would be beneficial to the community. 
My main concerns are:
1. The MedLFQA sets the answers in MUST HAVE and NICE TO HAVE and then calculates the hallucination and comprehensiveness metrics by comparing the generated text and the reference text. Although automatic hallucination detection and quantization are difficult, and it is worth exploring automatic evaluation methods, it is not persuasive that the current setting can effectively serve as the hallucination metric. The problems are: a) this setting only evaluates a subset of hallucination; the LLM generated incorrect facts that outside of the MH and NH will not be taken into account in the metrics; b) The calculation of contradicts and entails are based on a fine-tuned BioBERT model, which brings uncertainty of the evaluation, especially on the medical domain. The metrics are highly affected by the performance of the fine-tuned BioBERT model, so a sensitivity analysis needs to be tested. In total, it is not persuasive that the current hallucination and comprehensiveness metrics are sufficient. 
2. It is no surprise that the proposed OLAPH framework can improve the three metrics (Word composition, Semantic Similarity, and Factuality), as these three metrics are directly optimized during the training (similar to the comparison that fine-tuned model is better than non fine-tuned). 
3. Question: In Table 5, when setting the alpha_3 as 1.0, the performance seems better than other smaller alpha_3 scores. So if we enlarge the alpha_3 over 1 (2,3,5 or more), will the model's performance continue to improve? Furthermore, it would be interesting to see the performance of the single loss function (i.e., set two other alpha as zero) to confirm which loss function contributes the most.

strengths:
The study topic in this paper is important as the hallucination problem is critical when applying LLMs in the health domain. The manual evaluation and the dataset would be beneficial to the community.

weaknesses:
1. The MedLFQA sets the answers in MUST HAVE and NICE TO HAVE and then calculates the hallucination and comprehensiveness metrics by comparing the generated text and the reference text. Although automatic hallucination detection and quantization are difficult, and it is worth exploring automatic evaluation methods, it is not persuasive that the current setting can effectively serve as the hallucination metric. The problems are: a) this setting only evaluates a subset of hallucination; the LLM generated incorrect facts that outside of the MH and NH will not be taken into account in the metrics; b) The calculation of contradicts and entails are based on a fine-tuned BioBERT model, which brings uncertainty of the evaluation, especially on the medical domain. The metrics are highly affected by the performance of the fine-tuned BioBERT model, so a sensitivity analysis (with different models other than fine-tuned BioBERT) needs to be tested. In total, it is not persuasive that the current hallucination and comprehensiveness metrics are sufficient. These evaluation limitations need to be discussed. 
2. It is no surprise that the proposed OLAPH framework can improve the three metrics (Word composition, Semantic Similarity, and Factuality), as these three metrics are directly optimized during the training (similar to the comparison that fine-tuned model is better than non fine-tuned).

questions:
Question: In table 5, when set the alpha_3 as 1.0, the performance seems better that other smaller alpha_3 scores. Why not to enlarge the alpha_3 over 1 (2,3,5 or more) to test the model performance? Furthermore, it would be interesting to see the performance of the single loss function (i.e. set two other alpha as zero)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732667271723

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Area Chair and Reviewers,

We would like to extend our heartfelt thanks to you and the reviewers for your thoughtful and constructive feedback on our submission. Your comments and suggestions have been instrumental in helping us improve the clarity and quality of our work.

We deeply appreciate the significant time and effort that the reviewers dedicate to the review process. With only a few days remaining in the discussion phase, we are eager to engage further and address any remaining questions or concerns.

We fully understand the time constraints that reviewers face, but we sincerely hope they might have an opportunity to review our revisions before the discussion phase concludes.

Thank you once again for your time and kind consideration.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work concerns using LLMs to answer clinical queries directly. The main contribution is the compilation of a meta-resource for medical question answering, which is a composition of existing corpora that the authors call MedLFQA. The authors then follow what appears to be a fairly standard training recipe (SFT followed by preference tuning via DPO), informed by a composite metric that combines multiple facets (Eq 4). They refer to this training approach as OLAPH, though I agree with reviewers DEzX and Au2s: This is not particularly novel, methodologically (the authors also seem to acknowledge this in their response). 

I have two major concerns about this work. First, the framing. This paper is about "addressing patients’ questions", and takes for granted that we want to use LLMs to do this. Do we? Is adopting LLMs in this way worth the risk? Don't we want providers in the loop to inform patient care? No discussion of this is found in the paper; it is simply taken for granted that using LLMs to directly answer patient medical questions is a good idea. In the response period, the authors write that they are "aiming to develop a conversational agent that provides personalized medical diagnoses based on patient characteristics."; this is a very thorny proposition, given the risks, but again there is scant discussion of this in the paper. That said, I understand that there is precedent for this as a "task" (hence the availability of existing datasets that the authors compile into MedLFQA) and I also do appreciate that making technical progress on increasing factuality is a valid goal unto itself, and medical QA provides a compelling context for investigating this. So, to be clear, while I take issue with the framing, this would not be means on its own for rejecting the paper in my view. 

Second, and more practically, I share the technical concerns raised by UBp6, and do not feel these were adequately addressed in response and revision. Specifically:

— I am not convinced that (Bio)BERT trained for NLI is likely to be reliable at for the inference task here. The authors point to Table 11 in the Appendix as "proof" of sufficiency, but this is not particularly compelling. This table reports results of GPT-3.5-Turbo and BioBERT *fine-tuned* apparently on MedNLI (the authors write "Further details about this model will be provided in the appendix of the revised paper" in the current draft, so I can only speculate). The latter does OK, the former does poorly. But of course the authors are using the fine-tuned model out of domain, so these numbers are extremely optimistic and not applicable to the samples under consideration. The authors acknowledge this as a weakness in their response, but there is no plan to address it. 

— The evaluation is with respect to the same aspects that are used for the composite objective which informs preference tuning via DPO. Therefore, it is indeed unsurprising that "their" model "outperforms" other approaches: It was trained to optimize the scores being measured. The authors respond to this critique by saying "since our experimental setup ensures a clear separation between the training and test sets—where the model is trained solely on the training set without exposure to the test set—it seems that this does not pose a significant issue.". But this misses the point, which is that if you optimize a model for some measure and compare it to models that were not (which is the case here), you will be all but guaranteed that said model will "win"; this basically just tells us that the optimization was successful.