3D-CT-GPT++: Enhancing 3D Radiology Report Generation with Direct Preference Optimization and Large Vision-Language Models

Authors: Hao Chen, Wei Zhao, Yingli Li, Wenjun Li, Zhuoyi Li, Ning Zhu, Tianyang Zhong, Yisong Wang, Youlan Shang, Lei Guo, Junwei Han, Tianming Liu, Jun Liu, Tuo Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: Radiology Report Generation, 3D Medical Imaging, Direct Preference Optimization, Multimodal large Language Models

Abstract:
Automatically generating radiology reports from three-dimensional medical images, such as 3D CT scans, plays a crucial role in modern diagnostics. Current approaches for generating 3D reports often adopt video processing methods, which struggle to effectively capture the relationships along the Z-axis. Additionally, multimodal large language model-based methods for generating 3D image reports face significant limitations, particularly in terms of the image encoder’s ability to represent 3D structures and the hallucinations that arise in generated content. To address these challenges, we propose the 3D-CT-GPT++ model. This model integrates the optimized 3D image encoder CTViT-V, specifically designed for chest CT scans, and builds upon the LLaVA-1.5 architecture. Furthermore, we introduce \textit{Direct Preference Optimization (DPO)}, where GPT-4 is used to score the outputs of our fully fine-tuned (SFT) model, creating a preference dataset for subsequent DPO training. DPO significantly reduces hallucinations in the report generation process, ensuring the generated reports are more aligned with clinical needs. We fine-tuned the model on both high-quality private and public datasets to ensure clinical relevance. Extensive experiments were conducted using standard natural language generation (NLG) evaluation metrics, including BLEU, METEOR, ROUGE-L, and GREEN, to assess the report generation performance. Experimental results demonstrate that 3D-CT-GPT++ significantly outperforms existing methods in terms of accuracy, fluency, clinical factual consistency, and clinical relevance, advancing the automation of 3D medical report generation.

Decision: reject
Reviewer scores: 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730098623858

thread_type:
official_review

<thread_object>
creator: reviewer_a4sf

<content>
summary:
This paper proposes the 3D-CT-GPT++ model. 
The main contributions include 3D image encoder CTViT-V, report generation model 3D-CT-GPT++, and the Direct Preference Optimization(DPO) technique.

strengths:
Originality: The original method combines contrastive learning for image encoder pretraining, spatial transformer+slice transformer for 3D vision encoding, LLaVA architecture to support report generation from images, and DPO technique to optimize the quality of generation.

Quality: A complete framework, sufficient workload on method design, and effective performance

Clarity: Clear figures and writing about methods

Significance: Limited technological progress

weaknesses:
* Bad figure quality: Misalignment of line; Overlap of text and border; Case is not unified.

* Unclear experiment setting. The main text of the paper just describes the source and preprocessing of datasets, without mentioning the training and evaluation setting of datasets which is essential for the experiment section.

* Unfair comparison.  RadFM and M3D results in Table 1 are based on a different evaluation setting. The notation of "Literature Results" is not enough to describe the relationship.

* Computational overhead reduction of CTViT-V is listed as one of the major contributions, but there is no evaluation of it.

* Compared to the entire 3D ViT of M3D, what is the advantages of 2D slice encoding + Z-fusion?

In summary, the content of the main text has a lot of room for improvement and needs to be carefully polished.

questions:
Weaknesses have contained the part of questions. Please consider them seriously.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730494610975

thread_type:
official_review

<thread_object>
creator: reviewer_3ewp

<content>
summary:
The paper titled "3D-CT-GPT++: Enhancing 3D Radiology Report Generation with Direct Preference Optimization and Large Vision-Language Models" presents a novel approach for automating radiology report generation from 3D CT scans, addressing key challenges in current methods. Traditional models, often adapted from video processing, struggle to maintain spatial consistency along the Z-axis and are prone to inaccuracies ("hallucinations") in generated content. 

An optimized 3D image encoder, CTViT-V, is designed to improve spatial and temporal feature extraction specific to 3D CT images. This encoder integrates a slice Transformer and relative position encoding to better capture global dependencies across CT slices.

Built upon the LLaVA-1.5 architecture, the model efficiently processes 3D CT data and enhances the generation of radiology reports, providing a more accurate and context-aware representation.

To mitigate hallucinations without the high costs associated with Reinforcement Learning from Human Feedback, DPO is employed. GPT-4 scores model outputs, creating a preference dataset that aligns generated reports with clinical relevance and accuracy.

strengths:
The work is original in both its application and the methodology employed for 3D CT scan analysis. Unlike many radiology models that focus on 2D images, this paper tackles the more complex problem of 3D report generation, which requires understanding inter-slice dependencies across the Z-axis. The introduction of Direct Preference Optimization (DPO) for reducing hallucinations in generated reports is particularly innovative. Instead of relying on human feedback, which is costly and often impractical for large-scale applications, DPO utilizes GPT-4 for automated scoring, presenting a scalable approach to aligning model output with clinical relevance. Furthermore, the integration of the CTViT-V encoder optimized for 3D imaging enhances both spatial and temporal feature extraction, making this an effective and creative combination of existing model architectures adapted for a challenging domain.

The model's ability to handle 3D CT data and generate clinically relevant radiology reports holds significant potential for the medical field, especially in reducing radiologists’ workload. Addressing hallucinations in generated reports is a critical advancement, as inaccurate information in medical reports can lead to severe consequences. By demonstrating substantial improvement over existing models and applying DPO to radiology for the first time, this work positions itself as a valuable contribution. The approach’s scalability, thanks to automated feedback from GPT-4, enhances its significance by making it feasible for broad deployment in clinical settings. Future extensions to other imaging modalities, such as MRI and X-ray, as discussed in the conclusion, underscore its adaptability and potential long-term impact.

weaknesses:
Lack of Detailed Hyperparameter Configuration: More information about the choices made for hyperparameter training configurations would be beneficial. This detail could help with model reproducibility and provide clarity on the specific adjustments that impact performance.

Comparison with 2D Models: Although the paper focuses on introducing a 3D model, it would be helpful to include a comparative analysis with existing 2D models for similar tasks. This comparison could better highlight the specific advantages and limitations of the proposed 3D approach.

questions:
Could you provide more details on the hyperparameter choices and training configurations? For instance, what specific hyperparameters were optimized, and how were they chosen or tuned?

Given the focus on a 3D model, did you evaluate any baseline comparisons with existing 2D models for similar tasks?

How does DPO specifically reduce hallucinations compared to Reinforcement Learning from Human Feedback (RLHF)? Are there examples where RLHF failed but DPO succeeded?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629963465

thread_type:
official_review

<thread_object>
creator: reviewer_m2vv

<content>
summary:
This paper proposes a novel approach to automatically generate radiology reports from 3D medical images such as 3D CT scans. Traditional methods mostly use video processing technology, which is difficult to effectively capture the relationship between images along the z axis, and multimodal large language models have limitations in representing 3D structures and avoiding the illusion of generating content. To address these issues, the authors proposed the 3D Structure-CT-GPT ++ model, which integrates an optimized 3D image encoder CTViT-V designed for chest CT scanning and is based on the LAVA-1.5 architecture. Furthermore, this work introduces Direct Preference Optimization (DPO), which uses GPT-4 to score model outputs to reduce hallucinations and ensure reports are more aligned with clinical needs. The model was fine-tuned in two high quality conditions.

strengths:
1. This work proposes the 3D-CT-GPT++ model, which integrates an optimized 3D image encoder specifically designed for chest CT scans. This novel approach addresses the limitations of existing methods in representing 3D structures and reducing hallucinations in generated content.
2. This paper is well-organized and clearly presents the problem, methodology, experimental setup, and results. The use of appendices provides additional details, enhancing the readability of the paper.

weaknesses:
1. The format of references needs to be uniform. Some citations lack journal names. Such as "Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.", "Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.". In addition, on line 320 of page 6, the CT-RATE dataset is missing references. Authors need to carefully check and format the references.
2. 3D-CT-GPT++ models use complex architectures such as CTViT-V encoders, which can lead to high demands on computational resources and time. The authors mention that they reduce the computational overhead associated with processing high-resolution 3D volumes. However, the design of specific methods to reduce computing costs is not described in detail. The authors are advised to elaborate further on the design used to improve computational efficiency. It is better to show the computational cost through quantitative comparisons of computational requirements (e.g., GPU memory usage, training time) between the proposed model and baseline approaches. 
3. To further enhance the clinical relevance of the generated reports, the authors could consider incorporating additional clinical context or patient-specific information into the model.

questions:
1. What strategies were used to mitigate potential biases in the generated medical reports?
2. How does the integration of the optimized 3D image encoder improve the representation of chest CT scans compared to traditional 2D image encoders?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660736524

thread_type:
official_review

<thread_object>
creator: reviewer_8v4y

<content>
summary:
The paper introduces 3D-CT-GPT++, a novel model for radiology report generation that leverages an optimized 3D image encoder (CTViT-V) specifically designed for chest CT scans. They apply Direct Preference Optimization to reduce hallucinations and increase factfulness. They show that 3D-CT-GPT++ outperforms existing methods.

strengths:
1. The paper presents a novel CTViT-V encoder, specifically optimized for 3D CT images, addressing the challenges of spatial coherence and slice dependency, which could contribute to improved clinical relevance and accuracy​
2. The use of GPT-4 for preference optimization provides a scalable, cost-effective alternative to human feedback, potentially reducing hallucinations and aligning generated reports more closely with clinical needs​
3. Despite limitations in using NLG metrics, the proposed model outperforms existing benchmarks in BLEU, ROUGE, and METEOR scores

weaknesses:
1. The evaluation relies on standard NLG metrics like BLEU and ROUGE, which do not capture clinical relevance or diagnostic utility. Using the GREEN metric[1] or RadFact[2] for evaluation would allow assessing clinical relevance
2. The authors propose a multi-stage training process. The community would benefit from more ablation and more details for each stage: 
2.1 For Stage 1 and 2 it would be interesting to see how their CTViT-V encoder performs in comparison to other 3D CT image encoder, such as CTViT and CT-CLIP. 
2.2 Stage 2 pre-training is not well described. It's not clear how the custom-built dataset looks like, nor how the interactive approach was performed. 
2.3. Ablation study scoring LLMs would be interesting to see how sensitive the performance is to the chosen LLM. 
2.4. Ablation study on 3D-CT-GPT++(LoRA+DPO) missing, especially since 3D-CT-GPT++(LoRA) achieves higher performance than 3D-CT-GPT++(SFT)
3. The experiments are not reproducible due to the reliance on private dataset in many steps. It's not clear to me, why the authors didn't conduct their experiment on the public CT-RATE dataset. 
4. Comparison to Existing Models: The literature results are not comparable to the fine-tuned 3D-CT-GPT(++) results since the literature results are without finetuning on the dataset, therefore it's hard to judge the works improvements over other methods.

[1] Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., ... & Delbrouck, J. B. (2024). GREEN: Generative Radiology Report Evaluation and Error Notation. arXiv preprint arXiv:2405.03595.
[2] Bannur, S., Bouzid, K., Castro, D. C., Schwaighofer, A., Bond-Taylor, S., Ilse, M., ... & Hyland, S. L. (2024). MAIRA-2: Grounded Radiology Report Generation. arXiv preprint arXiv:2406.04449.

questions:
1. What are the various reconstructions techniques described in 3.4 Dataset, which expands the dataset? (line 320-322)
2. The authors use a subsection of the CT-RATE dataset, it's not described how they do this subselection and what's the motivation behind it. Why don't they use the whole dataset? 
2. Are you planning to open source the training code/weights? 
3. Will the authors publish the Dataset-XY?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731521382110

thread_type:
public_comment

<thread_object>
creator: ~xinkai_yuan1

<content>
title:
Question about this paper

comment:
The manuscript introduces the 3D-CT-GPT++ model, aiming to enhance automatic generation of 3D medical imaging reports by optimizing the 3D image encoder CTViT-V and incorporating Direct Preference Optimization (DPO). However, several areas require improvement:

1. **Originality and Citation of Related Work:**
   - **Slice Transformer Innovation:** The authors claim originality for the Slice Transformer; however, similar concepts have been presented in "M3T: Three-Dimensional Medical Image Classifier Using Multi-Plane and Multi-Slice Transformer" . Additionally, the CT-CLIP training framework aligns with innovations from "Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography" . The authors should explicitly cite these works and delineate distinctions and advancements made in their study.
   - **Fine-Tuning Methodology:** The fine-tuning approach resembles that in "LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation" . Proper citation is necessary, along with clarification of the novel contributions of this study.
   - **Application of DPO:** The manuscript applies DPO as described in "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"  without appropriate explanation. This study appears to apply DPO directly without methodological innovation. This seems to be the only innovation in this paper, which is also not proposed by authors.

2. **Experimental Design and Fairness:**
   - **Dataset Utilization:** The model is trained on a private dataset but compared against the M3D model, which is trained on a public dataset. Moreover, the study does not evaluate performance on the extensive public dataset used by M3D. This comparison may lack fairness, affecting result credibility. It is recommended to conduct experiments on public datasets to ensure fair comparisons and reproducibility. Additionally, unlike M3D, this study lacks downstream tasks such as segmentation and Visual Question Answering (VQA). The private dataset's quality is uncertain, and its size is significantly smaller than M3D's public dataset, yet it reports substantial performance improvements, suggesting potential overstatement.

3. **Result Validation and Model Reproducibility:**
   - **Model Weight Disclosure:** The authors report significant performance gains but do not disclose model weights, hindering result verification. In academic research, sharing models and data facilitates validation and advances the field. It is advisable to release model weights and training data to enhance credibility and scholarly value.

**Recommendations:**
The authors should explicitly cite relevant prior work and clarify the study's novel contributions. Experiments on public datasets are recommended to ensure fair comparisons and result reproducibility. Additionally, releasing model weights and training data would improve the study's credibility and academic significance. 

Jang, J., & Hwang, D. (2022). M3T: Three-Dimensional Medical Image Classifier Using Multi-Plane and Multi-Slice Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 20641-20650). 

Lee, S., Kim, W. J., Chang, J., & Ye, J. C. (2023). LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation. arXiv preprint arXiv:2305.11490.

Bai, Y., Jones, A., Lukosuite, K., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv preprint arXiv:2305.18290.

Hamamci, I. E., Er, S., Almas, F., Simsek, A. G., Esirgun, S. N., Dogan, I., Dasdelen, M. F., Durugol, O. F., Wittmann, B., Amiranashvili, T., Simsar, E., Simsar, M., Erdemir, E. B., Alanbay, A., Sekuboyina, A., Lafci, B., Bluethgen, C., Ozdemir, M. K., & Menze, B. (2023). Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography. arXiv preprint arXiv:2306.00983. 

Bai, F., Du, Y., Huang, T., Meng, M. Q.-H., & Zhao, B. (2023). M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models. arXiv preprint arXiv:2307.12345.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731785828787

thread_type:
public_comment

<thread_object>
creator: ~song_luhui1

<content>
title:
QUESTION ABOUT THIS MANUSCRIPT

comment:
The paper's most glaring issue lies in its unconventional approach to experimental validation. 
The authors make a fundamental error by using their private dataset results to compare against models trained on different public datasets. 
Most surprisingly, the paper presents comparison experiments as ablation studies, which demonstrates a serious misunderstanding of experimental design principles. 
Such methodology violates basic scientific principles of fair comparison and reproducibility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731797024350

thread_type:
public_comment

<thread_object>
creator: ~reiisk.med1

<content>
title:
Dataset concern

comment:
The reliance on private datasets in this paper is a glaring flaw that severely undermines the credibility of its findings. While the authors boast significant performance improvements, these claims are impossible to verify or trust without public access to the dataset. This lack of transparency raises the suspicion that the dataset's quality or curation may have artificially inflated the results. Furthermore, the omission of evaluations on standard public benchmarks, such as those used by the M3D model, makes the comparisons inherently biased and unfair. This practice is a disservice to the research community, and until public datasets are used and results are validated on established benchmarks, the claims made here should be treated with skepticism.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces 3D-CT-GPT++, a novel model for radiology report generation from 3D CT scans, combining the optimized 3D image encoder CTViT-V for enhanced spatial feature extraction with Direct Preference Optimization to reduce hallucinations and ensure clinically relevant outputs.

Reviewers pointed out the strength of the paper for its innovative approach to 3D CT radiology report generation, integrating the optimized CTViT-V encoder and Direct Preference Optimization (DPO) with GPT-4 to achieve scalable, clinically aligned outputs. On the other hand, reviewers raised major concerns about the paper's reliance on standard NLG metrics like BLEU and ROUGE instead of clinically relevant ones like GREEN or RadFact (Reviewer 8v4y). The multi-stage training process lacks details on encoder comparisons, pre-training datasets, and ablation studies on sensitivity to different LLMs and configurations (Reviewer 8v4y). Reproducibility is limited due to private datasets, and comparisons to existing models are based on differing evaluation settings (Reviewers 8v4y, A4sf). Computational efficiency, cited as a contribution, is not quantitatively evaluated (Reviewers m2Vv, A4sf), and comparisons with 2D models to highlight the benefits of the 3D approach are missing (Reviewer 3EwP). Poor figure quality, inconsistent references, and limited details on experimental settings further detract from the paper's clarity and reproducibility (Reviewers A4sf, m2Vv). These issues hinder the study's clarity, applicability, and impact.

During the discussion period, the authors conducted additional experiments with clinically relevant metrics like GREEN to assess model performance (Reviewer 8v4y).They provided additional details about the pre-training datasets and clarified encoder comparisons, noting that the enhanced CTViT-V encoder was compared with the baseline. However, ablation studies on sensitivity to different LLMs and configurations were still in progress, and full results were not included in the rebuttal (Reviewer 8v4y). While the authors provided some clarification about their reliance on private datasets and their decision to use a subset of a public dataset due to time and resource constraints, the core issue of limited reproducibility remains unresolved (Reviewers 8v4y, A4sf). For computational efficiency, the authors did not provide quantitative evaluations or comparative metrics in their response (Reviewers m2Vv, A4sf).

This AC has carefully reviewed the paper, the reviewers’ initial comments, the authors’ responses, and the final feedback. While the authors provided detailed responses and addressed some concerns, significant issues remain unresolved. The reliance on private datasets raises questions about experimental design, fairness, and reproducibility, as key comparisons with existing models are based on differing evaluation settings, and public datasets were not fully utilized for validation. This limitation undermines the credibility and generalizability of the reported results. Additionally, while computational efficiency is cited as a contribution, it has not been quantitatively evaluated, leaving the claimed advantages unsubstantiated. The multi-stage training process still lacks sufficient detail, with incomplete ablation studies and limited comparisons to 2D models, which are necessary to highlight the benefits of the proposed 3D approach. The authors partially addressed the evaluation metrics concern by providing initial GREEN metric results, but their response lacked thorough validation to fully complement or replace standard NLG metrics like BLEU and ROUGE.