M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

## Abstract

Abstract Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data , comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed , a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench , which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D .

## 1 Introduction

Medical scenarios [] encompass a vast amount of multi-modal information, including patient information, diagnostic reports, and medical images of various modalities. Diagnostic reports paired with medical images provide accurate and detailed descriptions, findings, and diagnoses, considered as high-quality annotations. These medical images and texts are saved into the database along with the physician’s diagnostic workflow at scale and without additional cost. How to make full use of these image and text data to build a medical image diagnosis model is a key issue.

In recent work [, , , ], multi-modal large language models (MLLMs) have demonstrated remarkable performance in various multi-modal tasks, effectively integrating image and text data. By combining the perceptual abilities of vision models [, , ] with the generative capabilities of large language models (LLMs) [, , , , , ], MLLMs have garnered significant attention from researchers, particularly in medical image analysis. Existing medical MLLMs [, , , ] fine-tune publicly available 2D MLLMs on medical image and text data to achieve tasks such as image-text retrieval, report generation, and visual question answering. These models are proposed as powerful tools for understanding and reasoning about 2D medical images. However, when confronted with widespread 3D medical images, such as CT and MRI, which contain rich spatial information, these methods often struggle, either requiring costly slice-by-slice analysis or failing outright.

In this work, we focus on 3D medical images and extend the use of MLLMs to analyze them. To this end, we collect a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs covering various diseases and tasks. This dataset is the largest public 3D multi-modal medical dataset to date and can advance related research. Additionally, we propose M3D-LaMed, a versatile 3D MLLM for medical image analysis. It can perform tasks such as image-text retrieval, report generation, and visual question answering, and also includes tasks such as vision language positioning and segmentation for the first time. Leveraging a pre-trained 3D vision encoder under a CLIP-like[] strategy and an efficient 3D spatial pooling perceiver, it can understand and reason about 3D images directly. For the first time, M3D-LaMed is combined with a 3D promptable segmentation model to achieve referring expression segmentation of 3D medical images. Furthermore, to evaluate the model’s ability in 3D medical analysis, we propose a multi-modal medical benchmark, M3D-Bench, which includes 8 tasks covering various aspects of 3D medical image analysis. This is the first comprehensive benchmark in 3D medical image analysis. In addition to traditional metrics, we have introduced LLM-based evaluation, allowing M3D-Bench to automatically and accurately evaluate the model’s performance.

In summary, our contributions are as follows:

- •
Establish M3D-Data, a large-scale 3D medical dataset containing 120K image-text pairs and 662K instruction-response pairs.
- •
Propose M3D-LaMed, a versatile MLLM for 3D medical image analysis, applied to various 3D multi-modal tasks.
- •
Create M3D-Bench, a comprehensive 3D multi-modal benchmark for 8 tasks.

**Table 1: Comparisons of M3D-Data with other datasets. M3D-Data contains image-text pairs (M3D-Cap) and instruction-response pairs (M3D-VQA, M3D-RefSeg, and M3D-Seg), the largest 3D medical dataset, and involves the most tasks. VQA: Visual Question Answering under closed- and open-ended. ITR: Image-Text Retrieval. RG: Report Generation. REC: Referring Expression Comprehension. REG: Referring Expression Generation, SS: Semantic Segmentation. RES: Referring Expression Segmentation.**
| Datasets | Types | Tasks | Images | Texts |
| --- | --- | --- | --- | --- |
| VQA-Med[] | 2D | VQA | 3,200 | 12,792 |
| MIMIC-CXR[] | 2D | ITR, RG | 377,110 | 227,835 |
| PMC-OA[] | 2D | ITR, RG | - | 1,646,592 |
| PMC-VQA[] | 2D | VQA | 149,075 | 226,946 |
| RP3D-Caption[] | 3D | ITR, RG | 51K | - |
| RP3D-VQA[] | 3D | VQA | - | 142K |
| M3D-Cap | 3D | ITR, RG | 120,092 | 42,496 |
| M3D-VQA | 3D | VQA | 96,170 | 509,755 |
| M3D-RefSeg | 3D | REC, REG, SS, RES | 210 | 2,778 |
| M3D-Seg | 3D | REC, REG, SS, RES | 5,772 | 149,196* |

## 2 Related Work

### 2.1 Medical Multi-Modal Datasets

In medical scenarios [], rich images of various modalities and texts are available. However, previous works [, ] have difficulty constructing large-scale medical multi-modal datasets due to privacy and restrictions. Inspired by CLIP[], PMC-OA[] obtained image and text data from medical papers through web crawling, resulting in 1.6M 2D image-text pairs. Additionally, MedMD[] aims to achieve multiple objectives: building 2D and 3D medical models, integrating public 2D medical datasets, and crawling 3D image and text data from medical professional websites. One of its 3D datasets, RP3D[], comprises 51K 3D image-text pairs and 142K VQA data generated from LLMs. In our work, we primarily focus on constructing large-scale 3D medical datasets by crawling medical professional websites. M3D-Data includes 120K 3D image-text pairs and 662K instruction-response pairs generated through an automatic and low-cost data generation pipeline. Furthermore, M3D-Data’s M3D-Seg component collects nearly 6K 3D images from 25 public medical segmentation datasets, facilitating tasks such as vision language positioning and segmentation. In summary, M3D-Data is the largest 3D medical multi-modal dataset, supporting various tasks, as shown in Table 1.

### 2.2 Medical MLLMs

Medical MLLMs [, , ] are typically fine-tuned from powerful 2D open-source MLLMs using medical multi-modal datasets. For instance, LLaVA-Med [], Med-PaLM M[], and Med-Flamingo[] are based on models such as LLaVA[], PaLM-E[], and Flamingo[], respectively. The availability of large-scale datasets like PMC-VQA [] has enabled training medical MLLMs from scratch, although initially limited to 2D images. While RadFM [] supports both 2D and 3D images, it is primarily used for text generation tasks such as VQA and has poor performance. In our work, M3D-LaMed serves as a generalist MLLM for 3D medical image analysis. It handles not only text generation tasks like report generation and VQA but also pioneers vision tasks, like vision language positioning and segmentation in 3D medical images, which are crucial for identification and localization in medical image analysis.

## 3 Dataset

The goal of M3D-Data is to provide a data source for various 3D multi-modal medical tasks. M3D-Data comprises 120K image-text and 662K instruction-response pairs covering 8 tasks, as shown in Table 1.

Figure: Figure 1: The generation pipelines for M3D-Data. (a) In the VQA data generation pipeline, we employ LLM to generate five types of questions from medical reports using a prompt-based method. Subsequently, we eliminate dirty data through self-filtering and check the test set by LLM and experts, achieving a pass rate of 99.4%. (b) In positioning and segmentation data generation pipelines, three feasible methods were implemented to construct image-mask-text triplets, including label-based instruction, definition-based instruction, and annotated instruction. The box coordinates required for positioning tasks can be generated directly from masks.

### 3.1 Image-Text Pair Data

Healthcare institutions, such as hospitals, maintain extensive repositories of medical images and diagnostic reports, often accompanied by text descriptions. However, disclosing such comprehensive image-text datasets presents challenges due to the sensitive nature of patient data and privacy concerns. To avoid these privacy concerns, we collect medical images and reports from publicly accessible professional medical websites (^1^11Radiopaedia: [https://radiopaedia.org/](https://radiopaedia.org/)). Specifically, each patient case in our dataset includes multiple images along with their corresponding reports, which are supplemented by meticulously peer-reviewed captions provided by experts from Radiopaedia^1. Focusing on 3D CT data due to its critical role in medical image analysis, particularly in diagnosing, positioning, and measuring lesions throughout the body, we successfully compiled a large-scale 3D medical image-text pair dataset, M3D-Cap, comprising 120K pairs. M3D-Cap supports tasks such as image-text retrieval and report generation.

### 3.2 Instruction-Response Pair Data

Instruction-response pairs data contains pairs of instructions or questions and corresponding responses or answers. These data are commonly used in various multi-modal understanding and generation tasks, such as VQA, vision language positioning, and segmentation. The sum of the instruction data for these tasks is 662K.

Figure: Figure 2: The data statistics of M3D-VQA on five question types. What, which, and where are 3 typical questions. Samples of 5 topics are displayed in word clouds.

#### 3.2.1 VQA Data

The acquisition and annotation of VQA data in medical scenarios often involve substantial expenses, primarily due to the requirement for specialized medical expertise. To mitigate these costs, we leverage public LLMs to analyze existing diagnostic reports and generate instruction-response pairs using a prompt-based approach, as illustrated in Figure 1(a). Additionally, we apply self-filtering techniques to eliminate noise data based on predefined rules. Following this process, the test set undergoes validation by both LLM-based analysis and expert review, resulting in an impressive pass rate of 99.4% (13729/13814). Considering the cost implications associated with utilizing ChatGPT [], we opt for the free and robust Qwen-72B [] for data generation. Specifically, we generate multiple-choice questions covering 5 key topics extracted from each diagnostic report: plane, phase, organ, abnormality, and location. The data statistics of M3D-VQA, including topic samples, are illustrated in Figure 2. Adopting the multiple-choice format offers convenience for both open- and closed-ended evaluation, facilitating comprehensive assessment.

#### 3.2.2 Positioning and Segmentation Data

Vision language positioning and segmentation require the integration of images, texts, and referring regions, typically represented as boxes and masks corresponding to the tasks. To streamline data handling, we adopt a unified data format consisting of image-mask-text triplets. This format provides resources for positioning tasks and segmentation by converting masks into coordinates of 3D boxes.
However, the rarity of lesion mask annotations during diagnosis in healthcare institutions makes the creation of a 3D image-mask-text pair dataset more costly due to the necessity for detailed region annotations. Therefore, we developed datasets using three distinct approaches, as depicted in Figure 1(b): (1) Label-base instruction data: Directly created from image-mask pairs of public segmentation datasets using label templates. (2) Definition-based instruction data: Constructed using a term dictionary containing multiple related definitions and descriptions generated by LLMs, with subsequent utilization of definition templates for instruction data creation. (3) Annotated instruction: Expert annotation of text descriptions about referring regions to produce image-mask-text triplets. Additionally, we leverage the low-cost yet powerful LLM, Qwen-72B, to enhance textual content and generate comprehensive instruction data based on annotations. In approaches (1) and (2), we compile the joint segmentation dataset, M3D-Seg, by amalgamating densely annotated 3D CT segmentation datasets obtained from public repositories, as shown in detail in Appendix Table 2. Meanwhile, in approach (3), we annotate a subset, M3D-RefSeg, from the Totalsegmentator[] dataset.

Figure: Figure 3: Overview of our M3D-LaMed model. (a) The 3D image encoder is pre-trained by cross-modal contrastive learning loss with image-text pairs, performing image-text retrieval. (b) In the M3D-LaMed model, 3D medical images are fed into a pre-trained 3D image encoder and an effective 3D spatial pooling perceiver to produce refined embeddings inserted into LLM. The output [SEG] token is used as a prompt to employ the promptable 3D medical segmentation model, thereby obtaining the 3D mask output. With our M3D-Data, M3D-LaMed can be leveraged on various 3D medical tasks.

## 4 Method

Due to unreliable 3D encoders, we train the vision encoder from scratch. As depicted in Figure 3 (a), we pre-train the 3D medical vision encoder using a CLIP-like[] strategy on M3D-Cap. Subsequently, we introduce end-to-end tuning to integrate 3D information into the LLM using instruction data, ensuring seamless interaction between the vision and language, shown in Figure 3 (b).

### 4.1 Model Architecture

#### 4.1.1 3D Image Encoder

Given a specific 3D image $I\in R^{C\times D\times H\times W}$, $C,D,H,W$ represent the channel, depth, height, and width respectively, we derive the image embedding $v=E_{img}(I)\in R^{n\times d}$. Here, $E_{img}$ denotes the image encoder, $n$ represents the image token numbers, and $d$ denotes the token dimensions. For versatility and generality, we utilize the 3D Vision Transformer (3D ViT) [] as the vision encoder. The 3D ViT comprises an $N$-layer transformer with attention mechanisms. Each layer operates on patches extracted from the input image, where the patch size is $P_{D}*P_{H}*P_{W}$. We can import standard 3D ViT directly from the MONAI (^2^22MONAI: [https://monai.io/](https://monai.io/)) library.

#### 4.1.2 3D Perceiver

Since 3D images inherently possess high dimensions and numerous tokens, direct input into an LLM results in significant computational costs. To alleviate this challenge, we propose a straightforward and efficient 3D spatial pooling perceiver designed to reduce the number and dimensions of embeddings, shown in Appendix Figure 1. Firstly, the output tokens from the vision encoder are reconstructed into 3D space for pooling. This step effectively decreases token numbers while retaining the original spatial information. Secondly, we employ a series of Multi-Layer Perceptrons (MLPs) to adjust the embedding dimensions, aligning them with the dimensions required by the LLM. By implementing these steps, the 3D perceiver not only mitigates computational costs but also ensures the preservation of spatial information.

#### 4.1.3 LLM

Large language models trained on extensive natural language corpora offer generalized embedding representations and powerful generation capabilities. In our study, we utilize the LLaMA-2-7B model directly as our base LLM due to its proven effectiveness in capturing linguistic patterns and generating coherent text across various domains.

#### 4.1.4 Promptable Segmentation Module

Inspired by LISA[], we leverage the capabilities of MLLMs to implement referring expression segmentation using a promptable segmentation module. Specifically, if an [SEG] token is present in the output tokens, we extract the last layer embedding of the [SEG] token as a feature. Subsequently, we map this feature into a prompt to drive the segmentation module through an MLP, ultimately producing the segmentation mask. For our implementation, we opt for SegVol[] as the promptable segmentation module due to its robust performance and compatibility with our framework.

**Table 2: Comparison on image-text retrieval. Our model outperforms previous models at various difficulty levels. IR means image-to-text retrieval. TR means text-to-image retrieval. R@1, R@5, and R@10 denote recall at 1,5,10.**
| Methods | PMC-CLIP[] | Our |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Test samples | 100 | 500 | 1000 | 2000 | 100 | 500 | 1000 | 2000 |  |
| IR | R@1 | 9.00 | 4.40 | 1.90 | 1.15 | 64.00 | 39.60 | 27.30 | 19.10 |
| R@5 | 28.00 | 12.80 | 7.60 | 4.35 | 95.00 | 76.20 | 61.10 | 47.45 |  |
| R@10 | 45.00 | 18.80 | 12.10 | 7.60 | 99.00 | 87.20 | 76.10 | 62.25 |  |
| TR | R@1 | 18.00 | 7.60 | 4.60 | 3.15 | 70.00 | 40.40 | 26.60 | 18.45 |
| R@5 | 47.00 | 20.20 | 13.00 | 8.55 | 95.00 | 74.20 | 61.80 | 47.30 |  |
| R@10 | 59.00 | 31.00 | 19.80 | 13.55 | 98.00 | 87.00 | 75.30 | 62.15 |  |

**Table 3: Comparison on report generation. In the LLM-based metric, we use Qwen-72B to score generations and references on an absolute value of 0-100 based on the content overlap.**
| Method | BLEU | ROUGE | METEOR | BERT-Score | LLM-Based |
| --- | --- | --- | --- | --- | --- |
| RadFM[] | 12.23 | 16.49 | 11.57 | 87.93 | 4.32 |
| Our (Linear) | 14.49 | 19.25 | 14.11 | 88.32 | 7.12 |
| Our (MLP) | 15.15 | 19.55 | 14.38 | 88.46 | 8.49 |

### 4.2 Vison Encoder Pre-training

Due to the lack of robust 3D medical image encoders, we adopt the architecture and training approach of CLIP[] to pre-train on M3D-Cap. As shown in Figure 3 (a), we utilize cross-modal contrastive learning loss in pre-training. The vision encoder is pre-trained from scratch, while the text encoder utilizes a pre-trained BERT [] as initialization.

### 4.3 MLLM Training

Upon obtaining the pre-trained 3D medical vision encoder, we integrate it into the LLM using a 3D perceiver for end-to-end training. Our training process consists of two main steps. Initially, we freeze the vision encoder and the LLM, focusing on fine-tuning only the 3D perceiver using image-text pairs. Subsequently, we fine-tune the vision encoder, 3D perceiver, LLM, and segmentation module using instruction data. If the [SEG] token is present in the output tokens, segmentation training is performed using Dice and BCE loss similar to []. Considering the training cost and retaining the original knowledge of LLM, we adopt the LoRA [] strategy for parameter-efficient fine-tuning on the LLM.

## 5 Evaluation and Benchmark

M3D-Bench is a comprehensive and accurate evaluation benchmark that covers 8 tasks from 5 major tasks: image-text retrieval, report generation, VQA, positioning, and segmentation.

### 5.1 Evaluation on Image-Text Retrieval

In 3D image-text retrieval, the model aims to match images and texts from a dataset based on their similarity, typically involving two tasks: text-to-image retrieval (TR) and image-to-text retrieval (IR). For evaluation, we selected a high-quality subset of 2000 pairs from M3D-Cap as the test set. This set is further categorized into four difficulty levels: easy (100 pairs), medium (500 pairs), difficult (1000 pairs), and very difficult (2000 pairs), based on the size of the retrieving data pool.
Evaluation metrics include recall at ranks 1, 5, and 10 for IR and TR, assessing the model’s ability to retrieve relevant images or texts from top-ranked results.

**Table 4: Comparison on 3D closed-ended VQA. Our model outperforms other 3D MLLM by a large margin in five types of questions.**
| Methods | Plane | Phase | Organ | Abnormality | Location | Mean |
| --- | --- | --- | --- | --- | --- | --- |
| RadFM[] | 19.65 | 28.70 | 16.80 | 18.92 | 14.88 | 19.79 |
| Our | 98.80 | 79.75 | 74.75 | 66.65 | 58.94 | 75.78 |

**Table 5: Evaluation on 3D open-ended VQA. Our model outperforms the other MLLM in five types of problems and four metric evaluations.**
| Method | Metric | Plane | Phase | Organ | Abnormality | Location | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- |
| RadFM[] | BLEU | 14.24 | 14.25 | 14.24 | 15.64 | 23.58 | 16.39 |
| ROUGE | 25.40 | 25.41 | 25.38 | 25.38 | 29.09 | 26.13 |  |
| METEOR | 20.62 | 20.63 | 20.61 | 20.60 | 24.19 | 21.33 |  |
| BERT-Score | 92.68 | 92.04 | 86.79 | 85.84 | 86.26 | 88.72 |  |
| Our | BLEU | 98.37 | 74.41 | 34.20 | 15.91 | 24.00 | 49.38 |
| ROUGE | 98.42 | 78.63 | 37.87 | 19.27 | 27.74 | 52.39 |  |
| METEOR | 49.20 | 63.58 | 23.78 | 12.83 | 18.50 | 33.58 |  |
| BERT-Score | 99.47 | 95.55 | 88.97 | 86.08 | 87.60 | 91.53 |  |

### 5.2 Evaluation on Report Generation

In report generation, the model produces text reports based on information extracted from 3D medical images. For evaluation, we use the same test set from the image-text retrieval task. We set 2000 high-quality image-text pairs as the basic test set and set 1000 image-text pairs as a small test set to facilitate user evaluation. All experiments are tested on the basic test set.
Due to the challenge of assessing content accuracy between generated reports and human references, we employ two types of metrics: traditional and LLM-based metrics. Traditional metrics include BLEU (Bilingual Evaluation Understudy)[], ROUGE (Recall-Oriented Understudy for Gisting Evaluation)[], METEOR (Metric for Evaluation of Translation with Explicit Ordering)[], and BERT-Score[]. These metrics measure aspects like n-gram overlap or variations to quantify text similarity, albeit with limited semantic understanding.
LLM-based metrics utilize LLMs with robust semantic understanding as evaluators, such as the public Qwen-72B. These metrics assess the aspects mentioned in generation and human reference and calculate the percentage of correct or partially matched aspects, assigning a score from 0 to 100.

Figure: Figure 4: Comparison on 3D positioning (REG & REC) with frozen or unlocked vision encoder in fine-tuning. Unlocking the vision encoder can improve the performance of the output box, the REG task.

**Table 6: Comparison on 3D segmentation. Our model not only surpasses previous methods in semantic segmentation tasks but also achieves RES tasks previously unattainable. ACT-1K: AbdomenCT-1K. TS: ToalSegmentator. CTOrg: CT-Oran.**
| Methods | Semantic Segmentation | Referring Expression Segmentation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ACT-1K[] | TS[] | CTOrg[] | ACT-1K [] | TS [] | CTOrg [] |  |
| SegVol [] | 79.06 | 44.28 | 77.78 | - | - | - |
| Our (Linear) | 73.64 | 58.30 | 81.27 | 73.63 | 58.50 | 83.48 |
| Our (MLP) | 73.37 | 59.70 | 81.01 | 73.36 | 72.38 | 83.49 |

**Table 7: Ablation study on closed-ended VQA.**
| Vision<br>pre-train | Spatial<br>pooling | MLP | Unlocked<br>vision | VQA<br>Mean |
| --- | --- | --- | --- | --- |
| ✘ | ✔ | ✔ | ✘ | 71.13 |
| ✔ | ✘ | ✔ | ✘ | 72.87 |
| ✔ | ✔ | ✘ | ✘ | 73.50 |
| ✔ | ✔ | ✔ | ✘ | 74.96 |
| ✔ | ✔ | ✔ | ✔ | 75.78 |

### 5.3 Evaluation on VQA

The VQA tasks involve generating text answers related to given images and questions, which typically fall into two categories: open-ended and closed-ended. In open-ended VQA, the model generates answers without constraints, while closed-ended VQA restricts the acceptable answers to a predefined set of choices.
To evaluate both forms, we organize M3D-VQA into multiple-choice questions with four answer choices (A, B, C, D). Subsequently, we provide two test sets with different sizes: the basic test set and the small test set. The basic test set comprises 2000 3D medical images and 13,791 question-answer pairs covering five types. The small test set comprises 1000 3D medical images and 5000 question-answer pairs covering five types. The results of this paper are based on the basic test set, and the small test set is prepared for faster and low-cost evaluation by users. After self-filtering, which removes erroneous or low-quality data, a combination of LLM and expert check is employed to identify issues, achieving a pass rate of 99.4%.
For closed-ended VQA, questions and choices serve as prompt inputs, and answers are supervision signals. The accuracy is directly computed by answers and choices. For open-ended VQA, the model is trained using questions and corresponding correct answers. Evaluation involves measuring the similarity between model-generated texts and references using metrics such as BLEU, ROUGE, METEOR, and BERT-Score.

### 5.4 Evaluation on Positioning

Positioning is crucial in vision language tasks [], especially those involving input and output boxes. Tasks involving output boxes, like Referring Expression Comprehension (REC) [, ], aim to localize a target object in an image based on a referring expression. In contrast, tasks with input boxes, such as Referring Expression Generation (REG) [], require the model to generate a description of a specific region given an image and a location box. In our M3D-RefSeg and M3D-Seg, the masks are converted into box coordinates representing the maximum bounding rectangle $(x_{1},y_{1},z_{1},x_{2},y_{2},z_{2})$. In evaluation, 20% of the data in AbdomenCT-1K [] from M3D-Seg is used as the test set. For tasks involving output boxes, we computed the Intersection over Union (IOU) between the output boxes and ground truth as the localization metric. The IOU score greater than 0.2 is considered as hitting the target, and positioning accuracy is reported accordingly. For tasks with input boxes, BLEU, ROUGE-L, METEOR, and BERT-Score are calculated as metrics for evaluating generated descriptions and answers.

Figure: Figure 5: Qualitative comparisons with our model and ground truth on image-text retrieval and report generation. The same colors marked in prediction and answer represent similar content.

Figure: Figure 6: Qualitative comparisons on closed-ended and open-ended VQA. Closed-ended VQA requires the model to pick the correct answer from the input options, while open-ended VQA has no option as input.

Figure: Figure 7: Qualitative comparisons on positioning and segmentation. The green box (mask) in the 3D image indicates the ground truth, while the red box (mask) indicates the prediction.

Figure: Figure 8: Case study on out-of-distribution (OOD) questions. We test the M3D-LaMed model on OOD dialogue, which means that all questions are NOT related to our training data. We found that M3D-LaMed has strong generalization and can produce reasonable answers to OOD questions instead of gibberish. In each set of conversations, the avatar and questions on the left come from the user, and the avatar and answers on the right come from M3D-LaMed.

### 5.5 Evaluation on Segmentation

The segmentation task is vital in 3D medical image analysis due to its recognition and localization capabilities. To address various text prompts, segmentation is categorized into semantic segmentation and referring expression segmentation. For semantic segmentation, the model generates segmentation masks based on semantic labels. Referring expression segmentation involves segmenting targets described by natural language expressions using the model. In M3D-Seg, 20% of the data from AbdomenCT-1K[], Totalsegmnetator [], and CT-Organ [] is allocated as the test set for both semantic segmentation and referring expression segmentation. Traditionally, Dice is employed as a metric.

## 6 Experiments

### 6.1 Implementation Details

We apply Min-Max Normalization consistently to preprocess the 3D CT images as input. Additionally, we resize and crop the 3D images to a standardized size of $32\times 256\times 256$. Our 3D vision encoder adopts a 3D ViT, using 12 layers of transformers with a patch size of $4\times 16\times 16$. The output embeddings are $2048\times 768$, representing 2048 tokens with 768 feature dimensions. After our 3D spatial pooling perceiver, the final vision tokens fed to the LLM are $256\times 768$. We utilize the LLaMA-2-7B as an LLM base and load pre-trained parameters.

In vision encoder pre-training, we employ BERT with 12 layers of transformers as the text encoder, with a maximum text length of 128. The $[CLS]$ tokens of both the vision encoder and text encoder serve as the global feature representation, and a linear layer is utilized for cross-modality contrastive training projection. Additionally, we adopt a batch size of $6\times 8$ for parallel training across 8 GPUs, with a learning rate of $10^{-4}$ and warm-up and cosine decay schedules.

There are two stages in MLLM training. Initially, we freeze the vision encoder and LLM and solely fine-tune the 3D perceiver with image-text pairs, employing a batch size of $12\times 8$, a learning rate of $10^{-4}$, and applying warm-up and cosine decay. Specifically, we explore two situations: 1-layer linear and 2-layer MLP in the perceiver. Subsequently, we fine-tune the vision encoder, 3D perceiver, LLM, and segmentation modules with instruction data, utilizing a batch size of $12\times 8$, a learning rate of $2\times 10^{-5}$, and employing warm-up and cosine decay. The segmentation module uses the parameters of SegVol[] as initialization. We consistently apply the parameter-efficient LoRA method for LLM fine-tuning, with LoRA parameters set to $r=16$, $\alpha=32$, and a dropout rate of 0.1. The maximum context length is defined as 512.

All our models are trained using the AdamW [, ] optimizer and leverage the bf16 Mixed-precision Training Strategy enabled by DeepSpeed to optimize the training process. Implementation is carried out in PyTorch, and training occurs in parallel across 8 NVIDIA A100 GPUs with 80 GB memory.

### 6.2 Results on Image-Text Retrieval

Since 3D CLIP-like models do not exist in medical image analysis, we consider using 2D models as baselines. During the evaluation, we sample 10 equally spaced 2D slices for each 3D image along the depth dimensions. When using a 2D model to perform retrieval on slices, we identify the 3D image whose slice has the highest similarity to the target. We attempted to use CLIP[], but it performed poorly in the medical domain with scores close to 0. Hence, we chose PMC-CLIP[] as the baseline model in the medical domain.
In Table 2, we find that the retrieval performance of our model far exceeds that of the 2D PMC-CLIP model at various difficulty levels, which lacks spatial information. For example, under the simplest settings (100 test samples and R@10 metric), our model outperforms PMC-CLIP by 54% in IR. Even under the most difficult setting (2000 samples and R@1 metric), our model exceeds PMC-CLIP by 17.95% in IR. Figure 5 qualitatively demonstrates that text retrieved based on images has similar content.

### 6.3 Results on Report Generation

Table 3 compares performance with RadFM and our model using Linear or MLP in the perceiver. Regardless of whether using traditional or LLM-based metrics, our model outperforms RadFM in report generation. For example, our model with MLP exceeds RadFM by 2.92% in terms of BLEU score. Based on the more accurate evaluation using the LLM Qwen-72B, our model even surpasses RadFM by 4.17%. Moreover, our model’s performance with MLP is slightly better than with Linear. Figure 5 qualitatively demonstrates the powerful generative capabilities of our model, in which the reports we generate have more in common with the correct answers.

### 6.4 Results on VQA

We assess the performance of the M3D-LaMed model on closed-ended and open-ended VQA utilizing the M3D-VQA dataset. In Table 4, for closed-ended VQA, our model performs far better than RadFM on all five types of problems. Referring to Table 5 for open-ended VQA, our model also significantly outperforms RadFM. In Figure 6, we qualitatively demonstrate our model’s understanding capabilities on open-ended VQA and closed-ended VQA.

### 6.5 Results on Positioning

Figure 4 depicts the evaluation of the 3D vision-language positioning task, considering two subtasks, REG (Referring Expression Generation) for output with a box and REC (Referring Expression Comprehension) for input with a box, respectively. We compare the frozen and unlocked vision encoders during fine-tuning. We found that the unlocked vision encoder significantly improved the REG task. For example, The accuracy increased by 29.25%, which requires generating box output from the 3D image. However, in the REC task, the unlocked vision encoder does not achieve consistent performance improvements. In Figure 7, we provide a qualitative showcase of our model’s prowess in the vision language positioning task.

### 6.6 Results on Segmentation

In Table 6, we evaluate the 3D segmentation task, considering SS (Semantic Segmentation) and RES (Referring Expression Segmentation). Leveraging the comprehension capabilities of multi-modal large models, our performance surpasses SegVol in several aspects. Furthermore, we possess RES capabilities that SegVol lacks. In Figure 7, we showcase the segmentation capabilities of our model qualitatively.

### 6.7 Ablation Study

In Table 7, we conduct ablation studies in our closed-set VQA task on four aspects: vision pre-train, spatial pooling, MLP, and unlocked vision. Omitting vision pre-training involves training from scratch. Omitting spatial pooling entails direct pooling on sequence tokens. Omitting MLP entails replacing it with a single linear layer. Omitting unlocked vision involves freezing the vision encoder during fine-tuning. Through detailed ablation experiments, we found that every part of the model is irreplaceable, and using visual pre-training as a starting point and unlocking the visual encoder in fine-tuning is a better training solution.

### 6.8 Case Study on OOD Questions

We aim to investigate the generalization ability of our model, specifically its capacity to answer OOD questions beyond the scope of the training set. To this end, we devise unconventional problems, illustrated in Figure 8. For instance, in a CT scan of the chest and abdomen, our model identifies the appendix as the smallest organ, a concept absent from the training data. Similarly, when presented with the grammatically awkward query ”smartest organ”, the model aptly responded with ”Brain”, despite this phrase not being part of the training data.
While our dataset already contains questions describing anomalies, we imposed stricter constraints, such as limiting queries to one, three, and five words. Remarkably, our model successfully addresses these constraints, even though it had not been explicitly trained for such scenarios. Furthermore, when presented with queries related to surgical planning or seeking life advice, the model generates pertinent responses, demonstrating its adaptability beyond the confines of the training data. In summary, our M3D-LaMed model exhibits robust generalization capabilities for OOD problems. This proficiency stems from our approach of performing lightweight LoRA fine-tuning on LLM rather than full-parameter fine-tuning, which preserves the LLM’s original understanding and knowledge. By leveraging LLM’s inherent capabilities and fine-tuning on new multi-modal datasets, MLLM emerges with enhanced professional and generalization capabilities. Hence, developing a medical MLLM based on a robust LLM foundation proves indispensable.

## 7 Conclusion

In conclusion, our study advances 3D medical image analysis with MLLM. Specifically, we construct a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K 3D image-text pairs and 662K instruction-response pairs tailored for 3D medical tasks. Additionally, we propose M3D-LaMed, a generalist model handling image-text retrieval, report generation, visual question answering, positioning, and segmentation. Furthermore, we introduce a comprehensive benchmark, M3D-Bench, meticulously designed for eight tasks. We assert that our approach establishes a robust foundation for MLLMs to comprehend the vision and language of 3D medical scenarios. The availability of our data and code will facilitate further exploration and application for 3D medical MLLM in future research.

## Appendix A Implementation Details

### A.1 Modules

We introduce our 3D spatial pooling perceiver in Figure 1. This architecture simplifies and efficiently reduces the number of 3D image tokens while aligning dimensions, enabling the injection of 3D image information into LLM.

Detailed module parameters of the M3D-LaMed model are presented in Table 1. Specifically, we explore two situations: 1-layer linear and 2-layer MLP. The overall model parameters amount to 6.9 billion, considerably smaller than RadFM[], which has 13 billion parameters. Although the LLM base constitutes 97% of all parameters, fine-tuning LLM with just LoRA during training is exceptionally cost-effective.

Figure: Figure 1: The architecture of the 3D spatial pooling perceiver. (1) To reduce the number of tokens, input tokens are first reconstructed into 3D space for pooling and then deconstructed into sequence tokens. (2) To align the dimensions, A projection layer, like linear or MLP, converts the sequence tokens to the same dimension as the LLM. The effect of different projection layers, linear and MLP, on performance is reported in our experiment.

**Table 1: The parameters of each module in our M3D-LaMed. Linear means using a 1-layer linear in the perceiver for dimensional alignment, while MLP means using a 2-layer MLP for alignment. We utilize 3D ViT with a 12-layer transformer as a 3D image encoder, LLaMA-2-7B[] as an LLM base, and SegVol[] as a segmentation module.**
| Modules | Parameters |
| --- | --- |
| 3D Image Encoder | 87.4M |
| 3D Spatial Pooling Perceiver<br>(Linear/MLP) | 3.1M/19.9M |
| LLM with LoRA | 6.7B |
| Segmentation Module | 117.3M |
| All | 6.9B |

**Table 2: Detailed dataset composition in M3D-Seg. M3D-Seg contains 5,772 labeled 3D CTs from 25 public datasets. All data, download links, and processing scripts will be made public. CAT: Category.**
| Datasets | Anatomical Targets | CAT | Train | Test | All |
| --- | --- | --- | --- | --- | --- |
| 3D-IRCADB[] | Liver and liver tumor | 47 | 16 | 4 | 20 |
| FLARE22[] | Thoracic and abdominal organs | 13 | 40 | 10 | 50 |
| AbdomenCT-1k[] | Liver, kidney, spleen, pancreas | 4 | 800 | 200 | 1000 |
| AMOS22[] | Abdominal organs | 15 | 192 | 48 | 240 |
| BTCV[] | Abdominal organs | 13 | 24 | 6 | 30 |
| CHAOS[, , ] | Abdominal organs | 1 | 16 | 4 | 20 |
| CT-ORG[, , , ] | Organs of the body | 6 | 112 | 28 | 140 |
| HaN-Seg[] | Organs of the head and neck | 30 | 33 | 9 | 42 |
| KiPA22[, , , ] | Kidney, renal tumor, artery, vein | 4 | 56 | 14 | 70 |
| KiTS19[] | Kidney and kidney tumor | 2 | 168 | 42 | 210 |
| KiTS23[] | Kidney, kidney tumor and cyst | 3 | 391 | 98 | 489 |
| LUNA16[] | Left lung, right lung, trachea | 3 | 710 | 178 | 888 |
| MSD-Colon[] | Colon tumor | 1 | 100 | 26 | 126 |
| MSD-HepaticVessel[] | Hepatic vessel and liver tumor | 2 | 242 | 61 | 303 |
| MSD-Liver[] | Liver and liver tumor | 2 | 104 | 27 | 131 |
| MSD-Lung[] | Lung tumor | 1 | 50 | 13 | 63 |
| MSD-Pancreas[] | Pancreas and pancreas tumor | 2 | 224 | 57 | 281 |
| MSD-Spleen[] | Spleen | 1 | 32 | 9 | 41 |
| Pancreas-CT[, , ] | Pancreas | 1 | 65 | 17 | 82 |
| QUBIQ[] | Kidney, pancreas and lesion | 3 | 65 | 17 | 82 |
| SLIVER07[] | Liver | 1 | 16 | 4 | 20 |
| TotalSegmentator[] | Organs of the whole body | 104 | 962 | 241 | 1203 |
| VerSe19[, , ] | Vertebrae | 28 | 64 | 16 | 80 |
| VerSe20[, , ] | Vertebrae | 28 | 48 | 13 | 61 |
| WORD[] | Thoracic and abdominal organs | 16 | 80 | 20 | 100 |
| Sum | - | - | 4610 | 1162 | 5772 |

### A.2 Datasets

In addition to the detailed introduction of our datasets in Section 3 of the manuscript, we provide additional details of the M3D-Seg dataset in Table 2. M3D-Seg comprises 5,772 3D CTs and their corresponding masks collected from 25 public segmentation datasets. These datasets offer training and evaluation data for vision language positioning and segmentation tasks.

## Appendix B Prompts and Templates

In our work, we frequently utilize various prompts and templates to guide LLM in different tasks, including data generation, data checking, model evaluation, and task instruction. Figures 2 and 3 depict prompts for data generation, while Figure 4 illustrates prompts used for assessing the quality of generated VQA data. The evaluation of model performance using LLMs is demonstrated in Figure 5. Additionally, instruction templates for different tasks are provided in Figures 6, 7, 8, and 9. Figure 10 presents samples from our term dictionary.

Figure: Figure 2: The prompt of VQA data generation. Specifically, we insert the image file name and report text into the placeholders ({}) within the prompt and feed it to LLM. Subsequently, we post-process the output of LLM to extract VQA data. Additionally, we observed that Qwen-72B[] and ChatGPT[] perform similarly in our data generation experiments, leading us to adopt the more cost-effective Qwen-72B model.

Figure: Figure 3: The prompt of data generation for referring expression segmentation. Specifically, we insert the report description for a mask into the placeholders ({}) within the prompt and feed it to LLM. Subsequently, we post-process the output of LLM to extract instructions. This involves generating diverse descriptions and inferential questions from simple diagnostic reports. Qwen-72B is selected for this task due to its efficiency and performance.

Figure: Figure 4: The prompt for VQA data checking. Manual verification of large datasets is highly resource-intensive, especially in medical scenarios. Therefore, we employ LLM to automatically check the generated data. Specifically, we insert the generated data (questions, choices, and answers) and original data (image file names, report text) into the placeholders ({}) within the prompt and feed them to LLM. Subsequently, we assess the validity of the data by analyzing LLM’s output. Qwen-72B is selected for this task due to its efficiency and performance.

Figure: Figure 5: The prompt for model evaluation. Traditional metrics struggle to evaluate the model’s performance semantically. Therefore, we introduce an LLM-based metric for evaluation. In detail, we insert ground truth and predictions into the placeholders ({}) within the prompt and input them to LLM. Subsequently, we extract the absolute scores directly from the output of the LLM. Qwen-72B is chosen for this task due to its efficiency and performance.

Figure: Figure 6: Examples of instructions for report generation. These instructions typically include prompts or guidelines for generating specific sections or content within the medical reports. These instructions, along with corresponding images, are input into the MLLM together to facilitate the report generation process.

Figure: Figure 7: Instruction templates for referring expression comprehension. These templates guide the construction of instruction data for the referring expression comprehension task. The data for this task is sourced from M3D-Seg, a segmentation dataset providing categories and bounding boxes. In category questions, categories are inserted into question templates’ placeholders ({}) as input, while bounding boxes are inserted into answer templates’ placeholders ({}) as output. In description questions, categories are converted into descriptions using the term dictionary. These instruction templates facilitate the generation of instruction data for referring expression comprehension.

Figure: Figure 8: Instruction templates for referring expression generation. These templates facilitate the construction of instruction data for the referring expression generation task. In category questions, bounding boxes are inserted into question templates’ placeholders ({}) as input, while categories are inserted into answer templates’ placeholders ({}) as output. Similarly, in description questions, categories are converted into descriptions using the term dictionary. The model is expected to output both the target and its description as answers.

Figure: Figure 9: Instruction templates for segmentation tasks. In semantic segmentation, categories are inserted into question templates’ placeholders ({}) as input. For referring expression segmentation, descriptions are inserted into question templates’ placeholders ({}) as input. In both cases, all answers include a special token [SEG], which instructs the segmentation module. This token is crucial for guiding the segmentation process based on the provided input.

Figure: Figure 10: Examples from the term dictionary. The term dictionary contains multiple descriptions for each medical term. These descriptions are generated through ChatGPT. With numerous medical terms included, this dictionary is crucial in transforming semantic categories into detailed descriptions. These descriptions are essential for facilitating vision language positioning and segmentation tasks.

## Appendix C Qualitative Analysis

To further demonstrate our model’s performance and generalist ability on 3D multi-modal medical tasks, we add additional qualitative analysis on 8 tasks: image-text retrieval (Figure 11), report generation (Figure 12), closed-ended VQA (Figure 13), open-ended VQA (Figure 14), referring expression comprehension (Figure 15), referring expression generation (Figure 15), semantic segmentation, (Figure 16) and referring expression segmentation (Figure 16).

Figure: Figure 11: Qualitative comparisons on image-text retrieval. In each case, text with the same color represents identical content, while text with different colors signifies different content. The top-ranked samples exhibit similarities to ground truth (GT) in both image-to-text retrieval (IR) and text-to-image retrieval (TR) tasks.

Figure: Figure 12: Qualitative comparisons with another method in report generation. Text with the same color indicates identical content, while different colors signify differing content. Our model exhibits superior performance to RadFM by generating more answer-identical content. We attempted to include GPT-4V in the comparison but encountered limitations, as it struggled to generate medical-related diagnostic recommendations.

Figure: Figure 13: Qualitative comparisons with other methods in closed-ended VQA. We compare methods across five types: plane, phase, organ, abnormality, and location, highlighting the superiority of our approach. In closed-ended VQA, GPT-4V sometimes relaxes the restrictions, because we provide choices that may turn medical advice into a multiple-choice question.

Figure: Figure 14: Qualitative comparisons with other methods in open-ended VQA. Similarly, our method demonstrates superior performance across five types. However, questions related to abnormality topics in open-ended VQA remain restricted by GPT-4V. In cases where no valid answer can be obtained, ”-” is used to indicate this limitation.

Figure: Figure 15: Qualitative analysis on positioning tasks. We demonstrate two task forms: box output and box input, representing referring expression comprehension and referring expression generation, respectively. This demonstrates our model’s effectiveness in completing the vision language positioning task. In the visualizations, the green box represents the ground truth, while the red box represents the prediction.

Figure: Figure 16: Qualitative analysis on segmentation tasks. We show two task forms: semantic segmentation and referring expression segmentation, highlighting our model’s proficiency in segmentation tasks. In the visualizations, the green mask represents the ground truth, while the red mask represents the prediction.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: This is a really interesting paper and I think it is a great contribution on how to adapt 3D medical image features into LLMs, but I have a few concerns on the benchmarking part:
1) For image-text retrieval task, you only perform PMC-CLIP as one of your baselines. However, it should demonstrate your 3D model's performance is outperformed 2D aggregation baselines then we should move on to 3D. There is still a lot of embedding generating approach and use embeddings to perform image-to-text / text-to-image retrieval, such as:
- Zhang, Sheng, et al. "BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs." arXiv preprint arXiv:2303.00915 (2023).
- Zhang, Kai, et al. "Biomedgpt: A unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks." arXiv e-prints (2023): arXiv-2305.

2) For 3D segmentation task, you also only perform SegVol as one your segmentation baseline. Still, there is a lot of 3D baselines out there e.g. SwinUNETR, 3D UX-Net, nnUNet. Also, as you have done pretraining with CLIP, you should also demonstrate that your finetuned model with LLM is better than purely supervised models. Right now, your experiments for Table 6 is like ablation studies, changing different LLMs to see the difference in segmentation performance and search for the best LLMs. It will be great if you can add some of your experiments into this paper and their code is already public for usage:
- Hatamizadeh, Ali, et al. "Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images." International MICCAI brainlesion workshop. Cham: Springer International Publishing, 2021.
- Lee, Ho Hin, et al. "3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation." arXiv preprint arXiv:2209.15076 (2022).
- Isensee, Fabian, et al. "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation." Nature methods 18.2 (2021): 203-211.
soundness: 3
strengths: The strength of this paper can be concluded as follows:
1) Significant contribution on creating a public dataset for different tasks, as there is limited datasets that can be used in public, especially referring expression segmentation and image-text retrieval
2) This paper proposes a new model completely focused on 3D and demonstrate the success of using simple way to adapt 3D features into large language models (LLMs) for finetuning with LORA, which shows an interesting contribution on how to link 3D features with LLMs
3) The generalist model demonstrates a great variety on performing different tasks, showing a powerful generalizability of the learned semantics inside the foundation model
summary: Due to the expensive training cost and the limited of text-labeled data, limited research have been proposed to train a 3D medical image model direct for analysis, and more focused on 2D image slices. Therefore, this paper proposed an interesting idea to tackle 3D medical image and has three main contributions:
1) Proposed a general medical foundation model M3D-LaMed that can handle 8 3D medical task, including image-text retrieval, report generation, visual question answering, positing and segmentation.
2) Proposed a new dataset for public usage with 120K image-text and 662K instruction-response task
3) Performed extensive benchmarking with the current foundation models and the proposed generalist model
weaknesses: The weakness of this paper can be concluded as follows:
1) As the model is focused on using 3D image analysis, there is a lot of 3D foundation backbone proposed such as 3D swin transformer, 3D large kernel convolution, there is limited discussion in this paper on why it is important to use ViT here.
2) As the paper benchmarks the performance across different models, only a few baselines comparisons is performed with the adjustment on LLMs and one model with respect to the corresponding task.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: •	How each of the three components in M3D-LaMed contribute to performance?
•	Could your model include real-word metrics and how could we assess its impact in clinical settings?
•	How generalisable is the current model to out-of-domain data, and other modalities such as MRI and ultrasound?
soundness: 3
strengths: •	M3D-Data is an interesting contribution to the community. This large-scale dataset enables further research into 3D vision and language for medical applications.
•	The addition of the spatial pooling perceiver in the 3D image encoder proves performant.
•	Evaluation: M3D-LaMed is thoroughly evaluated across several tasks strengthening the validity and usefulness of the current architecture. Several LLM evaluation metrics are used to quantitatively assess the results
summary: The paper proposes M3D-LaMed, a general model tailored for 3D medical image analysis. It includes a specialized 3D vision encoder, a 3D spatial pooling perceiver, and a large language model (LLM). Moreover, the authors propose M3D-Data, a  3D medical image-text pairs that can be used for several tasks i.e segmentation,  report generation, VQA, positioning, and image-text retrieval.  Comparisons with baselines models (PMC-CLIP and RadFM) reveal the superiority of M3D-LaMed.
weaknesses: •	Computational Efficiency: handling 3D images and text is known to be computationally demanding. Authors could have developed more on the problems that could arise if using longer text sequences or higher resolution images.
•	How well M3D-LaMed works on other medical imaging modalities?
•	The architecture of M3D-LaMed includes a 3D image encoder, a perceiver, and LLM but the interaction and importance of those three components is not fully explained.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could you specify the resizing method used to preprocess 3D CT images to the standard dimension of 32×256×256? For example, was a specific interpolation technique applied, or were aspect ratios adjusted?

Did you compare the 3D Perceiver module directly with a simpler alternative, like an MLP, to assess its contribution to token reduction and spatial information preservation?

Although the focus is on 3D image analysis, did you evaluate M3D-LaMed’s performance against state-of-the-art 2D models on similar tasks to establish a clearer benchmark for the benefits and limitations of 3D approaches?

Could you provide more detail on the demographic diversity (e.g., age, gender, ethnicity) and disease types included in M3D-Data? Was any effort made to ensure the dataset’s representativeness across these dimensions?

M3D-Data primarily consists of CT images. Did you consider evaluating M3D-LaMed’s performance across other 3D imaging modalities, such as MRI or PET, to assess the model’s robustness in handling different data types?

While Dice score is used to measure segmentation performance, did you evaluate M3D-LaMed’s segmentation accuracy with respect to clinically relevant boundaries or subtle anatomical variations?

Could you provide details on the computational requirements for deploying M3D-LaMed in clinical settings? How do its processing times compare to those of 2D models, and what are the implications for clinical feasibility?

Could you clarify the licensing and accessibility of M3D-Data and M3D-Bench? Will these resources be publicly available to the community, and if so, under what conditions?

Given that M3D-LaMed still faces challenges with high-resolution 3D data, do you foresee any future improvements to the model architecture or data processing to better handle long token sequences in 3D?
soundness: 3
strengths: M3D-LaMed represents a highly original application of multi-modal large language models (MLLMs) to the under-explored domain of 3D medical imaging. By moving beyond traditional 2D image analysis, it highlights the added value of 3D data in medical diagnostics, addressing a significant limitation in prior work.

The introduction of a 3D spatial pooling perceiver to efficiently process 3D tokens while maintaining spatial information is a novel approach that creatively adapts existing methods to suit the unique demands of 3D medical data.

M3D-LaMed’s generalist approach and performance across multiple complex tasks make it a significant contribution to the field of medical imaging, as it offers a versatile tool adaptable to various clinical applications.

The publicly available M3D-Data and M3D-Bench will serve as valuable resources for the broader community, facilitating the development and evaluation of future models in 3D medical image analysis.

By addressing spatial and contextual complexity in 3D images, M3D-LaMed enhances diagnostic capabilities, which could translate into better clinical decision-making and improved patient outcomes.
summary: The paper introduces M3D-LaMed, a generalist large language model designed for comprehensive 3D medical image analysis. The study tackles challenges in 3D medical imaging, such as data scarcity and high computational costs, which previously limited exploration in this field. M3D-LaMed is specialized for eight critical tasks, including image-text retrieval, report generation, visual question answering (VQA), positioning, and segmentation.

Key contributions of the paper include:
M3D-LaMed Model: A multi-modal large language model (MLLM) specifically designed for various 3D medical imaging tasks.
M3D-Data: The largest public 3D multi-modal medical dataset to date, containing 120,000 image-text pairs and 662,000 instruction-response pairs, covering a range of diseases and tasks.
M3D-Bench: The first 3D multi-modal benchmark that evaluates performance across eight 3D medical tasks, with specific metrics for image-text retrieval, VQA, positioning, and segmentation.
The results demonstrate that M3D-LaMed surpasses existing models across these tasks, establishing a robust framework for 3D medical image analysis and promoting further research in medical imaging applications​
weaknesses: Lack of Explanation for Resizing Method: The authors mention using Min-Max normalization and resizing 3D CT images to a standard dimension of 32×256×256, but the specific resizing technique used (e.g., interpolation method, aspect ratio considerations) is not clarified. Without this information, it is difficult to assess how the resizing might affect spatial integrity in 3D images, especially for subtle structures critical in medical analysis.
No Comparative Study on Resizing: It would be beneficial if the authors conducted a comparative study to evaluate the effects of different resizing methods on performance, as variations in preprocessing can impact model accuracy on medical data.

Lack of Comparative Analysis with MLP: The paper introduces a 3D Perceiver module to handle token reduction for efficient 3D processing, but it does not compare this approach directly against a simpler alternative, such as a Multi-Layer Perceptron (MLP). A controlled comparison would demonstrate the added value of the 3D Perceiver and validate its effectiveness in preserving spatial information while reducing computation.
No Ablation Study on the 3D Perceiver: Including an ablation study that isolates the impact of the 3D Perceiver module would strengthen the paper by empirically verifying its contribution to the model’s performance.

Limited Comparative Analysis with 2D Models: Although the paper focuses on 3D models, a more detailed comparison with state-of-the-art 2D models on similar tasks (e.g., segmentation or retrieval) would highlight the advantages and limitations of the 3D approach. By doing so, the authors could clarify whether the shift to 3D improves performance in specific areas, such as spatial recognition, and where it might face challenges, such as increased computational costs.

Lack of Discussion on Demographic and Disease Variety: While M3D-Data is substantial in size, the paper does not address whether it includes diverse demographic groups or a wide range of diseases, which are crucial for generalizable models in healthcare. An analysis of the dataset’s coverage across demographic and pathological categories would provide insight into its representativeness and the potential for biases.
Absence of Evaluation for Dataset Representativeness: Ensuring that M3D-Data includes representative samples across age, gender, and disease types would enhance the model’s applicability. Without this, there’s a risk that M3D-LaMed may perform less effectively on underrepresented groups, limiting its utility in real-world clinical applications.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: *	How was noisy VQA data eliminated? What was the rubric and evaluation criteria? The data quality did not include any analysis of open ended questions, which is likely of most importance of all tasks. Further, who were the experts used? Board certified radiologists?
*	Did interpolating the images to a lower resolution hurt performance of the model? This is a relevant ablation study. 
*	What metric is shown in Table 4?
*	It is not clear why natural language generation metrics should be used to assess open ended VQA when the correct answers are available in the closed form solution.
soundness: 2
strengths: *	Vision-language pretraining for 3D CT scans can be computationally prohibitive. This paper tackles this approach using a novel 3D ViT and perceiver-based approach. This allows for good image-text alignment. 
*	The proposed model has a nice unified training paradigm and interface across tasks
*	A variety of tasks are evaluated across task types, including VQA, segmentation, report generation, etc. Tasks like report generation and report generation currently have mid-to-high level clinical importance. 
*	Multiple examples of task performance is provided in appendix
summary: This paper presents the development of a 3D vision-language model focusing on the interpretation of 3D computed tomography imaging. A CLIP-guided multi-modal pretraining approach is proposed and evaluations are performed across a variety of clinically relevant and adjacent tasks. Some relevant baselines are included. No trained models or source code is provided.
weaknesses: *	For the training of their proposed model, the authors used the Radiopedia dataset. However, radiopedia explicitly prohibits scraping the data from their website for such uses. https://radiopaedia.org/terms . It is mentioned that "Radiopaedia is not designed or intended to be used as an imaging data-set for Machine Learning/Artificial Intelligence research, nor is it available for bulk download using an automated script or robot or any other process whereby many cases are downloaded." Copyrights to data is clearly a very active topic currently, and it is problematic that the model developed here does not abide by the original copyrights of a crowd-compiled dataset. Overall, it does not seem that the authors have the ability to download and freely distribute the dataset as claimed. 
*	No comparisons are performed with more relevant task specific models or more stat of the art 3D vision language models. From a baselines perspective, it would be beneficial to compare to the BiomedCLIP model for retrieval as well as training. For segmentation, using more common models like nnunet or swinunetr would be more fair. For general 3D VL training, there is no comparison to a couple of recent papers that are very related: 1. Hamamci et al. Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography. And  2. Blankemeier et al. Merlin: A Vision Language Foundation Model for 3D Computed Tomography 
*	The M3D-RefSeg is the TotalSegmentator dataset. All that was done as a mapping between class number and the text description. Rebranding it as a unique contribution is not appropriate. 
*	Table 1 describes a Texts column - it is unclear what this is. Tokens? Also, referring to segmentation masks seems unlikely to be considered as text since its a specific class anyway at the end of the day.
*	CLIP struggles with long context and fine grained learning. It does not seem that there was any additional method used besides the traditional CLIP in this study, which may be problematic because of the high density of text information and low variability of the images.
*	It is unclear to me what the value of the benchmarking positioning task is, when one could simple perform the segmentation for the same tissues anyway? Since the bounding box is generated from the segmentation label, there is limited new value provided.