MedPLIB-BRISC

Joint brain tumor classification and segmentation from MRI with an adapted multimodal large language model.

This repository holds the weights we trained on top of MedPLIB-7b-2e for the BRISC 2025 brain tumor MRI dataset. Given an MRI slice and a fixed instruction, the model answers with one of four classes and a <SEG> token. The four classes are glioma, meningioma, pituitary tumor and non-tumorous. The hidden state of the <SEG> token is decoded into the tumor mask, so the class and the mask come from the same response.

Project page · Code on GitHub · Base model

Files

File	Content
trainable.pt	All trained weights, 286.8M parameters in total: LoRA on the language model, the image projector, the router, the <SEG> projector, the SAM-Med2D mask decoder and the SAM-Med2D encoder adapters
adapter_meta.json	LoRA settings and the list of trained tensors
assets/	Figures used on this page

These weights are not a complete model. They are loaded on top of the original MedPLIB-7b-2e checkpoint, CLIP ViT-L/336 and SAM-Med2D-B with the code from our GitHub repository.

How to use

git clone https://github.com/Peilin-FF/Tumor && cd Tumor
scripts/setup_env.sh                      # conda environment "medplib"
scripts/download_models.sh                # MedPLIB-7b-2e, CLIP ViT-L/336, SAM-Med2D-B
huggingface-cli download Sssunset/MedPLIB-BRISC --local-dir checkpoints/medplib-brisc

# interactive demo
python -m medplib_bt.demo_app --adapter checkpoints/medplib-brisc --port 7860

# reproduce the test results, which also needs scripts/download_data.sh
scripts/run_infer.sh 0,1,2,3 outputs/preds/medplib-brisc --adapter checkpoints/medplib-brisc

Training

The model is trained on the 5,000 training slices of the official BRISC 2025 split with 8 NVIDIA A100 GPUs.

Stage	Epochs	Trained modules	Trainable parameters	Loss
A: Alignment	1	image projector, router, <SEG> projector, mask decoder	43.1M, 0.37%	CE + 2 BCE + 0.5 Dice
B: Adaptation	12	Stage A modules, LoRA with rank 16 on attention and both experts in all 32 layers, SAM-Med2D encoder adapters	286.8M, 2.44%	CE + 2 BCE + 2 Dice + IoU

Half of the training prompts ask for the class and the mask, a quarter ask for the class only and a quarter ask for the mask only. Non-tumorous slices are trained with an empty mask. At test time, each of the four valid answers is scored by its log-likelihood and the best one is kept. The mask comes from the <SEG> state of that answer, and no rule removes it afterwards.

Results

Results on the 1,000 BRISC test slices, from one training run with seed 42:

Metric	Value
Accuracy	0.980
Macro-F1	0.982
Dice on the 860 tumor slices	0.854
Dice on all 1,000 slices	0.874
Masks drawn on the 140 non-tumorous slices	0
Slices where the class and the mask disagree	0

Two separately trained models, EfficientNet-B0 and U-Net, draw a mask on 61.4% of the non-tumorous slices and contradict each other on 9.0% of all slices. Trained on both tasks instead of segmentation alone, the same MedPLIB model stops drawing false masks entirely. The project page has the full comparison and case studies.

Limitations

• The model is trained and tested on 2D slices from one public dataset. The dataset has no patient identifiers, so slices from one patient may appear in both splits.
• A dedicated U-Net still outlines tumors slightly better on average. Gliomas and small lesions are the weakest cases.
• Free text answers became very short after adaptation.
• This is a course research prototype. It is not a medical device and must not be used for diagnosis or treatment decisions.

Team

GP8001 Group 3, Nanyang Technological University: Simon Tong Sing Hee, Zeng Yi, Feng Peilin, Ye Xiaomeng, Lyu Muyang, Timothy Aw Bang Hao.

Acknowledgements

This work builds on MedPLIB, SAM-Med2D and the BRISC 2025 dataset. We thank their authors for releasing code, weights and data.