m3d-clip-mri-ct-aligned

Fine-tuned vision encoder derived from GoodBaiBai88/M3D-CLIP, contrastively aligned (InfoNCE) to understand MRI volumes, using paired MRI-CT data from SynthRAD2025 (Task 1) and Gold Atlas Male Pelvis.

A frozen copy of the original GoodBaiBai88/M3D-CLIP vision encoder was used as the CT "teacher" throughout training; this checkpoint is the MRI "student", trained so that encode_image() on an MRI volume produces an embedding close to the original model's embedding of that same patient's CT.

Usage

Input format matches the original M3D-CLIP: shape (1, 32, 256, 256), values min-max normalized to [0, 1].

from transformers import AutoModel
model = AutoModel.from_pretrained("anuragpradhan/m3d-clip-mri-ct-aligned", trust_remote_code=True)
mri_embedding = model.encode_image(mri_volume_tensor)[:, 0]

Note: the text encoder was not fine-tuned and retains its original CT-report alignment from GoodBaiBai88/M3D-CLIP. This checkpoint is intended for MRI <-> CT embedding alignment, not MRI-text retrieval.

Verification (held-out patients)

Retrieval check: for each held-out MRI embedding, is the correct CT volume (same patient) actually its nearest neighbour among all held-out CT embeddings?

Metric	Value	Chance level
Top-1 accuracy	0.353	0.020
Top-5 accuracy	0.745	-
Median rank	2	-
n (val patients)	51	-