RepConTUR-1B

RepConTUR is a report-conditioned volumetric multimodal model for abdominal tumor segmentation. It reads a whole abdominal CT volume and either a short referring phrase or the full narrative radiology report, and segments liver, kidney and pancreas tumors natively in 3D. The same backbone also generates reports and answers questions about the scan.

• Code: https://github.com/Oncosoft-Research/RepConTUR
• Data: https://huggingface.co/datasets/Oncosoft-Research/RepConTUR
• Paper: RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation (link coming soon)

Model

Component	Detail
3D CT encoder	CT-ViT from CT-CLIP, pretrained with a CLIP objective on CT-RATE chest and Merlin abdominal image–report pairs; frozen
Visual tokens	Attentional pooler with 256 learnable queries, projected to the LLM width
Language model	InternVL2.5-1B, initialized from UniBioMed, adapted with LoRA (merged into this checkpoint)
Segmentation module	VoxTell decoder, prompted by the hidden state of the [SEG] token through a two-layer projection
Input grid	CT-ViT: 140 × 240 × 240 at (3.0, 1.5, 1.5) mm, HU clipped to [−1000, 1000]; VoxTell: RAS, 1 mm, z-scored

Tasks

Task	Input	Output
Referring segmentation (RES)	CT + phrase, e.g. "Please segment the tumor in hepatic segment 7."	[SEG] + 3D mask
Report-grounded segmentation (GSEG)	CT + narrative report + phrase	[SEG] + 3D mask
Radiology report generation (RRG)	CT	narrative report
Visual question answering (VQA)	CT + question	yes/no, short answer or multiple choice

The model was trained jointly on all four tasks.

Training data

AbdomenAtlas 3.0 Mini, restricted to the 3,798 CT volumes with a liver, kidney or pancreas tumor (3,425 train, 373 test). Tumor masks were decomposed into suborgan-level targets such as liver Couinaud segments using the structured reports. VQA pairs were generated from the structured and narrative reports and checked by a second language model. The reports in AbdomenAtlas 3.0 are synthetic, produced by the RadGPT pipeline, not written by radiologists. The annotations are released at Oncosoft-Research/RepConTUR; the CT volumes and masks come from AbdomenAtlas 3.0 Mini.

Results

In-domain test split; DSC is image-wise Dice, and R/P/F1 are instance-wise recall, precision and F1 at IoU ≥ 0.10. Means are macro averages over the three organs, with 463 liver, 251 pancreas and 524 kidney GT-positive cases.

Organ	Mode	DSC	R	P	F1
Liver	RES	0.522	0.616	0.369	0.461
	GSEG	0.542	0.611	0.424	0.500
Pancreas	RES	0.386	0.624	0.336	0.437
	GSEG	0.399	0.644	0.376	0.475
Kidney	RES	0.621	0.781	0.487	0.600
	GSEG	0.627	0.775	0.501	0.609
Mean	RES	0.510	0.674	0.397	0.499
	GSEG	0.523	0.677	0.433	0.528

On the language tasks over the same 373 test volumes, overall VQA exact-match accuracy is 0.732, and RRG reaches BERTScore-F1 0.593 and ROUGE-L 0.377.

Usage

The model ships its own modeling code, so it loads with trust_remote_code=True. The CT preprocessing and prompt templates live in the GitHub repository, so set up the environment there first:

git clone https://github.com/Oncosoft-Research/RepConTUR.git && cd RepConTUR
bash setup_env.sh                      # uv environment under projects/repcontur
source projects/repcontur/.venv/bin/activate

Interactive demo:

python -m projects.repcontur.gradio.app Oncosoft-Research/RepConTUR

Python, run from the repository root:

from projects.repcontur.gradio.app import build_runtime, load_case, strip_response
from projects.repcontur.gradio.task_prompts import Task, build_task_prompt

runtime = build_runtime('Oncosoft-Research/RepConTUR')         # model, tokenizer and both CT preprocessors
case = load_case(runtime, 'ct.nii.gz')                 # any 3D abdominal CT in NIfTI
report_text = open('report.txt').read()                # the narrative report for GSEG

prompt = runtime.image_token_string + '\n' + build_task_prompt(
    Task.GSEG, 'tumor in the right kidney', report_text)
input_ids = runtime.tokenizer.encode(
    f'<|user|>\n{prompt}<|end|>\n<|assistant|>\n', add_special_tokens=False)

output = runtime.model.predict_volume(
    pixel_values=case.pixel_values.cuda(), input_ids=input_ids, g_pixel_values=case.grounding_volume)
text = strip_response(output['prediction'], runtime.model.template)   # '[SEG]' for segmentation tasks
mask = output['prediction_mask']                                       # (D, H, W) uint8 on the 1 mm RAS grid

Use Task.RES with an empty report for phrase-only segmentation, Task.VQA with a question, or Task.RRG for report generation. A single 24 GB GPU is enough for inference.

Fine-tuning

The repository's projects/repcontur/configs/repcontur_in25_1b_finetune.py starts from this model. It builds the InternVL2.5-1B backbone, loads all weights from this model's pytorch_model.bin (expected in pretrained/RepConTUR-1B), and attaches a fresh LoRA on top. The VoxTell decoder is built from this model's plans.json. Download the model first:

hf download Oncosoft-Research/RepConTUR --local-dir pretrained/RepConTUR-1B
bash tools/dist.sh train projects/repcontur/configs/repcontur_in25_1b_finetune.py 4

Limitations

• Research use only. The model is not a medical device and has not been validated for clinical decisions.
• Trained on a single public dataset with synthetic reports. Performance on other scanners, protocols, populations or real radiologist reports may differ.
• Only liver, kidney and pancreas tumors are supported.
• Generated reports and VQA answers can be wrong even when fluent; check them against the image.

License

Released under CC BY-NC-SA 4.0, the license of the AbdomenAtlas 3.0 data the model was trained on: attribution, non-commercial use only, and adaptations under the same license.

Citation

@article{lee2026repcontur,
  title  = {RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation},
  author = {Lee, Seunghwan and Park, Sangjoon and Lee, Jongmin and Kim, Yoon Jo and Cho, Wonyoung and Kim, Jin Sung and Chae, Han Joo},
  year   = {2026}
}

Acknowledgements

Built on Sa2VA, VoxTell, CT-CLIP / CT-RATE, InternVL, UniBioMed and AbdomenAtlas 3.0 / RadGPT.