arxp/multimodal-clinical-reasoning

Multimodal Clinical Reasoning for Chest X-Ray Reports

Research card — no model weights or clinical system are released here.
This repository documents an academic deep-learning project and links to its reproducible notebooks, evaluation artifacts and report.

Status at a glance

Item	Availability
Research description	Available
Architecture and ablation results	Available
Training/evaluation notebooks	Companion GitHub repository
Model weights/checkpoints	Not released
MIMIC-CXR data	Not redistributed
Clinical use	Not permitted or validated

Project overview

The project studies image-conditioned radiology report generation with a frozen medical vision encoder and a language decoder. Cross-attention connects chest X-ray patch representations to generated report tokens.

Chest X-ray image → BiomedCLIP vision encoder → patch features
                                                    │
                                                    ▼
Report tokens → GPT-2 decoder with cross-attention → generated report

Four controlled variants examine whether attention regularization changes generation quality and alignment behavior:

Variant	Experiment
baseline	Cross-attention report generator without alignment regularization
entropy_only	Adds attention-entropy regularization
sparsity_only	Adds sparsity-oriented attention regularization
full_alignment	Combines entropy, sparsity and alignment losses

Evaluation snapshot

The companion repository reports held-out evaluation on 1,000 samples.

Variant	BLEU-1	METEOR	ROUGE-L	CIDEr	Finding overlap
baseline	0.1391	0.1359	0.1137	0.0099	0.6429
entropy_only	0.1370	0.1344	0.1108	0.0090	0.6732
sparsity_only	0.1301	0.1263	0.1094	0.0101	0.5606
full_alignment	0.1361	0.1336	0.1105	0.0080	0.6439

The regularized variants did not consistently outperform the baseline across text-generation metrics. The entropy-only variant produced the highest reported finding-overlap score, illustrating why multiple evaluation dimensions are necessary for medical report generation.

Contribution and attribution

This is a collaborative academic project. The source repository's recorded contribution history attributes the following work to Arda Çiftçi (arxp / OxyOxygen):

• VQA-RAD data cleaning and exploratory/statistical analysis;
• engineering 34 text, image and clinical features across 2,244 cleaned examples;
• dataset documentation and research-summary artifacts.

The public project later evolved into the MIMIC-CXR report-generation and ablation pipeline summarized above. This card distinguishes Arda's recorded contribution from the complete team's final system.

Intended use

This card is intended for:

• inspecting an academic multimodal deep-learning workflow;
• reviewing an ablation-study design;
• understanding the limits of automated radiology report generation;
• navigating to the reproducible notebooks and project report.

It is not an inference endpoint, diagnostic device, clinical decision-support system or substitute for a qualified radiologist.

Limitations and risks

• No clinical validation, prospective study or regulatory review was performed.
• Generated text may omit, hallucinate or misstate clinically important findings.
• BLEU, METEOR, ROUGE and CIDEr do not fully measure clinical correctness.
• Dataset composition and institutional practices may limit generalization.
• MIMIC-CXR access and redistribution remain subject to the dataset provider's terms.
• Checkpoints and source medical data are intentionally excluded from this Hugging Face repository.

Reproducibility resources

• Source repository and notebooks
• Project report
• Arda Çiftçi on GitHub
• Istanbul Mobility & Mood Explorer Space

Citation

If you reference this educational artifact, cite the companion GitHub repository and its contributors. This Hugging Face page is a research summary, not a released model package.