Multi-Modal Disease Detection (research prototype)
EfficientNet-B4 image encoder + BioBERT text encoder with bidirectional cross-attention fusion, scoring 25 findings across chest X-rays (NIH ChestX-ray14), dermoscopy (HAM10000) and brain MRI. The paired text is real recorded metadata (age, sex, X-ray view, lesion site), never the label.
Held-out test results (patient-, lesion- and duplicate-disjoint splits, scored once):
| Mode | Chest macro AUROC | Skin balanced acc. | Skin macro AUROC | Brain accuracy | ECE |
|---|---|---|---|---|---|
| multimodal | 0.775 | 0.613 | 0.957 | 0.954 | 0.006 |
Full per-mode results, confidence intervals and the training history are in multimodal_config.json.
Code, training pipeline and evaluation: https://github.com/ARRY7686/Multi-Modal-Disease-Detection-System
Not a medical device. Trained on public research datasets and not clinically validated. Licensed CC BY-NC 4.0 because HAM10000 is; check each dataset's own terms before reuse.