ELIXR: Towards a general purpose X-ray artificial intelligence system
  through alignment of large language models and radiology vision encoders
Authors: name: Shawn Xu; name: Lin Yang; name: Christopher Kelly; name: Marcin Sieniek; name: Timo Kohlberger; name: Martin Ma; name: Wei-Hung Weng; name: Attila Kiraly; name: Sahar Kazemzadeh; name: Zakkai Melamed; name: Jungyeon Park; name: Patricia Strachan; name: Yun Liu; name: Chuck Lau; name: Preeti Singh; name: Christina Chen; name: Mozziyar Etemadi; name: Sreenivasa Raju Kalidindi; name: Yossi Matias; name: Katherine Chou; name: Greg S. Corrado; name: Shravya Shetty; name: Daniel Tse; name: Shruthi Prabhakara; name: Daniel Golden; name: Rory Pilgrim; name: Krish Eswaran; name: Andrew Sellergren
Published: 2023-08-02 17:59:45

Abstract: Our approach, which we call Embeddings for Language/Image-aligned X-Rays, or
ELIXR, leverages a language-aligned image encoder combined or grafted onto a
fixed LLM, PaLM 2, to perform a broad range of tasks. We train this lightweight
adapter architecture using images paired with corresponding free-text radiology
reports from the MIMIC-CXR dataset. ELIXR achieved state-of-the-art performance
on zero-shot chest X-ray (CXR) classification (mean AUC of 0.850 across 13
findings), data-efficient CXR classification (mean AUCs of 0.893 and 0.898
across five findings (atelectasis, cardiomegaly, consolidation, pleural
effusion, and pulmonary edema) for 1% (~2,200 images) and 10% (~22,000 images)
training data), and semantic search (0.76 normalized discounted cumulative gain
(NDCG) across nineteen queries, including perfect retrieval on twelve of them).
Compared to existing data-efficient methods including supervised contrastive
learning (SupCon), ELIXR required two orders of magnitude less data to reach
similar performance. ELIXR also showed promise on CXR vision-language tasks,
demonstrating overall accuracies of 58.7% and 62.5% on visual question
answering and report quality assurance tasks, respectively. These results
suggest that ELIXR is a robust and versatile approach to CXR AI.