Towards Generalist Biomedical AI

Authors:
raw_author_name: Shekoofeh Azizi
raw_affiliation_string: Google DeepMind
raw_author_name: Danny Driess
raw_affiliation_string: Google DeepMind
raw_author_name: Mike Schaekermann
raw_affiliation_string: Google Research
raw_author_name: Mohamed Amin
raw_affiliation_string: Google Research
raw_author_name: Pi-Chuan Chang
raw_affiliation_string: Google Research
raw_author_name: Andrew Carroll
raw_affiliation_string: Google Research
raw_author_name: Ryutaro Tanno
raw_affiliation_string: Google DeepMind
raw_author_name: Ira Ktena
raw_affiliation_string: Google DeepMind
raw_author_name: Anil Palepu
raw_affiliation_string: Google Research
raw_author_name: Basil Mustafa
raw_affiliation_string: Google DeepMind
raw_author_name: Aakanksha Chowdhery
raw_affiliation_string: Google DeepMind
raw_author_name: Yun Liu
raw_affiliation_string: Google Research
raw_author_name: Simon Kornblith
raw_affiliation_string: Google DeepMind
raw_author_name: David Fleet
raw_affiliation_string: Google DeepMind
raw_author_name: Philip Mansfield
raw_affiliation_string: Google Research
raw_author_name: Sushant Prakash
raw_affiliation_string: Google Research
raw_author_name: Renee Wong
raw_affiliation_string: Google Research
raw_author_name: Sunny Virmani
raw_affiliation_string: Google Research
raw_author_name: Christopher Semturs
raw_affiliation_string: Google Research
raw_author_name: S. Sara Mahdavi
raw_affiliation_string: Google DeepMind
raw_author_name: Bradley Green
raw_affiliation_string: Google Research
raw_author_name: Ewa Dominowska
raw_affiliation_string: Google Research
raw_author_name: Blaise Aguera y Arcas
raw_affiliation_string: Google Research
raw_author_name: Joelle Barral
raw_affiliation_string: Google DeepMind
raw_author_name: Dale Webster
raw_affiliation_string: Google Research
raw_author_name: Greg S. Corrado
raw_affiliation_string: Google Research
raw_author_name: Yossi Matias
raw_affiliation_string: Google Research
raw_author_name: Pete Florence
raw_affiliation_string: Google DeepMind
raw_author_name: Vivek Natarajan
raw_affiliation_string: Google Research

Abstract: BackgroundMedicine is inherently multimodal, requiring the simultaneous interpretation and integration of insights between many data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence systems that flexibly encode, integrate, and interpret these data might better enable impactful applications ranging from scientific discovery to care delivery. MethodsTo catalyze development of these models, we curated MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks, such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduced Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. To further probe the capabilities and limitations of Med-PaLM M, we conducted a radiologist evaluation of model-generated (and human) chest x-ray reports. ResultsWe observed encouraging performance across model scales. Med-PaLM M reached performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. In a side-by-side ranking on 246 retrospective chest x-rays, clinicians expressed a pairwise preference for Med-PaLM Multimodal reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. ConclusionsAlthough considerable work is needed to validate these models in real-world cases and understand if cross-modality generalization is possible, our results represent a milestone toward the development of generalist biomedical artificial intelligence systems. (Funded by Alphabet Inc. and/or a subsidiary thereof.)