Capabilities of Gemini Models in Medicine
Authors: name: Khaled Saab, name: Tao Tu, name: Wei-Hung Weng, name: Ryutaro Tanno, name: David Stutz, name: Ellery Wulczyn, name: Fan Zhang, name: Tim Strother, name: Chunjong Park, name: Elahe Vedadi, name: Juanma Zambrano Chaves, name: Szu-Yeu Hu, name: Mike Schaekermann, name: Aishwarya Kamath, name: Yong Cheng, name: David G. T. Barrett, name: Cathy Cheung, name: Basil Mustafa, name: Anil Palepu, name: Daniel McDuff, name: Le Hou, name: Tomer Golany, name: Luyang Liu, name: Jean-baptiste Alayrac, name: Neil Houlsby, name: Nenad Tomasev, name: Jan Freyberg, name: Charles Lau, name: Jonas Kemp, name: Jeremy Lai, name: Shekoofeh Azizi, name: Kimberly Kanada, name: SiWai Man, name: Kavita Kulkarni, name: Ruoxi Sun, name: Siamak Shakeri, name: Luheng He, name: Ben Caine, name: Albert Webson, name: Natasha Latysheva, name: Melvin Johnson, name: Philip Mansfield, name: Jian Lu, name: Ehud Rivlin, name: Jesper Anderson, name: Bradley Green, name: Renee Wong, name: Jonathan Krause, name: Jonathon Shlens, name: Ewa Dominowska, name: S. M. Ali Eslami, name: Claire Cui, name: Oriol Vinyals, name: Koray Kavukcuoglu, name: James Manyika, name: Jeff Dean, name: Demis Hassabis, name: Yossi Matias, name: Dale Webster, name: Joelle Barral, name: Greg Corrado, name: Christopher Semturs, name: S. Sara Mahdavi, name: Juraj Gottweis, name: Alan Karthikesalingam, name: Vivek Natarajan

Excellence in a wide variety of medical applications poses considerable
challenges for AI, requiring advanced reasoning, access to up-to-date medical
knowledge and understanding of complex multimodal data. Gemini models, with
strong general capabilities in multimodal and long-context reasoning, offer
exciting possibilities in medicine. Building on these core strengths of Gemini,
we introduce Med-Gemini, a family of highly capable multimodal models that are
specialized in medicine with the ability to seamlessly use web search, and that
can be efficiently tailored to novel modalities using custom encoders. We
evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art
(SoTA) performance on 10 of them, and surpass the GPT-4 model family on every
benchmark where a direct comparison is viable, often by a wide margin. On the
popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves
SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search
strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU
(health & medicine), Med-Gemini improves over GPT-4V by an average relative
margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context
capabilities through SoTA performance on a needle-in-a-haystack retrieval task
from long de-identified health records and medical video question answering,
surpassing prior bespoke methods using only in-context learning. Finally,
Med-Gemini's performance suggests real-world utility by surpassing human
experts on tasks such as medical text summarization, alongside demonstrations
of promising potential for multimodal medical dialogue, medical research and
education. Taken together, our results offer compelling evidence for
Med-Gemini's potential, although further rigorous evaluation will be crucial
before real-world deployment in this safety-critical domain.