SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context
  Learning
Authors: name: Melanie Rieff, name: Maya Varma, name: Ossian Rabow, name: Subathra Adithan, name: Julie Kim, name: Ken Chang, name: Hannah Lee, name: Nidhi Rohatgi, name: Christian Bluethgen, name: Mohamed S. Muneer, name: Jean-Benoit Delbrouck, name: Michael Moor

Multimodal in-context learning (ICL) remains underexplored despite
significant potential for domains such as medicine. Clinicians routinely
encounter diverse, specialized tasks requiring adaptation from limited
examples, such as drawing insights from a few relevant prior cases or
considering a constrained set of differential diagnoses. While multimodal large
language models (MLLMs) have shown advances in medical visual question
answering (VQA), their ability to learn multimodal tasks from context is
largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL
benchmark for medical tasks. Eleven medical experts curated problems, each
including a multimodal query and multimodal in-context examples as task
demonstrations. SMMILE encompasses 111 problems (517 question-image-answer
triplets) covering 6 medical specialties and 13 imaging modalities. We further
introduce SMMILE++, an augmented variant with 1038 permuted problems. A
comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit
moderate to poor multimodal ICL ability in medical tasks. In open-ended
evaluations, ICL contributes only 8% average improvement over zero-shot on
SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant
in-context examples: even a single noisy or irrelevant example can degrade
performance by up to 9.5%. Moreover, example ordering exhibits a recency bias,
i.e., placing the most relevant example last can lead to substantial
performance improvements by up to 71%. Our findings highlight critical
limitations and biases in current MLLMs when learning multimodal medical tasks
from context.