μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Authors: name: Alejandro Lozano; name: Jeffrey Nirschl; name: James Burgess; name: Sanket Rajan Gupte; name: Yuhui Zhang; name: Alyssa Unell; name: Serena Yeung-Levy
Published: 2024-07-01 20:30:26

Abstract: Recent advances in microscopy have enabled the rapid generation of terabytes
of image data in cell biology and biomedical research. Vision-language models
(VLMs) offer a promising solution for large-scale biological image analysis,
enhancing researchers' efficiency, identifying new image biomarkers, and
accelerating hypothesis generation and scientific discovery. However, there is
a lack of standardized, diverse, and large-scale vision-language benchmarks to
evaluate VLMs' perception and cognition capabilities in biological image
understanding. To address this gap, we introduce {\mu}-Bench, an expert-curated
benchmark encompassing 22 biomedical tasks across various scientific
disciplines (biology, pathology), microscopy modalities (electron,
fluorescence, light), scales (subcellular, cellular, tissue), and organisms in
both normal and abnormal states. We evaluate state-of-the-art biomedical,
pathology, and general VLMs on {\mu}-Bench and find that: i) current models
struggle on all categories, even for basic tasks such as distinguishing
microscopy modalities; ii) current specialist models fine-tuned on biomedical
data often perform worse than generalist models; iii) fine-tuning in specific
microscopy domains can cause catastrophic forgetting, eroding prior biomedical
knowledge encoded in their base model. iv) weight interpolation between
fine-tuned and pre-trained models offers one solution to forgetting and
improves general performance across biomedical tasks. We release {\mu}-Bench
under a permissive license to accelerate the research and development of
microscopy foundation models.