No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language
  Models
Authors: name: Min Woo Sun, name: Alejandro Lozano, name: Javier Gamazo Tejero, name: Vishwesh Nath, name: Xiao Xiao Sun, name: James Burgess, name: Yuhui Zhang, name: Kun Yuan, name: Robert Tibshirani, name: Sean Huver, name: Serena Yeung-Levy

Embedding vision-language models (VLMs) are typically pretrained with short
text windows (<77 tokens), which forces the truncation of long-format captions.
Yet, the distribution of biomedical captions from large-scale open source
literature reveals that a huge portion of captions far exceed 77 tokens. To
this end, we investigate the impact of pretraining on long-format biomedical
captions by extending the context length of text encoders in VLMs. We find that
longer context (thus, enabling additional supervision provided in long-format
captions) correlates with better retrieval and classification performance.
Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M
image-caption pairs enriched with context-aware descriptions from full-text
articles, providing longer and additional textual supervision. Using
BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a
text encoder supporting windows of up to 512 tokens. Our model extends context
capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption
retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in
Recall@1 and +2% average improvements in classification, while also converging
faster than short-context. Our results demonstrate that long-context modeling
is a promising direction for advancing biomedical VLMs.