NeurIPS 2025

TROVE: Discovering Error-Inducing Static Feature
Biases in Temporal Vision-Language Models
Maya Varma1
Jean-Benoit Delbrouck1,2
Sophie Ostmeier1
Akshay Chaudhari1
Curtis Langlotz1
1Stanford University
2HOPPR
mayavarma@cs.stanford.edu
Abstract
Vision-language models (VLMs) have made great strides in addressing temporal
understanding tasks, which involve characterizing visual changes across a sequence
of images. However, recent works have suggested that when making predictions,
VLMs may rely on static feature biases, such as background or object features,
rather than dynamic visual changes. Static feature biases are a type of shortcut
and can contribute to systematic prediction errors on downstream tasks; as a
result, identifying and characterizing error-inducing static feature biases is critical
prior to real-world model deployment. Existing approaches for identifying such
systematic failure modes in trained models (i) are typically designed for non-
temporal settings and (ii) are challenging to evaluate in temporal settings due to
the lack of quantitative evaluation frameworks. In this work, we address these
challenges by introducing TROVE, an automated approach for discovering error-
inducing static feature biases learned by temporal VLMs. Given a trained VLM
and an annotated validation dataset associated with a downstream classification
task, TROVE extracts candidate static features from the dataset and scores each
feature by (i) the effect of the feature on classification errors as well as (ii) the
extent to which the VLM relies on the feature when making predictions. In
order to quantitatively evaluate TROVE, we introduce an evaluation framework
consisting of 101 trained temporal VLMs paired with ground-truth annotations for
learned static feature biases. We use this framework to demonstrate that TROVE
can accurately identify error-inducing static feature biases in VLMs, achieving
a 28.6% improvement over the closest baseline. Finally, we apply TROVE to
7 off-th...