ICLR 2025

Published as a conference paper at ICLR 2025
UNCERTAINTY HERDING:
ONE ACTIVE LEARNING
METHOD FOR ALL LABEL BUDGETS
Wonho Bae
University of British Columbia & Borealis AI
whbae@cs.ubc.ca
Danica J. Sutherland∗
University of British Columbia & Amii
dsuth@cs.ubc.ca
Gabriel L. Oliveira∗
Borealis AI
gabriel.oliveira@borealisai.com
ABSTRACT
Most active learning research has focused on methods which perform well when
many labels are available, but can be dramatically worse than random selection
when label budgets are small. Other methods have focused on the low-budget
regime, but do poorly as label budgets increase. As the line between “low” and
“high” budgets varies by problem, this is a serious issue in practice. We propose
uncertainty coverage, an objective which generalizes a variety of low- and high-
budget objectives, as well as natural, hyperparameter-light methods to smoothly
interpolate between low- and high-budget regimes. We call greedy optimization of
the estimate Uncertainty Herding; this simple method is computationally fast, and
we prove that it nearly optimizes the distribution-level coverage. In experimental
validation across a variety of active learning tasks, our proposal matches or beats
state-of-the-art performance in essentially all cases; it is the only method of which
we are aware that reliably works well in both low- and high-budget settings.
1
INTRODUCTION
In active learning, rather than being provided a dataset of input-output pairs as in passive learning,
a model strategically requests annotations for specific unlabeled inputs. The aim is to learn a good
model while minimizing the number of required output annotations. This procedure is generally
iterative: a model is initially trained on a small, labeled dataset, then selects the most “informative”
data points from an unlabeled pool to annotate. This is particularly useful when labeling is expensive
or time-consuming. For example, manual annotations of medical imaging by radiologists or pathol-
...