Question: Why does the random baseline make small progress even with hundreds of datapoints? + Relation to train-test mismatch.
Evidence: Modern pretrained models can be adapted to new tasks with remarkably little data, enabling downstream applications for tasks with only tens or hundreds of examples. However, an important but neglected challenge that is especially salient in few-shot settings is task ambiguity, when the desired behavior is not uniquely specified by the provided examples. Task ambiguity can manifest in different ways: For example, the class-relevant features of an input (e.g., the shape of an object in Figure) may be spuriously correlated with other features predictive of the training labels (e.g., the color of the object), making the desired task unclear. In addition, task ambiguity may arise from an underdiverse training set, causing models to be unsure of the desired behavior for minority groups or during distribution shifts that occur at test-time.
We consider whether the task ambiguity problem can be addressed through active learning, where models select informative examples for users to label. In principle, active learning allows models themselves to assist in resolving task ambiguity by identifying examples whose labels would be informative; for example, in fig., asking for the label of the blue square helps determine that the desired task is to predict the object shape not the object color. Since it may be challenging in general for users to intuit possible sources of task ambiguity, much less how to address them with example selection, an automated active learning approach may be desirable.
In traditional settings with smaller, unpretrained models, several challenges often prevent active learning from seeing success in practice, including label noise, unlearnable examples, and a lack of generalizability across models and tasks. However, using pretrained models in few-shot or low-data settings has several key differences from traditional ML pipelines which may make them particularly well-suited for active learning. First, the effect of an individual data point (and the difference between an informative vs. an uninformative one) is magnified, compared to the thousands or millions of examples typically acquired through standard active learning pipelines. Second, pretrained models excel at learning high-level representations of inputs, which better surface relevant features and may encourage active learning to select examples that disambiguate between these high-level features (e.g. shape and color).
We consider the use of active learning on a range of spuriously correlated, imbalanced, or domain shifted datasets where task ambiguity is salient. We compare an active learning approach with a random-sampling baseline, and compare the difference in performance with and without the use of pretrained models. Our contributions are:
1. Identifying and motivating task ambiguity as a unified concern across a range of real-world datasets, and an especially salient problem for low-data settings.
2. Proposing active learning as a potential solution, along with a simple yet practical recipe that does not require adjustment for new datasets or tuning on validation data.
3. Showing that active learning can enable large gains (up to 6× reduction in data points, +12% absolute gain for the same labeling budget) and presenting scaling trends demonstrating that this is an emergent property of pretraining.
To examine the impact of task ambiguity from the perspective of the unlabeled dataset, we construct variants of the waterbirds dataset where the percentage of mismatched examples range from 95% to 50%. 4 We then proceed with AL and report results in Figure. 5 We find significantly larger gains on versions of the dataset with smaller fraction of mismatched examples (gains average 4% for 50-70% shift, and 10% for 75-95% shift) keeping. This provides stronger evidence that active learning is actually helping the model identify the true task.
However, an important but neglected challenge that is especially salient in few-shot settings is task ambiguity, when the desired behavior is not uniquely specified by the provided examples.****To examine the impact of task ambiguity from the perspective of the unlabeled dataset, we construct variants of the waterbirds dataset where the percentage of mismatched examples range from 95% to 50%.
Answer: This is correct! Pretraining is no cure-all, and we believe applications that feature task ambiguity (which may manifest through distribution shift) are one of the most important outstanding challenges. To capture the precise effect of these shifts, we conduct a "dose-response" experiment where we vary the fraction of mismatched examples in Waterbirds from 5% to 50%. See the main comment for more details.