Question: What explains the strength of image results over text results? Is this due to reduced task ambiguity?
Evidence: We also consider the role of the pretraining dataset and model size. We conduct active learning experiments with 3 pretrained models: BiT-S-R50x1, BiT-M-R50x1, BiT-M-R101x1, and their corresponding non-pretrained versions. For all pretrained models (except BiT-M R-101 for Treeperson) Review rating uncertainty acquisition outperformed random acquisition (Figure, Figure, Figure). Importantly, we did not see any gains from AL on the unpretrained models, even when exploring a range of different hyperparameter configurations.
Scaling trend Interestingly, the BiT-S-R50x1 model, which was pretrained on a smaller dataset than the BiT-M models (ImageNet-1k vs ImageNet-21k) achieves less of a boost from active learning on iWildCam vs the other two models. This suggests that the gains from active learning may continue to grow as pretained models are trained for longer on more data. However, we did not see a difference between BiT-M-50x1 and BiT-M-101x1, which were trained on the same dataset but have different numbers of parameters. We encourage future work that more thoroughly examines these behaviors and scaling laws across a range of models and datasets.
Impact of pretraining on acquisition patters Pretrained models also acquire disambiguating subgroups much more efficiently than their unpretrained counterparts. See Appendix G for additional figures and results illustrating this point.
Why does pretraining improve AL for resolving task ambiguity? We investigate the hypothesis that pretrained representations enable AL models to select objects based on higher-level features (e.g. object foreground and background) through various analyses on Waterbirds.
We investigate the presence of these high-level features by training linear classifiers on the second to last layer of BiT models. The classifiers are trained to predict each image's bird type and background type (4 classes, each comprising 25% of the data). As shown in Figure, these classes are more linearly separable in pretrained models both before any finetuning, as well as after finetuning on a seed set and then acquired examples from the original 2class Waterbirds dataset. This demonstrates that pretrained models can better identify both the causal and spurious features in the data, potentially explaining why they can choose them preferentially. Figure: Uncertainty sampling only provides gains when using pretrained models. S-R50, M-R50, and M-R101 correspond to the BiT-S-R50x1, BiT-M-R50x1, and BiT-M-R101x1 pretrained models, respectively, while R50-NP and R101NP correspond to ResNet models which are not pretrained. Shaded regions represent 95% CIs (Gaussian approx.).
We visualize the second-to-last layer embeddings of BiT (without any finetuning) using t-SNE (van der Maaten & Hinton, 2008), shown in Figure. We find that examples selected by uncertainty sampling fall closer to the decision boundary for the pretrained model than for the unpretrained model. We also see that the pretrained model exhibits far more separation by class, suggesting it is better able to identify useful features in the input. These conclusions help further characterize how pretraining provides a feature space more amenable to active learning.
Figure 5: Uncertainty sampling upsamples both visible and latent minority subgroups. Fraction of Amazon examples acquired by random and uncertainty sampling, stratified by star rating and product category. Uncertainty sampling preferentially acquires examples with lower star ratings and rarer product categories, despite the latter attribute not being visible to the model.****As shown in Figure 6, these classes are more linearly separable in pretrained models both before any finetuning, as well as after finetuning on a seed set and then acquired examples from the original 2- class Waterbirds dataset. This demonstrates that pretrained models can better identify both the causal and spurious features in the data, potentially explaining why they can choose them preferentially.
Answer: Yes—comparing Figures 5 and 6, we see the sampling ratios between the majority and minority classes are more extreme for the image datasets than the text dataset, suggesting either that the shift is less extreme or that the RoBERTa model is less adept at identifying informative examples than the image models we considered.