Before revision:  We find that long-tailed zero and few-shot learning markedly benefit from increasing 'dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
After revision:  We find our method outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's fine-tuning time .