Before revision:  To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts. To this end, we propose a novel `dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision. We also find empirical evidence that zero and few-shot learning markedly benefit from adding more ` dataset-internal' , self-supervised training signals, which is of practical importance when retrieving or computing on large external sources of such signals is infeasible .
After revision:  To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many minority concepts. We find that long-tailed zero and few-shot learning markedly benefit from adding more ` dataset-internal' , self-supervised training signals, which is of practical importance when retrieving or computing on large external sources of such signals is infeasible .