Before revision:  However, this methodology is being critiqued for: exceptional compute and pretraining data requirements ; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences. The core belief behind current methodology, coined `the bitter lesson' by R. Sutton, is that `compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption. We thus approach pretrainingfrom a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels. To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts.
After revision:  However, this methodology is being critiqued for: exceptional compute and pretraining data requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learning or preserving minority concept (long-tail) prediction performance along with accordingly designed evaluation scenarios remain open challenges. We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining, which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail, few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts.