Before revision: For natural language processing ' text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on massive external datasources, which incurs exceptional pretraining data requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learningor preserving minority concept ( long-tail ) prediction performance along with accordingly designed evaluation scenarios remain open challenges .
After revision: For natural language processing ' text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on massive external datasources, which incurs exceptional pretraining data (X) efficiency; (2) zero to few-shot label (Y) efficiency; and (3) long-tail ) prediction performance along with accordingly designed evaluation scenarios remain open challenges .