UT5: Pretraining Non autoregressive T5 with unrolled denoising
Authors: name: Mahmoud G. Salem; name: Jiayu Ye; name: Chu-Cheng Lin; name: Frederick Liu
Published: 2023-11-14 21:28:10

Abstract: Recent advances in Transformer-based Large Language Models have made great
strides in natural language generation. However, to decode K tokens, an
autoregressive model needs K sequential forward passes, which may be a
performance bottleneck for large language models. Many non-autoregressive (NAR)
research are aiming to address this sequentiality bottleneck, albeit many have
focused on a dedicated architecture in supervised benchmarks. In this work, we
studied unsupervised pretraining for non auto-regressive T5 models via unrolled
denoising and shown its SoTA results in downstream generation tasks such as
SQuAD question generation and XSum.