PLANNER: Generating Diversified Paragraph via Latent Language Diffusion
  Model
Authors: name: Yizhe Zhang; name: Jiatao Gu; name: Zhuofeng Wu; name: Shuangfei Zhai; name: Josh Susskind; name: Navdeep Jaitly
Published: 2023-06-05 01:36:39

Abstract: Autoregressive models for text sometimes generate repetitive and low-quality
output because errors accumulate during the steps of generation. This issue is
often attributed to exposure bias - the difference between how a model is
trained, and how it is used during inference. Denoising diffusion models
provide an alternative approach in which a model can revisit and revise its
output. However, they can be computationally expensive and prior efforts on
text have led to models that produce less fluent output compared to
autoregressive models, especially for longer text and paragraphs. In this
paper, we propose PLANNER, a model that combines latent semantic diffusion with
autoregressive generation, to generate fluent text while exercising global
control over paragraphs. The model achieves this by combining an autoregressive
"decoding" module with a "planning" module that uses latent diffusion to
generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed
method is evaluated on various conditional generation tasks, and results on
semantic generation, text completion and summarization show its effectiveness
in generating high-quality long-form text in an efficient manner.