Before revision:  We propose GOLD (generation by off-policy learning from demonstrations): an  algorithm that learns from the off-policy demonstrations by importance weighting and does not suffer from degenerative solutions. We find that GOLD outperforms the baselines according to automatic and human evaluation  on summarization, question generation, and machine translation , including attaining state-of-the-art results for CNN/DailyMail summarization .
After revision:  We propose GOLD (generation by off-policy learning from demonstrations): an algorithm that learns from the off-policy demonstrations by importance weighting . Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation on summarization, question generation, and machine translation , including attaining state-of-the-art results for CNN/DailyMail summarization .