Text Generation with Efficient (Soft) $Q$-Learning
Authors: Han Guo
Bowen Tan
Zhengzhong Liu
Eric Xing
Zhiting Hu
Venue: ICLR 2022 Submitted (2022)
Keywords: text generation
reinforcement learning for text generation

Abstract: Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models. Reinforcement learning (RL) on the other hand offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only at the end of sequences. In this paper, we introduce a new RL formulation for text generation from the soft Q-learning (SQL) perspective. It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-policy updates, and learn effectively from sparse reward. We apply the approach to a wide range of text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation. Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods.

Under review as a conference paper at ICLR 2022
TEXT GENERATION WITH EFFICIENT (SOFT) Q Q-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Maximum likelihood estimation (MLE) is the predominant algorithm for train- ing text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such generating ad- versarial attacks or generating prompts to control language models. Reinforce- ment learning (RL) the other hand offers more flexible solution by allowing users to plug in arbitrary task metrics as reward. Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only the end of sequences. In this paper, we introduce new RL formulation for text generation from the soft Q-learning (SQL) perspective. It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-polit fectively from sparse reward. We apply the approach to a wide range of text gener- ation tasks, including learning from noisy/negative examples. adversarial attacks, and prompt generation. Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods.
updates, and learn ef-
INTRODUCTION
Recent natural language generation systems have made remarkable progress in producing well- formed coherent text. especially with the massive pretrained language models (LMs) (Radford al., 2019 Brown et al. Lewis et al., 2020 Raffel et al. (2019). Those models are typically trained using maximum likelihood estimation (MLE) with a large amount of data supervisions. Despite its successful outcomes, the standard training method suffers from limited applicability to many emerging text generation problems, where little or no supervised data is available. Prominent exam- ples of such low-data problems include generating prompts to control the massive LMs (Yin et al.) 2019 Shin et al.) 2020) Zhong et al., 202T). learning text generation from noisy or even negative data, generating adversarial text attacks for robustness study (Wallace et al.) 2019 Atanasova al. 2020). and others (Figure I right). Due to the failure of standard MLE, people have had to devise specialized algorithms for those problems respectively.
On the other hand, reinforcement learning (RL) (Sutton & Barto 2018) offers an alternative prin- cipled framework for learning from arbitrary reward functions, and has achieved great advances in robotic and game control. However, RL by far has made limited success for training text generation, primarily due the key challenges of sparse reward (i.e., single reward signal is received only after the whole text sequence is generated) and large action space (i.e., a vocabulary of millions of words). For instance, a popular family of RL algorithms studied extensively for text generation is the pasec (Williams) 1992) or actor-critic based (Bahdanau et al.) 2016 Rennie et al.) 2017) al- gorithms, with policy gradient (PG) being the most prevalent example (Ranzato al. 2016 Li al,,
2016 Rennie et al.) 2017, Tan et al.) 2018) Pasunuru & Bansal 2018 Paulus et al. 2018). 8). Those algorithms train the model with on-policy updates, i.e., the text samples used for estimating policy gradients are from the target model itself. Due to the exponentially large space of sequences, on- policy updates often suffer from extremely high variance and low data efficiency (e.g., most model samples are not useful for learning). Thus directly training with PG from scratch is usually impos- sible. In practice, the model has to be initialized by MLE training, followed by PG as finetuning, which often leads to limited improvement (Choshen al. 2020
Wu et al. 2018).

---

Under review as a conference paper at ICLR 2022
large vocab
Q-values (simultaneously updated)
Learning from Noisy (Negative) Text
Dataset
Posiiive Examplis
Tran
Model
Positive Ouipuns
learning signal
Examples
Adversarial Text Generation
Model
Generated Adbersarial Text
Classifier
skiing
down
the
...
<E
Prompt Generation for Controlling Pretrained LMs
r=
0
97.3
into
Pictrolned Model
Model
Off-Policy Data Samples
On-Policy Model Samples
[Dutou: from the composition the chemical
[Prompi "the chemical
An overview of the proposed SQL algorithm for text generation. Text generation challenging due to sparse reward (i.e., the rewards of all intermediate steps are 0) and large tion space (i.c., large vocabulary). Our SQL formulation enables several key algorithmic features highlighted with yellow color, including (1) the combined and off-policy updates for the best of both, (2) bridging the final non-zero reward to directly supervise the Q-value estimation at inter- mediate steps for learning stability, and (3) simultaneously updating the Q- Q-values of all candidate actions for efficiency. Right: We explore diverse applications of the text-generation RL algorithm.
Figure I: Left:
Another set of work has resorted to poficy RL. The key advantage is that samples from other sources, e.g., human written text, can be used, making them more data efficient than on on-policy methods. Previous work has used either importance weighted (Pang & He 2021; Zhou et al. 2017 Kandasamy et al. 2017 or Q-learning based algorithms (Guo) 2015 Jaques et al. 2020) Narasimhan et al. 2015). However, off-policy methods have been considered to be less stable. For example, the Q-learming performance relies heavily how accurate the learned Q Q-function assesses the quality of intermediate subsequences challenging task due to the sparse reward signals.
In this paper, we develop (Figure [I left). We reframe the text generation problem from the soft Q-learning perspective origi- nally developed in robotics (Haarnoja et al. 2017 [Schulman al. (017). The resulting connection allows us to seamlessly take advantage the latest successful techniques from the RL literature. In particular, introduce and adapt the principled path consistency learning (Nachum et al.) (017)) to text generation, that (1) offers a natural way to train the model with both on- and off off-policy updates, hence combining the best of the strategies, (2) bridges the sparse reward signal directly su- pervise the Q function learning, leading to more accurate Q estimation and credit assignment, and (3) makes efficient updates to Q-values by considering all candidate actions together.
RL formulation for text generation that tackles the above issues
The generality and efficiency of the proposed method allows us to train text generation in wide range of applications: (1) With noisy and negative truining examples, our approach learns to gener- ate accurate entailment text that greatly improves upon the data itself well as other various training methods; (2) Our approach also manages to train an effective adversarial text generator for robust- ness test for classifiers; (3) We train prompt generator with algorithm to achieve controllable generation of pretrained LMs in terms of topics. On all the three tasks, our approach consistently improves not only previous RL algorithms for text generation, but also diverse sask-specialized methods designed specifically for each of the problems, respectively. In the appendix 1.4,, we also show that standard supervised tasks where MLE prevails, our approach is competitive train text generation models from scratch, which was usually impossible for previous RL algorithms.
BACKGROUND AND CHALLENGES
The goal of text generation is to produce coherent text y (yo. ... ...)y) of certain properties for given task, where yi is a token from vocabulary V, and T is the text length. The generation can condition arbitrary input context, which omit for simplicity of notations. We aim learn
2

---

Under review as a conference paper at ICLR 2022
generation model po(y) which is typically decomposed autoregressively po = II pe (tr ycr). where Ycce = (y...... 9c-1) is the prefix, and the distribution at each step t is obtained by applying the softmax function the output logits:
exp fo yee))
(y you)
(I)
Eyevv exp fo(y' yyr)
Here fo yee)) is the logit of token y computed by the generation model.
Given a training example y", maximum likelihood training (MLE) updates the model with the gra- dient VoCML.(()) (0) Ern E V log Po (yf y2.). Despite its popularity, MLE-based training only applies when clean supervised data y² is available, and cannot be used to optimize arbitrary task metrics (e.g., BLEU, entailment score) which are typically the goal in many text generation tasks.
2.1 REINFORCEMENT LEARNING (RL) FORMULATIONS FOR TEXT GENERATION
Notations. Previous research has formulated text generation as an RL problem by considering the following finite-time Markov Decision Process (MDP). each time step I, let the "state" be 8, = yсt, namely the partial sequence generated so far. The model, also known as the "agent", takes as input the current state & and outputs token, also called "action", Oe E V according to a policy <a, 8t). The agent then receives reward Tr r(8t, ax) and deterministically transitions to next state $1+1 (i.e., the concatenation of the tokens in 8, and the new token a,).
Following the notation convention in RL, let be the trajectory (i.e., text sample) generated by the policy. The agent's objective is to maximize the accumulative reward, J(x) = Ernr Eizo 7'r where y E (0, is the discount factor. A central concept in RL is the Q-function of policy n, defined as Qa (s,, 0z) = E. - 7 Tt' 8t, 0r, which is the expected future reward of taking action 0e (i.e., generating token ax) in state 8; and continuing with the policy
Challenges. Text generation poses significant challenges to RL, particularly because (1) the reward signal is usually sparse, i.e,, T2 = O, vt < T and the agent receives non-zero reward rr only after it generates the full sequence, (2) the action space (i.e., the vocabulary V) is extremely large, often containing millions of words. The challenges have led to difficulties of the two major families of RL approaches applied to text generation problems, as detailed below.
Policy-hased RL techniques directly parameterize the policy wo with parameters 0. Thus the policy (a s;) exactly corresponds to the above generation model Pa(y: ycr). Policy gradient (PG) one of the most widely used algorithms for text generation (Ranzato al. 2016). optimizes the cumulative reward with the policy gradient:
VoJ(Fo) = -EENNO En' Q(s, as Vo To (a: 8i)
where Q(sr, ax) is the estimated QTo value with sample r. Notice that the expectation taken w.rt. the policy wo, which makes PG an on-policy algorithm, meaning that the sample needs to come from the the current policy Te itself. In practice, however, optimizing this objective alone from scratch is unlikely going to work because most samples та are just gibberish with zero reward, failing to provide meaningful training signals for updating the policy. Previous literature either initializes the policy Te with MLE training, and/or which often leads to marginal gains in practice (Wu et al. 2018 Choshen et al. (020).
combination of MLE and PG updates,
Value-based RL techniques, such Q-learning, implicitly learn the policy x by approximating the value Q (s, a) directly. Specifically, let Q*, max, Q* (s, a) denote the optimal value over policies. Thus the optimal policy x¹ is simply taking the action of maximal Q" value at each state. The approximation of Q" is based on the well-known Bellman temporal consistency:
Q" Q"(8,,a.)
maX Q" (((++1,4++1).
Deep Q-learning Mnih et al. 2013) parameterizes the Q-function as Q9(2, a) (e.g., neural net- work), and train the parameters by minimizing the following regression objective:
(())= E. 00.5 (r+ max Q0(s441, Qo(st,
(4)
3

---

Under review as a conference paper at ICLR 2022
Coga, pct (0)
Feg
furg mi Fyle: m
Cag. rot (0)
Bi
(- 11
7 log melrest Sc-4)
q Q-valoes
reward
Qovaloes
P.-0
E+1
Single-Ster PCL Training
Stlt PCL Training
Figure 2: Soft Q-Learning with path consistency learning (PCL) objectives, where we illustrate with a vocabulary of size 3. Left: Single-step objective (Eq 9 where for each (8t, az), the computation involves step and (+1. Dashed boxes in dark green and gray indicate the regression target, where the intermediate reward ry is often due to sparsity. The gradient is applied to parameters at step t (indicated by orange color). Right: Multi-step objective (Iq(II) which aggregates from step t all the way to T. In this way, the final-step non-zero reward Fr is used as the regression target.
where the parameters of the targer Q-network, which is a slow copy of 0 and considered as constant for gradient computation of 0. Here T' is an behavior policy which can be an arbitrary distribution over text, such as the data distribution or replay buffer (Mnih et al. (013). This makes Q-learning an off-policy algorithm because of its ability to samples coming from other policies. After learning Qo. one can induce a policy и from that takes arg max, Qo a) at each state 8. Jaques et al. (2017) instead sample tokens from the softmax function applied to Qa-
However, the training can be unstable and inefficient due to several challenges: (1) The bootstrap- ping nature of the above regression problem make the training unstable. That is, the regression target F + maxa... Q0(8++1 a++1) itself is derived from the Q-function to be learned (Kumar al. 2019). The problem is exacerbated in the presence of sparse reward in text generation, where the real observed signal r, is zero for all intermediate < T; (2) The large action space (e.g.. 10*) in text generation results in slow updates. In particular, notice that Eq. (4) applies the gradient update to the Qa-value of the only one particular token de (out of the 10 candidate tokens in the vocabulary), making the training inefficient; (3) Besides, pure off-policy updates could be highly sensitive the quality of training data, and miss the opportunity of on-policy exploration that maximizes the reward of interest more direct way.
THE SOFT Q-LEARNING FRAMEWORK
In this section, combat the difficulties of previous RL methods by introducing the soft Q-learning (SQL) formulation of text generation. We show that the formulation is seamlessly compatible with the common architecture of text generation model [Eq[]) permitting easy implementation (3.I).
The formulation further allows us to integrate the latest advances in RL, notably path consistency learning (Nachum al. 2017) that makes the RL training efficient and stable in practice ($3.2) Figure 2 and Algorithm]] summarizes the resulting SQL framework for efficient training.
3.1 SOFT Q-LEARNING FORMULATION FOR TEXT GENERATION
Soft Q-learning (Haarnoja et al.) 2017 Schulman al. 2017 Nachum et al.
2017 is an maximum-
entropy (MaxEnt) extension to the standard (hard) Q Q-learning (Mnih et al.
2015 Sutton & Barto
2018
Under this framework, the agent is encouraged optimize the reward while staying as
MT + a7l (M (- st)) which augments the vanilla J(T) with the additional Shannon entropy term 7 with coefficient This is appealing because it seamlessly connects the Q-values to the familiar output logits of text genera- tion model, which enables straightforward implementation of the SQL formulation.
stochastic as possible, with the objective (T) Ersr
Q-values as Generation Model Logits. We show the connection of the Q-values with the logits, i.c., the model outputs right before the softmax layer. Concretely, with the SQL objective, the following relationship between optimal policy πT" and action-value Q" holds (Haarnoja et al. 2017
Schulman et al. 2017
exp Q' (a, a) Σ. exp Q" (s, (s)a')
(a
(5)
WLOGG can assume a=1, it he folded into the reward function by scaling the latter with 1/a.
4

---

Under review as a conference paper at ICLR 2022
This form is highly reminiscent of the softmax layer of the generation model in Eq. The con- nection suggests that we can naturally parameterize the Q-function in SQL as the generation model logit function, ie.. Q,(s,a) fo(a s). In other words, the model output fo(a s). originally interpretted as the "logit" token a given the preceding tokens 8, is now re-interpretted as the Q- value of action in state When achieving optimality, Jo- (a 8), namely Q" (s, a),, represents the best possible future reward achievable by generating token a in state 8. Similarly, the full generation model pa (a 8) in Eq. that applies softmax fo now precisely corresponds to the policy тe induced from Qo a). That is,
exp (8) To (a 8) - exp Qo (s,a') Σa exp_ fo (a' x)
exp fa(a 8)
= po a).
(6)
We could further gain even more intuitive interpretation of the above generation policy x¹ from the lens of advantage function (Sutton & Barto 2018). Specifically, in SQL. the optimal state-value function is the oog-nooaalizer of the optimal Q-values (Haarnoja al. 2017 Schulman et al. 2017).
This allows us to rewrite Eq. into more concise form:
(s) log Σ,, exp Q' (s,d'), x" (a s)= (Q, (s)
(s.
where A is the optimal advantage function. The equation says that, in the proposed text generation SQL formulation, the optimal policy generates token a in state s according to the token's 's advantage.
3.2 EFFICIENT TRAINING WITH PATH CONSISTENCY
The above section has described parameterizing the Q-function with the common generation model with parameters O. Now we present how to learn the Qo function within the SQL framework. Vanilla training based on the Bellman temporal consistency can suffer from the instability and inefficiency issues similar the conventional Q-learning ($2.1). as we discuss more in the appendix ($A.3.2). Fortunately, our SQL formulation allows us to import latest advances of RL techniques to the text generation setting that overcome the difficulties.
Specifically, we adapt the unified path consistency learning (PCL) that has excelled in game con- trol (Nachum et al.] (017). The PCL-based training updates Q-values of all tokens once through connection between the value function and the induced policy. More specifically, it is shown in
Nachum et al. (2017) that the optimal policy л² (Eq(5) and the optimal state value function V ( in SQL must satisfy the following consistency property for all states and actions:
(a.) 7V* ($1+1)
log x (a. s.), Vsa; ae.
(8)
Accordingly, the PCL-based training attempts to encourage the satisfaction of the consistency with the following regression objective:
2 ( (--vo -V (8t) + 7V (8.+1) + log m. (ax sc) )}
EsQu, (0) = Egy
(9)
where me is the induced policy defined in Eq. (6): Vi is defined similarly in Eq. but depends on the target Qa network (i.e., a slow copy of the Qe to be learned), and recall that n'' is an arbitrary behavior policy (e.g., data distribution). Please see Figure 2 (left) for an illustration. Crucially, notice that the gradient update is applied to 0 through the log na term which explicitly involves the Qo-values of all tokens a the vocabulary. This shows an important difference from the above vanilla training in conventional Q-learning ($2.T) where Qo is updated only through the particular a, token. The PCL training thus offers more efficient updates for the Qa function.
Multi-step PCL for Sparse Reward. The above PCL objective Eq.(9) alone does not resolve the potential instability issue due to the bootstrapped Vg value and the sparse reward (i.e., r(st, ax) = 0 for < T). Our SQL formulation allows us to additionally incorporate the multi-step variant of the PCL training (Nachum et al.) (017) to resolve the issue. Specifically, by applying telescoping sum on the consistency equation (Eq starting from up to T, we arrive at the multi- step temporal consistency:
T-t V" (st) - -7 T-VV" (ST+1) ΣT 7 (/eett -
log n (aste st4)))
(10)
5

---

Under review as a conference paper at ICLR 2022
where the value of past-terminal state is zero, V* ($1+1) = 0; and the rewards only available 'rr. the end. EL - 7TT =- " We can then come the following multi-step objective function,
LSQL, ms (0) Er (-Vo (s:) + +
T- o log no (ac41 BL.:)) ) Σ i==
(11)
We can see the objective side-steps the need to bootstrap intermediate value functions Vs((s:) for t' > t. Instead, it directly uses the non-zero end reward ry derive the update for 6. Please
Figure 2 (right) for an illustration. In practice, we combine the single- and multi-step objectives (Eqs 9 and together for training.
Joint On- and Off-policy Training. Finally, we highlight that the behavior policy n' involved in the objectives Eqs.()) and (II) can an arbitrary policy (i.e., distribution over text sequences), from which we can draw trajectories (i.e., text samples). For example, ní can be (possibly noisy) text dataset, or a set of text samples produced by other generation models, resulting in off-policy training. We can also set x' to be the current generation model To be learned, resulting in on- policy training. In practice, we could first train the model with only off-policy data for warming up, and then continue with joint and off-policy training further maximize the reward.
Algorithm Efficient Soft Q-Learning for Text Generation
Input: Qu function (i.c., generation model logit function fo in Eq Reward function r(s, t)
Training examples D (for off-policy updates; optional)
1:
Initialize and target model parameters
2: repeat
3:
Draw batch of off-policy samples {Tar) ~>
4:
Draw batch of on-policy samples {Tea J by decoding with policy ma(ae st) (Eq
5:
Compute Qo(ss, az) values (i.e., the model logits) and target Qa(se: for (8e, ae) E (Taa} [Tor)
Compute the objectives Eqs.(9) and [T]
7:
Update the model parameters @ via gradient descent
8: Update the target model parameters 0 by 0 + p + (1 -1 p) with update rate p
9:
until convergence
Output: The trained Qs- function and the induced generator T&-
4
APPLICATIONS AND EXPERIMENTS
4.1 LEARNING FROM NOISY (NEGATIVE) TEXT
The popular MLE algorithm learns by (blindly) imitating training data. However, it is often ex- pensive curate clean quality data. It thus highly desirable to be able learn from data with noises, or even negative examples. With the guidance of task metrics (rewards), the model can even learn to "outppeform"" the training data and achieve desired generation behaviors. To this end, we consider the task of entailment generation (Pasunuru & Bansal (2017). Given a sentence (premise), the goal to generate new sentence (hypothesis) that logically follows the premise. For example, given source sentence "Sophie is walk walking a dag out sutside her house", the hypothe- ses "Sophi: i is outdoor" considered entailed, but "Sophie is inside her house" is not and even negative (contradictive) sentence.
Setup (more in the appendix $A.2.1). We sub-sampled 50k training examples from the SNLI
dataset (Bowman et al. 2013). a commonly used entailment classification dataset. The hypotheses have an average entailment probability of only 50%, and over 2/5 of them less than 20% (nega- tive/contradictive examples). This poses a significant challenge for the models to learn from the noises. The rewards used in RL algorithms include (1) the entailment score of the generation mea- sured by a robust entailment classifier (Nie et al. 2020), (2) the log-likelihood of the generation indicator of language quality measured by GPT-2 language model (Radford et al. 2019). and (3) BLEU score w.r.t the input premises as another language quality reward that avoids trivial outputs. We sum together all rewards with weights 1.0.
We compare our approach with a broad range of baselines, including (1) the standard MLE training (MLE);; (2) rewa rd, where
the reward function to filter examples; (3) joint MLE and

---

Under review as a conference paper at ICLR 2022
100
66-
544
520
Entailment Rate
MLE+PG HEER
10-
mopee a-
Perplexity
426-
66-
326-
M.C
H.E-rovord
46-
HLE-PG
N XER
216-
persloxity
20
C0LD5
S:51
20
SQX (iraet
9.. cusl
8
14
Diversity
Diversity
Diversity
Figure 3: Left: entailment generation performance plotted against diversity (average of H1 and H2). Circles represent results of top-p sample outputs, and triangles represent results of beam- search outputs. Right: entailment attack performance against diversity (average of Hj and H2). Only few MLE+PG dots are visible because the model is not able to generate more diverse samples even with increasing value in top-p decoding, i.e., the model collapses.
PG training with MLE initialization (MLE+PG), where we initialize the model with MLE training, then train it with combined MLE and PG losses; previous lext-generation RL algorithms including (4) MIXER (Ranzato et al.) 2016). (5) Sel f-critii (Rennie et al.] 2017), and (6) one of the latest methods GOLD-8 8 (Pang & He 202T) which is a pure off-policy method based importance- sampling PG. To ablate the effect of multi-step training ($3.2). we additionally compare with a simplified variant of our approach that uses only vanilla single-step PCL training (SQL (single)) ) In the appendix I...I.I) we compare and discuss more baselines such as MLE weighted by rewards.
We evaluate generation results in terms of entailment rate, language quality (perplexity), and diver- sity which is measured by the Shannon entropy over unigrams and bigrams (H₁, H2) (Gehrmann al. (02T). Since text generation models intrinsically trade off diversity and quality (Caccia al.
2019 Hashimoto et al., 2019). we vary the generation diversity by generating samples via top-p sampling (Holtzman et al., (019)) with different p values, and plot the entailment rate and perplexity against diversity, resp. We also evaluate the samples produced by cearcscarchh decoding.
Results. Figure (left) shows the results. First, notice that MLE performs poorly, while MLE+reward improves upon it. This is not surprising as the training data contain noisy/negative examples. Similarly, since the pure off-policy algorithm GOLD-s relies heavily on the data distri- bution, observed that it achieves sub-optimal performance. The on-policy MLE+PG with MLE initialization gives better entailment rate. In comparison, our full SQL framework achieves the best -idiereentdiversitty trade-off. The comparison between SQL and SQL (single) highlights the importance of having the multi-step objective which directly uses the end reward rather than boot- strapping intermediate Q-values for supervision.
4.2 Universal ADVERSARIAL. ATTACKS
We next study the application in text adversarial attacks, where again supervised data is available. Adversarial attacks is an increasingly important research topic as they reveal models' vulnerabili- ties and flaws. This is especially true for universal attacks (Wallace al., 2019: Atanasova al., 2020). where we want to generate universal examples that trick the model on all possible inputs. For instance, consider the context of entailment classification. Our goal is to find universal human- readable hypotheses that are going to be classified as "entailment" with as high probability as pos- sible, regardless of the input premises. This more challenging setting compared to previous instance-specife attack (Moriss et al.) [2020] Jin et al., 2020: Ebrahimi et al. 2017) where the attack model conditions on a premise and generates an adversarial hypothesis specific the premise.
Setup (more in the appendix A.2.2 We aim to attack one of the most popular MultiNLI (Williams et al. (018)) entailment classifiers on HuggingFaceHHub] The attack gener- ation model generates adversarial text without conditioning on any inputs so that the generated attacks are universal to all premises. We compare our SQL with MLE+PG. We use all hypotheses in the MultiNLI dataset as the training data for the MLE training in MLE+PG and the off oof-po updates for our SQL. We do not compare with previous specialized adversarial text attack methods, because they either not applicable to the challenging universal attack setting (Morris al. 2020
roberta

---

Under review as a conference paper at ICLR 2022
Prompt Generator A
"science"
"the chemical microscope In summary"
prompt (model's output)
imput sentence
Reward Function
Generated
reward: average score
Topic Classilier
Sentence
Pretrained LM
Generated
Perplexity
Figure 4: Conditioning on a topic (e.g., "science'' ") the prompt generator automatically pro- duces short piece of text (i.e., prompt) such that, by prepending the prompt to the imput text, the pretrained LM will generate continuation sentences of the particular topic. The subsequent compo- nents the reward functions train the prompt generator. The discrete steps (highlighted in red) make previous gradient-based prompt tuning approaches not applicable here.
PPLM
GeDi
MLE (5) SQL. (off, 5)
60-
13.07
123.88
25.70
25.77
MLE+PG (5/10/15)
SQL (5/10/15, ours)
30
25. 25.52/28.16/28.77 16/28.71
05.94/26.95/29.111
Table 1:
Language perplexity results averaged across topics. The lower, the more fluent the gen- erated continuation sentences.
MLE talendd)
sot rest tarensy
musind talend)
six rilest manyy
HiL PRS (td talendd
sig masnts cony
MET en D1 taren
sqt tokenss oom)
Model PPLM GeDi
SQL
Seconds 5.58
1.05
0.07
Figure 5: Average topic accuracy.
Table Average sentence generation time cost.
Jin et al. 2020 Ebrahimi al. 2017). were not designed generate human-readable sentences (Wallace et al. 2019). We use similar settings as $4.1 to explore the diversity- trade-off by plotting the entailment rate and perplexity against diversity, respectively. The entailment classifier to be attacked is used as entailment score reward functions. We additionally include a token-level repetition penalty reward for readability.
Results. Figure 3 (right) shows the results, and Table shows samples. We can see that SQL outperforms MLE+PG consistently across different diversity values. The outputs from MLE PG are not diverse even with high p's, indicating the model collapses and can only generate small set of unique adversarial examples. The model by SQL discovers the pattern "saint-pierre et-stint saint-paull paul" (an entity name), and exploits this to generate samples with high universal entailment rate.
4.3 PROMPT GENERATION FOR CONTROLLING PRETRAINED LANGUAGE MODELS
reward function does not just have to a metric like the BLEU score, but also a complicated pipeline that eventually returns score. To demonstrate this, consider the emerging task of prompting large pretrained LM for controllable generation (Hu et al., 2017] Radford et al. 2019] Brown al. 2020). The goal is learn generate text prompts that sentences of certain desired attributes (e.g., topics). The problem of controlling the generation of pretrained LMs was previously approached through specialized algorithms such modifying the LM hidden states during decoding (Dathathri et al. 2020 Krause et al. [2020] Qin et al. 22200). Here we show that prompts offer an easier, faster, more effective way for controlled generation.
the LM to generate
Learning to generate/tune prompts is gaining increasing attention recently. It side-steps the needs for expensive LM fine-tuning, and adapts LMs to new scenarios with prompt as the (compute- friendly) interface. Most existing approaches (Wallace et al.) 20199 Li & Liang 2021; Lester et al.) 2021] rely on gradient hackpropagation and are applicable only when the whole training pipeline differentiable. This does not hold for the text generation setting, as illustrated in Figure 4 In contrast, the RL framework is generally applicable to any differentiable or discrete pipelines.
Following (Dathathri et al. 2019), aim to control the generation to have one of 7 topics (e.g., "science"); the generated prompt is prepended to one of 20 input sentences for the pretrained LM to generate continuation sentences. Figure shows the
Setup (more in the appendix $A.2.3).
8

---

Under review as a conference paper at ICLR 2022
architecture of prompt-based controllable generation. We compare our SQL method with MLE+PG as before. Since the prompt length could impact the generated sentences, we conducted experi- ments with maximum prompt length 5, 10, and 15. As ablation study, also evaluate the SQL algorithm with only off-policy updates (i.e., without on-policy exploration), denoted as SQL (off), and compare it with vanilla MLE training. Finally, we also compare with two specialized control- lable generation techniques based on pretrained LMs, namely PPIM (Dathathri et al., 2019) and GeDi (Krause et al. 2020), following similar procedures using their open-sourced code. We use distilled GPT-2 mode as the pretrained LM to be controlled. For rewards, we use the topic accu- racy of the continuation sentences measured by zero-shot classifier, plus the the log- logelikel of continuation sentences as the language quality reward measured by distilled GPT-2.
Results Figure 5 shows the topic accuracy of the controlled LM outputs averaged across the topics, and Table [ shows the respective language quality results. More detailed topic accuracy results and samples are provided in the appendix ($3.1.3) (where GeDi obtained low accuracy on 2 of the 7 topics, possibly because the topic tokens are tokenized into two subwords for which the model released by the authors was not specifically trained). We can see that the prompts generated by our SQL cause the LM to generate sentences with high topic accuracy while maintaining low perplexity in most settings. Increasing the prompt length positively impacts the topic accuracy, which makes sense because longer prompts give more flexible for steering the LM. The comparison between MLE and SQL (off) shows that the off-policy component of SQL is better than standard MLE training, as it incorporates reward signals instead of just blindly following the (noisy) data.
Next, comparing with the previous steered decoding such as PPIM and GeDi, we can see the prompt-based control trained with RL achieves better trade-off between topic accuracy and language quality. Moreover, once a prompt is produced, we can use the pretrained LM to generate text of de- sired topics efficiently, with the same time cost as standard non-controlled decoding. In comparison, the dedicated steered decoding is often orders-of- magnitude slower, as shown in Table 2
5 RELATED WORK
Standard RL algorithms maximizing the external rewards can sometimes be over-sensitive to the randomness in the environment. Recent works have considered maximum-entrog RL (MaxEnt RI.) extensions, such as the soft Q-learning (SQL.) (Haarnoja et al., 2017; Nachum et al., 2017 Schulman et al.] (017)), that maximize the entropy of policy besides the rewards, and have demon- strated substantial improvement in robotic and game control (Zebart et al., 2008 O Donoggue et al.) 2017 Nachum al. 2018 Eysenbach & Levine, 2021). Our work is the first to adapt SQL and its advanced variants (in particular the path consistency learning (Nachum et al. 2017)) to the chal- lenging text generation problem and show significant results on diverse applications. Applying RL for text generation has been discussed in alleviating the exposure bias problem and optimizing task metrics (Li et al.) 2016; Wu et al., 2016 Rennie et al.) 2017 Paulus et al., 2018; Chen & Bansal, 2018). For example, Ranzato et al. (2016) used the REINFORCE algorithm (Williams) 1992). and Bahdanau al. (016) used the actor-critic algorithm: Guo al. (2018) and [hh et al. (2018) tried to relieve the sparsity problem via hierarchical and inverse RL methods, resp. They are all on-policy RL algorithms with the need of pretraining their models using MLE. Another line of work focused mostly on using only off- off-policy data, often for offline training of chatbots (Jaques et al. 2020 Kan- dasamy et al. 2017| Zhou al.] 2017; Pang & He| (02TT) As result, the opportunity of directly improving the reward (as in on-policy updates) for other rich tasks is missed. Our proposed frame- work combines on- and off-policy training, and further offers solutions for efficient training from scratch in the presence of large action space and sparse sequence-level reward in text generation.
6 CONCLUSION
We develop a new RL. formulation for text generation based on soft Q-learning and path consistency learning. We conduct experiments on learning with noisy and negative data, black box adversarial attack, prompting a pretrained language model for controllable generation, and finally, on standard supervised tasks. The RL formulation opens up enormous new opportunities to integrate more advances made in the fertile RL. literature to improve text and other sequence generation problems.
htttp::/hugginggaaaaee co/iic..cccc...... i 1 1

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
This work develops a new RL. formulation for text generation. While we demonstrate the framework in four applications, it could be adapted to other (emerging) applications. One major component in these applications is the design of the reward function, which influences the behavior of the trained agent. While we believe the MaxEnt RL framework is more robust against reward misspecifica- tion (Eysenbach & Levine (201), the potential failures of sub-optimal reward functions are widely known and discussed To this end, deploying this model to the wild requires careful and extensive examination, using tools such as Ribeiro et al. (2020) Further, we highlight the application for (black-box) adversarial attacks in the paper, with the intention of using adversarial attacks to un- derstand the model's inner workings. That being said, this could potentially be misused to conduct malicious attacks against systems. Hence, users of this framework might want to conduct adversarial attacks against their models to avoid being attacked by other people with bad intentions.
REPRODUCIBILITY STATEMENT
We provide code in the supplementary materials, and additional experiment details in the appendix.
REFERENCES
Pepa Atanasova, Dustin Wright, and Isabelle Augenstein. Generating label cohesive and well- formed adversarial claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 3168-3177, 2020.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.00086. 2016.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Iyy Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: httt::://prroceddnngs neurssnneuuiiie ips ccc/paaperr222///ileee htt.a///.....................66-
pdf
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language GANS falling short. In International Conference on Learning Representations, 2019.
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 675-686, 2018
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforce- ment learning for neural machine translation. In International Conference Learning Represen- tations, 2020. htt httttp:s:///penrevieee net./ forum?
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2019.
10

---

Under review as a conference paper at ICLR 2022
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to con- trolled text generation. In International Conference Learning Representations, 2020. URL
htttps://gpeneevieww net www.:rrummuoouuuuommemoo forum? d=H1 edEy3KDS)
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv: 171v 2017.
Benjamin Eysenbach and Sergey Levine. Maximum entropy (provably) solves some robust rl problems. arXiv preprint arXiv:2103.062577 2021.
Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pp. 202-211, 2016.
Sebastian
Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.0167,, 2021.
Hongyu Guo. Generating text with deep reinforcement learning. arXiv preprint arXiv:1510.09202. 2015.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1361- PMLR, 2017.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1689-1701, 2019.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2019.
Zhiting Hu, Zichao Yang, Xiaodan Liang, R. Salakhutdinov, and E. Xing. Toward controlled gener- ation of text. In International Conference on Machine Learning (ICML), 2017.
Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, et al. Texar: modularized, versatile, and extensible toolkit for text generation. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics: System Demonstrations, pp. 169,164, 2019.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hermández-Lobato. Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation mod- cls with kl-control. In International Conference on Machine Learning, pp. 1645-1654. PMLR, 2017.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandcharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985-4003, 2020.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Kirthevasan Kandasamy, Yoram Bachrach, Ryota Tomioka, Daniel Tarlow, and David Carter. Batch policy gradient methods for improving neural conversation models. In ICLR, 2017.

---

Under review as a conference paper at ICLR 2022
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint (tX.h:2009.066677 2020.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-78800 2020.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep rein- forcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1 1192-1202, 2016.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/90) 2021.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGin: A constrained text generation challenge for generative com- monsense reasoning. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1823-1840, Online, November 2020. Association for Computational Linguistics. doi: 10. 10/5533v11222200 1/wp:/..1222../ 1/0.6/////122200..nnnnnn--mmm/mm fiw.iingseemnlp.1666.5 URL 2020. Fen
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132552 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):5995555 -53,, 2015.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pp. 11911116 2020.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. NIPS, 2017.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-PCL: An off-policy
trust region method for continuous control. In International Conference on Learning Representa- tions, 2018. http::ttt////peeeeee//
net umrumm Hidemmmeeeeeeyyyyyeeeee
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1-11, 2015.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, 2020.
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2c dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, 2017.
Brendan O' D'Donoguue, R. Munos, K. Kavukcuoglu, and V. Mnih. Combining policy gradient and q-learning. In ICI.R, 2017.
12

---

Under review as a conference paper at ICLR 2022
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In Interna- tional Conference on Learning Representations, 2021. htpptt///pppeeeeieeee..cot net//
Eorum? waw.ummdd-oooouu11ee
Ramakanth Pasunuru and Mohit Bansal. Multi-task video captioning with video and entailment generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1273-1283, 2017.
Ramakanth Pasunuru and Mohit Bansal. Multi-reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies. Volume (Short Papers),
pp. 646-653, 2018.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018. URL https:
Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D Hwang. Ronan Le Bras, Antoine Bosselut, and Yejin Choi. Backpropagation-oon based decoding for unsupervised counter- factual and abductive reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 794-805, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In ICLR, 2016.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7008-7024, 2017.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Be- havioral testing of nlp models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902 4912, 2020.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft Q- learning. arXiv preprint
2017.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 4361-4367, 2018.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Bowen Tan, Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric Xing. Connecting the dots between mle and rl for sequence prediction. arXiv preprint arXiv: 1.09740. 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30:5998-6000 2017.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.--JCCNI.P), pp. 2153-2162. 2019.
13

---

Under review as a conference paper at ICLR 2022
90-
MLE+ (LE-)reeardfiieeeed
MLE (reward(wweightted)
MLE+PG
MIXER
60
300
SCST
GOLD
(sing
30-
100
10
10
Diversity
Diveriity
Figure 6: Entailment generation performance plotted against diversity (average of Hi and H2).
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pp. 11 12222 2018.
Ronald J Williams. Simple statistical grolient-folloinng algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-22) 256, 1992.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie Yan Liu. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3612 2018.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 1r09vv1609.000 0444, 2016.
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pp. 3905-3914, 2019.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, 2019.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Meta-tuning language models to answer prompts better. arXiv preprint urXiv:2104.04677 2021.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. End-to-end offline goal-oriented dialog policy learning via policy gradient. arXiv preprint 17Xiv217/2.02838 2017.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
A APPENDIX
A.1 APPLICATIONS AND EXPERIMENTS
A.1.1 LEARNING FROM NOISY (NEGATIVE) TEXT
Please see Table 5 for beam search results, Figure 6 for additional results for MLE+zeward, and Table for examples.
14

---

Under review as a conference paper at ICLR 2022
E2E
E2E
CommonGen
15.0
20.0
Mavir
PG from scratch
54
20
Figure 7: Training curves on validation sets. Left: Training curves on E2E with best hyperparameter configurations. Middle: Training curves on E2E with varying reward scale. Right: Training curves on CommonGen with varying reward scale.
A.1.2 Universal ADVERSARIAL ATTACKS
Please Table 4 for examples.
A.1.3 PROMPT GENERATION FOR CONTROLLING PRETRAINED LANGUAGE MODELS
Please for detailed results breakdown, and Table 11 for examples. Examples in the format: topic: [prompt] input sentence generated text.
A.1.4 SUPERVISED TEXT GENERATION TASKS
Finally, we conduct experiment on standard generation tasks where clean supervised data is avail- able. The study is to examine the capabilities of the proposed RL method to train text generation model from scratch, which has been considered as exceedingly challenging for previous RL algo- rithms.
Setup. We study on two tasks, E2E (Novikova)
et al. 2017) and CommonGiN (Lin et al. 2020),
Model
MLE
PG
MLE++G
SQL (ours)
and use the respective datasets pre-processed by
(Gehrmann et al. 2021) which allow sequence-
val
45.67
0.00
49.08
47.04
to-sequence modeling with standard transform- ers. We run four sets of methods: the standard MLE training (MLE); PG training from scratch (PG); joint MIE and PG training. with MIE ini- tialization PML)++G);; and our SQL training from scratch with both off-policy and on-policy updates (SQL). We use the standard BLEU as reward. We additionally investigate the training stability and sensitivity w.r.t hyperparameters, in particular the scale of reward. To this end, for MLE+PG and SQL, we vary the reward scale in {1, 10, 50, 100, 500, 1000} and evaluate the respective perfor- mance under different scales.
test
41.75
0.00
42.26
41.70
Table 3: BLEU results on the E2E val/test sets.
Results. Table 3 shows the performance on E2E of different models whose hyperparameters are picked using the validation set. We can see the proposed SQL that trains models from scratch achieves competitive results with the common MLE and MLE+PG. In contrast, the PG algorithm alone without MLE fails the training. Figure 7 (left) shows the respective training curves (on the validation set), demonstrating that SQL converges in efficient and stable way as MLE.
We further demonstrate the sensitive of MLE+PG and SQL w.r.t the reward scale as key hyper- parameter. Figure 7 (middle and right) shows the training curves of the two methods with varying reward scales. We can see SQL is significantly more robust as reward scale changes, while MLE+PG tends to collapse with improper reward scale configurations.
15

---

Under review as a conference paper at ICLR 2022
A.2 SETUP DETAILS
Our evaluation follows the GEM Benchmark (Gehrmann et al. 2021) when applicable and other- wise same with the reward function used in training. We use transformer model (Vaswani al. 2017) based on Texar-Pytorch (Hu et al. (019) by default, with 64 hidden dimension, 3 blocks, and heads. For experiments that involve policy gradient training, we initialize the model with maxi- mum likelihood training by default unless specified otherwise. We train soft Q-learning model from scratch with both off-policy (using data) and on-policy (using samples) by default except in $4.1 and $4.3 in which we find it beneficial to warm-up the model with just off-policy training. We apply similar tuning budgets to both soft Q-learning model, and and top-k), based on performance the validation dataset and sample qualities.
(mostly the reward scale
Reward Functions We use the robust entailment classifier Nie al. 2020) in $4.1 one of the most used entailment classifiers on HuggingFaceHub $4.2 and a zero-shot classifier based on BART (Lewis et al. 2020) compute the topic score in 4.3 To compute perplexities, use GPT-2 model (Radford et al. 2019) fine-tuned on the corresponding datasets for computing perplexity in $4.1 and 4.2, and distilled GPT-2 model in $44. without fine-tuning We simply set reward weights to 1. .0, except in $4.2 where we changed the entailment weight to 0.5, log-likelihood and repetition penalty weight to 5.0.
A.2.1 SETUP DETAILS: $4.1
We study using the SNLI dataset (Bowman et al., 2013), a dataset commonly used in training an entailment classifier. The original dataset contains (premise, hypothesis) sentence pairs, where the hypothesis may or may not entail the premise. We sub-sampled 50, 000 training examples from the corpus such that the hypotheses have average entailment probability of only 50% in terms of the premises, and over 2/5 examples have entailment probabilities less than 20%, which can be seen negative (contradictive) examples. The resulting training poses significant challenge for the models to learn from the noises.
The RL algorithms (including PG and ours) permit to plug in arbitrary reward functions drive learning. Based on the goal of the task, we use the following intuitive rewards to ensure entailment accuracy and language quality: (I) robust entailment classifier (Nie et al. 2020) that measures the entailment score of a generation in termss of the input premise. (2) a GPT-2 language model (Radford jet al., 2019) that measures the log- lideliool of the generation as an indicator of language quality, and (3) BLEU score w.r.t the input premises another language quality reward that avoids trivial outputs. We sum together all rewards with weights 1.0.
A.2.2 SETUP DETAILS: $4.2
We study the task of attacking an entail- ment classifier. In particular, we aim to attack one of the most popular entailment classifiers on HuggingFacoHub The at- tack generation model generates adversar- ial text without conditioning on any inputs so that the generated attacks are universal to all premises. The generation model is trained with mostly the same setting as in $4.1 where the entailment classifier to
Model
Generation
Rate
MLE+PG SQL (ours)
it
90.48
the person saint-pierre-o siieere-ee-saiie et-st-nt-
97.40
paul is saint-pierre-et-sanl saaeerre---sant--aal
Table 4: Entailment attack samples and respective en- tailment rates across all test premises. For example, the adversarial sample by SQL is considered to entail 97.40% test premises by the entailment classifier.
R3-nli
This clas-
sifier is ranked (as of http://wwww....aaaaaaaaa/aaa//aa
https://qi thub. htttp://ww..aaoooccmiaa/r//re/er
which
ranked #1 as of May 20, 2021 based on hwc./nhcccc///ggggggaoeo/o//oo//
16

---

Under review as a conference paper at ICLR 2022
attacked is used as entailment score reward functions. Besides, additionally include token- level repetition penalty reward, which empirically benefits readability. Finally, we use the MultiNLI dataset (Williams et al. 2018) which includes more diverse examples than the SNLI used above.
We compare our SQL with MLE+PG, We use all hypotheses in the MultiLII dataset as the training data for the MLE training in MLE+PG and the off-policy updates for our SQL. We do not compare with previous specialized adversarial text attack methods, because they either are not applicable to the universal attack setting (Morris al. 2020; Jin al. 2020 Ebrahimi al. 2017). or were not designed to generate human-readable sentences (Wallace et al. 2019). Besides, it is worth noting that the general RL algorithms have an additional advantage
doing black-box attacks. That is, the algorithms only require the ability to query the entailment classifier for entailment probability, without need of knowing the internal structure of the classifier (e.g., for computing gradients) as in previous attack algorithms (Ebrahimi et al. 2017 Wallace et al., 2019).
For top-p sampling results, we sample hypothesis for each premise and measure the average attack rate across the dataset. This is because sampling multiple hypotheses, each for all premises, and measure performance are expensive. Since the hypotheses are sampled input-independently this should be good approximation.
A.2.3 SETUP DETAILS: $4.3
Following (Dathathri et al., 2019), we aim to control the generation to have one of 7 topics (e.g., "science"); the generated prompt prepended to one of 20 input sentences (Figure 4 for the pre- trained LM to generate continuation sentences. There is no direct supervision data available for training the prompt generator. We randomly create some noisy text as the training data for MLE baselines below and for off-policy updates for our algorithm. Specifically, the noisy text is created by sampling keywords and topics from the list used in (Dathathri et al. 2020) and a paraphrase generation model.
Figure shows the architecture of prompt-based controllable generation. We compare our SQL method with MLE+PG as before. At training time, for each generated prompt sample, the pretrained LM generates 2 continuation sentences for evaluating average reward. We use a zero-shot classifier evaluate the topic accuracy of the continuation sentences. That is, we do not assume access to classifiers pretrained on topic-specific sentences, because generating such topic-specific sentences the goal of the task in the first place. We additionally use an LM evaluate the log likeelhodood of continuation sentences for measuring language quality. Since the prompt length could impact the generated sentences, we conducted experiments with maximum prompt length 5, 10, and 15. As ablation study, we also evaluate the SQL. algorithm with only off- updates (i.e., without on- policy exploration), denoted as SQL (off). and compare with vanilla MLE training. At test time, given a topic, the trained prompt generator produces one prompt using beam search decoding. For each generated prompt, the pretrained LM generates 100 sentences using top-k decoding (with k = 50) for evaluation. Finally, we also compare with two specialized controllable generation techniques based on pretrained LMs, namely PPLM (Dathathri et al. 2019) and GeDi (Krause et al.) 2020), following similar procedures using their open-sourced code. We use distilled GPT-2 mode the pretrained LM to be controlled. We use the paraphrase generation model based on Zhang
2019) [2
During decoding, we include no.repeat.ngee ang.ammttnggammiiii size= 2, which improves
readability
A.3
THE SOFT Q-LEARNING FRAMEWORK
A.3.1 COMPARISON WITH MLE OBJECTIVE
It is interesting to take a closer look at the above objective and compare with the common MLE training. Specifically, we notice the relations between the optimal Q', V*, and A functions: A (sr,a,) Q" (st, ax) V*(s))
тi + + the definition of A" (see [7 and the second equation due to Eqs. and 7 We thus can
V* (st), where the first equation is
see the regression target in the above objective as approximation to the advantage function:
17

---

Under review as a conference paper at ICLR 2022
Model
Entl. Prob
Entl. Rate f
PPL
H1 t
H2 1
MLE
75.62/75.80
79. 79.75/80.25
5.49/5.45
5.46/5.42
8.47/8.40
GOLD-s (Png He 2021
74.55/76.08
78. 78.69/79.89
5.55/5.50
5.50/5.49
8.48/8.45
MLE+PG
90. N0.16/89.77 73
95. 18/948/44.. 13
6.38/6.31
5.23/5.20
8.02/7.99
SQL
91.94/91.55
96. 26/96.6/21 21
8.41/8.42
5.59/5.58
8,20/8.21
SQL (single)
89.90/89.92
94.94/94.82
214.42/214.42
0.00/0.00
0.00/0.00
Table 5: Beam search results on entailment generation, the format val/test. 1/ indicates
higher/lower better. 'SQL (single) achieves zero in Hj/H2 generates a single token.
Ag (s) Uz) == VV (8x) + 7V8 (8++1) r.. Therefore, by optimizing the regression objective, log Thl (a, 8t), which is the log probability generating token a, given preceding tokens 8t, is encouraged to match the approximate advantage value Ag (8,, Qz), no more and no less. This is different from the objective of MLE where the model is trained to (blindly) increase the probability of the observed token a, given 8, and decrease the probability of the rest.
A.3.2 VANILLA TRAINING WITH TEMPORAL CONSISTENCY
Much like the Bellman temporal consistency in standard Q-learning 3). in SQL, the optimal action-value function follows the softmax form of the temporal consistency (Ziebart et al. 2008
Ziebart 2010 Fox et al. 2016 Nachum et al. 2017):
Q' (s:, = r + log Σ exp Q' (8L+1, 4041).
(12)
We thus can derive regression objective similar to the standard Q Q-learning (Eq
(rs + 7 log E., exp Qo (s.41, Qu (st, ai) )
LSQL. vanilla (0) E
(13)
Recall that n' is an arbitrary behavior policy (e.g., data distribution), and Q3 is the target Q-network which is a slow copy of the Qa to be learned and is held fixed during the gradient updates. However, the above objective is inefficient due to exact the same reasons in standard Q-learning discussed earlier, namely the unstable per-step bootstrapping-sty.. training with sparse reward signals, plus the slow updates w.r.t only one token a out of the large vocabulary (action space).
18

---

Under review as a conference paper at ICLR 2022
Length Model
legal
politics
computers
space
religion
science
military
Average
Topic Scores
PPLM
16.52
25.09
13.35
26.23
5.39
38.87
19.33
20.68
GeDi
10.51
83.40
9.32
70.90
18.69
12.46
86.40
45.96
5
MLE
17.28
13.44
7.26
12.27
15.24
39.31
63.75
32.65
5
SQL (off)
23.79
61.11
21.07
7.91
61.77
64.67
67.83
44.45
5
MLE+PG
29.45
74.16
72.49
57.39
65.62
74.31
76.86
64.33
5
SQL
70.57
66.37
58.80
65.60
83.15
60.79
10
MLE+PG
17.72
75.29
71.01
73.92
58.29
80.85
80.84
65.42
10
SQL.
29.62
86.58
75.72
58.38
71.29
81.05
91.40
70.58
15
MLE+PG
40.18
81.47
47.14
82.64
76.21
84.82
89.31
71.68
15
SQL
48.08
77.94
70.04
87.43
75.46
85.94
77.36
74.61
Perplexity
/
PPLM
13.52
12.81
12.79
13.56
12.98
12.43
13.38
13.07
/
GeDi
204.44
80.01
132.82
116.94
132.19
90.00
110.77
123.88
5
MLE
24.52
25.05
23.79
26.26
26.07
25.63
28.56
25.70
5
SQL (off)
25.48
22.70
25.10
26.64
25.84
27.45
27.19
25.77
5
MLE+PG
24.12
22.60
27.74
23.17
25.38
21.84
30.50
25.52
5
SQL
25.31
21.15
26.40
24.31
27.02
25.73
28.67
25.94
10
MLE+PG
28.25
23.49
27.82
26.88
31.62
25.31
33.74
28.16
10
SQL
25.23
25.37
26.20
26.97
25.02
27.11
32.76
26.95
15
MLE+PG
28.38
28.24
28.16
27.21
26.43
29.99
32.54
28.71
15
SQL
35.16
27.72
29.70
31.89
24.04
28.46
26.74
29.10
Table 6: Prompt generation results. Note that some of the numbers from GeDi are low because the topics are tokenized into two subword tokens, which the model was not trained with.
19

---

Under review as a conference paper at ICLR 2022
Input: two men on bicycles competing race
Generated: two men are riding bikes
Input: families waiting line an amusement park turn ride.
Generated: families at amusement park.
Input: man in a black suit, white shirt and black bowtie playing an instrument with the rest of his symphony surrounding him.
Generated: man is playing music..
Input: a white dog with long hair jumps to catch red and green toy.
Generated: dog is jumping
Input: man in black shirt is playing golf outside.
Generated: man playing golf
Input: man wearing sunglasses is sitting on the steps outside reading magazine.
Generated: is sitting outside
Input: young child is jumping into the arms of woman wearing black swimming suit while in pool.
Generated: child jumping into pool.
Input: carefully balanced male stands Generated:
foot near clean ocean beach area.
the beach
Input: two young boys swimming trunks are in the water looking each other. Generated: two bays are the water.
Input: woman riding bicycle past and group of people a sidewalk.
Generated: woman rides bicycle.
Input: female sings and plays into microphone and male in green striped shorts plays hand drum Generated: woman is playing music
Input: little girl wearing cardboard diner hat is finishing off some onion rings a restaurant Generated: little girl is a restaurant.
Input: young woman celebrates getting strike during bowling game Generated: young woman is present
Input: black greyhound dog racing down a track. Generated: a dog runs down track.
Input: several men on stage having discussion.
Generated: men are talking on stage.
Table 7: Entailment generation samples from SQL (beam search, validation dataset).
20

---

Under review as a conference paper at ICLR 2022
legal: legal space religion and space In summary, good understanding of these concepts is that by giving an explicit understanding to person, they provide an avenue to be studied and studied. But the concept of one person being a space is also very confusing, and can be very difficult to obtain. So, polities: the primary referendum is In summary, the outcome will be referendum on the EU membership for the first time of kind for EU citizens, and the full extent of the benefits of single market and flexible single EU state."
computers: aacintoshinoohhintottnnoeel In summary, it appears that the company and IBM products are currently in need of upgrading the computer. This can be seen in detailed review of the Macintosh version of Windows Vista and XP. However, when looking at the changes made by the HP Macintosh hardware and software versions of
space: legal space science and space In summary: n in The purpose of this paper is to investigate and test the theory of space space and other objects. This project will support NASA.s efforts to demonstrate these theories, and propose other relevant new theories. \n This paper provides the following arguments for the
religion: space legal religion religion religion In summary, to the author the current discussion is the po- sition of the Church and the community. While we acknowledge that we should not be commenting upon claims such as our recent cases the other ones that contradict view, conclude it is appropriate to include these cases. Further
science; the chemical microscope is In summary, the most sophisticated of these experiments a tech- nique that gives no obvious, no apparent way of revealing that the material was obtained. In this study, we examine how the compounds in the samples in question make up the composition of the chemical and its properties. The chemical composition
military: arms defense battalion battalion cavalry summary: 6th Panzer Field Division. Second Di- vision.) \,)) n The main task of the battalion in the main counterinsurgency campaign was to counter the enemy in any counter-incursion. The main objective of this campaign is to climinate enemy groups and the remnants of
legal: legal space religion and space This essay discusses the idea of space and time as space, in both theoretical and conceptual terms, as not an individual time period or anything else. The emphasis on time itself, rather than having a fixed central space. Space was the object of the first chapter, and politics: the primary referendum is This essay discusses the nature of the EU referendum. The purpose of this essay is to shed light on the importance of public referendum, on question of whether the decision of an EU member states to remain in the European Union is constitutional and thus accord with constitutional guarantees of sovereignty
This essay discusses hardware devices and software systems for Mac OS X, MacOS X and Linux. To view the latest version of Macintosh OS: Mac 8.7.x) |n in For more information for information about Macintosh systems, visit Mac MacSystems. \n More space: legal space science and space This essay discusses science for teens, adults and teenagers. \n)n When the idea of studying space was first implemented as a method to test. the question was: What if student has been "comfortable" with space without its body? What would their body like to be alternative religion that focuses on the role of particular religion and views some form of religious ethics as the form when the law is applied to that particular religious community This discussion is concerned with the status of faith for individuals or groups which may be members and members
computers: macintoshintochhntoshhttt
religion: space legal religion religion religion This essay discusses
science: the chemical microscope is This essay discusses the mechanisms of reaction with focus on the molecular structure of nucleite and of enzymes within the cytoskeleton, thus making it easier to understand process metabolism and other elements cellular life. this essay, we use techniques such as the photochemical transfer
military: arms defense battalion battalion cavalry This essay discusses three main themes:) nnnn 1) Lack of uniformed soldiers is an unacceptable and unconscionable strategy for the Army \n 2) Poor and inade- quate training does not compensate the soldiers, and may deprive them of the necessary and competitive training from their instructors
Table 8: Prompt samples from SQL.
21

---

Under review as a conference paper at ICLR 2022
legal: there liable injunction In summary, the court decided that defendants had a right to petition the high court intervene, and therefore granted the injunction. However, Court of Appeal decision in the US District Court ruled with no effect on the petitioned by the plaintiffs, citing two reasons. \nn, The US politics: the primary referendum was In summary, the majority of Russians would support the idea of banning all human rights, and that would be major part of the government's effort to build new liberal economy, which President Vladimir Putin says would boost Russian tourism. \n),n The results were published in the computers: the macintoshintosh summary, the kernel has the best quality of life, and kernel security is at the heart of any OS development. The first time I Linux released was when 1386 released the Macintosh version, which had a rather low standard. Other issues like the introduction of the OS6 operating system space: comet asteroid spaceship In summary, this asteroid asteroid ship is about make its maiden flight to explore deep-space. |n) "The asteroid craft was a very early attempt at the discovery of a new home for the world's 's first black hole," NASA said in a news release. "Today we
religion: faith faith faith salvation In summary, Christian beliefs are not new way to use the time spent thinking about God's world source for faith. Faith is an effort to think of the world without fear that might become dangerous place for the human family. Because it represents the very essence that science: climate research chemistry In summary of the study, this review aims to determine how a single study where the same number of data was analysed, a new methodology is needed to better understand who produced a different graph than the one suggested. The paper will be published in issue #5, Issue #18.
military: the cavalry battalion In summary, the army are a unit the same type and in all, so there no need to declare one. The unit does not constitute a cavalry unit or for use on troops. \n),n The army is not under the command of a brigade from the front. For
legal: there liable injunction This essay discusses the potential legal consequences of stay in the United States for an indefinite period of time if the government continues delay the process of de-instituting it. To apply such a request, all applicable laws shall apply either the same terms as the existing statutes. In politics: the primary referendum was This essay discusses the electoral strategy against a candidate for governor of the Commonwealth. \n (n The survey of British voters in this survey provides an overview of what the candidates for the United Kingdom will be seeking in the next Parliament. In the general election few seats will lead up to a computers: the macintoshintosh This essay discusses the various problems of the Macintosh, the first two- year running environment. An early version of this paper was originally published in 1982. The MacSX was not designed and managed by Kia. /nnn Macintosh n The mac has heen family invention space: comet asteroid spaceship This essay discusses topic: the impact of two of the Farth's 's two-thirds comet-sized moon Charon on Earth, and why asteroids are so close to the sun; why people are looking for ways to find a way to keep Earth-shaper asteroids out of orbit.
religion: faith faith faith salvation This essay discusses the impact religion has on the American experience and in American culture. Since the beginning of my career have found that faith and belief have often been linked to cconomic growth, social development and education. I believe that all people need to know that there is no reason for
science: climate research chemistry This essay discusses the role of molecular information and its inter- action with the general organism and human health. An "The idea of biological information is not really new concept. We used genetic information as medium define, identify, and store information about biology and biology." explains military: the cavalry battalion a This essay discusses the potential for the development of a small infantry brigade as an infantry regiment. It is also contribution to the larger cavalry corps as it would require larger brigade for battle. For more information see the original article on this page.
Table 9: Prompt samples from MLE+PG.
22

---

Under review as a conference paper at ICLR 2022
legal: In summary Currently: In 1966 the Act amended into state of law through amendments. \n' n')n',n Defent No. 695 [The character in question for judicial decision purposes; participation concerned you; "but not acceptance.") "n n Generally held: Just
politics: In summary Senate candidates, senator (Republican); senator (Democrat); and opinion-former (2002-08). 2012 Senate results based on the federal Election Commission's 's October 2016 Current Opinion Polling Reports. Key figures: Open Gallup poll Most Americans view the computers: In summary: 12-16 add-on chips. Trace out the type predefined ORDER parameters, and write to /dev/tty with them. )
.n),n') an) (n) n')nn,n mnnnnnn' n'nnin Roundset sizes with mmm831xx8 18800 x870 x81f); space: In summary Space Station Farm Station (1985 by Mike Lazarra) Here is an article developed by Maregnus Spirit Experimentator
Win Viotrv An exploration benefit for compute- eerruee- array data densities (UPERS). thesis home
religion: summary nice things about Android 6.1 Jelly Bean!),n Searching for OP lag fixes one my clles or some other improvements that's fixing a bug due to this nerf! (model causing Huge Frame Decay!) It also fixed upper turret hook
science: In summary Computer Age Experience Overview's (a)n) (nl veloped skill for Interneuhumb members at universities at this time. computer science, we are introducing various gatekeepers to intellectual property ownership and cyberware acquisitions. entry pro- gram makers post
Networking skills the most de-
military: In summary Army Sgt. Harold Tolbard (5/16/2018) 18) Lt. Gen. Michael Bachaes Dickie Powell Lt. Zachary Bram *Gen. Robert Eisen: Warfighter Soldier + (amourred)pn'n' Senior Bush
Doctrine
legal: This essay discusses Illinois cases on issues such as drug trafficking and drug Social Security.
politics: This essay discusses federal ethics as the key area on which current and past state and local governments have been operating.
computers: This essay discusses very development of alternative technology for young people.
space: This essay discusses NASA's StarHubble satellite mission development. Transcript here.
religion: This essay discusses various aspects of the relays of mediocality and Hammazanna.
science: This essay discusses Limux desktop computing, and IRI video-game applications. \n)n The after the GNOME transition
zooming in have run a software operating system so vital that any hacker can mine it military: This essay discusses military courage that included in the combat operations Iraq and Afghanistan.
end, all about figuring out how you
Table 10: Prompt samples from GeDi,
23

---

Under review as a conference paper at ICLR 2022
legal: In summary we have published in the journal Nature Neuroscience: A systematic review human brain tissue has found no evidence for any association between the presence of the presence of a partic- ular form of human neuropathy in the brain, a condition that is not normally associated with cognitive impairment. We found that politics: In summary we have list of 10 of the hest and most common types of drugs for people with HIV. This is a very short list of recommendations from a national and international community. )')nn' n'mm,n This article has been updated to make the official state of the EU state of computers: In summary. we believe that the current system has way of doing anything about it. Aa/aa/annnnnn The following steps are taken get the system working.) ) 1. Install new operating system with a Linux Mint operating system' (n 2. Start new Linux Mint operating space: In summary we have some important news from the moment of the year and some important information about these two major plancts. This new discovery is the first to confirm this important planct has active life in its home planet, planet with religion: In summary, we believe that the current administration has no way of doing anything about the Benghazi attacks. This a very interesting story, and think has been a very nice surprise. This is very nice and well thought out piece that is a must for the science: In summary we use this approach to evaluate if the number of data points (in the dataset) that are relevant for each data is the same (in this case, the data are not in data set). In this approach can test the data points in different way.
of about 5.8 billion tons. It
military: In summary we have some important news from the moment of the year and some important information from the moment of the year. (n),) n'nn',n We' ve also added an additional update for our new feature, which includes:) n Improved access and access in all of the main legal: This essay discusses how you can build community of dedicated people. If you're 're a member of community of people who want to contribute to the environment, you'l 'll also be helping them build communities in order to support the local economy, and the future of the city. The latest report politics: This essay discusses how we can build on previous research findings about the role religion plays in human development in human development. This is a very interesting and highly entertaining story. What is an "independent" political party in the United States, the U.S. political party, and the United computers: This essay discusses how you can build a new browser to view and share your favorite web sites. \n' n A browser that is open source can also be built from a web browser, which can he a browser that does not allow browser extensions (e.g. Firefox, Chrome, Opera
space: This essay discusses how you build life with a healthy diet and how you you're ready to move forward. It's a very simple approach to building a life with a healthy diet and what means to be healthy and healthy for the
it when
religion: This essay discusses how you can build new game without having to play the original game, and how you can make new title that is completely different to the original. It has been around since 2007, when the first game, The Elder Scrolls IV: Oblivion, was released in the PlayStation
science: This essay discusses how we can build on previous research findings about the role of obesity in human metabolism and how we can improve our health. (n),)) n))n n In this essay, we explore why eating a whole whole diet does not help prevent obesity (I). We find that whole food diet military: This essay discusses how you can build community with the help of friends and fam- ily. nn n 'The people around me are the ones who need help. They are the ones who need help. They are the ones who are not alone." Michael n "It's
Table 11: Prompt samples from PPLM.
24


Reviews:
[{"review_id": "gS0M-NCc5Fw", "reviewer": "Reviewer_2uap", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper models text generation as a reinforcement learning problem, and uses the existing reinforcement-learning techniques (soft Q-learning and path consistency learning) to relieve the sparsity reward and large action space problems in the existing reinforcement learning based text generation methods. \n\nStrong Points:\n- The paper is well-written.\n- The illustration of this paper is clear and good.\n- The proposed methods achieve moderate improvement over previous methods.\n\n\nWeak Points:\n- The novelty of this paper seems limited as the techniques used in this paper are directly from the reinforcement learning area (soft Q-learning and path consistency learning). Considering that the reinforcement learning methods had been widely used in the text generation methods, e.g, MIXER [1] SeqGAN [2], the idea of using reinforcement learning methods for the text generation seems less novel.\n\n- One of the motivations of this paper is to relieve the sparsity problem in text generation. However, this problem had been clearly pointed by the previous text generation methods, e.g. LeakGAN [3] and IRLGAN [4]. Specifically, IRLGAN tried to relieve this problem using the inverse reinforcement learning method, which is highly related to the submission. However, the paper does not compare with them and even does not mention them. This is limiting.\n\n- How significant is the sparsity problem in the chosen three text generation tasks? If the authors emphasise the sparsity reward problem is significant in the text generation, the related experiments, e.g. generating different lengths of texts, should be performed. However, the related analysis is absent in the paper.\n\n- One of the baselines in this paper is MLE+PG. There are many MLE+PG text generation methods, but authors do not clearly state which one they use in the paper, which leads to confusion.\n\n- The paper shows the generation results of the prompt generation which is an interesting application of text generation. However, there are many meaningless and less fluent generated prompts in the shown examples, e.g. macintoshintoshintoshintosh in Table 7. What causes this phenomenon?\n\n- In addition to the prompts results, It would be better to show generated examples of other tasks.\n\n[1] Ranzato, M. A., Chopra, S., Auli, M., & Zaremba, W. (2015). Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.\n\n[2] Yu, L., Zhang, W., Wang, J., & Yu, Y. (2017, February). Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence (Vol. 31, No. 1).\n\n[3] Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., & Wang, J. (2018, April). Long text generation via adversarial training with leaked information. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).\n\n[4] Shi, Z., Chen, X., Qiu, X., & Huang, X. (2018, July). Toward diverse text generation with inverse reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (pp. 4361-4367).", "summary_of_the_paper": "This paper proposes a new text generation framework based on the existing soft Q-learning of reinforcement learning. The experiments demonstrate that the proposed text generation method achieves superior performance to baselines. ", "summary_of_the_review": "In summary, this paper is well-written but it lacks relevant analysis and experimental evidence for the problem they want to solve. Further, the method they proposed is directly from the reinforcement learning area, which significantly reduces the contribution of this paper. Last, the paper lacks the experimental comparison with the related work aiming to solve the same problem. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q8j4qC01TTS", "reviewer": "Reviewer_FPpH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I think when mentioning text generation, people will recall language modeling or machine translation. This paper only did experiments on some tasks which I don't think it's central to the topic. I think it's a misleading title and content. Since authors used this title, I think at least one of these 2 central tasks should be added in experiments.\n\n2. I think overall all the technical details are not new (correct me if I am wrong, and in this case kindly pointed out what's the contribution). It reads to me that the major contribution is trying some existing technical methods in these sort of problems. If that's the case, I feel it's more appropriate to submit to EMNLP I guess. \n\n3. On technical issues, I feel like the claim is not well justified. In many places, authors claims SQL is the important thing in overcoming the existing problems. However, in SQL the major concept to me is the entropy term. If that's the case, that means we want the term to be large and it translates to we want an evenly distributed actions space and in this thinking, normal SARSA or Q-learning or with larger epsilon greedy search should also work. I think if SQL is the reason, this should also be verified.\n\n4. Somehow it reads to me that SQL itself is not even enough to overcome the challenges so path consistency learning is the most important. So I think 1) there should be an ablation analysis on SQL only, and 2) I feel path consistency learning can be applied to other RL problems as well, which should also be included. And if it cannot be combined, please discuss the reason.\n", "summary_of_the_paper": "Authors proposed to use soft Q-learning (SQL) to formulate the RL problem in sequence generation. The stability can be increased when techniques of path consistency learning is corporated.", "summary_of_the_review": "Overall, I have the experience in using RL for machine translation so I agree with the authors that those problems mentioned in the paper are indeed challenging. But I am not familiar with SQL but from the derivation provided in the paper I am skeptical on the effectiveness of SQL. Since most if not all technical stuff are existing work, I think the novelty is limited and thus justification of each component in the method is important to me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Em37eNsBBA", "reviewer": "Reviewer_jb7t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: This paper establishes a connection between the text generation problem and path consistent learning. It leads to a principled and elegant RL-based text generation method that naturally inherits many desirable advantages from PCL algorithm, such as being capable of using both on- and off-policy samples. It is widely applicable to text generation tasks where MLE training is not directly applicable, and demonstrate superior performance on 3 such tasks.\n\nWeakness: The work is almost a direct application of PCL to text generation, and there is no newly developed RL algorithm for text generation. This may not be a serious weakness given that this paper mainly focuses on RL-based text generation problem, and the good performance could be encouraging for research on RL-based text generation methods. The experiments on standard MLE-based tasks are relatively small scale. Demonstrating competitive performance of SQL on standard machine translation tasks would be more convincing.\n\nComments: \n- It would be helpful to explain the intuition of the single-step PCL loss (9) and the multi-step PCL loss (10) in the text generation context. In particular, compare its intuition to the MLE loss could make it better received in NLP community.\n\n- It seems that the perplexity of the proposed method (SQL full) is not as good as the others (Figure 3 Middle and also Table 1). The authors may need to discuss it more thoroughly.\n\n- Compared to the original PCL learning, the SBEED algorithm [Dai et al 2018] provides a provably-stable algorithm for path consistency learning when there is nonlinear function approximation (e.g., by using deep neural networks to parameterize the Q-function, the state-value function and the policy here).\n\n[Dai et al 2018]: “SBEED: Convergent reinforcement learning with nonlinear function approximation”, Proc. ICML 2018.\n", "summary_of_the_paper": "This paper considers the problem of learning text generation models using reinforcement learning. The problem is challenging in that RL algorithm becomes inefficient or unstable when dealing with large action space and the sparse reward situations in text generation. To address these problems, this paper adapts the path consistency learning approach to the text generation setting. It allows to train the model with both on- and off-policy samples, bridge the sparse reward signal to directly supervise the Q-function learning, and makes efficient updates to Q-values by considering all candidate actions together. Experiments result show that the proposed approach are effective in solving a wide range of applications where MLE are not applicable, where it achieves better performance than previous RL-based methods as well as task-specific methods. In addition, on standard MLE-based tasks, the proposed approach could also achieve competitive performance when training the models from scratch.", "summary_of_the_review": "This paper establishes a connection between text generation and path consistent learning, which leads to an elegant RL-based text generation algorithm that inherits many advantages from PCL algorithm. The performance of the algorithm is encouraging for RL-based text generation, although having more large-scale experiments would be more convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes an approach to improve non-ML based methods of text generation. It reformulates the problem with the soft Q-learning approach from RL instead of standard hard RL formulations from previous text generation work. By doing this, the work allows application of path consistency learning. This is an elegant formulation. However, this reformulation into soft Q-learning appears quite straightforward and so the application of path consistency learning does not require much change to be used for text generation. This limits the novelty of the work. The experiments are also relatively small-scale and consists of some non-standard tasks such as prompt generation (which is typically evaluated indirectly, the response to the prompts rather than the prompt itself). As the reviewers mention, evaluating on more large-scale standard tasks such as summarisation or dialog would be more convincing. Finally the work lacks references to recent works in the field, such as LeakGAN.
Decision: Reject