Controlled Text Generation as Continuous Optimization with Multiple Constraints
Authors: Sachin Kumar
Eric Malmi
Aliaksei Severyn
Yulia Tsvetkov
Venue: NeurIPS 2021 Poster (2021)
Keywords: controllable text generation
constrained optimization
style transfer

Abstract: As large-scale language model pretraining pushes the state-of-the-art in text generation, recent work has turned to controlling attributes of the text such models generate. While modifying the pretrained models via fine-tuning remains the popular approach, it incurs a significant computational cost and can be infeasible due to a lack of appropriate data. As an alternative, we propose \textsc{MuCoCO}---a flexible and modular algorithm for controllable inference from pretrained models. We formulate the decoding process as an optimization problem that allows for multiple attributes we aim to control to be easily incorporated as differentiable constraints. By relaxing this discrete optimization to a continuous one, we make use of Lagrangian multipliers and gradient-descent-based techniques to generate the desired text. We evaluate our approach on controllable machine translation and style transfer with multiple sentence-level attributes and observe significant improvements over baselines.

Controlled Text Generation as Continuous Optimization with Multiple Constraints
Sachin Kumar
Eric Malmi Aliaksei Severyn R Yulia Tsvetkoy
Language Technologies Institute. Camegie Mellon University, Pittsburgh, PA, USA
Google Research
Paul G. Allen School of Computer Science & Engineering, University of Washington
sachinkécs. cmu.edu, {emalmi, soveryn)@google.com, yuliatsécs. washington edu
Abstract
As large-scale language model pretraining pushes the state-of-the-art in text gen- eration, recent work has turned controlling attributes the text such models generate. While modifying the pretrained models via fine-tuning remains the popu- lar approach, it incurs a significant computational cost and can be infeasible due lack of appropriate data. As an alternative, we propose MUCoCO flexible and modular algorithm for controllable inference from pretrained models. We formu- late the decoding process as an optimization problem which allows for multiple attributes we aim to control to be easily incorporated as differentiable constraints to the optimization. By relaxing this discrete optimization to continuous one, we make use of Lagrangian multipliers and gradient-descent based techniques to generate the desired text. We evaluate our approach on controllable machine translation and style transfer with multiple sentence-level attributes and observe significant improvements over baselines
Introduction
Recent advances in language models [11 52] trained large-scale web text corpora have led to great improvements in state-of-the-art on many natural language processing (NLP) tasks including the ability to generate increasingly coherent text [3]. However, once such models trained, they are prone to degeneration [66) and biased, non-factual outputs [16] 45] as it is difficult to control the characteristics attributes of the generated text without architectural modifications
26 28 35 and
fine-tuning the models on attribute-specific corpora 229] 7 This can be even more challenging multiple attributes are involved labeled data for each combination of attributes can be difficult to obtain.
We focus controlled text generation where the goal is decode from a text generation model such that the outputs satisfy certain constraints, which the model was not necessarily trained on. For example, given dialogue generation model, additionally constraining the generated responses to be polite, although the model was not optimized for politeness during training. Recent works address this problem with left-to-right autoregressive decoding, and modify the vocabulary distribution every step directly using classifiers or language models trained on attribute specific corpora 72 40 [3]] or
indirectly via backpropagating gradients through model activations [8]. While exhibiting high level of attribute control, by design these methods can only work with categorical attributes (typically only one attribute) and condition only on the left context while decoding. Additionally, they often require several heuristics to work and are prone to adversarial outputs [64].
'The code is available at hatttp:///gttuub...mm//hhhhiiiii
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To address these concerns, we propose the following decoding algorithm. Given a pretrained language model, we posit decoding from it as an optimization problem. First. we relax this discrete optimization problem to a continuous one by representing each token as a simplex on the target vocabulary [19] This allows us to use continuous optimization techniques like gradient-descent considering each token distribution as parameters; while keeping the language model's ' parameters fixed ($2). Second, we represent each target attribute to control as a differentiable function. We formulate controllable decoding as a multi-objective optimization problem, with maximizing the log-probability of the language model as well as target attributes as objectives. To make this optimization feasible via gradient-descent, repurpose it to constraint optimization problem and solve the dual using the modified differential method of multipliers [48]. We call the algorithm MUCoCO, for incorporating multiple constraints through continuous optimization.
We validate MUCoCO on three conditional text generation tasks with different types of sentence level constraints: (1) Adding formality and cross-lingual similarity in a machine translation model; (2) Ensuring transfer and content presentvttietetoooonn in a style-transfer model, and finally (3) Incorporating multiple styles and attributes (e.g., formality, sentiment magnitude, writer's age group) in para- phrasing model. With automatic as well as human evaluations we find that our proposed method outperforms strong baselines.
2 MUCoCO: Constrained Decoding as Multi-Objective Optimization
For given language generation task, let G model the conditional probability p(y|x) of the output sequence y . Vn. given the input sequence x .......:: -Tn- This model can be parameterized using any differentiable architecture like Transformers [62] or LSTMs (20) and trained with any loss function [0]]0]] either autoregressively or non-auloregreessieely [17. Traditionally, given input x, decoding from such a model requires finding the output sequence with the highest probability or the lowest negative log-probability, y* arg minyey - log P(y|x) Here Y is the set of all possible output sequences. In practice, searching )' to find the highest probability generation is intractable as the space of possible sequences grows exponentially with sequence length and has also been shown to produce undesirable solutions [60]. In most prior work, simple heuristics like beam search, or sampling have been adopted find approximate solutions, where the text is generated one token at time (usually left to right) with the output of step f. being fed to the input at step + 1.
In this work, however, given G and an input sequence we are interested in finding an output sequence y that not only maximizes the output probability but also optimizes multiple objectives defined over and y. More formally, we seek to find a y that minimizes all of the following objectives
y" = arg min( (ainn-lloye (loppoloox))))))_)mo .... (.)),, July), y). 9., yey
(1)
Here each f; is function defined over the output sequence for example, the negative log- log-probyyility of an attribute (e.g., formality) classifier we want the output sequence to satisfy. And each gj is a function defined over both the input and output sequence, for example, semantic similarity between x and y [55] We assume all fa and gj are differentiable. This is a multi-objective optimization with several possible solutions.
Since there are many objectives to minimize, a left-to-right decoding strategy like beam search or sampling will simply not work due to several reasons. First, the objectives S; and gy are sentence-level and hard to define accurately only on generated left-context [72 40) Even if we are able define them, as we add more objectives this process becomes very computationally expensive. Following prior work (19 [50] we formulate this as a continuous optimization process instead of standard discrete one, and then use standard algorithms for continuous optimization (like gradient descent) for decoding. We maintain noft-representation of the sequence y, ý (f1...., ... ....)..,...... where each UA E By is a simplex over the target vocabulary of size V, representing the probability of the k-th token. To decode a sentence, we initialize each ÿi uniformly over V, and treat the entire output sentence as the parameters for gradient descent keeping the parameters of G, Jú: 9j fixed. After gradient descent has converged, we generate discrete text by selecting the token with the highest probability in yr. We provide more details on the optimization procedure in 2.2

---

To make optimization feasible, a multi-objective problem generally yields itself to the following formulation:
arg min a log poy
A + Σ 199, (x.
(2)
for some statically or dynamically computed weights A, and H4 for each and j. where a + Ej X E /4 Although this weighted summation formulation is intuitively appealing. it typically requires expensive grid-search over the various scalings or of heuristic (25 6 [8] Fur- thermore, this formulation by definition assumes a trade-off between the different objectives by essentially assigning an importance weight each of them. This problem further exacerbated when different objectives have widely varying scales with smaller scale objectives just getting ignored. More concretely, multi-objective formulation as we define (I) admits several possible "optimal" solutions also known as the Pareto set [9. The image of the Pareto set is called the Pareto front. Since we define all objectives using neural networks, the Pareto front in our case non-convex, where linear combinations of objectives are shown to be unsuccessful in finding good solutions (see figure for an example).
36 37 I0)
194)
350
3/5
(a) Linear combinations.
(b) MUCOCO without damp.
(c) MUCOCO
Figure I: Loss curves for gradient descent for different configurations for an example of machine translation with a cross-line moos-llnggul semantic similarity constraint (XSIM < 0.15). For each experiment, gradient descent (for clarity, we plot the loss values for every 10 steps). See $3.2 for detailed results. Left: all cases one the objectives is favored while other fails decrease. Middle: We observe fluctuations in two losses. Right: The losses decrease much more smoothly leading to a better minimum.
do 100 steps of
Ideally, our goal is tunable optimization algorithm that finds solutions on the Pareto front, i.c., every solution on the Pareto front should have hyperparameter value for which the optimization algorithm finds that solution. In order to achieve this, we reframe our optimization problem as a Lagrangian optimization problem instead. We choose one of the losses the primary objective and consider other losses constraints. The goal is to minimize the primary loss subject to the secondary losses, each below threshold value. More formally,
arg min - log Ply(x) subject to
fil)) S G, E {I,.... ,u] 9, (x & En.j (.. .v.
Here and ε, are tunable hyperparameters whose values' change can result in different solutions on the Pareto front. This formulation leads to an intuitive interpretation generated text from the model G should satisfy the constraints while being as faithful to the primary objective as much as possible Consequently. original total loss linearly combined as in (2) given by
decoding process that the
Lagrangian we end up with looks similar to our
Cly,Ar..... Nus Mi, (.)) logp(y(x)) EMCE - f(()) ΣMS (x v))
(3)
For example, classifier log probabilliies are in inf) while sentence similarities usually lie [0.1].
'For example, defining fily) = p(aly) as the probability a desired attribute in y leads to natural threshold of fr > 0.5. For well-calibrated fr, an even higher threshold could be used for inducing highly indicative features of in y.

---

where Ar. are Lagrange multipliers, and an optimal output y" can be obtained as y" arg miny 20, (, Ag. However, the traditional method of solving the dual func- tion to find Mi that matches 6i; E, respectively, again leads to linear trade-off between the various objectives. When the Pareto front is non-convex as in our case, with gradient-descent, the constraints can be ignored and we still cannot always find optimal solutions by tuning <4 E [48]
2.1 Modified Differential Method of Multipliers
The fundamental issue in both linear combination of objectives and solving the dual is that fixed scalings A, and μi (manually pre-determined or obtained by solving the dual) do not work well with gradient descent to minimize for y. Following prior work on differential method of multipliers [8]. we propose to use a single gradient descent to optimize for both Lagrangian multipliers and y simultaneously as follows:
y()) = - V,
1/2
V Vx p = 14,
72 u..C
(4)
We follow the gradient of L downwards for the y (descent) and upwards for the multipliers (ascent) while making sure that the multipliers remain positive (by setting the multipliers to 0 whenever they become negative). Intuitively. this algorithm works by increasing the value of the multiplier with each gradient step as long as the constraint is violated. But when the constraint is suddenly satisfied and the multiplier is still large, it might take a number of gradient steps before the gradient descent pushes it to 0, thus causing the solution to be pushed further away from the constraint. As soon as the multipliers become 0 (or negative), the constraint is ignored and the process continues. However when the optimization hits the constraint again, this whole cycle repeats, resulting in "oscillations". We introduce a dampening parameter to each of the multipliers to reduce these oscillations (again following Platt and Barr (8)) and update the Lagrangian as follows:
Cly, Xs, Mb) -lggply/x)
Σ (A. -G)GG( f((r) - Imn ky) 9 (x, y)),
(5)
where S; = d stop-grad grappeggadien((e-- fi(y)). V/3 = d stop- shoo--rrdieen(() (z, and d is hyperparameter. d does not affect the final y. just how quickly the algorithm converges to it (We use d = in all experiments). computational graph and does not contribute to the gradient computation. When a constraint is not satisfied (e. fi( < O. hence G 0), the dampening parameter Ci being negative incurs higher penalty on the violation than when not using any dampening, without actually increasing the value of X too much. But when the constraint is satisfied, it helps quickly reduce the value of penalty being
(oop-rradien(()) indicates that the argument is detached from the
incurred on the constraint while the multiplier converges to
2.2 Optimization: Exponentiated Gradient Descent
Our goal is to generate a sequence of discrete symbols y = y1...., ..... where y/ from the target vocabulary. To make continuous optimization like gradient descent feasible, we adopt soft-relaxation 19 to represent each yk as a probability simplex, ÿR Ay (i.e. S GR S 1 and
Σ 0M1 = 1). Intuitively, it gives the probability of each token in the vocabulary. To compute the loss L during forward pass, we first convert dA one-hot vector ia via straight through estimator (2). This allows gradients to be applied to ÿA during the backward pass. More formally, soop-gradient.
+ Øk. During the forward pass, the input embedding tables corresponding to G and each of the constraints' models receive one-hot vector ik at each step k. and the input embedding is computed as a weighted-sum of the embedding weights. But in the backward pass, the gradients are applied to jv F
dx one- hoeehot(( (arg max ÿı+)
This relaxation, however, adds another constraint the objective L that each parameter ÿu should be a simplex. We use exponentiated gradient descent (2 19 to solve this problem which modifies the gradient-descent update shown in (4) as: Vk
(t)
(t-1)
exp( m V i. L). After every descent step,
vk is normalized to make it simplex.
Unlike prior work 19 50 59
we do not feed ÿ directly to the model as in our early experiments we found
that it leads to slow convergence.

---

2.3 Preventing ad adversarial solutions: Annealing the thresholds
Finally, it is well known that most neural network based models are not robust to noise and in fact gradient-based methods have been used generate adversarial examples for text classifiers (59]. We find in our early experiments that using these models to define constraints can also lead to such cases where the constraints are rapidly satisfied but the generated sentences are disfluent. To prevent this issue, we introduce an annealing schedule [47] during the gradient descent where we start with relaxed thresholds e,, Ey such that they are all satisfied and only the primary loss - log p(y |x) is active. As the optimization progresses, we gradually decrease the value of the thresholds causing the constraints to get violated resulting in the optimization gradually shifting to updating y to satisfy them. The exact schedule we use is described in the next section.
The final decoding algorithm we use in all our experiments is described in the Appendix algorithm I
3 Experimental Setup
We evaluate MUCoCO on the following controlled generation tasks: reinforcing target style in text generated by style transfer model $3.1 and adding formality to a machine translation model (:3.2). Additionally, we conduct a qualitative analysis of rewriting product review adhere multiple expected attributes like formality, sentiment magnitude, and age group of the author ($4). These tasks include constraints corresponding to both expected attributes in the target sentence (like formality) as well as both source and target sentences (like semantic similarity) with up to constraints per task.
Implementation Details For a given sentence length T. we initialize each simplex J1.-... Dr uniformly over the vocabulary. We use exponentiated descent learning rate of 77, 50 for y and ascent learning rate of 7/2 2.0 for the multipliers, and run the optimization for 100 steps. Given all intermediate solutions y*t), we choose the one which satisfies the constraints and has the minimum value of the primary objective. For each constraint, we use the following annealing schedule: we start with an initial value and linearly decrease it at step 40 until it reaches the desired value at step 80, after which we keep it constant. Additionally, since the length of the target sequence is not known in advance, we first greedily decode from G till the end-of-sentence token is generated resulting in sequence of length L. We then use our approach for each T E { - 5, ..... L + 5} and choose the one which (a) satisfies all the constraints and (b) has the minimum value of the primary objective. However, this optimization objective is highly non-convex and may get stuck in a local minimum
where constraints are not satisfied. If none or partial constraints are satisfied, we choose the output based on (b).
3.1
Style Transfer
We begin with style-transfer task, task aiming to faithfully and fluently rewrite a given sentence such that desired writing style is reflected in the generation. This task has been widely studied [22]
58 29, among others] and differs from related tasks like sentiment transfer 661 33 34] where flipping the sentiment usually comes at the cost of changing meaning.
Style transfer is usually evaluated across three dimensions: (1) does the output sentence conform to the expected style; (2) does the output sentence preserve the input's ' meaning; and (3) the generated sentence fluent. Most prior work in style transfer focused on devising training objectives serving as proxy for the desired outcomes, for example, back-translation 49(3] or paraphrasing (2 for content preservation and language modeling for style and fluency. But depending on training algorithm and available data, there is often an observed trade-off between transfer and content- peatent-resservttoon [49 33) To that end, we add the desired attributes via explicit constraints when decoding from existing style transfer model.
More specifically, we consider the task of informal to formal transfer 53 with the state-of tfite-of-thee -he unsupervised model STRAP from Krishna al. (29]. This model trained in an unsupervised fashion by (1) generating (which results in a demotion of stylistic attributes), and (2) training an inverse-paraphrase model to translate paraphrases back to the original formal style. At test time, given an informal input sentence x, the model first generates its paraphrase z, then using an awvers-paraphree model to generate the output y. We train this model by fine-tuning GPT2 (345M) [5I] with the GYAFC Corpus
-aaralle corpus by paraphrasing each formal sentence in the training set
5

---

(Entertainment/Muscc domain; around 50K formal sentences) 531 and evaluate it on the provided test set containing 1312 informal sentences. Krishna al. (9 report best results with greedy decoding. In MUCoCO we modify the decoding algorithm by considering the negative log- probability of given z according to the model as the primary objective, and incorporate the following constraints:
We train a binary classifier PFORMAL (y) by fine-tuning GPT2 on the same GYAFC training corpus, following default hyperparameter choices provided in Hugging Face [70]. This classifier outputs the formality probability of a sentence y. We add this output as a constraint to the decoder as
Formality:
log (PrORMALL (y)) < - (0g(0. In other words, the constraint is satisfied if the classifier assigns at least 0.5 probability of the output being formal. We initialize the threshold to 10.0 which is later annealed to l0g().
Semantic Similarity: Since the baseline style-transfer model takes as input the paraphrase z and not the original text x; it is susceptible losing some of the original content in x while generating y. To ensure content preservation we incorporate two kinds of objectives: (1) USIM(X, y) cosine (x), AM (1)) [55] where M outputs a continuous vector representation of given sentence. Similarity between x and is measured by cosine similarity of their respective representations. (2) WMD(x, y) takes as input bags of word embeddings of the two sentences and computes the Word Mover's Distance between them 32). This distance is computed by solving linear program. We adapt the alternating optimization procedure described in [3] to make this loss differentiable through the program. Intuitively. while USIM computes similarity between sentences taking context into account, it can be less robust certain missing or repeating tokens, whereas WMD measures lexical overlap between input sentences acting as proxy for coverage. We discuss the two losses in more detail in Appendix [D To compute the thresholds for constrained optimization, we compute the average value of the two functions on the development set in the same corpus. We use USIM S 0.15 and WMD S 0.4 as the final constraints (with initial threshold values of 2.0 for each).
Baselines and Evaluation Metrics We compare MUCOCO with the following baselines:
NO-CONSTRAINTS: We decode directly from the model greedily without any constraints. This replicates the best result reported by Krishma et al. [29]. We do not use continuous optimization to do unconstrained decoding it has been shown to perform similarly to left-to-right decoding in prior work 19).
FUDGE: Introduced by Yang and Klein 72. this method decodes in an autoregressive manner. modifies the output vocabulary distribution at every step by interpolating the language model probability with that of formality classifier. This classifier is trained to predict the probability of entire sentence being formal given only a prefix (we train it similarly to PFoRMaI(y) by fine-tuning GPT2). This method only works with categorical features like formality and is not extensible to constraints like semantic similarity. We decode using the hyperparameters recommended in [72].
To show the benefits of the constrained optimization setup, we show additional comparisons with a linear combination of objectives in Appendix
Following the baseline model Krishna et al. (29] we evaluate the generated sentences with the following metrics: (a) fluency grammatical wellformedness measured by the accuracy of RoBERTa-based classifier model [4]] trained on CoLA (65), averaged over all outputs, (b) transfer: measured by a RoBERTa-based classifier model [41] trained on the GYAFC training corpus, and finally (c) WSIM [68] a subword embedding based similarity model trained on a large-scale para- phrase corpus which performs well on STS benchmarks 4 as well. We measure this metric both with respect to the input and the provided references F In addition, we also report USIM.
Results The style transfer results summarized in table [ If we only incorporate formality constraint, we observe that compared to FUDGE our method significantly improves transfer accuracy at the expense of content preservation. Adding semantic similarity constraints the other hand improves both transfer as well as content preservation with the largest gains achieved when all the constraints are considered together. Qualitative analysis shows that MUCoCO's 's outputs are typically more fluent and have stronger formality signals, but all of the models are prone to propagating errors from the paraphrasing model (see examples in the Appendix table 4
'Eac input sentence has 4 references, choose the highest WSIM value compute the average.

---

Content Preservation (w.rt. imput)
Content Preservation
(wrft. ref)
Method
Constraint
Fluency
Transfer
WSIM
USIM
WSIM
USIM
STRAP
None
91%
78%
0.69
0.77
0.72
0.80
FUDGE
FORMAL(y)
90%
85%
0.71
0.77
0.73
0.81
MUCoCO
FORMAL(y)
89%
93%
0.67
0.75
0.72
0.78
MuCoCO
USIM(X,
92%
85%
0.71
0.78
0.74
0.81
MuCoCO
USIM(X, y). WMD(X, y)
92%
87%
0.73
0.79
0.77
0.86
MuCoCO
SIM(X, y), WMD(X, y), FORMAL(y)
93%
92%
0.71
0.79
0.75
Table I:
Automatic evaluation of fluency, formality transfer, and content preservation for informal-to- formal style transfer models.
3.2 Style-controlled Machine Translation
We now evaluate MuCoCO in the task of formality transfer machine translation. Given a trained MT model, decoding is often done using beam search and the highest probability beam candidate is chosen as the final output. Prior work has explored adding rule-based or heuristic constraints such length penalty coverage [7] to rerank beam candidates, and adding lexical constraints like penalizing n-gram repetitions [21] In this experiment, we target sentence-level constraints which are otherwise difficult to incorporate
left-to-right decoding process. Given a trained MT model and the source text I, we use negative log-probabiling of the translation under the MT model as our primary objective and incorporate the following constraints for decoding in different combinations:
Cross-lingual Similarity Similar to USIM, we define XSIM(x. = cosine (CMM CMM()) where CM is multilingual encoder trained by distilling monolingual model like M described ear- lier [56]. More details training are available in the Appendix D Averaging across the development set, 0.2 the threshold for the constraint.
Formality Unlike style transfer, where the goal is to rewrite text the desired style, here we seek to generate translations in desired style directly from an MT model which was not explicitly trained to conform to specific style. We train a classifier PRORMAL (y) similarly to one described in previous section by fine-tuning GPT2, but with different input-embedding table to match the vocabulary of the decoder the MT model. Again, we use log PrORMAL (y) > log(0.5) as the constraint.
Baselines and Evaluation Metrics We compare MuCoCO with the following two baselines:
BEAMSEARCH: We decode directly from the translation model with beam search of size 5.
defined similarly as in the style transfer task but trained to match the decoder vocabulary. As mentioned before, FUDGE only works with categorical attributes like formality and is not easily extensible to constraints like cross- lingull similarity. We use the recommended hyperparameters by Yang and Klein 772 for decoding.
FUDGE 722
In Yang and Klein 72,, the authors also compare FUDGE with other baselines such as PPLM [ and BEAMSEARCH followed by style transfer. They show that FUDGE vastly outperforms these baselines. Hence, we only show comparisons with FUDGE in this work. We evaluate along the following metries: (a) BLEU [46]: a standard metric for evaluating MT, (b) BERTScore [75]: an embedding- based metric which more robust to changes in surface forms of the words than BLEU. (b) transfer: the same RoBERTa-baen
formality classifier as in our style transfer experiments. We also report XSIM, the constraint we use for decoding.
We experiment with French to English translation with a subset of the OpenSubtitles test set [38 containing 1360 sentence pairs This test set contains informal spoken language for both source and target. For the primary objective, we use the Marian Transformer based French (fr) to English (en) model (24] through Huggingface. We summarize the results of this experiment in table 2 with selected examples in the Appendix 5able
'We create this subset by filtering the original test set to contain only sentence pairs for which beam search translations are classified informal.
7

---

Method
Constraint
BLEU
BertScore
Formality(%) (%)
XSIM
BEAMSEARCH
None
42.1
0.932
0%
0.85
MUCoCO
XSIM y)
42.7
0.939
4%
0.88
FUDGE
FORMAL(Y)
39.2
0,922
6%
0.83
MUCOCO
FORMAL(Y)
37.5
0,913
30%
0,83
MUCoCO FORMAL(Y), xSIM(x,
39.8
0.935
23%
0.86
Table 2: Results of style-controlled machine translation experiments.
Results By just using a cross-lingual similarity metric without modifying the model at all, we observe +0.6 improvement in BLEU score well as BERTScore. Adding formality constraint leads to considerable gain in formality of the outputs with a drop in BLEU; using both XSIM and FORMAL helps recover some of the drop. The drop in BLEU is unsurprising: since BLEU a surface-level metric it naturally penalizes the translations that are rephrased to conform to formality constraints. Indeed, as shown table [3] adding a formality constraint leads changes in sentence structure and vocabulary. On the other hand, we see improvements in BERTScore which is an embedding based metric, more robust to paraphrasing.
To further validate our results, we conduct a human evaluation of the generated translations. We randomly sample 100 source sentences and their translations generated by beam search and MuCoCO with both FORMAL and XSIM constraints. Two annotators (highly proficient in French and English) to rank the translations on faithfulness (is the source meaning reflected in the translation?? and formality, The options are randomized. On the translation pairs where both annotators agree (79 out of 100), the ones generated by our method were favored by annotators 37% percent of the time, while beam search translations were favored only 18% of the time, and 21% translations were equally favored.
4 Discussion
Simultaneously controlling several attributes One of the main advantages of our proposed ap- proach is its flexibility introduce any number of constraints (as long as they are differentiable) to the decoding objective. To illustrate this advantage we consider the following problem: given sentence annotated with following attributes: age group of the author, formality, and sentiment magnitude, rewrite it such that any chosen combination of the attributes are modified while keeping the others fixed and the content preserved [42 33). For our primary objective, we use a inverse- paraphrasing model as defined in $3.1 which we train on corpus of Yelp Reviews
449 First, we paraphrase each sentence in the corpus described in Krishna et al. (29 creating a pseudo-parallel model to translate the paraphrases back the original reviews. We use USIM and WMD for semantic similarity constraints and three classifiers for (a) age group of the author (binary; < 30 years or > 30 years); (b) formality of the review (binary: informal or formal); (c) sentiment magnitude (five-class classifier ratings of to 5). Here we focus on sentiment amplification rather than transfer. That is, changing the 4-star rating of an input to 5 (or to 1). Details of the classifiers and the data used are provided Appendix D.2 Table examples of generated sentences with differrntt combinations of attribute values. We do not focus on sentiment transfer in this setting (c.g. changing a 1-star review to 5-star review) because it also changes the meaning of the utterance making semantic similarity and sentiment constraints incompatible with each other where satisfying one violates the other.
corpus (of reviews and their paraphrases) and train G as an lnverse-prrapher
Finding other solutions on the Pareto front As described in $ the thresholds e, are tunable hyperparameters that allow us find different solutions on the Pareto front. our experiments so far, based expected outcomes and how the constraints are defined, we showed results with only one threshold for each constraint. For example, ideally for a well-calibrated text classifier based constraint, this technique should be able to find solutions for any probability as threshold, but most neural-network based classifiers are not well-calibrated and predict the highest probability output as the label, hence natural threshold for clasiy- is label probability > 0.5. In Appendix
7 This corpus is sentence tokenized and lowercased with 2.2M sentences not labeled for any attributes. Duue to lack of an established benchmark for this task and due to many possible combinations of attributes. we do not report quantitative results.
8

---

table we show how the outputs change if we modify this threshold to different values. We observe that most cases the optimization converges generate words more commonly associated with formality. On the other hand, semantic similarity between two sentences is even harder to define, is less robust to noise, and varies with writing styles of the input sentences. As shown, increasing this threshold for semantic similarity can lead to repetitions and disfluency.
Speed and memory requirements The presented decoding algorithm treats each token in the output sequence y as parameter for gradient-desenn which involves multiple forward and backward passes through the primary generative model G well as attribute models. Given an expected sequence length L, it optimizes L x V parameters which is both memory and time intensive compared to left-to-right decoding, For example, on a single GeForce RTX 2080 Ti (12GB) on which we run all presented experiments, with a batch size of 1, our approach takes approximately 90 minutes on average to decode around I 220 sentences compared to around 20 minutes for FUDGE [2] with a single constraint. For reference, unconstrained beam-search takes 2-5 minutes. Given enough GPU capacity, however, this approach can easily be extended to larger- to improve decoding speed. We do not conduct this experiment due to limited available resources. Using 16-bit floating point operations, this can further be improved. Another way of improving memory efficiency would be to optimize not for tokens directly but instead optimize for token embeddings (3)]. This formulation also removes the requirement for all the models to share a vocabulary. We plan to investigate this in future work. Finally, given the capability of this approach to incorporate multiple constraints, it can also be used to generate pseudo pseudo-parallel parallel data with different attribute combinations which then could be used to train supervised models for attributes for interest resulting in faster models at inference.
5 Ethical considerations
Language generation is growing research area, and state-of-the-art techniques are still not powerful enough to facilitate fine-grained control over generated content. In the current form, large language models have the potential to generate harmful and biased language. For example, language generators are prone to generating toxic [1] and non-factual content [45]. especially when used maliciously [63] 64 [7]. Controlled text generation techniques can be used to mitigate many such problematic biases already encoded in large language models [6 40). They also have many other positive use-cases, for example, anonymizing personal attributes in written text [54], and even aiding authors in avoiding implicit biases their writing 44 14.. However, none of the existing approaches, including ours,
can sufficiently address these issues yet.
We also caution that there are additional risks of adversarial applications of controlled text generation research. The same algorithms that help us control for content preservation and mitigate biases can be used maliciously, generate misinformation, incorporate pernicious biases, target specific individuals to influence public opinion and seed polarization via manipulating the generated content. For example, when style transfer techniques are used in conjunction with users' personal attributes such as gender, they can amplify harmful social biases. We thus opted not to include gender transfer in our experiments.
Nevertheless, these issues should not discourage the scientific exploration that will advance the sfate-of-the-art in many positive usages of controlled text generation, including in machine translation, question answering, summarization, dialogue, etc. In parallel, future research should focus on developing better defense methods against mis-using these models maliciously, in a way that could cause societal harms 77).
Related Work
Recent work on controllable text generation can be divided into two categories. The first focuses models either through fine-tuning pretrained models with surribute-specffi corpora [B3 29 5 or via training conditional generative networks [33] 76 49 73).
on directly training
More broadly, this includes methods for text style transfer. Unlike MUCoCO, these methods are not easily extensible and require training or fine-tuning a new model to incorporate new attributes. For example, CTRL. (26 train large scale language model (1.6B parameters) from scratch with 55 control codes capable of generating high-quality text but is very expensive to train. The second line of work, in line with MUCoCO, aims to incorporate control in pre-trained models without

---

retraining them. For example, GEDI [28 trains smaller conditionl LMs and uses them as discriminators for guided generation. More recently, FUDGE 772 and DEXPERT 40 propose changes to left-to-right decoding in language models by modifying the vocabulary distribution at every step using attribute classifiers and ensemble of language models trained on attribute- speciffc corpora. Although lightweight, these approaches are, by design, prone to trade-off between preserving content and enforcing the attributes in the generated text. Also related to this work NEUROLOGIC DECODING [43] which uses approximate search in the discrete space (similar to beam search) to satisfy predicate logic constraints. Our work is most closely related to Plug and Play Language Models [8] which use gradients from the attribute models to update the prediction. They work by updating the model activations rather than token probabilities which limits their applicability to only unconditional language models. Furthermore, due to their autoregressive nature, these approaches do not guarantee sequence-leve control as they only look at the prefix generated up to certain step. These are also limited to categorical attributes and can not enforce real-valued controls like semantic similarity.
Gradient-descent based optimization generate text has been explored in prior work for improving
machine translation [9], paraphrasing [50] and generating adversarial examples [59]. These methods
however rely linear combinations of various objectives which as we discuss in $2 are not optimal for non-convex neural-networt based models. This phenomenon has also been studied in multi-task learning [36 37 57 where linear combination of multiple task losses is the most common approach and approaches for multi-objective gradient descent have been proposed. These approaches can also be explored for text generation in the future.
7 Conclusion
We present MUCoCO, a decoding algorithm for controlled generation from (conditional) language models that flexibly combines pretrained LMs with any differentiable constraints. With experiments on style transfer and controlled machine translation, and multiple combination of constraints, show the effectiveness of this approach. In addition to its potential applications in factual rewriting and debiasing text, this work holds promise in making language generation models personalizable and adaptive to different dialects or even individual speakers, since MUCoCO re-uses pre-trained LMs without adaptation and can incorporate constraints (e.g., dialect or user properties) trained on very little data. Future work will explore more sophisticated optimization techniques to improve the computational efficiency of our approach, and gradient-descen based methods for sampling [67 which will allow to sample from the language models with constraints.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grants No. IIS2125201 and II82040926, and by the Google faculty research award. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. We thank Sebastian Gehrmann for detailed, helpful feedback, Kayo Yin and Artidoro Pagnoni for evaluating our model outputs, Biswajit Paria for helpful discussions, and the anonymous reviewers for much appreciated feedhack.
References
[1] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, On the dangers of stochastic parrots: Can language models be too big? In Proc. FAccT, 2021.
[2] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
[3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Knueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are sho- learners. In Proc. NeurlPS, 2020.
10

---

[4] D. Cer, M. Diab, E. Agire,, I. Lopez- Gazzio, and L. Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proc. SemEval, 2017.
[5] A. Chan, Y.-.. Ong, B. Pung, A. Zhang, and J. Fu. Cocon: A self-supervised approach for controlled text generation. In Proc. ICI., 2021.
[6] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Proc. ICMI,, 2018.
[7] Y. Cheng, Z. Gan, Y. Zhang, O. Elachqar, D. Li, and Liu. Contextual text style transfer. In Findings of EMNLP, 2020.
[8] S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: simple approach to controlled text generation. In Proc. ICLR, 2020.
[9] G. Debreu. Valuation equilibrium and Pareto optimum. Proceedings of the National Academy of Sciences, 1954.
[10] J. chine learning algorithms tunable.
Degrave and I. Korshunova.
How we
can
make ma-
blog/ how-we- www.maaaeaaaaaa waw.a---aaccheeeeerinnngn algori wwwsstttaalleeeeeee 2020.
hstps://www.oe
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.
[12] Edunov, M. Ott, M. Auli, D. Grangier, and M. Ranzato. Classical structured prediction losses for sequence to sequence learning. In Pruc. NAACL, 2018.
[13] J. Ficler and Y. Goldberg. Controlling linguistic style aspects in neural language generation. In Proc. Workshop on Stylistic Variation, 2017.
[14] A. Field and Y. Tsvetkov. Unsupervised discovery of implicit gender bias. In Proc. EMNLP,
2020.
[15] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. PealToxicityPrompt::: Evaluating neural toxic degeneration in language models. In Findings of EMNLP, 2020.
[16] Gehman, S. Gururangan, M. Sap, Y. Choi, and
Smith, Realtoxicitypromps::: Evaluating neural toxic degeneration in language models. In Findings of EMNLP. 2020.
[17] J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher. Non-autoregressive neural machine translation. In Proc. ICLR, 2018.
[18] M. Guo, A. Haque, D.-A. Huang, S. Yeung, and Fei-Fei. Dynamic task prioritization for multitask learning. In Proc. ECCV, 2018.
[19] C. D. V. Hoang, G. Haffari, and T. Cohn. Towards decoding as continuous optimisation in neural machine translation. In Proc. EMNLP, 2017.
[20]
Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 1997.
[21] C. Hokamp and Q. Liu. Lexically constrained decoding for sequence generation using grid beam search, 2017.
[22] Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing. Toward controlled generation of text. Proc. ICML, 2017.
[23] X. Huang and M. Paul. Neural user factor adaptation for text classification: Learning to generalize across author demographics. In Proc. SEM 2019), 2019.
[24] M. Junczys-Dommun,, R. Grundkiewicz, T. Dwojak, H. Hoang, K. Heafield, T. Neckermann, Seide, U. Germann, A. Fikri Aji, N. Bogoychev, A. F, T. Martins, and A. Birch. Marian: Fast neural machine translation in C++. In Proc. ACL System Demonstrations: 2018.
[25] A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proc. CVPR, 2018
[26] N. S. Keskar, B. McCann, L. Varshney, C. Xiong, and R. Socher. CTRL- Conditional
Transformer Language Model for Controllable Generation. arXiv preprint arXiv: (ete:::09999599 2019.
[27] Kivinen and M. Warmuth. Exponentiated gradient versus gradient descent for linear predictors. Information and Computation, 1997.

---

[28] B. Krause, A. D. Gotmare, B. McCann, N. S. Keskar, S. Joty, R. Socher, and N. F. Rajani. GeDi: Generative discriminator guided sequence generation. 2020.
[29] K. Krishna, J. Wieting, and M. lyyer. Reformulating unsupervised style transfer paraphrase generation. In Proc. EMNLP, 2020.
[30] S. Kumar and Y. Tsvetkov. Von Mises-Fisher loss for training sequence to sequence models with continuous outputs. In Proc. ICLR, 2019.
[31] S. Kumar, S. Chakrabarti, and S. Roy. Earth mover's distance pooling over siamese LSTMs for automatic short answer grading. In Proc. IJCAI, 2017.
[32] M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger. From word embeddings to document distances. In Proc. JCML, 2015.
[33] G. Lample, S. Subramanian, E. Smith, L. Denoyer, M. Ranzato, and Y.-L. Boureau, Multiple- attribute text rewriting. In Proc. ICLR, 2019.
[34] J. Li, R. Jia, H. He, and P. Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proc. NAACL, 2018.
[35] X. L.. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proc.
ACL, 2021.
[36] X. Lin, H.-L. Zhen, Z. Li, Q--E Zhang, and Kwong, Pareto multi-task learning. In H. Wallach,
H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Proc. NeurIPS, 2019.
[37] X. Lin, Z. Yang, Q. Zhang, and S. Kwong. Controllable pareto multi-task learning, 2021.
[38] P. Lison and J. Tiedemann. OpenSubtitles206:: Extracting large parallel corpora from movie and TV subtitles. In Proc. LREC, 2016.
[39] A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. DExperts: Decoding-tine time controlled text generation with experts and anti-experts. In Proc. ACL, 2021.
[40] A. Liu, M. Sap, X. Lu, S. Swayamdipta,
Bhagavatula, N. A. Smith, and Choi. DExperts: Decoding-ting time controlled text generation with experts and anti-experts. In Proc. ACL, 2021.
[41] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and
V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019.
[42] L. Logeswaran, H. Lee, and S. Bengio. Content preserving text generation with attribute controls. In Proc. NeurIPS, 2018
[43] X. Lu, P. West, R. Zellers, Le Bras, C. Bhagavatula, and Y. Choi. NouroLogic decoding: (un)supervised neural text generation with predicate logic constraints. In Proc. NAACL, 2021.
[44] X. Ma. M. Sap, H. Rashkin, and Y. Choi. rowerTransformer: Unsupervised controllable
revision for biased language correction. In Proc. EMNLP, 2020.
[45] A. Pagnoni, V. Balachandran, and Y. Tsvetkov. Understanding factuality in abstractive summa- rization with FRANK: benchmark for factuality metrics. In Proc. NACCL, 2021.
[46] K. Papineni, S. Roukos, T. Ward, and W.-J Zhu. BLEU: method for automatic evaluation of machine translation. In Proc. ACI. 2002.
[47] B. Paria, C.-K. Yeh, I. F. H. Yen, Xu, Ravikumar, and B. Póczos. Minimizing FLOPS to learn efficient sparse representations. In Proc. ICLR, 2020.
[48] J. Platt and A. Barr. Constrained differential optimization. In Proc. NeurIPS. American Institute of Physies, 1988.
[49] S. Prabhumoye, Y. Tsvetkov, R. Salakhutdinov, and A. W. Black. Style transfer through back-translation. In Proc. ACL, 2018
[50] L. Qin, V. Shwartz, West, C. Bhagavatula, J. D. Hwang, R. Le Bras, A. Bosselut, and Y. Choi. Back to the Future: Unsupervised backprop-based decoding for counterfactual and abductive commonsense reasoning. In Proc. EMNLP, 2020.
[51] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models unsupervised multitask learners. 2019.
[52] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
12

---

[53] S. Rao and J. Tetreault. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metries for formality style transfer. In Proc. NAACL, 2018.
[54] S. Reddy and K. Knight. Obfuscating gender in social media writing. In Proc. Workshop NLP and Computational Social Science, 2016.
[55] N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using siamese bert-
networks. In Proc. EMNLP, 2019.
[56] N. Reimers and I. Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proc. EMNLP, 2020.
[57] O. Sener and V. Koltun. Multi-task learning as multi-objectiive optimization. In Proc. NeurIPS, 2018.
[58] T. Shen, T. Lei, R. Barzilay, and T. Jaakkola. Style transfer from non-parallel text by cross- alignment. In Proc. NeurIPS, 2017.
[59] C. S.nng, A. Rush, and V. Shmatikov. Adversarial semantic collisions. In Proc. EMNLP, 2020.
[60]
Stahlberg and B. Byrne. On NMT search errors and model errors: Cat got your tongue? In Proc. EMNLP, 2019.
[61] A. Sudhakar, B. Upadhyay, and A. Maheswaran. "Transforming" delete, retrieve, generate approach for controlled text style transfer. In Proc. EMNLP, 2019.
[62] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention all you need. In Proc. NeurIPS, 2017.
[63] E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proc. EMNLP, 2019.
[64] E. Wallace, M. Stern, and D. Song. Imitation attacks and defenses for black-box machine translation systems. In Proc. EMNLP, 2020.
[65] A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. arXiv preprint 1805. /2471, 2018.
[66] Welleck, Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. In Proc. ICLR, 2020.
[67] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proc. ICML. 2011.
[68] J. Wieting, T. Berg- Korn-KKirkttrikk, K. Gimpel, and G. Neubig. Beyond BLEU: Training neural machine translation with semantic similarity. In Proce. ACL, 2019.
[69] A. Williams, N. Nangia, and S. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. NAACL, 2018.
[70] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-ar natural language processing. In Proc. EMNLP System Demonstrations, 2020.
[71] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Fukasz Kaiser, S. Gouws, Y. Kato, Kudo, H. Kazawa, K. Stevens, G. Kurian, Patil, W. Wang, C. Young, J. Smith, Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. Google's neural machine translation system: Bridging the gap between human and machine translation, 2016.
[72] K. Yang and D. Klein. FUDGE: Controlled text generation with future discriminators. In Proc. NAACI, 2021.
[73] L. Yu, W. Zhang, Wang, and Y. Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In Proc. AAAI, 2017.
[74] R. Zellers, A. Holtzman, H. Rashkin, Bisk, A. Farhadi, Roesner, and Choi. Defending against neural fake news. In Proc. NeurlPS, 2019.
[75] T. ZZang*, V. Kishore* F. Wu K. Q. Weinberger, and Y. Artzi. BERTScore: Evaluating text generation with bert. In Proc. ICLR, 2020.
[76] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and
G. Irving. Fine-tuning language models from human preferences, 2020.
13


Reviews:
[{"review_id": "Lp3ITWiQAXm", "reviewer": "Reviewer_deJ9", "summary": "In this paper, the authors propose a controllable decoding mechanism for language generation models (e.g., GPT2) using constrained optimization. The proposed method explicitly formulate the sentence decoding as a constraint optimization problem where the free parameters are the distribution over the vocabulary at each step and two types of constraints are defined one over the generate text itself (f(Y)) and one over the similarity between the input and the outputs (g(x,y)). Please refer to Eq. 1 where the problem is clearly formalized. Since directly solving the problem in Eq 1. is not computationally feasible the author reformulates the problem using the Lagrangian formulation for container optimization problems (Eq. 3). However, this min-max formulation introduces several problems (e.g., the tradeoff between different objectives, hard to find opt for non-convex functions). To cope with these issues, the authors proposed a gradient descent/ascent solution (Section 2.1), a soft relaxation for each token using the probability simplex (exponential gradient descent + normalization) (Section 2.2), and an annealing threshold (Section 2.3). \n\nThe authors benchmarked the proposed decoding schema in both style transfer and style-controlled machine translation. In the style transfer, they follow the methodology in Krishna et al. 2020 and they study two contains such as formality and the semantic similarity between the input and the output. Similarly in the style-controlled machine translation, they study both the formality style and the cross-lingual similarity between input-output. In both settings, the author compared the proposed method with other decoding techniques (greedy and beam-search) and other controllable language generation techniques (e.g., FUDGE) using both automatic and human evaluation.  Finally, the authors show some preliminary results in controlling multiple attributes (Age, style pos/neg), and discuss the speed and memory requirements of the model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "***Originality***\n\nTo the best of my knowledge, the proposed method is novel in controllable language generation. Moreover, constrained optimization is very interesting in this setting and this is very different from existing approaches. However, I would suggest including some relevant work as [1,2]\n\n***Quality***\n\nThe paper is technically sound, but I would have expected more experiments and comparisons with existing work. Indeed, this method is general and it could be used with any discriminator (e.g., positive/negative, topics, toxicity etc.). Several existing works (e.g. GEDI, PPLM etc.) run these controllable language generation experiments. Adding these experiments would have made the paper much stronger and more general. The results on formalization are not very exciting, although the result is extremely good. Moreover, I am a bit suspicious when checking Appendix Table 5,  where the authors show the multiple constraints example, in that example, I would be curious to see what happen when rewriting a review with an original score of 2 to a review with a score of 5. Is this possible? \n\nMoreover, it is also not clear the human evaluation for the style-controlled machine translation (Line 295). Why not running the same human eval as in the style transfer experiments? and the results in percentage are really confusing. Could you please clarify the human evaluation. \n\n***Clarity***\nThe paper is extremely well written,  the notation is clear and easy to follow also with a minimal background in optimization. \n\n\n***Significance***:\nYes, this proposed method is extremely important for any general NLG model. In general, enforcing constraints in text generation is extremely hard and solving this problem could be crucial for safely deploying large language models. \n\n\n[1] NEUROLOGIC DECODING: (Un)supervised Neural Text Generation with Predicate Logic Constraints (Lu et. al. 2020) \n[2] Gradient-guided Unsupervised Lexically Constrained Text Generation (Sha et.al. 2020)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author clearly states the limitation, i.e., a very slow inference time, and possible workaround in future work. Moreover, they also address possible negative societal impacts in the ethical considerations section. ", "review_text": "", "extra_scores": {"ethical_concerns": "The authors addressed this in the ethical considerations section. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "IYk7pSs1eD", "reviewer": "Reviewer_LkvM", "summary": "This paper presents a decoding algorithm for controlled text generation. The problem is formulated as finding the most likely output $y$ under given constraints. The authors adopt the method of Lagrange multipliers, and introduce dampening in gradient descent to reduce oscillations. To deal with discrete $y$, the authors choose to optimize over probability simplex $\\tilde y$ using exponentiated gradient descent. The authors also report that converting $\\tilde y$ to a one-hot vector via a straight through estimator helps to converge faster.\n\nExperiments include increasing formality in style transfer and machine translation. Compared with greedy decoding/beam search, the proposed method with formality and semantic similarity constraints achieves better content preservation/BLEU score and better transfer accuracy. It also compares favorably to the recent FUDGE method for controlled generation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Constrained text generation is an important problem that has received increasing attention. This paper proposes a multi-objective optimization approach for constrained decoding, and motivates the choice of each technique used.\n\nIs the formality classifier used in optimization also used for evaluation? If so, as you set the probability threshold to 0.5, if the constrained optimization was successful, shouldn't you achieve formality 100%? Why in Table 2 it's only 30%? Or the optimization failed and ended up with Lagrangian multipliers of extreme values?\n\nIn the method section, the authors discussed the shortcomings of using linear combinations of multiple objectives and advocated their Lagrangian approach. Nevertheless, I still would like to see how linear combinations perform in the experiments. Also, if you apply your continuous optimization in the unconstrained case (so maximizing likelihood is the only objective), how would it compare to beam search? I know it's slower than beam search, but will its BLEU be better or worse? I think this result will help to better understand the comparison between continuous optimization and combinatorial search.\n\nTypo: line 127, \"$(\\mu_j-g_j(x,y))$\" should be $(\\xi_j-g_j(x,y))$", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are discussed in Sec 4.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "B4T-WCOwqE-", "reviewer": "Reviewer_CUjF", "summary": "This work tries to solve a general type of text generation tasks, generating text with constrains. The author proposed a a new algorithm MUCOCO to formulate the decoding process as a constrained optimization problem. The author then use continuous relaxation, Lagrangian multipliers and exponential gradient descend techniques to solve the problem. The author validated the algorithm with style transfer and style-controlled machine translation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper formulates the constrained text generation problem into a discrete constrained optimization task. This idea is quite novel and interesting. The proposed setup of Lagrangian multiplier, straight through and exponential gradient descent are natural tools to use for this setup.  \n\nI do not get the reason why the author emphasis the proposed framework is for pretrained model. From the method itself, I do not see a limitation on this. \n\nAs the proposed algorithm is to improve over auto-regressive model which is hard to follow the constrains. Then the other type of framework,  non-autoregressive text generation should be interesting to be considered in this work, at least as a baseline for MT task.\n\nI do not see details in inference stage, will the proposed algorithm effect decoding stage?\n\nThe method's effectiveness seems strongly relay on the auxiliary optimization objective f and g. This may leads to a limitation to the proposed method, how to define a differentiable objective can be a hard task. Also in the experiments, seems the function threshold needs carefully selection, which may make the proposed algorithm practically hard to use. An empirical study on these should be shown in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "AmLRA-FqcJJ", "reviewer": "Reviewer_DZSf", "summary": "This paper proposes a flexible controllable decoding algorithm called MUCOCO, which incorporates multiple control attributes as differentiable constraints to the optimization. The authors relax the discrete optimization problem to a continuous one, and use continuous optimization techniques such as Lagrangian multipliers and gradient-descent based techniques to generate texts towards the desired attribute. Experiments on three conditional text generation tasks including text style transfer, machine translation and paraphrasing show the superior performance of the proposed model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The formulation of constrained decoding is intuitive and interesting. It provides a new perspective to solve the text generation problem under multiple constraints.\n2) Experiments on three different text generation tasks show the generalization ability of the proposed method.\n\nWeaknesses:\n1) My main concern is about the focus of this paper. I understand that the formulation itself is interesting. But the following techniques such as continuous relaxation and gradient-based optimization which are the main content of the method are all common techniques in the machine learning community, which don’t provide new insights. So, I feel that the focus should be addressed clearly.\n2) The authors choose FUDGE as the main baseline, but this model can only support a single constraint. The authors should consider the baselines which can handle text generation with multiple control attributes such as CoCon [1].\n3) In my view, one of the major problems in text generation with multiple control attributes is the compatibility of multiple constraints. I wonder how the proposed model can explicitly deal with this problem.\n\n[1] COCON: A SELF-SUPERVISED APPROACH FOR CONTROLLED TEXT GENERATION. ICLR 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately address the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes MUCOCO, a constrained text generation method. The constrained text generation is formulated as generating text with multiple controlling attributes. MUCOCO turns a discrete constrained decoding into differentiable optimization using techniques of  continuous relaxation, Lagrangian multipliers and exponential gradient descend. Experiments on three conditional text generation tasks including text style transfer, machine translation and paraphrasing show the superior performance of the proposed MUCOCO.

The author may adjust the main focus a bit accordingly. Additional discussion and/or comparison with other highly related methods (methods that can also handle constrained sentence generation with multiple constraints) could be added as pointed out by reviewers. The authors may explain the inference clearly. Additional discussion on limitations of MUCOCO could be added (e.g. conditions for auxiliary objectives and compatibility of multiple constraints) . 
Decision: Accept (Poster)