Why is constrained neural language generation particularly challenging?
Authors: Cristina Garbacea
Qiaozhu Mei
Venue: Accepted by TMLR ()

Abstract: Recent advances in deep neural language models combined with the capacity of large scale datasets have accelerated the development of natural language generation systems that produce fluent and coherent texts (to various degrees of success) in a multitude of tasks and application contexts. However, controlling the output of these models for desired user and task needs is still an open challenge. This is crucial not only to customizing the content and style of the generated language, but also to their safe and reliable deployment in the real world. We present an extensive survey on the emerging topic of constrained neural language generation in which we formally define and categorize the problems of natural language generation by distinguishing between conditions and constraints (the latter being testable conditions on the output text instead of the input), present constrained text generation tasks, and review existing methods and evaluation metrics for constrained text generation. Our aim is to highlight recent progress and trends in this emerging field, informing on the most promising directions and limitations towards advancing the state-of-the-art of constrained neural language generation research.

Published in Transactions on Machine Learning Research (02/2025)
Why is constrained neural language generation particularly challenging?
Cristina Gârbacea Data Science Institute University of Chicago
gartacea@uchiongo.ce
Qiaozhu Mei
qmeiCrumich. edu
School Information, Deportment of EECS University of Michigan
Reviewed on OpenReview: https: openrevvee............mum net/ net/forum? id= jkbykky
Abstract
Recent advances in deep neural language models combined with the capacity of large scale datasets have accelerated the development of natural language generation systems that produce fluent and coherent lexts (lo various degrees of success) in a multitude of lasks and application contexts. However, controlling the output of these models for specific user and task needs is still an open challenge. This crucial not only customizing the content and style of the generated language, but also to their safe and reliable deployment in the real world. We present extensive survey the emerging topic of constrained neural language generation in which we formally define and categorize the problems of natural language generation by distinguishing between conditions and construints (the latter being testable conditions on the output text instead of the input), present. constrained lext generation lasks, and review existing methods and evaluation metrics for constrained text generation. Our aim is to highlight recent progress and trends in this emerging field, informing on the most promising directions and limitations towards advancing the state-of sfate-of-the-art of constrained neural language generation research.
1 Introduction
Recent advances the field of natural language generation (NLG) Gatt Krahmer 2018 have resulted in models able produce realistic, coherent, and fluent texts multitude of natural language processing tasks. Powerful large scale language models can be readily used perform unconditional language genera- tion, however these models provide little control over attributes of the generated texts. Unlike conventional methods which were able to provide fine-grained control over many aspects of the system output including incorporating domain-speciff dictionaries, terminology or certain words in the generated output, neural end-to-end approaches remove many of these knobs and switches (Post. & Vilar 2018). However, impos- ing constraints on the output generated by these models crucial for achieving useful and safe language generation in multitude of real world application scenarios. For example, can help avoid generic and meaningless responses in dialogue systems (See et al.) 2019), personalize dialogue agents based on user fea- tures that lead to more engaging and meaningful conversations (Zhang et al. 2018a), ensure non-offensive sentence completion and friendly communication (Liu et al. 2019b), intervene the system output in in- teractive secnarios where domain specific terminology must be included in the generated texts (Crego et al. 2016), or aid in crealive applications such as poetry generation or assisted story writing (Peng et al. 2018). Moreover, controlling a generic pretrained language model in order to satisfy certain desiderata helps avoid generating toxic content, prevents demographic biases, can steer generations towards a desired topic or style [Khalifa et al. (222)), and helps communicate intentions in suitable manners for different situations, target audiences and environments (Lample et al. 2018 Li et. al. 2018 Incorporating prior knowledge and target

---

Published in Transactions on Machine Learning Research (02/2025)
side constraints in text generative models has numerous applications in many natural language processing areas, including dialogue systems, machine translation, question answering, text summarization, text simpli- fication, image captioning, etc. Unquestionably, constrained text generation is important in many real-world applications, but compared to other instances of natural language generation, constrained text generation using neural networks remains an open challenge.
We identify the following reasons why constrained neural text generation represents a much harder problem compared to other instances of neural text generation: i) lack of model expressiveness: current models are not expressive enough to incorporate arbitrary constraints, defined as testable conditions on the output text, into the objective function at training time; di) lack of suitable evaluation metrics: while one can verify whether an output satisfies a constraint or not, it is usually hard to measure to what exlent an output satisfies a constraint, and it is even harder to jointly evaluate this with other properties of the generated text (such as relevance or coherence); ti difficulty in constrained oplimization: even if constraints can be expressod and added to the objective function, they are usually non-differentiable, especially at the token level, which is bad since most methods model and generate text as sequence of lokens; in) lack of constrained texl generation datasets that are diverse and representative enough of the variety of practical constraints.
Commonly used sequential text generation methods and architectures assume a rigid modeling of the output sequence based on word ordering, with tokens generated progressively one at a time in a standard left-to- right manner (Chan et al. 2019). Such autoregressive models cannot easily express constraints at arbitrary positions in the generated sequence or satisfy constraints involving multiple input objects. addition to these issues, it is generally more challenging to incorporate multiple and heterogeneous constraints, which conform to given rules, topics, sentiments, lexical constraints, or pre-defined stylistic and content attributes.
Our work focuses on the emerging problem of neural natural language generation with coists We first define the problem and differentiate between the ambiguous use of conditions and constraints in natural lan- guage generation, including examples that represent instantiations of the constrained neural text generation problem. We then survey approaches, learning methodologies and model architectures employed for gener- ating lexts with desirable attributes, and corresponding evaluation metrics. We conclude with open research problems and limitations of current models. Our work aims to draw clear boundaries between the confusing terminology used in the neural language generation literature, highlight the main approaches and discuss how they suffer from the general challenges of rainer text generation, and serve as an informative guide for solving these general challenges and advancing meaningful, useful, and safe constrained NLG research.
2 Problem Definitions
We formally define the problem of natural language generation, accounting for context, conditions, and con- straints placed on text generative models. First, we aim to articulate the key difference between condition and constraint since the distinction between these concepts is rather blurred in the natural language pro- cessing literature. Given a lext generation task defined as x(X) + X', we define condition as a testable statement of the imput X, and constroint as a testable statement of the output X'.
Accounting for the distinction above, we divide the text generation problem into thrce categories: i) generic or free-read Lead generalion which we present in Section 2.1 ii) conditional text generation which we introduce in Section 2.2 and tüi) construined tezt generation which we outline in Section 2.3 The focus of our work is the particular problem of constrained text-to-text generation, lcaving aside text generation tasks from other types of inputs such as data-to-text generation or ln-ge-lo-text generation which are conditional in nature according to our definitions.
2.1 Generic/Free- Text
Text Generation
The problem of generic text generation considers the intrinsic history of words generated until the current timestep in the sequence as context, and does not place any external user-defined conditions or constraints the model output. We formally define it in what follows.
2

---

Published in Transactions on Machine Learning Research (02/2025)
Given discrete sequence of text tokens x (2.,,2..... Fx) as imput where each 2, is drawn from fixed set of symbols, generic text generation aims to learn the unconditional probability distribution p(z) of sequence æ. This distribution can be auto-regressively factorized using the chain rule of probability (Bengio et al. 2003) into a product of conditional probabilities p(æ) = Ih= p(a)( to perform density estimation and generation of text data. When p(æ) is modeled by a neural network with parameters 0, the neural network is trained to minimize the negative log-lik logllikelihood L(D) = - k= Σxn po (= over a collection of samples D {a² :..... ,21)), where N₂ is the length of m To generate new samples, each token x, is iteratively sampled from pz( and is fed back into the model as the input for the next timestep.
Large scale models for generic text generation show promising abilities to imitate the distribution of natural language and generate long-term realistic and coherent texts, however such free-text generation models place a lot of burden on the generative model to capture complex semantic and structural features underlying the data distribution; this can often result in repelitive, cont radittorr, and largely randomized generated lexts (Holtzman et al., 2020). Notably, the content generated by free-text generative models cannot be controlled with respect to particular attributes and modes of the data distribution. This inability to control which regions of the data. distribution are generated is particularly problematic considering there is significant toxicity, hate, bias, and negativity present in the large-scale web crawled datasets text generation models are commonly trained on. Imposing conditions or constraints on the generation process results in safer and more useful generated texts for downstream application tasks (Krause et al. 2021
2.2 Conditional Text Generation
Conditional text generation manipulates attributes of the generated content depending on specific contexts or user needs, and it allows the data generation process to focus on specific modes of the data. Conditioning the generalive model on additional information makes it possible to generate lexts which satisfy given impul conditions and meet desired attributes. In the literature conditional text generation is sometimes referred to as context-dependent text generation. While the word context may carry different semantics for different readers, in this survey we consider as context only attributes which are inherently external to the model itself; model intrinsic attributes such as for example, the history of past generated words, is already included in the formulation of generic text generation. For example, context attributes used for conditioning generated texts are the source sentence in machine translation, the conversational history in dialogue systems, the input document. in text summarization and lext. simplification, the inpul question in question answering systems, the prompt given to a large language model, or contextual information such as product, time, and location in review generation. Essentially, conditional text generation a form of eqquence-to-sequenee generation, text sequence and the goal is to generate another output sequence.
given the input condition
Conditional text generation models add contextual variable attribute code e to the probabilistic model p(æ) transforming it into conditional probability model p(æ|c), which can be auto-regressively decomposed
using the chain rule of probability p(z|c)- = II= p|<//=<< c). When p(æ|c) is modeled by a neural network with parameters& 0, the model minimizes the negative log-likelihood loss function accounting for the attribute code C(D)- - I Ee (og l c*). Besides generation, conditional models can also be used as generative classifiers to compute p(cltes) by applying Bayes rule.
No
2.3 Constrained Text Generation
The problem of constrained text generation is focusing on generating coherent and logical texts that do (not) cover lexical concepts (for eg., pre-defined nouns, verbs, entities, phrases sentence fragments) desired to be (not) present in the output, as well as generate outputs that abide to specific format, semantic, syntactic or utility rules to reflect the particular interests of the system user. Constraints impose restrictions on the generalive model that must be satisfied by any solution to the optimization problem and their fulfillment can be tested accordingly. In the literature the distinction between conditional, controlled, and constrained text generation is not clearly defined, and these terms are often used interchangeably. In fact, the first work that proposed generating constrained text is actually referring to the task as "controlled" generation (Hu et al. 2017). In what follows we formally define the problem of constrained text generation.
3

---

Published in Transactions on Machine Learning Research (02/2025)
Let us consider we are (optionally) given an unordered or ordered set of n concepts c fa, (c,,,,,.... c2,...,, Cn. E C, where C denotes the space of all concepts, and G E C' is a concept belonging to the concept vocabulary. In addition, let us assume we are also (optionally) given a set of m rules = = 22,....... 2m E Z, with 2; E R, where R denotes the space of all rules, and each 2 is a text generation constraint expressed in logical form. We formulate constrained text generation as learning the structured predictive function f CUU 2 + X, where C Z + Φ which maps a set of concepts and/or constraint rules to a generated sentence. Therefore, constrained text generation methods impose constraints on the generated sentences and produce output in the form of grammatical sentence y y which contains all concepts present in c and all constraint rules specified in 2. The probability p(ylf) can still be modeled autoregressively p(ylf) = I p(u: le f); when plyl is modeled by a neural network with parameters 0, the negative log likelihood function can be minimized while leveraging f for constraint satisfaction C(D) = - E log log (o l)..
The matching function f manipulates the probability distribution and indicates to which extent the con- straints are satisfied. In the literature, constr text generation methods can be either i) Soft-construined (priming), when the matching function f is a soft measure of semantic similarity and only requires the generated sentences to be semantically related to the given constraints, or ti) Hand-construined, when the matching function f is a binary indicator which rules out the possibility of generating infeasible sentences that do not meet the given constraints. Hard-constrained text generation notably a more challenging task compared to coft-constrained text generation, and it requires specialized approaches and architectures en- sure the constraints in the output sentence. In contrast, soft-constrained text generation models are usually easier to design, e.g., with the use of existing copy and attention mechanisms for soft enforcing constraints and annotated keyword-text pairs; nevertheless, even soft constraints are likely to be lost during generation, especially if multiple weakly correlated (lexical) constraints must be included (Zhang el al. 2020b).
Compared to generic text generation which assumes no conditions on input or output other than existing context, and comparod to conditional text generation which places conditions on the input which can be considered at training time, constrained text generation places conditions on the output which is a consider- ably more difficult and challenging problem to solve. Unlike input conditions, output conditions cannot be considered at training time and their satisfaction is assessed after training has completed by sampling and inspecting the generated outputs. In addition, standard sequence generation architectures are not designed to easily accommodate or incorporate output constraints. Given the model structure itself cannot express output conditions, it becomes challenging to evaluate the extent to which constraints are satisfied by a model, objectively compare and contrast the performance of different models, and mcasure overall success to inform on progress in constrained natural language generation. Due to these limitations, current methods proposed to address constrained text generation are neither satisfactory nor sufficient. The main machine learning challenge is that it is hard to evaluate the objective function for constrained text generation, and very few studies have approached the problem from the prism of editing the objective function to incorporate con- straints at training time. Even if constraints were to be added to the objective function itself, constrained optimization would be another challenge. In general, reinforcement learning approaches are used in the context. of lext generation to optimize non-differentiable reward functions computed at the loken level, for eg., BLEU in machine translation or ROUGE in text summarization. However, optimizing such automatic mcasures that focus on local n-gram patterns often results in deteriorated textual outputs despite increased automatic scores (Bosselut et al. 2018 Pasunruu et al. 2020). Moreover, applying reinforcement learning to text generation at the word level leads to difficulty in proper temporal credit assignment. for long-term textual rewards (Saleh et al.] 2020). Given that the enviromment provides only delayed rewards as the agent executes a sequence of actions, it is often impossible to know whether the agent succeeds in achieving a task until the end of the episode, at which point the agent needs to determine which of the actions in the sequence are to be credited with producing the resulting reward (Gao et al.] (019a)). Adding constraints on top of existing reinforcement learning issues would be detrimental to the learning process, if not make learning close to impossible: the objective function would be even harder to optimize, rewards would be delayed, sparse and non-informative. Despite these open problems and limitations, we argue neural constrained text generation is an important research area which deserves a lot more attention.
Constrained text generation is useful in many scenarios, such as incorporating in-domain terminology in machine translation (Post& & Vilar 2018), improving semantic corectness (Balakrishnan et al. 2019), avoiding
4

---

Published in Transactions on Machine Learning Research (02/2025)
generic and meaningless responses in dialogue systems using grounding facts (Mou et al. 2016 paraphrase generation in monolingual text. rewriting (Hu et al. 2019: Kajiwara 2019), incorporating ground-truth text fragments (such as semantic attributes, object annotations) in image caption generation (Anderson et al, 2017), creating a story (Fan et al. 2018b) or poem Ghazvininejad et al.) 2017) using a pre-defined set of keywords, or re-writing a user search query as a fluent sentence. Typical altributes used to generate constrained natural language are the tense and the length of the summaries text summarization (Fan et al., 2018a), the sentiment of the generated content in review generation (Mueller et al., 2017). language complecity in text simplification or the style in text style transfer applications. In addition, constrained text generation is used to overcome limitations of neural text generation models for dialogue such as genericness
and repetitiveness of responses (See et al. 2019 Serban et al. 2016).
Nevertheless, generating text under specific lexical constraints is challenging. Common models and archi- tectures employed for natural language generation are autoregressive in nature, generating tokens one by one in soquential manner from left to right; by design, these models lack fine control over the generated sequence and cannot easily support constraints arbitrary positions in the output constraints involving multiple imput objects (Zhang et al.] 2020b; Hsieh et al.) 2021). While for humans it is straightforward to generate sentences that cover a given set of concepts or abide to pre-defined rules by making use of their commonsense reasoning ability, generative commonsense reasoning with constrained text generation task is more challenging for machine learning models (Lin et al. 2019).
3 NLG Constraints
Natural language generation models place restrictions on the generated output to produce texts that reflect cerlain user preferences. In Table 1 we present NLG tasks distinguishing between conditions and constraints. We broadly group existing constraints into the following categories:
Lexical constraints Lexical constraints allow to include specific keywords, phrases or entities at arbitrary positions in the output, and can be specified as a word (a single token) or phrasal constraint. (a multi-word phrase). They are useful in tasks such as dialogue ppettry generation, machine translation, story telling, etc.
Format constraints Format constraints such as number of sentences, length of sentences, order of words, number of syllables, etc. serve to denote preferences on the form and appearance of the generated output. Format constraints are particularly useful in tasks such as poetry generation generated poem, for example quatrain or regulated verse, length of the poem, rhyme and rhythm. In text summarization tezt simplification, length constraints define the length of the generated output to be strictly less than the length of the input document, while in dialogue generation they help define the level of verbosity of the dialogue agent.
specify the form of the
Semantic constraints Semantic constraints are used to define the topic and sentiment of the generated content, or control fine-grained aspects such as removing toxicity. Topic constraints are particularly useful in dialogue generation, where the goal is to generate on-topic responses that are safe, non non-harmful, unbi- ased, relevant. to the dialogue context and particular user needs; in story telling or poetry generation, topic constraints help define the theme. Generating language that conveys particular positive, neutral or negative sentiment aims to endow artificial agents with human-like traits such as compassion, empathy, and enables agents to react with appropriate emotion in diverse social situations; constraining on specific sentiment is important in many tasks such as dialogue generation, review generation, story telling, poetry generation or text style transfer. Furthermore, increasing politeness of a dialogue system or reducing toxicity of generated language are important aspects with respect to human-centered metrics of conversation quality.
Syntactic constraints Syntactically constrained text generation produces sentences with desired synlax by incorporating syntactic templates and rules in the training of the text generative model. Syntactic constraints are useful in paraphrasc generation, where given a sentence and target syntactic form (c.g., a constituency parse), a system must produce a paraphrase of the sentence whose syntax conforms to the target ([yyer et al. 2018). Generating texts that convey the same meaning but with different expressions has
5

---

Published in Transactions on Machine Learning Research (02/2025)
Table Overview of constrained NLG tasks, differentiating between conditions and constraints.
Task
Condition
Constraint
Lerioal
Format
Semonfic
Syntactic
Utility
Machine Translation
input
words
topic
paraphrase
target language
phrases
sentiment
politeness
entities
gender protoums
factuality/fiithl /aithfulles
helpfulness, harmlessness
Dialogne Generation
past utterance(s)
words phrases entities
length verbosity
topic
paraphrase gender protoums
politeness
sentiment toxicity
personality traits
factuality/ factuality/faitbhuunes faithfullssssssss
helpfulness, harmlessuees
Text Summarisation
imput (ocument(ss)
words phrases entities
length
topic
paraphrase
factuality faitalittyyathfuuleeesss helpfulness, harmlessness
Text Simplification
Input text
words phrases entities
length
topic
paraphrase
simpler vocabulary
readability
factuality/laihhffi huithlulsss helpfulness, harulessuees
Text Style Transfer
source text
words phrases entities
length
topic
paraphrase
style factuality /fathfuulttss
sentiment
tense
gender pronoums
Question Answering
imput question
words phrases entities
length
topic
paraphrase
factuality ffaithfulnssss
tense
politeness helpfulness, harmlessness
gender pronoums
Narrative Generation/
words phrases entities entities
longth
topic
paraphrase
readability
Story telling
sentiment
tense
fact uality /lathhfuleess helpfulness, harmlessness style
gender pronouns
Poetry Generation
words
length rhyme rhythm
topic sentiment
paraphrase
readability factuality Caatulltyyfaiithuun fiithfulees style
phrases entities
tense
gender pronouns
numerous applications in many natural language generation tasks, including monolingual transduction tasks such as tezt simplification, tezt compression, or text style transfer, as well as in tasks like text summarization. machine translation or question answering where alternative ways of expressing the same information capture the inherent language variations.
Utility constraints Utility constraints capture holistic properties of the generated output, for example, stylistic, readability, faithfulness, helpfulness, harmlessness or politeness aspects. Preserving the information content texts while manipulating attributes such as style, readability level, personality traits of the user or specific gender pronouns allows customize generated texts diffcrent audienees and make them relevant in wide variety of end-user applications. Stylistic constraints are inmedialely relevant to the task of text style transfer, with applicability in many tasks, including dialogue generation, machine translation, tert simplification, story telling, poetry generation, review generation.
Constraining text generation on altributes such as readability and level of text complexity serves to adapt the generated output to users of different age, backgrounds and educational levels. Reducing complexity of texts while preserving the information content is the main goal of text simplification; in addition, in tasks such as dialogue generation, text summarization, story telling, poetry generation, question answering is important to customize texts for various literacy levels.

---

Published in Transactions on Machine Learning Research (02/2025)
In many languages the degree of politeness is an important aspect of inter-personal communication, and honorifies are used to express courtesy, social distance, or the relative social status between the speaker and their addressee(s) (Semnrich et al.) 2016). Politeness constraints on the output are used in machine translation, dialogue generation, story telling, and text style transfer.
Faithfulness constraints enforce similarity between generated text sequence and its corresponding input, requiring models to generate texts that are faithful, factual and preserve the original information content. Such constraints are important in many tasks, including tert summarization, machine translation, tert sim- plification or dialogue generation, where models are vulnerable to producing hallucinated content.
Language constraints are useful when translating texts between different languages such as in machine Iranslation, or from complex language into simple language such as in leat simplification.
4 Constrained Natural Language Tasks
In what follows we briefly describe major NL. lasks and differentiate between the roles of conditions and constraints these tasks.
Machine Translation Machine translation is focusing the automatic translation of textual content from one language into another language, and is typical example of both conditional and construined text generation, as it conditions on the imput text in the source language and constraints the model to generate fluent and faithful output in the larget language. Additionally, constraints can be placed on the degree of formality and politeness, the use of gender-specific pronouns, the inclusion in the target sentence of named entities or specific concepts from the source sentence.
Dialogue Systems A dialogue system, also known a conversational agent, is computer system de- signed to converse with humans using natural language. Dialogue generation is an instance of conditional text generation where the system response is conditioned on the previous user utterance and frequently on the overall conversational context (e.g, a prompt given to Large Language Model based Chatbots). Dialogue generation can also be an instance of constrained text generation - it is desirable that generated dialogues incorporate explicit personality traits (Zheng et al. 2019), control the sentiment (Kong et al. 2019), topic, degree of formality and politeness of the generated response to resemble human-to-human conversations. In addition, dialogue responses may need to incorporate text excerpts from past dialogue history or entities such as locations, persons, institutions, etc. From an application point of view, dialogue systems can be calegorized inlo: i) lask-oriented dialogue agends, designed to help users complete particular lask, or ii) non-task oriented dialogue agents (chat-bots) designed to carry entertaining conversations with their users on a wide range of open domains. A common problem in dialogue gencration systems is that they tend to gen- erate safe, universally relevant respouses that carry little meaning (Serban el al.) 2016 Li et al. 2016a Mou et al.] 2016). Moreover, they can fail to take turns asking questions and balance specificity with genericness of the output (See et al. 2019).
Text. Summarization Text summarization facilitates a quick grasp of the essence of document and produces condensed version of its content, by copy-pasting the relevant portions from the input as in extractive summarization (Nallapati et al. 2017), or by generating novel content as in abstractive summa- rization (Rush et al. 2015 Nallapati et al. 2016 See et al. 2017), or via hybrid approaches (Liu et al., 2018 that combine both tochniques. Text summarization is a conditional terf generation task where the condi- tion is represented by the given document (s); additional conditions are used in remainder sunmarization to flexibly define which parts of the document(s) are of interest, for eg., remaining paragraphs the user has not read yet, or in source-specific summarization to condition summaries on the specific input source and style of writing, for eg., newspapers, books or news articles. Text summarization is also a constrained tert generation task considering that the length of the summary is fixed, pre-determined, and strictly less than the original document. The goal of text summarization is to allow the user to digest information at different levels of granularity and detail according to personal needs, interests and time budget. Morcover, constraints can be placed on specific concepts to include in the summary, such as named entities, or on explicitly picking sentences from the original document as in extractive summarization.
7

---

Published in Transactions on Machine Learning Research (02/2025)
Text Simplification Text simplification is designed to reduce the text complexity, while preserving its original meaning. In the literature, simplification has been addressed at multiple levels: i) lerical simplifi- cation focused on replacing complex words or phrases with simpler alternatives; ii) syntactic simplification alters the syntactic structure of the sentence; iii) semantic simplification paraphrases portions of the text into simpler and clearer variants. End-to-end models al attempl to combine all these steps. Text simplification is both conditional and constrained text generation; we are conditioning on the input complex text to gener- ate a simpler version, accounting for constraints such as higher readability, simpler vocabulary, and shorter sentence length than the complex input.
Text Style Transfer Style transfer has its origins in computer vision applications for image-to-image translation and more recently has been used in natural language processing applications for machine transla- tion, sentiment modification to change the sentiment of sentence from positive to negative and vice versa, word substitution decipherment and word order recovery (Hu et al. 2017). Text style transfer is designed to preserve the information content a source sentence while altering the way it is delivered to meet desired presentation constraints. Textual content is disentangled from the style in which is presented, and manip- ulating stylistic attributes can be done without parallel aligned data between source and target styles. Text style transfer is an instance of both conditional and constrained teat generation given that condition the given source lext and constrain the transferred sentences to stylistically match larget examples.
Question Answering Question answering systems are designed to find and integrate information from various sources to provide responses to user questions (Fu & Feng 2018 While traditionally candidate answers consist of words, phrases or sentence snippets retrieved and ranked appropriately from knowledge bases and textual documents (Kratzwald et al. 2019). answer generation aims to produce more natural answers by using neural models to generate the answer sentence. Question answering both conditional and constrained text generation task; the system conditions on the user question, and simultaneously ensures that concepts needed to answer the question are present in the generated output. Diverse question answering systems are proposed in the literature addressing for eg., medical information needs (Wiese et al. 2017). mathematical questions (Schubotz et al. 2018), quiz bowl questions ([yyer et al. 2014), cross-lingual and multi-lingual questions [Loginova et al. 2018). Notably, in practical applications users are not only interested in learning the exact answer word or phrase, but also in how it relates to background information and to previously asked questions and answers FFu Feng 2018).
Narrative Gencration/Story Telling Neural narrative generation is an important step towards com- putational creativity [Gervás] 2009) and represents a long-form open-ended text generation task which simultaneously addresses the selection of appropriate content ("what to say") and the surface realization of the generation ( ("how to say it (Wiseman et al., 2017). Narrative generation is construined teat gener- ation task that places explicit constraints on concepts to steer the narrative in particular topic directions and expands the few keywords specified as the story title, beginning or ending. While existing models can generate stories with good local coherence, generating long stories is challenging. Difficulties in coalescing individual phrases into coherent plots and in maintaining character consistency throughout the story lead to rapid decrease in coherence as the output length increases (van Stegeren & Theune 2019). Hierarchical models for story generation break down the generation process into multiple steps: first modelling the action sequence, then the story narrative, and finally entities such story characters (Fan et al. 2019 Neural narrative generation combining story-writing with human collaboration in an interactive way improves both story quality and human engagement (Goldfarb- Tarrant et al. 2019).
Poetry Generation The poem generalor operates in an interactive context where the user supplies the model with a set of ordered concepts that reflect her writing intent, as well as the format of the poem, for eg, quatrain or regulated verse. Poctry generation is a constroined text generation problem since user defined concepts need to be included in the generated poem, and a conditional text generation problem given the explicit conditioning on stylistic attributes. For detailed overview of poetry generation Oliveira 2017
8

---

Published in Transactions on Machine Learning Research (02/2025)
5 Constrained NLG Methods
Accounting for the different types of constraints introduced in Section we distinguish the following method- ologics commonly employed in the constrained text generation literature: i) decoding approaches, ii) fine- tuning approaches, wii discriminative approaches, iv) edit-based approaches, v) adapting existing models and architectures to accommodate constraints on the generated output, and vi) prompting large language models. In what follows we present each approach in detail, outlining the main associated challenges.
5.1 Decoding approaches
The most popular approach to text generation in the literature has been supervised learning with task- specific datasets; nevertheless since many real-world applications require diverse and potentially evolving constraints, it is infeasible to annotate task-specific training data for every combination of constraints (Qin et etal. 2022p. Even if collecting the data was not a bottleneck, re-training large language models that are extreme in scale for each new constraint or combination of constraints is undesirable. The alternative to fine- tuning language models with task-specific datasets is to enrich decoding algorithms so as to accommodate constraints on the fly. We present decoding approaches to constrained text generation below.
Lexical constraints Levically constrained (guided) decoding aims to restrict the search space decoding time to sequences which contain pre-defined lexical constraints only. These lexical constraints can be specified in the form of a word constraint (a single token) or a phrasal constraint (a multi-word phrase, i.e. a sequence of two or more contiguous tokens). To this end, the beam search decoding algorithm is modified to enforce the inclusion of pre-specified words and phrases in the generated output by allowing the model distribution to not only account for the given lexical constraints, but also to generate parts of the output sequence not covered by the constraints. In general, the decoder can more easily place multiple sequential tokens in a phrasal constraint (where the permutation order is fixed) on the generated output opposed to placing multiple separate, independent constraints. In addition, the lexically constrained decoding approach assumes lexical constraints are pre-determined, which may not always be the case; if so, the open question is where to get lexical constraints from.
Early work on constrained decoding in machine translation relies on the placeholder approach designed to recognize identifiable elements (numbers and named entities) in the source sentence, temporarily replace these with corresponding placeholders during preprocessing, and then substitute the assigned placeholders with the original sourco-language strings during beam search docoding (Crego et al. 2016 Iso 2024). Nevertheless, such approach is limited and unable to model the source lokens in larget language specific terminology or domain. Prefix decoding represents modification of beam
the vocabulary from a new search to first ensure that a user defined target prefix is generated first, and only after build hypotheses for the suffix that maximize the coverage of the remaining source-side tokens. As decoding progresses from left to right, the decoder transitions from a constrained prefix decoding mode to unconstrained beam search. For example, the start of the sentence symbol 
 can be easily included as the first word of a constraint [Knowles & Kochn 2016; Wuebker et al. 2016). In the context of text summarization, an essential property of summarization system is the ability to generate a summary with desired length. Grid beam search [Hokamp & Liu 2017) extends beam search decoding to allow for the inclusion of arbitrary target side hard lexical constraints at any position in the generated sequence. Given C' input constraints, the algorithm maintains C+1 separate beams By, Bi..... : Bg that group together hypotheses which meet the same number of satisfied constraints. Decoding runs similar to beam search, with an additional dimension added to keep track of how many constraints are met by each hypothesis at every timestep; the highest scoring hypothesis in beam B, is ultimately generated. However, grid beam search is impractical as decoding complexity is linear in the number of constraints, i.e. beam size increases proportionally to the amount of constraints and changes for every sentence. Constrained beam search (Anderson et al. (017)) guarantees the inclusion of input constraints in the generated sentences by extending beam search with a finite state machine whose states mark completed subsets of the input set of constraints; however, decoding complexity has an exponential cost in the number of constraints, making infeasible in many applications. Dynamic beam allocation (Post & Vilar 2018) improves upon the runtime complexity of grid beam search and constrained beam search by decoding with constant complexity O(1) in the number of constraints. The algorithm still groups
9

---

Published in Transactions on Machine Learning Research (02/2025)
together hypotheses that have met the same number of constraints by using single fixed-size beam which is dynamically divided at each time-step according to how many constraints have been met. Despite being more efficient, dynamic beam allocation does not necessarily outperform conventional beam search {Lin et al, 2019). In addition, the generation of hypotheses that only partially satisfy a phrasal constraint needs to be aborted to unwind to the tokens in the constraint. Neurologic decoding (Luu et al., 2021j modifies beam search to enforce the satisfaction of lexical constraints expressed under predicate logic in conjunctive normal form (CNF). Given the intractability of exhaustive beam search to optimize CNF constraints, the algorithm searches for approximately-optimall output sequences in which all clauses are satisfied, inchuding both positive and negative constraints (i.e. words that must be generated, respectively omitted in the output sequence). The method is applied to cooking recipe generation, where the task is to generate cooking instructions given a dish name and a list of ingredients, and to data-grounded dialogue response generation where a response is generated given a query and a list of facts to convey.
In general, lexically constrained decoding methods have high computational complexity and force the inclu- sion of specific words in the generaled sentence at every timestep of the generation process with no prior examination of these specific words before generation begins (Latif et al. 2020); this unnatural way of gen- crating sentences can impact the quality and naturalness of the generated output (Liu et al. 2019a Post & Vilar 2018). In addition, there is a trade-off between the generated text quality and hard constraint satisfac- tion 1so 2024)- In a. lack of suitable evaluation metrics, there is no commonly agreed criteria for objectively assessing the quality of the generated sentences and conducting comparisons across text generation models.
Format constraints Fized length decoding (Kikuchi et al. 2016) constrains the length of generated summaries in two ways: i) by preventing the decoder from generating the ond-of-sentence tag until the length of the generated sequence excoods the desired length, and ii) by defining the minimum and maximum longth range of the sequence and discarding out-of-range sequences. Non-monotonie decoding approaches allow tokens to be inserted at any position in the generated sequence during decoding, therefore accommodating flexible orderings of the output. Unlike left-to-right autoregressive generation that produces a single word at a time, non- mon-monot decoding can satisfy lexical constraints at multiple locations in the output sequence allowing for highly parallel generation and faster decoding times. Nevertheless, such approaches assume the generated sequence length is known a priori, preventing it from being dynamically adjusted as generation proceods. Moreover, such models assume conditional independence between output tokens, i.c. tokens are generated independently, may be inconsistent and agnostic to each other. Consequently, this approach may hurt the expressiveness of the model and lead to potential performance degradation, impacting the fluency and naturalness of the output. In addition, non-monotonic sequence decoding approaches can terminate prematurely before constraints are satisfied in the output sequence (Zhang et al.) 2020b Hsich et al. 2021 The main limitation of this approach is the lack of model expressiveness in accommoaating constraints.
Insertion Transformer (Stern et al., 2019) proposes a flexible sequence generation framework based on re- peated insertion operations into an initially empty output sequence until a termination condition is met. The model adopts a progressive masking approach based on token importance in the original text and is trained to generate missing token between every two tokens in the input. To this end, the original Trans- former (Vaswani et al. 2017) decoder is modified to allow insertions not just at the end but anywhere in the output sequence. The model can decode sequences serially one token at a time, or it can decode sequences in parallel with simultaneous insertions at multiple locations. A similar approach is considered in InDIGO (Gu et al. 2019a) which extends Transformer for insertion- based decoding with inferred generation order. Token generation order for the output sequence is modeled as a latent variable, and at each decoding step the model predicts both the generated word and its position in the output sequence; nevertheless, strong conditional independence is assumed between the output tokens which hurts output quality. An iterative refinement step based on latent variables is added to the Transformer decoder to refine a target sequence gradually over multiple steps until a predefined stopping criterion is met [Lee et al. 2018). Progressive In- sertion Transformer (Zhang et al. 2020b) uses non-auloregressive modeling based on a top-down progressive structure for lexical hard-constrained text generation. Given lexical constraints as input, the model inserts tokens progressively according to word importance to generate the target sequence, as follows: first it gen- crates high-level words in a sentence such as nouns, adjectives and verbs, then uses these as pivoting points to insert details of finer granularity and finally completes the sentence by adding connecting words which
10

---

Published in Transactions on Machine Learning Research (02/2025)
carry less information, such as pronouns and prepositions. Entity Constrained Insertion Transformer (Hsich et al. 2 builds upon previous models considering hard lexical constraints in the form of entities in the output sequence. Similar approaches train the Transformer decoder to insert missing tokens in a partially complete sequence without relying on a pre-specified factorization of tokens (Chan et al.). 2019; Gu et al. 2019b); based on the information available in the sequence, the insertion-based generative model is able to dynamically infer the remaining parts irrespective of their arbitrary order.
Syntactic and Semantic constraints Distributional constraints (Baheti et al. 2018 on topic and semantic similarity are used to incorporate source side-information at decoding time in neural conversational systems and encourage the generation of more diverse responses. Moreover, constraints over topics and syntax are used to generate matching or semantically similar statements in response to the user input (Niul & Bansal 2018). Lexically constrained decoding from pre-trained language models aims to steer language models in useful and safe directions so as to minimize the risks associated with these models generating biasod, offensive and toxic content (Sheng et al. 2019 Holtzman et al. 2020). Weighted decoding methods employ prefix-prepende prompt) to
linear combination of output logits from multiple prompts (raw prompt 2023 Zhang & Song 2022)
vary the strength of desired target attributes in the output text (Pei et al.
Energy-based constrained decoding allows the specification of style and lexical constraints through an energy function and performs differentiable reasoning through gradient-basel sampling (Qin et al. 2022
(Mireshghallah et al. 2022). Constrained text generation is viewed as an optimization problem, where the goal is to itcratively sock text with lower energy. The sampling process uses gradients of the energy function to update continuous relaxation of text data, which is then mapped back to the discrete space of natural language via discretization approach. For streering the generation towards desired constraints, biases are applied to the logits of the pre-trained model output layer, which is also found to improve the speed of the decoding process (Liu et al., 2023b). Nevertheless, sampling from mergy-baset models requires many iterations to converge to plausible text.
5.2 Fine-tuning approaches
Semantic and Utility constraints Controlling the output of pre-trained language models is crucial in a wide-range of safety-critcaal applications, including mental health support chatbots, sentiment controlled text generation, language detoxification, etc. To this end, fine-tuning approaches are used for fine-grained control over individual stylistic aspects (for eg., length, professional and descriptive style, tense, personal voice, gender) and content aspects (for eg., sentiment and topic) of the generated texts (Ficler & Goldberg, 2017) [Lample et al. 2018). Typically, the pre-trained model is fine-tuned separately for each attribute of interest, which poses the challenge of how to learn disentangled latent representations of style and content in neural language models (John et al., 2019) and isolate the desired attribute from the distribution shift between the generative model and the fine-tuned dataset. The lack of datascts that are diverse and representative of constrained criteria encountered in practice represents an open challenge for fine-tuning pre-trained models.
CTRL (Keskar et al. 2019) uses control codes to generate texts that meet user-defined constraints on domain, style, topics, dates, entities, relationships between entities, plot points, and task-related behavior. These pre- defined codes are appended at the beginning of raw text sequences to define task-specific training data and create controllable task-specific behaviour at sampling time. Similarly, fine-grained semantic control codes are used to steer generation towards targeted attributes (Ross et al.) 2022). Decoding Experts (DExperts) [Liu et al.) 2021a) is a decoding time method for constrained text generation which combines a pre-trained language model with both an "expert" and "anti-cxpert" language model in a product of experts. The "expert" models desirable aspects of the generated text (for eg,, positive sentiment), while the "anti-expert" plays the antagonistic role of modeling undesirable attributes to be avoided (for eg., toxicity); each one of the three language models is conditioned on the same user prompt. While the method highlights the promise of customizing decoding from pre-trained language models in safe and efficient ways, gathering large amounts of toxic data to model undesirable attributes may be challenging. In general, adding negativity to a positive prompt is a much easier task than adding a positive turn to a negative prompt (Madotto et al., 2020). Augmenting the set of positive examples commonly used for LLM training with a set of negative examples, i.c. completions given a prompt that model should not generate, helps reduce the likelihood of repetitive model generations and negative tokens occuring in the output (Adolphs et. al. 2023). Similarly,
11

---

Published in Transactions on Machine Learning Research (02/2025)
token-level or sequence-level objectives are used to discourage LLM models from assigning high probabilities 2021b). Aligning LLM models with user preferences at inference time is important for personalizing LLM models to their users without the need to retrain the model for each new target attribute. SteerLLM (Dong et al.) 2023) allows end-users to customize responses during inference by conditioning a supervised fineluned model on a mulli-Qimennionall set of user altributes.
to certain tokens or sequences (Lu et al. 2022 Liu et al.
Reinforcement learning from human feedback (RLHF) (Christiano et al. 2017 Stiennon et al., 2020 is a key component in improving the instruction-followeee following and generation abilities of LLM models. Aligning AI models with human preferences is considered crucial for safely deploying artificial systems in the world and ensuring they exhibit behaviors consistent with human values (Ouyang et al. 2022 OpenAT 2022: Ziegler et al. 2019 Shen et al. 2023 Wang et al., 2024b). RLHF algorithms further finetune under a KL-constrained RL objective a language model that has already undergone supervised fine-tuning (referred to as the reference model); this objective encourages the model to maximize the reward and simultancously discourages high KL divergence between the language model and the reference model. The reward model derived from human preferences on text continuations with positive sentiment or vividly descriptive language, while the KL constraint is used to prevent the fine-tuned policy from drifting too far from the reference policy. In dialogue systems, KL control has been used to retain prior information and penalize divergence from the pre-trained model during RL fine-tuning (Jaques et al. 2019). Despite the promise of RLHF in aligning LLMs to user preferences, fine-grained control over large language models remains a significant challenge (Wang et al. 2024a). Since RLHF training can be difficult and unstable, alternative approaches finetune language models to mimic the Best-of-N (BoN) (Nakano et al. 2021 Touvron et al., (023) distribution by minimizing the KL. divergence between the language model and the BoN distribution, for example when generating movie reviews with positive sentiment (Amini et al. 2024) in text summarization (Gui et al., 2024). Nevertheless, KL does not capture attributes of generated text that humans judge to be salient, such as the length of the generated response (Singhal et al. 2024). The lack of suitable evaluation metrics that correlate with human judgements is a bottleneck in generating high quality outputs.
Pre-trained OpeuAI-GPT2 (Radford et al. 2019) model is used to re-write a story through counterfactual reasoning and generate a narrative consistent with the imposed constraints (Qin et al., 2019). In abstractive summarization, OpenAI-GPT2 is used in reinforcement lcarning setting which trains the summarization agent to maximize coverage and fluency of the generated content constrained on a pre-defined length (Laban et al. 2. RecipeGPT (H. Lee et al., 2020) fine-tunes the GPT-2 pre-trained language model for generating cooking instructions when hard constraints are placed on the recipe title and ingredients; the model can also generate the list of ingredients for a recipe when constrained on the recipe title and specific cooking rnstroct Infilling by language modeling is used to complete variable length text spans (e.g- words, n-grams and sentences) by fine-tuning a pre-trained language model on sentence pairs that contain both artificially-masked text and the correspouding original text. (Donahue et al. 2020).
While fine-tuning models on task specific datasets has become the dominant paradigm for constrained text generation from pre-trained large language models, these models generally fail to reliably incorporate the underlying constraints in the generated texts even when supervised with large amounts of task-specific examples (Lu et al.) 2021). Moreover, the superficial alignment hypothesis (Zhou et al. 2024 argues that that almost all knowledge in large language models is learned during pre-training and that alignment tuning only teaches the base LLM in which data format and language style to interact with its users. Further analysis of this hypothesis finds that fine-tuning approaches only influence small number of tokens focused primarily stylistic elements (Lin et al. 2024a). The lack of model expressivity to incorporate constraints in an important challenge for fine-grained constrained text generation.
Format constraints Due to length biases in reward models, LLM models fine-tuned with RLHF tend to suffer from verbosity issues and generate long answers that are not necessarily of high quality (Singhal et al. 2024 Park el al. 2024b Dubois et al., 2024b Kabir et al.] 2023 Nakano el al., 2021 Sun el al., 2023b Wu et al.) 2024a). Offline preference optimization algorithms with implicit reward are known to exploit evaluator length biases and quickly increase the length of the generated text during training without capturing more complex features of human preferences (Rafailov et al. 2024 Li et al. 2023b Dubois et al.,
2024a). To disentangle verbosity from quality, length-controlled direct preference alignment methods employ
12

---

Published in Transactions on Machine Learning Research (02/2025)
an additional regularization term in the loss function that governs the token length of the generated response (Park et al. 2024b). Online RLHF approaches that first train reward model include a similar length regularization term in the reward modeling stage (Chen et al. 2024a). Open problems with current RLHF frameworks include their limited ability to capture and adapt to the complexity of user-dependent preferences in the real world (Wang el al. 2024a). Since human preferences can change over time depending on users and their expectations, methods relying on multi-objective reward models are used to capture desirable aspects of the generated texts (eg., verbosity, factuality, helpfulness, harmlessness) (Pan et al.) 2023: Ramee et al., 2024a; Dong et al. 2023 Bakker et al.) 2022 Wu et al. 2024a3. To enhance the control and personalization of single LLM model across desired range of attributes, different user preferences are encoded as unit vectors and embedded numerically into the system prompt (Wang et al.) 2024a Yang et al. 2024b Dong et al. 2023). The analysis of the alignment tuning process reveals that RLHF simply Leaches the base LLM model to select a sub-distribution of data formats for interacting with the user (Lin et al. 2024a). Notably, most distribution shifts occur with stylistic tokens (transitional phrases, discourse markers, safcty disclaimers) instead of content-bearing words, which supports the superficial aligument hypothesis (Zhou et al., 2024).
5.3 Discriminative approaches
Utility constraints One of the early works proposing constrained text generation learns disentangled latent representations by combining variational auto-encoders with attribute discriminators (Hu et al. 2017). Semantic structure is imposed on the latent codes by using global discriminators, one for each attribute, to guide the learning of the discrete text generator and force it to allocate one latent dimension per attribute code. The model is used to manipulate the sentiment. and tense of the generated sentences.
Weighted decoding (Holtzman et al. 2018) relies on a mixture of discriminative models to guide a recurrent generator towards incorporating attributes that enhance the overall coherence, style, and information content of the generated text. The discriminators complement each other and their weighted contributions form the final decoding objective from the generator. Similarly, stylistic configurations are revised and polished for generated pocms by adding additional weights during decoding to control the style of generated poem. including the repetition, alliteration, word length, cursing, sentiment, and concreteness (Ghazvininejad et al. 2017). Nevertheless, modifying the scoring function used for generation as in weighted decoding often leads to sacrificing fluency and coherence of the generated text (See et al.) 2019). Selective sampling (Wang et al., 2017) relies on a sample selector (multilayer perceptron for binary classification) which outputs whether the current sample should be accepted or rejected based on the presence of desired target words that define the output style and topic in the generated sequence. The robustness of evaluation metrics is directly correlated with model performance, therefore it is crucial to focus on developing metrics that capture diverse aspects of text quality during training and sampling time.
Gencrating texts with desirable attributes from a pre-trained unconditional language model P(X) is a non- trivial lask. Most approaches resort to either training from scratch a new conditional model P(X for desired attribute a, or fine-tuning ((X) on additional data representative for the attribute a. Theoretically, rejection sampling could also be used to sample P(X(a) from P(x),, but this approach is highly inefficient in practice. Fudge (Yang & Klein 2021) generates text conditioned on desired attribute (for eg., topic control in language generation, degree of formality in machine translation) while only accessing the output probabilities P(X) of generative model G. Given an incomplete sequence prefix, the model trains binary discriminative models for one or multiple desired attributes to predict whether the attribute(s) will be fulfilled in the future complete sequence, therefore evaluation is an important, challenge. The output probabilities of the discriminator(s) are then multiplicd with the output logits of the generator G to adjust the original probabilities of G accounting for desired altribute(s) a and model P(X(a) via Bayesian decomposition.
PPLM (Dathathri et al. 2020) combines pre-trained language model with attribute classifiers that guide generation towards specific topics and sentiment styles. These classifiers are trained on top of the last hidden layer of the pre-trained language model, and gradients from the classifiers are backpropagated to update the hidden representations of the language model and steer generation in desirable directions. While PPLM achieves fino-grained control of content and style attributes via a simple gradient-based sampling mechanism, the approach is computationally intensive and inefficient as it requires multiple forward and backward passes for each generation step. Plug-and-play methods have been used to control large pre-
13

---

Published in Transactions on Machine Learning Research (02/2025)
trained conversational models such as GPT-2 Radford et al. 2019) using a variety of styles (positive and negative sentiment) and topies (Question, Sport, Business, Finance) (Madotto et al. 2020 Liu et al. 2020). Plug-in Language Model (PiLM) (Yang et al. 2024a) manipulates the latent state of the language model using a regression model to generate texts that adhere to specific topics or sentiment. More effort needs to be locused on collecting datasets for constrained lext generation that capture real-world constraints.
GeDi (Krause et al. 2021) guides language generation from large language models towards desired attributes by using generative discriminators to compute classification likelihoods for all candidate next tokens on the fly at generation time. Given conditonn language model conditioned both on desired attribute ct and an undosired attribute ct, GeDi-guided contrastive generation uses the two instances of the model as discriminative classifiers to contrast and filter out common attributes between the two classes et and c-; then aspects of the desired attribute ct are transferred across domains via weighted decoding and filtering, The contrast between positive and a negative class conditional distribution is employed both at training and inference time control the bias, toxicity and negativity of GPT-2 (Radford et al. 2019) and GPT-3 BBrwnn al. 2020). Recent approaches explore the role word embeddings can play in debiasing LLMs and steering their generations in particular sentiment directions (Subramani et al. 2022 Turner et al. 2023 Li
& Liang 2021| LM-Steer (Han et al.) 2023) leverages the fact that lincar transformations in output word embeddings are equivalent to style changes in LLM generation; the method linearly transforms output word embeddings at decoding time using learnt parameters representative for each target style.
5.4 Edit based approaches
Utility constraints Edit based approaches rely on the key idea that changing only few words or phrases which are indicative of particular attribute are sufficient to alter the style of a given piece of text. For example, the sentiment of sentence can be altered from negative to positive by first identifying negative attribute markers ( ("bad', "worst", 'disappointed'), deleting these negative attributes while keeping other content words fixed, and then generating the final output via a recurrent decoder which conditions on the extracted content words and the target attribute (Li et al. 2018). Leaving from the observation that humans write text in incremental passes with multiple revisions, a prototypo-then-cde model first samples a prototype sentence from the training corpus and then edits it conditioned on an edit vector (Guu et al. 2018). Noticeably, text generation based on editing a prototype is much casier compared to generating text from scratch. Also building upon the 'Delete Retrieve Generate' framework, the Generative Style Transformer (Sudhakar et al.] 2019) incorporates neural mechanism to delete style attributes from the source sentence based on the attention weights of a Transformer model (Delete Transformer), and then generates sentences in the desired target style by decoding with a pre-trained GPT-2 Radford et al. 2018) model.
Activation editing (Li et al. 2023a Hernandez et al. 2024: Ti al. 2024c) methods discover directions in the representation space that correspond to encodings of specific attributes (such as sentiment, topic, style, fac- tual information, truthfulness, etc). When these encodings are added to the internal representations of large language models, they act as knowledge editors by manipulating the generated output to be consistent with desired constraints (Kong et al.) generation without the need to rely on textual prompts, while enhancing LLM control and intepretability.
2024h. The advantage of representation edits is that they enable constrained
Steering vectors (Subramani et al. 2022 Turner et al.] 2023) are added to the hidden states of a language model to generate texts with desired style or sentiment. Other editing approaches manipulate model weights instead of representations by inserting updated factual knowledge directly into model specific parameters
(Meng et al.
2022;
Meng et al. Mitchell et al. 2022 Dai et al.) 2022 Rawat et al. 2021 Ihharco et al., 2023 Nevertheless, open challenges with these models include indeterminate/ /unerraainnnnnc edit-
Orgad et al.
2023
ing boundarics, failing to account for contextual information and entailed consequences of edited facts {Liu et al. 2024b Cohen et al. 2024: Zhong et. al., 2023). Evaluations focused on assessing the success of model edits mostly consider a few tokens generated after an input prompt, and do not measure the consistency of edits over a long generation of text; failure modes observed in long-form generation include topic drift, lexical cohesion issues, gradual and catastrophic forgetting of previous edited facts these aspects limit the usefulness of model editing methods at scale (Rosati et al. 2024
Gupta et al. 2024 Li et al. 2023d)
14

---

Published in Transactions on Machine Learning Research (02/2025)
Lexical constraints Mctropolis- Haatopolis--astingss sampling (Miao et al. 2019 first inserts all constraint keywords in template in random order, then samples local edit operations (word replacement, deletion or insertion) to perform at specific positions for improving sentence fluency. The probability of each edit operation being accepted or rejected is determined by a language model, however individually sampling cach token results in slow convergence. Instead of randomly sampling edit operations, the gradient. of differentiable objective function used to determine where and how to edit (Sha (020). The majority of editing approaches model a single edit step, unlike humans who do iterative refinement and editing of the content. Modeling the whole process of iteratively generating sequences leverages neural network models to describe the likelihood of multi-step edits, with improved performance over modeling single-order edits (Reid & Neubig, 2022). Self- correction (Welleck et al. 2023) incorporates a learnt mechanism to iteratively revise LLM model outputs and correct imperfect generations that. violate lexical, mathematical reasoning and loxicily constraints.
5.5 Adapting existing models and architectures to accommodate constraints
It is non-trivial to impose constraints on existing deep learning models while maintaining high generation quality since their model architecture is designed to generate sentences sequentially from left to right. While current doep learning models are lacking the expressiveness to incorporate constraints at training time and at arbitrary positions in the generated sequence, well known models and architectures are adapted to accommodate constraints through set of custom engineered approaches. We present these methods below.
Lexical constraints Current architectures used for language generation produce texts sequentially from the first word to the last. word, and it is non-trivial to impose lexical constraints on lest-to-right generation while maintaining high output quality for natural and fluent texts, Current workarounds for hard lexically constrained text gencration address this limitation by generating texts in a non-monotonic fashion when employing brrtaard-bakkwoo language models. The backward language model takes lexical constraint as input and generates the first half of the sentence backwards conditioned forward language model takes as input the sequence generated by the backward generator and produces its sentence completion in normal order conditioned on the backward generated sequence. While the topic word can occur at any position in the sentence, this approach can only generate output constrained on at most one lexical constraint; generating sequences with multiple lexical constraints is an open research problem. These approaches adapt existing frameworks for constrained text generation by splitting a sentence into two parts, which is unnatural and also hurts fluency when generating half of the sequence in reverse order.
the topic word, while the
Given a topic word at an arbitrary position in scientific paper title, a recurrent language model is tasked with generaling both past and future words in the litle conditioned on the given topic (Mou el al., 2015). Similarly, on-topic dialogue responses that satisfy hard lexical constraints are generated with a "sequence to backward and forward sequencos" (seq2bf) model (Mou et al. 2016) which first predicts a keyword noun that reflects the gist of the response, then decodes the response backward and forward starting from the given word. BEGAN (Liu et al., 2019a) employs GANs for lexically constrained text generation (product reviews, conversational responses). The model incorporates three modules, namely a backward generator and forward generator which collaborate generating lexically constrained sentences, and discriminator which guides the joint training with policy gradient of the two generators.
Generating a fluent sequence which simultaneously satisfies multiple lexical constraints employs a backward- forward LSTM language model to first generate the sequence from a user-defined verb constraint and then satisfy other lexical constraints by word embedding substitution based on cosine similarity between generated tokens and desired constraints (Latif et al., 2020). Nevertheless, the approach assumes verb constraint is always specified in the set of lexical constraints.
Semantic and Utility constraints Steering neural models in specific directions is achieved by: i) adding special tokens at the beginning or end of the source text, ti) incorporating additional conditions into the decoder hidden states and iii) connecting the conditions directly to the decoder output layer. A topic aware sequence-to-senn to-seueence model is used to generate on-topic conversational responses by conditioning the decoder on specific topic words (Xing et al.) 2016). Imposing conversational goals on dialogue agents aims to guide the conversation towards designated target subject by combining coarse- grained topic constraints with discourse-level rules (Tang et. al. 2019). Generating emotional responses
15

---

Published in Transactions on Machine Learning Research (02/2025)
in neural conversational systems is achieved by feeding the emotion category embedding to sequence- to-sequence decoder (Zhou et al. 2018). Personalized chit-chat dialogue agents that display consistent personalities, viewpoints and are configurable depending on attributes of the system user are used to produce more personal, specific and engaging dialogue responses (Wang et al.) 2017 Bosschut et al. 2018 Zhang et al., 2018a). Nevertheless, finding the proper balance between fluency, engagement, consistency and persistent personality remains an open challenge for current dialogue models due to lack of a measurable objective function and correspondingly suitable evaluation metrics. While it is possible to judge whether or not an output satisfies one constraint, it is hard to judge the extent which ("how much") the constraint is satisfied; it is even harder to jointly model/measiee measure multiple constraints. Moreover, accounting for repetition and diversity is important as these models often get stuck in an infinite loop of redundant, dull, generic and universally relevant responses that carry little meaning (Li et al. 2016c See et al. 2019 Mou et al., 2016).
For integrating factual knowledge into open-ended conversational systems, factoid and entity-rich web docu- ments are encoded altogether with the conversation history into the same representation which is passed to an attentional neural decoder that generates the response tokens. Similarly, speaker-level representations are integrated into seq2seq conversational models for generating personalized conversation responses (Li et al. 2016b). Fact-guided sentence modification for dynamically rewriting, updating or correcting articles ac- cording to changing information is an instance of constrained text generation which presents the particular challenge that the rewritten sentence needs to be consistent with an input claim while at the same time preserving uon-contradicting content (Shah et al. 2020). Given the claim and an old sentence, an updated sentence is produced by first identifying contradictory components in the input sentence, masking these, then using the residual sentence and the claim as input into two encoder saquence-to-sequenee model with copy attention to produce the update sentence consistent with the claim. Syntactically controlled paraphrasc generation produces paraphrases of an input sentence by constraining the system on the target synlactic form [yyer et al. 2018 however not many syntactically constrained datasets to learn from are available.
Controllable story generation with RNNs is used to influence the story ending valence (whether happy or sad) and the storyline (specified as a sequence of words) (Peng et al. 2018 Story-telling methods commonly use a hierarchical approach to thematically consistent story generation, by first generating a prompt describing the topic for the story, and then constraining on the prompt for generating the story content (Fan et al., 2018b); additionally, constraints on the presence of entities are included as well (Clark et al. 2018). Open- domain story generation requires compasing coherent natural language texts that describe plausible sequence of events and is more challenging compared to generating stories in a narrow domain given an existing plot.
Unsupervised machine translation methods are adapted for the task of text-style transfer by incorporating stylistic constraints a neural seq2seq model with attention and using a style classifier to guarantee the accuracy of style transfer (Zhang et al. 2018c), or for control over multiple style attributes, includ- ing gender, sentiment or product type 4Lample et al., 2018). In machine translation, honorifies constraints are important for producing socially appropriate forms of address and controling the level of courtesy (Sen- nrich et al.] 2016); the system user defines the desired level of politeness of the translation, however these user-defined constraints are only soft constraints and can be overridden by the attentional encoder-decoder machine translation system whenever the source text provides strong politeness clues.
For effective imposition of semantic structure constrained text generation, latent space representations need to be disentangled (John et al.) 22019), such that varying an individual latent code will only change single desired attribute. VAEs can achieve meaningful latent representations with designated semantics when combined with attribute discriminators and optimized end-to-end with differentiable softmax ap- proximation (Hu et al.] 2017]; this allows to generate sentences with constraints on sentiment and tense Given an input sequence and a set of labels, sequence transduction with multi-space variational autoencoders (Zhou& & Neubig 2017) generates an output sequence that alters the content of the input sequence accord- ing to the constraints specified by the labels; the method is used for morphological inflection in multiple languages. In general, constrained text generation approaches assume that constraints need to be known a priori; however, this is not always possible, for eg,, when suggesting alternative phrases for search queries in real-time, when generating responses in dialogue systems according to the dynamics of the conversational context. Recent constrained text generation approaches control attributes of generated sequence based on another sentence example: given two sentences X and Y, the goal is to generate a new sentence Z that
16

---

Published in Transactions on Machine Learning Research (02/2025)
follows the semantics of X and the syntax of Y. To this end, VAE model with two latent variables is used to achieve disentanglement in the continuous latent space between syntax and semantics (Chen et al. 2019 Bao et al., 2019). Topic guided VA (Wang et al. 2019) use a Gaussian mixture model prior where each mixture component corresponds to a latent topic extracted from data as opposed to using pre-defined parameter settings which do not incorporate semantic meaning into the latent codes; the model is used for text summarization with designated topic guidance. Abstractive and extractive sentence compression with VAEs assumes the existence of a background language model from which a latent summary sentence is drawn first, and then the observed sentence is generated conditioned on the latent summary (Miao & Blunsom 2016;; the model is able to balance copying a word from the source sentence with generating it from the background distribution. Iterative refinement of a sequence to transform it into another sequence with desired attributes exploits geometry of the latent space to produce incremental higher-quality revi- sions with theoretical guarantees in the combinatorial space of sequence elements (Mueller et al.) 2017 Shen ct al., 2017). Such latent variable manipulations can rewrite modern text in the language of Shakespeare, improve sentence positivily, address word substitution and word order recovery lasks without need for any revision examples. Constraints the use of metaphor and personification in poems are incorporated in a conditional VAE with a rhetorically controlled decoder trained to emit meaningful and diverse rhetoric and overcome generic sentences (Liu et al. 2019b). Variational neural machine translation (Zhang et al. 2016) incorporates a continuous latent, variable to model the underlying semantics of sentence pairs. Nevert efficiently performing posterior inference and large-scale training during the incorporation of latent variables remains an open challenge for constrained VAEs. Finding structure in the latent space that corresponds to particular sentiment makes it possible to steer the generation towards desired sentiment by adding the sentiment direction vector to the residual stream when generating sentence completions (Tigges et al. 2023
Modilying textual attributes of sentences including sentiment, style, tense, voice, mood and negation is achieved by incorporating conditioning information into a neural encoder-decoder model, and optimizing a reconstruction loss which interpolates between auto-encoding and back-translation components to encourage content compatibility, as well as an adversarial loss which encourages sentence-level stylistic attribute compatibility (Logeswaran et al., 2018). The model allows simultaneous conditioning on multiple textual attributes, however the extent to which the generated sentences match the conditioning information roquires new objective evaluation metrics for attribute accuracy and content compatibility / prescorvation.
Style transfer between scientific papers and newspapers is performed with separate style decoders, or by generating both content and style from the same decoder (Fu et al.] 2018). In poetry generation, it is common to impose hard constraints on rhyme, rhythm, and topic (Ghazvininejad et al.] 2016; 2017). Given a user-supplied lopic, the poetry generation algorithm first generales a large set of on-lopic words and phrases, assigns rhyming words and phrasos to specific lines, and then combines finito-state machinery with an RNN language model to score plausible poems that meet the desired constraints. While augmenting an RNN with a working memory to explicitly maintain a limited history of generated topics and context, coherence in meaning and topics across the overall poem remains an important challenge (Zhang & Lapata 2014). Constrained recurrent models are also used to generate online product reviews of certain topic, sentiment, style and length (Ficler & Goldberg, 2017), affective dialogue responses (Ghosh et al. 2017), or for modeling participant roles and topics in conversational systems (Mei et al. 2017
Alternative non-autoregreesive architoctures basod on continuous diffusion models are adapted for text gen- eration with semantic and syntactic constraints (Yang el. al.] 2023 Austin et. al.) 2021 Gong et al.) 2022). Diffusion-LM (Li et al. 2022) gradually denoises a sequence of Gaussian noise vectors into word vectors, resulting in a hicrarchy of continuous latent reprosentations which enables gradient-based based methods to steer the text generation process. Nevertheless, training of diffusion models is slower to converge and decoding from these models takes longer time. To speed up the inference process, adaptive sampling strategies are applied for different generation stages in the context of story generation (Tang et al. 2023).
Format and Utility constraints Text simplification models parameterized on constraints such as length, amount of paraphrasing, degroe of lexical and syntactic complexity are used for generating texts casier to read and understand with simpler grammar and structure (Martin et al., 2020). Towards a similar goal of controlling the degree of lexical complexity, the training loss function is changed to assign weights
17

---

Published in Transactions on Machine Learning Research (02/2025)
to words based on their complezity level (Nishihara et al. 2019). In text summarization, constraints on the output sequence length for neural encoder-decoder models are specified as length embeddings and are passed as additional input to the decoder (Kikuchi et al. 2016).
Faithfulness in abstractive text summarization is enforced in a seq2seq model by conditioning on both the source text and extracted factual descriptions (Cao al. 2018): this helps avoid generating false facts the output summary. Hybrid text summarization approaches combine an unsupervised sentence extractor
which selects salient sentences from the input document with sentence abstractor that paraphrases cach extracted sentence to overcome limitations of parallel aligned datasets (Nikolov & Hahnloser 2020
Reinforcement learning is used for constrined NLG to directly optimize non-differentiabble reward functions and evaluation metrics. While any user-defined reward function can be employed for training, most. frequently optimized metrics with RL are BLEU for machine translation (Ranzato et al. 2016), ROUGE for text
summarization (Ranzato et al. 2016 Paulus ct al.] 2018 Wu & Hu 2018 Gao et al. 2019b or human- defined conversation metries locused on coherence, inlormativenesss sentiment, politeness, Loxicily, question,
repetition or semantic similarity (Li et al.) 2016c Saleh et al.) 2020 Wu & Hu 2018). However, manually defined reward functions based on heuristics cannot cover all crucial aspects of a natural realistic conversation [Bossclut et al. 2018 Gao et al. 2019b). In addition, rewards are commonly modeled at the word level accounting for the probability of generating each word in a sentence (Ranzato et al., 2016 Jaques et al. 2019); such low-level control makes credit assignment challenging since the number of actions available to the RL agent is equivalent to the number of words in the vocabulary. Defining a global score that measures complex aspects of text quality beyond local n-gram patterns and which can reliably approximate human judgments of text quality remains open challenge (Bosselut et al. 2018).
In the RL framework the generative model is seen as an agent with parameters that define a policy and which interacts with an external environment by taking actions, receives a reward once it reaches the end of a sequence and updates its internal state consoquently. To this end, policy gradient methods are used to train text generative models and alleviate issues such as exposure bias and loss functions which do not operate at the sequence level. However, policy gradient algorithms present. large variance and generally struggle in settings with large action spaces such as natural language generation. In addition, they take very long time to converge (Choshen al., 2020) and the improvement in the optimized metrics is not always reflected in human evaluations of text quality. Training RL models to optimize n-gram evaluation measures based on local patterns provides only a limited and myopic perspective of overall text quality and does not necessarily lead to better text quality, overall coherence or discourse structure (Bosselut et al. 2018). Moreover, fine- tuning on such measures may yield deteriorated outputs despite increased automatic scores, while difficulty in constrained optimization with RL often leads to sparse, non-informative and delayed reward signals.
Learning RL rewards from human preferences aims incorporate human feedback in text generation and teach models to follow human instructions (Ouyang et al., 2022 OpenAl 2022: Achiam et al. 2023). Neural reward learning schemes train neural teachers that learn to score an ordered sequence of sentences and formulate rewards that guide coherent long text generation (Bossclut et al. 2018): the approach is used for generating cooking recipes given the dish title and the set of ingredients as constraints. Learning-to-res algorithms are used to approximate ground-truth oracle rewards in extractive multi-document summarization to indicate the quality of a summary or preferences over summary pairs (Gao et al. (019b)). Machine learnability of human rewards in neural machine translation models is approached by first training reward estimators on rewards collected from offline logs, then integrating these reward estimators in an off-policy RL setting (Kreutzer et al. 2015). Similarly, implicit human reactions such sentiment or length of conversation are used to learn rewards for fine-tuning off- off-policy RL models for dialog (Jaques et al., 2019). Nevertheless, human leedback is noisy, not well-defined, complex and inconsistent. Using RL. to improve system outputs with respect to human-centered metrics of conversation quality is highly dependent on developing robust metrics for the particular application domain, for example increasing politeness or reducing toxicity of generated responses in dialogue generation.
Hard-constrained text generation in a non-monotonic order relies on tree-based text generation scheme, where a word is generated at an arbitrary position in the sentence, then binary trees of words to its left and right are recursively generated (Welleck et al. 2019). Learning proceeds in an incremental fashion in an
18

---

Published in Transactions on Machine Learning Research (02/2025)
imitation learning framework, where the policy gradually moves from imitating the oracle to reinforcing its own preferences and generating lexts without pre-specified word order. Nevertheless, the time complexity of the approach is O(n), same as for autoregressive models and the constructed tree does not reflect a high-lovel to low-level hicrarchy of concepts. Constraint satisfaction problems with solutions that can be automatically verified are used to evaluate how well LLMs adhere to logical constraints(Lin et al.] 2025 Due to token mislalignment, enforcing strict constraints during generation can nevertheless lead significant decrease in reasoning performance and downstream accuracy (Beurer-Kellner et al. 2024).
5.6 Prompting Large Language Models
Prompt-bascd learning, which became popular with the release of OpenAI GPT-3 (Brown et al., 2020) demonstrates it is possible to elicit. factual and commonsense knowledge from large language models and steer them towards desired behaviours via a textual prompt. Instead of adapting models to downstream tasks via objective engineering as it is common during fine-tuning, prompt-based learning reformulates downstream tasks to resemble those encountered during the language model pre-training phase where a fill-in-the blanks objective is used (Liu et al. 2023a ). While prompting allows for manipulating the model behaviour to predict desired output, sometimes even without additional task-specific training, model performance on a given task is highly dependent on the quality of the prompt used to steer the model and how much conditioning text can fit into the model's input. In general, identifying the most appropriate prompt for a task is challenge. While prompting provides a natural interface for humans to communicate with machines, human users have little knowledge of which instructions are compatible with a given model and need to experiment with a wide range of discrete prompts to find suitable ones that elicit desired behaviours (Zhou et al.) 2022b). Given that plain language prompts do not always produce the intended results, automated methods for prompt design are proposed in the literature, including scarching over the discrete space of words guided by training data (Shin al.] 2020), prefix tuning which optimizes a task-specific continuous vector (Li & Liang 2021 Hambardzumyan et al. 2021), prompt tuning which learns soft prompts via backpropagation
[esterr et al.
2021 Qin & Eisner 2021). natural language prompt engincering where large language models
themsellver ves generate meta-prompts for solving a wide range of tasks [Reynolds & McDonell 2021 Zhou
2022bb or inverse prompting which uses the generated text to inversely predict the prompt (Zou jet al., 2021 Directional Stimulus Prompting (Li et al., (023c) guides black-box language models such
ct al.
as ChatGPT (OpenAI, 2022) towards desired outputs by optimizing a policy model trained to maximize rewards that measure the alignment. between the generated text and desired topics and keywords on tasks such text summarization and dialogue response generation. Stecring LLMs to generate texts that reffect multiple perspectives and diverse opinions is achieved by first modeling data-driven personas when embedding individuals and their viewpoints into a continuous vector space, then using soft prompting techniques to map persona embeddings to specific tokens (Li et al. 2024b Hwang et al. 2023 Santurkar et al., 2023).
While there is not much theoretical understanding behind the reasons why and how prompting works, it is assumed that prompting provides way to steer large language models in particular directions by helping locate a specific task in the pre-trained model's existing space of learned tasks, phenomenon evidenced by the superior performance of some prompts over others (Reynolds & McDonell 2021). Nevertheless, prompting large language models is far from sufficient for robust and reliable constrained text generation. Prompting approaches must be employed with caution, as models can deviate from the original prompt, fail to maintain the coherence and produce texts on unrelated topics (Zou et al. 2021 Hernandez et al., 2024), and may even degenerate into toxic text. from seemingly innocuous prompts (Gehman et al. 2020). In improving prompting reliability, it is important to account for generalization outside of distribution, reducing social biases, ensuring fairness to different demographic groups, calibrating output probabilities and updating the model's factual knowledge and reasoning chains (Si et al., 2023 Bach et al. 2022). Adapting to new constraints without the need for model retraining can be done by verbalizing the constraints into natural language instructions, then appending these constraint verbalizations to natural language sentences (Zhou et al. 20235).
Prompting Considerations LLMs leverage vast amounts of information they learn from web-scale pre-training datasets which they store in their parameters, resulting in improved performance on many knowledge-intensive tasks (Brown et al., 22200] OpenAI 2022; Achiam et al., (023). Nevertheless, it is im- portant to understand what kind of knowledge L.LSs actually capture. In factuality assessments of LLMs,
19

---

Published in Transactions on Machine Learning Research (02/2025)
it is found that current systems tend to hallucinate and make up facts (Maynez et al. 2020 Tam et al, 2022 Zhou et al.] 2021 Lin et al. 2022). and this behaviour becomes more predominant as the rarity of entities increases (Min et al. 2023). There is a strong correlation between the correctness of answering factoid questions and the number of pro-training documents relevant to that question (Kandpal et al. 2028 models are more accurate on instances whose lerms are more prevalent in the training dala, and struggle on questions containing long-tail terms with low document count. Similarly, mathematical reasoning capa- bilities are correlated with training data frequency, and the selection of the training corpus does impact the few-shot performance of LLMs (Razeghi et al., 2022 Shin et al., 2022) These findings suggest that low- order co-occurrence statistics in the pre-training dataset have a significant impact on model performance, leaving the open question of how much current models generalize beyond their training data. Ideally, a general purpose language model can generalize not only to unseen instances of known lasks, but also to new tasks. LLMs tend to rely on narrow, non non-transferable procedures for task solving specialized to tasks seen during pre-training (Wu et al.] 2024bb; in counterfactual scttings their performance degrades considcrably, indicating overfitting to training lasks.
6 Constrained NLG Evaluation
Evaluation of constrained text generation is performed using the same evaluation approaches and method- ologics available in the natural language generation literature. In general, evaluation of the generated text is largely an unsolved and notoriously difficult problem (Borji 2019). Currently, there is no well-established consensus on how NLG systems should be evaluated, (van der Lee et al. 2019 Gkatzia & Mahamood 2015)) and the lack of meaningful quantitative evaluation metrics to accurately assess the quality of trained models is detrimental to the progress of the field. In the absence of well established evaluation measures, natural lan- guage evaluations are carried in a rather ad-hoc manner with lot of variability across the proposed models and tasks on inconsistent benchmarks, resulting in misleading performance measures. Subjective evaluations based on visual inspection of the generated samples often lack scientific rigour, making it difficult to quantify and judge precisely the quality of generative model (Hashimoto et al., 2019). In what follows we review the main methods for constrained text generation evaluation.
Lexical constraints Measuring how many of the given lexical constraints are included in the generated outputs is done using concept coverage {Lin et al. 2019 Lu et al. 2021]; ; the metric is computed as the the average percentage of input concepts that are present in the lemmatized outputs.
Semantic and syntactic constraints Surface similarity based on n-gram overlap metrics, such as BLEU (Papinoni et al. 2002). , ROUGE (Lin 2004), METEOR (Bancrjee & Lavic 2005) measure to what exlent the generative model can preserve content by retaining words commonly shared between the generated output and ground-truth references. Such metrics are commonly used to measure response relevance in dialogue systems (Galley et al.) 2015: Li et al.] (016b), translation quality in ncural machine translation (Sennrich et al. 2016), summary quality in text summarization (See et al. 2017). In general, the correlation between word overlap metrics and true text quality is widely debated topic (Li et al.) 2016c). Evaluation metrics based on local n-gram patterns only provide a limited, myopic perspective of overall text quality and
are notoriously poor at evaluating dialogue systems (Liu et al. 2016 See et al. 2019 Bosselut et al. 2018
Perplemity (Jelinek et al. 1977) based evaluation metrics are used to evaluate and compare language models, and measure the fluency and diversity of the generated samples (Madotto et al. 2020 Bosselut et al. 2018 Li et al.] 2016b). Reverse Perplexity (Zhao et al.) 2018) and Forward Perplexity (Kim et al. 2017 scores are calculated by training language models on synthetic samples, respectively real samples, and then using these trained models to mcasure perplexity real samples, respectively generated samples. Nevertheless, perplexity is a model dependent metric, and "how likely a sentence is generated by given model" is not directly comparable across different models unless properly normalized; finding the right normalization a challenge that could potentially improve the evaluation of constrained text generation. Moreover, numerous studies find perplexity to be an inadequate measure of text quality {Theis et al. 2016 Fedus et al. 2018), since models with high likelihood can generate low-quality samples, while samples of
20

---

Published in Transactions on Machine Learning Research (02/2025)
good quality can present low likelihood. In addition, infinite perplexity can still be obtained from perfect model even when its ability to generate test sentences is removed (Hashimoto et al. 2019).
P, R, F1 measure the distance of the generated samples to the real data manifold Lucic et al. 2018 ). When precision is high, the generated samples are close to the data manifold; when recall is high, the generator outputs samples that cover the manifold well. Metrics that aggregate precision and recall such as Fg, a generalization of the Fi score, quantify the relative importance of precision and recall (Sajjadi et al.) 2018)) However, the non-syuthetic data manifold is unknown and therefore impossible to compute in practice.
Content diversity measures how different the generated sentences are from each other, by either considering word choice, topic and meaning |Vijaykuuar et al. 2016; Gimpel et al. 2013 pppolito et al. 2018), or by looking at the level of sentence interestingnes or unlikeliness (Hashimolo et al. 2019). Perplexity on a reference set, n-gram diversity (Li et al.) 2016a) and Self-BLEU (Zhu et al. 2018 are commonly used mcasures of the diversity of the generated samples. In addition, BLL (Shi et al. 2018) evaluates test data using the generated samples as reference; the higher the score the more diverse the generator output. Lexical diversity (Bache et al.] 2013) calculates the ratio of unique tokens to the total number of generated tokens. Similarly, Distinct-k or Dist-k (Li et al.) 2016aa measures the total number of unique k-grams normalized by the total number of generated k-gram tokens to avoid favoring long sentences. Nevertheless, the Dist-k metric ignores the fact that infrequent k-grams contribute more to diversity than frequent ones and assign same weight to all k-grams that appear at least once. Distinct-1 and Distinct-2 are used to measure the diversity of constrained conversational responses (Baheti et al. 2018 Zhang et al.] 2018b) and rhetoric constrained generated poems (Liu et al., 2019b). Entropy based metrics such as Ent-k (Zhang al. 2018b) reflect the frequency difference of k-grams and to analyze the information content of the generated responses in dialogue systems (Serban et al. 2017 Mou et al.) 2016).
Unlike traditional evaluation metrics based on heuristics, learnable metrics train machine learning models on human annotated datasets to learn a scoring function that reproduces human judgements. Fully-learnt metrics leverage existing datasets of hurna ratings to learn automated evaluation metrics that fit the human data distribution, and can be tuned to measure specific properties of the generated texts, such as fluency, style, grammaticality, fidelity, etc. Linear regression based on human judgements is used learn model for scoring system summaries (Peyrard et al., 2017 RUSE (Shimanaka et al., 2018) combines sentence embeddings in a multi-layer perceptron regressor model. ESIM (Chen et al.] 2017 Mathur et al., 2019) feeds the encoded representations of the candidate and the reference sentence into a feedforward regressor. BLEURT (Sellam et al. 2020) fine-tunes BERT (Devlin et al.) 2019) on human ratings datasets for similarity score prediction. MAUDE Sinha et al. 2020 is proposed for the evaluation of online dialogue conversations and leverages sentence representations from pre-trained BERT to train text encoders which can distinguish between valid dialogue responses and fake examples. BARTScore (Yuan et al. 2021) formulates the evaluation of generated text as text generation task from pre-trained language models and measures the weighted probability of the generated text given another text as input or output. GPT Judge {Lin ct al. 2022) finc-tunes GPT3 (Brown et al.) (020) model on human annotated data to clasify answers of QA systems as true or false, evaluating factuality and truthfulness. The same evaluation metric, this time based on GPT-4 (Achiam et al. 2023), is used to establish via prompting whether texts generated by GPT-4 are more similar to human-written reference answers or GPT-3 machine-gee -eeeerrteegerereed texts. Nevertheless, the GPT-4 evaluator known to be biased in its preferences towards answers with longer length (Singhal et al., 2024 Wang al. 2023b Li et al. 2023b). GPTScore (Fu et. al. 2024) computes the conditional probability of generating the target text given specific context. FactScore (Min et al. 2023) breaks generation into atomic pieces of information and evaluates the factual precision of long-form text by mcasuring the percentage of atomic facts supported by a reliable knowledge source. Other evaluation metrics based on probabilities inferred from pre-trained masked language models include InfoLM (Colombo et al.] 2022). CTRLEval (ee et al. 2022), MaskEval (Liu et al., 2022). Hybrid metrics combine learnt elements with human-defined logical rules, for example, contextual embeddings with token alignment rules. BERTscore (Zhang et al., 2020a) evaluates generated text against. gold standard references using soft-string similarity matches (i.e. cosine similarity) computed on pro-trained contextualized BERT (Devlin et al.) 2019) token embeddings. MoverScore (Zhao et al. 2019) combines contextualized representations of system and reference texts with semantic measures of distance computed using Word Mover's Distance (Kusner et al.) 2015 the metric
21

---

Published in Transactions on Machine Learning Research (02/2025)
is extended to evaluate multi-sentence texts (Clrk et al. 2019 Human and statistical evaluation are combined in HUSE (Hashimoto et al. 2019), an evaluation framework which estimates the optimal error rate of predicting whether a piece of text is human-written or machine--eneratee A limitation of learned evaluation metrics is that they often fail to generalize across different systems (Chaganty et al. 2018
Utility constraints A commonly used approach in the literature to assess whether generated texts have desirable attributes is to rely on an attribute classifier and measure the classiftcation score, i.e. the fraction of outputs generated by the model having the desired attribute (Hu et al. 2017 Shen et al. 2017 Li et al. 2018 Adversarial evaluation (Bowman et al. 2016 Kannan & Vinyals) 2017 employs an evaluator trained to distinguish machine geririatd from aumann-writted texts, analogous to the discriminator in GANs (Goodfellow et al.] 2014). On this note, pre-traéned attribute classifiers and class-specific discriminators measure how well the generated samples match the conditioning labels on attributes such
sentiment, tense, voice, mood and negation (Logeswaran et al. 2018 Li et al.) 2017 Bruni & Fernández, 2017), guarantee the accuracy of stylistic text transfer (Zhang et al.. 2018c Shen et al. 2017), or are used to evaluate biases against cortain demographics and quantify model fairness in downstream settings (Cao ett al. 2022 Mathew et al., 2021: Kurila el al. 2019). GLEU (Napoles el al. 2015) was originally proposed for grammatical error correction, and later adopted for the evaluation of text style transfer since both tasks
roquire localized edits to the input sentence;; GLEU is found to present a reasonable balance between target style match and content retention (Sudhakar al. 2019).
Readability metrics such as Flesch-Kineaid Grade Level (Kincaid et al. 1975 and Flesch Reading Ease of a piece of text. Both metrics are computed as linear combinations of the number of words per sentence and number of syllables per word with different weighting factors. Although these metrics are fast and easy to compute, they should not be used on their own but in combination with metrics that capture the grammaticality and meaning preservation of the generated output (Wubben et al., 2012). In addition, they were not designed for measuring text readability in scientific or specialized domains, and are only available for the English language.
(Flesch 1979) are used to mcasure the reading
All constraints While automated evaluation helps assess generated texts quickly and cheaply, the use of aulomated evaluation metrics is dependent upon their correlation with human judgements of text quality [Fomicheva & Specia 2019). Human evaluations remain the gold-standard in natural language generation; automated evaluation metrics can be used as a proxy for human judgements only when there is reasonable correlation with human decisions. Ideally, automated evaluations are carried sinultaneously with human annotation studies, and not as replacement of human evaluations. In text style transfer, human evaluations are conducted to determine how accurately constrained text generation methods identify stylistic textual attributes in the source input and replace these with desired target attributes in generated sentences (Sud- hakar et al. 2019h. Tn conversations systems, responses generated by open-domain chatbots are evaluated across two dimensions: i) humanness, as a proxy for the fluency and coherence of the generated responses, and ii) attribute consistency, to determine whether the style and topic enforced by the generation model are well captured (Madotto et al.) 2020). Human evaluations are also carried to determine the plausability of the generated response, its content richness and how much new information it adds to the conversation (Bahoti let al.] 2018). Outputs generated by neural conversational systems are also assessed for quality, style and topic determine whether the acquisition of styles of famous personalities, characters, or professionals is achievable, and whether the conversational topic can be steered in particular directions (Wang et al. 2017).
Limitations of current evaluation metrics Given the wide diversity of evaluation paradigms, it be- comes challenging to objectively compare models and research progress when different evaluation metrics are employed in each work. By far, human-quality texts are considered the ground-truth for evaluating the output of NLG systems, serving human-quality texts and/or or soliciting human judgements of text quality is a costly and time-consuming process which requires careful design choices. Often times automated metrics that present reasonable cor- relation with human evaluations of text quality are used as a proxy for human judgements, however these motrics come with their own limitations. A common complaint is the lack of good ways to encode what constitutes human-quality output in an automated metric (Clark et al. (021). In addition, shortcomings of current evaluation metrics include poor correlations with human judgements, lack of interpretability of their
upper bound measure of their performance. However, collecting
22

---

Published in Transactions on Machine Learning Research (02/2025)
scores, the presence of complex biases in their evaluations, poor adaptability across tasks and inability to capture mancess(Dubers et al.) 2024a Khapra & Sai 2021). In what follows we discuss limitations of existing evaluation metrics, hoping to inform on the development of more robust evaluations for NLG systems.
Word Overlap metrics measure the lexical overlap between the model generated text and a set of human- written references. Metrics such as BLEU (Papineni et al. 2002), ROUGE (Lin 2004) and METEOR (Banerjee & Lavie, 2005) allow for fast and inexpensive development cycles and have been widely adopted for evaluating the output of natural language generation systems based on their correlation with human judgements at the time they were introduced, nevertheless their use is not without problems. On the one hand, the choice and quality of references is critical for improving the correlation between human and automated evaluation {Freitag et al. 2020). Current evaluation metrics are biased towards assigning higher scores to outputs that share a similar style with the reference, therefore collecting only a single style of references fails to reward systems that produce alternative but equally accurate outputs (Popović 2019); besides, collecting numannwritten references for new tasks is costly. On the other hand, these metrics assume that valid machine-generated responses present. a significant degree of overlap with ground-truth references; this is problematic for open-ended text generation tasks that require diversity and creativity (for eg., dialogue generation), and in such cases their correlation with human judgements is relatively low (Liu et al.) 2023c
2016 Graham et al. 2019 Sellam et al. 2020). For the evaluation of text simplification, BLEU presents weak or no correlation with grammaticality and meaning preservation for sentence splitting operations, therefore penalizing simpler sentences (Sulem et al., 2018). In addition, improvements in BLEU do not necessarily reflect an improvement in machine translation quality and there is a huge amount of variation for identically scored hypotheses (Callison-Burch et al. 2006) (i.e. wide variety of candidate outputs receive the same score when they present the same degree of overlap with the reference although they grcatly vary). Word overlap metries are also insufficient for measuring factual correctness of text sunmarization and fail to correlate with human judgements of factuality (Falke et al. 2019 Kryscinski et al., 2019 Pagnoni et al. 2021). Even more concerning is that the great majority of automated metrics, and in particular conventional reference-based metrics such as BLEU (Papineni et al., 2002) and CIDER (Vedantam et al.) 2015), are found to overrate machine-gent gencrne-generated text. over human--ritten text even though the machine text falls short of humans (Kasai et al., 2022).. In addition, BLEU and ROUGE fail to accurately measure content quality, capture syntactic errors and do not reflect the reliability of NLG systems (Reiter & Belz 2009 Stent et al., 2005 Using such evaluation metrics to compare systems may lead to drawing inaccurate conc conclusions, gives the false impression of progress and actively discourages the development of stronger generative models.
Since BLEU is based on n-gram precision, lexical differences between the hypothesis and references are aggressively penalized even when they are similar or synonymous to the reference. Given that no partial crodit is given if an n-gram does not exactly match a sub-sequence of the reference, BLEU is also hard to optimize due to the fact that learning objective is flat and cannot hill-elimb through intermediate hypotheses that have high semantic similarity synonymy, but low n-gram overlap (Wieting et al. 2019). Alternative metries based on word embeddings are easier to optimize as they output continuous values and capture fine- grained distinctions between similar outputs (Wicting et al. 2019). When used for mcasuring the quality of back-Iranslations for data augmentation, ion, BLEU only shows significant improvements for test examples if the source itself is a translation (Edunov et al. 2020); whenever references are translations and the source itself is natural tcxt, BLEU fails to capture human preference for source original sentences. While the use of multiple references substantially improves reference-based metrics, evaluations are often conducted using single human-written reference per instance; in such cases strong referenceless metrics frequently achieve higher correlation with human judgements (Rei et al.) 2020). Developing evaluation metrics that correlate well with human judgements on an instance level could serve to augment and validate human annotations.
To overcome the limitations of reference-based evaluation metrics, reference-free natural language evaluators are proposed (Fu et al.) 2024 Wang et al.) 2023a). Simultaneously, evaluating the quality of generated texts based on form-filling paradigm leverages large language models with of-in-os- (Liu et al., 2023c : given prompt that defines the evaluation task and desired evaluation criteria, the language model generates a chain-of-thoughtt with detailed evaluation instructions based which it will then score the generated lext according to the defined criteria. While LLM-based metrics seem to outperform reference- based and reference-fres evaluation metrics in terms of correlation with human judgements for open- open-ended
23

---

Published in Transactions on Machine Learning Research (02/2025)
and creative NLG tasks, they are very sensitive to the instructions and prompts given. Moreover, they tend to prefer I.M--eeneraeed texts over high quality human-written texts, which leads to biased predictions especially when used as reward signal for improving themselves. Using language models for "self-evaluation" indicates their predictions are well calibrated for token probabilites in-distribution, but they struggle with calibration in settings outside of the data distribution (Kadavath et al., 2022
Model-Based Evaluation metrics are becoming increasingly popular for NLG evaluation due to powerful representations learnt by pre-trained language models and high correlations with human judgements of text quality. However, current language models have well-known flaws and limitations, for example they assign high likelihood to degenerate texts, i.c. output that is bland, incoherent, or repetitive (Holtzman et al. 2020). can be insensitive to perturbations such as word order randomization (Pham et al. 2021), negation (Ettinger 2020) or named entity replacements (Balasubramanian et al., 2020), exploit superficial cues through the use of the self-attention mechanism (Pham et al. 2021) and exhibit naive understanding of the meaning of sentences without complex reasoning (Ribciro et al. 2020). To investigate the extent to which model basod evaluation metries suffer from the same limitations are used to complement human correlation tests and detect the blind spots of evaluation metrics (He et al., 2023) The authors construct a noised hypothesis set by applying different synthetic errors to ground-truth human-writter written references; if this noised hypothesis set is not scored worse than the original unperturbed set, it means the evaluation metric fails the corresponding stress test. Stress tests reveal that model-based evaluation metrics can be insensitive to errors at the start and middle of the generations when based on pre-trained models that do not encode long-range context, their judgement can be misled by simply injecting valueless text spans into the hypotheses, are biased towards frequent n-grams, present self-evaluation bias by unfairly ranking generations from their underlying base pre-trained language model higher than better quality generations from larger models, fail fluency tests (lemmatizing verbs, removing articles, prepositions or tokens at the end of the hypothesis) and consistency tests (sentence switching, replacement or negation). Complex biases, including a strong preference for longer outputs, are a common issue when relying on LLMs models to estimate response quality (Li et al.) 2023b Dubois et al. 2024a). While some model-based metrics perform better than others, is important to recognize their limitations and use each metric with awareness of its blind spots. To mitigate the risks of drawing inaccurate conclusions based on a single metric, using combinations of evaluation metrics that cover cach other's blind spots is recommended. For example, evaluation metrics based on pre-trained language models that encode long range context could be more robust to errors in the beginning or middle of the generations, valueless text span injections can be identified by word-overlap based metrics such as ROUGE Lin 2004), biases towards frequent n-grams can be detected by using diversity metrics, and truncation errors can be recognized via precision, recall and F1 scores. To mitigate unfair biases, is desirable to avoid using the same pre-trained language model for generation as well as base for the evaluation metric, or comparing different pre-trained models using an evaluation metric that relies one of these models. Finally, adding explainability on top of black- evaluation metrics can help identify system quality issues and increase trust in the evaluation of NLG systems (Leiter et al. 2022
black-box pre-trained language models, stress tests
Human Evaluation is considered the gold standard for the evaluation of NLG systems, however there is no consensus on how these human studies should be conducted (Gkalzia & Mahamood] 2015 van der Lee et al. 2019). The large variability in the design of human evaluations leads to difficulty in comparing results across different studics and also impacts the reliability of the inferred conclusions. The lack of consistency in human evaluation can be attributed to different factors such as the level of expertise of human annotators, their cognitive biases, ambiguity of the annotation task itself, or the actual wording of questions and instructions presented to participants ("how something is asked as opposed to what is asked") (Schoch et al. 2020). Untrained human evaluators may provide inconsistent results and contradictory reasons behind their judgments: "all that's human is not gold" &Clark et al., 2021]. Unsurprisingly selecting a different subset of annotators can lead to different conclusions due to variations in individual annotators' understanding of the annotation scheme (Amidei et al. 22020). In general, it is hard to decompose, interpret and validate crowdworker evaluations (Kasai et al.) (022). Depending on the evaluation setup, may be sensible to use qualified evaluators who have gone through extended training and can provide more reliable annotations. Moreover, improving the robustness and transparency of human evaluation guidelines is essential for increasing the reliability of human annotations. As the fluency of generated texts is improving,
24

---

Published in Transactions on Machine Learning Research (02/2025)
it is important to not only focus on surface-level aspects of text quality in human evaluations, but also to assess the informativeness and usefulness of generated texts in downstream settings (Clark et al. 2021
Future Outlook While so far we have reviewed limitations of existing evaluation metrics, we would also like to note the metrics that are missing or are rnder-represented in the literature, particularly metrics for measuring the trustworthiness, factuality, fairness, bias, toxicity, efficiency, diversity, uncertainty quantifica- tion, calibration and robustness of text information systems. In addition, is important for the community to focus on the interpretability aspect of evaluation metrics, particularly for model-based evaluations that currently function as black-box (Leiter et al. 2022). In the era of large language models, aspects such as knowledge, reasoning, memorization/ /copyright and disinformation are becoming increasingly important to quantify and analyze for NLG systems (Liang et al. 2023). Special attention also needs to be paid to existing datasets used to evaluate the generalization abilities of state-of-the-ar methods to ensure there is no overlap between the train set and the test set; in such cases, evaluations inadvertently mcasure memoriza- tion instcad of the model's ability to generalize, giving the false impression of improvements in performance (Elangovan et al. 2021 J. Large language models in particular are known to memorize parts of their training data, phenomenon which becomes more predominant with increasing the model capacity and the repetition of training examples (Carlini et al.)
2022 Razeghi et al. 2022). On top of this, given that LLMs are trained on web-scale datasets, evaluations are subject to potential data contamination issues (Wu et al.) 2024b; Therefore, interpreting evaluation results must. be done with caution accounting for the source pre-training data in determining to what extent current models generalize vs simply memorize training examples (Razeghi et al.] 2022); this also highlights the need to reconsider and redefine evaluation schemes for I.L.s, and focus on debiasing current evaluation metries (Li et al., 2023b; Dubois et al. 2024a). Finally, it is important to consider how advances in generative models can benefit and inform the development of more suitable evaluation techniques, and vice versa. Bidimensional leaderboards (Kasai et al. 2022) that simultaneously track progress in language generation models and evaluation metrics can bridge the gap between generation modeling and evaluation rescarch. As generation models contimue to improve, it is important to keep reassessing and updating evaluation metrics so that they accurately reflect the target objectives and correlate with human language use the real world (Zellers et al. 2021).
Dodge et al. 2021 Magar & Schwartz 2022
7 Constrained NLG Benchmarks and Datasets
Datasets that capture a wide diversity of constraints and are representative of many real world situations are critical for advancing safe and robust constrained text generation. Existing benchmarks focused on politeness (Madaan et al.] 2020), formality (Rao & Tetreault, 2018), sentiment (Shen et al., 2017), writing style (Jhamtani et al. 2017) are rather limited in nature and do not offer fine-graine control over stylistic attributes. StylePTB Lyu et al. 2021) aims to allow compositional transfer over a wider range of fine-grained stylistic constructs, including lexical, semantic, stylistic and thematic transfers.
CommonGen [Tin et al. 2020) benchmark proposes the task of constrained text generation with generative commonsense reasoning, where given a set of concepts the task is generate coherent sentence describing everyday scenario using the given concepts. To do this successfully, the generative model must reason over commonsense relations between the given concepts (relational reasoning), and infer novel combinations of familiar concepts (compositional generalization). Preliminary analysis shows that current state-of-the- art pre-trained models struggle at the task and generate implausible sentences by large margin. Other benchmarks proposed in the literature focus on avoiding model hallucinations and assessing the veracity and factuality of current models (Hendrycks et al. 2021 Bhakthavatsalam et al., 2021 Talmor et al., 2019). TruthfulQA (Lin et al., 2022) benchmark is proposed for measuring the factual accuracy and truthfulness of QA systems. Surprisingly, in their preliminary experiments the authors find that larger language models are less truthful than smaller language models from the same family, neverthleless they are more informative. RealToxicityPrompts (Gchman et al., 2020) aims to mcasure the extent to which toxic degeneration of large language models can be avoided, and the effectiveness of steering text generation algorithms away from producing racist, sexist and toxic content. Ideally, we want to have NLG models that are controllable, truthful, informative and perform well in the real world, however current pre-trained large language models can degenerate into toxic texts even from seemingly innocuous prompts. EiarructionFFolowwiii Eval (IFEval) (Zhou et. al. 2023a for large language models aims to measure to what extent L.M models can generate
25

---

Published in Transactions on Machine Learning Research (02/2025)
texts that satisfy specific lexical, format and utility constraints, such as for example "write in more than 400 words" and "mention the keyword of AT at least 3 times". However, performance of current models on existing benchmarks is not necessarily representative of their real-world performance. This issue is amplified by the use of biased automated evaluators, for example towards models that generate longer outputs (Li et al. 2023b; Dubois et al. 2024a). The research community not only needs belter evaluation metrics (as outlined in Section G), but also better benchmarks. Given the fragility of current NL. benchmarking practices, fallacious interpretations can be derived (Dehghani et al. 2021). To minimize the discrepancy between model performance on a given benchmark and its actual usefulness when deployed in real-life situations, benchmarks used for assessing the capabilities of current. NT.G systems should accurately reflect the end task of interest, as well as the wide diversity of scemarios and constraints encountered in practice. Motivated by the observation that new advances in melrics and models should more directly inform and benefit. each other, bidimensional leaderboards (Kasai et al. 2022) are proposed to track progress in both generative models and evaluation metrics for constrained text generation tasks such as machine translation, text summarization and image captioning. Follow (Jiang et al. 2024) evaluates the ability of LLM models to follow instructions with fine- ane-grained constraints. Single constraints are added incrementally to the initial instruction, allowing to estimate the difficulty level at which models fail to satisfy multiple consecutive constraints. Overall, state- of-the-art LLM models are limited to following instructions with at most several constraints, which illustrates the difficulty of the multiple-constraintt satisfaction problem and suggests there is significant potent ial for further improvement. In addition, only few instructions can be fully verified objectively and automatically, since edge cases make it hard to determine if an instruction is followed (Zhou et al. 2023a Using a diverse pool of atomic, verifiable instructions with constraints that are relevant real-world applications can help enhance the clarity and objectivity of the constrained NL. evaluation process on the proposed benchmarks.
Factuality benchmarks (Jacovi et al. 2025 Chen et al. 2023 Muhlgay et al., 2024 Iqbal et al., 2024) aim to evaluate the factual accuracy of the generated responses in information-seekinn scenarios. Enhancing LLM factuality requires finding delicate balance as it can compromise other desirable attributes, for example creativity and novelty. Factuality is expected to remain a research challenge for the forcsccable future, particularly in long-form lext generation lasks. Cognac (Chen el al. 2022) measures whether LLMs conform to lexical level constraints by guiding models on what topics to generate, while also imposing knowledge- intensive constraints what aspects the model should not to generate. Prompt-based approaches show a lot of promise in steering instruction-tuned LLM models away from generic outputs for stylistic tasks, but tend to perform less well for lexical and format constraints (Ashok & Poczos 2024
Sun et al. 2023a
A larger issue in terms of natural language evaluation is the gap between how humans use language in the real world, and what current benchmarks can mcasure (Zellers et al. 2021]. In addition, many datasets are not an effective indicator of model generalization and real world performance, particularly in the presence of overlap between the train and test sets, leading to inflated evaluation results (Elangovan et al.] 2021; datasets used to train large language models are often
Dodge et al. 2021). Besides, since massive "contaminated" with downstream test sets, it is important to conduct in-depth analyses to disentangle gen- uine progress in natural language understanding/ generalization from rote memorization (Magar & Schwartz, 2022); overlooking the impact of pre-training data can result in misleading interpretations of model perfor- mance (Razeghi et al. 2022). Finally, we would like to draw attention on the lack of resources (datasets and evaluation metrics) for many languages other than English.
In summary, we outline below the reasons behind the mismatch between constrained NLG benchmarks and real-world use case scenarios:
LLM models are trained on web-scale datasets with minimal curation to ensure data quality. Many pre-training datasets also contain various evaluation benchmarks that are used for assessing and comparing trained models. Due to these factors, interpreting evaluation results must be done with caution accounting for the source pre-training data in determining to what extent current models generalize vs simply memorize training examples. Auditing LLM models for test set contamination via statistical significance tests reveals verbatim contamination, i.e. LLM models are trained directly on the test sets they are evaluated on (Oren et al.] 2024). Bocause of this, LLM performance is likely overestimated on existing benchmarks and their behavior in practice is likely much inferior to reported results. In the context of constrained text generation, even if part of the benchmarks
26

---

Published in Transactions on Machine Learning Research (02/2025)
is revealed in training, it will significantly simplify the task the constraints are no longer only testable on the model outputs (in other words, the model can find a shortcut by mimicking part of the training data that already satisfies these constraints). In addition, the inflated evaluation results on benchmarks are not an effective indicator of model gencralization abilitics and their real world performance. Indeed, recent. work shows that emergent abilities of LLMs on many benchmarks are a mirage (Schaeffer et al., 2024). appearing due the rescarcher's choice of evaluation metric rather than due fundamental changes in model behavior with scale.
Current benchmarks are not representative of the diversity of real-world use cases and constraints, and often only have limited data coverage they are evaluating models on. Certain tasks such question-answering are overly used in bechmarking LLMs. Due to lack of holistic evaluations, it may appear that LLMs perform well in controlled environments, however they fail in critical real- world applications and constraints, posing safety risks such as perpetuating bias, making unsafe decisions, or being vulnerable to manipulation and adversarial attacks (Williams et al. 2024 Dong et al. 2024 Shayegani et al. 2023). In addition, recent analysis of stale-of-the-art LLM benchmarks finds that they suffer from significant limitations, including biases, difficulties in measuring gen- uine reasoning, adaptaablitty, implementation inconsistencies, prompt engineering complexity, lack of evaluator diversity, and the overlooking of cultural and ideological norms (Mclntosh et al. 2024).
Typically, benchmarks evaluate model performance across one single dimension and summarize re- sults in the form of a single scalar value; this not only offers an incomplete picture of the model performance, but is also misleading when used to compare across models and rank model submis- sions. Besides, single-value benchmarks can often lead to "reward-hacking" and exploiting spurious features, such as annotators' preference for more verbose responses (Sorensen et al. 2024).
Going beyond monistic benchmarks that measure model performance on single target objective, it is important to evaluate model performance on multiple constraints. Pluralistic benchmarks with more than one target objective to maximize aim to capture the entire spectrum of model performance across different attributes, making it feasible to compare and rank diffcrent models across multiple dimensions (Sorensen el al., 2024)) Pluralistic benchmarks can be calegorized into: i) malti-objecttive benchmarks, reporting evaluations across all objectives for all solutions; ii) trade- off stecrable benchmarks, designed to mcasure stecrability of models and encourage models to trade off between different objectives at inference time, and fit) bury-pluralistic benchmarks which model diverse human ratings, and allow to explicitly reason over which users or groups models are being aligned to for more fair outcomes. In our view, multi-objective benchmarks and trade-off steerable benchmarks are particularly important for further advancing multi-objective constrained NLG.
In general, there is lack of research consensus on how to properly benchmark models and measure scientific progress. In-depth analysis of inadequacies of LLM benchmarks reveals significant limitations, including bi- ases, difficulties in measuring genuine reasoning, lack of adaptability, implementation inconsistencies, prompt engineering complexity, limited evaluator diversity, overlooking cultural and ideological norms (McIntosh) et al. 2024). Moving away from evaluations on static benchmarks to dynamic behavioral profiling, adopting standardized methodologics, regulatory certaintics and ethical guidelines should be prioritized. We encour- age more research in these directions to bridge the gap between current constrained NLG evaluations and the model performance in real-world settings.
8 Discussion
In what follows we summarize the main challenges for constrained NLG and outline open problems, then we present the most promising research directions in the authors' opinion for advancing the state-of-the-art for safe and reliable constrained NLG.
8.1 Open Challenges
In our view, constrained text generation is a more difficult problem compared other instances of text gen- eration. The difficulty arises from multitude of factors, including lack of model expressiveness which makes
27

---

Published in Transactions on Machine Learning Research (02/2025)
it difficult for current models to incorporate constraints into the objective function, lack of suitable evalua- tion metrics to assess the extent to which constraints are satisfied (which becomes even more challenging in the presence of multiple constraints), difficulty in the constrained optimization of non-difeerentiable reward functions, and finally lack of constrained text generation datascts that are illustrative of a wide diversity of constraints. Due to these pressing issues, constrained lext generation remains an open challenge in the research community. Advancing the statc-of-the-art requires considerable collective and focused effort.
Multiple constraint satisfaction Most approaches proposed for constrained text satisfaction focus generating sentences that meet one single desired constraint, nevertheless generating sequences that simulta- neously satisfy multiple lexical constraints is an important open research problem in text generative models (Liu et al. 2019a; Latif et al., 2020 Hsieh et al. 2021). While incorporating one constraint is already hard enough due to lack of model expressiveness, incorporating multiple constraints poses significant challenges in terms of defining the loss function accounting for all the desired constraints, difficulty in optimizing it and evaluating whether cach constraint is satisfied. Approaches that convert the multiple constraint sat- isfaction problem into allowing the inclusion of pre-specified lexical constraints at decoding time are not optimal either: on the one hand, decoding complexity increases exponentially or linearly in the number of constraints, and on the other hand forcing constraints at every step of the generation process impacts the quality and naturalness of generated texts (Post & Vilar 2018). Moreover, many model architectures are designed for sequential sentence generation only (vs. non-monotonic text generation) and it is non-trivial to impose decoding time constraints while maintaining optimal text generation quality (Miao et al. 2019).
Prompting methods are used to evaluate to what extent multiple constraints are satisfied by instruction- tuned LLM models (Jiang et al. 2024). Single constraints added to instructions in incremental fashion, allowing to estimate the upper limit of instruction following capabilities in LLMs and assess the difficulty level at which models fail to follow instructions with multiple constraints, Overall, the more constraints are added to an instruction, the more rapid the decrease in performance of state-of-the-ar LLM models; on average, at most three constraints are satisfied. Constraints such as role-playing, reasoning in complex situations, numerical planning, suggestion generation, recognizing and following patterns are identified as the most difficult constraints to satisfy. Prompting with multiple, verifiable, fine-grained constraints is proposed for assossing discropancies/ ceapppanccis/misundddii misiddnstttaddiiggn in following instructions that lead to unintended outputs (Zhoul et al. 2023a Sun et al. 2023a). Nevertheless, making constrained NLG evaluations relevant to real-world applications is crucial for improving the reliability of conclusions drawn from current benchmark evaluations.
Dynamically defined constraints Current approaches to constrained text generation assume there is prior knowledge of the constrained textual attributes and the finite set of values these attributes can take. Nevertheless, there are situations when it may be desirable to impose constraints dynamically, for eg. in con- versational systems depending on the system user's statements, reactions and emotions. When dynamically defining constraints, the main challenges are the lack of model expressiveness and robust ways to evaluate whether these constraints are satisfied. In the literature, controling the realization of sentence based on another's 's sentence syntax and semantics is less explored sctting for constrained text generation with dy- namic constraints which does not require prior knowledge of all the values the control variable might take on CChn et al.) 2019). Disentangled latent space representations of syntax and semantics are essential for the manipulation sentence attributes in tasks such as unsupervised paraphrase generation and syntax-transfer generation (Bao et al., 2219)). In the context of LLM models, handling dynamic and complex application constraints remains challenging and relatively under-explored area of research. Commonly used solutions for incorporating dynamic constraints leverage the reasoning and planing capabilities of LLM models via strate- gics such as model fine-tuning and reflection- basod reasoning, for example Chain Chain-of-Thought (Wei et al. 2022), Tree-of-Thoughy (Yao et al., 2024 or Self-Play fine- toning (Chen et al.) 2024b). Nevertheless, these approaches typically address constraints on a case-by-case basis, limiting their generalizability. Open ques- tions are how to represent constraints for LLMs effectively, guide them to reason within those constraints, and accurately assess the correctness or fallacies in their reasoning process (Wei et al. 2024).
Generative reasoning Current large-scale text generation models display impressive ability to generate fluent texts, nevertheless composing realistically plausible sentences in the presence of constraints remains a significant open challenge. This is illustrative of all challenges associated with constrained text generation, including lack of model expressiveness, lack of suitable evaluation metrics, difficulty in constrained opti-
28

---

Published in Transactions on Machine Learning Research (02/2025)
mization and lack of constrained text generation datasets. Endowing generative models with commonsense reasoning abilities is an important milestone lowards advancing machine understanding and intelligence. In general, the great majority of models proposed in the literature only exploit superficial cues via self-attention to solve NLP tasks, without relying on syntactic information or complex reasoning (Pham et al. 2021 h.
LLM models trained with reinforcement learning can perform complex reasoning tasks by thinking through a problem, producing series of intermediate reasoning steps that allow to recognize and correct mistakes before attempting to give the final answer. Prompt-based reasoning with LLMs has lead to rapid advancements many constrained NLG tasks, including mathematical, logical and commonsense reasoning problems, code generation, question answering, text summarization, machine translation, etc. (Plaat et al. 2024 Liu et al. 2025a)). Despite the strong performance of LLMs on certain reasoning tasks, the extent to which LLMs are actually capable of reasoning and handling complex deductive problems (vs. exploiting superficial cues and
shallow patterns the data) remains uncertain (Huang & Chang 2023 Hosseini et al., 2024). Compared to human performance, LLMs multi-step logical and commonsense reasoning performance is lagging behind humans in various real-world domains such as murder mysteries, object placement or leam assignment (Sprague et al., 2024). Significant decline in LLM reasoning performance is reported as the complexity of constraint satisfaction problems increases, a phenomenon referred to as "the curse of complexity for reasoning" (Lin et al. 2025). Scaling up the number of reasoning tokens generated during inference, and ensuring the correctness and verifiability of the reasoning chain may alleviate the issue to some extent.
Attribute specific datasets The lack of annotated datasets for attribute specific text generation constitutes a bottleneck in the development and adaptation of models for tasks that require fine- ane-grained control over style and topics. For example, in dialogue systems the absence of attribute annotated conversational datasets that can be used for fine-tuning large scale pre-trained models limits control over the generated responses for a desired attribute (Madotto et al. 2020). Moreover, such attribute annotated datasets can help with the personalization of dialogue systems, make dialogues safe, supportive and engaging (Serban et al. 2015 Zhang et al., 2018a Ge et al., 2024b. Personalized dialoguc agents that display consistent personalitics and viewpoints overcome the ussatislying experience of a persona-free chit-chat model and empower practical applications such as personalized conversations. Nevertheless, imposing conversational goals on a dialogue agent for learning target-guided strategies requires keyword-angmented conversation datasets for lcarning how to steer the conversation towards a designated target subject (Tang et al. 2019
Rule constraints While most research that is currently trying to address constrained text generation is focusing on the incorporation of pre-defined utility or lexical constraints to various degrees of success on simple tasks with narrow scope (Ashok & Poczos 2024 Sun et al.) 2023a Zhang et al. 2023), the satisfaction of rule based constraints is equally relevant, particularly when used to define format and syntactic conditions on the output. However, the lack of model expressiveness makes it challenging
incorporate rule based constraints into the loss function at training time. We encourage more effort in this direction likely to open plethora of new possibilities in how constraints are specified, incorporated and satisfied in models particularly designed for constrained neural text generation.
Prompt-based control for lexically constrained text generation remains challenging for the following reasons: i) current LLM models tend to display position bias (Liu et al. 2024c), and only satisfy lexical constraints that appear within specific positions in the input. äi) decoding parameters lack of sensitivity to incorporate lexical constraints, and üi) complesity of compound word constraints, which are often misinterpreted or altered in meaning by LLM models. Overall, LLMs struggle to adapt to increasingly complex lexical constraints with prompt-based control (Li et al.) 2024a]; the more lexical constraints are added to the prompt, the more significant the decrease in performance of LLM models for constrained text generation (Jiang et al. 2024). In addition, there is an inherent trade-off between generating text of high quality and satisfying hard constraints [so] 2024). LLMs often struggle to meet fine-grained hard constraints (Sun et al. 2023a), and evaluations on out-o--distribution constraints aiming to differentiate constraint-followiii abilities from over- fitting (IFEval (Zhou et al., 2023a) vs. IFEval-OOD (Lambert et al. 2024)) report that a lot of the claimed success of prompts targeted for constrained instruction following may be simply attributed to overfitting.
Evaluation of constrained text generation In general, evaluation of text generative models is open challenge. The field is missing robust automated evaluation metrics that correlate with human judgements
29

---

Published in Transactions on Machine Learning Research (02/2025)
across multiple dimensions of text quality. Evaluation of models for constrained text generation is currently done using the same flawed existing metrics commonly used in unconditional and conditional text generation evaluation, or in an informal way often times in the absence of a rigorous evaluation procedure. Human evaluation remains the gold standard way to assess text quality, however designing evaluation metrics tailored specifically al assessing whether generated lexts meet desired constraints altogether with new benchmark datasets for the evaluation of constrained text generation are important next steps (Latif et al., 2020: Ruan
et al. 2024 Chen & Wan 2023:
Zhou et al., 2022a Hu et al., 2024).
Adversarial Attacks Adversarial examples exploit vulnerabilities in text generation models and represent an active rescarch area. Adversarial triggers in the form of input-agnostic sequences of tokens concatenated to any input dataset can trigger a pre-trained language models to produce biased, racist and discriminatory outputs even when these models are carefully fine-tuned and optimized against adversarial triggers (Wallace ct al. 2019). Gradient- based adversarial trigger phrase search techniques are used to generate input prompts to a pre-language model that induce biases in the generated output and allows to study strategies for bias miligation (Sheng al.) 2020). Constrained text generation models that are robust to adversarial atlckk are needed for the beneficial use of machine learning and artificial intelligence technology in real world applications, as well as to mitigate any potential socictal harms and biases associated with the deployment of large language models (Chowdhury et al. 2024 Gallegos et al. 2024).
Other important open challenges include the use of constrained text generation for personalized agents in a wide variety of contexts (Zhang et al., 2024b Liu et al.) (022b), such as in dialogue settings (Zhang et al. 2018a), and new benchmark datasets that are reflective of worllworld constraints for both training/ fine-tuning
and evaluating constrained text generation models (Ziyu et al. 2023 McIntosh et al. 2024 Xu et al. 2024)
8.2 Promising Research Directions for Advancing Constrained NLG
Despite the many open challenges, we believe there are promising approaches in the literature that merit special attention for advancing constrained NLG. We present these below.
Reinforcement Learning from Human Feedback (RLHF) is the predominant paradigm for aligning LLMs to human preferences given helpfulness, harmless and safety constraints (Ouyang et al. 2022). h. RLHF performance is strongly dependent upon the quality of the reward model, and defining rewards for real-world tasks, especially with the presence of constraints, is non arivill Reward functions are fragile and notoriously difficult to specify, particularly for tasks with complex goals (MeKinney et al., 2023). An outstanding RLHF challenge is the issue of reward hacking, where LLM policies learn to exploit failures of the reward model and achieve seemingly high rewards without meeting the underlying objectives (Rame et al. 2024b).
Despite these challenges, RLHF can play a key role in advancing constrained NLG, conditioned on the design of more robust and reliable reward functions. To mitigate reward hacking in particular, it is necessary to have reward models that can reliably score generations despite distribution shifts, are robust to label noise and inconsistencies in human preferences. Preliminary approaches that explore prediction ensembling (Christiano et al.) 2017) or weight averaging (Rame et al., 2024b) of multiple reward models are designed to act regularization preserving only those mechanisms that are invariant across runs, helping reduce reliance on spurious features and memorization of corrupt/note
training examples. While these approaches may help delay reward hacking to some extent, they do not fully solve the problems of reliability under distribution shifts and robustness to noisy labels. We believe there is a lot of space to explore more efficient solutions for training robust reward models for real-world constraints that accurately reflect human preferences, entirely prevent reward hacking (instead of just delaying it), and generalize to oft-t--distributiin scttings.
RL policies that perform well for diverse reward functions (not just one reward model) can accommodate diverse user preferences and advance NLG with multiple constraints. Multi-objective reinforcement learning (MORL) algorithms can be used to learn Parcto-optimal policies and control the learnt policics accounting for multiple objectives (Liang et al. 2024). For example, multi-objective reward modeling allows to dynamically control the trade-off between diverse user preferences via arithmetic operations in the vector reward space (Wang et al. 2024a). However, in many real-world scttings it may be difficult to accurately specify constraints mathematically; in such situations, it may be possible to learn constraints directly from user provided demonstrations, even when the reward function is unknown [Lindner et al. 2024 Malik et al. 2021).
30

---

Published in Transactions on Machine Learning Research (02/2025)
Mechanistic Interpretability Recent works aiming to understand the inner workings of LLMs find that these models have internal representations that encode concepts in disentangled manner. Tf one can identify which part of the representation subspace corresponds to a given concept, then it its possible manipulate the concepts expressed by the model through algebraic manipulation of the representation (Wang et al. 2024c). The linear representation hypothesis (Park el al., 20244] posits that high-level concepts are represented lincarly as dircctions in the representation space. Assuming it is possible to identify these lincar concept representations, linear algebraic operations can be performed on the representation space for fine-grained control of LLM outputs. Interventions using steering vectors that change the value concept takes without changing other concepts show that in carefully designed test cases it is possible to change, for eg- the output from English to French by adding a suitable English//rren French steering vector. If the linear representation hypothesis were to hold true, this could potentially open up new methods that advance constrained NLG. Open problems are the identifiability of learned representations, to what extent they capture real-world structure, and what assumptions need to be made about the geometry of the representation space.
Causal Interventions/ / Causal Probing / Inference Time Interventions Recent work shows that it is possible to perform direct manipulation of computational mechanisms inside LLMs. In particular, causal tracing approaches first identify neuron activations corresponding to particular concept, then edit the corresponding weights to change model outputs in a desirable way (Meng et al. 2022 Meng et a..). However, the connection between causality- localization and model editing is still unclear, as localization performed by causal tracing is not indicative of which layer to select for model editing (Hase et al. 2024). Developing more reliable causal mechanisms for localizing where knowledge stored inside a neural network, well as robust ways to edit. the internal knowledge of LLM models to ensure generated outputs satisfy given constraints, can pave the way for advancing constrained NLG.
Causal probing can be used to analyze how intervening on latent properties of the model's representation have an impact on the model outputs (Canby et al.) 2024). For example, linear probing has been used to prodict whether the answer will be truthful or not before it is actually generated (Joshi et al. 2024). While measuring the effectiveness of probing interventions in LLM models is an open research area, causal probing is a promising direction to explore whether constraints are present in the generated output before generation even begins. Inference-time intervention (ITT) (Li et al. 2024c) locates directions that correspond to specific concepts (for eg., truthfulness) and shifts model activations along these directions at inference time. Potentially integrating ITTI with causal probing and intervention mechanisms, and a better understanding of the geometry of the representation space, could help enforce complex prributes//onstrainne in the output.
Reasoning-based approaches / Constrained Decoding approaches / Rejeecion Sampling Graph- constrained reasoning (Luo et al., (024)) aims to connect the unstructured reasoning in L.Ms with the structured knowledge found in knowledge graphs (KG); the model constrains the LLM decoding process to reasoning paths that encode KG information. Generating KG-grounded paths helps alleviate reasoning issues due to lack of knowledge, mitigates hallucinations and enhances the faithfulness of generated responses. Constrained chains of rcasoning (Lin et al.] 2024b) leverage domain knowledge and the causal relations between concepts to construct reasoning chains that improve consistency of the generated responses.
Constrained decoding approaches enforce adherence to constraints during generation, while (ideally) mini- mally intervening on other non-target aspects to avoid misalignment (Beurer-Kellner et al.] 2024). Guided decoding (Lu et al.] 2021) employs an auxiliary evaluation function that captures to what extent partial outputs satisfy given goals; the method can be combined with search algorithms to generate outputs that satisfy specific constraints. Combining inference-time search algorithms such Monte-Carlo Tree Search (MCTS) with RLHF fine-tuned models (Liu et al. 2024a) demonstrates that valuo-guided decoding with MCTS is a crucial component for achieving model steerability and constraint satisfaction (for eg., senti- ment steering, toxicity reduction, helpful and harmless chatbots). Value models trained as byproducts when aligning LLMs to human preferences have only recently been employed as evaluation functions for scoring partial/incomplet sequences and steering LLM models; it would be interesting to use MCTS as policy optimization operator to search for contrained sequences with high rewards. Overall, we believe that com- bining reasoning- based approaches with guided decoding and MCTS has lot of potential to improve the state-of-th state-of-the-art for constrained NLG, including more faithful, consistent generations and less hallucinations.
31

---

Published in Transactions on Machine Learning Research (02/2025)
Statistical rejection sampling can be used to discard partial samples that do not meet given constraints; this technique has been widely employed in LLAMA-3 family of models (Dubey et al. (024), and is found to improve the alignment with human preferences in constrained optimization settings (Liu et al.) 2024d). Best-of-N rejection sampling (Stiennon et al., 2020) draws n samples from the LLM, ranks them on the target attribule of interest, and returns the best sample. Despite its simplicity, this strategy has been found to be surprisingly effective in practice, however Best-of-N sampling comes at considerable inference cost. Approaches trained to mimie this distribution achieve high win-rates while minimally affecting other off- off-target aspects of the generation (Gui et al. 2024), allowing for better control of LLM models.
Improving model architectures The mismatch between how LLM models are trained with a next-word prediction objective and how they are used in practice (for eg,, for long-term open-ended dialogue with users) leads to inconsistency in their behaviour over long horizons. The Transformer attention mechanism decays over long exchanges, causing chatbots to stray away from prompted behaviour and resulting in instruction drift that degrades the quality of the outputs over lengthy dialogues cLi al.). Improving instruction stability in LLMs, particularly in long-form conversat ions, can lead to more stable and robust. prompting, improve the performance of current models and their abilities to generate texts accounting for given constraints. There is a need for better understanding how LLMs use input context and potentially come up with novel designs for attention mechanisms that more robustly capture information within long input contexts.
Autoregressive LLMs fail to generalize in surprising ways. The reversal curse (Berglund et al. 2024 is one instance where a model trained on sentence of the form "A is B" will not automatically generalize to the reverse direction "B is A", therefore failing to deduce the reverse relationship this directly impacts the LLM models' ability to generate constrained texts. Further analysis of LLMs generalization using influence functions Grosse et al. 2023) finds that training examples that match the order ("A precedes B") are far more influential than examples with reverse order ("B precedes A"). Fine-tuning and data augmentation approaches are used to alleviate the issue, however it does point to a basic failure of logical deduction in LLMs training. Incorporating logical doduction and causal-based rcasoning during LLM training could help.
Accounting for constraints / Better guidance during training Classifier guidance (Dhariwal & Nichol 2021) introduces an extra trained classifier to guide diffusion model generations in particular desirable directions. Methods aiming to improve this approach propose classificr-fres guidance (Ho & Salimans 2022) showing it is possible to steer LLMs using pure generative model; a conditional and an uncondil it ional diffusion model are jointly trained, and their score estimates are combined to achieve fine-grained control over the generated outputs. There is a lot of potential to extend the use of such approaches beyond the vision domain to the text domain for more robust constrained NLG.
Multi-objective / pluralistic benchmarks Aligning LLM models to pluralistic human values requires the capability to accurately steer models in directions representing a diverse set of human values and per- spectives (Sorensen et al.] 2024). Steerable pluralism, i.e. faithfully steering LLMs to represent particular attributes or perspectives, plays key role in personalizing and customizing models to varions users or tar- get populations. Nevertheless, current benchmarks for constrained NLG are monistic, focusing on a single objective. Pluralistic benchmarks have more than one objective to maximize (each objective is measured separately) and allow for more explicit trade-offs between constraints at inference time.
Hybrid Auman-AI Collaborative Approaches Intoractive systems that allow LLM models and their end users to write collaboratively have the potential to enhance constrained text. generation by empowering users to choose which (partial) model generations are in line with their needs and expectations. CoAuthor (Loo et al. 2022) proposes a collaborative human-Al approach for text generation where a writer and a model take turns interactively in writing story and editing it. Such methods empower users with fine-grained control over the model outputs and open up new opportunities in assistive writing in a steerable fashion. Causal inference can play a key role in modeling the human-Al collaboration byanswering counterfactual "what-if" questions on how the outcome of the collaboration would change if humans employed different text teettinng/refnemenl strategy. For example, causal estimands (Tncremental Stylistic Effect. (Zhang et al., 2024al can be used to mcasure the average impact of infinitesimally shifting text towards specific style, such as increasing the degree of politeness or formality.
32

---

Published in Transactions on Machine Learning Research (02/2025)
9 Conclusion
In this work, we have presented the reasons why constrained natural language generation is an important, yet highly challenging and largely unsolved research problem. Our first contribution consists in clarify- ing the difference between the ambignous use of unconditional, conditional and constrained terms in the natural language generation literature, and draw clear boundaries between these concepts by exemplifying instances of natural language generation tasks with their associated conditions and constraints. Among dif- ferent paradigms of text. generation, we consider constrained text generation to be particularly challenging (if not the most challenging), yet also extremely useful. We identify general reasons why constrained nat- ural language generation deserves significant more attention in the research community, including the lack of model expressiveness in incorporating constraints into the objective function at training time, difficulty in constrained optimization algorithms, the lack of suitable evaluation metrics for robustly assessing, com- paring model outputs and claiming success in constrained natural language generation, as well as the lack of constrained lext generation dalasets/benchnnan benchhmark that are represenlative of a wide range of real-world constraints for training, fine-tuning and evaluating these models. We then survey representative body of recent literature on constrained text generation using neural networks, presenting the main approaches and methods used, as well as their limitations. Our work serves as an informative guide for both researchers and practitioners to become familiar with the current methodology and main challenges, in the hope of advancing state-of-the-at constrained NLG. We invite future work in solving the outlined challenges for better, useful, safer and more robust constrained natural language generation and evaluation.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint Xrv: 08774, 2023.
Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. The cringe loss: Learning what language not to model. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Papers), pp. 8854 8874, 2023.
Jacopo Amidei, Paul Piwek, and Alistair Willis. Identifying annotator bias: A new irt-based method for bias identification. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4787-4797, 2020.
Afra Amini, Tim Vieira, and Ryan Cotterell. Variational best-of-n alignment. arXie preprint arXiv:407.000577, 2024.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 936 945, 2017.
Dhananjay Ashok and Barnabas Poczos. Controllable text generation in the instruction-tuming era. arXiv preprint Xre:224405 2024.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93 104, 2022.
Kevin Bache, David Newman, and Padhraic Smyth. Text-based measures of document diversity. In Proceed- ings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 23-31. ACM, 2013.
33

---

Published in Transactions on Machine Learning Research (02/2025)
Ashutosh Baheti, Alan Ritter, Jiwei Li, and William B Dolan. Generating more interesting responses in neural conversation models with distributional constraints. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 3970 3980, 2018.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Luey Campbell-Gillingharl Jan Bal- aguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38176-38189) 2022.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. Constrained decoding for neural nlg from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 831 844, 2019.
Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhiject Awasthi, and Sunita Sarawagi. What's in name? are bert named entily representations just good for any other name? In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 205-2122 225-214, 2020.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved corre- lation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine tronslation and/or or summarization, pp. 65-72, 2005.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annaull Meeting of the Association for Compulational Linguislies, pp. 6008 6019, 2019.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistie language model. Journal of machine learning research, 3(Feb):1137-1115) 2003.
Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024.
Luca Beurer-Kellner Marc Fischer, and Martin Vechev. Guiding lims the right way: fast, non-invasive constrained generation. In Proceedings of the 41st International Conference on Machine Learning, pp. 0058-3673, 2024.
Sumithra Bhakthavatsslam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer ques- tion answering? try are-da, the direct-answer ai2 reasoning challenge. arXiv preprint wxXi::2102.03355, 2021.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-60 15- 2019.
Antoine Bosselut, Asli Çelikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse- aware neural rewards for coherent text generation. In NAACL-HLT, 2018.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Elia Bruni and Raquel Fernández. Adversarial evaluation for open-domain dialogue generation. In Proceedings of the 18th Annual SIGdial Meeting Discourse and Dinlogue, pp. 284 284-288, 2017.
34

---

Published in Transactions on Machine Learning Research (02/2025)
Chris Callison-Buree Miles Osborne, and Philipp Kochn. Re-evaluation the role of bleu in machine transla- tion research. Tn 1 11th Conference of the European Chapter of the Associalion for Compulational Linguis- tics, 2006.
Mare Canby, Adam Davies, Chirag Rastogi, and Julia Hockenmaier. Measuring the reliability of causal probing methods: Tradcoffs, limitations, and the plight of mullifying interventions. ar preprint arXiv:2408. 15510, 2024.
Yang Trista Cao, Yada Pruksachatkum, Kai-Wa Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language rep- resentations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 561-570, 2022.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. Faithful to the original: Fact aware neural abstractive summarization. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 643 643-653, 2018
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. Kermit: Generative insertion-based modeling for sequences. aixiv preprint arxiv:1906 100600006444 2019.
Howard Chen, Huihan Li, Danqi Chen, and Karthik Narasimhan. Controllable text generation with language constraints. arXiw preprint 2r. 10466 2022.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In International Conference Machine Learning, pp. 7935-795 79522 PMLR, 2024a.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5972-5984, 2019.
Qian Chen, Xiaodan Zhu, Zue Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced Istm for natural language inference. In Proceedings of the 55th Anruall Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1657-1668, 2017.
Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. Felm: benchmarking factuality evaluation of large language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 44502-44523, 2023.
Xiang Chen and Xiaojun Wan. A comprchensive evaluation of constrained text generation for large language models. arxiv preprint or arXiv:2310. 163443 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In International Conference on Machine Learning, pp. 6621 6642. PMLR, 2024b.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machinc translation. In International Conference on Learning Representations, 2020.
Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vinija Jain, and Aman Chadha. Breaking down the defenses: A comparative survey of attacks on large language models. arXiv preprint ar Xiv:22400.00 047804 2024.
35

---

Published in Transactions on Machine Learning Research (02/2025)
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. Tn Advances in Neural Information Processing Systems, pp. 4299 4307, 2017.
Elizabeth Clark, Yangfeng Ji, and Noah A Smith. Neural text generation in stories using entity representa- tions as context. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, Volume 1 (Long Papers); pp. 2250 2200, 2018.
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2748-2222 2760, 2019.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that's 'human'is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7282 7296, 2021.
Roi Cohon, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effeets of knowl- edge editing in language models. Tronsactions of the Association for Computational Linguistics, 12: 283 29-, 2024.
Pierre Jean A Colombo, Chloé Clavel, and Pablo Piantanida. Infolm: A new metric to evaluate summariza- tion & data2text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10554-10562, 2022.
Josep Crego, Jungi Kim, Guillaume Klein, Anabel Rebollo, Kathy Yang, Jean Senellart, Egor Akhanov, Patrice Brunelle, Aurelien Coquard, Yongchao Deng, et al. Systran's pure neural machine translation systems. arxiv preprint ar Xix::6100005500, 2016.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493 (883-850,,42 2022.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosaune Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020.
Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107. 07002, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Tn Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics; luman language technologies, volume 1 (long and short popers), pp. 4171 4186, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ihaarco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 1286 1005, 2021.
Chris Donahuc, Mina Lee, and Percy Liang. Enabling language models to fill the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2492-2501. Association for Computational Linguistics, 2020.
36

---

Published in Transactions on Machine Learning Research (02/2025)
Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute con- ditioned sft as an (user-steerable) alternative to rlhf. In Findings of the Associalion for Compulational Linguistics: EMNLP 2023, pp. 11275 11288, 2023.
Zhichen Dong, Zhanhui Zhou, Chao Yang. Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for 1lm conversation safety: A survey, In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024.
pp.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404004445, 2024a.
Yann Dubois, Chen Xucchen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Sergey Edunov, Myle Out Marc' Aurelio Ranzato, and Michael Auli. On the evaluation of machine translation systems trained with back-translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2836 2846, 2020.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. Memorization vs. generalization: Quantifying data leakage in ulp performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1325 1335, 2021.
Allyson Etlinger. What bert is not: Lessons from a new suile of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. Ranking gener- ated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 2214- 2019.
Angela Fan, David Grangier, and Michael Auli Controllablc abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 45 54, 2018a.
Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical neural story generation. In Proceedings of the 56th Annual Meeling of the Association for Compulational Lánguistics (Volume 1: Long Papers); pp. 389-898. 2018b.
Angela Fan, Mike Lewis, and Yann Dauphin. Strategies for structuring story generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2650 2660, 2019.
William Fedus, Tan Goodfellow, and Andrew M Dai. Maskgan: Better text generation via filling the In International Conference on Learning Representations, 2018.
Jessica Fieler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. EMNLP 2017, pp. 94, 2017.
Rudolf Franz Flesch. How to write plain English: A book for lawyers and consumers. Harpercollins, 1979.
Marina Fomicheva and Lucia Specia. Taking mt evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515-558, 2019.
Markus Freitag, David Grangier, and Isaac Caswell. Bleu might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 61 71, 2020.
37

---

Published in Transactions on Machine Learning Research (02/2025)
Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapler of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6556 6576, 2024.
Yao Fu and Yansong Feng. Natural answer generation with heterogeneous memory. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers); pp. 185-195, 2018.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Isabel O Gallegos Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey, Computational Linguistics, pp. 1-79, 2024.
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. deltablou: discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeling of the Association for Compula- tional Tinguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pp. 445 450, 2015.
Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational AI: Question answering, task-oriented dialogues and social chatbots. Now Foundations and Trends, 2019a.
Yang Gao, Christian Meyer, Mohsen Mesgar, and Iryna Gurevych. Reward learning for efficient reinforcement learning in extractive document summarisation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 2350-2355555 2019b.
Albert Galt and Emiel Krahmer. Survey of the state of the art in natural language generation: Core Lasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65-170, 2018.
Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint (rXic:2406.2009) 2024.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts:: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356-3369, 2020.
Pablo Gervás. Computational approaches to storytelling and creativity. Al Magazine, 30(3):49 49, 2009.
Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. Generating topical poetry. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1183 1191, 2016.
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. Hafez: an interactive poetry generation system. In Proceedings of ACL 2017, System Demonstrations, pp. 43 48, 2017.
Sayan Ghosh, Mathieu Chollet, Eugene Laksana, Louis- Philis-hillippe Morency, and Stefan Scherer. Affect-Im:
A neural language model for customizable affective text generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 634 142, 2017.
Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1100-1111, 2013.
Dimitra Gkatzia and Saad Mahamood. A snapshot of nlg evaluation practices 2005-2014. In Proceedings of the 15th European Workshop Natural Language Generation (ENLG), pp. 57-60, 2015.
Scraphina Goldfarb-Tarrant, Haining Feng, and Nanyun Peng. Plan, write, and revise: an interactive system for open-domain story generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties (Demonstrationn)) pp. 89 97, 2019.
38

---

Published in Transactions on Machine Learning Research (02/2025)
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Fleventh International Conference on Learning Represen- tations, 2022.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 1872-2680, 2014.
Yvette Graham, Barry Haddow, and Philipp Koehn. Translationese in machine translation evaluation. arXiv preprint ar XrX::1900..0 09833, 2019.
Roger Grosse, Juhan Bae, Cem Amil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Essi Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. Xiv preprint arXiu:2308.00 0322096 03296, 2023.
Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based decoding with automatically inferred generation order. Transaclions of the Association for Compulalional Linguislics, 7:661-69 676, 2019a.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in neural information
processing systems, 32, 2019b.
Lin Gui, Cristina Gárbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. In The Thirly-eighth Annual Conference on Neural Information Processing
Systems, 2024.
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. In Findings of the Association for Computational Linguisties ACL 2024, pp. 15202 15232, 2024.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association of Computational Linguistics, 6:437-450, 2018.
Helena H. Lee, Ke Shu, Palakorn Achanamuparp, Philips Kokoh Prasetyo, Yue Liu, Ec-Peng Lim, and Lav R Varshney. Recipegpt: Generative pre-training based cooking recipe generation and evaluation system. Tn Companion Proceedings of the Web Conference 2020, pp. 181-184, 2020.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogram- ming. In Proceedings of the 59th Annual Meeling of the Association for Compulational Linguislics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 4921-4933, 2021.
Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarck Abdelzaher, and Heng Ji. Lm-switch: Lightweight language model conditioning in word embedding space. arXiv preprint SrXin:2355.12700 2023.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandcharioun. Docs localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Newral Information Processing Systems, 36, 2024.
Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1789, 2019.
Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov. On the blind spots of model-based evaluation metries for text generation. In ACL: Annual Meeting of the Association for Compulational Linguistics, 2023.
39

---

Published in Transactions on Machine Learning Research (02/2025)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. In First Conference on Language Modeling, 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXic:2207.1250 2022.
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search.
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1535-1546, 2017.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638 1649, 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degener- ation. In International Conference on Learning Representations, 2020.
Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not all Ilm reasoners are created equal. arxiv preprint 1r.i0:2410.01745 2024.
Lee Hsun Hsieh, Yang- Yin Lee, and Ee-Peng Lim. Enconter: Entity constrained progressive sequence generation via insertion-based transformer. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3590 3599, 2021.
J Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 839-850, 2019.
Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yichong Chen, Teng Xu, and Xiaojun Wan. Are Ilm-based evaluators confusing nlg quality criteria? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9530-9577 2024.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1587-1596. JMLR. org, 2017.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In 61st Annual Meeting of the Association for Computational Linguisties, ACL 2023, pp. 1049 1065. Association for Computational Linguistics (ACL), 2023.
EunJeong Hwang, Bodhisatiwa Majumder, and Niket Tandon. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5906-59069 5919, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh, International Conference on Learning Representations, 2023.
Daphne Ippolito, Reno Kriz, Joao Sedoc, Maria Kustikova, Chris Callison-Burch, Reno Kriz, Eleni Milt- sakaki, Marianna Apidianaki, Chris Callison-Burch, John Hewitt, et al. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2018.
Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Nenkov Georgiev, Jiahui Geng, Iryna Gurevych, and Preslav Nakov. OpenFactCheck A unified framework for factuality evaluation of LLMs. In Delia Irazu Hernandez Farias, Tom Hope, and Manling Li (eds.), Proceedings of the 2024 Conference Empirioal
40

---

Published in Transactions on Machine Learning Research (02/2025)
Methods in Natural Longuage Processing: System Demonstrations, pp. 219-229, Miami, Florida, USA,
November 2024. Association for Computational Linguistics. doi: 10.18653/v/ /22//22.mmmlp-dmmoo... URL emnlp-demo. 23/
httpss//acc.antoooogy.oor///////
Hayate Iso. Autotemplatee A simple recipe for lexically constrained text generation. In INLG, 2024. URL
28387
Mohit Tyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. A neural network for factoid question answering over paragraphs. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 633 644, 2014.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Länguistics: Human Language Technologies, Volume (Long Papers), pp. 1875 1885, 2018.
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, et al. The facts grounding leaderboard: Benchmarking 1lms' ability to ground responses to long-form input. orXiv preprinl Nit:2501.032000 2025.
Natasha Jaques, Asma Ghandeharioum Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXív preprint orXic:1907.00450 2019.
Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity?a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63-S63 1977.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Fric Nyberg. Shakespearizing modern language using copy-enriched sequence-to-sequnnc models. EMNLP 2017, 6:10, 2017.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4667-4688, 2024.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non-paraen non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424-434, 2019.
Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas a way to model truthfulness in language models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 6346-6359, 2024.
Samia Kabir, David N Udo-Imch, Bonan Kou, and Tianyi Zhang. Who answers it better? an in- depth analysis of chatgpt and stack overflow answers to software engineering questions. arXiv preprint arXiv:2308.0233 2023.
Saurav Kadavath, Tom Conerly, Amanda. Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
Tomoyuki Kajiwara. Negative lexically constrained decoding for paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6047 6052, 2019.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 15696 15707. PMLR, 2023.
Anjuli Kannan and Oriol Vinyals. 1rXiv:1701.1 17010081988 2017.
Adversarial evaluation of dialogue models.
orXiv preprint
41

---

Published in Transactions on Machine Learning Research (02/2025)
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A Smith. Transparent human evaluation for image captioning. In Proceedings of the 2022 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, pp. 3464-3478, 2022.
Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlic Huang. Ctrleval: An unsupervised reference- free metric for evaluating controlled text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2306-2319, 2022.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprinl arXiv: 1909..088888 2019.
Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. A distributional approach to controlled text generation. In International Conference on Learaing Represenlations, 2021.
Mitesh M Khapra and Ananya B Sai. A lutorial on evaluation metrics used in natural language generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials, pp. 15-19, 2021.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling oulput length in neural encoder-decoders In EMNLP, 2016.
Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, et al. Adversarially regularized autoencoders for generating discrete structures. arXiv preprint arXiv: 106. 04223, 2:12, 2017.
Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975.
Rebecca Knowles and Philipp Kochn. Neural interactive translation prodiction. In Twelfth Conference of The Association for Machine Translation in the Americus, pp. 107 120. Association for Machine Translation in the Americas, AMTA, 2016.
Lingkai Kong, Haorui Wang, Wenhao MU, Yuanqi Du, Yuchen Zhuang, Yifei Zhou, Yue Song, Rongzhi Zhang, Kai Wang, and Chao Zhang, Aligning large language models with representation editing: A control perspective. Advances in Neural Information Processing Systems, 97:37356-37384, 2024.
Xiang Kong, Bohan Li, Graham Neubig, Eduard Hovy, and Yiming Yang. An adversarial approach to high-quality,
neural dialogue generation. arXiv preprint Xxx::1901.001999 2019.
Bernhard Kratzwald, Anna Eigenmann, and Stefan Feuerriegel. Rankqa: Neural question answering with answer re-ranking. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pp. 6076-6085 2019.
Ben Krause, Akhilesh Decpak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp.
2021.
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit foedback for pequence-to-sequene reinforcement learning, In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1777-1788, 2018
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNI.P-ICNNLP) pp. 540-55) 051, 2019.
42

---

Published in Transactions on Machine Learning Research (02/2025)
Keita Kurita, Nidhi Vyas, Ayush Parcek, Alan W Black, and Yulia Tsvetkov. Measuring bias in contextu- alized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pp. 166 172, 2019.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document dis- tances. In International conference on machine learning, pp. 957-966 PMLR, 2015.
Philippe Laban, Andrew Hsi, John Canny, and Marti A Hearst. The summary loop: Learning to write abstractive summaries without examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticss, volume 2020.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint ar 2rXiv2241..1512, 2024.
Guillaume Lample, Sandcep Subramanian, Eric Smith, Ludovic Denoyer, Aurcllo Ranzato, and Y-Lan Boureau. Multiple-attribute text rewriting. In Intervational Conference on Learning Represenlations, 2018.
Seemab Latif, Sarmad Bashir, Mir Muntasar Ali Agha, and Rabia Latif. Borwward-forward sequence gener- alive nelwork for multiple lexical constraints. In IFIP International Conference on Arlificial Intelligence Applications and Innovations, pp. 39-50. 50. Springer, 2020.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregreesive neural sequence mod- cling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1173 1182, 2018.
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI conference human factors in computing systems, pp. 19, 2022.
Christoph Leiter, Piyawal Lertvillayakumjona Marina Fomicheva, Wei Zhao, Yang Gao, and Stelfen Eger. Towards explainable evaluation metrics for natural language generation. arXiv preprint 3rXiv:2203.11191 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045- 3059, 2021.
Bingxuan Li, Yiwci Wang, Tao Meng, Kai-Wei Chang, and Nanyun Peng Control large language models via divide and conquer. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15240-15256. 2024a.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promotiin objective function for neural conversation models. In Proceedings of NAACL-HLT, pp. 110 119, 2016a.
Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and William B Dolan. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 994 1003, 2016b.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1102-1202, 2016c.
Jiwei Li, Will Monroc, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, 2017.
43

---

Published in Transactions on Machine Learning Research (02/2025)
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1865 1874, 2018.
Junyi Li, Charith Peris, Ninarch Mchrabi, Palash Goyal, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. The steerability of large language models toward data-driven personas. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7283 7298, 2024b.
Kenneth Li, Tianle Liu, Naomi Bashkansky, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Measuring and controlling instruction (in) stability in language model dialogs. In First Conference on Language Modeling.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. In The Eleventh International Conference on Learning Representations, 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-timee inter- vention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024c.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in Newral Information Processing Systems, 35:4328-433 4343, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pp. 4582 4597, 2021.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsuori B Hashimoto. Alpacaeval: An automatic evaluator of wow.woiiiiioooooiiiogiooo models, 2023b.
Zekun Li, Baolin Peng, Pengcheng He, Michel Galley, Jianfeng Gao, and Xifeng Yan. Guiding large language models via directional stimulus prompting. Advances in Neural Information Processing Systems, 36:62630- 62656, 2023c.
Zichao Li, Ines Arous, Siva Reddy, and Jackie Chi Kit Cheung. Evaluating dependencies in fact editing for language models: Specificity and implication awareness. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1923-7636, 2023d.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transac- tions on Machine Learning Research, 2023.
Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for 1ms.. arXiv preprint arXiv:2404.04 102, 2024.
Bill Yuchen Lin, Ming Shen, Yu Xing, Pei Zhou, and Xiang Ren. Commongen: A constrained text generation dataset towards generative commonsense reasoning. Xr preprint orXie:1911.03771 03705, 2019.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823 1840, 2020.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Mclanic Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base 1lms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, 2024a.
44

---

Published in Transactions on Machine Learning Research (02/2025)
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogie: On the scaling limits of Ilms for logical reasoning. arXie preprint orXim:2502.0111 100, 2025.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falschoods. In Proceedings of the 60th Annual Meeting of the Associalion for Compulational Linguistics (Volume. 1: Long Papers), pp. 3214 3252, 2022.
Zizheng Lin, Chunkit Chan, Yangqiu Song, and Xin Liu. Constrained reasoning chains for enhancing theory- of-mind in large language models. In Pacific Rim International Conference on Artificial Intelligence, pp. 354-360. Springer, 2024b.
David Lindner, Xin Chen, Sebastian Tschiatschek, Katja Hofmann, and Andreas Krause. Learning safety constraints from demonstrations with unknown rewards. In International Conference on Artificial Intelli- gence and Statistics, pp. 2386 2394. PMLR, 2024.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yojin Choi. Dexperts: Decoding- timenng--imeeeee time controlled text generation with experts and anti-experts. In Proceedings of the 59th Anroual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021a.
Chia-We Liu, Ryan Lowe, lulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirieal Methods Natural Language Processing, pp. 2122-2132, 2016.
Dayiheng Liu, Jie Fu, Qian Qu, and Jiancheng Lv. Bfgan: Backward and forward generative adversarial networks for lexically constrained sentence generation. TEEE/ACM Transaclions on Audio, Speech, and Language Processing, 23(1)) 2350-2211, 2019a.
Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang, Logical reasoning in large language models: A survey. arXiv preprint orXiv:2502.09100, 000000 2025a.
Jiachong Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannanch Hajishirzi, and Asli Celikyilmaz. Don't throw away your value model! generating more preferable text with value-guided monte-carlo tree search decoding. In First Conference on Language Modeling, 2024a.
Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang, and Irwin King. A survey of personalized large language models: Progress and future directions, 2025b. URL http:///axxiv..rr/bbbeoe 220.. 11528
Jiateng Liu, Penglei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, and Heng Ji. Evedit: Event-based knowledge editing with deductive editing boundaries. arXiv preprint 2xx.:2402.11324, 2024b.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Compulational Linguistics, 12:157 173, 2024c.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 35, 2023a.
Peter J Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Conference on Learning Represen- lations, 2018.
45

---

Published in Transactions on Machine Learning Research (02/2025)
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. Data boost: Text data augmentation through reinforcement learning guided conditional generation. Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
2020.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Represenlations, 2024d.
Xin Liu, Muhammad Khalifa, and Lu Wang. Bolt: Fast energy-based controlled text generation with tunable biases. In The 61sl Annual Meeling Of The Associalion For Compulational Linguistics, 2023b.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-220 1522,, 2023c.
Yixian Liu, Liwen Zhang, Wenjuan Han, Yue Zhang, and Kewei Tu. Constrained text generation with global guidance sadddnnc-casee study commongen. ar preprint arXiv:2103.0711 2021b.
Yu Lu Liu, Rachel Bawden, Thomas Scaliom, Benoit Sagot, and Jackie Chi Kit Cheung. Maskeval: Weighted mim-based evaluation for text summarization and simplification. arXic preprint arXiv:2205 222:2205.12294 2022.
Zhiqiang Liu, Zuohui Fu, Jie Cao, Gerard de Melo, Yik-Cheung Tam, Cheng Niu, and Jie Zhou. Rhetorically controlled encoder-decoder for modern chinese poetry generation. In Proceedings of the 57th Annual Meeling of the Association for Compulational Linguistics, pp. 1992 2001, 2019b.
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. In Advances in Neural Information Processing Systems, pp. 5103-5113, 2018.
Ekaterina Loginova, Stalin Varanasi, and Günter Neumann. Towards multilingual neural question answering. European Conference on Advances in Dalabases and Information Systems, pp. 274 285. Springer, 2018.
Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic (ecoding:(un) supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4288 4299, 2021.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquct. Are gans created equal? a large-scale study. In Advances in neural information processing syslems, pp. 700-709, 2018.
Linhao Luo, Zicheng Zhao, Chen Gong, Gholamreza Haffari, and Shirui Pan. Graph-constrained reasoning: Faithful reasoning on knowledge graphs with large language models. arXie preprint orXiv:2410. 18000, 2024.
Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Barnabés Póezos, Ruslan Salakhutdinov, and Louis- Philippe Morency. Styleptb: A compositional benchmark for fine-grained controllable text style transfer. In Proceedings of the 2021 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, pp. 2116 2138, 2021.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabás Póczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. Politeness transfer: tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1869 1881, 2020.
Andrea Madotto, Etsuko Ishii Zhaojiang Lin, Sumanth Dathathri, and Pascale Fung. Plug-and-play con- versational models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 2422-2433, 2433, 2020.
46

---

Published in Transactions on Machine Learning Research (02/2025)
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Popers), pp- 157 165, 2022.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. International conference on machine learning, pp. 7390 7399. PMLR, 2021.
Louis Martin, Eric Villemonte de la Clergerie, Benoît Sagot, and Anloine Bordes. Controllable sentence simplification. In Proceedings of The 12th Longuage Resources and Evoluation Conference, pp. 4689 4698, 2020.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14867-14875, 2021.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pp. 2799 2808, 2019.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1906 1919, 2020.
Timothy R McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint 2020:2400.008800 2024.
Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave, On the fragility of learned reward functions. arXiv preprint Xix::2301006633 2023.
Hongyuan Mei, Mohit Bansal, and Matthew R Walter. Coherent dialogue with attention-based language models. In Thirty- First AAAI Conference on Artificial Intelligence, 2017.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In The Eleventh International Conference on Learning Representations.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17872 2022.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. Cgmh: Constrained sentence generation by metropolis- hastings sampling. In Proceedings of the AAAT Conference Artificial Intelligence, volume 33, pp. 6834-6842, 2019.
Yishu Miao and Phil Blunsom. Language as a latent variable: Discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2016.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Tyyer, Luke Zettle- moyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.
2023.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Kerk-arik., Mix and match: Learning-fin free controllable text generationusing energy language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 401 415, 2022.
Eric Mitchell, Charles Lin, Antoine Bossclut, Chelsea Finn, and Christopher Manning. Fast model editing at scale. In International Conference on Learning Regresentations, 2022.
47

---

Published in Transactions on Machine Learning Research (02/2025)
Lili Mou, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Backward and forward language modeling for constrained sentence generation. arviv preprinl or Xie:15 161:0200000,, 2015.
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence to backward and forward sequences: approach to generative short-text conversation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguisties: Technical Papers, pp. 13349-335, 2016.
A
Jonas Mueller, David Gifford, and Tommi Jaakkola. Sequence to better sequence: continuous revision of combinatorial structures. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2536 2544. JMLR. org, 2017.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Bolinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 49 66, 2024.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, et al. Webgpt: Browscr-ussisted question-answering with human leedback. arXie preprint arXiv:21 2.09332, 2021.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to--sequenee rmns and beyond. In Proceedings of the 20th SIGNLL Conference on Compulational Natural Language Learning, pp. 280 290, 2016.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. Summarunner: A recurrent neural network based se- quence model for extractive summarization of documents. In Thirty-First AAAT Conference on Artificial Intelligence, 2017.
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp.
2015.
Nikola I Nikolov and Richard Hahnloser. Abstractive document summarization without parallel data. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 6638 6644, 2020.
Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase. Controllable text simplification with lexical constraint loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 260 266, 2019.
Tong Niu and Mohit Bansal. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389, 2018
Hugo Gonçalo Olivcira. A survey intelligent poctry generation: Languages, features, techniques, reutilisa- tion and evaluation. In Proceedings of the 10th International Conference Natural Language Generation, pp. 11 11-20, 2017.
OpenAT. Introducing chatgpt. https: // openai. com/ blog/ chattpt/ 2022.
Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024.
Hadas Orgad, Bahjat Kawar, and Yonatan Belinkov. Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7053 70611, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730 27744, 2022.
48

---

Published in Transactions on Machine Learning Research (02/2025)
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Longuage Tech- nologies, pp. 4812-4829, 2021.
Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International Conference on Machine Learning, pp. 26837 26867. PMLR, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311 318. Association for Computational Linguistics, 2002.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024a.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4998-5017, 2024b.
Ramakanth Pasumuru, Han Guo, and Mohit Bansal. Dorb: Dynamically optimizing multiple rewards with bandits. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7766-7780, 2020.
Romain Paulus, Caiming Xiong, and Richard Socher. deep reinforced model for abstractive summarization. In International Conference on Learning Represenlations, 2018.
Jonathan Pei, Kevin Yang, and Dan Klein. Preadd: Prefix-adaptive decoding for controlled text generation. In The 61st Annual Mecting Of The Association For Computational Linguistics, 2023.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. Towards controllable story gener- ation. In Proceedings of the First Workshop Storylelling, pp. 43 49, 2018.
Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 74 84, 2017.
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguislics: ACL-IJCNLP 2021, pp. 1145-1160, 2021.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint orXiv:2407.11511 2024.
Maja Popović. On reducing translation shifts in translations intended for mt evaluation. In Proceedings of Machine Translation Summil XVII: Translalor, Project and User Tracks, pp. 80 87, 2019.
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the As- socialion for Compulational Långuislies: Human Language Technologies, Volume 1 (Long Papers), pp. 1124,124,444 2018
Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2021.
Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. Coun- terfactual story reasoning and generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNI.P-ICCNLP), pp. 5046-5055 2019.
49

---

Published in Transactions on Machine Learning Research (02/2025)
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9553 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Nya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1:8, 2019.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Rame, Guilluune Couairon, Corentin Dancette, Jean- Cean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-oplimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024a.
Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Tazeeozi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. In International Conference Machine Learning, pp. 42048 42073. PMLR, 2024b.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Represenlations, ICLR, 2016.
Sudha Rao and Joel Tetreault. Dear sir or madam, may introduce the gyafc dataset:
Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Lánguisties: Human Language Technologies, Volume (Long Papers), pp. 129 140, 2018.
Ankit Singh Rawat, Chen Zhu, Daliang Li, Felix Yu, Manzil Zaheer, Sanjiv Kumar, and Srinadh Bhojanapalli, Modifying memories in transformer models. In International Conference Machine Learning (ICML), volume 2020, 2021.
Yasaman Razeghi, Robert L Logan Iv, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Compulational Linguistics: 2022.
EMNLP 2022, pp.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: neural framework for mt evaluation. Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685 2702, 2020.
Machel Reid and Graham Neubig. Learning to model editing processes. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5822-3832, 2022.
Ehud Reiter and Anja Bolz. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529-55) 0552,9555,,11 2009.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstructs of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of ulp models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2020.
Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. Long-form evaluation of model editing. In Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume
1: Long Popers), pp. 3749 3780, 2024.
50

---

Published in Transactions on Machine Learning Research (02/2025)
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. In 60th Annual Meeting of the Association for Compulational Linguistics, ACL 2022, pp. 3194 3213. Association for Computational Linguistics (ACL), 2022.
Jie Ruan, Xiao Pu, Mingqi Gao, Xiaojun Wan, and Yuesheng Zhu. Better than random: Reliable nlg human evaluation with constrained active sampling. In Proceedings of the AAAI Conference on Artificial 2024.
Intelligence, volume 38, pp.
Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for sentence summariza- tion. Tn Conference on Empirical Methods in Natural Language Processing, EMNIP 2015, pp. 379-389
Association for Computational Linguisties (ACL), 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, pp. 5228 5237, 2018.
Abdehrhman Saleh, Natasha Jaques, Asma Ghandeharioum Judy Shen, and Rosalind Picard. Hierarchi- cal reinforcement learning for open-domain dialog. In Proceediigg of the AAAT conference on artificial intelligence, volume 34, pp. 8741 8748, 2020.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pp. 29971 30004. PMLR, 2023.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances Newral Information Processing Systems, 36, 2024.
Stephanie Schoch, Diyi Yang, and Yangfeng Ji. "this is a problem, don't you agree?" framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pp. 10 16, 2020.
Moritz Schubotz, Philipp Scharpf, Kaushal Dudhat, Yash Nagar, Felix Hamborg, and Bela Gipp. Introducing mathqa: math-aware question answering system. Information Discovery and Delivery, 2018.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, 2017.
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. What makes a good conversation? how controllable attributes affect human judgments. In Proceedings of NAACL-HLT, pp. 1702 1723, 2019.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metries for text generation. In Proceedings of the 58th Arnual Meeling of the Association for Compulational Linguisties, pp. 7881-7892, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compulational Linguislies: Human Languoge Technologies, pp. 35 40, 2016.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pincau. Building end-to- end dialogue systems using generative hierarchical neural network models. In Thirlieth AAAT Conference on Artificial Intelligence, 2016.
Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. A survey of available corpora for building data-driven dialogue systems. arXiv preprint arXiv:1512.0577 057442 2015.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joclle Pincau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty- First AAAT Conference on Artificial Intelligence, 2017.
51

---

Published in Transactions on Machine Learning Research (02/2025)
Lei Sha. Gradient-guided unsupervised lexically constrained text generation. In Proceedings of the 2020 Conference Empirioal Methods in Natural Language Processing (EMNIP), pp. 8692 8703, 2020.
Darsh J Shah, Tal Schuster, and Regina Barzilay. Automatic fact-gnided sentence modification. In AAAT, pp. 8791 8798, 2020.
Erfan Shayegani, Md Abdullah Al Manun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.1088 10844, 2023.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: survey. arXiv preprint arXiv:2309. 15025, 2023.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pp. 6830 6841, 2017.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empérical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-LJCNLP), pp. 3407 3412, 2019.
Emily Sheng, Kai-Wa Wei Chang, Prem Natarajan, and Nanyun Peng. Towards controllable biases in language generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3239 0294-,24, 2020.
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 1361-4367. AAAI Press, 2018
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence embed- dings for automatic machine translation evaluation. In Proceedings of the Third Conference Machine Translation: Shared Task Papers, pp. 751-758, 758, 2018.
Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung- Woo Ha, et al. On the effect of pretraining corpora on in- context learning by a large-scale language model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5168-5186 2022.
Taylor Shin, Yasaman Razoghi, Robert L Logan IV, Eric Wallace, and Samcer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNIPP), pp. 4222-42222 -225,, 2020.
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. In The Eleventh International Conference on Learning Representations, 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. In First Conference on Language Modeling, 2024.
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L Hamilton, and Joelle Pineau. Learning an unreferenced metric for online dialogue evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2430-22 1330-2441, 2020.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: A roadmap pluralistic alignment. In Forty-first International Conference on Machine Learning, 2024.
52

---

Published in Transactions on Machine Learning Research (02/2025)
Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024.
Amanda Stent, Matthew Marge, and Mohit Singhai. Evaluating evaluation methods for generation in the presence of variation. In international conference on intelligent text processing and computational linguistics, pp. 341-351. Springer, 2005.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Usækoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning, pp. 5976 5985, 2019.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human leedback. Advances in Newral Information Processing Systems, 33:3008-300 2020.
Nishant Subramani, Nivedita Suresh, and Matthew E Peters. Extracting latent steering vectors from pre- trained language models. In Findings of the Associalion for Computational Linguislics: ACL 2022, pp. 166-581, 2022.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNLP), pp. 3269-3279, 2019.
Elior Sulem, Omri Abend, and Ari Rappoport. Bleu not suitable for the evaluation text simplification. In Proceedings of the 2018 Conference on Empirioal Methods Natural Language Processing, pp. 738 744, 2018.
Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Wieting, Nanyun Peng, and Xuezhe Ma. Evaluating large language models on controlled generation tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3155-3168 3168, 2023a.
Simeng Sun, Dhawal Gupta, and Mohit lyyer. Exploring the impact low-rank adaptation on the perfor- mance, efficiency, and regularization rlhf. arXiv preprint XrXi:22309.0 09055,, 2023b.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensoqa: A question swering challenge largeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 4149-4158, 2019.
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. Evaluating the factual consistency of large language models through sunmarization. arXie preprint or 2rX1v:2211.08412. 2022.
Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. Target-guides open-domain conversation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5624-5634, 2019.
Zecheng Tang, Pinzheng Wang, Keyan Zhou, Juntao Li, Ziqiang Cao, and Min Zhang. Can diffussio model achieve better performance in text generation? bridging the gap between training and inference! Tn Findings of the Association for Computational Linguistics: ACL 2023, pp. 11359 11386, 2023.
L Theis, A van den Oord, and M Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations (ICLR 2016), pp. 1-10, 2016.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. Xiv preprint 210 15154, 2023.
53

---

Published in Transactions on Machine Learning Research (02/2025)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arxic preprint orXiv:2307.09288 2023.
Alex Turner, M Monte, David Udell, Lisa Thiergart, and Ulisse Mini. Steering gpt-2-xl by adding an activation vector. In AI Alignment Forum, 2023.
Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. Best prac- tices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 355 368, 2019.
Judith van Stegeren and Mariët Theune. Narrative generation in the wild: Methods from nanogenmo. In Proceedings of the Second Workshop on Storytelling, pp. 65-74, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Aduances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, C Tawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4575, 2015.
Ashwin Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint Xrc::166100000 02444, 2016.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Tanguage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
Wang, Nebojsa Jojie, Chris Brockett, and Eric Nyberg. Steering output style and topic in neural response generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2140 2150, 2017.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang, Arithmetic control of ilms for diverse user preferences: Directional preference alignment with multi- objective rewards. In 62nd Annual Meeling of the Associalion for Compulational Linguistics, ACL 2024, pp. 8642 8655. Association for Computational Linguistics (ACL), 2024a.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou Is chatgpt a good nlg evaluator? a preliminary study. In Proceedings of the 4th New Frontiers Summarization Workshop. Association for Computational Linguistics, 2023a.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang. Dinghan Shen, Changyou Chen, and Lawrence Carin. Topie-guided variational auto-encoder for text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 166-177, 2019.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764- 74786, 2023b.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mchrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment techniques: Rlhf, rlaif, ppo, dpo and more. Xix preprint 240 16216, 2024b.
54

---

Published in Transactions on Machine Learning Research (02/2025)
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. Advances in Newral Information Processing Systems, 36, 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Edl Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information 2022.
processing systems,
Timothy Wei, Annabelle Mii, and Anastasia Miin. Optimizing large language models for dynamic con- straints through numan-in-the-long discriminators. arXiv preprint arXiv:2410.15163 2024.
Sean Welleck, Kianté Brantley, Hal Daumé Iii, and Kyunghyun Cho. Non-monotonic sequential text gener- ation. In International Conference Machine Learning, pp. 6716-66666 0726. PMTR, 2019.
Sean Welleck, Ximing Lu, Peler West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023.
Georg Wiese, Dirk Weissenborn, and Mariana Neves. Neural domain adaptation for biomedical question answering. In Proceedings of the 21st Conference on Compatational Natural Language Learning (CoNLL 2017); pp. 281-289, 2017.
John Wieting, Taylor Borg-Kirkpatrick, Kevin Gimpel, and Graham Neubig, Beyond bleu: Training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4344 4355, 2019.
Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dra- gan. On targeted manipulation and deception when optimizing Ilms for user feedback. arXiv preprint or arviv:24 11.22/1111111 0a.22411102066 02306, 2024.
Sam Wiseman, Stuart M Shieber, and Alexander M Rush. Challenges in dota-to-do generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253-2263 2017.
Yuxiang Wu and Baotian Hu. Learning to extract coherent summary via deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Sulır, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024a.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob An- dreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual lasks. In Proceedings of the 2024 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1819-1862, 2024b.
Sander Wubben, Antal Van Den Bosch, and Emicl Krahmer. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Anruual Meeling of the Associalion for Compulational Linguistics: Long Papers- Volume 1, pp- 1015 1024. Association for Computational Linguisties, 2012.
Joern Wuebker, Spence Green, John DeNero, Sasa Hasan, and Minh-Thang Luong, Models and inference for prefix-constrained machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, 2016.
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei- Ying Ma. Topic augmented neural response generation with a joint attention mechanism. arXiv preprint arXiv: 08340, 2(2), 2016.
Ruijic Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. Xiv preprint or 2rxiv:2444 18824, 2024.
55

---

Published in Transactions on Machine Learning Research (02/2025)
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapler of the Association for Compulalional Linguistics: Human Language Technologies, pp. 3511 3535, 2021.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Compuling Surveys, 56(4):1-33) 39, 2023.
Nai-Chi Yang, Wei-Yun Ma, and Pu-Jen Cheng. Plug-in language model: Controlling text generation with a simple regression model. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2165-2181, 2024a.
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in- context: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceed- ings of the 4 41st International Conference on Machine Learning, pp. 56276-56297, 2024b.
Shunyu Yao, Dian Yu, Jeffrey Zhaoo Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Aduances in Newral Information Processing Systems, 34, 2021.
Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. Turingadvice: A generative and dynamic evaluation of language use. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. Variational neural machine translation. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 521-530, 2016.
Bohan Zhang, Yixin Wang, and Paramveer S Dhillon. Causal inference for human-language model col- laboration. In Proceedings of the 2024 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1630- 1647, 2024a.
Hanqing Zhang and Dawci Song. Discup: Discriminator cooperative unlikelihood prompt-tuning for control- lable text generation. In Proceedings of the 2022 Conference on Empirical Melhods in Natural Language Processing, pp. 3392 3406, 2022.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1-37, 2023.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Sulam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2204-2213, 2018a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020a.
Xingxing Zhang and Mirella Lapata. Chinese poetry generation with recurrent neural networks. In Pro- ceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 670 680, 2014.
Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xinjun Li, Chris Brockett, and Bill Dolan. Generating informative and diverse conversational responses via adversarial information maximization. In Advances in Neural Information Processing Systems, pp. 1820, 2018b.
56

---

Published in Transactions on Machine Learning Research (02/2025)
Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and William B Dolan. Pointer: Constrained progressive text generation via insertion-based generative pre-training. Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8649 8670, 2020b.
Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Porsonalization of large language models: A survey. arXiv preprint (rXit:241100022 2024b.
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. Style transfer as unsupervised machine translation. arXiv preprint arXiv: 10X8:.9008.07894 2018c
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. Adversarially regularized autoen- coders. In 35th International Conference on Machine Learning, ICML 2018, pp. 1405-9420 International Machine Learning Society (TML.), 2018.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Nalural Language Processing (EMNLP-IJCNI.P), pp. 563 578, 2019.
Yinhe Zheng, Guanyi Chen, Minlic Huang, Song Liu, and Xuan Zhu. Personalized dialogue generation with diversified traits. Xiviv preprind urXiv:1901. 006672, 2019.
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15686-15702, 2023.
Chunting Zhou and Graham Neubig. Multi-space variational mucoer-decoders for semi- superviie labeled sequence transduction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 310 320, 2017.
Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Associalion for Compulational Linguislics: ACL-IJCNLP 2021, pp. 1393 1393-1404, 2021.
Chunting Zhou, Pengfci Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for aligmment. Advances in Nearal Information Processing Systems, 36, 2024.
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAT Conference on Artificial Intelligence, 2018.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and
Le Hou. Instruction-folloiing evaluation for large language models. arXiv preprint 1rX.v:2311.07911, 2023a.
Kaitlyn Zhou, Su Lin Blodgett, Adam Trischler, Hal Daumé III, Kaheer Suleman, and Alexandra Olteanu. Deconstructing nlg evaluation: Evaluation practices, assumptions, and their implications. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 314-324, 2022a.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell, and Mrinmaya Sachan. Controlled text generation with natural language instructions. In International Conference on Machine Learning, pp. 42602 42613. PMLR, 2023b.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022b.
57

---

Published in Transactions on Machine Learning Research (02/2025)
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference Research & Development in Information Retrieval, pp. 1097 1100. ACM, 2018
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fino-tuning language models from human preferences. arXiv preprint arXiv:1909.08500 2019.
Zhuang Ziyu, Chen Qiguang, Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. Through the lens of core competency: Survey on evaluation of large language models. In Proceedings of the 22nd Chinese National Conference Computational Linguistics (Volume 2: Frontier Forum), pp. 88 109, 2023.
Xu Zou, Da Yin, Qingyang Zhong, Hongxia Yang, Zhilin Yang, and Jie Tang. Controllable generation from pre-trained language models via inverse prompting. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2450 2460, 2021.
58


Reviews:
[{"review_id": "nv4n06LBDx", "reviewer": "Reviewer_ndnQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The paper provides a comprehensive coverage of prior work on multiple axes relating to constrained NLG, including different types of approaches to this problem & their limitations, datasets & metrics, etc. I appreciate the fact that the surveyed approaches encompass many different time periods, from the pre-LLM era & the current set of more recent work, including LLM prompting, etc.\n\n2. I like the fact that the paper dedicates a section on evaluation metrics & datasets and benchmarks, and highlights how these factors constitute a key bottleneck to constrained NLG today (in particular, the lack of a reliable automated evaluation metric, the lack of a standardized protocol for conducting human evaluation, and the fact that current constrained NLG datasets are not necessarily representative of how constrained NLG is done in real-world applications).\n\n# Weaknesses\n\n1. Some of the notations used in the paper are confusing / inaccurate. For example, the negative log likelihood objective at the top of page 3 has a summation only over the different **sentences** in the dataset, where each sentence is indexed by $k$. However, in standard, autoregressive language models, we also need to do a summation over the different **token positions** in the document, which is currently neglected in the current notation. The correct language modelling objective is: $\\ell(D) = \\sum_{k=1}^{|D|} \\sum_{i=1}^{N_k} \\log p_{\\theta}(x_{i}^{k} \\mid x_{<i}^{k})$, where $N_k$ denotes the length of the $k$-th sentence / document in the dataset $D$. The same point also holds for the equation in Section 2.2, and also Section 2.3 (top of page 4). The notation also does not exactly follow standard convention of denoting vectors / sequences of tokens with bold (e.g. $\\boldsymbol{x}_{<i}^k$ is denoted in bold because it denotes all the previous tokens at position $<i$), and individual token with non-bold (e.g. $x_i^k$ is denoted in bold because it is a single token). Also the model parameters $\\boldsymbol{\\theta}$ should ideally be denoted in b old, because it refers to the full set of model parameters, rather than a single scalar.\n\n2. I find some sections of the paper to be rather belaboured, and does not add too much value to the overall theme of the paper. For example, section 4 provides an overview of constrained NLG tasks like machine translation, dialogue systems, etc. I do not find this section to be necessary, because (i) many TMLR readers will already be broadly familiar with the tasks that are discussed here; and (ii) the types of NLG constraints for each task are already well-summarized in Table 1, rendering a comprehensive description in the text unnecessary. Another similar section that can be made clearer & more concise is the evaluation section. Page 22 of the paper outlines the limitations of current evaluation metrics, including word overlap metrics like BLEU, etc. This part reads to me like a rather generic overview of these automated metric's limitations, which are already quite well-known in the field. A lot of these limitations are generic, and therefore not particularly relevant to the problem at hand (constrained NLG).\n\n3. There are a few interesting points that this paper raised that warrants further discussion in the paper, which are still missing from the current version.\n    - The first point relates to the promising **approaches** for constrained NLG. There is already a lot of discussion on the paper about what is still missing from the setup & current work & datasets & metrics, but an equally important angle is: What approaches should the field try in order to get better results in this problem. Given that: (i) the evaluation metric is noisy & hard to hill-climb on; (ii) accurately measuring our progress is markedly difficult; and (iii) real-world constrained NLG applications will be **even more challenging** than what is reflected in the current datasets of today (e.g. real-world applications will require fulfilling multiple different constraints on noisy datasets), what do the authors think as the main way forward? Should we hold off on coming up with new models until we come up with better evaluations? Or are there approaches that the field is not trying enough of? Should we invest more in e.g. diffusion models? Or something else?\n    - The paper highlights the mismatch between the constrained NLG evaluation benchmarks & real-world use cases, which in my opinion is a great point and a timely problem. However, the paper does not dive into more detail on the key points of this mismatch. What kinds of mismatch are there? And based on this, what kinds of datasets & metrics should we develop next in order to more accurately measure progress in this direction? Should we focus on problems with multiple constraints? Or should we focus on more realistic (e.g. real-world LLM use cases) & diverse domains for evaluations? Or something else?\n    - What are the areas that we should revisit? Is there any prior work from the pre-LLM era that we should revisit, because they are likely to combine nicely with recent LLM advances? It would be great if this survey paper does not solely focus on what has been done and what is still missing, but also take a stance on the most promising ways forward.\n\n4. I find the assertion in Section 6, \"Perplexity\" paragraph, page 20 to be dubious. In particular, it states that \"... how likely a sentence is generated by a given model is not directly comparable across different models.\" I would argue to the contrary, that one can, indeed, compare perplexity from different models (we have been doing it since the days of the Penn Treebank language modelling evaluation). The main thing is that we need to divide the total negative log likelihood from each model with the same denominator (typically the number of tokens or characters or bytes in the whole evaluation dataset). If we do it this way, we can indeed compare perplexity from different models, including those with a different tokenization scheme, pretraining corpus, etc. In fact, many datasets like the Hutter Prize does exactly this. Given how correlated perplexity is with downstream performance (not perfectly correlated, of course, but it is quite highly correlated with a lot of downstream performance, even in the case of LLMs), I would like to disagree with this assertion.\n\n5. There are still a number of typos in the paper, so it could benefit from more copy-editing. This is a minor point (primarily because it is a long paper and it is hard to catch all the typos), and I find the paper to be generally clear and well-written. I also have some presentational suggestions:\n    - In Section 2.2, is \"conditional text generation\" basically **sequence-to-sequence generation**? The field used to use this term to refer to any problem that has a sequence-like input and an output, including machine translation, summarization, dialogue systems, etc. If so, might be good to mention the connection.\n    - In page 7, \"... it is desirable generated dialogue ...\" -> \"it is desirable **that** ...\"\n    -  In page 10, \"In lack of suitable ...\" -> \"In **a** lack of ...\"\n    - In page 16, \"while at the same time preserve ...\" -> \"**preserving**\"\n    - In page 17, \"...allow to rewrite...\" -> \"allow **us** to rewrite...\"\n    - In page 18, the reinforcement learning part and why it is difficult might warrant its own subsection, which would improve clarity.\n    - In page 24, \"... can be mislead ...\" -> \"**misled**\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a survey on **constrained neural language generation**. It begins with a definition off constrained neural language generation, and highlights its important difference with the **conditional** language generation problem. Concretely, the former is defined as imposing constraints on the output of the model (e.g. the model needs to follow a particular style or constraint), whereas the former is defined as conducting neural language generation, conditional on some input (e.g. the input condition can be a sentence in the source language, in the case of neural machine translation). The paper then proceeds to give examples of constrained NLG tasks, including the typical input condition & output constraints for each task, such as format constraints, syntactic constraints, semantic constraints, etc.\n\nAfter defining the problem and the typical types of constraints & conditions for each task, the paper then surveys prior approaches on constraint NLG, including decoding approaches, fine-tuning approaches, discriminative approaches, edit-based approaches, adapting existing models & architectures to accommodate the constraints, and also prompting-based approaches for LLMs. The paper then outlines current evaluation metrics, benchmarks, and datasets for constrained NLG evaluation, and highlights why this remains a challenging open problem in the field (e.g. noise in automatic metrics, lack of standardized procedure for human evaluation, etc.).  The paper then summarizes the key open challenges in constrained NLG, including the key problems that the field should tackle in order to make progress on more realistic constrained NLG setups (e.g. satisfying multiple different constraints at once, rather than focusing mostly on one type of constraint, as is done in most constrained NLG evaluation datasets today).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Critical**: Refactor parts of the paper to be made more concise (Weakness point 2).\n2. **Critical**: Improve the notations (Weakness point 1).\n3. **Critical**: Expand the discussion on: (i) what promising approaches we should try more for constraint NLG, given that the problem is very much a challenging one & will likely require more modelling / architectural innovations; (ii) what mismatch there is between current constrained NLG evaluation metric, datasets, and benchmarks, in contrast to real-world applications of the problem (this will highlight what kinds of datasets & metrics we should develop next as a community); and (iii) what the areas & approaches we should revisit are in the LLM era (e.g. are there any techniques that we should combine with recent LLM progress, and why). (Weakness point 3).\n4. **Important**: Clarify the discussion with respect to the perplexity point, and how it is potentially comparable across different models (Weakness point 4).\n5. **Important**: Improve the presentation of the paper & fix the typos (Weakness point 5).", "broader_impact_concerns": "No broader impact concerns come to mind.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JuFmLxakCe", "reviewer": "Reviewer_r3Cn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Insufficient Novelty: The paper is primarily a survey and does not propose new methods, datasets, or evaluation frameworks.\n2. Limited Practical Insights: While challenges are listed, actionable strategies for overcoming these limitations are underexplored.\n3. Superficial Metrics Discussion: The analysis of evaluation metrics for constrained NLG lacks depth and does not suggest comprehensive solutions.\n4. Dataset Diversity: Although dataset limitations are acknowledged, the survey does not provide a detailed catalog of available datasets or gaps.\n5. Heavy Focus on Challenges: The paper dedicates significant space to challenges without sufficiently emphasizing progress or success stories in the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides an extensive survey on constrained neural language generation (NLG), focusing on formally defining the challenges, methodologies, and applications of imposing constraints on output text. It categorizes NLG tasks into generic, conditional, and constrained paradigms and highlights the distinction between conditions (testable statements about the input) and constraints (testable statements about the output). The work comprehensively reviews many approaches (decoding, fine-tuning, discriminative, edit- based, and architecture adaptation) and the challenges associated with constrained NLG, such as lack of model expressiveness, suitable evaluation metrics and datasets.\n\nSummary of Contributions:\n1. Definition and Categorization: The paper provides a clear distinction between conditions and constraints, offering a structured framework for understanding constrained NLG.\n2. Extensive Survey: It consolidates a broad range of methods for constrained text generation, detailing their strengths and limitations.\n3. Highlighting Challenges: This work identifies key barriers to advancing constrained NLG, including the non-differentiability of constraints and limited datasets.\n4. Applications and Metrics: Authors in this survey outline real-world applications, such as dialogue systems, machine translation, and summarization, and discusses existing evaluation metrics for constrained NLG.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Propose New Metrics: Include detailed discussions on potential novel evaluation metrics for constrained NLG tasks.\n2. Expand Dataset Review: Provide a categorized list of existing datasets and specific gaps that need to be addressed.\n3. Highlight Positive Trends: Balance the discussion of challenges with examples of successful constrained NLG applications and breakthroughs.\n4. Suggest Research Directions: Offer concrete, actionable suggestions for addressing identified challenges, such as integrating reinforcement learning with constraints or developing hybrid architectures.", "broader_impact_concerns": "1. Bias and Safety: The survey mentions mitigating toxicity and bias in generated content\nbut does not delve into ethical concerns of deploying constrained NLG systems.\n2. Dual-Use Risks: Constrained NLG can be misused to generate persuasive but harmful content, such as disinformation or propaganda.\n3. Societal Applications: The broader implications of constrained NLG in sensitive domains like healthcare or legal advice need more exploration.\n\nComments, Suggestions, and Typos:\n\n1. Comments:\nClarify the distinction between soft and hard constraints with examples. Expand on how constraints can be dynamically adjusted during generation. Provide a visual taxonomy of methods and applications for constrained NLG.\n\n2. Suggestions:\nUse tables or charts to summarize approaches, applications, and challenges for better readability.\nInclude a dedicated section for \"Future Directions\" to organize actionable insights more effectively.\n   3. Typos and Formatting Issues:\nPage 3: \"generative commonsense reasoning with a constrained text generation task is more challenging for machine learning models\" — rephrase for clarity.\nPage 7: \"stylistic constraints are immediately relevant to the task of text style transfer\" — redundant phrasing, streamline.\nSeveral sections use \"eg.,\" without periods; standardize to \"e.g.,\" for consistency.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ckAojYhyY", "reviewer": "Reviewer_Umg2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is a very interesting topic covered from a compilation point of view (no experiments).\n\nThe writing is clear and understandable; however, there are points that need to be reviewed urgently.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The following contribution provides an extensive review of natural language generation, delving into the various constraints, metrics, and conditions that critically influence the quality of generation and the evaluation processes of current neural language models. This detailed analysis aims to shed light on the key factors that drive the performance and effectiveness of these models in real-world applications, highlighting both the advancements and the ongoing challenges faced by researchers and practitioners in the field.\n\nGenerally, the contribution is well-formed and has a clear and concise structure. However, some unclear points need to be examined thoroughly.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The contribution proposes a multidimensional survey of the NLP world. Although it was well written and organized, there are points that make the contribution of little value. \nIn particular:\n\n- The introduction is clear, but research questions or bullet points should be used to clearly understand the direction of the work;\n- The citations are truly archaic; it is not possible to present a work without citations from at least the year preceding the submission\nIt may be important to make a hierarchical map of the works (which still need to be reviewed).\n\nThe contribution has a good structure but definitely needs updating to be competitive and in a good form.", "broader_impact_concerns": "The impact is not decisional. Although the paper is interesting and deals with important topics, it needs updating.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As with many survey papers, it is difficult to define exactly whether the "claim" of providing a survey is supported. In general, I think it is in this case, although there are several revisions that would make the paper stronger.

Reviewer ndnQ gives an extremely detailed review with some very concrete recommendations. Please include the discussion in the paper and make the required notation fixes. I would appreciate it if you can highlight the changes for ease of reviewing.

Reviewer Umg2 complains about the datedness of cites. The citations to older work are, in my view, commendable, and not a problem at all.  However, I think the survey feels a bit dated in its orientation at times (that is, preferring older work and older topics to the exclusion of newer ones). For one thing, there is relatively little treatment of the most canonical approach for constrained NLG these days, which is prompting.

Section 5.2 discusses fine-tuning approaches including modern LLMs, and section 7 discusses some relevant datasets like IFEval (which is a modern take on most of the constrained generation literature like CommonGen).  However, the future work directions do not make enough mention of this in my view:
* Multiple constraint satisfaction doesn't mention prompting, yet this is the go-to solution these days. Can prompted models satisfy multiple constraints? At what LLM scales? Benchmarks like IFEval, FollowBench, etc. have sought to answer this question and contain some evaluation. This should be mentioned.
* The future direction on "rule constraints" would do well to mention this as well: does prompting models for lexical constraints work? If so, do we need methods that enforce these constraints in a hard way? If not, where would c
onstrained NLG methods be most useful?
* Finally, dynamically-defined constraints and generative reasoning both tie in with modern LLM methodology (and the latter is one of the biggest topics in the field, with whole surveys dedicated to it like https://arxiv.org/html/2407.11511v1 ). I think these sections should be a bit better contextualized: this survey is neither introducing these nor really calling special attention to them, as they are already well-studied problems.

Nevertheless, the survey provides a substantial treatment of the subject and will be valuable to readers who aren't familiar with this area.

Requested changes:

1. ndnQ's changes
2. Provide better contextualization of the future work as requested above
3. Fix the references style; citep should be used almost everywhere, so the cites look like (Krause et al., 2020).
Decision: Accept (poster)