Multi-Attribute Constraint Satisfaction via Language Model Rewriting
Authors: Ashutosh Baheti
Debanjana Chakraborty
Faeze Brahman
Ronan Le Bras
Ximing Lu
Nouha Dziri
Yejin Choi
Mark Riedl
Maarten Sap
Venue: Accepted by TMLR ()

Abstract: Obeying precise constraints on top of multiple external attributes is a common computational problem underlying seemingly different domains, from controlled text generation to protein engineering. Existing language model (LM) controllability methods for multi-attribute constraint satisfaction often rely on specialized architectures or gradient-based classifiers, limiting their flexibility to work with arbitrary black-box evaluators and pretrained models. Current general-purpose large language models, while capable, cannot achieve fine-grained multi-attribute control over external attributes. Thus, we create Multi-Attribute Constraint Satisfaction (MACS), a generalized method capable of finetuning language models on any sequential domain to satisfy user-specified constraints on multiple external real-value attributes. Our method trains LMs as editors by sampling diverse multi-attribute edit pairs from an initial set of paraphrased outputs. During inference, LM iteratively improves upon its previous solution to satisfy constraints for all attributes by leveraging our designed constraint satisfaction reward. We additionally experiment with reward-weighted behavior cloning to further improve the constraint satisfaction rate of LMs. To evaluate our approach, we present a new Fine-grained Constraint Satisfaction (FineCS) benchmark, featuring two challenging tasks: (1) Text Style Transfer, where the goal is to simultaneously modify the sentiment and complexity of reviews, and (2) Protein Design, focusing on modulating fluorescence and stability of Green Fluorescent Proteins (GFP). Our empirical results show that MACS achieves the highest threshold satisfaction in both FineCS tasks, outperforming strong domain-specific baselines. Our work opens new avenues for generalized and real-value multi-attribute control, with implications for diverse applications spanning natural language processing and bioinformatics.

Published in Transactions on Machine Learning Research (04/2025)
Multi-Attribune Constraint Satisfaction via Language Model Rewriting
Ashutosh Bahetio.4r; Debanjana Chakraborty", Faeze Brahman Ronan Le Bras* Ximing Lu Nouha Dziri*, Yejin Choi Mark Riedl Maarten Sap*.* 9 Georgia Institute of Technology, University Washington, The Ohio State University,
Carnegie Mellon University, AAleen Institute for Artificial Intelligence ababett196@gatech.ede
Reviewed OpenReview: http:://goaanoroeeeeeee///rrrrrr
Abstract
Obeying precise constraints on top of multiple external attributes is common computational problem underlying scomingly different domains, from controlled text generation to protein engineering. Existing language model (LM) controllability methods for multi-attribute constraint satisfaction often rely on specialized architectures or gradient-based classifiers, limiting their flexibility work with arbitrary black-box evaluators and pretrained models. Current general-purpose large language models, while capable, cannot achieve fine-grained multi-attribute control over external attribules. Thus, we create Multi-Attributt Constraint Satisfaction (MACS), a generalized method capable of finetuning language models on any sequential domain to satisfy user-specified constraints on multiple external real-value attributes. Our method trains LMs as editors by sampling diverse multi-attribute edit pairs from an initial set of paraphrased outputs. During inference, LM iteratively improves upon its previous solution to satisfy constraints for all attributes by leveraging our designed constraint satisfaction reward. We additionally experiment with reward-weighted behavior cloning to further improve the constraint satisfaction rate of LMs. To evaluate our approach, we present a new Fino-grained Constraint Satisfaction (FINECS) benchmark, featuring two challenging lasks: (1) Text Style Transfer, where the goal is to simul simultaneously modify the sentiment and complexity of reviews, and (2) Protein Design, focusing on modulating fluorescence and stability of Green Fluorescent Proteins (GFP). Our empirical results show that MACS achieves the highest threshold satisfaction in both FINECS tasks, outperforming strong domain-specifie baselines. Our work opens avenues for generalized and real-value multi-attribute control, with implications for diverse applications spanning natural language processing and bioinformatics.
1 Introduction
Multi-attribute constraint satisfaction is a challenging problem that holds many useful applications in the domains of natural language processing (NTP),, drug design, and protein engineering. In NL.P, numerous classifiers and regressors cxist for detecting individual linguistic attributes such as fluency, scntiment, formality, and complexity. Enabling fone-grained granular control over such attributes will allow users to personalize any text with their desired style (Kumar et al. 2021 2022). In the realm of medicine and biotechnology, fine-grained control of multiple physicochemical properties opens avenues for engincering of novel drugs and proteins, for example, antibiotics with increased efficacy and reduced toxicity (Wong et al. 2023 and specialized proteins with manipulated attributes like fluorescence, binding affinity (Shen el al.
2014 and
stability Chan et al. 2021b.
Conventional methods for multi-attribute control often rely on mechanisms such as class-conditioned LMs
Keskar et al. 2019 Lu et al. 2022 Hallinan et al. 2023) latent attribute embeddings (He et. al. 2020 Russo

---

Published in Transactions on Machine Learning Research (04/2025)
Multi-Attribute Constraint Satisfaction (MACS)
Sequential Domain Dataset
Real-Value Attribute Evaluators (c)
Define Threshold Boundaries
Multi-step LM rewriting to satisfy all constraints
Train LM editor w/ SFT or WBC
Sentiment
Cya
Yo
ye p
Complexity
Astriburet
V, X
2 x
Fluency
Reviews
Cy
Y3
Flucrescence
Altribute?
Stability
Y4
Construct Edit Pairs (Yar X,)
Proteins
Figure 1: MACS framework starts with sequential domain datasets (customer reviews or proteins) and set of real-value attribute evaluators (such as sentiment, complexity regressors, or protein folding models). We then define fino-grained dhresholdwwindow boundaries for every attribute and create edit pairs distributed across the multi-attribule landscape. We train the LM editor lop of the edit pairs by leveraging supervised fine-tuning (SFT) or weight-weighted behavior cloning (wBC). Subsequently, LM editors can achieve the desired fine-grained constraints by employing prioritized editing that maintains a priority queue of past edits ordered by their proximity to the target threshold constraints.
et al. 2020 Riley et al. 2021 Gu et al. 2022 Liu et al. 2022 Ding al. 2023 However, these approaches are generally proposed to handle attributes with categorical values and may not effectively generalize to those with continuous/scall scalar values, such as gradually changing the complexity readability of sentence or modifying the activity of a protein within specified bounds. Techniques that do support real-value constraints satisfaction often require computationally expensive gradient-base bbaeed decoding optimizations (Dathathri et al.,
2020 Kumar al.] 2022 2022 Qin al.] [202 Li et al.] (2022) or seergy-based sampling (Mireshghallahi et al. 2022 Liu et al. 2023). These methods suffer from slow inference and fixed-length outputs, limiting their widespread adaption to downstream applications.
We introduce the Multi-Attribute Constraint Satisfaction (MACS) framework, generalized method for training LMs from diverse sequential domains towards fine-grained constraint satisfaction. Here, the LM is conceptualized as an editor tasked with navigating the multi-attribute landscape, iteratively refining its outputs to meet desired constraints. Unlike online and off-policy reinforcement learning (RL) methods (Lu) lt al. 2022; Hallinan et al. (023), which require LM genteregee data during training, our approach utilizes initial set of paraphrased outputs and directly trains the LM editor them by sampling edit pairs. The initial paraphrases are obtained externally. In the case of text style transfer, for example, through few-shot prompting in the language domain. In the case of protein design, through randomized mutations in the protein domain. As part of the framework, we introduce a generalized reward function for constraint satisfaction and experiment with offline rewire-weighted behavior cloning (Norouzi et al. 2016) to train the fine-grained LM editor on sampled edit pairs. Finally, we introduce a reward-prioritized multi-step inference stratogy to enhance constraint satisfaction while reducing overall inference computational cost. We provide an overview of our learning and evaluation process of MACS in Figure [
To comprehensively assess the effectiveness of MACS, we introduce novel Fine-grained Constraint Satisfac- tion (FINECS) benchmark, that consists of two challenging controllability tasks. The first task, Text Style Transfer ($4), requires precisely modilying the sentiment. and complexity of a given text while preserving its fluency and content similarity. We sub-divide sentiment and complexity attribute ranges into five threshold window constraints each, resulting in 25 different multi-attribute threshold combinations. The socond task, Protein Design ($5), focuses on mutating the Green Fluorescent Protein (GFP) achieve the desired fluorescence and stability, with both attributes divided into four threshold windows, leading to total of
2

---

Published in Transactions on Machine Learning Research (04/2025)
16 threshol--window combinations. During evaluation, the LM editors are tasked with satisfying every multi-attribute constraint within a fixed inference budget. We analyze the trade-off between different input conditions, inference strategies, and objective functions.
A systematic comparison of our framework against preexisting domain-specific baseline: shows that LM editors trained with MACS yield the highest constraint satisfaction success rate in both FINECS tasks. Our findings show the potential of adapting language models from diverse domains as fine- grained editors that allow navigating across the multiple-attribute landscape and discovering novel sequences. We release the code at https:///ithuu: c abaaet1955MMAS
2 Related Work
Precise Multi-Attribute Control While controlled text generation and style transfer have been widely studied problems in NLP literature, enabling fine-grained constraint satisfaction still proves to be quite challenging. A prominent approach is to incorporate attribute signals in gradients during decoding to allow
satisfying multiple attribute constraints on them (Dathathri et al. 2020 Kumar et al. 2021 2022
Qin et al.,
2022; Li et al. 2022 Liu et al. 2023) However, there are three major limitations of these mothods, (1) they require white-box access to evaluators for gradient computation, (2) their decoding speed is slow and memory intensive, and (3) their output length needs to be predefined to tractably compute the gradients. Other
studies have proposed architecture augmentations and specialized loss functions (Russo et al. 2020 Riley
et al. 2021 Gu el al. 2022 Liu et al. 2022: Ding et al. 2023 Hu et al. 2023) to perform multi-altribuie
control of language models. However, they typically cannot work with arbitrary external attribute evaluaton
and some also require expensive on-policy or off-policy samples during training. Recently, Mireshghallahi
itt al. (022) proposed probabilistic energy models to allow black-box attribute scorer constraints, but can only use masked language models for out sampling- In contrast. to all the above methods, our framework leverages offline learning to offer the most flexibility in terms of external scorers, LM architecture choice, and training data sources.
Iterative Refinement via verbal feedback LLMs may not generate the best output on their first attempt. Therefore, many recent prompting methods have been introduced for LLMs to iteratively improve model outputs while incorporating internal /nd/or external LLM evaluators as verbal feedback (Shinn et al,
2023 Zhang et al. 2023 Madaan et al. 2023: Dhuliawala et al.
2023 Akyurek et al., 2023: Gou et al.
2024). However, these methods implicitly expect the availability of expert large language model (LLM) which
may become costly during inference. Studies also find inompting LLMs with only scalar feedback is not as effective as using both scalar and verbal feedback (Peng et al.) 2023). These methods are further limited by unrecognizable language or non-language sequential data sources (for example, DNA, protein, or chemical sequences) due to lack of domain knowledge (Ouyang el. al. rewriting approach like MACS.
2024 motivating the need for general-domain
Iterative Refinement via fine-tuning To reduce inference costs, a few studies have demonstrated single attribute improvement across a diverse set of tasks via finetuning approaches for small LMs (Padmakumar et al. 2023 Welleck et al. 2023). Typically, a corrector a small LM edits the previous response from itself or an external LLM to improve downstream task performance. These correctors are supervised finetuned on edit pairs obtained from off-policy sampling paraphrasing techniques (mask and infill). We built upon these works to provide a unified framework for fine-grained control of multiple external attributes while only using offline data.
Data-driven approaches for Protein Engineering Designing proteins with desirable functionalities using limited data has been a longstanding challenge in biotechnology. Recent works have successfully leveraged machine learning and deep learning methods on assay-labeled data to find new protein sequences
with enhanced propertics such as fluorescence, binding affinity, stability, assembling, and net charge content.
Hsu al.] 2022 Sinai et al. 2020 Ren et al. 2022 Padmakumar al. 2023 Kirjner et al. 2024 Sternke
We only focus on offline methods which exclusively use preexisting data, thus avoid comparison with online and off ofl-policy RL methods our study which typically require some form of expensive LM exploration.
3

---

Published in Transactions on Machine Learning Research (04/2025)
& Karpiak 2023). However, most of these approaches are limited to unidirectional optimization of only single attribute that may compromise other physicochemical properties. Fine-grained control of protein sequences can allow simultaneous tuning of multiple properties of interest and provide deeper insights into sequence-strrccurr--ne funntcttttrreneiiiionnnoo relationships of proteins across these properties. For example, understanding
the impact on activity (Huang et al. 1996
Guo et al. 2004). lluorescence (Shaner et al.; 2007 Amat &
Nifosi 2013), stability (Rabbani et al.
2023
Schlinkmann al. 2012 Childers Daggett, 2017]. solubility
Bolognesi et al.; 2019), assembly (Garcia Seisdedos et al. 2022 Bryant et al. 2022)) and binding affinity
[tarr et al. 2020 Whitehead et al. 2012 under different physiological conditions.
3
Multi-Attribute Constraint Satisfaction
3.1 Problem Definition
We aim lo solve multi-attribute constraint satisfaction for any sequential data as a multi-step LM rewriting task. Formally, the language model is the actor the Markov Decision Process (MDP), that learns to navigate across a multi-attribute space defined by a set of attribute evaluators C = {ci, .2.... c2 Cx (which can be classifier probability, regressor, embedding similarity, protein attribute predictors, etc). All attribute evaluators convert sequential inputs into a scalar value within finite range (cy(-) E [Uj, 1 mj,,)r)). Fach MDP episode begins with the initial state containing a context I (that can be empty), starting sequence yo and its altribute location C(yo) and a set of threshold window constraints T = {41, 1,,,,,,,, (x), where ty = (tj,start, (nnen)) is the threshold boundary for attribute cj. The rewriting language model M iteratively odits the previous sequence until it satisfics the given threshold constraints, i.c., PM (ys-111 12, yis C(ys), T Here, each edit y/4 + V is considered an action, with a deterministic transition to the next state. During inference, the goal is to generate series of consecutive edits starting from yo yas such that C(yn) E T.
3.2 MACS Approach
Edit Pairs Construction Even though the rewriting process is inherently multi-step during inference, we can isolate individual edits and train language model rewrite using offline pairs. For example, given any pair of similar sequences Va and y/b which have distinct attribute locations C(ya) and C(ya), we can construct a training instance by asking the language model to edit Ya + Y/ and artificially selecting threshold windows T that encourage M to move from C'yo)) towards C(y)) (Andrychowicz et al., 2017). We can similarly define another training instance going from yo - Ya- Assuming m variations of a particular sequence are
available (y1, 1/2 -.-m);; we can construct Pm trainable edit pairs from them. In 44 and $5 create edit pairs for languages and proteins respectively.
we show how we
Constraint Satisfaction Reward We want to encourage the rewriter LM to make edits that move closer to the usev-provided multi-attribute threshold boundary. If the initial sequence is already inside the target threshold boundaries, we expect the LM to paraphrase the sequence. Basod on these two aspects, for each attribute (c)(.) [ujmmin Uj)maz]) and its corresponding threshold boundary (tj = (Ij,starts (jnn))), we define its constraint satisfaction reward the sum of the two components,
R(ym No: (,())t))= = (lc((W))ttt) + -
(1)
Satisfaction Score
Change Satisfaction Score
Here Yn and Yo represent the new and the old sequence respectively, while f(.) E [0. is the threshold satisfaction scoring function that shows the deviation of the attribute score from its threshold boundary. We set the satisfaction score as 1 if its attribute location satisfies the threshold and linearly decreases to 0 as it moves towards the extreme ends,
(c((o)-y..iaa) [fjatort - mjmin))
if ey((y) < (j,start if Li,stant S se(()) S lj,end otherwise
f(c,(v),t))
(2)
(1).ma--:(()) [8),max - (y,end)
2C(w) represents a vector of attribute scores for an intermediate output yi
$Selecting the threshold window that satisfies C(y6) works best for training the language model editor.
4

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm 1: Multi-Attribute Constraint Satisfaction Training pseudo code
Data: Edit Pairs Offline set = Un D.ye.. to {(z, Yas yb, To-+)), Attribute Evaluators C = {c1, .2.,..c Initial rewriting language model M, SFT steps N1, wBC steps N2, Learning rates 01, 02
Obtain attribute values for all sequences yı D
M1 t- M
for i i to Ni do
4
Sample edit pair (x,, Vas yd, To-+b) from D (random or k-NN sampling) LSFT(M) = In PMr, (yoke, Ya. C(ya), Ta-sh)
5
6
M1 + M1 - 01 M., LSFT (M)
end
7
M2 +- M1
8
for - to N2 do
9
Sample edit pair (x,, ya yb, To-bb) from D (random or k-NN sampling) R(yb. y C, Ta-bb) x In PM2 (y6ler, Bay Ta-))
M2 + M2 02 V W BC M22)
end
Evaluate M1 and M2 with multi-step inference strategies
The total multi-attribute reward is defined as the sum of satisfaction reward for all attributes,
R(yns No: C,T)=
Σ R(yns No: No: (.))
Given a collection of edit pairs D Uz Me Mo. {(x, ya. yb. Ta-+h}} we obtain an LM editor by employ- ing supervised finetuning, e-g. with the negative log-likelihood loss LsFT(M) In PM (26 Yo: C(y.), Ta-s)).
Learning
To improve beyond the supervised finetuned model, experiment fine-tuning it further with the offline
reward-weigee
bchavior cloning objective (Norouzi et al. 2016
et al.
2018
Wang
lt al.
2020
Ghosh et al., 2021 Ramachandran et al.] 2022
Yang et al. 2023 Feng et al.
2023
Baheti
et
al.
2024).
that directly multiplies the reward with SFT objective LwB = R(y,, Va, C,Ta.
In PM (36 rr No. C(ya), Ta-45). We provide the pseudo-code of the MACS training process in Algorithm
Multi-Step Reward Prioritized Inference Satisfying multiple precise constraints T across diverse attributes C' is a challenging task for which the editor language model, PM (Mi+ll,,,, yi, Clvs), may not get the correct answer in one try. A trivial solution is to employ best-of-N inference strategy. To improve beyond best-of-N, previous solutions to single attribute iterative improvement propose using an iterative editing strategy, where the rewriter LM generates a trajectory of edits sequentially (3o + 91... + yn) (Padmakumar ct al. 2023 Welleck et al. 2023). However, this naive editing stratogy doesn't interact with the attribute evaluators and cannot verify if the intermediate edits are moving toward the threshold constraints or not. We instead propose maintaining a priority qucue of edits using the generalized reward function we defined equation In this strategy, the LM generated subsequent edit y/ + 9i+1 is only retained if it moves closer to the threshold satisfaction, i.e. R(3++1, yo. C,T) > R(y, Yo: C)T). We call this strategy prioritized inference and compare its performance against best-of-N and naive editing.
In the subsequent sections, we extensively and systematically evaluate MACS and baselines on a new ( (FINECS) benchmark, which comprises two fine- ane-grained control tasks:
Fine-grained Constraint Satisfaction Text Style Transfer $ and Protein Design
$
4 FineCS Text Style Transfer
While foundational large language models are capable of solving a variety of general language tasks via prompt engineering (Ouyang et al. 2022 OpenAI et al.) (0244) they incur large computation overhead during inference and often underperform in directly incorporating external real-value feedback (Peng et al. 2023
5

---

Published in Transactions on Machine Learning Research (04/2025)
2
Change complexity
Change Complexity
-1
-1
-3
-4
Change in Sentiment
Change in Sentiment
(a) Random Sample Density
(b) K-NN Sample Density
Figure 2: Sentiment and Complexity attribute edit pair distribution via random sampling vs. proposed k-NN sampling. k-NN sampling yields much more diverse set of edit pairs better suited for simulating editing in all directions.
To mitigate their limitations, develop MACS to fine-tune small language models that enable constraint satisfaction external signals via iterative refinement (Padmakumar et al. 2023 Welleck el. al. 2023). To evaluate these methods, we create FINECS Text Style Transfer task, where the goal is to precisely modify the sentiment and complexity of Yelp reviews while preserving fluency and content similarity.
To obtain the sentiment and complexity evaluators, we train RoBERTa-large (Liu et al. 2020 regressors on Yelp reviews (Zhang et al. 2015) and the SWiPE Wikipedia. simplification dataset [Laban et al., 2023). The output range of the sentiment regressor E [], while the complexity regressor is within the range [-2, _ Subsequently, we defined five threshold boundaries for sentiment as follows:
Attribute Evaluators
(1, 1.5) very negative, (1.5, 2.5) negative, (2.5, 3.5) neutral, (3.5, 4.5) positive, (4.5,5) very positive and five threshold boundaries for complexily as follows, (-2, 1.55 very simple, (-1.5, 0.5) simple, (-0.5, 0.5) normal, (0.5, 1.5) complex, (1.5,2) 2) very complex. In total, these results in 25 different multi-attribute threshold combinations.
We further include two more evaluators to encourage fluency and content preservation: (1) fluency classifier probability (E [0, 1 and cosine text embedding similarity scord between the previous and the new output (E [0, I)). Since we always want to maximize both properties, add their scores directly in the constraint satisfaction reward function (eqn. 3 as two additional components.
Creating Attributed Variations and Edit Pairs To synthetically obtain a diverse set of paraphrases
previous studies have proposed various techniques such as mask-then-infill (Xu et al. 2018 Li et al. 2018 Ma
et al. Padmakumar et al. 2023), back-translations (Prabhumoye et al., 2018 Zhang et al. 2018 Lample
2020
ct al. (Welleck et al. 2023). our preliminary experiments, these methods did not yield many diverse attribute variations. Instead, we use few-shot prompted LLMs to generate alternate attributed variations of reviews. In particular, for both sentiment and complexity attributes, first sample an equal number of reviews from each threshold boundary (1K from each label, 5K total for each attribute). Then, construct few-shot
2019 Luo et al. 2019), paraphrasing models (Krishna et al. 2020) and generating multiple samples
"Training dctails of sentiment and complexity regressors provided in Appendix A.I ROBERRT -ase classifier from the Corpus of Länguistic Acceptability Warstadt https//allmmmttteeseeeeee baseev2
2019 textattack/robbrta- CoLA

---

Published in Transactions on Machine Learning Research (04/2025)
LLM prompts that propose five alternate variations of each review, one for each sentiment (or complexity) label. We employ nucleus sampling (Holtzman et al. 2019h on the few-shot prompts (top,- = 0.95) to generate 25 variations for each review, ~ 125K total variations for each attribute Considering the original review and its 25 new proposed variations, we can construct at most P2 trainable edit pairs for each review ($3.2). The final dataset simply combines all the edit pairs from sentiment and complexity variations. We choose Llama2-7B parameter model (Touvron et al. 2023) as our base LLM and provide the prompts designed for sentiment and complexity attributes in the Appendix A.2
Within the language domain, edit pairs from syuthetic variations are not uniformly distributed. Therefore we propose k-NN Sampling to obtain evenly distributed edit pairs in the multi-attribute space. In multi-step editing via LM, consecutive edits can lead to large drift in content from the original text. Subsequently, we propose an Anchor Conditioned Inference strategy to mitigate this problem. We discuss both algorithmic modifications below.
k-NN Edit Pair Sampling An ideal data distribution should have edit pairs from every multi-attribute location to every other location. However, in practice, this is not always true. Given an edit pair Yo + yb, we visualize its attribute change by converting the difference into vector C(ys) C(va). We show the distribution of edit pairs when sampled randomly from our synthetic variations in Figure 2a It shows that most of the edits change only one attribute on average. To obtain a more balanced coverage, we propose using a k-nearest neighbors (k-NN) sampling strategy as follows, (1) sample two multi-attribute threshold boundaries at random, (2) uniformly sample attribute locations from both boundaries (representing start and end) (3) find k-nearest neighbors (k- 30) of the sampled transition from the available edit pairs and randomly select one of them. We find the edit pairs sampled with the k-NN strategy to be much more evenly distributed (Figure 2bp.
L Anchor Conditioned Inference To reduce content drift in multi-step rewriting, we propose to include the original text in the context of the LM's prompt which we call anchor conditioning. During multi-step inference, when a new output sequence is generated, we still retain the original text in the context of subsequent rewrites. To train the rewriter LM with anchor, we augment D's edit pairs (Na + yb) by sampling an anchor Ve such that R(ye: Vo: C.T 2 R(y.. Va. C.Ta-b). In the experiments, we evaluate the effectiveness of this anchor conditioning in content preservation over multiple rewrites.
4.1 Text Style Transfer Evaluation
For the Text Style Transfer task on Yelp Reviews, we design a fixed inference budget evaluation setup, i.c., cach method will have a fixed number of allowed rewrites to satisfy all multi-attribute constraints. Subsequently, we construct a test set of 250 total reviews (10 from each of the 25 sentiment and complexity threshold combination)). The task to generate 25 attributed paraphrases for every test review within 5 rewrites (250 x 25 x 5 R 31.2K total inference budget). For every baseline and our models, we compare the constraint satisfaction success rate of multi-step inference strategies: best-of-N, naive rewriting, and reward-prioritived rewriting. We report the average satisfaction rate, fluency, and embedding similarity of paraphrases that satisfied the given constraints.
Basclines and MACS models As a bascline, we use few-shot prompted Llama2-7B (Touvron et al., 2023) and Llama3-8B Mela 2024) models as fine-grained editors for our Text Style Transfer lask. For every transition from one threshold combination to another, we find 10 edit pairs as few-shot demonstrations (a total of 25 25 x 10 6250 edit pairs). For fine-tuning methods, we use a smaller TinyLlama (Zhang) et al. 2024] 1.1B parameter model as the multi-attribute rewriter LM. Among finetuning baselines, we compare with Control Tokens (Keskar et al. 2019h that simply convert each threshold combination into style tokens. We allocate 10 total style tokens (5 for each attribute) and simply append the style tokens of the target threshold windows in the prompt along with the target response as follows: ya [Sentiment
TTe I.M-generated variations do not always agree with the target thresholds but provide a good spread of paraphrases in the multi-attrbbute space. We filter out variations that yield fluency soore or embedding similarity score 0.7.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table I: FINECS Text Style Transfer task evaluation: Paraphrase each test review 25 times into fine- grained Sentiment and Complexity threshold constraints while maintaining fluency and content preservation. We compare the Control Tokens baseline with supervised fine-tuned and wBC models each with 3 different inference strategies: Best-of-N, naive rewriting, and roward-prioritized rewriting. We report the average satisfaction rate for each model and inference strategy and average fluency and embedding similarity of the paraphrases that satisfied the constraints. Takeaway: Our proposed sward-prioritized rewriting combined with anchor conditioning (L) and wBC obtains the highest satisfaction rate. However, its differences with (L) and SFT are not statistically significant', indicating that anchor conditioning leads to most improvement in multi-step editing.
Inference Type
Bost-of-N
Naive Rewriting
Train Sample
Satisfaction
Fluc-
Emb.
Satisfaction
Flue- Emb. Sim.
Satisfaction
Fluc- Emb. ney
Method
Rate"
ney
Sim.
Rate*
ney
Rate*
Sim.
baselines
10-shot Llama2-7B
.478 + .141
.93
.85
10-shot Llama3-8B
.594 + .10
.93
.85
Control Tokens
random
.774 1 .083
.92
.80
.783 1 .068
.78
.792 .061
.93
.79
Control Tokens
k-NN
.888 + .063
.92
.80
.809 + .046
.91
.78
.828 + .048
.92
.79
SFT
K-NN
.824 + .061
.92
.80
.809 + .056
.92
.78
.827 .054
.93
.79
MACS
SFT + BBC
k-NN
820 +: .072
.92
.81
815 + .063
.92
.79
.835 +: .051
.93
.80
t + SFT
K-NN
.833 + .065
.92
.81
.849 + .054*
.80
.847 + .052*
.92
.80
+ SFT + wBC
k-NN
835 = .65
.92
.81
.840 + .061
.92
.80
.855 + .059*
.92
.80
Token] [Complexity Token] yo. For LMs trained with MACS, construct a text-only prompt that doesn'l. use any special tokens as follows:
Review:
Review's Sentiment: (Va) and Complexity: c2(Ua)
Paraphrase the review such that its Sentiment is within: 11,2-5 and Complexity is within:
22
Paraphrased Review: yb
We prepend the above text prompt with Ye and its attribute locations for anchor conditioning (+) training.
We train both control tokens and text-prompted models with supervised fine-tuning (SFT) for 200K steps and a batch size of 16. For control tokens, we experiment with both randomized and k-NN edit pair sampling, whercas we only use k-NN edit pair sampling for text-based models. For weighted behavior cloning (wBC) objective, we continue training the supervised finetuned models for an additional 50% steps (100K steps).
4.2 Text Style Transfer Results
We present the performance of all baselines and MACS models with the three inference types in Table I Among few-shot methods, the newer Llama3 model outperforms Llama2, however, both struggle to achieve very high satisfaction rates and show high variance across different threshold combinations. In comparison, the control tokens- based finetuning baseline works much better than few-shot prompting and gains a further boost in overall satisfaction rate when trained with our proposed k-NN edit pair sampling. Interestingly, naive rewriting is occasionally worse than best-of-N inference, indicating that models may not consistently move toward the threshold boundaries. The reward-prioritized rewriting improves over naive rewriting by leveraging the external scorers and our reward function to guide its search process.
Among our methods, the text-only finetuned model matches the performance of the control tokens bascline when trained with the proposed k-NN edit pair sampling. For models without anchor conditioning, we notice that multi-step rewriting can drift away from the original content indicated by drop in embedding similarity when switching from best-of-N to rewriting. Anchor conditioning (L) resolves the content drift problem and subsequently improves satisfaction rate and final embedding similarity when employing rewriting inference strategies. Finally, we notice that models trained with wBC outperform their counterpart SFT-only models
8

---

Published in Transactions on Machine Learning Research (04/2025)
Avg. satisfaction 83.47%
Avg. satisfaction 85.47%
1.00
1.00
0.84
0.90
0.90
0.81
0.86
0.95
0.82
0.86
0.89
0.80
0.91
0.95
0.90
0.90
0.86
0.86
0.92
0.87
0:91
0.92
0.89
0.92
0.85
0.85
Sentiment
Sentiment
0.86
0.86
0.84
0.73
0.76
0.80
0.88
0.90
0.83
0.74
0.79
0.80
0.75
0.75
0.77
0.72
0.73
0.82
0.85
0.73
0.83
0.70
0.70
0.68
0.85
0.89
0.85
0.92
0.65
0.74
0.86
0.93
0.88
0.92
0.65
0.60
0.60
-2
-1
1
2
.2
-1
1
2
Complexity
Complexity
(a)
E SFT + wBC Best-of-N inference
(b) L + SFT + wBC Reward-Prioritizz inference
Figure 3: Comparing best-of-N vs. reward-prioritized inference constraint satisfaction rate of Sentiment and Complexity attributes. Takeaway: reach constraints i.e, edges of the satisfaction matrix.
inference has better satisfaction rates in harder to
T= Sentiment (1, 1.5,, Complexity (-2, -1.5)
T= Sentiment (1, 1.5), Complexity (-5, .))
T Sentiment [1, 1.5), Complexity (1.5, complete
The restaurent's culnaay ofterings dissppainment, with prosoiutto salad and French dip being the acla bright spota. The burgers the burger palty posssssing unique flavor that was unpleasent upan consumptioon awarded the restaurant one star solety for the service, as the from of of-house standards were for above average. However, the bathrooms were abyomal, and the coffee was utterly devoid flovor, which added to the overal disappointent.
This place was a complete dlsappoinment, The prosoiutto saled was the only redeoming quality, but wasn't enough to make up for the diegusting burgers and spicy chill. The French dip was mees, and couldn't even finish it. The service was average, and the bathrooms were dirty. The coffee was mediocre, and would not recommend this place to anyone.
The restaument diessterr Some disbes almost mediocre, like the prosciutto asled and Frunch dip. The burger was tamril,, and the hot nature of made uncomtortuble to cat. The service good, but the bathrooms smolled like sanitorium The coffee was bad, and the hard surfaces wore uncomfortable.
travesty, with
Sentiment 1.3, Complexity Fluency ,9,, Similarity .84
Sentiment 1.2, Complexity -.48
C Sentiment Complexity 1.7, Fluency .94, Similarity .88
Flueney ,96, Similarity
T= Sentiment (2.5, 3.5), Complexity (1.5.
T= Sentiment (2.5, 3.5), Complexity (-2, -1.5)
Original Review
My experience eatery was charactertoed
lows. proeclutto bossting captivating were nighe echibiting delightiul burgars
eatery had dual nature, while others faltored. prosolutto salad and French disappointmant, outstanding but the burgars detracted from unplessent spicy experience, bathrooms were very clean, but the coffee was only possoble.
dishes excelling
This mestarmnt is good but not great. Iiked the prosciutio axttd and the Franch dip. The burgers were bed, though. They ware too spicy and dide't taaie good. The service was akey. The battrooms were nica. The coffee was avarage.
spice expactations, sullering
Service adequate.
adequate,
notable
Unfortunately, the coffee
satistying experience that entirely without Despite shortoomings, shall but with disseming palate
C Sentliment 3.1, Complexity -2, Fluency Similrtty 82
C Sentiment 3.1, Complexity .82,
refined level cophistication.
Fluancy 933
C Sentiment 2.9, Complexity 1.9,
Fluency 94, Similarity .90
T Sentiment (4.5, 5) Complexity (-2, -1.5)
= Sentiment (4.5, 5) Complexity (-5,
T Sentiment [4.5, 5) Complexity (1.5.
This restaurent a culinary mastarpicce, with each dish showoasing pure and rich flavor profile that nothing short of beeathtaking. The prosolutto salad and Fronch dip are particular highlights, with the former boasting subtie yet satistying crunch and the latter a bold and sproy kiok that will leave you wanting mone, Desphte the burger being a bit out of place, the meal is delight from start to finis,, with the service being frinndl and attentive. The clearlinses of the bethrooms a bonus, and the coffee is nothing short of subline.
This metaurant exceeded my aspectations. lowed every single dish, from the proscittoo sslad to the French dip. The burgers were cooked to perfection, and the spicy kick was delicouus The service was exceptional, and the bethrooms were clean and wal-maintaied. The coffee was rich and flavorful, and it's dlear that they take pride in their product. Overall, would highy resommend this restaurant.
This place awesome! love the prosoiutto salad and the French dip. The service great, and the bathrooms are clean. The coffee is good, and it's like Starbucks or something better Thy it and add to your dtnner.
Sentiment 4.9, Complexity Fluency ,3, Similarity. .75
C Sentiment 5, Complextly-2 -11, Fluency 96, Similarity
C Sentiment 4.9, Complexity 1.7,
Fluency .95, Similarity .85
Figure 4: Showing 8 attributed paraphrases of a test review for varions thresholds generated by L + SFT + wBC model with roward-pri rewriting.
in reward-prioritized rewriting. However, the performance differences are not statistically significant (p z 0.2) according to the two-proportions z-test (Fleiss et al.] 2013). We show the detailed threshold satisfaction matrix of best-of-N vs. inference achieved better satisfaction rate than best-of-N in most of the threshold constraints, especially for harder-to-reach threshold constraints (corners of the threshold satisfaction matrix). We also present example generations from the best method in Figure 4 showcasing the difficulty of the task.
inference for our wBC model in Figure 3 Reward- privoit-iiooitizeed
9

---

Published in Transactions on Machine Learning Research (04/2025)
5 FineCS Protein Design
Unlike language, where text can be paraphrased in many different ways, fine-grained editing in protein space is challenging due to (a) the uneven distribution of assay-labeled data across multiple attributes and (b) the existence of limited potential solutions in nature for a given set of attribute constraints (Sternke &
Karpiak 2023). Moreover, for any given set of constraints, obtaining multiple novel and diverse candidates is important to maximize the chances of success in wet lab experiments (Jain et al. 2022 To evaluate fine-gres ane-grained control of MACS framework in protein space, we create FINECS Protein Design, where the task is to simultancously modulate fluorescence and folding stability of Green Fluorescent Protein (GFP), (a protein widely investigated and used as biosensors in life sciences research).
Fluorescence and folding Stability Evaluators We obtain the dataset of x 51.7K mutants of the GFP wild-type (i.e., the protein sequence that occurs in nature) (Sarkisyan et al. 2016 Gonzalez Somermeyer et al. 2022). The dataset contains fluorescence levels on logarithm 10 scale for every mutant. sequence E [1228, 4.12]. Due to a lack of assay-labeled data for a second attribute, we calculate the theoretical folding stability values (AAG or ddG) of each mutant with respect to the wild-type structure using FoldX Sottware(Schymkkee et al., 2005) The wild-type ddG is 0 and any mutant with negative ddG more stable than wild-type. The overall distribution of ddG for all the mutants is E [-5.66, 60.75]. We train ESM2-based regressors (Lin) et al. 2023) evaluators for both attributes using mean squared error loss g fuoorscencc and ddG are 0.974 and 0.987 respectively To
The test set correlation for
Attribute Distribution and Edit Pairs We plot the distribution of log fluorescence and ddG of all the GFP mutations in Figure g in the Appendix. Unlike language data, protein mutants are even more unevenly distributed across the multi-attribute landscape, with the bulk of the mutants clustered near the wild-type (WT) GFP sequence (which has 3.72 log fluorescence and ddG). To effectively navigate this skewed distribution, we define four threshold boundaries in log fluorescence, (< 3.0) very low, (3.0, 3.4) low, (3.4,3.7) medium, (> 3.7)- bright and four threshold boundaries in ddG, (< 0) more stable than WT, (0.0, 0.5)- as stable WT, (0.5, 2.0) slightly destabilized, (> 2.0) highly destabilized (Dill et al. 2008).
The limited viable solutions in certain regions (< 10% of proteins have < 0 ddG) make the protein design task very challenging, especially when learning from an offline dataset of mutations. Here, all GFP mutants are considered poraphrases of each other, and thus, total possible edit pairs are v P P31.7K To train the LM rewriting models to edit all possible directions, we employ the following edit pair sampling strategy: (1) pick two multi-attribute threshold boundaries, (2) sample a mutant at random from both of the selected threshold constraints and (3) construct an edit pair by trcating the first as the source and the second as the larget mutant.
5.1 Protein Design Evaluation
Unlike the Style Transfer task, where we only care about one solution for each constraint, the goal of the Protein Design task is to find the maximum number of new mutants in every multi-attribute constraint under fixed inference budget. For each threshold constraint, we initiate multiple random walks of different lengths starting from wild-type GFP sequence (WT 91... + ya). We assign a total 3000 inference budget which results in (1) 3000 x I-hops, (2) 1000 x 3-hops, and (3) 300 x 10-hops random walks. We expect duplicated predictions under specific constraints since cortain regions will have naturally very few solutions. Among the 3000 predictions in each inference method, we calculate the lotal success rale: ratio of distinct mulants that. satisfy the constraints according to our evaluators and unique success rate: ratio of unique successful mutants outside of the offline training data. We also compare with reward-prioriticn the LM generated intermediate edit y/ + Ve+1 is only retained if it moves closer to the threshold constraints, > R(y, WT, C,T). We experiment. with reward-prioritized walks in 1000 x 3-hops
walks from wild-type ($3) where
i.e. R(H++1, (T,++1,W,CC
Foldx uses empirical force field determine the effect of nutations the protein folding, We note that FoldX-generrated values are not an accurate representation of real experimental folding stability and only used as a proxy. We follow best practices recommended in the previous rescarch and compute ddG on am average of five FoldX calculations Chan et al. 2021 We divide the dataset into 50% train, 15% validation, and 35% set for both fluorescence and ddG attributes. 10 1mmplemmntttion details in Appendix B.1
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 2: FINECS Protein Design task evaluation: Starting from GFP wild-type, discover the maximum possible unique mutants across 16 multi-attribute constraints of log fluorescence and ddG within 3000 total inferences. We compare the ProtGPT2 LM editor fine-tuned with SFT and wBC with 5 different inference strategies: three random walks and two reward-prioritized walks with different hop lengths. After discarding all duplicate solutions, we report the average rate of mutants that satisfy the threshold constraints (total success rate) and the average rate of successful mutants that are outside the training data (unique success rate). We also report the average edit distances between all pairs of successful mutants for each method. Takeaway; wBC \w entropy, another variant of MACS method, discovers the most number of novel mutants. However, the differences wetween different inference methods are not statistically significant".
Total
Unique
Edit
Total Success Rate
Unique Success Rate
Edit Dis- tance
Total
Unique
Edit
Success Rate
Success Rate
Dis- tance
Success Rate
Success Rate*
Dis- tance
Random
Recombine
Unique Recombine
baseline
8.2
4.4 + 2.2
36.5
30.0
+ 1.6
39.5
39.5
3.8 + 1.6
SFT
SFT + WBC
SFT + wBC \w entropy
random walk 3000 1-hop
41.3
38.6
4 2.0
43.6
40.2
4.0 1.9
44.3
41.1'
4.4 2.3
random walk 1000
3-hop
41.3
38.6
4.2 + 2.1
43.6
40.1
4.0 1.9
44.5
41.2!
4.5 + 2.3
random walk 300
10-hop
41.8
39.0
4.2
2.1
43.8
40.4
4.0 1.9
44.7
41.5'
4.5
priority walk 1000 3-hop
41.6
4.2
43.5
40.1
4.0 1.9
44.4
41.1'
4.5 2.3
priority walk 300 x 10-hop
41.1
38.4
4.2 1:
2.1
43.5
40.1
4.1
44.6
41.5'
4.4 2.3
and 300 x 10-hops settings. In total, for 16 multi-attribute threshold constraints of log flnorescence and ddG, we have a total budget of 16 x 3K 48K decoding in every inference method.
Basclines We compare our method with two baselines: (1) Random proteins are randomly mutatod based on the edit-distance distribution of the train-set sequences and (2) Recombine - a previous method that samples new diverse sequences by shuffling and merging pairs from an initial seed set (Otwinowski et al.j 2020: Sinai et al.) 2020). Recombine can sample many duplicate sequences when the seed set is small (certain threshold combinations have fewer than 100 original sequences). We also compare with stronger variant of Recombine where we ensure that every 3000 newly sampled sequences for a specific multi-attribute constraint. are unique and outside the seed set. We call this stronger bascline Unique Recombine.
MACS training We finetune the ProtGPT2 LM (Ferruz et al. 2022 which is a 738M parameter protein language model as the rewriter for this task. Since ProtGPT2 does not have English words as tokens, we prompt the mutant edit pair to the LM as follows: Ya [A Fluorescence] eı (Ba) [A ddG] c2(o) [Target Fluorescence]
[Target ddG] /2.a>b [edit] yb, where intermediate key-words are special tokens added to the model's vocabulary. Using the edit pair sampling strategy described earlier, we train ProtGPT2 with SFT for 20K steps with batch size 16 and learning rate 10-4, We then further continue finetuning with the wBC objective for additional 10K steps and learning rate 10-5, 5, Since want to encourage the LM editor to generate diverse candidates in this task, we separately also train with wBC objective augmented with entropy penalty (coefficient 7 = 0.05) 12 We present additional implementation details for our methods and the baselines in Appendix B.2
5.2 Protein Design Results
We report the evaluation results of baselines and different variants of MACS in Table 2 Random mutation shows the worst performance as expected, while Recombine is a strong baseline that finds more unique and successful mutants. With Unique Recombine, we establish an upper bound on the baseline's performance by only retaining unique sequences. However, the offline wBC model outperforms both the Recombine baseline and the SFT model across all inference strategics. When augmented with entropy penalty, we observe boost in success rate for the wBC model and larger spread of edit distances indicating more diverse
/httpe://uuggingffccccco http://gnaccccccooattt/////ooogo
W. /e entropy M) (. 7(PM (vnle 0..2. Wa. C(va), Ta-se) PMr (url,, Va. C(ya), T Ta-i)))
11

---

Published in Transactions on Machine Learning Research (04/2025)
Top Mutotional Hotepots for 3.7 log Flucrescence and + deG
1.0
Distibutioon 53725
1300>
20090
+2.7
2002
1627
1824
2302
10090
1382
2.034
4644
811
190
1523
5003
<300
20126
402
100
<3.0
2927
700
152
542
528
0.32
1023
+20
52
0.50
0.545
<80
Train Set (hotai 585)
LM Génantted notal 1824)
0.0
(a) Train Mutant Distribution
(b) SFT + wBC \w entropy priority 300 x 10-hop
(c) Comparing Mutational Hotspots
Figure 5: Analyzing new mutant discovery of reward-prioritized walk compared with training set distribution. Takeaway: Even with very few training instances in many of the regions, LMs trained with MACS discover many novel candiclates. Analysis of the top 15 mutational hotspots reveals that LMs can extrapolate beyond the mutational patterns seen in the training set.
mutants. Although reward-prioritized and naive multi-hop walks yield the best performance, we do not notice significant difference between different inference strategies with two proportions z-test.
Finally, we compare the distribution of train set mutant vs. the newly discovered mutants via the reward- prioritized walk (300 x 10-hop) from wBC \w entropy model in Figure 5 Despite small sample sizes of training data in cortain regions, our mothod can extrapolate beyond the original training set and find diverse sequences even with offline training. Finally, when comparing mutational hotspots and their distribution across GFP structure, our LM-generated sequences show a different distribution and occasionally novel mutations compared to train set sequences, shown in Figure 5c (and Figure g in the Appendix).
6 Conclusion
We create Multi-Attribute Constraint Satisfaction (MACS) framework to cheaply train LMs fine- grained editors by sampling edit pairs from offline sequential datascts. We also create new Fino-grained Constraint Satisfaction (FINECS) benchmark to evaluate our method, comprising two challenging fine-grained controlla- bility tasks. In the FINECS Text Style Transfer task, LM editors trained with weighted behavior cloning paired with proposed k-NN edit pair sampling, and multi-step reward- paraadd-rioiiizeed editing outperform their SFT counterparts and other inference methods. We boost its performance further with anchor conditioning and achieve the highest constraint satisfaction rates compared to previous fine-tuning and few-shot, prompted baselines. Interestingly, in the FINECS Protein Design task, MACS can train protein language models to discover novel proteins outside the training data with high success rates while highlighting different mutational hotspots Our study demonstrates the potential of LMs as fine-gre fineggrained writing assistants and protein engineering models that can aid in the creation of novel proteins with fine-grained properties.
References
AI@Meta. Llama model card. 2024. URL http::///gggaab..ooo///ee////aaa
CARD.md
Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. RLAF: Generating natural language feedback with reinforcement learning for repairing model outputs. In and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting
Anna Rogers, Jordan of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7716-7733, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/vv /1153//v//// 2211//223aa0 ael-llnng.40 long.427. URL https://acllnnhooogg..ooo org/ http:/222.....11oog/.e acc-llogg.e.. 427
12

---

Published in Transactions on Machine Learning Research (04/2025)
Pictro Amat and Riccardo Nifosi. Spectral "fine" tuning in fluorescent proteins: The case of the gfp-like chromophore in the anionic protonation state. Journal of Chemical Theory and Compulation, ((1))497-500 2013. doi: 10.1021/tt300455. URL https://doi.ors 011g/1001022/33007455
PMID: 26589050.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Picter Abbecl, and Wojeiech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advonces in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: //pooceed///oeeiggss http:///r/22//111e///faaaaaaaaaa
neurips... http:ppp.cppppprrreeee
Paperr-ap..... pdf
Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, and Mark Riedl. Leftover lunch: Advantage-based offline reinforcement learning for language models. In The Twelfth Internaiiona Conference on Learning Representations, 2024. URL https:// wttp://oooooo///1-000-ooP---
Benedetta Bolognesi, Andre J Faure, Mircia Seuma, Jörn M Schmiedel, Gian Gaetano Tartaglia, and Ben Lehner. The mutational landscape of prion-like domain. Nature commumications, 10(1):41183) 2019.
Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick F Riley, George M Church, Lucy J Colwell, and Eric D Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology. 39(6):691-696, 2021.
Alvin Chan, Ali Madani, Ben Krause, and Nikhil Naik. Deep extrapolation for attribute-enhanced generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neurol Information Processing Systems, 2021. URL
Matthew Carter Childers and Valerie Daggelt. Insights from molecular dynamics sinulations for computational protein design. Molecular systems design & engineering, 2(1):9 -3-, 2017.
Michael Collins and Terry Koo. Discriminative Reranking for Natural Language Parsing. Computational Linguisticss, (1(1):25-70) 03 2005. ISSN 0891-2017. doi: (0.16220091201053666 URL https://doi org/10. 10.
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Represendations, 2020. URL http:://pppmeeeiieeeee///forrrre H1edEyBKDS
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-voriffcatiion reduces hallucination large language models, 2023.
Ken A. Dill, S. Banu Ozkan, M. Scott Shell, and Thomas R. Weikl. The protein folding problem. Annual Review of Biophysics, (7(1):289-311 2008. doi: (8.1146/anmun http://aww..ooooos........ //doi. 10. 1146/annuree biophys. 092707777999.... 153558
URL https
PMID: 18573083.
Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Maclasa: Multi-aspect controllable text generation via efficient sampling from compact latent space, 2023.
Yihao Feng, Shentao Yang, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, and Huan Wang. Fantastic rewards and how to tame them: A case study on reward learning for task-oriented dialogue systems. In The Eleventh International Conference Learning Representations, 2023. URL https://openreviee http://ww..e.rrrrrroooooooaaaaaa
Noclia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 a deep unsupervised language model for protein design. Nature communications, 13(1):434 2022.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. Statistical methods for rates and proportions. john wiley & sons, 2013.
Hector Garcia Seisdedos, Tal Levin, Gal Shapira, Saskia Freud, and Emmanuel D Levy. Mutant libraries reveal negative design shielding proteins from supramolecular self- self-assembly and relocalization in cells. Proceedings of the National Academy of Sciences, 119(5):c201117111 2022.
Sayan Ghosh, Zheng Qi, Suigdha Chaturvedi, and Shashank Srivastava. How helpful is inverse reinforcement learning for table-to-text generation? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 71-79, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 10/0//11/2000 wtt/2221alllss/aaaaeaee acl-short.11. URL
https://aclantthooogy.oooo org/ 2021
o0.. acl-short. 11.
Louisa Gonzalez Somormeyer, Aubin Fleiss, Alexander S Mishin, Nina G Bozhanova, Anna Igolkina, Jens Meiler, Maria- Elisenda Alaball Pujol, Ekaterina V Putintseva, Karen S Sarkisyan, and Fyodor A Kondrashov. Heterogeneity of the gfp fitness landscape and data-driven protein design. eLife, 11:e75842, may 2022. ISSN 2050-084X. doi: /77554/eiife.588/ URL https://doi org/10. wttp://0..55//......eee..
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Represenlations, 2024. URL htttps:/oopenrevier net/ www.::eetfforriie wtt.:///ooorrrr/dddx00883.13-
Yuxuan Gu, Xiaochong Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. A distributional lens for multi-aspeet controllable text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 1023 1043, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/
URL /ettps://aclante https:///aclathoolog) org/2022. emnlp-main. 67
Haiwei Henry Guo, Juno Choc, and Lawrence A. Loeb. Protein tolerance to random amino acid change. Proceedings of the National Academy of Sciences of the United States of America, 101 25:9205 10, 2004. semanticscholar URL https: http:s///ppp..sssttcccccol.a... org/ :ttg/CorpuSI:::::0 7391571
Skyler Hallinan, Faeze Brahman, Ximing Lu, Jachun Jung, Sean Welleck, and Yejin Choi. STEER: Unified style transfer with expert reinforcement. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7546 7562, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 2ww....iniingsoor..
URL https :
//aclanthology... www.gyy..rg/2....../ findings-emnlp. 506
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg- Kirg-Kirkpatrick. A probabilistic formulation of unsupervised text style transfer. In International Conference on Learning Representations, 2020. URL https: het//forum2idd=HJAACCCEE
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jemnifer Listgarten. Learning protein fitness models from evolutionary and assay-labeled data. Nature Biotechnology, 40(7):1114-1122, Jul 2022. ISSN 1546-1696. URL htttps://doi. wrw..o..g....... 111.:001111/11110000000001111110
doi: 10.
Zhe Hu, Zhiwei Cao, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Jinsong Su, and Hua Wu. Controllable dialogue generation with disentangled multi-grained style specification and attribute consistency reward. FEEF/ACM Trunsactions on Audio, Speech, and Tanguage Processing, 14:188-199, 2023. doi: 10.1109/
Wanzhi Huang, Joseph Petrosino, Marc Hirsch, Peter S. Shenkin, and Timothy Palzkill. Amino acid sequence determinants of 3-lactamase structure and activity. Journal of Molecular Biology, 258(4))688-77))) -03, 1996. http:////.....0/////...0000...00 1996...... URL buttps://vee http://wwwwwcccncciicc.cooo
ISSN 0022-2836. doi:
science/ cctaac///ar.ccc////////202288899
14

---

Published in Transactions on Machine Learning Research (04/2025)
Moksh Jain, Emmanuel Bengio, Alex Hernandez- Garcia, Jarrid Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Fkbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceediings of Muchine Learning Research, pp. 9786 9801. PMLR, 17-23 Jul 2022. URL htttps://procceednggs mlr. wttp:/sswwrrrrrss/eereeeeee ppess/v162. jain22a..........aaaa... .html.
Marcin Junezys- Dowmunt, Roman Grundkiewiez, Shubha Guha, and Kenneth Heafield. Approaching neural grammatical error correction as a low-resource machine translation task. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pp. 595 606, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/ 10.18653/v1/N14. /1/118 1055. URL http::/aaccaantoooo.oooo////////
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. CTRL - A Conditional Transformer Language Model for Controllable Generation. arXiv preprint ar 1rx:::9000 000.8, 2019.
Andrew Kirjner, Jason Vim, Raman Samusevich, Shahar Bracha, Tommi S. Jaakkola, Regina Barzilay, and lla R Fiete. Improving protein optimization with smoothed fitness landscapes. In The Twelfth International Conference on Learning Represenlations, 2024. URL btttps://pperrevier net mett//forummddrrxl222222at
Kalpesh Krishna, John Wieting, and Mohit. Tyyer. Reformulating unsupervised style transfer as paraphrase generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 737 762, Online, November 2020. Association for Computational Linguistics. doi: 11//666///1/222.emm//pp. //aclanthology. www.anooooo.or////0000 org/2020. amnlp-main.95 55
URL https
Sachin Kumar, Eric Malmi, Aliaksei Severyn, and Yulia Tsvetkov. Controlled text generation as continuous optimization with multiple constraints. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and
J. mortman Vaughan (eds.),, Advances in Neural Information Processing Systems, volume 34, pp. 14542- 14554. Curran Associates, Inc., 2021. URL httppp///pproeeeingggeerrr.....o ottp://rrrrrpeecpppppeeeeeeeeppp hto/:/ttw//aacaa20t0000////66622
pdf
Sachin Kumar, Biswajit Paria, and Yulia Tsvetkov. Gradient-based constrained sampling from language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2251-2277, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1 2t0p://111/2222em/-mmmaa 144. 144. URL org/2022 emnlp-main. main. 144
Philippe Laban, Jesse Vig, Wojeiech Kryscinski, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. SWiPE: A dataset for document-level simplification of Wikipedia pages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Länguistics (Volume 1: Long Popers), pp- 10674-10695, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v1 /2033//v1/222.aa// lollo..559
URL https://aclaatthollge/ org//222
acl-long 596
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multiple-atrribute text rewriting. In International Conference on Learning Representations, 2019.
URL https::/openrerier.. http://wwww.ee.oooo.o/iecdttaaaa
Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generale: a simple approach to sentiment and style transfer. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1865-1874, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10. 18665/v11/18-111111 URL http:://belaattooogg..r////
15

---

Published in Transactions on Machine Learning Research (04/2025)
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 4328 4343. Cur- ran Associates, Inc., 2022. URL https::/pproceeddngg nwwrass...........p... babbe:/ceeeeeebbc222bbooa----a--
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smettnnin Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Exolutionary-scale prodiction of atomic-lovel protein structure with a language model. Science, 379 1130, 2023. doi:
URL https://wwr https://ww.sccience org. /oi/ ade277 Earlier versions as preprint: bioRxiv 1022.07.20.50090
Guangyi Liu, Zeyu Feng, Yuan Gao, Zichao Yang, Xiaodan Liang, Junwei Bao, Xiaodong He, Shuguang Cui, Zhen Li, and Zhiting Hu. Composable text control operations in latent space with ordinary differential equations. Xix preprint arXiv:2208.0000 2022.
Xin Liu, Muhammad Khalifa, and Lu Wang. BOLT: Fast energy-based controlled text generation with tunable biases. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proccedings of the 61st Annual Meeting of the Association for Compulational Lánguistics (Volume 2: Shorr Papers), pp- Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2100 /ttp////222....sorr..ttt
URL https://aclanthollegg 2rtp://atttooooooor/22233333 wwwp://o2..lllsoort.... 18
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [b(bert A robustly optimized {bert) pretraining approach, 2020.
URL httpp:://ppanrrvveee.eeftetoo
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. QUARK: Controllable text generation with reinforced unlearning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps://openrevice http:///pppppeeeeeeeeeeerrrrrnne iite/EErrrmiidd-SHaaaaaauab
Fuli Luo, Peng Li, Jie Zhou, Pengeheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI 2019, 2019.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. Powertransformer: Unsupervised controllable revision for biased language correction. In EMNLP, 2020. URL https://www.accled wwwg/aggglllaa. 2020. emnlp-main 602
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhunoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majunder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-foedback, 2023.
Fatemehsadat Mireshghallah, Kartik Goyal, and Taylor Berg-Kirkpatrick. Mix and match: Learning-fres controllable text generationusing energy language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lánguistics (Volume 1: Long Papers), pp. 001-415, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1 /22022.acl-looog/eeeee 31. URL https::/accaattholl
org/ 2tgg/022..eo cong.nng... .31
Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prodiction. In D. Leo, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL http:://prroceeeing...eom www.pps..ccappppppppppp ftte://ppw//oofff////8o0eee2eee2
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello,
16

---

Published in Transactions on Machine Learning Research (04/2025)
Jake Berdine, Gabriel Bernadett-Shapite Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Tuisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecosset, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johamnes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantamu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanilscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanic Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichrr Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Hcnrique Ponde do Oliveira Pinto, Michael, Pokorny, Micholle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Fric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Tan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, 1lya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Jakub Otwinowski, Colin H. LaMont, and Armita Nourmohammad. Information- optimization with natural selection. Entropy, 22(9), 2020. ISSN 1099-4300. doi: 10.3300/e22090967. URI. https:
//www.mdpi.. 100m/1099-43300 040199994000/////////83
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:://penrevvie http:///ppaaeeeeeeeeeffoomotdddd
Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei Han, and Lianhui Qin. Structured chemistry reasoning with large language models, 2024.
17

---

Published in Transactions on Machine Learning Research (04/2025)
Vishakh Padmakumar, Richard Yuanzhe Pang, He He, and Ankur P Parikh. Extrapolative controlled sequence generation via iterative refinement. Fortäeth International Conference on Machine Learning (TCML., 2023.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback, 2023.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. Style transfer through back-translation. In Iryna Gurevych and Yusuke Miyao (cds.), Proceedings of the 56th Annual Meeting of the Association for Compulalional Linguistics (Volume 1: Long Pupers), pp. 866 876, Melbourne, URL.
Australia, July 2018. Association for Computational Linguisties. doi: 2011653/v1/P18-10/ htttas:/aaclanchollog..ooo
Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy-basede constrained text gencration with langevin dynamics. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 9538 9551. Cur- ran Associates, Inc., 2022. URL https::/ppooceeddggs wwt.:nwuuur...aappeeeee fttes//aperr/222//EI 6et5ddaffttsceee00tttrrd444eerr-
Gulam Rabbani, Ejaz Ahmad, Abrar Ahmad, and Rizwan Hasan Khan. Structural features, temperature adaptation and industrial applications of microbial lipases from psychrophilic, mesophilic and thermophilic
origins. International Journal of Biological Macromolecules, 225:822-888 839, 2023. ISSN 0141-8130. doi: https:
//doi.org/10.1000//..bbbbbb wit.:::ommmmm............. pi1/500111813022227776
Govardana Sachithanandam Ramachandran, Kazuma Hashimoto, and Caiming Xiong. [CASPI] causal-aware safe policy improvement for task-oriented dialogue. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lánguistics (Volume 1: Long Popers), pp. 92-102, Dublin, Ireland, May 2022. Association for Computational Linguistios. www.0.....oo..........
doi: 10.18653/v1 https//1112222clll//// 8. URL https://acclatthooogyy
Zhizhou Ren, Jiahan Li, Fan Ding, Yuan Zhou, Jianzhu Ma, and Jian Peng. Proximal exploration for model- guided protein sequence design. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18520-18536. PMLR, 17-23 Jul 2022. URL https://proceeddng/ mlr. wttppp//ee//rre222222 hmml html
Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. TeX:SETT:: Few-shot text style extraction and tunable targeted restyling. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3786-3800, Online, August 2021. Association for Computational Linguistics. doi: URL https: wwg.:/llooooooo.o/oo///// bbg. 293
10.18653/
Giuseppe Russo, Nora Hollenstein, Claudiu Cristian Musat, and Ce Zhang. Control, generate, augment: A scalable framework for multi-attribute text generation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguisties: EMNLP 2020, pp. November 2020. Association for Computational Linguisties. doi: 2220//3//1122.mmmiiinn https://acllnthhooog..oor
Online, URL
findings-ennnnl
Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Tvankov, Nina. Bozhanova, Mikhail Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature, 533(7603))397 401, 2016.
Karola M. Schlinkmann, Annemarie Honegger, Esin Türeci, Keith E. Robison, Daša Lipovšek, and Andreas Plückthun. Critical features for biosynthesis, stability, and functionality of g protein-coupled receptor uncovered by all- versusuullll mutations. Proceedings of the National Academy of Sciences, 109 9815, 2012. doi: 10. 10.1073/ /pa.....sTT 1202107109. URL https: www.. http:/ppwa..r///ooo//aa/////.//p 1202107109
18

---

Published in Transactions on Machine Learning Research (04/2025)
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousscau, and Luis Serrano. The foldx web server: an online force field. Nucleic acids research, 33(supp 2):W382-WW W388, 2005.
Nathan C. Shaner, George H. Patterson, and Michael W. Davidson. Advances in fluorescent protein technology. Journal of Cell Science, 120(24):4247-4200 12 2007. ISSN 0021-9533. doi: 10.1242/jes.005501. URL
https:///di. org/ 10. 1ee2/ 005.80001
Wen-Jun Shen, Hau-San Wong, Quan- Wu Xiao, Xin Guo, and Stephen Smale. Introduction to the peptide binding problem of computational immunology: new results. Foundations of Computational Mathematics, 14:951-984 1944, 2014.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-sevente Conference on Neural Information Processing Systems, 2023. URL
Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D. Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design, 2020.
Tyler N Starr, Allison J Greaney, Sarah K Hilton, Daniel Ellis, Katharine HD Crawford, Adam S Dingens, Mary Jane Navarro, John E Bowen, M Alejandra Tortorici, Alexandra C Walls, et al. Deep mutational scanning of sars-cov-2 receptor binding domain reveals constraints on folding and ace2 binding. cell, 182 (5):1295 1310, 2020.
Matt Sternke and Joel Karpiak. ProteinRL: Reinforcement learning with generative protein language models for property-directed sequence design. In NewrIPS 2023 Generative AI and Biology (GenBio) Workshop, 2023. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstcin, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurclien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tunel chat models, 2023.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 7768 7778. Curran Associates, Inc., 2020. URL
pdf
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Lènguistics, 7:625-641, 2019. doi: 10.1162/tacl _02000
00290.
URL http://allantllllooorrr/r///////
Scan Wellock, Ximing Lu, Peter West, Facze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct In The Eleventh International Conference on Learning Representations, 2023. URL httpps://ppenreviev.. http://wwww...eeeee..oo.ccecaaaa
Timothy A Whitehead, Aaron Chevalier, Yifan Song, Cyrille Dreyfus, Sarel J Fleishman, Cecilia De Mattos, Chris A Myers, Hetunandan Kamisetty, Patrick Blair, Ian Wilson, et al. Optimization of affinity, specificity and function of designed influenza inhibitors using deep sequencing. Nature biotechmology. 30(6): 543 548, 2012.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Felix Wong, Erica J Zheng, Jacqueline A Valeri, Nina M Donghia, Melis N Anahtar, Satotaka Omori, Alicia Li, Andres Cubillos-Ruiz Ruizz, Aarti Krishman, Wengong Jin, et al. Discovery of a structural class of antibiotics with explainable deep learning. Nature, pp. 1 9, 2023.
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. Unpaired translation: A cycled reinforcement learning approach. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 979-988, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10. 18653/v1 1 1/118-1090
CRL https://aclaathooogyy http://atttoooooooo/eeeeee----ee 1090
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference- grounded token-level guidance for language model fine-tuning, 2023.
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. Summlt: Iterative text summarization via ChatGPT. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10644 10637, Singapore, December 2023, Association for Computational Linguistics. doi: http///v1//2.../niiiii emnlp.714. URL https://acllatthlllgs wrw.://hoooogg.../222...... org/2023. fineiiigiiinsn ceciiinng-emmllpp emnlp.71.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett. (eds.), Aduances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https::/prroeeeeggg.nn
www.ing...nuuiiiccomm cc/
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. Style transfer as unsupervised machine translation, 2018.
A FineCS Text Style Transfer Implementation Details
A.1 Sentiment and Complexity Regressor Training
Sentiment Regressor We train Sentiment regressor on Yelp reviews (Zhang et al. 2013). The original data contained 650K train and 50K test reviews divided evenly across five labels (1 - very negative, negative, 3- neutral, 4 positive, and 5 very positive). We filter reviews that are non-Englis) (> 200 tokens). After filtering, we obtain 464K train reviews and RO 36K test reviews. We randomly sample ~ 36K reviews from the train set for validation and train RoBERTa-large (Liu et al. 2020 regressor on the remaining instances for 4 epochs using mean squared error loss. The final regressor obtained a 0.92 test correlation (and 0.37 mean absolute error). During inference, we clamp the predictions from the regressor such that its output range is E 51.
long
Complexity Regressor To obtain the Complexity regressor we train a ranking model on top of the SWiPE Wikipedia simplification datasct (Laban et al. 2023). The SWiPE datasct contains z 143K pairs of simple to complex Wikipedia paragraphs. However, many instances were low quality (very long, very short, high repetition, bad words, non-Fnglish, etc.). After filtering these instances, we are left. with 79K train, 1K validation, and 1.8K test simple complex pairs. We train a RoBERTa-large (Liu al.) 2020) regressor this pairwise data using the ranking objective (Collins& & Koo| 2005) for 8 epochs. The best checkpoint emitted raw scores in the range of E [-17.1, 17.1] and obtained 98.2% accuracy on the test set (comparing the raw scores of simple and complex passage pairs). We lincarly interpolate this output range to be E [- 2] such that we can subdivide the output range from the regressor into five fine-grained threshold boundaries (to match the Sentiment regressor labels).
13 Using external language identification classifier classfferntttpss://uggggggnn http:////tto//o....../////////// haaprarrrrrbbaa-lllaagg--geeteee
20

---

Published in Transactions on Machine Learning Research (04/2025)
[I>ST <<sV>>>
You are an advanced stylistic paraphrasing AI that is designed to generate high- quality, grammatically correct, non-repetitive, semantically similar and stylistically diverse paraphrases. Follow the user's prompt structure precisely and retain most of the lexical and topic content of the original input when changing the style.
<</SY>>>
You are Sentiment changing paraphraser for yelp reviews. when paraphrasing, do not deviate too far from the original review in terms of lexical and topic coverage. Reviews on yelp contain 5 levels namely: 1 (strongly negative), 2 (negative), 3 (positive),
(strongly positive). Given human written review of particular level, modify it to generate 5 variations foor each sentiment level (one per line) as follows:
(neutral),
Original Review: <review>
Sentiment Level: <level>
Variation (strongly negative): <variation1>
Variation (negative): <variation2>
Variation 3 (neutral): <variation3>
Variation (positive): <variation4>
Variation (strongly positive): <variation5>


Figure Llama2 Sentiment paraphrasing prompt
A.2 Sentiment and Complexity Few-Shot Prompts
To generate attributed variations of Yelp reviews the Sentiment and Complexity axis we use few-shot prompting on top of Llama2-7B parameter model [Touvron et al. 2023). The prompt used for Sentiment and Complexity given in Figures f and respectively. We augment both prompts with their 3-shot. demonstrations and generate 5 samples for each review using nucleus sampling (top, 0.95).
B FineCS Protein Design Implementation Details
B.1 Fluorescence and ddG Regressor Training
To train the protein evaluators, we randomly split the se 51.7K mutant sequences into 50% train, 15% validation, and 35% test sequences. Along with the ESM2 model-basod regressors, we also experimented with traditional CNN regressors (Dallago al. 2021 The learning rate for both models is le-4 where the ESM2-based regressor trained for 12 epochs and the CNN regressor was trained for 40 epochs. Despite additional training time, the test set correlation for fluorescence and ddG for the CNN regressors are 0.892 and
mneta-llama/TTlams. 7b-chat-hf
21

---

Published in Transactions on Machine Learning Research (04/2025)


 <<sYs>>
You are an advanced stylistic paraphrasing AI that is designed to generate high- quality, grammatically correct, non-repetitive, semantically similar and stylistically diverse paraphrases. Follow the user's prompt structure precisely and retain most of the lexical and topic content of the original input when changing the style.
<</SYs>>
You are Linguistic Complexity changing paraphraser for yelp reviews, When paraphrasing, do not deviate too far from the original review in terms of lexical and topic coverage. Reviews on yelp can vary on 5 levels of complexity namely: (very simple), (simple),
(complex), 5 (very complex). Given human written review of particular level, modify it to generate variations for each complexity level (one per line) as follows:
(normal),
Original Review: <review>
Complexity Level: <level>
Variation 1 (very simple): evariation 1>
Variation 2 (simple): <variation 2>
Variation 3 (normal): <variation 3>
Variation (complex): <variation 4>
Variation 5 (very complex): <variation 5>


Figure Llama2 Complexity paraphrasing prompt
0.933 respectively, that are much lower than ESM2-based models (0.974 and 0.987 respectively). Subsequently, we use the ESM22-asse regressors as evaluators our experiments.
B.2 Protein Design Baselines and LM editor Implementation
The 51.7K GFP train mutants are unevenly divided across the 16 multi-attribute threshold combinations as seen in Figure 5a In the Random mutation baseline, when predicting new mutant sequences from a particular threshold combination, we maintain the edit distance distribution of the train sequences within the same threshold combination. The Recombine baseline uses recombination strategy where pair of sequences are mixed (shuffling each position with recombination rate 0.5) to create two sequences. When generating new sequences for particular threshold combination with the Recombine bascline, we set the train sequences within the same thresholds as the seed set, randomlly shufile them, and iteratively apply the recombination strategy until we get 3000 new sequences. Since some threshold combinations have very low seed sequences (<200) there may be duplicates when generating the 3000 new sequences with this strategy. We improve upon this baseline, we create Unique Recombine where we keep generating sequences with the recombination strategy until we get 3000 unique sequences that don't overlap with the training set.
22

---

Published in Transactions on Machine Learning Research (04/2025)
60
10
50-
50
060
20
10
2.0
2.5
4.
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
lng
log fluorescence
(a) Mutant edit distances from wild-type GFP
(b) Attribute distribution of GFP mutants
Figure 8: Log Fluorescence and ddG distribution of 51.6K GFP mutants
Top Mutational Hotspots for log Fluerescence and <0 ddG
Top Mutational Hotspots for 3.7 log Floorsscenca and ddG
1.0
1.0
0.0
0.0
Train Set (total 100)
LM Ganerated (total 542)
Train Set (total 942)
LM Generated [total 2081)
(a) low fluorescence high slability
(b) high fluorescence low slability
Figure 9: Comparing the top 15 mutational hotspots from the train set vs. LM predicted demonstrating that MACS can extrapolate beyond the mutational patterns seen during training.
For the Protein Language Model editor models trained with our MACS framework, we use the nucleus sampling [Holtzman et al. 2019 with (topp 0.95). We also had to increase the generation temperature to 1.2 to encourage more diverse sequences. During our early experiments, we sampled edit pairs from each threshold combination uniformly, leading to overfitting the low-density regions of multi-attribute space, i.c., most LM-generated sequences duplicated. To mitigate this bchavior, downsampled the threshold combinations containing fewer than 400 sequences
15
18ff target threshold combination sequences, reduce edit-pair sampling weights n/r reduce overfitting in the sparse region.
23

---

Published in Transactions on Machine Learning Research (04/2025)
C Limitations and Future Work
MACS is an casy-to-implement framework to train domain-specific language models fine-grained editors in an offline setting. However, there are a few limitations. Due to the oflline nature of our method and our sampling strategy, it unable to extrapolate well to regions within the multi-attribute space with low or no data points. To train good multi-attribute LM editors, MACS requires a good initial domain-specific pretrained language model. In our preliminary experiments with antibody generation task (Wong al., 2023), chemistry LM To trained with MACS not able to generate many novel candidates, likely due to its small size and poor data coverage.
In the future, we aim to extend our method such that it can use both offline and on-policy samples to improve its performance and diversity in the fine-grained control task. Further research is also needed to support categorical and lexical constraints in MACS.
24


Reviews:
[{"review_id": "8NFas3WjBz", "reviewer": "Reviewer_p7fm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is reasonable and the authors focus on the multi-attribute constraints, which is more challenge and important than previous works.\n2. Several proposed techniques are interesting and meaningful, such as the Anchor Conditioned Inference and Multi-Step Reward Prioritized Inference.\n\nWeaknesses:\n1. The main challenge is the novelty of this work, though the authors criticize previous works, such as the iterative refine and inference method, the proposed method is actually in a same way with small modifications. \n2. Besides, the results are not good as satisfied, even as the author claimed by themselves, some of the results are not significant. Also, the compared baselines are not enough, which is quite limited.\n3. Further, though the overall framework is reasonable, the authors use RoBERT-large to train the evaluator. Since the evaluator is a regression-based training model, the regression-model would be not so precise as expected, this happens in all regression-based models. Therefore, the evaluator may not be so good. Though this is a hard problem, it should not be ignored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission proposes to control the LLM output with satisfied properties/constraints in real value. The authors propose an attribute evaluator and an iterative inference method to rewrite the language model output. Further, the authors propose two benchmarks with multi-attribute constraints and evaluate their approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. More experiments are required.\n2. Differences between this method and existing methods are required in detail.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mIKApWttVD", "reviewer": "Reviewer_VzBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strength of the paper lies in that it seems that using LLM to do rewriting for constraint satisfaction of multiple attributes in text-based domains seems to be an interesting and relatively under-explored application. Despite the fact, that using LLM for rewriting is itself not a super novel design choice.\n\nThe main weakness might be that the technical evaluation does not seem to yield significant gains over certain baselines in the empirical comparison, posing the question as to whether offline RL is needed at all in this process or does SFT suffice. As mentioned above, LLM rewriting is itself not a very novel design choice, and the technical contribution of the paper relies a lot on the algorithmic part of the empirical design, which seems to be a bit lacking in the current paper. The presentation can also use improvement in its clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to carry out rewriting using LLM to solve constraint satisfaction problems for certain text based domains, such as text style transfer and protein designs. The paper proposes a RL approach: they describe a dataset collection process, reward design as well as loss and algorithm for offline RL followed by SFT. They extensively evaluate their approach with a number of datasets and show improvements over baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "=== *Presentation clarity* ===\n\nOverall, I find the paper lacks clarity in Presentation. The description of the offline RL method should be better highlighted in the presentation and draws connection to the weighted BC literature. In its current form, the paper spends lots of text for description rather than formula description, making it a bit harder for technical readers to follow. An algorithm box would help.\n\n=== *Reward design* ===\n\nThe reward design process is critical for offline RL. From the description of the work, the authors propose to label randomly chosen pairs of data with some instructions such that they can be understood as some form of text rewriting data. I find this process to be remindful of hindsight experience replay [1] line of work, which can be of interest to discuss.\n\n[1] Andrychowicz et al, Hindsight experience replay, 2017\n\n=== *Overall design* ===\n\nMy overall concern with the paper is the technical depth and solidity of the overall method, since using LLM for rewriting itself is not a very novel application, and despite the interesting design on data collection and reward design, it feels more like a bag-of-tricks rather than a coherent framework to solve the constraint satisfaction problem. The paper also discusses lots of specific points regarding applications, such as domain-specific designs for protein design and text style transfer, e.g., knn-sampling seems to be coupled wiht text style transfer. This makes it a bit difficult to disentangle what constitutes a more general algorithmic design of the paper, and what constitutes an application-specific design choice.\n\n=== *Evaluation for text transfer* ===\n\nIn text style transfer in table 1, we find the new approach generally outperforms the previous baseline. However, I notice that SFT itself is already bringing most of the benefits and in fact in certain cases wBC (offline RL) does not add much on top of SFT, or the improvement from SFT to SFT + wBC is generally not statistically significant (within error bar, despite highlighted in the table). This paints the general picture that SFT is the main contributor to the performance gain, which makes sense since all other baselines do not make use of domain-specific SFT fine-tuning. It then poses the question of whether wBC or offline RL is needed at all in this process, since SFT feels like all you need. \n\nI think the evaluation comparison here should highlight the importance of wBC over SFT, because SFT is a common process and performance improvement is expected compared to few-shot or control token baselines. wBC is a key contribution of this paper, whose role is not clear from the current eval.\n\n=== *Evaluation for protein design* ===\n\nIn protein design eval in table 2, it is nice that wBC yields general gains over SFT. But it is interesting to see that entropy regularization seems to generally increase the edit distance (contrary to wBC alone). Can you elaborate more on this?\n\nCan you also provide error bar for the evaluation in order to assess statistical significance.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qWkUcJBzgU", "reviewer": "Reviewer_JCoR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths:\n\n1. The paper presents a robust solution for Real-Value Attributes Constraint Satisfaction, a significant challenge in the field. The MACS framework's ability to handle step-by-step refinement for such constraints is particularly noteworthy and addresses a crucial gap in current methodologies.\n\n* Weaknesses:\n\n1. One of the weaknesses is the difficulty in constructing the datasets needed for training the step-by-step refinement process. The paper would benefit from a more detailed discussion on how such datasets can be effectively created and utilized.\n\n2. The paper could improve in detailing the baseline configurations, and algorithmic procedures. Providing a step-by-step pseudo code would enhance understanding and reproducibility of the methods described.\n\n3. While the experiments conducted are thorough within the presented tasks, expanding the scope to include more tasks would strengthen the paper's claims. Demonstrating the framework's versatility across a broader range of applications would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a framework, Multi-Attribute Constraint Satisfaction (MACS), which aims to enhance the capability of language language models (LLMs) to satisfy multiple user-specified constraints on external real-value attributes. The authors propose a method that involves fine-tuning LLMs in any sequential domain, leveraging weighted behavior cloning to iteratively improve outputs and meet constraints across various attributes. The experimental results on text style transfer and protein design tasks demonstrate MACS's effectiveness in achieving high constraint satisfaction rates, outperforming existing domain-specific baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Hard Constraints Satisfaction: Can the MACS method strictly satisfy hard constraints, or are there scenarios where it may fail to meet these constraints? If so, how does the method handle such failures?\n\n2. Offline RL Formulation Clarification: The paper mentions using Offline Reinforcement Learning for training the language model editor. Could the authors clarify how the rewriting process is formulated as a Markov Decision Process? It seems that the method described leans more towards imitation learning via Weighted Behavior Cloning, which is traditionally distinct from offline RL approaches [1]. How does this method reconcile with the principles of offline RL?\n\n3. Comparison with Self-Prompting: The paper could benefit from a comparison with multi-round self-prompting, which are also designed to iteratively improve language model outputs. How does MACS differ from or compare to such self-prompting methods in terms of effectiveness and efficiency?\n\n4. Emphasis on Real-Value Attributes: The paper appears to place significant emphasis on the satisfaction of real-value attributes constraints. Reflecting this focus in the title or abstract could potentially highlight the paper's contributions more clearly. Would the authors consider adjusting the title to better reflect this unique aspect of their work?\n\n[1] Rashidinejad, Paria, et al. \"Bridging offline reinforcement learning and imitation learning: A tale of pessimism.\" Advances in Neural Information Processing Systems 34 (2021): 11702-11716.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I am a new AE reassigned to this paper as of today.

The reviewers praise a number of aspects of this work, including the multi-aspect rewriting setting and aspects of the methodology. I found the paper to be interesting and well-written.

The reviewers raise a number of criticisms, but I don't think these are critical for TMLR.

1. One of the main criticisms from reviewer p7fm is novelty, which is not in scope as a critique for TMLR.

2. Insufficient comparison to baselines is a more germane critique, which potentially impacts the supportedness of the claims. I partially agree that additional methods could be used.  For instance, FUDGE (Kevin Yang et al.) uses steering classifiers which I believe could be trained given the data in FineCS.  Moreover, the technical limitation that the method of Mireshghallah et al. can only use masked language models for generation was relaxed by follow-on work ( https://arxiv.org/pdf/2312.04510 ). However, these approaches don't precisely match the setup of the paper here. The approach here has certain simplicity advantages that I think make it fair for it to stand on its own without comparison to these.

3. I do think that JCoR's suggested baseline of self-prompting is also relevant. Small models are increasingly able to do this kind of refinement. However, as I understand the data condition and the models here, this baseline would not work in the protein design domain, as the authors point out in the response. I think it's fair to only consider approaches that cover both domains here.

4. VzBx's critiques about the lack of generality and ad hoc nature don't really resonate with me. Although aspects of the implementation like kNN sampling do strike me as ad hoc in a certain sense, the framework itself is general.

5. VzBx expresses concern about the small sizes of the gains in the results and some of the gains not being statistically significant, particularly as they relate to wBC over SFT.  Whether wBC is needed over SFT is not a critical part of the claimed approach in my view. Although it is part of the framework, wBC is not a novel algorithmic contribution of this work either; I view the use of SFT, wBC, or the combination as a bit like a hyperparameter choice, and it is not critical for a certain value of the hyperparameters to work better for the work to be accepted.

A few cites (Andrychowicz et al. on page 4, Peng et al. at the bottom of page 5) use citet when they should use citep.
Decision: Accept (poster)