Deep Extrapolation for Attribute-Enhanced Generation
Authors: Alvin Chan
Ali Madani
Ben Krause
Nikhil Naik
Venue: NeurIPS 2021 Poster (2021)
Keywords: extrapolation
generative modeling
controllable generation
protein design

Abstract: Attribute extrapolation in sample generation is challenging for deep neural networks operating beyond the training distribution. We formulate a new task for extrapolation in sequence generation, focusing on natural language and proteins, and propose GENhance, a generative framework that enhances attributes through a learned latent space. Trained on movie reviews and a computed protein stability dataset, GENhance can generate strongly-positive text reviews and highly stable protein sequences without being exposed to similar data during training. We release our benchmark tasks and models to contribute to the study of generative modeling extrapolation and data-driven design in biology and chemistry.

Deep Extrapolation for Attribute-nhhanced Generation
Alvin Chan Salesforce Research, NTU
Ali Madani*
Ben Krause Salesforce Research
Nikhil Naik Salesforce Research
Salesforce Research
Abstract
Attribute extrapolation in sample generation is challenging for deep neural net- works operating beyond the training distribution. We formulate new task for extrapolation in sequence generation, focusing natural language and proteins, and propose GENhance, a generative framework that enhances attributes through learned latent space. Trained movie reviews and computed protein sta- bility dataset, GENhance can generate trooggy--oosttion text reviews and highly stable protein sequences without being exposed to similar data during training. We release our benchmark tasks and models to contribute to the study of gen- erative modeling extrapolation and data-driven design in biology and chemistry: htt.s://gthtuu.cooo//uussrreegee
1 Introduction
Deep generative neural networks can generate realistic data across data-types, from sequences to images time-series data, with applications in domains such natural language processing (NIP),,
computer vision, and speech. Beyond canonical domains, the scientific application of synthetic design
of proteins, molecules, and materials
be cast generative modeling of sequences, graphs, or
images (nnadd & Huang 2018 De Cao & Kipi 2018 Madani et al. 2020 202T). Most often, the goal is to design generate sample that improves upon the attribute label of interest (Fig (left)) which we term attribute-enhance
generation. Examples include generating a protein sequence with higher binding affinity or a nanomaterial structure with an energetically favorable state, as compared to all of the samples in the training distribution. In these scientific fields, traditional methods for synthetic object design with improved attributes are iterative and expensive, relying on labor- or compute-intensive methods (Bepler & Berger 2021 Wu et al. 2021 Hie & Yang (021). Hence, deep generative models that can design new proteins, molecules, and materials with improved attributes have the potential to dramatically accelerate design research. Beyond scientific applications, extrapolation in generation has potential applications NLP, such as reducing toxicity or operating in low-resource settings.
is, however, a well-known challenge for deep neural networks to generate samples beyond the training distribution (Arora et al,, 2017 Radford et al. 2019: Xu et al. 2020). In this work, we develop a method for extrapolation, particularly for sequences. Our approach, called GENhance, is designed to generate an enhanced sequence using learned latent space. GENhance consists of generator (sampler) and a discriminator (ranker) that are jointly trained to minimize generation and discrimination losses, regularized by latent vector smoothing and cycle-consistency loss.
We evaluate GENhance in data domains. First, natural language benchmark containing movie reviews with five discrete sentiment attributes (Socher) al. 2013), show that GENhance generates strongly positive reviews, after training with no
the Stanford Sentiment Treebank (SST),
Equual Contribution
Correspondence to amadani @ @salesforee.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Protein
A4G
Review
Rating
Estrapcheeed
Number of samples
Range y Nr
Range
STEERGGKTLDCCNEEEEE...
langely bogge story.
Jiaze y-
STIEBGNKTRDKTHHHAEEE..
0.0
argety bogus story.
Wookl Magezie
Range
Siatribtiion
Gencratec
STEERGKKRLXKPHEEEEE...
6.6
An uplitting, yot langely bogus vtory
STEERMMTRLKKNDEEERR.
-74
relutiioely unjoyable stary.
Veeaky Posttvv
Train Distribution
Extepolated
-22
uptifting vhory heartwwarming story.
Soungly Pootive
Sample attribute value,
Ur
More Stable
More Positive
Figure I: Attribute-enhanced generation. The goal of extrapolation (left) is to generate samples whose attribute values exceed that of all training samples, yr- We explore target attribute extrapolation for protein sequences (center) and movie reviews (right), where more stable protein sequences and more positives text reviews are generated.
positive examples. Second, we develop protein stability dataset for the ACE2 protein (Chan et al. 2020) with change in free energy (ddG) continuous attribute, and show that GENhance can generate protein sequences with higher stability than the training set (Fig. D (right)). GENhance significantly outperforms baseline methods based on (i) generator-diserimnaat model with rejection sampling and (ii) an algorithm using Eetropolis-Hastting discriminator. GENhance's 's performance further improved when provided access to examples with attribute scores beyond the training distribution. Our contributions are summarized below:
Markov chain Monte Carlo sampling with a trained
We formalize the task of extrapolation for deep generative models, focused on enhancing attributes in sequence generation, with important scientific applications in synthetic object design.
We introduce GENhance, regularized encoder-decoder framework with learned latent space, and demonstrate its superior performance with respect to rigorous baseline techniques.
We curate extrapolation benchmarks in NL.P and proteins. We release the data, evaluation metrics, along with oraele models and scripts for automatic evaluation of generation quality.
2 Related Work
Generalization to Low Data Regimes: Previous approaches aim to generalize classification and regression to low data settings. Imbalanced classification methods upsample downsam-
ple Caasses(Chawii et al. 2002, Gareía & Herrera 2009) or reweight the training cost function (Huang et al.) 2016 Cao al. 2019 Cui et al. 2019). Yang al. (202T) improve the generalization of regression models in extrapolation and interpolation of the continuous data domain by smoothing both the label and features of the training data. Unlike prior work in this area, GENhance aims to generate samples in low/no data settings. Methods that can better generalize discriminators to these regions complimentary and orthogonal to work.
Data-Driven Design: Data-driven design aims to learn a distribution over high-dimensional input space that is optimized for a fitness function corresponding
desirable property. Design methods often iterate sampling from a generator, and then updating the generator to assign higher probability to inputs that a discriminator predicts to have higher fitness (Bedbrook et al., 2019 Biswas et al. 2021; Mansouri Tehrani et al. 2018). Auto-focused oracles (Fannjiang & Listgarten, 2020)
also adapt discriminators throughout this optimization process to using re-weighting of the training examples in the cost function to make them more reliable in the regions where the generator is more likely to generate. CbAS (Brookes et al., 2019) and DbAS (Brookes & Listgarten 2018) use fixed discriminator/oracle model and iteratively learns the distribution of inputs conditioned on desirable property using importance sampling. CbAS is an improved version of DbAS which also re-weights samples based on how close they are to the original training data. We view these techniques as complementary as GENhance proposes a -pdec-specific architecture for optimizing attributes.
Das et al. 1021) train VAE to learn latent space and use latent space classifiers to sample latent vectors through rejection sampling and decode them into sequences that would have the target attribute/label.
Hawkins Hooker et al. (222) also decode generations of VAE by conditioning latent vectors

---

that correspond to the target attribute/label. Hoffman et al. (2020) seek to optimize molecular designs by using zeroth-oue order optimization on query- prediction of candidate molecules' properties. Klomez-Bombarde et al. (2018) build Gaussian Process (GP) regression model trained with latent vectors to predict their inputs' labels and use gradient-ban based optimization on the GP to find sequences with target attributes. Compared with these previous works, the core difference in our approach is the combination of cycle-consistency and contrastive discriminatory objective to train the generator and discriminator one model.
Controllable Text Generation: Our work is also related to controllable text generation, which aims to generate text that corresponds to user-specified attribute (Kikuchi et al. 2016 Ficler & Goldberg 2017). CTRL (Keskar et al. 2019 generates controlled fluent texts through control codes which are meta-data prepended to the text during generation. Krause et al. (0220) use generative discriminator resulting from contrasting predictions from opposing control codes to guide generation. CoCon (Chan al. 202T performs zero-shot controllable text generation without attribute labels. (Ziellrr et al. 2019) optimizes language generation for desirable attributes via human in-the-loop reinforcement learning. Similarly to GENhance, PPLM (Dathathri et al. 2020) applies a discriminator on top the latent space generative model to guide generation, however, GENhance
use of
autoencoder rather than a language model. Lastly, text style transfer methods have used autoencoders with disentangled style latent representations (Shen et al., 2017; Hu et al.) 2017: Yang et al. 2018). Unlike text style transfer and previous approaches toward controllable text generation, GENhance differs, aside from its model formulation, in that the goal is to optimize and extrapolate particular attribute beyond the training distribution.
3 Methods
Our goal is to generate sequences with target attribute values that are better than the the training data. Formally, assume that there is ground-truth oracle (O) that maps each sample (x IRd) to the target attribute value (y E R). i.e. y = O(x). Given a dataset of oracle labeled samples (D), we aim to generate new sequences where its ground-truth attribute value is better than that of this dataset:
Vnew
VT+ v(x,V) E D V+
To generate samples that satisfy this criterion with high probability, we develop a caampling-rankinng framework that consists of a sampler S that proposes pool of candidate sequences and ranker R model to infer the relative scores of these candidates. First, we describe two baseline generation techniques that natural choices for this task and build on them to develop our GENhance model.
3.1
Rejection Sampling
The first baseline, Gen-Dise, rejection sampling approach that uses generator model as the sampler S and separate discriminator model as the ranker R. The generator is trained to model the training distribution p(x) through a language modeling objective where learns auto-regressively
(Manning et al. 1999 Bengio et al. 2003) construct the training samples,
p(z .(........ 2z 11 . Tt-1) II p(c.lop............ X 2i....,
where is the length of the training sequence.
The discriminator model is trained with an objective to predict the relative ranking of sequences from pair of samples, based on their attribute. Given two training samples (Xa. Ma) and (xx, yb), the pairwise contrastive loss is:
1
Ceontrast - log exp(ū. 26)
D. faisc
Ya > Yb
where fais denotes the discriminator which outputs scalar score value for each input sequence. We employ the contrastive loss term for this objective since can be applied both continuous- and discrete-labeled samples. After training, we sample candidate sequences from the generator model an auto-regressive fashion and use the discriminator to rank the sequences according the discriminator's 's output score values.
3

---

GENhance DEC
Lecoon
GENhance ENC
Latent Space
L Leye-con
DEC
smoohh
L
Ccontrast
2
Zu
Az
db
ENC
GENhance an mecoderd framework with latent space between the two. GEN- hance is trained to extrapolate beyond the training distribution of attributes, by learning the latent space using a combination of contrastive, smoothing and cycle consistency losses, in addition the reconstruction loss for autoregressive generation.
Figure 2:
3.2 Metropolis- Hastinss Markov Chain Monte Carlo
Traditional methods for data-driven design rely on iterative optimization of candidates with better attributes. To mimic this process, we design method that generates candidates using Metropolis- Hastings MCMC sampling from a population of better candidates. We start with an initial population of sequences, sampled from the training set. In the sampling step, new candidates are proposed by making edits to samples from this initial population, scored with the ranker R, and compared with the previous population of candidates. The probability that new generations are kept the population depends on the score predicted by R. The cycle of sampling and ranking repeats until terminated. The ranker R takes the form of a neural network, identical to the discriminator model the Gen-Disc setup.
3.3 GENhance
In the Gen-Disc framework, since the generator is trained only to model the training distribution, there is no direct way to steer the generation distribution towards certain direction beyond the training data. In the MCMC framework, it might be challenging to find desirable candidates with stochastic mutation operations, since the search space can be large and high-dimensional for many design problems (Kumar & Levine 2019). To overcome these limitations, we propose using learned latent space (Kingma & Welling 2013) to control the attributes of generated sequences.
Architecture: GENhance is an encoder-decoder framework with latent space between its encoder (ENC) and decoder (DFCC modules (Figure 2). The latent vector (z Rd,) ) of particular input sequence (x E Rdd)) is the output from the encoder module, i.e., z = ENC(x)) ). In our experiments, z the hidden state the location of cls token (Devlin al. 2018 that prepended the input sequence. Within the latent vector z, representation relevant and irrelevant to the attribute of interest stored in i| and z_ respectively, i.c., z [z]ji 11]. To train the encoder store information about the target attribute in , we train it with contrastive objective that aims to learn which of the two samples has the better value of the attribute:
Ceontrust - -log exp( 36)
0a f(j(aaa)); [Zo]]: Z.ai = ENC(x.), Yo >
(4) where (x., Ma) and (Xb: vb) are pair of training samples, each containing input sequence x and its label y- Sj is an operation that maps 2|| to a scalar value. Here we use Zl of dimension 1 and fjj simply identity operation.

---

We train GENhance to generate sequences using an objective where the decoder will autoregressively reconstruct a sequence while conditioned on the latent vector z. For an input sequence x of length 1
parameterizing the ENC with A and DEC with v/, we get the reconstruction loss:
Lreecn
logps (zilz, (..,.... ((-1))
log pa, (x)(x)
(5)
To ensure that the perturbed latent vector z would result in plausible generations, we include smoothing objective, the deterministic Wasserstein autoencoder-maximuut mean discrepancy (WAE- MMD) objective (Tolstikhin et al.) 2017). to train the latent space as it has shown to effective for discrete sequences. The WAE-MMD term (defined in the Supplement A.1 penalizes divergence of the latent vectors z from target prior distribution Pz. which a unit Gaussian in our case.
smooth
MMD(P2,
To help learn better latent space and stronger discriminator within GENhance, we propose cycle- consistency learning objective (Leyo-con) to train ENC' to correctly predict the relative rank between two reconstructed inputs:
Leyo-oon - log
+ exp - (6) da (2(1))))
(7)
[2ali [ ENC(,.),
X. DEC((EN((s.))
ya yb
The intuition behind this objective is two-fold. First, since the discriminator (ENC) is used to rank generated sequences during inference, by also training the discriminator (ENC) on generated sequences (x) during the training phase. Secondly, by backpropagating the Leye-eon term through GENhance, could learn a latent space which generates sequences that are easy for the discriminator to rank accurately. Combining all the training objectives, we can optimize using stochastic gradient descent to approximate the optimal parameters for GENhance:
improve its performance these synthetic sequences
0", v
arg min( (aannnDontattt + Apoccn + A + DayeeennCyy-co
(8)
0,4
To the best of our knowledge, this is the first instance of using cycle-consistency with contrastive loss train generative model.
Sampling & Ranking: After training, we can sample candidates from GENhance's ' latent space and rank the generated samples with the scalar scores output by the GENhance's 's ENC'. First, encode a training sample with ENC to sample latent vector z. To obtain the latent vector encoding for new candidate sequence with improved attribute, target satribute-aligned latent component Z||- At the final step, GENhance's 's DEC conditions on this perturbed latent vector (z') to generate the improved candidate X/:
make perturbation (Azi)) to the
X = DDC((z)) z'
[ AZ zi]
[11 z ENC(x)
0f (1
The perturbation Azi is determined as the direction that increases the fji 's score output, i.e., For linear layer f|]- this term is the weight of the layer while for our case where fj an identity operator, Azi simply scalar.
82
After generating a pool of candidates with GENhance, we can rank and filter out top-scoring candidates with the GENhance ENC's predicted score:
ÿ (((EN(()))
(10)
Experiments and Results
4.1 Experiments in Natural Language with SST-5
Dataset The Stanford Sentiment Treebank-5 (SST-5) (Socher al. 2013) contains movie reviews from Rotten Tomatoes, which are labeled with of the five ordinally increasing sentiment labels:
5

---

Table I: GENhance generates large fraction of
sequences for SST-5, when samples are present in the training set (< 4% of the training set), Metrics are computed for top-1000 ranked sequences. SST-5 test samples have a mean perplexity value of 101.3. Smoothing Latent Smoothing, CC = Cycle-consistencyy
200
Model
% Positive
% Strong-Positive (t better)
Perplexity (1 better)
[[%SP) (t better)
(f better)
Baseline Gen-Disc
90.6
26.7
63.9
14.2
MCMC-Random
17.6
0.5
49696
0.17
MCMC-T5
54.8
10.8
224
4.58
GENhance w/o Smoothing & CC
88.2
21.5
125
15.44
GENhance w/o CC
91.3
23.6
101
16.62
GENhance
98.7
49.7
90.5
44.33
"Strong-Negative', 'Negative', 'Neutral', 'Positive', and Strong-Positives This allows study enhancement approaches for discrete ground-truie truth labels. We curate two data splits. The first SST-5 200-Pos setup removes all Strong-Positive examples from the training set while keeping 200 randomly sampled "Weak-Positivs higher attribute samples. For the more challenging SST-5 No-Pos setup, both WWeakPPositvve' and "Strong-Positive" samples are removed. The 200-Pos and No-Pos training set have 5134 and 4934 samples respectively.
examples, to simulate the presence of a small amount of
Training For both the Gen-Disc and MCMC models, we train the discriminator model by finetuning publicly available (Wolf et al. 2019) pretrained bS-base encoder (Raffel et al. 2019). The generator module of both Gen-Disc and GENhance are trained by finetuning the whole pretrained T5-base encoder-decoder model. The Gen-Dise generator is trained with language modeling objective by feeding in an empty string as the encoder's input and minimizing the cross-entropy loss between the decoder's s output tokens and training sample's tokens through teacher forcing. For GENhance, the training samples are both fed in as the T5 encoder's 's input and used as the label for the decoder's output for the reconstruction objective. Further details on training settings four NVIDIA A100 GPUs are found in the Supplement A. to A.3
Evaluation: We generate 25,000 candidate sequences from each model and use their respective
discriminator module to rank the sequences into pools of top-100, 1000 and 10000 sequences. The percentage of candidates containing target attributes ("Strong-Positive & "eek-Posstiive ) are computed by using ground-truth oracle model. In our experiments, we use a pretrained BERT- large (Devlin al. 2018 model that finetuned on the full training SST-5 training (including "Strong-Posiine Positive & Weak-Positiin trained with batch size of 32 for 30 epochs and achieves an accuracy of 92.5 % for strong-positing vs neutral/negaatve classification. 'Neutral' -labeled SST-5 sequences are used as the initial sequences for the MCMC baselines and as the input sequence for the GENhance models. A2l perturbations of magnitude equal to 5% of the standard deviation of the training samples' 2|| are used for all GENhance generations.
samples), with a classification objective. This oracle model is
We develop additional performance metric, E[/SP]. the expected percentage of "Strong-Positive" generated sequences. The metric was developed to (i) have a statisticllly-eeleaant measure with an expectation value and (ii) use the 'Strong- Positive" labels alone maximize Oracle label fidelity, as 'Strong-Positive labels are almost perfectly distinguishable from the train labels of 'Neutral' and lower. It is computed with the following steps: a) Randomly sample 1000 of generations from the 25000 generations, b) filter out top-100 candidates based on discriminator's ranking, c) compute % top- 100 with ground-truth oracle model and d) repeat step a) c) for 100 rounds and average % -roonn-positive.
Strong-Posittvee
As proxy for text quality, we compute the perplexity value for each generation using pretrained GPT-2 large model (Radford et al. 2019) and average their values across the top-K pools. To guide the MCMC substitution step (MCMC-T5), substitution would degrade fluency. During mutation, a span of 1 or 2 tokens is masked and the masked sequence is fed into the T5 model generate a replacement.
a pretrained T5-base model, since random token
Finally, to further evaluate generated text aside from the oracle model's scores, we conducted human evaluation study to examine the positiveness and fluency of our text generations. The study

---

Table GENhance generates large fraction of attribute- enhanced sequences for SST-5, when no positive samples are present in the training set. Metrics are computed for top-1000 ranked sequences. SST-5 test samples have a mean perplexity value of 101.3. .3. Smoothing Latent Smoothing, CC Cycle-consistency.
Model
% Positive
% Strong-Positivy (1 better)
Perplexity (l better)
E[%SP] (t better)
(f better)
Baseline Gen-Disc
65.1
11.4
61.7
7.65
MCMC-Random
22.9
0.3
20924
0.28
MCMC-T5
46.4
6.2
125
5,81
GENhance w/o Smoothing & CC
42.3
5,6
596
5.46
GENhance w/o CC
69.5
9.3
126
7.8
GENhance
87.7
21.4
118
19.52
was formulated as an A/B test where three evaluators were asked to compare pairs of text blinded random manner to separately determine which text was more positive or more fluent than the other. The comparisons were between text generated by GENhance vs 1) the Gen-Disc baseline, 2) MCMC baseline, 3) SST5 train data labeled neutral, and 4) SST5 train data labeled positive. The evaluators compared 100 samples for each the four comparisons in the 200-Pos and No-Pos settings, totaling to 800 A/B test comparisons. For each comparison, the majority answer between the three evaluators was assigned as the final score.
Results: GENhance outperforms all baselines and ablation variants for all % positive metrics (Table D and D2. 49.7% of the more challeng- ing Strong-Positive' sequences generated by GENhance are correct, which is almost twice the % of samples generated by Gen-Disc, the strongest baseline. All models see performance drops in the % positive metries for the No-Pos training setup as compared to the 200-Pos setup, except for
GENhanoc
which is signifi- cantly worse in both cases. This reflects the greater challenge generating desirable candi- dates when there are positive samples in the training data. GENhance also outperforms other baselines in the top- top-1000 and top- 10000 pools of candidates (See Supplement A.3 Figure shows that the basclines and GEN GENhance
Lmbbll
MOMA-TS
gen- erate more positive sequences than the training, with GENhance showing the largest distribution shift towards candidates with enhanced positive- ness.
Figure 3: GENhance generations have higher pro- portion of 'Strong-Pose Positive' samples than the two baselines, and successfully extrapolate from the training data distribution. Shown here are top-1000 ranked generations for 200-Pos. Colored vertical lines show the generations' mean value.
GENhance generations also have lower perplex- ity values (i.e., better text quality) than the base- line and ablation methods, except for Gen- -isc, which explicitly models the training distribution (more in Supplement A..5). In fact, the average GENhance perplexity value (118) is close to the perplexity of SST-5 test samples (101.3).
According to the human evaluation (Table Bh. GENhance succeeds in generating positive, fluent text that outperforms baselines. In the setup where the models were exposed to 200 positive training samples (200-Pos), GENhance outperformed all baselines, including 'Neutral' and Weak-Positive training samples, the positiveness text generations. For the setting where the models were not exposed to any positive samples (No-Pos), GENhance comparable the positive train samples in

---

Table 3: GENhance enhances 'neutral" sequences from SST-5 to be Strongly-possitive as seen in these generated samples. Positive words in blue and negative words red.
Original Text (Attribute: 'Neutral")
Generated Text (Attribute: "Stronglyypositivee)
A melancholy, emotional film.
A melodramatic film, this powerful story.
An ambitious moving but bleak film.
ambitious and light-hearted film, is strong and moving story.
Some stunning visuals and some staggeringly boring cinema.
engaging collection fantastic visuals and yet at the same time stunningly striking.
You'll laugh for not quite hour a half, but come out feeling strangely unsatisfied.
You will laugh very well and laugh so much you will end up feeling great afterwards
A dark, dull thriller with parting shot that misfires.
A dark and compelling thriller ends with a bitter, compelling punch.
positiveness and outperforms all the other baselines. Likewise, the fluency of GENhance's generations either match outperform that of the training samples and baselines.
Table 4: In human evaluation experiment of text generations positiveness and fluency, GENhance outperforms baseline methods in both "positiveness' and 'fluncy' in most cases. Values reported in % (1 better for all metrics).
Model
200-Pos
No-Pos
Positiveness
Fluency
Positiveness
Fluency
Train Neutral
18
GENhance
91
52
70
49
Train Pesk-Positive
26
39
48
50
GENhance
63
39
45
34
Gen- Disc
42
35
44
GENhance
68
41
53
43
MCMC-T5
17
18
34
46
GENhance
61
56
Both latent smoothing and cycle-consistency objectives contribute to generating sequences with improved attributes and text quality. Without the cycle-consistency objective, observe a drop in performance across all metries, indicating that the objective is vital in helping the latent space and encoder generalize to sequences outside the training distribution. When the latent smoothing objective is removed, especially for the more challenging No-Pos setup, the generation quality drops, as indicated by the large increase in perplexity. This indicates that the smoothing objective important learning a latent space that amenable to perturbations that control its attributes while maintaining generation quality.
Ablation Study:
4.2 Experiments in Protein Design with ACE2 Proteins
Designing protein with an optimized property (e.g. stability) is of immense interest to synthetic biology and drug discovery. Here, we create new synthetic dataset of stability for mutations of human angiotensin-converiin enzyme (ACE2) protein. Since virus binds to ACE2 to gain entry into human organs, ACE2 has emerged as a promising target for COVID-19 19 therapeutic protein design (Chan et al. 2020). Our optimization problem is to generate an ACE2-like protein sequence that is more stable than samples in the training set. As proxy for experimentally measured stability of a protein sequence, we use the free energy calculation via FoldX (Schymkowitz et al. 2005) which provides automated, computational oracle for testing extrapolation methods in silico. In particular, we measure the change in free energy from wild-type, ddG AAG, between the folded and unfolded state of protein sequence with the known ACE2 structure. lower ddG value indicates a sequence that more stable.
Dataset:
BARS-CoV-2
We mutate the N-terminus subdomain of ACE2. The protein is represented amino acids starting from the N-terminus side, with vocabulary of 20 amino acids. We curate 250K ACE2 variants by mutating the wild-type (natural) ACE2 subdomain through substitutions and
sequence of 83

---

Table 5: GENhance generates large fraction of highly stable ACE2-like sequences, with better mean stability as compared to baselines. Metrics are computed for top- 100 ranked sequences. Smoothing Latent Smoothing, CC Cycle-consistency.
Model
ddG mean (4 better)
PCI,, (%)
[[min] better)
(1 better)
Baseline Gen-Dise
-4.05
-6.18
MCMC
-4.84
9
-6,89
DbAS
-3.48
-4.67
CbAS
-4.31
5
-6.49
GENhance w/o Smoothing & CC
-7.16
66
-8.31
GENhance w/o CC
-5.59
30
-7.68
GENhance
-7.34
77
-8.71
computing their ddG values. To keep local landscape of deviation from wild-type, amino acids were substituted by another amino acid with probability of P 4/L where L is the length of the protein's 's mutable region. Mutants with more than eight mutations are discarded and a constant region (NTNITEEN) is maintained. The ddG values for each sequence are computed the average over five FoldX simulations. More details are in Supplement A.4 We use this dataset to evaluate GENhance's ability generate protein sequences with lower ddG values than those found in the training distribution. In contrast to SST-5, the ACE2 ddG values lie on a continuous scale, allowing validate GENhance in the continuous label setting.
Training: initialize the weights of our models, we use a T5-base model that is pretrained on Uniref50 (Suzek al. 2015) with masked span objective of mean length 3. The discriminator model for both Gen-Disc and MCMC models are trained by finetuning the pretrained encoder on the full set of 250K sequences, with random 10% used the validation set while the generator modules of both Gen-Disc and GENhance are trained by finetuning the whole pretrained encoder-decoder model. Further details training settings four NVIDIA A100 GPUs are found in the Supplement.
generate 250,000 sequences from each model while elim- inating generations without the constant re- gion (NTNITEEN) or with different sequence length from the wild-type sequence. We then use the methods' respective discriminator mod- ules to rank the candidates into pools of top- 10, 100 and 1000 sequences. The top-K sequences' ddG values are then computed with the FoldX software, taking the average over five simula- tion runs. The top-5% most stable sequences are used as the initial sequences for MCMC and the input sequence for GENhance. Azli per- turbations of magnitude equal 25% of the std. dev. of the training samples' ž|| are used for all the GENhance models. Following Fannjiang & Listgarten (2020), we also measure percent chance of improvement (PCI,.) over the best la- bel (most negative ddG) in training data. To have aettistically-relevaaa metric, we developed the expected minimum ddG value (E) (mim))) which is computed by the following steps: a) Randomly sample 10000 of generations from the 250,000 generations, b) filter out top-10 candidates based on discriminator's ranking, c) compute ddG top- 10 with FoldX oracle software find the minimum ddG value among these 10 candidates and d) repeat step a) to c) for 100 rounds and average the minimum ddG values across these rounds.
Evaluation: For evaluation,
GENhance
0.4
Baseline
Train data
Demott
0.2
0.0
-7.54
4.54
-405
1.2
-10
-5
0
5
10
More Stable
Less
Stable
Figure 4: GENhance-generaan ACE2-like quences show the largest shift in ddG distribution (i.e., stability improvement) from the training set. Top-100 ranked generated sequences are shown. ddG values binned with interval of 1. Trian- gles denote the distributions mean values.

---

In addition to Gen- Gen-Disc and MCMC, we include CbAS (Brookes et al. 2019) and DbAS (Brookes & Listgarten 2018) as baselines for comparison in this task. Both CbAS and DbAS use the same model architecture as the baseline Gen-Disc. We first sample generations from the baseline Gen-Disc's generator then retrain the generator on the generations re- re-weighted by the discriminator's 's scores. We used the initial hyperparameters from CbAS and conducted grid search on a) M, number of generation per iterations (50, 100, 200), b) Q. percentile threshold (75, 90), c) temperature of a sigmoid score-based weight computation (0.1, 1, 10) and report the best results for CbAS, The DbAS' hyperparameter values mirror those of CbAS in our experiments.
Results: GENhance outperforms all baselines on all metrics (Table 5 in designing more stable sequences. GENhance sequences have the lowest mean ddG value, with significant fraction of generations more stable than the most stable sequence in the training set, as indicated by the higher PCI,, values. GENhance also has the lowest [[min] value, indicating that it may be well-suited to find stable protein candidates in laboratory experiments where only small numbers of candidates can be evaluated due to cost. Even though MCMC and CbAS fare better than the simpler baseline Gen-Disc setup, we observe that GENhance outperforms both baselines on all three metrics measured. The distribution of generated samples by GENhance shows the largest shift towards more stable sequences as compared to the original training distribution (Figure
Ablation Study: Similar to SST-5 experiments, GENhance outperforms its ablation variants on all metrics. Rather surprisingly, we observe a drop in performance when the latent smoothing objective is added, which speculate is due to the tension between the GENhance's ' reconstruction and its encoder's contrastive training objectives. With the sycle-consisteney objective, we see a boost to GENhance that outperforms the vanilla variant, indicating that this objective aids in stabilizing the convergence of these two training objectives. To further study their contribution to GENhance ' performance, we use GENhance's encoder to rank sequences generated by the generator in the baseline Gen-Disc setup and observe a boost in performance (Supplement Table T2). This suggests that GENhance's superior performance is due to both more accurate ranking by its encoder and better generation by its decoder.
There are two main features of GENhance that may contribute to its ability to generate attribute-cnhanced sequences. First, compared to the baselines which mainly rely on the discrimina- tor's prediction to filter out promising candidates, GENhance can additionally use its latent space to steer the general distribution of generated candidates towards a target region (e.g., more stable positive sequences). Second, unlike the discriminators in the baselines which were trained only on training samples, the encoder used in GENhance was also trained on GENhance GENhance-generated se- quences through the cycle-consistency loss. This may contribute to the better-ranking performance for GENhance, increasing the fraction of desirable candidates.
Discussion:
5 Conclusion
In conclusion, we formalize the task of attribute-enhanced generation that aims to create improved samples with target attributes beyond the training distribution. Scientific applications can include the design of proteins, materials, and molecules without expensive, iterative procedures for dis- covery. To achieve this, we proposed GENhance, a generative model with trained latent space, that generates sequences that outperform both the training data and baseline methods in natural language and protein engineering tasks. In the future. we aim to expand GENhance to other data types beyond sequences and study generation in scenarios where new data samples could be ac- tively acquired. We also open-source source our curated benchmark datasets with computational oracles along with all models and evaluation metrics/scripts to enable further research in extrapolation: http:///hhhoooooaaast////////e/e
Broader Impact: Extrapolation involves designing samples, whether text, proteins, molecules, or materials, that have attributes that are unseen in training. If our technique future iteration thereof adopted broadly, care should be taken in terms of the end use-cases of these designed/optimizzed samples and downstream effects to ensure safe, non-nefarious, and ethical applications. For projects in any domain, active oversight during project initiation, experimental optimization, and deployment phases should be put in place to ensure safe usage and limitation of unintended harmful effects.
10

---

References
Namrata Anand and Po-Ssu Huang. Generative modeling for protein structures. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7505-7516, 2018.
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). International Conference on Machine Learning, pp. 224-232, 2017.
Claire N Bedbrook, Kevin K Yang. Elliott Robinson, Elisha D Mackey, Viviana Gradinaru, and Frances H Arnold. Machine learning-guided channeirhodopsin engineering enables minimally invasive optogenetics. Nature methods,
2019.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 137-1155, 2003.
Tristan Bepler and Bonnie Berger. Learning the protein language: Evolution, structure, and function. Cell Systems, (2(6):654 669, 2021.
Surojit Biswas, Grigory Khimulya, Ethan Alley, Kevin M Esvelt, and George M Church. Low-n protein engineering with data-efficient deep learning. Nature Methods, 18(4):389-396, 2021.
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for
robust design. In International Conference on Machine Learning. pp. 773-782. PMLR, 2019.
David H Brookes and Jennifer Listgarten. Design by adaptive sampling. arXiv preprint arXiv:1810.03714, 2018.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-dissributionn-aaar margin loss. arXiv preprint arXiv: 19t.:vv9906007...3 2019.
Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu. Cocon: A self-supervised approach for controlled text generation. ICLR, 2021.
Kui K Chan, Danielle Dorosky, Preeti Sharma, Shawn A Abbasi, John M Dye, David M Kranz, Andrew S Herbert, and Erik Procko. Engineering human ace2 to optimize binding to the spike protein of sars coronavirus 2. Science, 16666 2020.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9268-9277, 2019.
Payel Das, Tom Sercu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann. Flaviu Cipcigan, Vijil Chenthamarakshan, Hendrik Strobelt, Cicero Dos Santos, Pin-Yu Chen, et al. Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering, 5(6):613-6)) 202.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. ICLR, 2020.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv: 11977 2018
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800,, 2018.
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. arXiv preprint arXiv:2006.0m 08052, 2020.
Jessica Fieler and Yoav Goldberg. Controlling linguistic style aspects neural language generation. arXiv preprint arXiv:1707.022 02663, 2017.
11

---

Salvador García and Francisco Herrera. Evolutionary undersampling for classification with imbal- anced datasets: Proposals and taxonomy. Evolutionary computation, 17(3):275-306, 2009.
Rafael Gómez-Bombarelll, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge Aguilea--parraguirre Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268- 276, 2018.
Alex Hawkins-Hooker Florence Depardieu, Sebastien Baur, Guillaume Couairon, Arthur Chen, and David Bikard. Generating functional protein variants with variational autoencoders. PLoS computational biology, 17(2)=e000733, 2021.
Brian I. Hie and Kevin K Yang. Adaptive machine learning for protein engineering. arXiv preprint 011::22066005466. 2021.
Samuel Hoffman, Vijil Chenthamarakshan, Kahini Wadhawan, Pin- Yu Chen, and Payel Das. Optimiz- ing molecules using efficient queries from property evaluations. arXiv preprint 1rXiv:2011.01921, 2020.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2017.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 1909.00899 2019.
2019.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Controlling output length in neural encoder-decoders. arXiv preprint arXiv:1609.0983 09552, 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv: 14, 2013.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367 2020.
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. arXiv preprint 1rXiv:1912.13444 2019.
Ruizhe Li, Xiao Li, Chenghua Lin, Matthew Collinson, and Rui Mao. A stable variational autoencoder for text modelling. arXiv preprint arXiv:1911.05345 2019.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint ar 2004:22000003497,, 2020.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z Sun, Richard Socher, et al. Deep neural language modeling enables functional protein generation across families. bioRxiv, 2021.
Christopher D Manning, Christopher D Manning, and Hinrich Schütze. Foundations of statistical natural language processing. MIT press, 1999.
Aria Mansouri Tehrani, Anton O Oliynyk, Marcus Parry, Zeshan Rizvi, Samantha Couper, Feng Lin, Lowell Miyagi, Taylor D Sparks, and Jakoah Brgoch. Machine learning directed search for ultraincompressible, superhard materials, Journal of the American Chemical Society, 140(31): 9844-9853, 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
12

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprini arXiv: 10683, 2019.
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Serrano. The foldx web server: an online force field. Nucleic acids research, 233pp2222 W388, 2005.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment, arXiv preprint arXiv: 100..00500000 770500965,, 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, Pp. 1631-1642, 2013.
Baris Suzek, Yuqi Wang. Hongzhan Huang. Peter B McGarvey, Cathy H Wu, and UniProt Consortium. Uniref clusters: comprchensive and scalable alternative for improving sequence similarity searches. Bioinformatics, 31(6)922-993, 2015.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv: /7/ 1.015858 2017.
Dinara R Usmanova, Natalya S Bogatyreva, Joan Ariño Bernad, Aleksandra A Eremina, Anastasiya Gorshkova, German M Kanevskiy, I. Lyubov R Lonishin, Alexander V Meister, Alisa G Yakupova, Fyodor A Kondrashov, et al. Self-consistency test reveals systematic bias in programs for prediction change of stability upon mutation. Bioinformatics, 34(21))3653-3658 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et Huggingface's transformers: 2019.
State-of-the-ar natural language processing. arXiv preprint
Zachary Wu, Kadina E Johnston, Frances H Arnold, and Kevin K Yang. Protein sequence design with deep generative models. Current Opinion in Chemical Biology, 65:18-27, 2021.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009. 11448, 2020.
Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang. and Dina Katabi. Delving into deep imbalanced regression. arXiv preprint Ttt::2/02.09554. 2021.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Kirkpairka..iic. Unsupervised text style transfer using language models as discriminators. arXiv preprint arXiv: 1805. 1749, 2018.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint ar
13


Reviews:
[{"review_id": "xWBDO3iwGN", "reviewer": "Reviewer_CSW1", "summary": "The paper considers the problem of generating sequences that have label values higher than those observed in training. An approach is proposed to encode information about the attribute to be optimized into the latent vector of an autoencoder through joint training with a discriminator and a cycle consistency objective. The model is evaluated on two different datasets, one on sentiment in movie reviews and the other on protein stability.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper works on an important problem (extrapolation to unseen values of a label of interest) and proposes an interesting approach. The incorporation of the cycle consistency objective is an interesting idea and shown to be valuable in the ablations. The protein design dataset/task is an interesting contribution to the biology community. Additionally, the paper is well written with clear explanations of models+setup and descriptive usage of figures. However, there are some concerns about how it is situated relative to the literature and prior work in the area, especially wrt to claims of novelty in the task/formal setting, and possibly missing baselines/comparisons in light of this.\n\nThe formal setting here doesn’t appear to be new, other work e.g. Fannjiang and Listgarten 2020,  Brookes et al. 2019 also consider extrapolation to unseen values. The paper could be better situated relative to prior work in this area. Discussion might include Gómez-Bombarelli et al., 2018, Gupta et al., 2019, Linder et al. 2020, Fannjiang and Listgarten 2020,  and Brookes et al. 2019. That being said, the protein stability is an interesting dataset and looks useful for the broader community.\n\nMore extensive comparison to prior work seems warranted. One example being the previous work CbAS (Brookes et al. 2019). The authors write in section 2 that \"Adaptive sampling (Brookes et al., 2019) uses a fixed discriminator/oracle model and iteratively learns the distribution of inputs conditioned on a desirable property using importance sampling.  We view the aforementioned techniques as complementary as GENhance proposes a model-specific architecture for optimizing attributes.\" Although GENhance has a model-specific architecture, there are still meaningful similarities in the motivation and problem setting of the two approaches that warrant a baseline comparison. This is further expressed by the fact that the fluorescence evaluation used by Brookes et al., 2019 is quite similar to the protein stability dataset presented in the paper. Although proper ablation studies have been performed (removing cycle-consistency and smoothing), it is ultimately difficult to judge the significance of the results in the absence of comparisons to other work in the literature. Other potential baselines include FB-VAE (Gupta & Zou 2019), GB-NO (Gómez-Bombarelli et al., 2018), RWR (Peters & Schaal, 2007), or CEM-PI (Snoek et al., 2012).\n\nOn the SST-5 experiments. Some questions around the comparison. Can the model fool the oracle by generating in a space that is adversarial for the oracle? It might be helpful to control for the fluency of generations e.g. as measured by perplexity. It looks like GENhance has significantly higher perplexity than Baseline Gen-Disc which might confound interpretation of results. Additionally, why are only 'Neutral'-labeled SST-5 sequences used as the initial sequences for the models? Why not insert Strong-negative and negative examples as well? So far, the results only show that the model learns a landscape from neutral to positive, but not necessarily negative -> neutral -> positive.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Discussion seems reasonable and they do not overstate the ability of their approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "vGrGg-PAZu", "reviewer": "Reviewer_XZaE", "summary": "This work introduces a framework for learning representations that can extrapolate attributes in text generation.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- About contribution and novelty. \n  - The idea proposed in this project is a special case of using discriminators to learn disentangled representations. \n  - Another question about this work is the controllability. Particularly, for the case of boosting text attributes from \"Neutral\" to \"Strongly-positive\", as both \"Strongly-positive\" and \"Weak-Positive\" were not in the training set, how did the model learn the difference between these two types of attributes? Furthermore, how can we control the model to generate a \"Strongly-positive\" sentence instead of a \"Weak-Positive\" one? \n  - Overall, I did not see any theoretical evidence that the proposed idea can handle the extrapolation problem discussed at the beginning of this paper\n- About modeling\n  - For smoothing the latent space, why not use word perturbation to create $\\tilde{x}_{\\alpha}$? What is the advantage of using this cycling strategy? \n  - What is the justification of simplifying $z_{||}$ as a 1-dimensional scalar and $f_{||}$ as an identity operation? (line 160 - 161)\n- About experiments\n  - There is no comparison between the proposed models and some existing works on controllable text generation. I noticed that in table 1, several baseline models were reported. However, none of them is from the existing codebase, which should not be difficult to find.\n  - There is a potential mismatch for the baseline Gen-Disc models. As explained in the paper, Gen-Disc was trained with an empty string as the input to the encoder (line 179), while for evaluation, \"Neutral\"-labeled sequences are used as inputs for generation (line 191). This mismatch may degrade the actual performance of baseline models. On the other hand, GENhance used text inputs during both training and evaluation. \n  - In addition to the average of 100 rounds of sub-sampling, it would also be interesting to report the variance and also the overall performance on the entire 25K generated texts.\n  - Recall the goal of this work is generating \"sequences with target attribute values that are better than the the data\" (there is an extra \"the\"). To verify this claim, at least for text generation, we should know the human evaluation performance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Part of the potential negative societal impact was addressed in the social impact section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.0"}, "extra_text": {}}, {"review_id": "gBc8TC1zkxq", "reviewer": "Reviewer_6v4B", "summary": "GenEnhance proposes a probabilistic sequence autoencoder combined with a discriminator for ranking sampled sequences, with the goal of enhancing capability of generating better sequences (in term of certain attribute). For this purpose, a disentangled auto encoder is used, which leverages a contrastive loss on top of  WAE-MMD loss, reconstruction loss and a cycle-consistency objective. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important area of controllable generation of better samples. The model formulation leverages known architectures and loss objectives. The performance of the model is shown on two different applications - positive sentence generation using a sentiment dataset and more stable protein sequence generation using a protein variant dataset. \n\nA number of baselines has been considered including a random--MCMC, MCMC from a T5 model, as well as a number of ablation experiments have been performed. The paper is nicely and crisply written. \n\nThe paper work does not compare Genhance with natural language baselines from other published works such as PPLM or GeDi. Also, the paper does not discuss or compare with a number of recent works on controllable biological sequence generation (e.g.  Nat Biomed Eng 5, 613–623, 2021. https://doi.org/10.1038/s41551-021-00689-x, https://doi.org/10.1371/journal.pcbi.1008736) as well as the line of work that performs optimization on the latent space to generate better sequences, particularly in the context of molecular/biological sequence optimization (http://proceedings.mlr.press/v97/brookes19a.html, https://pubs.acs.org/doi/10.1021/acscentsci.7b00572, https://arxiv.org/abs/2011.01921). The paper should include this line of work in the related work section. \n\nAlso, it is not apparent from the presented results to what extent generated sequences are novel (for both English and protein tasks). The authors should include a novelty analyses of the generated samples. \n\nFor both experiments, the work considers controlling generation by a single attribute. Can authors include experiments to show how Genenhance enhances better quality sample when one needs to control two attributes. \n\nAlso, can the authors investigate the effect on Genhance performance  when there are no/less number of stable sequences  in the training data.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hrs"}, "extra_text": {}}, {"review_id": "9w1sgHXIDX", "reviewer": "Reviewer_1cpZ", "summary": "The paper describes GenEnhance, a method for generating sequences with attributes that do not exist in the training distribution. GenEnhance is based on a discriminator / generator architecture trained with a cycle consistency loss similar to existing methods. The main innovation of GenEnhance is that the discriminator is trained with a contrastive loss for learning to rank generated samples by their attribute. \nThe paper is well written and combining a contrastive loss with a cycle consistency loss for controllable generation is new as far as I know. However, it is unclear how the cycle-consistency loss performs relative to regression losses and how GenEnhance performs relative to existing methods for model-based optimization in the visible and latent space.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Major comments\n1) I am considering the use of the contrastive loss for training the discriminator and combining it with the cycle-consistency loss as the main contribution of the paper. Using a cycle-consistency loss alone for training a generative model is not new. However, the ablation experiments do not show how cycle-consistency + contrastive loss performs relative to cycle-consistency + regression loss. How does the contrastive loss compare to traditional regression losses, e.g. mean-squared error?\n\n2) It is unclear how GenEnhance performs relative to existing methods for optimizing sequences in both the latent space (as GenEnhance does) and visible space (similar to the MCMC and Gen-Disc baseline). I would like to see the following baselines:\n* DbAs / CbAs (http://arxiv.org/abs/1810.03714, http://arxiv.org/abs/1901.10060): Train a discriminator (with regression loss and optionally also contrastive loss) on the training set and optimize it by cross-entropy optimization using a VAE. This baseline will be similar to the Gen-Disc baseline, with the difference that the generative model is updated by cross-entropy optimization.\n* Gomez-Bombarelli (https://doi.org/10.1021/acscentsci.7b00572): Train a VAE and discriminator in the latent space and optimize the discriminator in the latent space to generate new sequences. This baseline will show how GenEnhance performs relative to Bayesian optimization performed in the latent space of a VAE.\n\n3) How does the performance of the MCMC baseline depend on a) the mutation rate and b) the number of MCMC iterations (optimization steps)?\n\n4) Which and how were hyper-parameters optimized?\n\n5) The protein design experiment does not show if generative sequences are realistic. Please report the perplexity using a generative model (e.g. DCA or HMM model) fit on homologous sequences of the ACE2 parent sequence.\n\n6) The ACE2 dataset includes 250K training samples. However, the number of labeled sequences that can be used as a starting point for the optimization is smaller in practice (in the order of dozens to hundreds). I would therefore like to see how GenEnhance performs on a dataset with fewer training samples. For example, you may consider the GFP or stability dataset described in the Rao et al (http://arxiv.org/abs/1906.08230). Also perform two different dataset splits as you did for SST-5.\n\n# Minor comments\n7) l144-146: Please show experimentally that the performance of the discriminator improves (as you assume) when it is also trained with generated samples.\n\n8) l19: Books et al 2019 describe one specific method for protein design. I suggest to replace the reference by a more comprehensive review of the field, e.g. \nhttps://www.sciencedirect.com/science/article/pii/S2405471221002039\nhttps://www.sciencedirect.com/science/article/pii/S136759312100051X\nhttp://arxiv.org/abs/2106.05466\n\n9) Is the generator of the Gen-Disc baseline trained on all training sequences or only the best sequences (e.g. all sequences with y > 90% threshold)? Training it on only the best sequences will most likely increase the likelihood of generating sequences that are better than training sequences when sampling from it.\n\n10) l190, accuracy of the oracle model: What is the AUC-ROC and AUC-PR? What is the fraction of strong-positive vs. neutral/negative reviews? Accuracy is in appropriate to quantify the performance of the oracle if class is imbalanced. \n\n11) [optional]: I suggest combining Table 1 and Table 2 to make it easier for the reader to compare performances for the two dataset splits.\n\n20) [optional]: Using the perplexity of a generative model for quantifying the quality of generated sequences SST-5 sentences has known limitations. I suggest strengthening the experiment by also evaluating the top sequences using human ratings (e.g. from Amazon Mechanical Turk).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for generating sequences with attributes or characteristics that are not present in the training data. It is based on an encoder-decoder  model, trained according to a regularized loss. The applications concern two domains: more positive sentence generation and more stable protein sequence generation. The reviewers agree that the topic, extrapolating to unseen attribute values for sequence generation is important and that the technical contribution – through the combination of a contrastive loss and a cycle consistency loss is original.

The initial reviews highlighted weaknesses concerning two main aspects: the positioning w.r.t. the literature and the lack of experimental comparisons with SOTA baselines.  The authors provided a strong rebuttal answering with details to the different questions and remarks. They provided comparisons with two new baselines suggested by the reviewers and performed a qualitative human evaluation also suggested in the reviews. They additionally introduced new ablation analyses. The reviewers have appreciated the quality of the answers and in light of this, they all raised their scores. I propose an accept.
Decision: Accept (Poster)