Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: GenEnhance proposes a probabilistic sequence autoencoder combined with a discriminator for ranking sampled sequences, with the goal of enhancing capability of generating better sequences (in term of certain attribute). For this purpose, a disentangled auto encoder is used, which leverages a contrastive loss on top of  WAE-MMD loss, reconstruction loss and a cycle-consistency objective. 


main_review: The paper addresses an important area of controllable generation of better samples. The model formulation leverages known architectures and loss objectives. The performance of the model is shown on two different applications - positive sentence generation using a sentiment dataset and more stable protein sequence generation using a protein variant dataset. 

A number of baselines has been considered including a random--MCMC, MCMC from a T5 model, as well as a number of ablation experiments have been performed. The paper is nicely and crisply written. 

The paper work does not compare Genhance with natural language baselines from other published works such as PPLM or GeDi. Also, the paper does not discuss or compare with a number of recent works on controllable biological sequence generation (e.g.  Nat Biomed Eng 5, 613–623, 2021. https://doi.org/10.1038/s41551-021-00689-x, https://doi.org/10.1371/journal.pcbi.1008736) as well as the line of work that performs optimization on the latent space to generate better sequences, particularly in the context of molecular/biological sequence optimization (http://proceedings.mlr.press/v97/brookes19a.html, https://pubs.acs.org/doi/10.1021/acscentsci.7b00572, https://arxiv.org/abs/2011.01921). The paper should include this line of work in the related work section. 

Also, it is not apparent from the presented results to what extent generated sequences are novel (for both English and protein tasks). The authors should include a novelty analyses of the generated samples. 

For both experiments, the work considers controlling generation by a single attribute. Can authors include experiments to show how Genenhance enhances better quality sample when one needs to control two attributes. 

Also, can the authors investigate the effect on Genhance performance  when there are no/less number of stable sequences  in the training data.
limitations_and_societal_impact: See above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper considers the problem of generating sequences that have label values higher than those observed in training. An approach is proposed to encode information about the attribute to be optimized into the latent vector of an autoencoder through joint training with a discriminator and a cycle consistency objective. The model is evaluated on two different datasets, one on sentiment in movie reviews and the other on protein stability.
main_review: The paper works on an important problem (extrapolation to unseen values of a label of interest) and proposes an interesting approach. The incorporation of the cycle consistency objective is an interesting idea and shown to be valuable in the ablations. The protein design dataset/task is an interesting contribution to the biology community. Additionally, the paper is well written with clear explanations of models+setup and descriptive usage of figures. However, there are some concerns about how it is situated relative to the literature and prior work in the area, especially wrt to claims of novelty in the task/formal setting, and possibly missing baselines/comparisons in light of this.

The formal setting here doesn’t appear to be new, other work e.g. Fannjiang and Listgarten 2020,  Brookes et al. 2019 also consider extrapolation to unseen values. The paper could be better situated relative to prior work in this area. Discussion might include Gómez-Bombarelli et al., 2018, Gupta et al., 2019, Linder et al. 2020, Fannjiang and Listgarten 2020,  and Brookes et al. 2019. That being said, the protein stability is an interesting dataset and looks useful for the broader community.

More extensive comparison to prior work seems warranted. One example being the previous work CbAS (Brookes et al. 2019). The authors write in section 2 that "Adaptive sampling (Brookes et al., 2019) uses a fixed discriminator/oracle model and iteratively learns the distribution of inputs conditioned on a desirable property using importance sampling.  We view the aforementioned techniques as complementary as GENhance proposes a model-specific architecture for optimizing attributes." Although GENhance has a model-specific architecture, there are still meaningful similarities in the motivation and problem setting of the two approaches that warrant a baseline comparison. This is further expressed by the fact that the fluorescence evaluation used by Brookes et al., 2019 is quite similar to the protein stability dataset presented in the paper. Although proper ablation studies have been performed (removing cycle-consistency and smoothing), it is ultimately difficult to judge the significance of the results in the absence of comparisons to other work in the literature. Other potential baselines include FB-VAE (Gupta & Zou 2019), GB-NO (Gómez-Bombarelli et al., 2018), RWR (Peters & Schaal, 2007), or CEM-PI (Snoek et al., 2012).

On the SST-5 experiments. Some questions around the comparison. Can the model fool the oracle by generating in a space that is adversarial for the oracle? It might be helpful to control for the fluency of generations e.g. as measured by perplexity. It looks like GENhance has significantly higher perplexity than Baseline Gen-Disc which might confound interpretation of results. Additionally, why are only 'Neutral'-labeled SST-5 sequences used as the initial sequences for the models? Why not insert Strong-negative and negative examples as well? So far, the results only show that the model learns a landscape from neutral to positive, but not necessarily negative -> neutral -> positive.
limitations_and_societal_impact: Discussion seems reasonable and they do not overstate the ability of their approach.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work introduces a framework for learning representations that can extrapolate attributes in text generation.
main_review: - About contribution and novelty. 
  - The idea proposed in this project is a special case of using discriminators to learn disentangled representations. 
  - Another question about this work is the controllability. Particularly, for the case of boosting text attributes from "Neutral" to "Strongly-positive", as both "Strongly-positive" and "Weak-Positive" were not in the training set, how did the model learn the difference between these two types of attributes? Furthermore, how can we control the model to generate a "Strongly-positive" sentence instead of a "Weak-Positive" one? 
  - Overall, I did not see any theoretical evidence that the proposed idea can handle the extrapolation problem discussed at the beginning of this paper
- About modeling
  - For smoothing the latent space, why not use word perturbation to create $\tilde{x}_{\alpha}$? What is the advantage of using this cycling strategy? 
  - What is the justification of simplifying $z_{||}$ as a 1-dimensional scalar and $f_{||}$ as an identity operation? (line 160 - 161)
- About experiments
  - There is no comparison between the proposed models and some existing works on controllable text generation. I noticed that in table 1, several baseline models were reported. However, none of them is from the existing codebase, which should not be difficult to find.
  - There is a potential mismatch for the baseline Gen-Disc models. As explained in the paper, Gen-Disc was trained with an empty string as the input to the encoder (line 179), while for evaluation, "Neutral"-labeled sequences are used as inputs for generation (line 191). This mismatch may degrade the actual performance of baseline models. On the other hand, GENhance used text inputs during both training and evaluation. 
  - In addition to the average of 100 rounds of sub-sampling, it would also be interesting to report the variance and also the overall performance on the entire 25K generated texts.
  - Recall the goal of this work is generating "sequences with target attribute values that are better than the the data" (there is an extra "the"). To verify this claim, at least for text generation, we should know the human evaluation performance.
limitations_and_societal_impact: Part of the potential negative societal impact was addressed in the social impact section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes GenEnhance, a method for generating sequences with attributes that do not exist in the training distribution. GenEnhance is based on a discriminator / generator architecture trained with a cycle consistency loss similar to existing methods. The main innovation of GenEnhance is that the discriminator is trained with a contrastive loss for learning to rank generated samples by their attribute. 
The paper is well written and combining a contrastive loss with a cycle consistency loss for controllable generation is new as far as I know. However, it is unclear how the cycle-consistency loss performs relative to regression losses and how GenEnhance performs relative to existing methods for model-based optimization in the visible and latent space.
main_review: # Major comments
1) I am considering the use of the contrastive loss for training the discriminator and combining it with the cycle-consistency loss as the main contribution of the paper. Using a cycle-consistency loss alone for training a generative model is not new. However, the ablation experiments do not show how cycle-consistency + contrastive loss performs relative to cycle-consistency + regression loss. How does the contrastive loss compare to traditional regression losses, e.g. mean-squared error?

2) It is unclear how GenEnhance performs relative to existing methods for optimizing sequences in both the latent space (as GenEnhance does) and visible space (similar to the MCMC and Gen-Disc baseline). I would like to see the following baselines:
* DbAs / CbAs (http://arxiv.org/abs/1810.03714, http://arxiv.org/abs/1901.10060): Train a discriminator (with regression loss and optionally also contrastive loss) on the training set and optimize it by cross-entropy optimization using a VAE. This baseline will be similar to the Gen-Disc baseline, with the difference that the generative model is updated by cross-entropy optimization.
* Gomez-Bombarelli (https://doi.org/10.1021/acscentsci.7b00572): Train a VAE and discriminator in the latent space and optimize the discriminator in the latent space to generate new sequences. This baseline will show how GenEnhance performs relative to Bayesian optimization performed in the latent space of a VAE.

3) How does the performance of the MCMC baseline depend on a) the mutation rate and b) the number of MCMC iterations (optimization steps)?

4) Which and how were hyper-parameters optimized?

5) The protein design experiment does not show if generative sequences are realistic. Please report the perplexity using a generative model (e.g. DCA or HMM model) fit on homologous sequences of the ACE2 parent sequence.

6) The ACE2 dataset includes 250K training samples. However, the number of labeled sequences that can be used as a starting point for the optimization is smaller in practice (in the order of dozens to hundreds). I would therefore like to see how GenEnhance performs on a dataset with fewer training samples. For example, you may consider the GFP or stability dataset described in the Rao et al (http://arxiv.org/abs/1906.08230). Also perform two different dataset splits as you did for SST-5.

# Minor comments
7) l144-146: Please show experimentally that the performance of the discriminator improves (as you assume) when it is also trained with generated samples.

8) l19: Books et al 2019 describe one specific method for protein design. I suggest to replace the reference by a more comprehensive review of the field, e.g. 
https://www.sciencedirect.com/science/article/pii/S2405471221002039
https://www.sciencedirect.com/science/article/pii/S136759312100051X
http://arxiv.org/abs/2106.05466

9) Is the generator of the Gen-Disc baseline trained on all training sequences or only the best sequences (e.g. all sequences with y > 90% threshold)? Training it on only the best sequences will most likely increase the likelihood of generating sequences that are better than training sequences when sampling from it.

10) l190, accuracy of the oracle model: What is the AUC-ROC and AUC-PR? What is the fraction of strong-positive vs. neutral/negative reviews? Accuracy is in appropriate to quantify the performance of the oracle if class is imbalanced. 

11) [optional]: I suggest combining Table 1 and Table 2 to make it easier for the reader to compare performances for the two dataset splits.

20) [optional]: Using the perplexity of a generative model for quantifying the quality of generated sequences SST-5 sentences has known limitations. I suggest strengthening the experiment by also evaluating the top sequences using human ratings (e.g. from Amazon Mechanical Turk).
limitations_and_societal_impact: yes
review_title: null

Meta-review: The paper presents a method for generating sequences with attributes or characteristics that are not present in the training data. It is based on an encoder-decoder  model, trained according to a regularized loss. The applications concern two domains: more positive sentence generation and more stable protein sequence generation. The reviewers agree that the topic, extrapolating to unseen attribute values for sequence generation is important and that the technical contribution – through the combination of a contrastive loss and a cycle consistency loss is original.

The initial reviews highlighted weaknesses concerning two main aspects: the positioning w.r.t. the literature and the lack of experimental comparisons with SOTA baselines.  The authors provided a strong rebuttal answering with details to the different questions and remarks. They provided comparisons with two new baselines suggested by the reviewers and performed a qualitative human evaluation also suggested in the reviews. They additionally introduced new ablation analyses. The reviewers have appreciated the quality of the answers and in light of this, they all raised their scores. I propose an accept.
Decision: Accept (Poster)