Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a unified causal framework for controllable text generation tasks (attribute-conditional text generation and attribute text transfer). The authors propose a structural causal model (SCM) to formulate the two tasks as causal intervention and counterfactual reasoning in the context of causal latter, which thus provides a unified framework for studying controllable text generation.  The learning stage aims to integrate variational learning (VAE) and confounder disentanglement (and balancing). 

For the evaluation, the authors reused Yelp and Bios datasets, and then added synthetic spurious correlations, creating challenging data scenarios.  They evaluate the proposed model with a few simple baseline methods in terms of control accuracy, bias, fluency, and diversity. To support the automatic evaluation of these metrics, they use pre-trained attribute classifiers. They also conduct human evaluations to show the effectiveness of their proposed methods.  

Overall, the main contribution of the paper is a casual perspective for two main controllable text generation tasks via casual interventional inference and counterfactual reasoning. The method can be used to reduce toxic content (e.g., gender bias) in downstream applications that require automatic text generation. 

main_review: ***Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?***

The controllable text generation tasks,  particular causal framework, and learning methods are all existing techniques. The work is rather a novel combination of them and provides a novel and valuable perspective. The casualty-centric framework is adequately different from prior works in the same direction, most of which are also well cited. 


***Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?***

The proposed method itself is sound and complete. The evaluation section is rather inadequate. There are only two simple datasets and the attribute and confounding factors are very simple. The baseline methods for controlling attributes and debiasing are not compared. The authors only used a simple "Conditional LM" as the key baseline method. Therefore, the comparisons are not that convincing to me. 



***Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)***

Most of the parts are clearly written, while I would suggest the authors add more concrete examples of attributes vs confounding factors in the method section. Maybe the authors can simply add a problem formulation section and use the cases in the dataset description paragraphs (e.g., sentiment vs categories). Another improvement of the presentation could be adding a figure for the overview architecture of the proposed method, i.e., connecting the GPT2 and the attribute classifier and so on to illustrate how the method works in terms of the implementation. 


***Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?***

As stated above, the empirical results in the paper are inadequate, although the reported numbers show that the proposed method is much better than a simple baseline. It is, however, not convincing enough that the proposed method is significantly better than more sophisticated baseline methods --- either recent conditional text generation methods or the methods to remove spurious correlation (for NLG and other tasks). The style transfer experiment does not use other popular datasets in the community, so it is hard to directly compare it in a broader context. 

Also, the analysis of the results is also superficial and does not give any insights and implications why such a casual framework is significantly better. The experiment section is barely a report of the results. Adding more dedicated analysis with concrete examples would improve the paper a lot. 

***Other weaknesses. ***

The proposed method (and the evaluation) is only limited to a single target attribute and a single confounding factor. What are the key challenges for extending the method (and the experiments) to multiple target attributes with multiple confounding factors? The authors should have explicitly discussed the limitation and the possible ways to address them as this is a key weakness of the paper. 

Minor question: Would it be possible to apply the method to lexical-constrained text generation tasks such as CommonGen? 
limitations_and_societal_impact: ***Have the authors adequately addressed the limitations and potential negative societal impact of their work?***

This proposed method in this paper can be used to reduce the bias in automatic text generation tasks and thus can lead to a positive social impact. However, it is also possible that someone can use the method to generate even toxic text by controlling the attribute in a bad way.  In this aspect, I would suggest the authors add a simple paragraph in the final version to discuss the social impact from both aspects.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces methods to integrate causality in the task of controllable text generation, specifically on attribute-conditioned generation and text attribute transfer. The goal is to eliminate biases from spurious correlations with confounders (e.g., jobs when controlling the gender of the text). They do this by introducing a latent confounder variable and an observed proxy variable. They trained their models using a VAE objective, as well as three counterfactual objectives to help mitigate model collapse.
main_review: Strengths
* Controlling text while removing biases and spurious correlations is one of the most important problems in controllable text generation. The method introduced in the paper is intuitive and the results show that it works better than competing baselines. 

Weaknesses
* It would be very helpful to readers who are more familiar with NLG but less familiar with work on causality to provide definitions to terms on this area (e.g., outcome, treatment, confounders, and their relations).
* The ablation could have been more extensive. For example, what happens when only one of the three counterfactual objectives is used?

limitations_and_societal_impact: The paper does not have separate sections for limitations and societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the problem of the attribute (sentiment, gender) controlled text generation or text style transfer. The authors formulate the problem from a principled causal perspective, which models the two tasks with a unified framework. A framework of structural causal model is introduced to ground the idea. Experiments on text generation and style transfer are conducted and the experimental results show that the proposed framework can indeed improve the control accuracy.
main_review: This paper investigates the problem of the attribute (sentiment, gender) controlled text generation or text style transfer. The authors formulate the problem from a principled causal perspective, which models the two tasks with a unified framework. A framework of structural causal model is introduced to ground the idea. Experiments on text generation and style transfer are conducted and the experimental results show that the proposed framework can indeed improve the control accuracy.


Strength:

- The paper is well written and easy to follow. The authors describe the technical content detailly and readers can easily understand the implementation tricks.

- The proposed framework unifies the tasks of attribute-controlled text generation and style transfer which using one model can conduct two tasks simultaneously.


Weakness:

- Although the authors present the story from a principled causal perspective which is good certainly, it is not easy to figure out the novelty and technical contribution compared with the previous methods (such as CVAE model). The basic and main idea is still a VAE-based framework with some more variables or tricks to improve the performance, therefore, from the technical derivatives and descriptions, it is hard to feel excited and benefit more.  

- It is good to see that Conditional LM is introduced as the baseline method to conduct comparisons. I am wondering what is the performance of CVAE model which GPT2 is employed as the language decoder. This comparison is more important and convincing. 


limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The goal of the paper is to control an attribute of the text without changing another associated attribute. Let's use the biased YELP dataset the authors created as an exampe. In the dataset, 90% of positive reviews come from restaurants and 90% of negative reviews come from the other categories (e.g., shopping). Their goal is to control the sentiment (attribute a) without changing its category (confounding factor c). This is difficult because the conditional generator might just generate a restaurant review when the task is to generate a positive review. Similarly, if most of the nurses are female, the text generator might just generate nurses to satisfy the requirement of generating a female character.

In conditional generation such as a seq2seq model, even the controlled attribute is given as the condition, the text generator might still generate the text that might imply confounding factors or some other attributes because the confounding factors and controlled attribute are associated or the prior probabilities of those factors/attributes are high. The paper solves the problem using backdoor adjustment (removing the dependency between the attribute a and the hidden representation of the sentence z). My understanding of their method is like assuming confounding factors and controlled attributes are independent or ignore the prior probability of observing other attributes. This would make the generated text less fluent but can generate the text that would have the attribute in a high probability.

The experiments show that the proposed method is significantly better than conditional LM or other text attribute transferring approaches when the training data is biased (e.g., the review category is highly correlated with the sentiment of the review in YELP). When the dataset is unbias, one experiment seems to show that the proposed method is also better than other approaches.

main_review: I think the problem setting is practical. For example, people might want to remove an undesired attribute that has a high correlation with other desired attributes (e.g., remove the toxic attribute but keep the negative sentiment). The proposed method also seems to work pretty well. I did not know any other direct-related work. Thus, the originality and significance are good, which might lead to high impact. 

Nevertheless, the experiments and presentation still have sufficient room for improvement. First, the paper does not say very clearly whether the proposed method is better than existing approaches in the popular unbiased benchmarks. The evaluation setting of the unbiased YELP experiment is not specified clearly. Is the proposed model trained on the unbiased dataset in this experiment? How do you tune the hyperparameters of your model and other methods? Do you think where the improvement comes from? Do all the models use the same amount of resources and are trained on the same data? For example, does the proposed method use category information but other methods do not? Is your model size (number of parameters) larger than previous approaches (I suggest you put all the model sizes into your table)? Is the classifier given when you train other models? Do all the models use the same classifier? Could you also compare the diversity in the text attribute transfer task? Could you also compare your methods with other approaches in a more standard evaluation (e.g., Table 9 in https://arxiv.org/pdf/2010.12742.pdf)? You use the predicted confounding labels to train Conditional LM (full), so I am not sure whether it performs worse than your model due to the noise in the predicted labels or the effectiveness of your model. Did you try to train your casual model on such predicted labels rather than the ground truth label subsets?

If the authors cannot show that the proposed method is consistently better than other approaches in unbiased datasets in a fair comparison. I think emphasizing the focus of this paper earlier would make the paper much easier to read. For example, the current title and introduction give me the impression that this paper proposes a general approach to all types of controllable text generation tasks. I do not understand the main motivation behind the paper until I saw the creation of the biased YELP dataset in the experiment section. I recommend the author to mention some concrete examples (e.g., in your biased YELP dataset) in the introduction or Figure 2 to explain what a, c, x, and z mean and emphasize the focus of this paper. In addition, in Tables 1 and 2, the YELP dataset should be called biased YELP to avoid confusion.

Finally, I think more ablation studies are needed (The paper only tries another baseline in the ablation study). The architecture is complicated (e.g., line 162 says it even uses GAN) and you have seven loss terms. Why do we need to use GAN here? If the proposed method is better than other approaches in the unbiased setting, we would like to know how much each model design or each loss term contributes to the improvement.

If the rebuttal addresses my concerns about experiments and presentations well, I will vote acceptance (>=7) for this work. You don't have to answer all the questions I asked. I just want to know whether this method is generally better than state-of-the-art results in unbiased datasets in a fair comparison. If no, majorly revise your presentation. If yes, provide more pieces of evidence (and minorly revise the paper to make your motivation more concrete). 


Some minor suggestions:
1. More detailed description of model architectures should be included in the appendix. It would be even better to add a figure to explain architecture. This is especially important if you want to claim the proposed method is generally better than others in unbiased datasets. I only have a very high-level understanding of the model after reading the whole paper.
2. A paper I saw usually won't describe the baselines inside the result section/paragraph. Explain cf-z/c baseline before the result paragraph and mention the cf-z/c keywords that allow the readers to search.
3. In the paper, you mentioned that your method is related to disentangled representation studies. The relation of your work to the related text style transfer studies such as [1] could be discussed in the paper.
4. I think the perplexity numbers are usually positive values and smaller is better.
5. What's the n value you used in the distinct-n diversity metric?

[1] John, Vineet, et al. "Disentangled Representation Learning for Non-Parallel Text Style Transfer." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
limitations_and_societal_impact: One potential negative societal: Although neural style transferring could be used to remove toxic attributes but also could be used to inject toxic attributes, I do not think explicitly mentioning this possibility would have a positive social impact. Thus, I think the paper should keep as it is on this part.
review_title: null

Meta-review: The submission proposes a method for debiasing controllable text generation, allowing text to be generated that satisfies one desired attribute, without also introducing confounding attributes. The reviewers agree that this is an important problem, and that a causal perspective is valuable here. The major concerns raised by the reviewers focused on the evaluation, however the authors have provided additional experimental results in their response that significantly strengthen the conclusions. Therefore, I recommend acceptance.
Decision: Accept (Poster)