Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers a text style transfer task where no paired sentences are available. For example, we have positive Clothing reviews and negative Cell Phone reviews, the paper tries to generate negative Clothing reviews. Note that it is likely the model transfers not only the sentiment (from positive to negative) but also product category (from clothing to cell phone). To avoid that, the paper proposes to learn an invariant style classifier (which can classify the sentiment regardless of the product category). Besides, the paper also proposes to learn an orthogonal classifier which can monitor any style-independent changes (e.g. product category). 
main_review: The paper is tackling an important problem where we only want to change certain aspects of the text and maintains other characteristics. 
The main concern is about the effectiveness of the proposed method. According to the human evaluation in Table 5, the other two baselines are doing better or close to the proposed method. I would also suggest adding human evaluation experiments on M w/C_ERM and M w / C_s. In Table 4, we also see that ERM loss achieves the best sentiment accuracy.  It shows better category accuracy, however, the category classifier only has a test accuracy around 71%, which makes the Category Acc column not very informative. The paper has demonstrated relatively good results on the synthetic sentiment transfer task. On the other hand, it would make the paper stronger if more real tasks and datasets can be evaluated. 

Another missing part is the ablations studies of L_LM and L_BT in Eq 9. I understand that these two losses have been investigated in other works, it would still be good to know the benefits from them. I would also suggest adding qualitative results of ERM loss in Table 3 and Table 6, as well as the results of C_s in table 6. And a follow-up question is has the combination of C_ERM and C_o been tested ever? It might be interesting to take a look.
summary_of_the_review: The paper needs stronger empirical results. The proposed method is mainly built on top of Invariant Risk Minimization, thus lacks novelty. There are also ablation studies missing. I would recommend to reject this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a method that can work on a single text style transfer task with multiple datasets. One example scenario as exemplified in the paper is the sentiment transfer task with datasets on multiple product categories. The basic idea of this work is to design a cross alignment between datasets from different categories, so the learning algorithm can focus on the expected stylistic information and eliminate other confounding factors. The proposed methods were evaluated on the sentiment transfer task with two different setups to demonstrate the value. 
main_review: Strengths

- The idea of using Invariant Risk Minimization is interesting. 

Comments/questions on the proposed methods and evaluation

- In section 3.1.2, how to learn `an IRM classifier`? Or, to be specific, how this classifier can be implemented? 
- In table 2, if the sentiment classifier (84.3) can outperform human reference (76.8), I am not sure whether we should trust it or how we should interpret its results? 
- In section 4.2, it is a good idea of using BLEU with source sentences as a way to measure content preservation, however, this is not sufficient --- when input copy can give a perfect BLEU score, then we should not read too much from 59.2. Additional evaluation methods are needed here. 
- Overall, the proposed method seems to be similar to relevant to the idea of learning discriminators for generation tasks, which has been used in prior work of text style transfer. Based on the description, I am not sure whether the proposed method can be better than learning a collection of discriminators, where each discriminator by design will tell or not tell the difference within an environment.
- As a method that has the potentials to be used in many other problems (as described in the introduction), it is disappointing to see both evaluation tasks are sentiment transfer. Would it be possible to at least try some other text style transfer tasks? 


Comments on writing

- It would be great if the explanation of IRM in section 3.1 can be self-contained and based on the proposed task (text style transfer)
- In section 3.1.1, it says 
> style is a feature representation that elicits an invariant classifier across different environment

In general, I don't think this statement is valid. By constructing the `environments`, it can eliminate other factors, but there is no guarantee that style is the only factor left. 
summary_of_the_review: The major concerns of this work are the novelty of the proposed method and the sufficiency of evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This article deals with style transfer. It proposes an approach to generate texts with opposite sentiments in a adversarial context where the style is correlated with other contextual factors such as domain or noisy punctuation.
Given the fact that other contextual factors are weakly labeled, the authors introduce Invariant Classifers to extract the style from other attributes.
The general framework has been defined by (Arjovsky et al., 2019, IRM) and the basic classifier is (Kim, 2014). Section 3.2 describes the authors' contribution where multiple orthogonal classifiers enables them to transfer style while preserving other attributes.
The authors propose relevant baselines corresponding to hypotheses ablations. The text generator is a 1-layer LSTM.
The authors demonstrate that their approach is able to transfer sentiment while preserving domain or punctuation depending on the experiment.
main_review: Pros:

+ interesting application and synthetic dataset

Cons:

- outdated encoder and text generator

- less general framework than [Subramanian et al., 2018] 

==

I don't understand the synthetic noise addition by punctuation change: what are the changes?

Why relying on a so old model for text modeling? What would be the performance of BERT or T5 on the same task?
Gao, Z., Feng, A., Song, X., & Wu, X. (2019). Target-dependent sentiment classification with BERT. IEEE Access, 7, 154290-154299.

This framework is very reductive compared to [Subramanian et al., 2018] in which feeling is simply one attribute among others. One wonders what this approach would look like in the context of the experiments conducted in [Subramanian et al., 2018].

The authors propose to compare their approach with baselines that do not take into account all hypothesis... But they do not compare their work with existing approaches from the litterature.

Is it reasonnable to use a 1-layer LSTM for text generation purpose today? Even if GPT is very expensive, BART or T5 could have been investigated for such a task.

Regarding table 2, I understand that Puntuation ACC can be low but I don't get how Sentiment ACC can be so low whereas it is a binary classification task.

Generally speaking, sentiment accuracy seems not competitive with state of the art models, even if I am not familiar with the dataset from  Shen et al. (2017). 

Comparing authors approach with Krishna et al. (2020) seems not relevant (even according to the authors)
summary_of_the_review: This work is interesting and consistant but the fact that both the encoder and the generator are outdated seems critical regarding a pure NLP paper. Despite some clear qualities, I have to reject this contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Style transfer is a text generation task where a certain attribute of a sentence (like formality) is modified while preserving sentence content. It is generally studied in "unsupervised" settings --- no access to parallel data of sentences differing only in the target attribute. Nevertheless, algorithms assume access to corpora of unpaired sentences in each of the attributes, which may significantly differ in content. Prior works [1, 2, 3, 5] have noted this difference in content, which is encouraging models to modify non-target attributes like semantics (See Table 1 in [4]).

This paper is an attempt to fix this issue using invariant risk minimization (IRM) [6]. IRM encourages classifiers to be invariant across "environments", or datasets which share a target class but differ in other aspects which are not essential for classification ("spurious correlations"). In this work two spurious correlations are studied in sentiment transfer --- (1) a synthetic punctuation correlation; (2) product category of Amazon reviews. To perform style transfer, the authors first train two orthogonal IRM classifiers --- the first classifies the target attribute ("sentiment") irrespective of the confounder ("product category"), and the second for the confounder ("product category") irrespective of sentiment. These two classifiers are used to guide model generation, along with an LM smoothing and backtranslation objective (similar to [4]).

The authors compare their proposed approach against strong baselines and report promising results.

[1] - https://arxiv.org/abs/1910.03747  
[2] - https://arxiv.org/abs/2010.05700  
[3] - https://dl.acm.org/doi/10.5555/3016100.3016326  
[4] - https://arxiv.org/abs/1811.00552  
[5] - https://arxiv.org/abs/1905.13464  
[6] - https://arxiv.org/abs/1907.02893
main_review: 
**Strengths**

1. Style transfer is an important problem in natural language generation, with several practical applications (as discussed in [1]). The authors have identified a worrying issue in current style transfer systems (modification of non-target features like content), and attempted to fix this issue via invariant risk minimization (IRM).

2. The methods developed in this paper are quite novel --- to the best of my knowledge this is the first work using IRM for style transfer. IRM is a good fit for the confounder issue in style transfer, and I found the ideas in the paper interesting. The authors use an interesting set of loss functions to use these orthogonal classifiers to guide the model to perform style transfer without modification of confounders.

3. The authors test the approach on sentiment transfer, studying one synthetic confounder and one real-world confounder (product category). Experiments confirm the models trained by the model modify sentiment while keeping the confounder consistent. The authors test their method using both automatic and human evaluations, comparing it against strong baselines. On both kinds of metrics the authors see improvements using their proposed approach.

**Weaknesses**

1. My biggest concern with IRM is the need to identify environments across which the style classifiers should be consistent. In the sentiment transfer case, product category is certainly a valid confound which leads to semantic modification, but there could be several others, for instance intra-category variation of features of different products, writing style of author (formality / simplicity / usage of analogies). Obtaining these group annotations might be difficult in practice. The current approach also needs an orthogonal classifier, which needs a well-defined label space for the confounding factor. Many confounds related to semantics might be hard to precisely define & identify.

2. It would be useful to see the applicability of this approach on one more task which is not sentiment transfer. One possibility could be using the GYAFC formality transfer corpus, which has two categories of data (Entertainment & Music, Family & Relationships). Also while maybe this is out of the scope of this work, it will be interesting to see if confounding factors can be identified automatically. In a task like converting Shakesepare <---> Tweets (https://arxiv.org/abs/2010.05700), there will be several confounding features between the two styles. For instance, Shakespeare data will be on topics from the 1500s, whereas Tweets will mostly contain modern 21st century topics. These features will need to be disentangled from style (lexical / syntactic choice) for successful semantic preserving style transfer. Here it will be harder to segregate the corpus into suitable environments.

3. The ablation studies in Appendix B could be stronger --- what happens when you completely remove certain loss functions (weight = 0)? Also it's hard to compare between rows in this table since it's missing an "overall score" / aggregation like you did for the human evaluation.

**Minor**

1. The human evaluation is missing checking the accuracy of the confounding factor like you did for automatic evaluation ("CATEGORY ACC")

2. I would avoid branding this paper as "style transfer" since style generally refers to properties unrelated to semantics unlike sentiment (which is the focus of this work). The term "attribute transfer" might be more appropriate.
summary_of_the_review: Overall, I thought the paper was well-written, had very interesting ideas and a good evaluation / comparison against prior work. Weakness #1/#2 will prevent me from going to the next higher score (of 8), but I'm in favour of acceptance.
review_title: null

Meta-review: This paper studies text style transfer which aims to edit a given sentence to possess a desired style value (e.g., positive sentiment) while keeping all other styles and content unchanged. The paper specifically focuses on a challenging setting where besides the target style (e.g., sentiment) to transfer, there exists confounding attributes (e.g., product category) that correlate with the target style, making it hard to change only the target style while preserving the other. The proposed approach is to learn an invariant/unbiased style classifier using Invariance Risk Minimization (IRM), together with an orthogonal classifier for monitoring style-independent changes (e.g., product category), to supervise the generator training. The main concerns are on the experiments -- it's suggested to include experiments on other styles besides sentiment; human evaluation and/or other metrics are needed for more convincing comparison; it's also encouraged to experiment with large language models (e.g., GPT-2, BART) besides the small LSTM/CNN networks as in the present work.
Decision: Reject