Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the author proposed a transformer-based encoder-decoder framework for label-free text style transfer. The described task under the unsupervised setup is important and instructive for the text style transfer domain. The model architecture is well demonstrated and the writing is easy to follow up. The experiment results show satisfying performance even comparing with state-of-the-art supervised methods. 

However, I have some concerns that may lead to the weakness of the paper:

1. About the assumption: The author claimed the method is label-free. However, the "unsupervised" model is based on an assumption that two adjacent sentences should have the same style. With the assumption, the training of the model is actually weak-supervised because in each step the paired sentences are provided with the same style. This assumption is actually utilizing the context-level supervision instead of the sentence-level labels. This idea is also previously used in [1].

2. About the framework: The model adds the exacted style vector to all the hidden states of the encoder. How can the author guarantee that the encoder will not extract the style information of the input? Also, is it possible that the style vectors still contain the content information from context?

3. About the style vector: The model changes the style of the sentence by adding a direction from the source style vector to the target style vector. The approach may work under the assumption that the style vector space is linear to the semantic meanings. But there is no regularizer or training loss to guarantee the linearity assumption of the style extractor. Why didn't the author directly replace the sentence style vector with the target style vector?

4. About the dataset: The model is only evaluated on one dataset. It could be more solid if the author conduct experiment on other commonly-used style transfer datasets such as Yelp and Personality-Captions [2]. Besides, the split Amazon review dataset only has two sentiment classes as "positive" and "negative". It could be more persuasive if the model is tested on other datasets with multiple sentiments, to verify the effectiveness of the proposed re-styling strategy. 

5. About the evaluation: The author only reported the performance of content and style preservation (Acc and self-BLEU). The sentence generation quality is expected to report by testing the BLEU score of the generated sentences.

6. In Figure 4, there is no clear pattern between positive and negative sentence embedding. The difference in embedding space is mainly caused by different topics, which in my understanding are the content of sentences. This means the style vectors cannot eliminate the content information and also failed to separate sentences with different sentiments.


Reference:

[1] Zhang et al. Improving the Dialogue Generation Consistency via self-supervised Learning, 2019

[2] Shuster et al. Engaging Image Captioning Via Personality, 2018
review_title: The paper proposed a encoder-decoder text style transfer framework without requiring style labels in training. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles the problem of extracting and modeling text (writing) style without using labeled data.

Traditionally, modeling text style requires either paired sentences (supervised) or two pools of unpaired sentences (so-called unsupervised). This paper exploits 1) language model pretraining and 2) free supervision signals in text to achieve modeling text styles without labeled data.

For the 1st point, the authors (correctly) hypothesize that a large pre-trained language model (e.g. T5) already “knows” about style information and one can isolate the style information using the right fine-tun signal.
For the 2nd point, the authors assume that text style (e.g., sentiment) is slow-moving and consistent for adjacent sentences (I guess It’s a similar signal is exploit by Next Sentence Prediction in BERT, and CBOW in word2vec?). And this is used as the “free” supervision signal to their model finetune.

In the experiments section, the authors test their model on transfer learning tasks. The experiments (Fig 2 &3) seem to suggest that at at given high content preservation score ( > 50), the proposed model is not as accurate as other supervised models. But with low content preservation, the model can steadily improve accuracy by modifier more words (Fig 2). 

In Figure 2, the TextSETTR accuracy has almost (inverse) linear response wrt to the content preservation score. But in Figure 3, the plot for TextSETTR stopped at “30-50%”. What would happen if the modification percentage is higher? Would TextSETTR get closer to 100% accuracy?

Another small issue with Figure 3: I believe the task is binary (pos vs neg). It might be more useful to plot the accuracy from 50%~100% instead of from 0% ~100% since 50% is the practical lower bound performance.

review_title: Well motivated method

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Reasons for score: 


This paper proposes a novel approach to the label-free style transfer task where an input is corrupted via different strategies and fed into an auto-encoder which is additionally conditioned on its prior adjacent context sentence via a "style encoder" which adds its mean pooled hidden state to the former before decoding.  Both encoders are initialized to and leverage the strength of pre-trained T5 model.  Additionally the amount of addition/deletion of tokens is tunable at both training and inference time.  

The overall idea is quite compelling, but the paper's argument could be improved greatly with revisions to its existing experimental setup and more evaluation overall to better and more thoroughly back its claims.  

 
##########################################################################Pros: 
Pros:
1) The authors propose a novel approach to the label-free style transfer task that is based on evaluating how training under different combinations of 3 noising strategies  ( Noise, Back Translation and Noisy Back Translation ) on input texts can be used in conjunction with an auto-encoder and style encoder over the prior sentence context to then do inference given an input text and a small number of examplars of the source and target styles.  The idea is laid out fairly clearly both for training and inference though certain particulars there and in the experiments section were a little unclear and could have benefited from some formal notation ( see next section ). 

2) The quantitative results on the Amazon dataset for their best model on both the full data and few shot regimes are quite impressive compared with the other label-free style transfer paper they compare against ( Xu 2020 ) 

3) The few qualitative examples shown are impressive ( particularly the American <-> British ones ) 

4) The tuning hyper parameter is a useful addition ( though it'd be interesting to see how dataset dependent it is )
 
##########################################################################
Cons: 
1) Overall the writing was a little unclear at certain spots and could have benefitted greatly from some equations explicitly stating the setup.  For instance i was unclear if the context representation was added ( which the text suggests ) or concated to the noisy encoding before being decoded ( the later is suggested by Figure 1 especially since the 4 float values for the tuning rate ranges are said to be prepended ).  Similarly the sampling strategy used ( as opposed to greedy decoding )

2) Doing quantitative evaluation on only one dataset ( Amazon ) and then only showing examples of how the model does qualitatively over another dataset ( English Common Crawl "C4" ) without doing any human eval is a little disappointing.  The idea is novel enough where even just doing some more automated evals would be suffice for me.  For instance, why weren't automated metrics given for the English Common Crawl dataset?  Those results and having information on training set size and the average token size of each example for C4 should be given.  Also the authors compare against Lample 19 for the pos->pos and neg->neg setup for the Amazon data, why not show the results for the SYelp data as well?   Does the 20-40% add/delete tuning work better there as well or is it dataset dependent?

3) There are two issues with your use of the Amazon dataset.  First it doesn't really provide apples to oranges comparison against the prior papers as they train/test on the same data from Li 18 which has ~ 270 K training examples whereas the work here generates 23.6 M training examples.  It seems you should either see how those papers do with that much data or limit your dataset to be of at least comparable size to be fair.   Second, the Amazon test set is only of size 500 so assessing results on that alone seems in-suffice.

4) The paper hypothesizes that style is a "slow moving" feature consistent over large spans of text hence the use of only the prior adjacent sentence as context.  The paper shows that using just an adjacent sentence gives promising results, but doesn't show that its necessarily better than just using examplars or using a leading paragraph to derive the style from.   I don't think this is exactly necessarily to address here, but for future work it would be nice to see such a comparison.  Additionally, how would using a 1000 examplars as opposed to 100 at inference time affect performance?  A graph showing how accuracy and content preservation were affected by that would be interesting to gain better understanding.  Similarly showing how just the NBT strategy did alone (as opposed to N + NBT ) would be interesting. 

5) I didn't find the multiple aspect UMAP embedding visualization particularly convincing for how well the embeddings separate the "sentiment" aspect as there is substantial overlap within each category ( particularly software ).  I don't know if this is particularly necessary for your argument in my opinion ( especially compared with evals on other datasets), but if so then it'd be interesting to have quantitative numbers for those separations and compared with how it differs from just taking the T5 embeddings and doing the same UMAP? 

6) The "replace" noise strategy feels pretty arbitrary.  Is there any motivation behind using that as opposed to using a LM or another strategy to replace tokens?

7) A citation for using Self Bleu as opposed to Multi-Bleu in the Evaluation Procedure section would be helpful.  Additionally a citation of Ke Wang, Hang Hua, Xiaojun Wan "Controllable Unsupervised Text Attribute Transfer via Editing Entangled Latent Representation" ( Neurips 19 ) particularly for its "tunable" aspect could be an addition to the Related Work section.

8) This is nitpicky and probably for future work, but the use of examplars doesn't necessarily limit the user to a pre-defined set of styles ( like the unsupervised case does ), however it would be interesting to see what would happen given out of domain examplars for either the source or target classes at inference time

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above 
 
#########################################################################
Possible prior citation 

review_title: This paper proposes an approach to label-free style transfer where a noise augmented auto-encoder is conditioned on an input's prior sentence via a "style encoder" and user defined tuning rates with a final objective composed of reconstruction cross entropy and separate noise corruption losses.   The encoders are initialized to a pre-trained T5 weights and at inference time style transfer is performed given examplars of the source and desired target classes for a given input.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method for text-style transfer where they dont need label information of the interested style.
The extend t5 model to develop their architecture which models style extract a style vector from arbitrary text and
 use this vector to condition the decoder to perform style transfer.



However the current presentation of the paper is hard to follow. Which raises the following concern:

1. As they need to provide two sentences which has to be chronological sentences, it is not possible to obtain always. 
Hence, they randomly select sentence pair, but then two sentences may not bear same style. How the authors are incorporating the same?

2. For inference they need sentence exemplar both both style. This contradicts their previous claim.
They have not compared with

3. How noise introduction in helpful for style corrupted sentence generation? They do not use any heuristic and from a single sentence there can be multiple variation of the corrupted version, are all those taken at the time of training? Then the style it is learning is possibly not the intended one as different corrupted sentences might  need different style to reconstruct the sentence back.

4. Model portion is extremely cryptic. What is back translation etc? At least should be explained in one line.

5. Due to the unreadability of the model, I cannot provide judgement on the result section.
review_title: TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling

Meta-review: This paper proposes a new method for label-free text style transfer. The method employs the pre-trained language model T5 and makes an assumption that two adjacent sentences in a document have the same style. Experimental results show satisfying results compared with supervised methods.

Pros. • The paper is generally clearly written. • The proposed method appears to be new. • Experiments have been conducted.

Cons • The fundamental assumption of the method is not convincing enough. (Issue 1 of R3, Issue 4 of R4, Issue 1 of R2) • The proposed model is also not convincing enough. (Issues 2 and 3 of R3, Issue 3 of R2) • There are problems with the experiments. For example, it would be better to use more datasets in the experiments. (Issue 4 of R3, Issue 2 of R4)

Discussions have been made among the reviewers. The reviewers appreciate the efforts made by the authors in the rebuttal, including the additional experiments. However, they are not fully convinced and still feel that the submission is not strong enough as an ICLR paper.


Decision: Reject