SlowGuess/NAACL_2022_OCR huggingface dataset:
Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the EMNLP-IJCNLP 2019, pages 3573-3578.
Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wen Han Chao. 2020. Formality style transfer with shared latent space. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2236-2249, Barcelona, Spain (Online).
Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4873-4883.
Hu Xu, Lei Shu, S Yu Philip, and Bing Liu. 2020. Understanding pre-trained bert for aspect-based sentiment analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 244-250.
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 979-988.
Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. arXiv preprint arXiv:1903.06353.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 2020.
Yi Zhang, Tao Ge, and Xu Sun. 2020b. Parallel data augmentation for formality style transfer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3221-3228.
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation.
Chulun Zhou, Liang-Yu Chen, Jiachen Liu, Xinyan Xiao, Jinsong Su, Sheng Guo, and Hua Wu. 2020. Exploring contextual word-level style relevance for unsupervised style transfer. In Proceedings of the ACL 2020, pages 7135-7144.
Algorithm 1 Training process of the proposed semi-supervised text style transfer framework.
1: Given non-labeled datasets  $\mathcal{D}_S$  and  $\mathcal{D}_T$  in two different styles  $S$  and  $T$ , construct pseudo parallel dataset  $\mathcal{D}_{pseudo}$  with sentence pairs matched with lexical-based or semantic-based similarity
2: Pre-train a binary style classifier styleCLS on the two datasets  $\mathcal{D}_S$  and  $\mathcal{D}_T$
3: Pre-train the text style transfer model  $G_{\theta}$  using pseudo-parallel sentence pairs in dataset  $\mathcal{D}_{pseudo}$ , with MLE loss (Eq. 1).
4: for each iter  $i = 1,2,\dots,M$  do
5: Sample sentence  $\pmb{x}$  of source style  $S$  from  $\mathcal{D}_S$
6: Generate sentence  $\pmb{y}^{\prime}$  of target style  $T$  via model  $G_{\theta}$  by greedy decoding
7: Generate sentence  $\hat{\pmb{y}}$  of target style  $T$  via model  $G_{\theta}$  by sampling-based decoding
8: Reconstruction Reward Calculation (Content Preservation)
9: Given  $\pmb{y}^{\prime}$ , generate back-translated sentence  $\pmb{x}^{\prime}$  of source style  $S$  via model  $G_{\theta}$  by greedy decoding
10: Given  $\hat{\pmb{y}}$ , generate back-translated sentence  $\hat{\pmb{x}}$  of source style  $S$  via model  $\pmb{G}_{\theta}$  by greedy decoding
11: Compute reconstruction reward  $R_{cyclic}$  based on BLEU scores of the pair  $[\pmb{x},\pmb{x}^{\prime}]$  and the pair  $[\pmb{x},\hat{\pmb{x}}]$ , following Self-Critical Sequence Training (Eq. 2)
12:  $\triangleright$  Style Reward Calculation (Style Strength)
13: Compute style reward  $R_{style}$  of generated sentence  $\hat{y}$  using the style classifier styleCLS
14: Stepwise Reward Re-weighting
15: Compute the stepwise re-weighting values by max-pooling attentive scores from style classifier styleCLS on the generated sentence  $\hat{\pmb{y}}$
16: Expand  $R_{\text{style}}$  and  $R_{\text{cyclic}}$  from 1-D (sequence level) to 2-D (token level), and re-weight  $R_{\text{style}}$  with stepwise values
17: Compute the total stepwise reward  $R'$  by adding  $R_{\text{style}}$  and  $R_{\text{cyclic}}$ , based on Eq. 4
18: Update  $\theta$  using reward  $R^{\prime}$  based on Eq. 6
19: end for
Style Classifier
GPU Model
Library Version
Computational Cost
BART-Base (12-layer, 768-hidden, 16-heads, 139M parameters).
RoBERTa-base (12-layer, 768-hidden, 12-heads, 125M parameters).
Single Tesla A100 with 40 GB memory; CUDA version 11.0.
Pytorch==1.8.1; Transformers==4.8.2
Average 5 hours training time for one round. Average 3 rounds for each reported result (calculating mean of the result scores).
Hyper-parameter
Learning Rate and Batch Size
Beam Search Size
Style Reward Penalty  $\gamma$  (Eq. 4)
We set the learning rate and batch size according to regular language model fine-tuning strategy (Lewis et al., 2020).
We evaluated models on beam search sizes from 3 to 10, and 6 provided the best balance of performance and inference speed.
(1) In our experiment, we observed that the style reward  $R_{style}$  values given by the style classifier were up to 0.9 (indicating a high level of style transfer strength), while the cyclic reconstruction reward  $R_{cyclic}$  values were at a lower level (average was 0.5). Therefore, we added the  $\gamma$  to adjust the  $R_{style}$  to the same level of  $R_{cyclic}$ . (2) We evaluated values from 0.1 to 0.4 (0.1 as step), and empirically set the  $\gamma$  at 0.2. Training without the penalty  $\gamma$  did not produce significantly degraded results.
We evaluated both values with  $1.0 + / - 0.2$ , and empirically set  $\lambda_{cyclic}$  at 1.0,  $\lambda_{style}$  at 0.8. Setting at 1.0 by default did not produce degraded results.
For the comparison of using sequence level and stepwise rewards, we run experiments with the aforementioned parameter setting.
In our pilot experiment, we tried to combine both lexical and semantic pseudoparallel data, but this did not bring any improvement on the Yelp and Amazon. Presumably this is because the semi-supervised model only requires weak supervision from the pseudo-parallel data, and either the lexical and semantic data can provide sufficient information at the bootstrapping training stage.
Sequence-Level & Stepwise Reward
Combination of lexical and semantic pseudo-parallel data
Table 9: The detailed environment settings and search strategy of training parameters in our experiment. It is worth mentioned that our proposed semi-supervised approach with bootstrapping strategy and stepwise reward re-weighting is targeted to tackle the unstable learning issue of RL-based models.
Model	E&M Domain	F&R Domain
Accuracy*	BLEU	G2	H2	Accuracy*	BLEU	G2	H2
Human Reference (Rao and Tetreault, 2018)	58.7	100.0	76.6	73.9	51.4	100.0	71.6	67.8
Rule-Based (Rao and Tetreault, 2018)	11.4	72.4	28.7	19.6	52.1	65.8	58.5	58.1
Hybrid Annotations (Xu et al., 2019)	10.4	69.2	26.8	18.0	8.75	74.3	25.4	15.6
Semi-LM-MMI w/ BART (Chawla and Yang, 2020)	10.6	76.5	28.4	18.6	9.68	79.9	27.8	17.2
Rewarded BART-Large (Lai et al., 2021)	52.8	76.5	63.5	62.4	45.9	79.2	60.2	58.1
Only Labeled Data Supervision (Full)	55.2	71.2	62.6	62.1	47.3	72.5	58.5	57.2
Labeled Data + Reward-Learning (30K)	55.3	71.4	62.8	62.3	45.2	74.4	57.9	56.2
Labeled Data + Reward-Learning (Full)	58.1	71.0	64.2	63.9	50.3	74.2	61.0	59.9
Table 11: Examples of human references and generated sentences on the Yelp corpus from representative baseline models and our proposed framework. The text style is converted from negative to positive.
Model	Text
Source Sentence	it makes a buzzing sound when devices are plugged in.
Human Reference	it makes a useful buzzing sound when devices are plugged in.
Cross Aligned (Shen et al., 2017)	it s a nice, and easy to clean out.
Style Embedding (Fu et al., 2018)	it makes a bit different, while but num extend mode.
Template-Based (Li et al., 2018)	it makes a buzzing sound when devices are plugged in and use it to charge my.
Delete+Retrieve (Li et al., 2018)	it makes a buzzing sound when the devices are plugged in.
Direct Rewards w/ GPT-2 (Liu et al., 2021)	it makes a cooking faster than devices are plugged in.
Bootstrapping + Reward-Learning (Ours)	it makes a great sound when devices are plugged in.
Source Sentence	it was not as good as our much cheaper model.
Human Reference	its a great as before.
Cross Aligned (Shen et al., 2017)	it s not not worth the phone and very well.
Style Embedding (Fu et al., 2018)	it was worth it size but at least my product.
Template-Based (Li et al., 2018)	it was not as good as our much cheaper model and works just.
Delete+Retrieve (Li et al., 2018)	as using the much cheaper model as it is also much cheaper.
Direct Rewards w/ GPT-2 (Liu et al., 2021)	it was excellent as our much cheaper model.
Bootstrapping + Reward-Learning (Ours)	it was as good as our much cheaper model.
Source Sentence	i received the wrong color and it shreds easily.
Human Reference	i received the right color and it works well.
Cross Aligned (Shen et al., 2017)	i bought the phone and it s easy to.
Style Embedding (Fu et al., 2018)	i received the fact that and quickly is no clean.
Template-Based (Li et al., 2018)	i received the wrong color and it shreds easily to order more.
Delete+Retrieve (Li et al., 2018)	i received the wrong color and it looks very nice! he would highly recommend it easily.
Direct Rewards w/ GPT-2 (Liu et al., 2021)	i received the best cooking efficiently.
Bootstrapping + Reward-Learning (Ours)	i received the right color and it shreds easily.
Source Sentence	i am actually afraid to open the remaining jars.
Human Reference	I look forward to opening the remaining jars.
Cross Aligned (Shen et al., 2017)	i have to say and the other ones.
Style Embedding (Fu et al., 2018)	i am actually used the right over a container.
Template-Based (Li et al., 2018)	i am actually afraid to open the remaining jars highly recommend.
Delete+Retrieve (Li et al., 2018)	i am actually afraid to open the remaining jars this is great.
Direct Rewards w/ GPT-2 (Liu et al., 2021)	i am actually faster cooking than items.
Bootstrapping + Reward-Learning (Ours)	i am actually happy to open the remaining jars.
Table 12: Examples of human references and generated sentences on the Amazon corpus from representative baseline models and our proposed framework. The text style is converted from negative to positive.
Model	Text
Source Sentence	my dad likes action, my mom likes romance, but for me i like comedy.
Human Reference	My father likes action, my mother likes romance, but for me I prefer comedy.
Rule-Based (Rao and Tetreault, 2018)	My dad likes action, my mom likes romance, but for me I like comedy.
Hybrid Annotations (Xu et al., 2019)	My father likes action, my mother likes romance, but I like comedy.
Semi-LM-MMI w/ BART-large (Chawla and Yang, 2020)	My dad likes action, my mom likes romance, but for me I like comedy.
Rewarded BART-Large (Lai et al., 2021)	My dad likes action, my mom likes romance, but for me I like comedy.
Labeled Data + Reward-Learning (Ours)	My father likes action, my mother likes romance, but for me I prefer comedy.
Source Sentence	I want to be on TV!
Human Reference	I would like to be on television.
Rule-Based (Rao and Tetreault, 2018)	I want to be on television!
Hybrid Annotations (Xu et al., 2019)	I want to be on television.
Semi-LM-MMI w/ BART-large (Chawla and Yang, 2020)	I want to be on TV.
Rewarded BART-Large (Lai et al., 2021)	I would like to be on television.
Labeled Data + Reward-Learning (Ours)	I would like to be on television.
Source Sentence	BUT IT IS OKAY TO KISS ON THE FIRST DATE.
Human Reference	It is okay to kiss on the first date.
Rule-Based (Rao and Tetreault, 2018)	However, it is okay to kiss on the first date.
Hybrid Annotations (Xu et al., 2019)	It is okay to kiss on the first date.
Semi-LM-MMI w/ BART-large (Chawla and Yang, 2020)	It is okay to kiss on the first date.
Rewarded BART-Large (Lai et al., 2021)	However, it is acceptable to kiss on the first date.
Labeled Data + Reward-Learning (Ours)	However, it is acceptable to kiss on the first date.
Source Sentence	The same guy you wanna be in a relationship with?
Human Reference	Do you want to be in a relationship with the same man?
Rule-Based (Rao and Tetreault, 2018)	The same man with whom you would like to be in a relationship?
Hybrid Annotations (Xu et al., 2019)	The same guy you want to be in a relationship with?
Semi-LM-MMI w/ BART-large (Chawla and Yang, 2020)	The same guy you want to be in a relationship with?
Rewarded BART-Large (Lai et al., 2021)	The same man you want to be in a relationship with?
Labeled Data + Reward-Learning (Ours)	Is this the same man you want to be in a relationship with?
Table 13: Examples of human references and generated sentences on the GYAFC corpus from representative baseline models and our proposed framework. The text style is converted from informal to formal.
I. Scoring result on the Yelp corpus
Model	Style	Fluency	Content	Mean
Delete+Retrieve (Li et al., 2018)	3.25	2.72	2.86	2.94
IterativeMatch (Jin et al., 2019)	3.40	2.88	2.69	2.99
Direct Rewards w/ GPT-2 (Liu et al., 2021)	3.51	3.15	3.18	3.28
Bootstrapping + Reward-Learning (Ours)	3.49	3.29	3.25	3.34
II. Scoring result on the Amazon corpus
Model	Style	Fluency	Content	Mean
Template-Based (Li et al., 2018)	2.78	2.36	2.55	2.56
Delete+Retrieve (Li et al., 2018)	2.94	3.08	2.73	2.91
Direct Rewards w/ GPT-2 (Liu et al., 2021)	3.20	3.23	2.21	2.88
Bootstrapping + Reward-Learning (Ours)	3.31	3.28	3.12	3.23
III. Scoring result on the GYAFC corpus
Model	Style	Fluency	Content	Mean
Hybrid Annotations (Xu et al., 2019)	2.56	3.15	3.13	2.95
Semi-LM-MMI w/ BART (Chawla and Yang, 2020)	3.12	3.47	3.22	3.27
Rewarded BART-Large (Lai et al., 2021)	3.36	3.60	3.33	3.43
Labeled Data + Reward-Learning (Ours)	3.37	3.67	3.37	3.47
Table 14: Human evaluation are conducted on the Yelp, Amazon, and GYAFC style transfer datasets. Following previous work (Chawla and Yang, 2020; Liu et al., 2021), we evaluated the generated sentences from three aspects: style transfer strength (Style), text fluency (Fluency), and content preservation (Content), separately. The three aspects are rated with range [1, 5], then their average value is calculated and reported as Mean. For each corpus, we randomly selected 80 test samples and compared the outputs of representative and previous state-of-the-art models. Each candidate was rated by three linguistic experts, and we report the average scores. Our model achieves better overall performance when considering all three evaluation metrics on each dataset. Moreover, we observe that leveraging the pre-trained language models such as BART and GPT-2 is beneficial for the text fluency.
Hello, this is a text style transfer human evaluation rating form!
Thanks for your participation!
This form is to conduct the text style transfer rating.
For each item, please rate the candidate sentences from 1 (worst) to 5 (best).
All the data are only collected for research use.
Source Sentence (Negative-to-Positive Sentiment Transfer): "ever since joes has changed hands it's just gotten worse and worse."
Candidate 1# just since their sausages has changed it's just gotten worse
Please read this rating criteria of the three aspects (Style, Fluency, Content)
before your assessment:
#Content#
Annotators judge if the source and candidate sentence convey the same information on a scale of 1-5:
5: Completely equivalent.
4:Mostly equivalent, 10
3: Roughly equivalent; 4: Not available.
2: Not equivalent but share some details, 4: Not equivalent and no topic under
Fluency#
Score the clarity and ease of understanding of the translated sentence on a scale from 1-5:
5: Perfect,
4: Comprehensible,
3: Somewhat comprehensible, but similar to source sentence.
2: incomprehensible, 10
Style#
Rate the style strength of the translated sentence on a scale of 1 to 5:
5: Completely being classified in the target style,
4: Mostly being classified in the target style.
3: Partially in the target style,
2: Not in the target style, but neutral.
1: In the opposite style.
#Candidate 2# ever since joes has changed hands it 's just gotten better and better *
Style
Fluency
Content
Candidate 3# ever since joes has changed hands it's just so good!
Figure 3: Rating interface for the human evaluation. Text candidates are shuffled for each sample.
Model	Accuracy	Yelp Data	Amazon Data
BLEU	G2	H2	Accuracy	BLEU	G2	H2
Lexical Pseudo + Reward-Learning (30K)	81.1	50.4	63.9	62.1	71.2	36.1	50.6	47.9
Pure Reward Learning (30K)	70.8	41.3	54.0	52.1	61.2	26.1	39.9	36.5
Lexical Pseudo + Reward-Learning (100K)	86.2	59.4	71.5	70.2	73.1	46.3	58.1	56.6
Pure Reward Learning (100K)	75.5	46.1	58.9	57.2	65.6	26.5	41.6	37.7
Table 15: Ablation study of the proposed bootstrapping on the Yelp and Amazon datasets. Models are running in a RL-based unsupervised manner, and we used the same data sizes as the experiments in Table 3 and Table 4.
Figure 4: Attention heatmap examples of the attention scores with layer-level max-pooling. The 'ROBERTa-base' model is fine-tuned on the Yelp data for style classification. The higher scores denotes higher attention weights on the tokens, and the top layers (especially the 11-th layer) shows better attribute-specified correlation. At the token level, the attention values and the max-pooled step-wise values described in Section 3.4 are all in the range of  $(0,1)$ .