Table 10: Automatic evaluation scores on the GYAFC formality style transfer task of baseline models and our framework. Baseline results are reported with the model generations provided in published studies (Chawla and Yang, 2020). * The style accuracy is calculated with a TextCNN classifier.
Model	Text
Source Sentence	ever since joes has changed hands it ’s just gotten worse and worse .
Human Reference	ever since joes has changed hands it ’s gotten better and better .
Cross Aligned (Shen et al., 2017)	i recommend that has out to it ’s always great and fun .
Delete+Retrieve (Li et al., 2018)	ever since joes has changed hands it ’s just so good !
DualRL (Luo et al., 2019)	ever since dedicated has changed hands it ’s just gotten better and better .
IterativeMatch (Jin et al., 2019)	dominos has gotten better and better .
Deep Latent w/ LMs (He et al., 2019)	just since their sausages has changed it ’s just gotten worse and worse .
Direct Rewards w/ GPT-2 (Liu et al., 2021)	ever since joes has changed hands it ’s just gotten better and better .
Bootstrapping + Reward-Learning (Ours)	ever since joes has changed hands it ’s just gotten better and better .
Source Sentence	no , i ’m not at a scottsdale club .
Human Reference	this was a great club .
Cross Aligned (Shen et al., 2017)	great , i ’m so at a local business .
Delete+Retrieve (Li et al., 2018)	this is a great place to get a scottsdale club .
DualRL (Luo et al., 2019)	great job .
IterativeMatch (Jin et al., 2019)	i ’m so glad i found this place .
Deep Latent w/ LMs (He et al., 2019)	great food , great service at a scottsdale club .
Direct Rewards w/ GPT-2 (Liu et al., 2021)	great , nice and a scottsdale club .
Bootstrapping + Reward-Learning (Ours)	great , i ’m at a scottsdale club .
Source Sentence	french toast plate was good , mom said , but eggs were cold .
Human Reference	french toast plate was good , mom said , eggs were hot .
Cross Aligned (Shen et al., 2017)	their food tasted was good , juicy , and fries are very clean .
Delete+Retrieve (Li et al., 2018)	french toast plate was good , mom said , but eggs were amazing !
DualRL (Luo et al., 2019)	french toast plate was good , mom said , but eggs were delicious .
IterativeMatch (Jin et al., 2019)	the food was delicious and the eggs were fresh .
Deep Latent w/ LMs (He et al., 2019)	wow !
Direct Rewards w/ GPT-2 (Liu et al., 2021)	french toast plate was good , mom said , with amazing eggs are warm .
Bootstrapping + Reward-Learning (Ours)	french toast plate was good , mom said , but eggs were amazing .
Source Sentence	however , it turned out to be nothing like i thought it would .
Human Reference	this turned out exactly how i thought it would .
Cross Aligned (Shen et al., 2017)	however , it right out to be great , it is the place .
Delete+Retrieve (Li et al., 2018)	it turned out to be nothing like i thought it was so good !
DualRL (Luo et al., 2019)	however , it turned out to be nothing extraordinary it would thought it would
IterativeMatch (Jin et al., 2019)	it turned out i worried about nothing .
Deep Latent w/ LMs (He et al., 2019)	loved it !
Direct Rewards w/ GPT-2 (Liu et al., 2021)	although , it turned out to be great with i thought it will .
Bootstrapping + Reward-Learning (Ours)	however , it turned out to be great like i thought it would .