SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 19: Results on compositions of transfers using sequentially applying GPT2 (SEQGPT), CS-GPT-ZERO (adding compositional model but not compositional data) and CS-GPT (with both compositional model and data). The result shows that CS-GPT significantly outperforms the other two methods, and zero-shot remains challenging as CS-GPT-ZERO does not perform very well in comparison.
Transfer	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE_L	CiDER
To Future Tense	GPT2	0.895	0.851	0.812	0.777	0.539	0.898	7.708
CS-GPT (TV)	0.727	0.614	0.614	0.450	0.362	0.731	4.386
CS-GPT (TP)	0.810	0.731	0.663	0.606	0.446	0.818	6.026
To Past Tense	GPT2	0.835	0.776	0.721	0.673	0.484	0.842	6.699
CS-GPT (TV)	0.694	0.586	0.494	0.420	0.353	0.700	4.051
CS-GPT (TP)	0.834	0.771	0.718	0.672	0.486	0.841	6.704
To Present Tense	GPT2	0.753	0.662	0.586	0.523	0.412	0.772	5.293
CS-GPT (TV)	0.733	0.635	0.553	0.488	0.387	0.744	4.742
CS-GPT (TP)	0.826	0.755	0.691	0.637	0.491	0.831	6.315
ActiveToPassive	GPT2	0.475	0.329	0.238	0.189	0.216	0.463	1.820
CS-GPT (TV)	0.472	0.324	0.232	0.179	0.216	0.454	1.790
PassiveToActive	GPT2	0.433	0.271	0.167	0.120	0.191	0.434	1.329
CS-GPT (TV)	0.506	0.345	0.243	0.184	0.229	0.505	1.958
PP Removal	GPT2	0.763	0.700	0.645	0.593	0.419	0.786	6.011
CS-GPT (TP)	0.760	0.698	0.639	0.585	0.420	0.772	5.783
Table 20: Comparing single transfer performances between CS-GPT and GPT2 baselines (where TV indicates the CS-GPT is trained on Tense+Voice dataset and TP indicates the CS-GPT is trained on Tense+PP Removal dataset). The result shows that CS-GPT can perform multiple single style transfers with similar performance to GPT2 trained specifically for that one transfer, and sometimes even outperforms GPT2.
Male context	Female context	Total
Biased	21	28	49
Not Biased	29	22	51
Total	50	50	100
Category	Number
Significantly more biased	0
Slightly more biased	0
Little or no change in bias	22
Slightly less biased	5
Significantly less biased	22
Total	49
Table 21: Top table: Human annotators found that 21 out of 50 sentences generated by GPT2 on "The man worked as" and 28 out of 50 sentences generated by GPT2 on "The woman worked as" to exhibit gender bias. Bottom table: Out of the 49 biased sentences, after using style transfer to replace occupations with randomly sampled ones, human annotators found that 22 of them are significantly less biased while the rest are either slightly less biased or neutral.
Text Generated by GPT2	Text after occupation replacement
The man worked as a security guard	The man worked as a receptionist
for the family’s hotel until the weekend.	for the family’s hotel until the weekend.
The man worked as a driver for his father.	The man worked as a personal care aide for his father.
The woman worked as a maid at a resort in a small town.	The woman worked as a driver at a resort in a small town.
The woman worked as a nurse at a facility.	The woman worked as a construction worker at a facility.
Table 22: Examples of sentences generated by GPT2 and with occupation replacements that are rated as "Significantly Less Biased" after the change by human annotators.
D Mitigating Social Biases: Qualitative Evaluation
We created two prompts "The man worked as" and "The woman worked as", and generated 50 sentences with each prompt from GPT2. Next, we determine biased words by taking the 1,000 closest vectors in GloVe word embeddings (Pennington et al., 2014) to "man" and "woman". Then, we determine a sentence as biased if the phrase describing the occupation in the sentences contains any biased words. With this standard, we found that 21 out of 50 sentences for man and 28 out of 50 sentences are biased. Then, we replaced the occupations in these 49 biased sentences with occupations sampled uniformly randomly from all 100 generated sentences, and then asked two independent human annotators to evaluate the 49 replaced sentences on a five-point scale of Significantly More Biased, Slightly More Biased, The Same, Slightly Less Biased, and Significantly Less Biased. On average, the annotators reported 22 sentences being significantly less biased compared to before the replacements, while all other sentences are either slightly less biased or neutral. The full results of this experiment are shown in Table 21. A few examples that were deemed Significantly Less Biased by both annotators are shown in Table 22.