SlowGuess/NAACL_2022_OCR huggingface dataset:
where  $\oplus$  denotes column-wise concatenation,  $W_{4}\in \mathbb{R}^{2d\times d}$  and  $b_{4}\in \mathbb{R}^{d}$ .  $H^{\prime}\in \mathbb{R}^{n\times d}$  is the new hidden state, which fuses the old hidden state and the new knowledge aware hidden state.  $H^{\prime}$  is passed into the BART decoder.
A.3 Implicit Knowledge
A.3.1 Incorporating Implicit Knowledge using GPT
In this section, we provide formal details on incorporating implicit knowledge using a pretrained GPT models.
Let  $M_{\mathrm{tm}}$  be a BART model trained to predict the target minority of a given input post. Let the target minority string predicted by  $M_{[\mathrm{tm}]}$  be " $s_{[\mathrm{tm}]}$ " and let " $s_{[\mathrm{pr}]}$ " be a prompt. We provide a list of prompts in Table 8. We then prompt the GPT models with the following string "The  $s_{[\mathrm{tm}]}$ $s_{[\mathrm{pr}]}$ ". The GPT models complete the prompt with a generated string, " $s_{[\mathrm{gpt}]}$ ", so that the final string is "The  $s_{[\mathrm{tm}]}$ $s_{[\mathrm{pr}]}$ $s_{[\mathrm{gpt}]}$ ". For GPT generations, we choose hyper-parameters based on methods in Patrick von Platen (2020). For each input, we randomly select various prompts to generate  $k$  sentences of the form "The  $s_{[\mathrm{tm}]}$ $s_{[\mathrm{pr}]}$ $s_{[\mathrm{gpt}]}$ ", varying  $k \in \{3,15\}$ . Note that if there is no predicted
Prompt
were known for
were described as
were regarded as
were thought of as
were well-known for
were seen as a
have a reputation for
would frequently
worked as
have a job as
have a part-time job as
earned money by
started working as
have various hobbies such as
would regularly engage in
frequently talked about
behaved as though
liked to
target minority,  $s_{[\mathrm{tm}]}$  is the empty string and no sentence is generated using GPT. In this case, the input is paired with just the empty string.
We then pair the input with each of the  $k$  generated sentences (or the empty string if there is no predicted target minority) and train a BART model  $M$  to predict the generated sentences. Model  $M$  is then used as a pretrained model and is retrained to predict the implied stereotype given the same input. The retrained BART model is our final implied stereotype generator.
B.1 Implementation Details for BART Encoder Decoder Models
We train our BART models using a learning rate of  $5e - 6$  for 3 epochs with a batch size of 2 or 4, depending on the size of the input. This excludes the BERT classifier models, whose settings are given in Appendix B. The BART models have 406M parameters and all training is done on an Nvidia Titan V GPU, with 12 GB memory, a boost clock speed of  $1455\mathrm{MHz}$ , 640 Tensor cores
and 5120 CUDA cores. Training under this regime takes approximately 90 - 120 minutes. We remove all URLs, "RT" strings, and "@" mentions from the input post. We train these models on just a single seed and results are reported on just that seed, as we had limited time to train and test our models. The baseline GPT-2 and GPT models are trained for 5 epochs, as in the original paper by Sap et al. (2020). Following the paper, we perform minimal preprocessing to the input text before training and testing and only remove all URLs. During inference, we pass batches of input from the dev and test sets to the generate method of the huggingface BART model class. We use beam search for generation, with a beam width of 10 and a length penalty of 5.0.
B.2 Implementation Details for BERT Classifier Models
We train base BERT models, which have 110M parameters. Training takes approximately 30 - 40 minutes on the GPUs described in B. We train these models on just a single seed, as results did not vary much as the seed varied and we had limited time to train and test our models.
Table 8: Prompts for the GPT/GPT-2 biased sentence generators. The biased sentences are used to train the BART model, which is then retrained to produce the output stereotype. The predicted target minority of the input post is prepended to some of the prompts above, chosen at random. The GPT models then complete the prompt.
Model	LR	Batch Size	Epochs
Offensiveness	5e-6	32	2
Intent to Offend	5e-7	32	1
Lewdness	5e-6	32	1
Group Targeted	5e-6	32	2
Table 9: Training Settings for BERT Classifier Models. LR stands for Learning Rate.
BERT Classifier Model training settings are given in Table 9 and Table 10.
C Further Ablation Studies
In this section, we look at ablation studies conducted on the Expert Knowledge models and the Explicit Knowledge models.
C.1 Expert Knowledge Ablation Study
Recall that we use the last attention layer of a BERT classifier with the join embedding architecture introduced in Pryzant et al. (2020) to enhance the hidden states of the BART model performing stereotype generation. We perform ablation studies by replacing the classifier over a few different annotated categories, namely Offensiveness, Intent to
Model	dev	test
Offensive	Intent	Lewd	Group	Offensive	Intent	Lewd	Group
GPT	0.834	0.818	0.608	0.740	0.835	0.818	0.577	0.754
GPT-2	0.832	0.812	0.654	0.754	0.847	0.824	0.670	0.774
BERT	0.863	0.832	0.726	0.810	0.867	0.828	0.708	0.826
Table 10: GPT/GPT-2/BERT Classifier Performance on Dev and Test sets.
Models	dev BERTScore	test BERTScore
GPT	0.712	0.713
GPT-2	0.733	0.741
BART Base	0.624	0.638
EXPERT (OFFENSIVE)	0.759	0.767
EXPERT (INTENT)	0.761	0.764
EXPERT (LEWD)	0.757	0.764
EXPERT (GROUP)	0.765	0.776
EXPERT (ALL)	0.765	0.770
Offend, Lewdness, and Group Targeted. We also train a model that uses all of the classifiers. The results for the ablations are in Table 11.
It is important to note the relative performance of the models. The Expert Knowledge model using the Group Targeted BERT classifiers performs better than the other single classifier models, and performs on par with the Expert Knowledge model leveraging all of the classifiers. It's likely that the tokens used by the BERT model to identify whether a minority group is targeted aligns closely with the portions of the encoded input used to generate the stereotypes. This makes intuitive sense, since the set of posts targeting some minority group likely has some stereotype mentioned in the post and vice versa. The same cannot be said for the other categories. This intuition is further strengthened by the fact that the Expert Knowledge model using all the classifiers does not perform much better than the Expert Knowledge model using just the Group Targeted classifier. Thus the other classifiers may be contributing little additional knowledge to the stereotype generation task.
C.2 Explicit Knowledge Ablation Study
In this section, we specifically discuss the number of knowledge triples we use when training ex
Table 11: We report BERTScores of baseline models (first three rows) and the expert knowledge models. The Expert Knowledge Models leverage annotations of categorical variables on the input to enhance stereotype generation. The All variant leverages all of the categorical variables.
Model	dev	test
GPT	0.712	0.713
GPT-2	0.733	0.741
BART Base	0.624	0.638
EXPLICIT (INPUT) (3)	0.761	0.768
EXPLICIT (INPUT) (5)	0.748	0.757
EXPLICIT (INPUT) (10)	0.764	0.770
EXPLICIT (INPUT) (15)	0.769	0.772
EXPLICIT (INPUT) (20)	0.770	0.773
EXPLICIT (INPUT) (25)	0.752	0.763
EXPLICIT (ATTN) (5)	0.761	0.768
EXPLICIT (ATTN) (10)	0.759	0.765
EXPLICIT (ATTN) (20)	0.768	0.769
Table 12: We report BERTScores of baseline models (first three rows) and the explicit knowledge models.
plicit knowledge models and note trends in the BERTScore as  $k$  varies. The results are in Table 12. We discuss an additional attention based model not mentioned in the main paper. The detailed methodology for this model is given in Appendix A.2.2.
We note that when concatenated directly to the input, performance increases as  $k$  increases up to a point and then starts to decrease. We believe that this occurs because the usefulness of knowledge initially increases then decreases as  $k$  increases. In particular, as  $k$  increases, many of the latter triples tend to be synonymous with earlier triples or unrelated to the original input. We also note that incorporating knowledge as attention tends to produce a better BERTScore, while not performing as well on other metrics. We suspect that this is due to the numerbatch embeddings capturing more semantic meaning than the BART embeddings. The numberbatch embeddings draw from a variety of sources, in addition to the graph structure itself and they perform well on benchmarks measuring word similarity.
We provide a few examples of stereotype generation, comparing and contrasting the different model types in Table 13.
Error/Challenge Type	Input Post	Generated Stereotype	Explanation
Error Type 1 (Non-Existent Stereotype)	what do you call it when you fuck an apple? an inside job	makes light of child abuse	The post does not refer or imply anything about children in any way.
Error Type 2 ( Ignores Stereotype)	why do bitches hate so much tho? what's wrong with congratulating and giving people props sometimes?	“”	The post clearly refers to women in a derogatory manner.
Error Type 3 ( Incorrect Target Minority)	if harvey and irma are god's fidget spinners, wouldn't god be autistic?	trivializes harm to victims	The post refers to autistic folks, but the generate stereotype doesn't capture it.
Error Type 4 ( Incorrect Stereotype)	why don't black people go on cruises? because they already fell for that shit once.	black people deserve to die	The post refers to African American history (those arriving on slave ships), yet the generated stereotype does not mention this.
Challenge Type 1 ( Misunderstands Post)	if the operation to remove your tonsils is called a tonsillectomy, then what is the name of the operation for a female do male sex change? addadectomy	women are only good for sex	The generate stereotype has nothing to do with the original post.
Challenge Type 2 ( High Sensitivity)	why do jews have big noses? to smell the money.	jews were killed in the holocaust	The token “jew” triggers an irrelevant generation.
Challenge Type 3 ( Localized Generation)	how do you know the asian food is under-cooked? when it starts barking.	asians have no food	The post focuses on the first part of the sentence, but ignores the remainder.
Challenge Type 4 ( Does not Draw Con-nections)	enjoy having multiculturalism shoved down your throat, fighting for israel and having a black or asian as your new best friend.	non-whites are inferior	The generated stereotype understands that the stereotype is about non-whites, but does not connect it to the broader stereotype of non-white cultures having a negative impact.
Challenge Type 5 ( Misunderstands Sar-casm/Irony)	dating rules for women * don't be morbidly obese *	women shouldn't be skinny	The generated stereotype inverts the sarcasm of the original post and thus fails to capture the stereotype.
Challenge Type 6 ( Ignores Stereotype)	fight anyone that tells you there's nothing wrong with being fat. and help your body and bones and lose weight, seriously.	“”	The input post targets those who are obese, yet the model does not generate a stereotype.
Table 13: Examples of generation from each of the error types described in Sections 5.1 and 5.2
D.1 Error Analysis
Table 14 provides the counts for the Error Analysis. Clearly, the MIXGEN models minimize the total number of errors made, although MIXGEN may not have the minimal number of errors in each category.
D.2 Challenges in Stereotype Generation
Table 15 provides the counts for the Challenges faced by each model. Clearly, the MIXGEN models minimize the total number of challenges face, although MIXGEN may not have faced the minimal number of challenges within each category.
Error Type	GPT	GPT-2	Expert	Explicit	Implicit	MixGEN
1	45	46	37	44	30	24
2	8	7	1	0	3	4
3	13	11	11	10	10	8
4	28	37	41	24	27	27
5	106	99	110	122	130	137
Table 14: This table contains the counts for the error analysis. Though it is not quite an error and isn't mentioned in the main paper, Error Type 5 accounts for the examples in which the model outputs an accurate stereotype.
Challenge Type	GPT	GPT-2	Expert	Explicit	Implicit	MixGEN
1	8	6	6	4	2	4
2 (Non-existent Stereotypes)	44	46	37	44	30	24
2 (Remainder)	15	9	33	13	18	11
3	8	19	10	7	8	8
4	7	9	2	5	6	5
5	4	5	1	5	3	7
6	8	7	1	0	3	4
7	106	99	110	122	130	137
Table 15: This table contains the counts for the error analysis. Though it is not quite a challenge and isn't mentioned in the main paper, Challenge Type 7 accounts for accurate generations by the model. Challenge Type 2 is partitioned into two sub-categories, one which includes only cases where the model detects non-existent stereotypes and another which does include such cases.