On the Unintended Social Bias of Training Language Generation Models with News Articles
Authors: Omar U. Florez
Venue:  (2020)
Keywords: Fair AI
latent representations
sequence to sequence

Abstract: There are concerns that neural language models may preserve some of the stereotypes of the underlying societies that generate the large corpora needed to train these models. For example, gender bias is a significant problem when generating text, and its unintended memorization could impact the user experience of many applications (e.g., the smart-compose feature in Gmail).

In this paper, we introduce a novel architecture that decouples the representation learning of a neural model from its memory management role. This architecture allows us to update a memory module with an equal ratio across gender types addressing biased correlations directly in the latent space. We experimentally show that our approach can mitigate the gender bias amplification in the automatic generation of articles news while providing similar perplexity values when extending the Sequence2Sequence architecture.

Under review as a conference paper at ICLR 2020
ON THE UNINTENDED SOCIAL BIAS OF TRAINING LANGUAGE GENERATION MODELS WITH NEWS ARTICLES
Anonymous authors Paper under double-blind review
ABSTRACT
There are concerns that neural language models may preserve some of the stereo- types of the underlying societies that generate the large corpora needed to train these models. For example, gender bias is a sigmificant problem when generating text, and its unintended memorization could impact the user experience of many applications (e.g., the smart-compose feature in Gmail).
In this paper, we introduce a novel architecture that decouples the representation learning of neural model from its memory management role. This architecture allows us to update memory module with an equal ratio across gender types addressing biased correlations directly the latent space. We experimentally show that our approach can mitigate the gender bias amplification in the automatic gen- eration of articles news while providing similar perplexity values when extending the Sequence2Sequence architecture.
INTRODUCTION
Neural Networks have proven to be useful for automating tasks such question answering, system response, and language generation considering large textual datasets. In learning systems, bias can be defined the negative consequences derived by the implicit association of patterns that occur in a high-dimensional space. dialogue systems, these patterns represent associations between word embeddings that can be measured by a Cosine distance observe male- and female-related analogies that resemble the gender stereotypes of the real world. We propose an automatic technique to mitigate bias in language generation models based on the use of an external memory in which word embeddings are associated to gender information, and they can be sparsely updated based on content- based lookup.
The main contributions of our work are the following:
We introduce novel architecture that considers the notion of Fair Region to update subset of the trainable parameters a Memory Network.
We experimentally show that this architecture leads to mitigate gender bias amplification in the automatic generation of text when extending the Sequence2Sequence model.
MEMORY NETWORKS AND FAIR REGION
As illustrated in Figure I the memory M consists of arrays K and V that store addressable keys (latent representations of the input) and values (class labels), respectively in Kaiser al. (2017). To support our technique, we extend this definition with an array G that stores the gender associated each word, e.g., actor is male, actress female, and scientist is no-gender. The final form of the memory module is as follows:
M = (K, V,
neural encoder with trainable parameters receives an observation x and generates activations h in hidden layer. We want normalized (i.e., |b||| in the long-term memory module M

---

Under review as a conference paper at ICLR 2020
Key
Value Gender
0J(0)
Softmax
ōkj
kj
y1
1
0J(0)
hfair
K6
1
0kg
LSTM
Attention
0J(0)
[(x, yr. g1), ... (xD Vu g1)l
0k
1
y11
2
0J(0)
k15
V15
2
Figure 1: A Fair Region in memory M consists of the most similar keys to h given a uniform distribution over genders (e.g.. 1: male, and 2: female). The input consists of sequence tokens annotated with gender information, c.g., (The, O), (president, 0), (gave, 0), (her, 2), (speech, 0).
to increase the capacity of the encode. Hence, let iman be the index the most similar key
imar argmaar; (h. K K[]]]
then writing the triplet (ar, v. to M consist of:
K [maa = |/ + K Kiimaz]l
Vmmar = y
G[imaz] 9.
However, the number of word embeddings does not provide an equal representation across gender types because context-sensitive embeddings are severely biased in natural language, Zhao et al. (2017). For example, it has been shown in that man is closer programmer than woman, [Bolubbasi al. (2016). Similar problems have been recently observed in popular work embedding algorithms such as Word2Vec. Glove, and BERT, Kurita (2019).
We propose the update of memory network within Fair Region in which we can control the number of keys associated to each particular gender. We define this region as follows.
Definition 2.1. (Fair Region) Let h be an latent representation of the input and M be external memory. The male-neighborhood of h is represented by the indices of the n-nearest keys to K KNN NNN,,, n, male).
in decreasing order that share the same gender type male { ""T...., 2 Repeating the same process for each gender type estimates the indices if and (n resulting in the female and non-gender neighborhoods. Then, the FairRegion of M given consists of K K[TT"; ; éf; j"9].
The Fair Region of a memory network consists of subset of the memory keys which are responsible for computing error signals and generating gradients that will flow through the entire architecture with backpropagation. We do not want to attend over all the memory entries but explicitly induce uniform gender distribution within this region. The result is training process in which gender-related embeddings equally contribute in number to the update of the entire architecture. This embedding- level constraint prevents the unconstrained learning of correlations between a latent vector and similar memory entries in M directly in the latent space considering explicit gender indicators.
LANGUAGE MODEL GENERATION
Our goal is to leverage the addressable keys of memory augmented neural network and the notion of fair regions discussed Sectior guide the automatic generation of text. Given an encoder-decoder architecture Sutskever, Vinyals, and Le (2014); [Bahdanau, Cho, and Bengio (2015), the inputs two sentences z and y from the source and target domain, respectively. An LSTM encoder outputs the context-sensitive hidden representation enco based on the history of sentences and an LSTM decoder
2

---

Under review as a conference paper at ICLR 2020
Softmax
h fair
Key K
C0
Co
hg
h amco
h doco
Kfai
LSTM
LSTM
Attention
(xp 8 source
(yr g) target
Figure 2: Architecture of the neural dialogue model that incorporates the memory approach discussed in Section The figure shows the jth decoding step of the word y4 given the sparse update within Fair Region centered at h deco.
receives both aeco and y and predicts the sequence of words every timestep of decoding, the decoder predicts the ith token of the output ú by computing its corresponding hidden state hdeco applying the recurrence
Ng LST (M- h;TT)
Instead of using the decoder output Adeco directly predict the next word prediction over the vocabulary O, as in Miller et al. (2016). We combine this vector with a query to the memory module to compute the embedding vector h air We do this by computing an attention score Bahdanau, Cho. and Bengio (013) with each key of Fair Region. The attention logits become the unormalized probabilities of including their associated values for predicting the 1t token of the response ý. We then argmax the most likely entry in the output vocabulary O obtain the 7th predicted token of the response More formally,
fair Fair Region deco K,,))
Softmar h
decoo Kfair)
h feir W Wtanhlaa K
Softmar (h: foir)
ú Olargmaz; (ūDI))
Naturally, the objective function is to minimize the cross entropy of actual and generated content:
J(O) Σ of log p(ù))
where N is the number of training documents, m indicates the number of words in the generated output, and y is the one-hot representation of the 5th word in the target sequence.
4 BIAS AMPLIFICATION
Inspired by Zhao et al. 2017). we compute the bias score of a word considering its word embedding h (x) and gender indicators (words man and woman). For example, the bias score of scientist is:
h fair (sciientts))) | farr [man))))
(sccientist
[afa (scieentt)))) h (man) + hf a (scientist) hfarr ([frr(woman))
If the bias score during testing is greater than the during training.
pest (scieentiss man) - ptrain ftrain (scientist, man) 0,
then the bias of man towards scientist has been amplified by the model while learning such represen- tation, given training and testing datasets similarly distributed.
For Seq2Seq neural models, this word embedding is the output the decoder component h deem (x)

---

Under review as a conference paper at ICLR 2020
PERPLEXITY
BIAS AMPLIFICATION
MODEL
ALL
PERU
MEXICO
ALL
PERU-MEXICO
MEXICO-PERU
SEQ2SEQ
13.27
15.31
15.61
+0.18
+0.25
+0.21
SEQ2SEQ+ATTENTION
10.73
13.25
14.08
+0.25
+0.32
+0.29
SEQSEQQFAIRREGION
10.79
13.04
13.91
+0.09
+0.17
+0.15
Table 1: Perplexity and Bias Amplification results on the datasets of crawled newspapers.
EXPERIMENTS
5.1 DATASET
We evaluate our proposed method in datasets crawled from the websites three newspapers from Chile, Peru, and Mexico.
To enable fair comparison, we limit the number articles for each dataset to 20,000 and the size of the vocabulary the 18,000 most common words. Datasets are split into 60%, 20%, and 20% for training. validation, and testing. We want to see if there are correlations showing stereotypes across different nations. Does the biased correlations learned by encoder transfer the decoder considering word sequences from different countries?
5.2 BASELINES
compare our approach Fair Region, with the following baseline models:
an encoder-decoder architecture augmented with
Seq2Seq Sutskever, Vinyals, and Le 2014h: encoder-decoder architecture that maps between sequences with minimal assumptions the sequence structure and that is able to remember long term dependencies by mapping the source sentence into a fixed-length vector.
2. Deq2Seq+Attentee Bahdanau, Cho, and Bengio (2015): Similar Seq2Seq, this archi- tecture automatically attends to parts of the input that can be relevant to predict the target word.
5.3 TRAINING SETTINGS
For all the experiments, the size of the word embeddings is 256. The encoders and decoders bidirectional LSTMs of 2-layers with state size of 256 for each direction. For the Seq2Seq model, the number of memory entries is 1,000. We train all models with Adam optimizer Kingma and Ba 2014) with a learning rate 0.001 and initialized all weights from a uniform distribution in 0.01, 0.0TJ. We also applied dropout Srivastava al. 2014) with keep probability of 95.0% for the inputs and outputs of recurrent neural networks.
5.4 FAIR REGION RESULTS SIMILAR PERPLEXITY
We evaluate all the models with test perplexity, which the exponential of the loss. We report in Table the average perplexity of the aggregated dataset from Peru, Mexico, and Chile, and also from specific countries.
Our main finding is that approach (Seq2Seq+FairRegiin shows similar perplexity values (10.79) than the Seq2Seq+Attente baseline model (10.73) when generating word sequences despite using the Fair Region strategy. These results encourage the use of controlled region automatic technique that maintains the efficacy of generating text. We observe a larger perplexity for country- based datasets, likely because of their smaller training datasets.
4

---

Under review as a conference paper at ICLR 2020
5.5 FAIR REGION CONTROLS BIAS AMPLIFICATION
We compute the bias amplification metric for all models, as defined in Section to study the effect of amplifying potential bias in text for different language generation models.
Table shows that using Fair Regions is the most effective method to mitigate bias amplification when combining all the datasets (+0.09). Instead, both Seq2Seq (+0.18) and Seq2Seq+Attention (+0.25) amplify gender bias for the same corpus. Interestingly, feeding the encoders with news articles from different countries decreases the advantage of using a Fair Region and also amplifies more bias across all the models. In fact, training the encoder with news from Peru has, in general, larger bias amplification than training it with news from Mexico. This could have many implications and be product of the writing style or transferred social bias across different countries. We take its world-wide study as future work.
CONCLUSIONS
Gender bias is an important problem when generating text. Not only smart composer or auto-complete solutions be impacted by the encoder-decoder architecture, but the unintended harm made by these algorithms could impact the user experience in many applications. We also show the notion of bias amplification applied to this dataset and results on how bias can be transferred between country-specific datasets in the encoder-decoder architecture.
REFERENCES
Bahdanau, D.; Cho, K.; and Bengio, 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Bolukbasi, T.; Chang, K.-W.; Zou, J.; Saligrama, V., and Kalai, A. 2016. Man is to computer programmer as woman is homemaker? debiasing word embeddings. NIPS' 16, 4356 4364.
Kaiser, L.; Nachum, O.; Roy, A.; and Bengio, S. 2017. Learning to remember rare events.
Kingma, D. P., and Ba, J. 2014. Adam: A method for stochastic optimization. CoRR.
Kurita, K. 2019, Acl web. 166-172.
Miller, A.; Fisch, A.; Dodge, J.; Karimi, A.-H.; Bordes, A.; and Weston, J. 2016. Key-value memory networks for directly reading documents. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 1400-1409. Austin, Texas: Association for Computational Linguistics.
Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, L; and Salakhutdinov, R. 2014. Dropout: A simple way prevent neural networks from overfitting. Mach. Learn. Res. 15(1):1929-1955
Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence sequence learning with neural networks. In Ghahramani, Z.: Welling, M.; Cortes, C.; Lawrence, N. D.; and Weinberger, K. Q., eds., Advances in Neural Information Processing Systems 27. Curran Associates, Inc. 3104-3112.
Zhao, J.; Wang, T.: Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
5


Reviews:
[{"review_id": "ryl7v3dJcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of debiasing word representation in a trained language model using some sort of key-value memory structure with annotations of gender that is constrained to view only a subset of keys. \n\nThe current form of the manuscript contains too few details to evaluate the work in a meaningful way. Please include \n\n(1) a more detailed description of data input and output for system (i.e. where are gendered bits coming from), and more details of the memory (is it a global fixed memory? then how gendered bits assigned?)\n\n(2) a more complete definition of bias amplification -- beyond just an example (i.e. what training representation are you comparing to?). A formula would be useful here.\n\n(3) more complete experimental definition. What datasets exactly are you evaluating on?\n\n(4) overview sections to motivate your solutions and your setup (i.e. an introduction, related work)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJelztl6FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThe authors propose using attention with Fair Region in Memory Network for reducing gender bias in language generation. \n\nDecision:\nOverall, the paper is poorly written and lack of enough substance, even though the idea seems reasonable. I am inclined to reject this paper. \n\nSupporting argument:\n1. The authors use quite small dataset (also non-standard) for evaluation. It might be better to evaluate on some standard large-scale language model datasets. \n2. As mentioned in  Section 5.5, the proposed approach is not generalized to other countries. \n3. Since the task is language generation, the use of bias-amplification in embedding space is not well justified. \n4. What are the words used in the Fair Region?\n5. In section 3, the V, G of memory are not used. What's the output of FairRegion function?\n\nAdditional feedback:\n1. The usage of notation, citation is quite poor. \n- The usage of \\citep and \\cite are different.\n- In Table 1, what are PERU-MEXICO, MEXICO-PERU models?\n- Section2 -> Section 2", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Ske6R2YaYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the timely and important topic of unintended social bias in language models and proposes an approach to mitigate gender bias in the automatic generation of news articles. The automatic text generation is done by a sequence-to-sequence-model using word embeddings. The pattern of these embeddings in an multi-dimensional space can be associated with the stereotypes found in the corpora the models are trained on. To mitigate this bias the authors combine the network with a memory module storing additional gender information. This approach is inspired by the memory module used by Kaiser et al. (2017) to remember rare events, replacing the vector to track the age of items with a key representing the gender. \nThe proposed network architecture (\"Seq2Seq+FairRegion\") is evaluated using the bias score presented by Zhao et al. (2017) and compared to the sequence-to-sequende-models \"Seq2Seq2\" by Sutskever, Vinyals and Lee (2014) and \"Seq2Seq+Attention\" by Bahdanau, Cho and Bengio (2015).\n\nBesides some typographical errors (\"work embedding\") the paper is overall well written but unfortunately the flow of the text could be improved to make it easier to follow. The authors should perhaps consider to explain the structure of the paper in the introduction. There are some errors according the brackets of the in-text citations which should be corrected. \nThe two figures both illustrate the memory module and could be easily combined to one because the second figure just adds the encoding stage of the architecture. The caption of figure 1 could be used to further explain the \"Fair Region\" in the text (section 2). In section 4 the authors explain how to compute the bias score. Because these calculations are used to evaluate their methods, this section should perhaps be a subsection to section 5 (\"experiments\").\nTo evaluate the proposed method the authors use newspaper articles from Chile, Peru and Mexiko and compare the bias amplification metric for the trained models. The results are shown for the datasets from Peru and Mexico (Chile is missing) with a larger bias amplification for the Peru dataset. It could be justified more clearly why the experiments compare the bias amplification scores of different countries while the main focus of the paper is to mitigate gender bias in general.\n\nWhile this is an important topic, the contribution is rather minor and not well presented. Specifically the experiements should be extended.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers had a hard time fully identifying the intended contribution behind this paper, and raised concerns that suggest that the experimental results are not sufficient to justify any substantial contribution with the level of certainty that would warrant publication at a top venue. The authors have not responded, and the concerns are serious, so I have no choice but to reject this paper despite its potentially valuable topic.
Decision: Reject