# A Survey on Bias in Deep NLP
## 5 A General Methodology for Dealing with Bias in Deep NLP  
Up to this point, it is possible to conclude that the bias problem is of relevance for industrial deployments of artificial intelligence solutions. When putting a language model into production for a defined task like classification, dialogue or whatsoever, the engineer has to ensure that no bias could affect the expected behavior of the system so future troubles due to stereotyped decisions are prevented. This paper has made an effort in showing the state-of-the-art in bias related research, specially on deep learning models for natural language processing. In addition, a clear definition of this phenomena has been provided, detailing all the elements involved in spotting the bias with precision and completeness.  
In this section, we propose the use of all those elements to help the engineer to identify them in the subject of her study and to follow a structured method to tackle it in a software engineering process. With that purpose in mind, the following steps are proposed as a general methodology for dealing with stereotyping bias in deep language models generation and application:  
1. Define the **stereotyped knowledge**. This implies to identify one or more protected properties and all the related stereotyped properties. For each protected property, you have to develop its own ontology.
2. With the previous model at hand, we can overcome the task of identifying **protected expressions** and **stereotyped expressions**, so your **stereotyped language** is defined. It is equivalent to the process of _populating your ontology_ (i.e., your stereotyped knowledge). There are some corpora available, like the ones mentioned in this work, but you may need to define your own expressions in order to capture all the potential biases that may harm your system. Anyhow, it is here when different resources could be explored to obtain a set of expressions as rich as possible.
3. The next step is to **evaluate your model**. Choose a distance metric and compute overall differences in sequence probabilities containing stereotyped expressions with protected expressions as priors. Detail the benchmarked evaluation framework used.
4. **Analyze** the results of the evaluation to identify which expressions or categories of expressions result in higher bias.
5. Design a corrective mechanism. You have to decide which strategy fits better with your problem and with your available resources: data augmentation, a constraint in the learning process, model parameters correction, etc.
6. **Re-evaluate** your model and loop over these last three steps until an acceptable response is reached, or though out your model if behavior is not what is desired. Rethink the whole process (network architecture, pre-training approach, fine-tuning, etc.).
7. Report the result of this procedure by attaching model cards or similar document formalism in order to achieve **transparent model reporting**.  
Following these steps may help in getting a final system you understand better and with predictions not affected or marginally affected by stereotyping bias. For sure, this method can be adapted or extended according to the requirements of each specific AI project.