User: Does a difference in prediction outcomes between two ethnic/gender groups from a credit scoring algorithm constitute a bias in the data collection process or the algorithm itself?
Assistant: I argue that differences in outcomes of credit scoring constitute both a data issue and an algorithmic bias. Let’s first examine the bias in the dataset. I think of the model itself as a black box that generates outputs to fulfill certain objectives – in the case of credit scoring, the objective is the accuracy, recall, and precision of default risk predictions. The model at its untrained version does not discriminate against foreign workers, women, or young people: in fact, it does not conceive of these (socially constructed) labels the way humans do. Yet, the dataset that the model is trained on is fraught with socially constructed biases. For example, social norms cast women as incapable, foreign workers as untrustworthy, and young people as inexperienced, leading to higher unemployment rate and worse living conditions for these groups. Their resulting higher historical default likelihood stems less likely from the demographics-related differences in capability than from the vicious cycles generated by social stereotypes. Stereotypes render datasets biased, which in turns bias the algorithms: given the goal to maximize prediction accuracy, the model learns the “patterns” in the dataset and absorbs the socially constructed differences as a part of its prediction algorithm. Can we blame the algorithm? – After all, the model is constrained to the “history” and has no imaginative capacity for the potential of the historically discriminated groups. Essentially we have a “garbage in, garbage out” scenario caused by the historical and representation biases in the data inputs.

On the other hand, I argue that the outcome differences still constitute an algorithmic bias. I believe that we could still “blame” the model for the objectives and constraints that it has to fulfill (even though we humans lay them down in the first place). As illustrated above, with accuracy as the evaluation metric, the entire prediction/forecast algorithm built on historical datasets is meant to repeat the history without looking for potential or discerning social norms. Does the objective of maximizing accuracy constitute a bias itself? – If we can discern the vicious cycle that is both created by and reifying the socially crafted myths (weak women, reckless youth, and xenophobia), a model that actively learns and propagates certain kinds of societal stereotype should certainly be called out for its bias! However, we hard-wire the models to optimize those goals, and to address the inherent bias in the metrics, we humans are responsible for modifying the objectives and adding appropriate constraints. For example, in the programming assignment we encounter the mitigation technique that rejects predictions within a certain decision boundary. Besides techniques like active rejection, can we add constraints such as promoting the growth of underprivileged groups as an objective? Without these constraints, I would say the outcome differences also constitute an algorithmic bias in the part of objective specification.