Predictive Performance and Discrimination in Unbalanced Classification

In this thesis, we investigate two types of problems in classifier construction. First, we focus on the problem of class imbalance. We acknowledge that common accuracy metrics such as accuracy and precision are not appropriate to evaluate performance of a classifier for an imbalanced dataset, and therefore consider existing solutions including the ROC-curve and Kappa statistic, and extend the notion of the ROC-curve to a new single-value performance identifier “ROC-distance”. We employ several methods that reduce class imbalance, using variable support for itemsets, and synthetically oversampling the minority class, using techniques that preserve the original characteristics of a dataset, while creating an equal amount of samples for both classes. Secondly, we focus on the problem of fairness and discrimination in classifiers. Classifiers that learn from socially sensitive data, can learn certain decisions that may be considered discriminatory, due to biased data or even discriminatory choices that are recorded, which may lead to structural discrimination by the classifier. We show methods that quantify and remove discrimination from the classifier. The influence of discrimination removal on classifier performance is evaluated and we propose a different sorting precedence than proposed by CBA2 (Classification Based on Associations), which deals better with discrimination removal than the original CBA2 method. We develop an association rule mining classifier. During the development of this classifier, we used Bayesian Networks to visualize decision characteristics of the classifier and acknowledge that this is an effective way of analysing certain properties of a classifier, tuning it for more optimal performance, and understanding of the dataset and classifier in general. Finally we test, evaluate and optimize our classification system on a real, non-trivial to predict dataset including both imbalance and discriminatory aspects. And show that our extensions to the existing approaches improve their performance. Acknowledgements First, I would like to thank my thesis advisor dr. Mykola Pechenizkiy, whenever I was in doubt about results or the direction of the thesis, he was able to motivate me and open my eyes to different interesting directions. dr. Pechenizkiy consistently helped me when I had questions, even during travelling or at late nights via email, which is greatly appreciated. Secondly, I would like to express my gratitude to the other experts on the committee, who were involved in the evaluation of this thesis: dr. Nicola Zannone and dr. Alexander Serebrenik. Without their valuable input, the validation of this thesis could not have been conducted successfully. I would also like to thank my fellow master student Bob Giesbers for countless helpful discussions. And my father dr. Ben van der Zon for reading my thesis and providing valuable feedback. Furthermore, I want to thank my brothers and friends, both at home and at the University, for accompanying me during most valued coffee breaks and times of relaxation, which gave me the energy to passionately work on my thesis. Finally, I express my greatest gratitude to my parents who gave me unfailing support and encouraged me throughout the years of my study.