# Decorrelation-Based Deep Learning for Bias Mitigation
## 3 Materials and Methods
### Experimental Setup  
In order to evaluate our proposed generic method, we explore five different scenarios and different types of biases. To validate our proposed approach, we utilize a simulated biased dataset [7] generated specifically to check the performance of the model in mitigating the bias. Datapoints are generated using four Gaussians whose magnitudes \(m_{1}\) and \(m_{2}\) are controlled by sampling from two different uniform distributions to classify into two groups. We implement three layers of 3 \(\times\) 3 convolutions followed by tanh activation and max-pooling. This is followed by one hidden layer with 16 dimensions. The output of the last convolutional layer is used as feature \(F\), whereas \(m_{2}\) is a bias variable \(B\) since we assume \(m_{1}\) is the main reason for discrimination between two groups. We use a mini-batch size of 256 and the hyperparameter (\(\lambda\)) of 0.7. We consider commonly used standard datasets such as the German credit dataset and UCI adult dataset [24] which have been examined for biases. We consider age as a bias variable for the German credit datasets and gender as a bias variable for the UCI Adult dataset as shown in Table 1. The basic three- and two-layer DcANN model is implemented for the adult and German datasets, respectively, and compared with existing mitigation algorithms. For the German dataset, there are two layers with 50 and 10 dimensions with ReLU activations except the last layer with sigmoid activation. We use a mini-batch size of 100 with a dropout of 0.5 and the hyperparameter (\(\lambda\)) of 0.9. Detailed analysis of \(\lambda\) is give in Section 4. The regularization weight decay parameter is set to 0.05. For the adult dataset, we construct three layers with 200, 100, and 50 dimensions. The rest of the configurations used for the adult dataset is the same as for the German dataset except for the mini-batch size of 1024. The output of the last hidden layer is used as feature \(F\), whereas age and gender are used as bias variable \(B\) for the German and adult dataset, respectively. We also use the MNIST image dataset [25] to check the performance of the proposed approach of DcCNN. Since the dataset is grayscale images with no bias and specifically designed for digit classification, we decided to utilize [6] the approach of intentionally planting color bias in the MNIST dataset. A few examples from the color-biased MNIST dataset are shown in Figure 3a. The training dataset consists of colored digits which are randomly sampled from the normal distribution of the corresponding mean and variance. So, the ten colors with their mean color value are assigned to each digit. The variance values such as 0.02, 0.03, 0.035, 0.045, 0.05 are also explored in this experiment. The smaller variance value means more color bias. These variance values controlled the amount of color bias in the training dataset. The testing dataset is unbiased. Colors are assigned randomly to each digit in the testing dataset. For this dataset, we apply the DcCNN method and implemented the same network architecture as in [6], i.e., four convolution layers, followed by average pooling. We use softmax loss and decorrelation loss with \(\lambda=0.9\). For decorrrelation loss, we use the output of the first convolutional layer as features \(F\) whereas mean RGB color values are used as bias \(B\). RMSprop optimizer with a mini-batch size of 1200 and a learning rate (lr) scheduler with an initial lr of 0.01 with a decay of 0.5 is used. Another way of adding color bias in the MNIST dataset [26] is dividing the dataset to predict binary labels where 0-4 digits are assigned as labels 0 and 5-9 digits are assigned as label one and then flipping the label with a 25% probability and color with probability value which depends on the training environment. We combine these two training environments in one for our proposed method.