# Decorrelation-Based Deep Learning for Bias Mitigation
## 3 Materials and Methods
### DcANN and DcCNN  
ANNs are suited for modeling complex small datasets. For DcANN, we use the same architecture used in ANN which consists of an input layer, multiple hidden layers \(h^{1}\),..., \(l\), and an output layer with only the difference of using hidden layer output values as feature \(F\) in decorrelation loss function. The other input to the decorrelation loss function is bias \(B\) which can be \(N\)-dimensional and include more than one bias type. The framework of our model DcANN is shown in Figure 1. The output of first hidden layer [23] is given by:  
\[\begin{split} h^{1}_{j}&=\ \sum_{j=1}^{p}w^{1}_{ij}\ x_{i}+b^{1}_{j}\ \text{where}\ j=1,\ldots,s\\ F&=\ h^{1,\ldots,l}\end{split} \tag{3}\]  
Here, \(x\) is the input size of \(p\) and after applying a transfer function to \(f(h^{1}_{j})\) becomes the output of the first hidden layer, whereas \(w\) and \(b\) are weights and biases, i.e., \(\theta\) parameters of the neural network. The variable (\(s\)) denotes the total number of hidden units in the first hidden layer and (\(l\)) denotes the total number of layers. We use the output of the first hidden layer \(h^{1}\) as input \(F\) to our decorrelation function to reduce the dependencies of these output values on biases. Of course, it may be more appropriate to use just one or combinations of other layer outputs depending on the types of applications.  
For DcCNN or DcDNN, we follow the same technique used in ANN. We implemented decorrelation loss function and applied either a CNN or DNN architecture. We propose our DcCNN architecture as in Figure 2. The output of the first convolutional layer [23] is given by:  
\[\begin{split} Z^{1}&=\ W^{1}*X\\ F&=\ Z^{1,\ldots,l}\end{split} \tag{4}\]  
In Equation (4), \(X\) is the two-dimensional or three-dimensional input size of \(p\). \(*\) denotes the convolution operation. We apply a transfer function and max-pooling to \(Z^{1,\ldots,l}\) to the output of convolutional layers and concatenate them together to use as features for the decorrelation function. We use the first layer output \(Z^{1}\) as input \(F\) to our decorrelation function to reduce the dependencies of these output values on biases. As in the DcANN case, using just one or combinations of other layer outputs as features might be more beneficial depending on the complexity of task.  
Figure 1: Proposed DcANN architecture: Black dashed lines denote the output of hidden layers \(l\) which are combined together to represent learned features. Green dashed lines indicate the start of the learning process where backward arrows show back-propagation using their respective gradient values while forward arrows show forward paths with updated parameters. Network parameters are updated as per the objective function.  
Figure 2: Proposed DcDNN architecture. Black dashed lines denote the output of convolutional layers \(l\) which are combined together to represent learned features. Green dashed lines indicate the start of the learning process where backward arrows show back-propagation using their respective gradient values while forward arrows show forward paths with updated parameters. Network parameters are updated as per the objective function.