Maximum Categorical Cross Entropy (MCCE): A noise-robust alternative loss function to mitigate racial bias in Convolutional Neural Networks (CNNs) by reducing overfitting

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) offer state-of-the-art results in computer vision tasks He et al . ( 2016 ) ; Szegedy et al . ( 2015 ) ; Simonyan & Zisserman ( 2014 ) but are susceptible to inherent noises in the input training data preempting overfitting on the input data during information propagation . When new data is presented , overfit models do not generalize well and offer significantly lower classification performance , exacerbating the problem of bias towards a specific subset of data . The fundamental learning theory behind CNNs is to approximate an underlying d-dimensional interpolated function f ( X ) ∈ Rd by using information from n number of d-dimensional input vectors X = { x1 , x2 , · · · , xn } where xi = < x1 , x2 , · · · , xd > and i , d ∈ Z > 0 Maiorov ( 2006 ) . The problem of approximation is theoretically non-linear and there is empirical evidence to support the assertion that CNNs simply memorize the input training data Zhang et al . ( 2016 ) . Overfitting occurs when the internal parameters of a CNN model are finely tuned to the unique variances of the input training data that it perfectly models its characteristics Hawkins ( 2004 ) . Misclassification occurs when overfit models are unable to distinguish between overlapping variances for different classes of images . Reducing overfitting is also difficult since establishing a theoretical understanding or analyzing the mechanisms of learning in CNNs for non-convex optimization problems such as image classification is generally not well understood Shamir ( 2018 ) . A simple way to reduce overfitting is to train models using a very large number of images Shorten & Khoshgoftaar ( 2019 ) , such as the ImageNet dataset consisting of millions of training images used for the purpose of natural image classification . While using big data solutions might mask the underlying problem of model overfitting , acquisition of clean/noise-free labeled data for supervised model training is challenging . The problem of data acquisition is compounded further by ethical , societal , and practical concerns when dealing with facial datasets , especially for the task of race or gender classification . Another key challenge while creating datasets is the consideration that needs to be made on the distribution of data amongst the multiple classes along with the variability of data within an individual class . Unbalanced datasets where the data distribution of images is not equal for all the classes introduces bias during model training Ganganwar ( 2012 ) . The only viable solution to rectify imbalanced datasets is to augment or supplement datasets with new images which as mentioned before is an ongoing challenge . To the best of our knowledge , there is no research/work undertaken to optimize data distribution of the convolutional kernel weights during model training . We hypothesize that balancing convolutional kernel data , during model training could aide in mitigating bias and increase classification performance through alleviating the severity of inherent noise . Some researchers attribute racial bias of CNN models to noises in the training data and associated labels proposing alternate loss functions like Mean Absolute Error ( MAE ) Ghosh et al . ( 2017 ) to commonly used loss functions like Categorical Cross Entropy ( CCE ) , as explained in Section 2.1 . MAE was proposed as a noise-robust alternative to mitigate the susceptibility of CNNs to noise , but as Zhang & Sabuncu ( 2018 ) asserts , MAE is not applicable for complex natural image datasets like ImageNet and as such it is not considered in this paper . The task of classifying race in human faces is established to be more complex than natural image classification because there exists a narrow range of possible variations in features between human faces of different races , especially when skin tone is not the major determining factor for racial identity Fu et al . ( 2014 ) . In this paper , we explore the problem of overfitting with respect to racial classification by assessing the train-test divergence to quantify the degree of generalizability where a higher train-test divergence indicates a greater degree of model overfitting on the training data . We also propose a novel extension to the commonly used CCE loss function using Maximum Entropy ( ME ) Hartley ( 1928 ) measures , called Maximum Categorical Cross Entropy ( MCCE ) . MCCE loss calculations are determined by taking into account the distribution of convolutional kernel weights during model training and the traditional CCE loss . Most related works explore model over-parameterization Zhang et al . ( 2019 ) or under-parameterization Soltanolkotabi et al . ( 2018 ) with unrealistic assumptions made about the distribution of input data ; we do not make any such assumptions . The contributions of this paper are as follows : • We propose a novel extension to the Categorical Cross Entropy ( CCE ) loss function using Maximum Entropy ( ME ) measures known as Maximum Categorical Cross Entropy ( MCCE ) loss to reduce model overfitting . • We empirically validate the MCCE loss function with respect to model overfitting using traintest divergence as a metric and evaluate generalizability across datasets by using cross-validation testing . 2 BACKGROUND . Section 2.1 presents an understanding of how CCE loss is calculated . Sections 2.2 and ? ? detail how kernel regularization and batch normalization influence CCE loss with their limitations . Section 2.3 provides the theoretical background of Maximum Entropy ( ME ) and methods to calculate ME along with estimating the reconstruction loss . 2.1 CATEGORICAL CROSS-ENTROPY ( CCE ) LOSS . The most commonly used loss function is the Categorical Cross-Entropy ( CCE ) loss given in Equation ( 1 ) , which is a measure of difference between the probability distributions of one-hot encoded CNN computed class labels and ground truths . CNN classification uses a softmax function to calculate the required probability distributions Goodfellow et al . ( 2016 ) . H ( p , q ) = Σni=1 = p ( xi ) log q ( xi ) Where , xi ∈ X ( 1 ) In Equation ( 1 ) , q ( xi ) and p ( xi ) represent the probability distributions of the one-hot encoded CNN predicted class labels and ground truths respectively for an input data vector xi . Given that CNN model training introduces noises during convolutional operations or information propagation and that any inherent noise present in the input data can significantly affect model performance , a noise-robust alternative to CCE would help improve classification performance and mitigate bias . This is the reason why stochastic optimizers and gradient descent algorithms function using the framework of maximum likelihood estimation . 2.2 KERNEL REGULARIZATION . The intuition behind regularization is that of Ockham ’ s razor to penalize complex models and to promote simpler models during training . Unlike empirical risk minimization which only considers loss minimization , regularization was proposed to minimize structural risk which considers both complexity and loss minimization . The most prominent and simple kernels that greatly minimize loss are selected Bilgic et al . ( 2014 ) . Model complexity is represented in two ways , as a function of the total number of features with nonzero weights ( L1 ) or as a function of all the weights of all the features in a model ( L2 ) . L2 regularization is most commonly used in computer vision tasks for CNN models such as ResNet . Model complexity can be quantified using the L2 regularization formula given in Equation ( 2 ) , defined by using the sum of squares of all the feature weights as the regularization term Cortes et al . ( 2012 ) . ||ω||2 = ω21 + ω22 + ω23 + ω2n ( 2 ) In Equation ( 2 ) , the magnitude of the absolute value of the weight ω indicates complexity . Feature weights close to zero have no significant impact on model complexity , while large outlier weight values have a more pronounced impact on ω . The quantity of feature weights n determined using the number of trainable model parameters also contribute greatly to ω and model complexity . Furthermore , kernel regularization as it is implemented currently for CCE loss utilizes CNN computed label errors and does not take the data distribution of the convolutional kernels into account . 2.3 MAXIMUM ENTROPY AND RECONSTRUCTION LOSS . The use of Maximum Entropy ( ME ) for applications such as convolutional kernel analysis is justified since ME is the only consistent way of selecting a single discrete data point from the set of input data vectors to best fit the regression curve , proven axiomatically in Shore & Johnson ( 1980 ) ; Johnson & Shore ( 1983 ) . A method to approximate ME for digital images is through the use of distributed normalized histograms Gonzalez & Woods ( 2007 ) ; Jain ( 1989 ) . The open-source SciKit-image processing library written in Python can be used to calculate the ME measures for images Virtanen et al . ( 2020 ) . Entropy in images is related to the complexity contained in a given neighborhood , computed by using a circular disk with a radius of r. The disk is used to measure minute variations in local grayscale level distribution . The maximum entropy for an image depends on the number of gray levels , an 8 bit image has 256 gray levels ( 0-255 ) which has a theoretical maximum entropy of log2 ( 28 ) = 8 bits per pixel . Changing the value of r can invariable produce higher or lower ME measure as illustrated in Figure 1 . Similarly higher or lower ME values will be obtained while measuring convolutional kernel weights . A decrease in ME divergence can be observed in Figure 1 for r values of 5 and 50 relative to r values of 1 and 5 . A significant difference in spatial/semantic information in the images can be seen with greater r values , which suggests loss in precision during approximation . ME measures for color images require the computation on each of the three color channels , Red ( R ) , Green ( G ) and Blue ( B ) i.e . RGB separately and averaging the result . The averaged ME measures for images in the colorFERET and UTKFace datasets are 2.09 and 2.25 bits per pixel respectively using an r value of 1 . The amount of time taken to calculate the ME measures is insignificant as the ME calculation script can be executed in parallel on the CPU , while CNN model training occurs on the GPU , as evidenced in the supplementary data uploaded . Solutions other than ME for image reproduction/reconstruction from noisy or incomplete measurements such as , the use of non-linear variations on fourier transformations fail when convolutional kernels are incorporated Donoho et al . ( 1990 ) . Furthermore , ME reconstruction has been shown to provide superior noise suppression while mostly preserving de-emphasized structural noise near the baseline ( relative to high signal information ) Donoho et al . ( 1990 ) . Accurate reconstructions can be approximated using a 1D projection of any underlying function which is reduced to g ( X ) ∈ Rd such that xi ∈ X Reis & Roberty ( 1992 ) . As discussed in Section 1 , the underlying functional representation of the input dataset is f ( X ) , the difference between the true representation f ( X ) and the ME reconstruction approximation g ( X ) is the reconstruction loss for the input dataset . Results presented in Reis & Roberty ( 1992 ) , indicate that reconstructions using accurate and noisy data had insignificantly small variations compared to the original , attesting to the noise-robust ability of using ME measures for reconstruction . This noise averse characteristic of ME is especially important for race classification as lighting or ISO parameters of the input images can significantly affect the performance of CNN models . Reconstruction loss is described as the convolutional kernel data loss whereas CCE can be characterized as a class label loss .

Summary: 1.The authors propose an extension of the CE loss to reduce classification bias that occurs in present methods and datasets. They calculate Maximum Entropy (ME) for images on the entire training dataset and then calculate the reconstruction loss between this and the ME for convolutional kernels during training. Their experiments results show that minimizing this reconstruction loss along with CE speeds up convergence.