Maximum Categorical Cross Entropy (MCCE): A noise-robust alternative loss function to mitigate racial bias in Convolutional Neural Networks (CNNs) by reducing overfitting
Authors: Nidhi Gowdra
Roopak Sinha
Stephen MacDonell
WeiQi Yan
Venue:  (2021)

Abstract: Categorical Cross Entropy (CCE) is the most commonly used loss function in deep neural networks such as Convolutional Neural Networks (CNNs) for multi-class classification problems. In spite of the fact that CCE is highly susceptible to noise; CNN models trained without accounting for the unique noise characteristics of the input data, or noise introduced during model training, invariably suffer from overfitting affecting model generalizability. The lack of generalizability becomes especially apparent in the context of ethnicity/racial image classification problems encountered in the domain of computer vision. One such problem is the unintended discriminatory racial bias that CNN models trained using CCE fail to adequately address. In other words, CNN models trained using CCE offer a skewed representation of classification performance favoring lighter skin tones.

In this paper, we propose and empirically validate a novel noise-robust extension to the existing CCE loss function called Maximum Categorical Cross-Entropy (MCCE), which utilizes CCE loss and a novel reconstruction loss, calculated using the Maximum Entropy (ME) measures of the convolutional kernel weights and input training dataset. We compare the use of MCCE with CCE-trained models on two benchmarking datasets, colorFERET and UTKFace, using a Residual Network (ResNet) CNN architecture. MCCE-trained models reduce overfitting by 5.85% and 4.3% on colorFERET and UTKFace datasets respectively. In cross-validation testing, MCCE-trained models outperform CCE-trained models by 8.8% and 25.16% on the colorFERET and UTKFace datasets respectively. MCCE addresses and mitigates the persistent problem of inadvertent racial bias for facial recognition problems in the domain of computer vision.

Under review conference paper at ICLR 2021
MAXIMUM CATEGORICAL CROSS ENTROPY (MCCE): A NOISE-ROBUST ALTERNATIVE LOSS FUNCTION TO MITI- GATE RACIAL BIAS IN CONVOLUTIONAL NEURAL NET- WORKS (CNNS) BY REDUCING OVERFITTING
Anonymous authors Paper under double-blind review
ABSTRACT
Categorical Cross Entropy (CCE) the most commonly used loss function in deep neu- ral networks such as Convolutional Neural Networks (CNNs) for multi-class classifica- tion problems. In spite of the fact that CCE is highly susceptible to noise; CNN models trained without accounting for the unique noise characteristics of the input data, or noise introduced during model training, invariably suffer from overfitting affecting model gen- eralizability. The lack of generalizability becomes especially apparent in the context of ethnicity/racial image classification problems encountered in the domain of computer vi- sion. One such problem the unintended discriminatory racial bias that CNN models trained using CCE fail to adequately address. In other words, CNN models trained us- ing CCE offer skewed representation of classification performance favoring lighter skin tones.
In this paper, we propose and empirically validate novel noise-robust extension to the existing CCE loss function called Maximum Categorical Cross-Entropy (MCCE), which utilizes CCE loss and a novel reconstruction loss, calculated using the Maximum Entropy (ME) measures of the convolutional kernel weights and input training dataset. We compare the use of MCCE with CCE-trained models on two benchmarking datasets, colorFERET and UTKFace, using a Residual Network (ResNet) CNN architecture. MCCE-trained models reduce overfitting by 5.85% and 4.3% colorFERET and UTKFace datasets re- spectively. In cross-validation testing. MCCE-traines models outperform CCE-trained models by 8.8% and 25.16% the ColorFERET and UTKFace datasets respectively. MCCE addresses and mitigates the persistent problem of inadvertent racial bias for fa- cial recognition problems in the domain of computer vision.
INTRODUCTION
Convolutional Neural Networks (CNNs) offer state-of-the-an art results in computer vision tasks He et al. (2016): Szegedy al. (2015); SSionyyan & Zisserman (2014) but are susceptible inherent noises in the input training data preempting overfitting on the input data during information propagation. When new data presented, overfit models do not generalize well and offer significantly lower classification performance, exacerbating the problem of bias towards specific subset of data. The fundamental learning theory behind CNNs is to approximate an underlying d-dimensional interpolated function f(X) E Rd by using information from number of d-dimensional input vectors X (X1, ... , X,, where x; =< x! x ... xd > and i, d E Z Maiorov] 2006). The problem of approximation is theoretically non-linear and there is empirical evidence support the assertion that CNNs simply memorize the input training data [Zhng et al. (2016).

---

Under review conference paper at ICLR 2021
Overlitting occurs when the internal parameters of CNN model are finely tuned to the unique variances of the input training data that it perfectly models its characteristics Hawkins (004). Misclassification occurs when overfit models are unable to distinguish between overlapping variances for different classes of images.
Reducing overfitting is also difficult since establishing a theoretical understanding or analyzing the mecha- nisms of learning in CNNs for non-convex optimization problems such as image classification is generally not well understood S (2018).
simple way to reduce overfitting is to train models using very large number of images [Shorten & Khosh- goftaar 2019), such the ImageNet dataset consisting of millions of training images used for the purpose of natural image classification. While using big data solutions might mask the underlying problem of model overfitting, acquisition of clean/noise-free labeled data for supervised model training is challenging. The problem of data acquisition is compounded further by ethical, societal, and practical concerns when dealing with facial datasets, especially for the task of race gender classification.
Another key challenge while creating datasets is the consideration that needs to be made on the distribution of data amongst the multiple classes along with the variability of data within individual class. Unbal- anced datasets where the data distribution of images is not equal for all the classes introduces bias during model training Ganganwar (2012). The only viable solution rectify imbalanced datasets is to augment or supplement datasets with new images which as mentioned before is an ongoing challenge. To the best of our knowledge, there no research/work undertaken to optimize data distribution of the convolutional ker- nel weights during model training. We hypothesize that balancing convolutional kernel data, during model training could aide in mitigating bias and increase classification performance through alleviating the severity of inherent noise.
Some researchers attribute racial bias of CNN models to noises in the training data and associated labels proposing alternate loss functions like Mean Absolute Error (MAE) Ghosh et al. (2017) to commonly used loss functions like Categorical Cross Entropy (CCE), as explained in Section 2.1 MAE was proposed as a noise-robust alternative to mitigate the susceptibility of CNNs to noise, but as Zhang & Sabuneu (2018) asserts, MAE is not applicable for complex natural image datasets like ImageNet and as such it is not considered in this paper. The task of classifying race in human faces is established to be more complex than natural image classification because there exists a narrow range of possible variations in features between human faces of different races, especially when skin tone is not the major determining factor for racial identity Fu et al. (2014).
In this paper, we explore the problem of overfitting with respect racial classification by assessing the train-test divergence to quantify the degree of generalizability where a higher train-test divergence indicates a greater degree of model overfitting on the training data. We also propose novel extension to the com- monly used CCE loss function using Maximum Entropy (ME) Hartley (9928) measures, called Maximum Categorical Cross Entropy (MCCE). MCCE loss calculations are determined by taking into account the dis- tribution of convolutional kernel weights during model training and the traditional CCE loss. Most related works explore model wverpprrameterization Zhang et al.) (2019) or aader-parameterization [Soltanolkotabi et al. (2018) with unrealistic assumptions made about the distribution of input data; we do not make any such assumptions.
The contributions of this paper are as follows:
We propose a novel extension to the Categorical Cross Entropy (CCE) loss function using Max- imum Entropy (ME) measures known Maximum Categorical Cross Entropy (MCCE) loss to reduce model overfitting.
We empirically validate the MCCE loss function with respect to model overfitting using train- test divergence as a metric and evaluate generalizability across datasets by using cross-validation testing.
2

---

Under review conference paper at ICLR 2021
2 BACKGROUND
Section 2.1 presents understanding of how CCE loss is calculated. Sections 2.2 and ?? detail how kernel regularization and batch normalization influence CCE loss with their limitations. Section (2.3 provides the theoretical background of Maximum Entropy (ME) and methods to calculate ME along with estimating the reconstruction loss.
2.1
CATEGORICAL CROSS-ENTROPY (CCH) LOss
The most commonly used loss function is the Categorical Cross-Entropy (CCE) loss given in Equation 1 which is measure of difference between the probability distributions of one-hot encoded CNN computed class labels and ground truths. CNN classification uses softmax function to calculate the required proba- bility distributions Goodfellow et al. (2016).
H q) = E p(x:) log g(x:) Where, X, X
(1)
In Equation (1). g(3))) and p(z::) represent the probability distributions of the one-hot encoded CNN pre- dicted class labels and ground truths respectively for an input data vector x, Given that CNN model training introduces noises during convolutional operations or information propagation and that any inherent noise present in the input data can significantly affect model performance, a noise-robust alternative to CCE would help improve classification performance and mitigate bias. This is the reason why stochastic optimizers and gradient descent algorithms function using the framework of maximum likelihood estimation.
2.2 KERNEL REGULARIZATION
The intuition behind regularization is that of Ockham's razor to penalize complex models and to promote simpler models during training. Unlike empirical risk minimization which only considers loss minimiza- tion, regularization was proposed to minimize structural risk which considers both complexity and loss minimization. The most prominent and simple kernels that greatly minimize loss are selected [Bilgic et al, (2014). Model complexity is represented two ways, as a function of the total number of features with non- zero weights (L,) most commonly used in computer vision tasks for CNN models such as ResNet. Model complexity can be quantified using the La regularization formula given in Equation (2), defined by using the sum of squares of all the feature weights as the regularization term Cortes et al. (012)..
function of all the weights of all the features in model (L2). Lg regularization is
||||| = wf + we + w + Of
(2)
Equation (2). the magnitude of the absolute value of the weight w indicates complexity. Feature weights close have significant impact model complexity, while large outlier weight values have more pronounced impact on w. The quantity of feature weights n determined using the number of trainable and model complexity. Furthermore, kernel regularization
model parameters also contribute greatly as it is implemented currently for CCE loss utilizes CNN computed label errors and does not take the data distribution of the convolutional kernels into account.
2.3 MAXIMUM ENTROPY AND RECONSTRUCTION LOSS
The of Maximum Entropy (ME) for applications such convolutional kernel analysis is justified since ME is the only consistent way of selecting a single discrete data point from the set of input data vectors best fit the regression curve, proven axiomatically in Shore & Johnson (1980); Johnson & Shore (183)..
3

---

Under review conference paper at ICLR 2021
amage
2.0
50
Red Mice
MT pini
Difference in ME measurement for changes in the radius r used to measure the complexity con- tained in a local neighborhood.
Figure 1:
A method to approximate ME for digital images through the use of distributed normalized histograms Gonzalez & Woods (207); Jain (989). The open-source SciKit-image processing library written in Python can be used to calculate the ME measures for images) Virtanen et al. 2020)).
Entropy in images is related to the complexity contained in a given neighborhood, computed by using a circular disk with radius of r. The disk is used to measure minute variations in local grayscale level distribution. The maximum entropy for an image depends on the number of gray levels, an bit image has 256 gray levels (0-255) which has theoretical maximum entropy of (2g2(2*) = 8 bits per pixel. Changing the value of r can invariable produce higher or lower ME measure as illustrated in Figure [ Similarly higher or lower ME values will be obtained while measuring convolutional kernel weights. A decrease in ME divergence can be observed Figure r values of and 50 relative to r values of and 5. A significant difference in spatial/semantic information in the images can be seen with greater values, which suggests loss in precision during approximation.
ME measures for color images require the computation each of the three color channels, Red (R), Green (G) and Blue (B) i.c RGB separately and averaging the result. The averaged ME measures for images in the colorFERET and UTKFace datasets are 2.09 and 2.25 bits per pixel respectively using value of 1. The amount of time taken to calculate the ME measures is insignificant as the ME calculation script can be executed in parallel on the CPU, while CNN model training occurs the GPU, evidenced in the supplementary data uploaded. Solutions other than ME for image reproduction/reccostrricii incomplete measurements such as, the use of non-linear variations on fourier transformations fail when convolutional kemnels are incorporated Donoho et al. (199)). Furthermore. ME reconstruction has been shown to provide superior noise suppression while mostly preserving de-emphasized structural noise near the baseline (relative to high signal information) Donoho et al. (1990).
from noisy
Accurate reconstructions can be approximated using ID projection of any underlying function which reduced to (X) E Rd such that x; E X Reis & Roberty (1992). As discussed in Section the under- lying functional representation of the input dataset is f(X), the difference between the true representation S(X) and the ME reconstruction approximation gXX) is the reconstruction loss for the input dataset. Results presented Reis & Roberty (1992). indicate that reconstructions using accurate and noisy data had insignif- icantly small variations compared to the original, attesting to the noise-robust ability of using ME measures for reconstruction. This noise averse characteristic of ME is especially important for race classification as lighting ISO parameters of the input images significantly affect the performance of CNN models. Reconstruction loss described as the convolutional kernel data loss whereas CCE can be characterized as class label loss.
4

---

Under review conference paper at ICLR 2021
3 MAXIMUM CATEGORICAL CROSS-ENTROPE (MCCE)
The classification of data in CNNs primarily depends on the convolutional kernels represented by their weights. Optimization of kernel weights using a loss function is performed to ensure a closer approximation the underlying function f(X) is achieved. As discussed in Section 2.1 CCE is measure of difference be- tween two probability distributions, the ground truth and CNN computed label for a class C. The drawback of CCE is that it only considers class label errors and does not account for the distribution of the convolu- tional kernel weights. The estimation of kernel weight probability distributions is critical in knowing the state of model training and learning capacities which could enhance classification performance, and MCCE is proposed to rectify this limitation.
The Maximum Categorical Cross-Entropy (MCCE) loss function monitors the data distribution of convolu- tional kernel weights using ME measures along with traditional CCE loss and penalizes models which are overly complex. Apriori knowledge of the entropic distribution of the input data can be computed using ME measures which is used as baseline to monitor convolutional kernel weight distributions and penal- ize models with greater divergences. It is well understood that maximizing entropy measures using even partial information (such as from convolutional kernel weights) can enhance the estimation of probability distributions Macqueen Marschak (175)) used extensively to calculate CCE loss.
The main criterion for producing high quality reconstruction approximation is the incorporation of two- dimensional convolutions, which is traditionally a computational burden Wernecke et al.) (1977). CNN models implicitly two-dimensional convolutions produce feature maps therefore, the computational overheads are eliminated making the computation of MCCE loss very efficient. Furthermore, using MCCE loss 1 difference can be calculated between the reconstruction approximation g(X) and ground truth f (X) (CCE error).
Reconstruction егror be calculated using the apriori determined ME of the input dataset and recon- struction approximation g(X). Monitoring the divergence between the CCE error and reconstruction error provides indication in the degree of deviation of convolutional kernel weights and predicted class labels to the deviation of f(X) to determine if the convolutional kernels are stuck in global minima/maxima and thus indicating model learning has saturated.
3.1 ALGORITHM
The pseudo-code for MCCE loss is presented as Algorithm [ which requires the calculation of a 1D linear interpolation output for reconstruction loss is provided in Section 3.2
Algorithm 1 Maximum Categorical Cross Entropy (MCCE)
1: Input: One-hot encoded ground truth (ytrve) and CNN predicted (Upred) class labels, apriori ME of training images in the dataset (i.e. ME(X)).
2: Output: Probabilistic logarithmic loss of ypred with the ground truth Ytrue-
3: Initialize: A ME(X), f <- ME(w)
4: 7 -log( '"'
{CCE loss, Sp is the CNN score and sy the ground truth for the class C}
5: A μ (Convolutional Reconstruction (CR) loss}
6: K = Interpolation(r, (O,A), (0,1)) (1d linear interpolation to output x between 0 and 1 rather than in the range of 0 to A}
7: A = K (Maximum loss = CCE loss + CR loss}
8: return
5

---

Under review conference paper at ICLR 2021
3.2 ID LINEAR INTERPOLATION
A one-dimensional interpolation of the reconstruction error/loss is required as the MCCE loss is an extension CCE loss which outputs values between O and 1. A linear interpolant is the straight line between the two known points given by their coordinates (ae, by) and (b1.b1) Davis [1975). For any value i in the interval (do, 01), ) the value of j along the straight line can be calculated using the equation of slopes given in Equation
3
j bp
b
bp
Or j (a) i) + bi 00)
(3)
40
01
a1
a0
4 EXPERIMENTATION
Experimentation revolved around quantitatively measuring train-test divergence determine the degree of overfitting in trained CNN models for the traditional CCE loss and our novel MCCE loss using all of the tech- niques discussed in Section 2 valss-valiotiion tests by interchanging the datasets was performed to ensure CNN models trained using MCCE loss consistently generalize better were also employed. No modifications were made to our CNN model training regime compared to the original implementation presented He et al. (2016) apart from using different testing hardware and software frameworks (Keras with tensorflow backend).
4.1 DATASETS
To determine the effect of racial bias and the efficacy of our novel MCCE loss function, we select a balanced dataset (UTKFace Zhang al. (2017)) where each class of race/ethnicity has equal number of images and an unbalanced dataset (colorFeret Phillips et al. (1998)) where the distribution of data across the classes is unequal.
The colorFERET dataset contains 11,338
semi-controlled color images of 512x768 pixel size with 13 different poses from 994 test subjects. Due to our limited computing infrastructure, the images needed to be downsampled to 96x96 pixel resolution using cubic interpolation. The original dataset contains nine classes (Asian, Asian-Southem, Asian-Middle- Eastern, Black-or- Black-or-African-Anee African-American,, White, Hispanic, Native-American, Other and Pacific-Islander). Due the very limited number of subjects and images for four of the nine classes, the dataset was reduced to five classes (Asian, Asian- Master,
White, Hispanic) containing a total of 11,172 images.
The original UTKFace dataset contains 23,708 in-the-wild color images of 200x200 pixel size with five ethnic classes (White, Black, Asian, Indian and Others) of all age groups. Only the OECD definition for working age population (15-64) consisting of 18,095 images are considered since the facial variations are not severe enough to cause any unexpected errors like misclassification or underfitting. The images used in our experimentation were downsampled to 96x96 pixel resolution using cubic interpolation to accommodate our limited computing infrastructure.
4.2 EXPERIMENTAL SETUP
All experiments presented this paper were carried out with a single RTX 2080ti with IIGB of VRAM, generously provided by InfuseAl Limited (New-Zealand). All models datasets randomly shuffled when reading from storage into memory and a 20% allocation of the randomly allocated dataset reserved for testing. The training data was again randomly shuffled during model
trained from seratch with the
6

---

Under review conference paper at ICLR 2021
training to mitigate any variability in the input data. This process was repeated for all the three model training instances.
Table I: Results validating the efficacy of training models using the proposed MCCE loss function to miti- gate overfitting and reduce bias
Loss Function
Dataset
Train Acc.
Test Acc.
Train-Test A
Best Test Acc.
CCE
ColorFERET
97.57%
84.67%
12.90%
86.58%
MCCE (Ours)
colorFERET
97.44%
90.38% %
7.05%
92.22%
CCE
UTKFaae
94.77%
74.43%
20.34%
76.26%
MCCE (Ours)
UTKFace
91.09%
75.04%
16.05%
77.12%
Cross-Validation
Loss Function
Train Dataset ColorFERET ColorFERET
Test Dataset
Acc.
Best Acc.
score
CCE
UTKFace
17.36%
22.62%
0.29
MCCE (Ours)
UTKFace
26.16%
39.91 %
0.39
CCE
UTKFaae
colorFeret
19.74%
33.30%
0.26
MCCE (Ours)
UTKFace
colorFeret
44.90%
63.40%
0.57
4.3 RESULTS
The results in Table the averaged classification performance from the three model training instances and the best test accuracy using an early stopping patience of 10, monitored on the training accuracy. Table also presents the cross-validation results and scores highlight bias in trained models on new data. MCCE trained models outperform CCE trained models in all of the measured criteria. All links to the datasets and experimental scripts used are available online and can be accessed at: Uploaded as supplementary material.
5 DISCUSSION
Analyzing the data presented Table we clearly identify the effectiveness of MCCE in model training with respect overfitting and generalizability. Classification bias is evaluated by examining the weighted f score determined using the confusion matrix. Models trained using MCCE outperform standard CCE models by almost two times. Perhaps more concerning the fact that CCE trained models barely improve on random chance of 20% (five classes) on the balanced UTKFace cross-validation test. Although training models using MCCE might mitigate overfitting indicated by the relatively lower train-test difference. MCCE trained models still overfit to limited degree. The tendency of relatively lower degree of overfitting holds even for cross-validatie results where MCCE models achieve a greater weighted score and test accuracy.
Figures 3 and illustrate the confusion matrices for both the balanced ColorFERET and unbalanced UTK- Face datasets respectively. The MCCE algorithm provided enhanced classification performance for both datasets. The CCE loss was especially vulnerable to unbalanced datasets, suggesting implicit biases with respect to the training set significantly corrupts the final results. Similar patterns of implicit biases can be observed for the MCCE loss however not as pronounced relative to CCE trained models demonstrating the improved resilience of the MCCE algorithm.
MCCE trained models had a relatively higher loss of 1.36 and 1.5 compared to 0.35 and 0.38 for the col- oFFERET and UTKFace datasets respectively, suggesting a greater degree of L2 kernel regularization being employed by the MCCE algorithm. The higher loss indicates the MCCE trained models have not fully
7

---

Under review conference paper at ICLR 2021
Bits/Pixel
ME Measures of Conv. Kernel Weights
1.9
1.8
1.75
1.7
1.65
31
41
51
61
71
81
91 101 1111111222 121 131 141 151 161 171 181 191 201 211 221 2.11
ColorFERET
F Epochs
Figure 2: ME measures of the convolutional kernel weights during model training using r = for color- FERET and UTKFaae datasets
75.00
0.8
AiianWW-
1.097
0.67
0.0036
0.021
0033
0.001
0.09
00003
0.059
0.039
Slcc
:.009
20254
0.009
02022
020088
0.999
White
White
Mispani
2.011
50001
5,077
D40
0.62
Hispanic
0.0000
5.008
0.055
0.30
lotel
Fredited
Figure 3: Confusion matrix for the five classes of the balanced ColorFERET dataset; Asian-M-E: Asian Middle-Eastem
converged and greater improvements can be achieved with manual HP fine tuning. Furthermore, MCCE trained models generally converge faster taking an average of 176 and 204 epochs compared to 199 and 268 epochs for the colorFERET and UTKeaace datasets respectively. Faster convergence along with higher losses suggests an enhanced learning capacity of the MCCE models, which can be improved with exploration of additional techniques to improve convergence such as learning rates.
Figure illustraaes the convolutional kernel weight ME measures during model training for both datasets, which does not reach the 2.09 and 2.25 bits per pixel measures for colorFERET and UTKFaae datasets respectively. Examining Figure 2 highlights the MCCE model training process where large divergences from the maximum are quickly penalized and weight adjustments during the back boeck-ropagatton step correct these divergences quickly. In Figure 5 visualize the loss curves for CCE and MCCE loss functions during model training. MCCE and CCE loss functions generally follow the same curve relative to each other but differ in their final convergence; similar pattern can be observed in Figure ?? for model training.
8

---

Under review conference paper at ICLR 2021
W/100
0.91
0.03
0.055
0.033
0.0006
White
0.52
0.0309
0,005
9615
0.8
0.94
0.077
0015
0.0029
0.54
0.0302
6015
000)
0.6
Asimm
0.0094
0.004
0.89
0.0008
0,00009
Aetan
0:11
8/15
0.0023
0.056
8615
62
02
6.1
Obiaa
027
0.005
0.13
C.00
0.43
0.54
0.4
110303
01044
D411
Predised
Prodicted
MCCE
CCE
Figure 4: Confusion matrix for the five classes of the unbalanced UTKEaaee dataset
Prant METe
Figure 5: Loss curves for MCCE and CCE loss functions for ColorFERET and UTKFace datasets
6 CONCLUSION AND FUTURE WORK
In this paper, proposed novel extension the commonly used Categorical Cross Entropy (CCE) loss function known as Maximum Categorical Cross Entropy (MCCE). While CCE evaluates the probability distributions of the CNN predicted and ground truth class labels, MCCE extends this evaluation to include the entropic distribution of convolutional kernel weights during model training. MCCE provides a robust noise-averse method of calculating model loss since partial knowledge of the entropic distribution of the input data is determined by priori and large divergences from the maximum are penalized during model training.
MCCE loss takes into account the label loss and convolution kernel weight distribution loss or reconstruction loss penalizing model training if either of these distributions greatly diverge from optimal. MCCE loss has been empirically validated to minimize overfitting by 5.85% and 4.3% using ResNet architecture on the ColorFeret and UTKEaace datasets respectively. Furthermore, MCCE has shown to improve generalizability of trained models in cross-validation testing by 8.8% using the trained colorFeret models UTKFace and 25. 26.16% using the UTKFace trained models on ColorFERET. The knowledge of entropic distribution of convolutional kernel weights during model training can be used to determine the state of convergence of the model. This state determination can be used to adjust other model parameters like learning rates which are reserved future work.
9

---

Under review conference paper at ICLR 2021
REFERENCES
Berkin Bilgic, Itthi Chatnuntawech, Audrey P Fan, Kawin Setsompop, Stephen F Cauley, Lawrence L Wald, and Elfar Adalsteinsson. Fast image reconstruction with L2-regularization. Journal of magnetic resonance imaging, 40(1):181-191, 2014.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. L2 regularization for learning kernels. arXiv preprini arXiv:/205.265. 2012.
Philip J Davis. Interpolation and approximation. Courier Corporation, 1975.
David I. Donoho, lain M Johnstone, Alan S Stern, and Jeffrey C. Hoch. Does the maximum entropy method improve sensitivity? Proceedings of the National Academy of Sciences, 87(13):5066-500 1990.
Siyao Fu, Haibo He, and Long-Guagg Hou. Learning race from face: A survey. IEEE transactions on pattern analysis and machine intelligence, 36(1 36(12))2483-2509 2014.
Vaishali Ganganwar. An overview of classification algorithms for imbalanced datasets. International Journal of Emerging Technology and Advanced Engineering, 2(4):42 2012.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Rafael Gonzalez and Richard E Woods. Image processing. Digital image processing, 2, 2007.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. book in preparation for mit press.
URL; http://www. deepleamningook. org, 1, 2016.
Ralph VL Hartley. Transmission of information. Bell Labs Technical Journal, 7(3):535-555 563, 1928.
Douglas M Hawkins. The problem of overfitting. Journal of chemical information and computer sciences, 44(1):1-12, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint
2015.
Anil K Jain. Fundamentals of digital image processing. Englewood Cliffs, NJ: Prentice Hall,, 1989.
R Johnson and Shore. Comments on and correction to" Axiomatic derivation of the principle maximum entropy and the principle of minimum cross-entropy" 80 [Cores]Cr IEEE transactions on Information Theory, 29(6):942 944-9433,3339.. 1983.
James Macqueen and Jacob Marschak. Partial knowledge, entropy, and estimation. Proceedings of the National Academy of Sciences, 1211043319-3824, 1975.
Vitaly Maiorov. Approximation by neural networks and learning theory. Journal of Complexity, 22(1): 102-117, 2006.
P Jonathon Phillips, Harry Wechsler, Jeffery Huang, and Patrick J Rauss. The FERET database and evalua- tion procedure for face-recognition algorithms. Image and vision computing, 16(5):295-33) -36,, 1998.
MI Reis and Nilson Costa Roberty. Maximum entropy algorithms for image reconstruction from projections. Inverse Problems, 8(4):623, 1992.
10

---

Under review conference paper at ICLR 2021
Ohad Shamir. Distribution-speeffic hardness of learning neural networks. The Journal of Machine Learning Research, 19(1):1135-1163 2018.
John Shore and Rodney Johnson. Axiomatic derivation of the principle of maximum entropy and the prin-
ciple of minimum cross-entropy. IEEE Transactions on information theory, 26(1):26-37, 1980.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal Big Data, 6(1 ):60, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint 1409 2014.
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization land- scape of aver-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2): 742-769, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Er- han, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. O Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: https://doi.org/ 10. 00.1088/441592-019-0666
Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew
Stephen Wernecke al. Maximum entropy image reconstruction. IEEE Transactions on Computers, 26 (4):351-364, 1977.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: 161..3500 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C Mozer, and Yoram Singer. Identity crisis: Memo- rization and generalization under extreme overparameterizzatio. arXiv preprint arXiv:1902.04699 2019.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial autoen- coder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5810- 5818, 2017.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778-8788, 2018.
11


Reviews:
[{"review_id": "u6Y-6ZtfR7q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author studies the bias problem in race classification task with face data. Specifically, it first analyses the influence of kernel regularization and batch normalization to categorical cross-entropy loss and proposes a maximum categorical cross-entropy loss. Experiments on two face datasets colorFERET and UTKFace demonstrate the effectiveness of the proposed method. \n\nFrom the ethical aspect, the topic of this paper is important and interesting. But it seems the proposed loss is not specially designed for the racial bias problem, please consider evaluate the proposed loss on general image classification tasks. There have some losses for imbalanced training can be used in this topic (e.g. Focal loss, GHM-C loss). In this paper, the author only compares their approach with the traditional CCE loss which is not convincing. And also, to show the proposed approach can mitigate the bias problem in the race classification task, the author should show the accuracy for different races rather than an averaged accuracy. Overall, I think this paper’s topic is important but the approach seems not make sense and less relevant to the racial bias problem.\n\nPros.\n1.\tThe bias problem that this paper studied is an important problem for image classification, especially for race classification.\n2.\tThe results in the experiment section could partially demonstrate the effectiveness of the proposed MCCE loss.\n\nCons.\n1.\tThe writing of this paper is bad and hard to follow. The author uses several subsections (Section 2.1~2.3, 3.2) to introduce the cross-entropy loss, kernel regularization, batch normalization, and linear interpolation which are redundant.\n2.\tAlgorithm 1 is not aligned with the paper. The variable \\mu is not used in the algorithm but seems to be very important to the method (see Section 3). \n3.\tAccuracy, the key evaluation metric in the experiment part, cannot fully demonstrate the effectiveness of the proposed method. Please consider adding confusion matrix or per-class accuracy.\n4.\tThe discussion section (Section 5) seems not clear. The figures in that section (ME measures, loss curve, training accuracy) cannot support the conclusions.\n5.\tThere are some formatting problems in the paper (Page 7 Line 1 & 5). The figures in the paper look like screenshots from Excel which are not very clear. Please consider inserting the figures in a vectorized format.\n\nOverall,  I think this paper is far more below the ICLR acceptance bar.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DeG6T8ri8tJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n1.The authors propose an extension of the CE loss to reduce classification bias that occurs in present methods and datasets. They calculate Maximum Entropy (ME) for images on the entire training dataset and then calculate the reconstruction loss between this and the ME for convolutional kernels during training. Their experiments results show that minimizing this reconstruction loss along with CE speeds up convergence. \n2.The paper is thoroughly written with minor typos and is easy to follow.\n\n\nCons:\n\n1.How does minimizing Maximum Entropy in the form of reconstruction error help to improve the weights learned by the model more suited for unbiased performance ? As shown it might help in faster learning but it’s not very clear how and why it learns good feature maps? \n2.How is the ME entropy calculated? It is good to briefly discuss the method/formula to calculate that. \n3.In Algorithm 1, is there an error in line 5, it should be mu instead of gamma? The 1D interpolation seems like a good way to normalize, but is it the only way or the best way? \nAlso discuss some experimental results with per-class accuracy/precision/recall in case of unbalanced datasets.\n4.It seems like this loss acta as a regularizer on the CE loss only training, but does that also help as a prior knowledge or information to learn better information as discussed in paper several times. Some evaluation on this either comparing feature activations or particularly which category improves more compared to CE might give a better intuition.  \n5.Please discuss some related work  or compare against as baselines with papers also trying to reduce classification bias. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9IejdRQSqEs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: the paper proposes a new loss function, called MCCE to reduce the effect of overfitting to noisy examples. This involves calculating the Maximum Entropy (ME) of the input images as well as the filters (?). Experiments are conducted on standard datasets to validate the claims. \n\nReview: I find the current state of the paper very confusing and unclear. Specifically, it is unclear what the method is trying to optimize (other than adding some form of regularization term based on entropy). The only technical development of the algorithm is given in Alg. 1 and no justification is provided for the design choices (such as: how is mu = ME(w) used in the algorithm? What does convolutional reconstruction loss amount to? What is the purpose of the interpolation? ...). The general discussion up to Section 2 can be shortened significantly and devoted to the development of the method. Overall, the paper is poorly written on the technical side. \n\nAdditionally, it is not clear why the authors attribute the bias in the predictions to noisy examples. For instance, a poorly trained model or a model which overfits to certain examples can produce biased predictions. A number of recent work also aim to reduce the effect of overfitting to noisy examples. For instance, (Amid et al. 2019a) generalizes the GCE loss (Zhang and Sabuncu 2018) by introducing two temperatures t1 and t2 which recovers GCE when t1 = q and t2 = 1. A more recent work, called the bi-tempered loss (Amid et al. 2019b) extends these methods by introducing a proper (unbiased) generalization of the CE loss and is shown to be extremely effective in reducing the effect of noisy examples. Also, (Yang and Guo 2020) proposes peer-loss (which can be combined with CE, bi-tempered, etc. loss) for handling noise. Please consider referencing/comparing to these SOTA methods.\n\nAdditional references:\n\n(Amid et al. 2019a) Amid et al. \"Two-temperature logistic regression based on the Tsallis divergence.\" In The 22nd International Conference on Artificial Intelligence and Statistics, 2019.\n\n(Amid et al. 2019b) Amid et al. \"Robust bi-tempered logistic loss based on Bregman divergences.\" In Advances in Neural Information Processing Systems, 2019.\n\n(Yang and Guo 2020) Yang and Guo. \"Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates.\" In International Conference on Machine Learning, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3e_OHtMARuz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new extension to the categorical cross-entropy using maximum entropy (MCCE) loss function to reduce model overfitting. The goal is to stabilize the training with respect to overfitting and generalizability. \nStrengths:\n+The proposed method is simple and elegant. It is theoretically well-founded and easily implemented.\n+The paper provides good initial results, and the experiments are conducted on the various dataset: colorFERET and UTKFace.\n\nWeakness:\n+ The proposed method is not novel and just a combining of maximum entropy with cross-entropy.\n+ The authors claimed to upload the supplementary material, but it's missing.\n+ The authors should describe the detailed CNN models implemented with MCCE. And should report the computation cost for each experiment.\n+  More generalization analysis would be beneficial.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new loss, Maximum Categorical Cross-Entropy, which combines the usual cross-entropy loss with a maximum entropy regularisation term on the convolutional kernels, and is evaluated on image classification. The authors have trained a face classification algorithm on two datasets: UTKFace (https://susanqq.github.io/UTKFace/) and NIST colorFERET (https://www.nist.gov/itl/products-and-services/color-feret-database). The labels consisted in, respectively: White, Black, Indian, Asian, Others (over 18k images) and Asian, Asian-Middle-Easter, Black-of-African-American, White, Hispanic (over 11k images) (see section 4.1 of the paper).

From the meta-reviewer's perspective:
As stated in the title, abstract and in paragraph 3 of section 1, the motivation of the paper is to reduce model overfitting and racial bias towards one category. However, there is no further discussion about any "ethical, societal and practical concerns when dealing with facial datasets, especially for the task of race or gender classification". It seems to me that a paper that implements a "race classification" algorithm should at least devote a substantially long part of the discussion on the validity of such a task and of such a labelling process, as well as question the motivations and potential misuses. Who labeled these faces and based on what visual characteristics? Were the subjects of the photographs consenting and did they self-declare their ethnicities? Are the authors simply reproducing discredited phrenology assumptions about ethnicities and about "race", which is increasingly defined as a mere social construct? Given that there is nothing specific to face classification in the loss function, I wonder why did the authors decide to focus on ethnicity features? What exactly could a visual ethnicity classifier be used for? Given the sheer amount of questions raised by the paper, we have submitted it for review by the Ethics Board.

Summary of the reviews:
Reviewers gave scores scores 3, 4, 5, 5 (without rebuttal from the authors), raising concerns about the novelty and contribution of the method (as it is simply combining maximum entropy with cross-entropy), clarity of the explanation of the method, missing related work and baselines and evaluation metrics.

Based on the low scores, unfavourable reviews and an ongoing Ethics Board investigation, I recommend for this paper to be rejected.



While this paper is likely to be rejected (, I believe that these concerns should be raised and potentially reviewed by the Ethics Board (unless this is an obvious rejection). Thank you in advance for your time.
Decision: Reject