Exploiting Benford's Law for Weight Regularization of Deep Neural Networks
Authors: Julius Ott
Huawei Sun
Enrico Rinaldi
Gianfranco Mauro
Lorenzo Servadei
Robert Wille
Venue: Accepted by TMLR ()

Abstract: Stochastic learning of Deep Neural Network (DNN) parameters is highly sensitive to training strategy, hyperparameters, and available training data. Many state-of-the-art solutions use weight regularization to adjust parameter distributions, prevent overfitting, and support generalization of DNNs. None of the existing regularization techniques have ever exploited a typical distribution of numerical datasets with respect to the first non-zero (or significant) digit, called Benford's Law (BL). In this paper, we show that the deviation of the significant digit distribution of the DNN weights from BL is closely related to the generalization of the DNN. In particular, when the DNN is presented with limited training data. To take advantage of this finding, we use BL to target the weight regularization of DNNs. Extensive experiments are performed on image, table, and speech data, considering convolutional (CNN) and Transformer-based neural network architectures with varying numbers of parameters. We show that the performance of DNNs is improved by minimizing the distance between the significant digit distributions of the DNN weights and the BL distribution along with L2 regularization. The improvements depend on the network architecture and how it deals with limited data. However, the proposed penalty term improves consistently and some CNN-based architectures gain up to $15\%$ test accuracy over the default training scheme with L2 regularization on subsets of CIFAR 100.

Published in Transactions on Machine Learning Research (02/2025)
Exploiting Benford's Law for Weight Regularization of Deep Neural Networks
Julius Ott Technical University of Munich Infineon Technologies AG
juliusg.ott.@tumm ott de
Huawei Sun
Snaacii.su.ttummmmmm de
Technical University of Munich Infineon Technologies AG
Enrico Rinaldi
erinaldi. monnaldd..rrkk@@mmil.cco
Riken (THEMs
Gianfranco Mauro
gianfranco. mauro Winfinoooo.i...oomccom
Infineon Technologies AG
Lorenzo Servadei
lorensoo.eeruddiittam.ccc
Technical University of Munich
Robert Wille
robert. willeetummmmcccmm de
Technical University of Munich
Reviewed on OpenReview: /ttps: net/ forum? id= TnI59y=7lc
Abstract
Stochastic learning of Deep Neural Network (DNN) parameters is highly sensitive to train- ing strategy, hyperparameters. and available training data. Many state-of-the-art solutions use weight regularization to adjust parameter distributions, prevent overfitting, and support generalization of DNNs. None of the existing regularization techniques have ever exploited a typical distribution of numerical datasets with respect to the first non-zero (or significant) digit, called Benford's Law (BL). In this paper, we show that the deviation of the significant digit distribution of the DNN weights from BL is closely related to the generalization of the DNN. In particular, when the DNN is presented with limited training data. To take advantage of this finding, use BL to target the weight regularization of DNNs. Extensive experiments are performed on image, tabular, and speech data, considering convolutional (CNN) and Transformer-bassed neural network architectures with varying numbers of pa- rameters. We show that the performance of DNNs is improved by minimizing the distance between the significant digit distributions of the DNN weights and the BL distribution along with L2 regularization. The improvements depend on the network architecture and how it deals with limited data. However, the proposed penalty term improves consistently and some CNN-based architectures gain up to 15% test accuracy over the default training scheme with L2 regularization on subsets of CIFAR 100.
1 Introduction
The advent of Deep Neural Networks (DNNs) has revolutionized several domains by exploiting their adapt- ability and robust learning capabilitics without requiring full model interpretability. Although the number of large datasets in DNN research is constantly increasing, many industrial applications must deal with smaller, manually collected datasets. In addition, the choice of hyperparameters significantly affects the

---

Published in Transactions on Machine Learning Research (02/2025)
training stability and overall performance.
A notable area of research is weight regularization. Over the years, numerous studies have shown how ex- plicit weight regularization described in Van Laarhoven (2017), such as L2 or L1 regularization, can improve learning by imposing penalties to encourage smaller or sparser weight distributions. Implicit regularization techniques, such as Micup by Zhang et al. (2018) or Culmir by Yun et al. (2019), modify the input data to achieve similar effects. These strategics highlight the importance of weight regularization in improving the learning capabilities of DNNs. However, they are scale dependent and need to be fine-tuned for individual domains.
This paper explores a remarkable aspect of DNNs the distribution of the first significant digits of their weights, a phenomenon closely related to Beeford's Law (BL) Discovered independently by Newcomb (1881)
and [eeford 1938 BL describes a counterintuitive but common pattern in numerical data across several
domains: Lower significant digits occur more often than higher ones.
This scalo-invariant pattern described by hasS been obscrved in various domains, such as physical constants by Shao & Ma (2010) and stock prices by Ley (1996). In machine learning, the properties of BL have been exploited to detect anomalies in input data by O'Mahony et al. (2023) and synthetically generated images by Bonettini et al. (022)), underlining its relevance. Their motivation is derived from the observation that
natural datasets tend to exhibit the pattern of BL while synthetic data does not demonstrate this pattern. Sahu et al. 22021) show that the link between
BL and DNN weights is based on their mutual relationship to thermodynamics. In particular, the probability of energy states in closed systems is such that smaller energy states are more likely to occur than larger ones. This pattern consistent with that observed for BL Despite the highlighted correlations between DNNss and the Benford distribution, research has only focused observing whether the collected significant digits of trained weights follow [LL The authors of [Sahu et al. (202T) propose to use the correlation between significant digits of the weights and [L to estimate the generalization error of DNNs. A high correlation is then used as stopping criterion for Early Stopping, a mechanism that stops training when performance on a holdout validation dataset does not increase for a predefined number of consecutive epochs. This feature of BL is closely related to its relevance for bias detection in numerical data, such as synthetic images. In particular, data, especially with limited quantity, is always biased because it represents small snapshot of the real world. In the same way, this bias is reflected in the weights of DNNs Consequently, DNN trained with limited training data is expected to deviate from the ideal case of bias-free learning. One potential solution to enhance the performance of the DNN is to bias the weights towards the ideal case of bias-free learning by incorporating a penalty term into the loss function.
To date, no research has been published on the use of BL as a regularization technique. To analyze the effects of BL on DNNs weights regularization, this paper:
demonstrates the relation between ML datasets and BL
2. introduces a way to approximate BL via pradient-based optimization,
3. analyzes the effect of Benford regularization in training DNNs with reduced datasets compared to their standard L2 regularized training scheme.
Benford regularization has particular relevance for applications where a complex dataset is insufficient for a DNN or a DNN has limited capacity for a large dataset. To this end, our experiments utilize subsets of common image (CIFAR10/100) and audio datasets to emphasize the bias in the dataset. In this manner, we are able to draw comparisons between CNN-based architectures, such as DenseNet and ResNext, and Transformer architectures, including tiny VIT and tiny Swin Transformer, with respect to their performance random subsets of the data. The findings reveal consistent enhancements due to the Benford regularization when employed in conjunction with L2 regularization, opposed to L2 regularization alone. In particular, CNN-based architectures benefit from additional Benford regularization in subsets of the datasets with improvements of up to 15% Similarly, Transformer models exhibit enhancements, achieving up to 3% improvement through the integration of Benford and L2 regularization on the entire dataset.
Furthermore, experiments were conducted with small and hardware-optimizzed MobileNet models on the ImagenetIK dataset. It was observed that the larger-capacity model demonstratod a greater improvement in performance when utilizing Benford regularization, in comparison to the smaller model. The experimental
2

---

Published in Transactions on Machine Learning Research (02/2025)
results indicated an interaction between enhanced performance, model capacity, and data availability when employing Benford regularisation. The remainder of the manuscript is structured as follows. First, an investigation is conducted into related research on weight regularization and BL in connection to DNNs. Subsequently, an outline is presented of the relation between BL and physics, with an illustration of a close connection to DNNs. The proposed method introduces a differentiable approximation of the Benford distribution and demonstrates how BL. can be approximated via gradient descent. The following experiments are performed on the public image datasets CIFAR10/ 1000 (Krizhevsky] 2009), and Imagenet1K (Russakovsky et al., 2015). These experiments evaluate the performance of well-known DNNs trained from scratch with different dataset sizes. To complete the experiments, the regularizer is evaluated on different data domains, such as speech and tabular data. Following a thorough discussion of the experiments and their results, the limitations of the presented work are assessed. Finally, the paper concludes with an overview of its findings.
2 Considered problem and related work
Research on weight regularization focuses on specific favorable weight distributions. The goal of these approaches is to make DNN training stable while ensuring robustness against overfitting and noise in the data. This section discusses state-of-the-an art research that employs regularization and data augmentation for DNNs, and related work that features BL with respect to DNNs.
Regularization and data augmentation are of practical importance when training DNNs to avoid overfitting and foster numerical stability. Weight regularization, like L2 or L1, aims reduce the norm of the weights by adding penalty to the loss function. L2 regularization is mainly implemented as weight decay as described in Loshchilov & Hutter (2019). Alongside 1.2 regularization, the focus of seminal work is on implicit regularization, such as data augmentation methods reviewed by Shorten & Khoshgoftaar (2019) like Mirup by Zhang et al.] (2018) or Cutmix by Yun et al. (2019). Such techniques do not directly constrain the weights, but they affect their distribution. Dala augmentation distorts the input data to make the network robust against different conditions, such as lighting conditions for camera images, thus improving generalization. To achieve this, Mirup fuses two training samples and their respective label via convex combination to learn smooth decision functions. Cutmir extends the approach and cuts out patch from an input image and replaces the patch with information from different training sample. The targets are adjusted proportionally to number of switched pixels. An alternative regularization method, SAM by Foret let al. 2219)). ) reduces the sharpness of the loss landscape via second order gradient smoothing. Generalization is achieved by finding parameters that lie in a neighborhood with a low loss instead of suboptimal sharp minima. At last, Early Stopping is one of the most common methods in DNN rescarch. It describes a mechanism to stop the training when a monitored metric, like validation error, stops improving (Yao et al., 2007g. In particular, Early Stopping does not change the weights by adding penalty or changing the input data, so it is of relevance for training DNNs in any data domain and is commonly used. In the work of [Sahu] ct al. (2021), the deviation to [B has been used as metric when to stop the training. This metric makes a validation dataset obsolete, which in turn can be used as additional training data.
The distribution of significant digits has not yet been used to regularize the weights of DNNs via gradient-based optimization. To date, BL has been used to analyze the characteristics and the liability of the data. In Bonettini et al. (2021). ; the authors use the significant digits of the cosine transformation coefficients of an image to detect whether it was generated by Generative Adversarial Network (GAN). Similarly, BL has been used in O'Mahony et al. (2023) to discriminate between natural and corrupt data, leveraging it as a filter to detect out-of-distribution data or anomalous data points. These methods monitor whether the collected data or generated images follow and are motivated by the observat ion that natural datasets are known to obey BL As demonstrated by Sahu et al. (2021), the distribution of significant digits is a prodictor of the generalization of DNNs to the validation data and can function as an Early Stopping criterion. Notably, the incorporation of BL into the optimization process has been overlooked, consequently missing significant aspects of the Benford distribution.
3

---

Published in Transactions on Machine Learning Research (02/2025)
3
Motivation
Regularization and data augmentation are two prac- tical techniques with the polential to enhance the training and generalization of DNNs. However, these techniques must be customized to the spe- cifie data being utilized, which can lead to non- universal approach.
Benford's Law
DCT Cifar10
35
DCT Cifar100
DCT MNIST
Ocairence digi as fitss digit in
FFT Google Speech Commands
All Datasets
In various domains, researchers have observed that the significant digits of datasets often follow pe- culiar pattern, as shown in Figure [ known as BL Despite its widespread occurrence, the under- lying reason behind [B. remains a mystery (Wang & Ma 2023). Interestingly, random samples from randomly selected distributions have been shown to converge to BL (Hill 1995). This finding is also sup- ported by the original work of Benford, who noted that while some datasets deviate from BL their
20
15
10
5
union closoly aligns with it (Benford 1938 under- lining that unbiased data tend to adhere to BL To further explore this phenomenon, we analyzed the frequency spectrum of several image datasets monly used for DNN training. The coefficients of the Discreee Cosine Transformation (DCT) and Fast Fourier Transformation (FFT) are known to follow BL as cvidenced by previous studics, see Benford 2021b. b. As shown in Figure 1 not all Irequencies of these datasets follow BL However, the union of all datasets exhibits a strong correlation, providing further evidence that data from various unbiased distribu- tions obey BL Mathematically, the law is fulfilled when the frequency P(d) of any significant digit d is given
1
3
6
7
digit
Figure 1: Comparison of significant digit distribution of DCT/FFT coefficients in public image and speech datasots, the union of datasets, and Benford's Law.
((d) log, (1+ 1
2
where is the base of the number system. Alternatively, the authors in Berger Hill (2015) demonstrate that sequence satisfies BL if and only if "the fractional parts of its decimal logarithm are uniformly distributed between zero and one', shown in Eq.
P(d) logio (1+)) 2)
iff (X) mod U((00, 1),
(2)
where represents arbitrary numerical data. The described law is particular interest because it is scale- and baso-invariant, as shown by Berger & Hill (2021). To measure the deviation from BL we compute the Kullback-Leibler (KT.) divergence by (Kullback & Leibler 1951) between the significant. distribution of the
weights and BL BL KL), formally defined in Eq. 3
9 KL(Q II BL) = Σ Q(d) log d=1
(Qd) P(d)
(3)
where ((d) is the observed frequency of the d-th significant digit in the DNN weights. In the experimental
section, we show that networks trained with more data have weights closer to BL These observations are associated to theoretical results from Iafrate et al. 2015 on the partitioning of
numbers, subject within the realm of number theory, and its relation to BL
A partitioning process describes how fixed quantity distributed. To illustrate, partitioning describes the energy states of particles in closed systems, see Iafrate et al.) (2015) and is employed in the field of com- binatorics to describe the partitioning of an integer into positive smaller integers. Common DNN structures.
4

---

Published in Transactions on Machine Learning Research (02/2025)
such CNNs, employ fixed filters that partition the imput. Transformers, on the other hand, are employed to extract information from partitioned input data. Formally, we define quantity X that is divided into smaller parts:
X Σ nzJz.
(4)
with parts nj of size rj. The goal is to find the distribution p(n)) that explains how often we observe part of size in the subset that adds up to X. Intuitively, it is obvious that smaller numbers occur more often, which can be easily verified by inspecting the subsets that add up to ten: [[9, 1), {8, 2}, {8, f1, HH An illustrative example from Kafri (209)) describes the system of Eq. with a number X, represented as "boxes' with N non-interacting balls. The question is how many boxes have exactly n balls, which in turn gives information about the probability of integers a number. The combination of different boxes then forms number. As the probability of a ball being in box is the same for all boxes, the distribution of balls obeys BL This result is further generalized by Iafrate et al. (2015), showing that the distribution of parts approximalely follows simple power law p(n) ~ 1/x. The signilicant digit distribution Pa of the inverse power distribution obeys
BL as shown in Eq. 5
(dd1)110
dx/w
In(1 + 1/d] logi0 ( (1+
Pa
(5)
|100 dx/x
In(10)
The approaches of Iafrate et al.] (2015) and Kafri (2009) assume equal probabilities for each ball to fall in box the parts of each size. This assumption is based on the maximum entropy principle by Jaynes (1957). Whenever principle is the most unbiased assumption. Finding probability distributions that maximize the entropy is used in machine learning research for regularization in Haarnoja et al. (2018) and Chiang et al. (2005). Like partitioning, DNNs for classification adopt a top-down strategy that involves the division of the input into a smaller feature space by partitioning it with the trained weights. In DNNs, the number of features represents the parts, and the weights represent their respective sizes. With sufficient training data, we expect an unbiased partitioning of the input space into the feature space, resulting in weights that are approximately Benford distributed. However, in the absence of training data, the parts are biased towards fewer features with larger weights, leading to deviation from BL This provides an explanation for the data obtained from Figure 2 where the lack of information in the training data leads to less diversity in the feature space and, thus, to an increase in the distance to BL
further information about probability distribution is given, the maximum entropy
This motivates to find a way to exploit BL for weight regularization in DNNs. Specifically, our goal is to constrain the significant digit distribution of the weights closer to BL order to improve the test error independently of the task, focusing on smaller subsets of datasets. To the best of our knowledge, this is the first method that incorporates BL into the optimization process of neural networks via gradient-based optimization.
4 Approach
This section presents framework for incorporating BL into neural network regularization. To this end, we proposc differentiable approximation of BL that can be used to optimize the significant digit distribution of the DNN weights. The significant digits of the weights are updated via gradient descent to close the distance to BL To further improve the proposed approach, we utilize derived error functions on BL for exponential functions, thereby facilitating the relaxation of the optimization problem.
The approximation of BL is more tractable with the right-hand side of Eq. as it depends on the input data x and not on the significant digits. The modulo 1 operator is equivalent to the fractional part of number. The fractional part of any positive number x is defined as
frac(z) = x [x],
(6)
where [:] is the floor function and [x] denotes the next lower integer value of x. This function has disconti- muities at zero and one, but the gradient can be numerically approximated in those regions. To approximate the uniform distribution of the fractional part with gradient optimization, we utilize quantile regression
5

---

Published in Transactions on Machine Learning Research (02/2025)
described in Koenker & Hallock 2001). Quantile regression has been used lately in Reinforcement Learning (RI. to learn arbitrary distributions (Dabney et al. 2018). Specifically, we compare the quantiles of the weights to the quantiles of a uniform distribution U( 1). In terms of the cumulative distribution function (c.d.f.), the quantile p defines the probability that a random variable X evaluates smaller or equal to a
threshold n, as shown in Eq. 7
Fx(x) = Pr(X xx)=
The quantile function (p)) is defined as the inverse c.d.f, presented in Eq.
I(w): (1 + + = [n( if n (P) not an integer
k: = -n n() if n()) is an integer
Q(p) = (p)
(8)
Here, =(2), (.)).....,, (n) denote the ordered weights in ascending order and n is their total number. The L.] denotes the floor function. The expression (P)) is the index of the p-th quantile, and the value of k is the integer part of n()) If (P)) is not an integer, the p-th quantile is defined the average of the k-th and (k + 1)-th ordered observations. The p-th quantile of the standard uniform distribution 4(00 1) is already given by p. To measure the deviation from the standard uniform distribution, compare the quantiles of the weight distribution to the quantile value, as shown in Eq. g
K Σ -k) K
LBI(K,O 0)
(9)
where the number of quantiles and QK:: 0) denotes the k-th quantile of the weights 0. For an accurate approximation of the standard uniform distribution, the number of quantiles used for the regression is essential. The best cstimation be achieved when the number of quantiles is equal to the number of weights in the DNN. Since, quantiles computed based on the Quickselect algorithm by Hoare (1961), which has an average complexity of O(n), computing them for becomes infeasible for DNNs with millions of parameters. Thus, whenever needed, the quantile regression is computed for each layer sequentially. This loss formulation closcly related to isotonic regression for DNN calibration in Nieulescu- & Caruana (2005).
In Algorithm 1 we illustrate PyTorch-style pscudocode of the quantile regression steps. Finally, the quantile regression loss added to the objective function and consequently not limited to classification tasks.
Nevertheless, to enhance the stability of the proposed loss, we integrate the error function from Engel & Leuenberger (2003). The authors show that exponential functions of the form x AA obey BL within error bounds independent of X and x. The error on BL for the exponential function is defined as:
Er(f) = Σ e Adio" (1-e-110") 1og J.
to0
(10)
According to the calculation in Engel & Leuenberger 2003 the error is bound within Er(J) < 0.03. This result can be incorporated into the proposed Benford regularizer given that the unnormalized probability density of classification network defined as:
p(x, = e(f(n.))/T)
(11)
where fa, 8) defines the neural network output and r the temperature. Therefore, losses within the error bound are neglected.
5 Experiments
In this section, an evaluation of the Benford regularization is conducted on various datasets and model archi- tectures. The baseline consists of networks trained with L2 regularization, as this is an integral component of their original training scheme. The Benford regularizer is then evaluated top of the T.2 regularization,

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm 1: PyTorch-style pscudocode of computing the quantile loss for Benford regularization # W flattened model weights of shape Nx1
def quantile_loss _aalle___ss________ (W)
# Number of quantiles is equal to the number of weights N
n_ quantiles = shsape
Compute fractional part
(emainder(Loog1(0))11
Define quantile steps between and 1
quantile_stant steps (staatt end=1, steps-n_guantille # Compute quantiles for the weights i_quantlles- - quantile
conttllessepps
# Uniform quantiles are the quantile steps
uniform quaattileeeeeesseeee - saantile_tteps
Compare the quantiles with Mean Squared Error bl_loss mse_ mse_loss Bos_losss___aaa__a (W_quantiles, uniform__uantiles) return b1_loss
which contributes to numerical stability.
To this end, we train all networks from scratch on subsets of publicly available image datasots such as MNIST (LeCun) 1998), CIFAR 10/100 (Krizhevsky) 2009) and ImageNet (Russakovsky et al. 2015). In order to show invariance to different model architectures, former models. The Transformer models used throughout the experiments have 3 times more parameters than the CNN-based networks, rendering them predisposed to overfitting on limited datasets. At the same time, they have a larger capacity which is relevant when the experiments use the entire dataset. In addition to images, the experiments evaluate the performance of the M5 model from Dai et al. (2017) on the Google Speech Commands datasct (Warden] 2018) and two-layer MLP on the tabular Iris datasct (Fisher 1936) The hyperparameters used for training are defined in the respective sections. All reported results for CIPAR 10/100 obtained from 15 different seeds and the mean and standard deviation reported in the respec- tive tables. Due to limited computational resources, the results from ImageNet1K experiments are obtained from 3 different seeds. Thus, we illustrate the mean and standard deviation on the validation performance. Afterward, the table presents the test accuracy of the model with the highest validation performance. The implementation is based on PyTorch" v2.0 (Paszke et al. (019) and as processing unit, we used two NVIDIA Tesla R A30 GPUs. The implementation is publicly available.
evaluate the Benford regularizer on CNN and Trans-
5.1 MNIST experiments
In order to demonstrate that a reduction in the quantity of training data results in an increase in the KT. divergence, and consequently, a greater deviation from the expected value, an evaluation was conducted on LeNet (Lecun et al.] 1998) in conjunction with Benford and L2 regularization across various subsets of MNIST. The Benford regularizer was scaled to value of 0.1, and the mean test error curves across five seeds are presented in Figure 2 The results indicate that with fewer data points, greater improvements be achieved through Benford regularization. In addition to the substantial enhancement in test performance when training with only 1% of the data, the efficacy of this regularization diminishes as more data is incorporated into the training process. Concerning the discrepancy from BL, the proposed regularization effectively mitigates the BL KL divergence. However, does not fully compensate for the absent data. Networks with reduced BL KL demonstrate superior performance. It is noteworthy that the convergence speed of the BL regularized network is accelerated when utilizing only 10% and 1% of the data. However, during training, the BL KL values of both networks become comparable. This obscrvation underscores discernible correlation between the BL KL and the test error, thereby highlighting the significance of the proposed regularizer in facilitating convergence speed.

---

Published in Transactions on Machine Learning Research (02/2025)
test error in %
35
densenet121+
0.75
30
densenet1214
0.50
digit
25
0.25
0.00
BL KL
Occurence digit n as
0.10
0.08
15
0.06
10
0.04
0.02
100
epochs
100%
1%
1
3
5
100% L2+8L
10% L2+
1% L2+BL
digit
Figure 2: Comparing the test error in % and BL KL of L2 and 1.2 with Benford regularization when training LeNet on 100%, 10% and 1% of the MNIST dataset.
Figure 3: Comparison between Benford's Law and significant digits of DenseNet121 weights trained with 1.2 and Benford regularization and with L2 reg- ularization alone on CIFAR 10.
5.2 CIFAR experiments
In this study, we initially demonstrate the elficacy of the Benford regularizer on the CIFAR 10 and CIFAR 100 datasets. Both consist of 50, 000 training images and 10,000 test images with 10 and 100 different classes respectively. subset 10, 000 images from the training data used for validation. The evaluation of different dataset sizes is conducted by maintaining the validation and test data sizes constant. To this end, we present the results of CNN- and Transformer-based models introduced by Vaswani et al. (2017) For
this, the DenseNet by Huang et al. (2017) with depths 121 and ResNext with 29 layers by Xic et al. (017)) represent the CNN family. Furthermore, the tiny version of the Swin Transformer by Liu et. al. (2021 and the Vision Transformer (ViT) for small datasets proposed by Lee et al. (2021) represent the Transformer modols. The imput images are normalized to 32 x 32 images with a random crop and random horizontal flip during training, and for lesting, the images are only normalized. The CNN models are trained with an SGD optimizer, initialized with 0.9 momentum and 10-4 weight. decay, and an initial learning rate of 0.001. The Transformer models utilize an Adam optimizer with an initial learning rate of 0.0001 and cosine annealing learning rate schedule. Each network is trained for 200 epochs, and the learning rate of the CNNs is divided by 5 after [60, 120] epochs, following the settings in DeVries & Taylor 2017 For all models, the quantile regression loss for each layer is computed, and the average is added as regularization with a scaling factor of 0.1. As demonstrated in Figure 3 the proposed Benford regularizer not only minimizes the BL KL but also modifies the significant digits of the DNN weights to approach BL In contrast, a network optimized with L.2 regularization only exhibits deviation from BL. The results in Table [ and Table P report the mean and standard deviation of the test error obtained over 15 seeds. The results show that the proposed Benford regularizer improves the network performance when the mumber of data samples is limited. As additional illustration, Figure 4 show the improvements of the tiny ViT's and DenseNet121's test performance the subsets of CIFAR 10 and CIFAR 100 with the additional use of Benford regularization.
It is important to note that the ViT trained with L2 and Benford regularization on 80% of CIFAR 10 achieves a performance level comparable to that of the VIT trained on the entire dataset. Generally, the VIT benefits from the I.2 and Benford regularization across all datasct sizes. Given the substantially larger size of Transformers in comparison to the evaluated CNN models, the overall test accuracy is lower. However, the discrepancy between model size and dataset diversity elucidates the ViT's benefits, both on subsets and

---

Published in Transactions on Machine Learning Research (02/2025)
95
90
80
85
75
tost accurocy
80
70
65
60
derseneti21 L2 BL
65
dersenet121 L2
vit L.2
10%
40%
60% subset
80%
100%
subset
(a) DenseNet121 CTFAR 10
(b) VII CIEAR
55
70
50
45
te:t accuracy
50
40
test
30
30
dersenet121 L2
25
vit L2 + BL
dersenet111 L2
20
60% subsot
100%
10%
40%
60%
100%
subsat
(c) DenseNet121 CIFAR 100
(d) VIT CIFAR 100
Figure 4: The distribution of DenseNet121's test. accuracy in % (t) trained on subsets of CIFAR 10 (a) and CIFAR 100 (c) shows that it improves with BL regularization performance as L2 regularization on the full dataset, respectively. The distribution of tiny VIT's test accuracy on CIFAR 10 (b) and CIFAR 100 (d) improves on the subsets and on the full dataset over L2 regularization.
all subsets but achieves the same
the full dataset. Conversely, the effect of Benford regularization becomes negligible when the complexity of the datasct surpasses the capacity of the model and the available data, as evidenced by the performance of ViT on 10% of CIFAR 100.
In contrast, the CNN models exhibit enhancement of up to 15% on the subsets of CIFAR 100 and up to 10% on subsets of CIFAR 10. Additionally, the Benford regularization imposes constraints on the full dataset for the CNN models, where their performance approaches an optimal state for the respective architecture.
5.3 Imagenet experiments
In the final set of image classification experiments, the complexity of the task was amplified by employing the ImageNet1K dataset (Russakovsky et al., 2015).. The focal point of these experiments lies in the MobileNotV3 networks by Howard et al. (2099) which are optimized for mobile applications. In opposition to the preceding experiments, this section emphasizes reduced models for extensive datasets. To this end, have trained
9

---

Published in Transactions on Machine Learning Research (02/2025)
Table Average test error in % (4) and standard deviation on CIFAR 10. An evaluation the L2 regularization with (L2+BL) and without Benford regularization on the full dataset and random subsets of 80%, 60%, 40%, and 10%. We highlight the respective results in bold when the performance difference is least standard deviation.
Subset
Reg.
100%
80%
60%
40%
10%
Swin
L2
15.70
0.35
17.44 + 0.48
20.39 + 0.46
24.61 + 0.55
42.72 + 0.72
Transformer
L2+BB
15.17 0.42
16.66 = 0.59
19.94 = 0.41
24.55 = 0.50
41.72 + 0.53
Tiny VIT
L2
18.91 0.35
21.21 0.35
23.86 + 0.22
28.48 + 0.41
42.34 0.59
L2+BL
17.97 + 0.40
19.51 + 0.31
21.66 + 0.22
26.6 + 0.36
41.75 + 0.59
DenseNet121
T.2
5.61 + 0.24
+
12.86 + 0.27
16.30 + 0.80
33.65 0.47
L2+BL
5.71 + 0.245
7.6 + 0.61
9.01 + 0.27
12.04 + 0.74
27.62 + 0.64
ResNext29
L2
6.03 + 0.19
12.88 + 2.85
16.55 + 0.35
22.10 = 0.55
43.23 + 0.39
2x64
T.2+BL
6.03 + 0.11 9.04 + 0.29
10.49 0.38
13.3 + 0.55
32.35 + 0.39
Table 2: Average test error in % (L) and standard deviation on CIFAR 100. An evaluation of the L2 regularization with (L2+BL) and without Benford regularization on the full dataset and random subsets using 80%, 60%, 40% and 10% of the training data. We highlight the respective results in bold when the performance difference is at least one standard deviation.
Subset
Reg.
100%
80%
60%
40%
10%
Swin
L2
45.96
0.44
50.33 1.82
54.94 1.47
78.08 + 0.40
Transformer
L2+BB
44.08 0.47
48.09 = 0.46
52.41 + 0.77
1..87
77.65 + 0.34
Tiny VIT
L2
46.26 0.53
0.52
54.21 0.81
1.400
76.33 + 0.43
L2+BL
43.88 + 0.36
48.43 = 0.58
51.64
57.89 0.27
76.28 + 0.47
DenseNet121
L2
22.71 0.70
0.44
44.55
53.18 + 0.49
77.09 + 0.60
L2+BL
23.36 0.08
29.7 0.23
32.88 + 0.46
39.51 + 0.64
66.43 + 0.72
ResNex129
L,2
22.97 0.14
13.02+0.33
49.81 + 0.39
59.11 + 0.44
80.38 + 0.31
2x64
L2+BL
22.62 0.13
29.99 = 0.35
35.28 0.9
42.61 + 0.46
74.5 + 0.37
the small and large versions of MobileNetV3 on the ImagenetIK dataset with default data augmentation techniques, including horizontal flipping, random cropping, and color jittering. We have scaled the Benford regularization loss with factor of 10 which has yielded optimal results. The networks were trained for 90 epochs, which was the convergence limit, with initial learning rate of 0.1, which was divided by 10 after 30 and 60 epochs. Due to limited resources, networks were trained with batch size of 256 images, while networks proposed in the original work were trained with a batch size of 4096. As illustrated in Figure the mean and standard deviation of the validation error were calculated over three runs, with the best validation performance of cach network used for testing. As shown in Table and the validation error curve, the proposed regularization method resulted in a significant improvement.
Table 3: Imagenet IK test. accuracy in % (t) for the MobileNet networks.
Regularization
MobileNetV3 Large
L,2
66.96
L2+BL
67.82
MobileNet Small
L2
59.3
L2+ BL
59.77
10

---

Published in Transactions on Machine Learning Research (02/2025)
100
86
99
test accuracy
acturacy
95
+22 BL
Audio
50%
100%
50%
100%
subset
subset
Figure 5: Distribution of the test accuracy in % (f) trained on 100% and 50% subsets of Google Speoch Commands (Audio) dataset.
Figure 6: Distribution of the test accuracy in % (f) trained on 100% and 50% subsets the tabular IRIS dataset.
5.4 Speech and tabular datasets
To further demonstrate the effectiveness of the Ben- ford regularizer, we propose the evaluation of differ- ent data domains. For this purpose, we train the M5 model Dai et al. (2017) on the Google Speech Commands dataset (Warden 2018). The datasets contain recordings for a total of 85,511 training, 10, 102 validation, and 4, 890 test recordings. For training, we use the Adam optimizer with a weight decay of 0.01 and initial learning rate of 0.1 and scale the Benford regularizer by factor of 0.1. The model is trained for 100 epochs, and the learning rate is divided by 10 after 20 epochs, as described in Dai et al. (2017). For the experiments on the IRIS dataset (Fisher 1936). we use a neural net- work consisting of two fully connected layers with 10 and 3 neurons and a ReLu activation function (MLP). The dataset consists of 150 samples, ran- domly divided into 73 for training, 32 for valida- tion, and 45 for testing. The model is trained for 1000 epochs with the Adam optimizer and an initial learning rate 0.001. The Benford regularization scaled by a factor of 0.001. The results Table 4 show that the Benford regularization also improves the performance of DNNs regardless of the data do- main. Looking at Figure 5 in the case of the more complex audio datasct, the Benford regularization improves the average performance of the model regard- less of the dataset size. The enhancement. in performance achieved through training on a mere 50% of the audio data substantial, with an average improvement of 10% and a reduced variability. For the more straightforward IRIS datasot, Figure presents a boxplot illustrating that the performance of the 50% sub- set is enhanced by Benford and L2 regularization, while the performance of the full dataset remains nearly unchanged. In this scenario, the network approaches optimal state.
validation accuracy
65
45
MobileNetV3 Large L2
MobileNetV3 Large L2 BL
40
MobileNerV's Small L2
MobileNetV3 Small L2 BL
35
epochs
Figure 7: Validation accuracy (f) of MobileNetV3 small and large. Both models benefit from the Benford regularization.
11

---

Published in Transactions on Machine Learning Research (02/2025)
6 Limitations
The experimental findings indicate that regularizing the DNN weights towards BL improves the overall performance, particularly in scenarios with limited training data. The proposed regularization method con- sistently enhances performance in conjunction with L2 regularization. As illustrated in Figures 2 and 3
the network weights can be optimized to approach BL Given the scale invariance of Benford regularization, there are infinitely many ways to approximale it, which has led to the use of L2 and Benford regularization in tandem. Furthermore, experimental findings have demonstrated that Benford regularization is constrained by the amount of data, the complexity of the data, and the model capacity. In instances where the model is unable to learn reasonable features, Benford regularization is ineffective in compensating for this defi- ciency. Additionally, when the net is presented with sufficient data and it is near optimum, the Benford regularization has limited effect on the distribution of the weights because it is already closely following the Benford distribution. Since the applied crror bounds are related to exponential functions, such as the softmax function, the experiments were carried out on classification tasks. Energy- models learn an exponential energy function, which is used for generative modeling. This finding suggests that our approach is transferable to generative modeling, but not to regression problems, necessitating further analysis in this area.
Table 4: Average test error (L) and standard deviation on tabular and speech datasets. An evaluation of the Benford regularization on the full datasets and a random subset of 50%.
Subset
Regularization
100%
50%
M5 (Audio)
L2
14.12 = 0.39
17.14 + 0.38
Benford
14.11 + 0.31
16.04 + 0.35
MLP (IRIS)
L2
3.43 + 0.049
5.5 + 0.05
Benford
3.12 + 0.046
4.22 0.03
7 Conclusion and future work
While previous approaches only observed whether obtained data follows BL this paper is the first to pro- pose the Benford regularization, where BL is learned via gradient-based oplimization. The molivation is based on commonalities between DNNs, BL and thermodynamics and the intriguing features of BL It is scale invariant, thus independent of the data domain and is commonly used to detect bias and anomalics in measured datasets. The proposed method applies Benford regularization in combination with L2 regulariza- tion for numerical stability and presents substantial improvements. Extensive experiments were conducted on random subsets of common image datasets CIFAR 10/100 with CNN and Transformer-based models of varying number of parameters. The addition of Benford regularization boosted the performance up to 15% on subsets of CIFAR 10 and up to 10% improvement, on CIFAR 100 subsets. The limitations are observed when the model either reaches the optimum or the dataset is too complex for the limited amount of data. In these cases, the performance is the same as training with L2 regularization alone. Consequently, exper- iments on low-capacity MobileNetV3 models are evaluated on the large and complex ImageNetlk dataset. As expected, the larger capacity network benefits more from the additional Benford regularization as the smaller version due the larger amount of available data. This observation is further shown by additional experiments the tabular Iris and Google Speech commands datasets.
To summarize, our experiments demonstrate that Benford regularization yields the most significant improvements when a large-capacity model is paired with limited data or a lower-capacity model is paired with abundant data. However, we observe that there are limits to this approach, as overly complex models for simple data (e.g., Swin Transformer) or low-capacity models for complex data (e.g., MobileNetV3 Small) cannot effectively leverage this regularization.
interplay between model and dataset complexity, revealing
These results not only provide insights about model capacity but have practical implications for industrial DNN applications where data collection is challenging and model sizes are constrained by hardware design.
12

---

Published in Transactions on Machine Learning Research (02/2025)
References
Frank Benford. The law of anomalous numbers. 551 572, 1938.
Proceedings of the American phitosophical society, pp.
Frank Benford. Base dependence of benford random variables. Stats, 4(3):578-594, 2021.
Arno Berger and Thoodore P. Hill. An Introduction Benford's Law. Princeton University Press, 2015.
URL http://www. www.ww.sto...................... wwsttor.org/saalleee .ttp://ttaalllj..ctt........
Arno Berger and Theodore P Hill The mathematics of benford's law: primer. Statistical Methods & Applications, 30:779 795, 2021.
Nicolo Bonettini, Paolo Bestagini, Simone Milani, and Stefano Tubaro. On the use of benford's law to detect san-generated images. In 2020 25th international conference on pattern recognition (ICPR), pp. 5-955-5502, IEEE, 2021.
Yun-Wei Chiang, Peter P. Borbat, and Jack H. Freed. Maximum entropy: A complement. to tikhonov regu- larization for determination of pair distance distributions by pulsed esr. Journal of Magnetic Resonance, 177(2):184-196, 2005.
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Arlificial Intelligncee volume 32, 2018.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 421 425. IEEE, 2017.
Terrance DeVrics and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprinl arXiv:1708 045504 2017.
Hans-Andreas Engel and Christoph Leuenberger. Benford's law for exponential random variables. Stalis- tics and Probability Letters, 63(4):361-38) 365, 2003. ISSN 0167-7152. doi: http:://ooi.org11.1110
50167-7155(03300000--3
Ronald Fisher. The use of multiple measurements in taxonomic problems. Annals of eugenics, 7(2): 179-188, 1936.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Vienna, Austria, May 3-7, 2021, 2019.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Theodore P. Hill. A Statistical Derivation of the Significant- Digit Law. Statistical Science, 10(4):354 363, 1995. doi: 112214/5/11170009555
C. A. R. Hoare. Algorithm 65: find. Commun. ACM, 4(7):321-322, 1961. ISSN 0001-0782. doi: 10.1145/ 366622.366647. URL https:///ooi: https:///doooor////// 114...66621131 366647
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314 1324, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Joseph R Iafrate, Steven J Miller, and Frederick W Strauch. Equipartitions and a distribution for numbers: A statistical model for benford's law. Physical Review E, 91 (1):00633, 2015.
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev., 106:620 630, May 1957. doi:
h..1003/hysRev. 106.62.... URL https://link... aps. http:/iwa......./oo//////./ http:r//oo/......////eee.....ee 1103/PhysRev...o
Oded Kafri. Entropy principle in direct derivation of benford's law. arXiv preprint arXiv:0901.3047, 2009.
Roger Koenker and Kevin F. Hallock. Quantile regression. Journal of Economicc Perspectivess, 1 1444))14-11 December 2001. doi: 10. no.125//jpp.1...111
143
A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, Univ. Toronto, 2009.
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22 (1):79 86, 1951. doi: 10. 10.1214/aamm/1177299933 URL. https://dooooor///// 1tt.:/..111///am///a. /////////////////////
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basse learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-233) 2324, 1998.
Yann LeCun. The mnist database of handwritten digits. http://yarm. lecun, com/exdb/mmiis 1998.
Scung Hoon Lee, Seunghyun Lee, and Byung Cheol Song. Vision transformer for small-size datasets. arXiv preprinl Xiv:22 12. 13492, 2021.
Eduardo Ley. On the peculiar distribution of the us stock indexes' digits. The American Statistician, 50(4): 311-313, 1996.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022,, 2021.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Simon Newcomb. Note on the frequency of use of different digits in natural numbers. American Journal of Mathemalics, 4:39 40, 1881.
Alexandru Miculescu-Mizz and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 625-632, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: URL https://doi. org/10. 1145/1102351, 1102430
Laura O'Mahony, David JP O'Sullivan, and Nikola S Nikolov. On the detection of anomalous or out-of- distribution data in vision models using statistical techniques. In The 3rd International Conference on Artificial Intelligence and Computer Vision (AICV2023), March 5 7, 2023, pp. 426 435. Springer, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalía Gimelshein, Tuca Antiga, Alban Desmaison, Andreas Köpf, Edwwad Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanc deep learning library. ArXiv, abs/1112.00170) 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-25) 2015. doi: 10.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Surya Kant Sahu, Abhinav Java, and Arshad Shaikh. Rethinking neural networks with benford's law. In Workshop at the 35th Conference. on Neural Information Processing Systems (NewrIPS), 12 2021.
Lijing Shao and Bo- Qiann Ma. The significant digit law in statistical physics. Physica A: Statistical Mechanics and its Applications, (89(16):3109 3100--11661 2010.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1 48, 2019.
Twan Van Laarhoven. L.2 regularization versus batch and weight normalization. ar preprint arXiv: 706. 10X0v01706.05350, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Luohan Wang and Bo-Qiang Ma. concise proof of benford's law. Fundamental Research, 2023. ISSN 2667-3258. doi: https://doi.or// fww.::33..ree2233..0..
P. Warden. Speech Commands: A Dataset for Limited- Limited-Vocabulary Speech Recognition. ArXiv e-prinds, April 2018. URL https://arxiv.orgg wrtp://///////0.....//// 022/00099900009
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep ncural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492 1500, 2017.
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnctto. On early stopping in gradicnt descent learning. Constructive Approzimation, 26(2):289-31) 315, 2007.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVE international conference computer vision, pp. 6023 6032, 2019.
Hongyi Zhang, Moustapha Ciss6, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancoueer, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings, 2018.
15

---

Published in Transactions on Machine Learning Research (02/2025)
A Appendix
A.1 Ablation on weight decay parameters
In this set of experiments, the effect of weight decay magnitude on performance is studied. Due to the small gap wet the performance of L.2 and regularized liny ViT, this model chosen for the ablation study. The objective to determine whether a stronger L2 regularization can close the gap to the regularized version using BL. The CIFAR. 100 experiments in Table 2 utilize a weight docay rate of 5 10-4 Consoquently, the experiments in Table employ 10-4, 10-3, and 10-2 to assess the impact of varying weight decay magnitudes. In general, the tiny ViT benefits from larger weight decay parameters, phenomenon that is also observed in the regularized version of the BL. In the final experiment, the L2 regularization removed. As anticipated, the elimination of L2 regularization led to a decline in the performance of the regularized version of BL, given BL's inherent scale invariance, which necessitates its use in conjunction with L2 regularization. In summary, the efficacy of weight decay in enhancing performance when confronted with constrained datasets noteworthy. However, it does not surpass the regularized version of BL when employing equivalent training parameters.
Table 5: Average test error (4) and standard deviation on CIEAR 100 for different weight decay magnitudes.
Subset
decay rate
80%
40%
10%
Tiny VIT
10-4 4
49.34 0.34
60.09 + 0.58
76.23 + 0.16
10-3
49.58 0.73
0.016 59.69
76.14 0.27
10-2
48.76 0.41
75.83 0.30
Tiny ViT + BL
10-4
#7.84
0.206
76.37
10-3
47.48 + 0.46
58.18+0.33
+
10-2
47.02 + 0.68
58.17
75.39 0.22
Tiny VIT + BL only
49.84 0.41
0.48
76.47 + 0.14
16


Reviews:
[{"review_id": "m6asMGC4IX", "reviewer": "Reviewer_UuAS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper tackles regularization, a fundamental problem relevant to all neural networks.\n- The regularization method is novel, being the first to use Benford's Law for weight regularization rather than just observation.\n- The approach is well motivated through connections to other disciplines. The motivations section is really enjoyable to read because it builds from statistical principles to neural network applications.\n- Results are comprehensive and convincing. The validation is done across multiple domains (image, speech, tabular) and testing on diverse architectures.\n\n**Weaknesses**:\n- Limited benefits in optimal conditions, like when models are already well-performing, or when sufficient training data is available. In these cases, performs similarly to standard L2 regularization\n- Current theoretical framework mainly applies to classification tasks\n- The \"Approach\" section could benefit from more intuitive explanations before the mathematical formulations:\n  - The introduction of the fractional part (eq. 6) needs better connection to eq. 2\n  - The quantile regression method could use more intuitive motivation (not sure how feasible this is though)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel perspective on neural network regularization with cross-domain, cross-architecture applicability. The work is well-structured and follows a clear logical flow from theoretical motivation to practical implementation. While the approach shows most benefit in specific scenarios (limited data, matching model capacity), this specialization doesn't make it less valuable as I think many real-world applications face exactly these constraints. The technical contribution of making Benford's Law suitable to gradient-based optimization is significant. \nMain contributions are: \n- discussing Benford's Law connection to neural network weights (but this is also done in other works, as the authors have mentioned in the menuscript)\n- devising a method to use the Benford's prior to regularize the model, with experiments on different domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions/Doubts**:\n- Only L2 and L2+BL results are shown, but BL-only baseline is missing, why?\n- What is the computational cost or memory overhead compared to standard regularization methods (e.g. L2)\n\n**Suggestions**:\n- The \"Approach\" may be rewritten with more intuitive explanations and can benefit from some polishing.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rCfRsB7ZG0", "reviewer": "Reviewer_Epf9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The idea of considering BL for weights is novel and interesting to me. Moreover, a large number of experiments are presented to support the algorithm proposed by the authors. \n\nWeakness: There could be more reasoning, and explanations for the algorithm; it may also be helpful to discuss how we may apply it in practice. Besides, some parts of the paper are not written in a clear way. See my comments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper claims that the deviation of the significant digit distribution of DNN weights from Benford’s Law (BL) would affect the generalization of trained models. Based on some previous works, the paper introduces BL regularization to the training of neural networks. Experiments on images, speech, tabular data etc. are presented to demonstrate that adding the regularizer helps improve model accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Some sentences may have typos. I highlight the correction of potential typos in boldface. \n\n    i) “this bias **is** reflected in the weights of DNNs”. \n\n    ii) “where $**f(x,\\theta)**$ defines the neural network output”. \n\n    iii) “return **bl_loss**”. \n\n2. In eq (2), what is the $X$ mentioned in “$log⁡_{10} (X)  mod 1 \\sim U(0,1)$\"? \n\n3. The description of the algorithm should be clearer. For example, I am confused about\n\n    i)  What is the MSE loss mentioned in the pseudo algorithm (is it just eq (9)? But eq (9) is not MSE)? \n\n    ii)\tHow is the comparison between quantile and uniform distribution related to the distribution of significant digit? \n\n    iii) Why mention the error function from Engel & Leuenberger (2003)? How is it related to this paper (especially considering the fact that Engel & Leuenberger considered exponential random variables but not having a neural network setting). Why we use exponential random variable here? And in the definition eqs. (10), where does the temperature τ come from (is there a specific application behind it)? \n\n    iv) In the algorithm, what does $N$ refer to? And is there any word after the first \"#\"? \n\nThus, I strongly suggest the authors reorganize this description of the algorithm and make clear about the notations they use. For example, it would be much clearer if the complete formula for the loss function is presented. \n\n4. The authors did a wide range of experiments to demonstrate that adding BL regularizations would improve the accuracy of models. I suggest the authors further discuss when to add the BL regularizer, considering different models and different data size. For example, (a) on CIFAR 10/ CIFAR 100, the improvement of accuracy is small for a large portion of dataset, could you please explain why the performance differs on different models? (b) The experiments on Imagenet, speech and tabular data show only about 1% improvement, so is it still necessary to use BL regularization? \n\n5. As the authors mentioned in Section **6 Limitations**, the experiments consider errors based on Softmax function. It would be great if the authors give some explanations/guess on how the structure of training and testing loss functions affect the BL KL, or why it does not.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jz1irOuOtU", "reviewer": "Reviewer_3i5Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* Relevant topic: training and regularizing DNNs with limited data\n* experiments to verify claims\n\n\n# Weaknesses \n\n## approximation error\n* why is the fact that the union of multiple datasets follows BL is sufficient and how is it generalizable? one can always find a set of data sets where neither the distinct ones nor their union follows BL closely?\n\n\n## experiments\n* means and standard errors are based on 3 different seeds only. I consider this too little to obtain robust results\n* the only competitors are L2 regularization and L2+BL which does not seem convincing to me without benchmarking against other methods\n* when using transformer architectures the improvements seems rather marginal which indicates that the proposed refinement works only for specific architectures\n* similar in section 5.4 the improvement of BL is not really present in particular for small data sets\n\n\n## further comments\n* is (9) related to isotonic regression and post-calibration approaches?\n* am I correct that the proposed approach only works when being used on-top of the usual L2 regularization? why L2 regularization and not other weight regularization methods for DNNs?\n\n## mathematical flaws\n* equation (3) appears wrong. it should be either BL on the right-hand side or assumed that P is a distribution that follows BL exactly. \n* please increase brackets in (1) and (2)\n* the rounding operator in (6) is not properly introduced\n* the arguments of the uniform distribution below (8) should be non-bold\n\n\n## Language\n* the paper contains a number of grammatical errors and typos, e.g. \"gradient base optimization\" or \" significant digits is \"\n* some sentences are hard to follow and language should be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "this paper investigates the use of Benford's Law as a technique to conduct weight regularization in DNNs. The authors' approach is based on a recent finding by Sahu et. al that significant digits can be seen as a predictor of the generalization of DNNs to the validation data and can be used as early stopping criterion. Gradient-based optimization is practically used to conduct the regularization of weights.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* make contribution clearer. for me it reads like a refinement of L2 regularization that can help in specific architecture choices in classification tasks. However, the lack of theory and missing benchmarks does not support the currently made general claim. So I suggest to either extend the experiments or to narrow down the contribution to what the authors can show empirically. \n* language needs to be improved and mathematical inconsistencies correct", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We received three reviews on the submission. Reviewers provided a point-by-point response to most comments, although the reviewers did not interact further during the rebuttal period. At the end, 2 out of 3 reviewers are favourable, while 1 reviewer leans towards a rejection. There is a consensus that the technique presented in the paper is novel and might be of interest, as it touches a core topic in neural networks. The third reviewer, however, is concerned that the proposed experiments are limited, as they only consider a single baseline (l2 regularization).

Overall, I believe the paper is suitable for acceptance, but I would ask the authors for a few improvements in the camera ready version, based on the reviewers' comments. I believe these improvements are necessary for a complete validation of the algorithm, especially points 3 and 4.

1.  The language quality must be significantly improved, as there are many errors (e.g., "making them prune to overfit"), or sentences that do not make sense in English ("To the best of the authors notice"). Please proofread the manuscript professionally.
2. Math should be improved, especially by avoiding the inconsistent use of terms such as $log$ and $\text{log}$ (I suggest strongly to stick to the latter).
3. One reviewer was concerned about the lack of experiments using only BL. While the answer seem convincing, I believe adding a small ablation in this sense (with an explanation of the results), possibly as an appendix, could be beneficial.
4. The experiments consider a single setup for the weight decay and BL weights. I believe it is necessary to add an experiment showing the change in performance as these parameters are varied (e.g., can we recover the gap with BL by simply increasing the weight decay coefficient?).
Decision: Accept (poster)