Input Normalized Stochastic Gradient Descent Training for Deep Neural Networks
Authors: Salih Furkan Atici
Hongyi Pan
Ahmet Cetin
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose a novel optimization algorithm for training machine learning models called Input Normalized Stochastic Gradient Descent (INSGD), inspired by the Normalized Least Mean Squares (NLMS) algorithm used in adaptive filtering. When training complex models on large datasets, choosing optimizer parameters, particularly the learning rate, is crucial to avoid divergence. Our algorithm updates the network weights using stochastic gradient descent with $\ell_1$ and $\ell_2$-based normalizations applied to the learning rate, similar to NLMS. However, unlike existing normalization methods, we exclude the error term from the normalization process and instead normalize the update term using the input vector to the neuron. Our experiments demonstrate that our optimization algorithm achieves higher accuracy levels compared to different initialization settings. We evaluate the efficiency of our training algorithm on benchmark datasets using a toy neural network and several mature modern deep networks including ResNet-20, ResNet-50, MobileNetV3, WResNet-18, and Vision Transformer. Our INSGD algorithm improves ResNet-20's CIFAR-10 test accuracy from 92.57\% to 92.67\%, MobileNetV3's CIFAR-10 test accuracy from 90.83\% to 91.13\%, WResNet-18 on CIFAR-100 from 78.24\% to 78.47\%, and ResNet-50's accuracy on ImageNet-1K validation dataset from 75.60\% to 75.92\%.

Published in Transactions on Machine Learning Research (08/2024)
Input Normalized Stochastic Gradient Descent Training for Deep Neural Networks
Salih Atici, Hongyi Pan, Ahmet Enis Cetin Department of Electrical and Computer Engincering University of Illinois Chicago Chicago, Illinois, USA {hpan21, satici2, aecyy)Quic.edu
Reviewed on OpenReview:
Abstract
In this paper, we propose a novel optimization algorithm for training machine learning mod- els called Input Normalized Stochastic Gradient Descent (INSGD), inspired by the Normal- ized Least Mcan Squares (NLMS) algorithm used in adaptive filtering. When training com- plex models large datasets, choosing optimizer parameters, particularly the learning rate, is crucial to avoid divergence. Our algorithm updates the network weights using stochastic gradient descent with & and fz-based normalizations applicd to the learning rate, similar to NLMS. However, unlike existing normalization methods, we exclude the error term from the normalization process and instead normalize the update term using the input vector to the neuron. Our experiments demonstrate that our optimization algorithm achieves higher accuracy levels compared to different initialization settings. We evaluate the efficiency of training algorithm on benchmark datasets using toy neural work and several ma- ture modern deep networks including ResNet-20, ResNet-50, MobileNetV3, WResNet-18, and Vision Transformer. Our INSGD algorithm improves ResNet-20's CIFAR-10 test curacy from 92.57% to 92.67%, MobileNetV3's CIFAR-10 test accuracy from 90.83% to 91.13%, WResNet-18 on CIFAR-100 from 78.24% to 78.47%, and ResNet-50's accuracy on ImageNet-1K validation dataset from 75.60% to 75.92%.
1
Introduction
Deep Neural Networks (DNNs) have gained immense popularity and have been extensively applied across various research fields due to their convenience and ease of use in many machine learning tasks LeCun et al. (1995): He et al.] (0016); Krizhevsky et al.] (2017); Simonyan & Zisserman (2014); Long et al. (2015) Researchers from different domains can readily utilize DNN models for their work, these models can adapt their parameters to find the best possible solutions for wide range of problems, particularly in supervised learning scenarios. The parameters of a DNN model are updated using various optimization algorithms, and researchers have proposed different algorithms that offer fresh perspectives and address different conditions Ruder (2016). It. important to note that different optimization algorithms can yield different results in a given problem depending on the task hand.
Stochastic Gradient Descent (SGD) is widely adopted optimization algorithm for supervised learning in DNN models. It is simple, efficient, and parallelizable algorithm that produce very accurate results on large-scale datasots appropriate initial conditions Bottou (201)). Another popular algorithm Adam can outperform the SGD in some cases Kingma & Ba (2014). However, its initial learning rate is crucial. A relatively high value can lead to divergence. Optimization algorithms always play a significant role in training DNN models, but ensuring convergence of weights and finding the optimal solution for given problem is not always guaranteed. Therefore, the evaluation of an optimization algorithm should also consider its limitations. Robustness to variability is crucial attribute expected from any optimization algorithm.

---

Published in Transactions on Machine Learning Research (08/2024)
Given the inherent ability of the Normalized Least Mean Squares (NLMS) algorithm to exhibit resilience to environmental fluctuations, it is plausible to anticipate that an optimization algorithm derived from NLMS principles would demonstrate favorable performance characteristics in dynamic settings or uncertain conditions. In this paper, we propose a novel optimization algorithm called Input Normalized Stochastic Gradient Descent (INSGD), which draws inspiration from the Normalized Least Mean Squares (NLMS) algorithm used in adaptive filtering Mathews & Xie (1993): Chan & Zhou (011 Our study focuses on
demonstrating the innovation from NLMS to INSGD and the effectiveness of INSGD over various tasks.
The organization of the paper is follows. Sections and 1.2 review the SGD and NLMS, respectively, introduces the proposed Input Normalized Stochastic Gradient Descent (INSGD) algorithm. Sec- tion 3 presents simulation examples. Section draws our conclusion.
Section 2
1.1 Stochastic Gradient Descent
Stochastic Gradient Descent (SGD) is an iterative optimization method commonly used in machine learning to update the weights of a neural network model. It calculates the gradient of the weights based on the objective function defined to measure the error in the training phase and estimates the new set of weights using the gradients with a predefined step size. The SGD with convex loss function can converge to the optimal or sub-optimal set of weights with the correct initial settings Li & Orabonal (201)). The gradual convergence provided by gradient descent boosts optimizing the weights for any type of machine learning model.
Assume a pair of y) composed of an arbitrary input x and an output y. Given a set of weights w E W where W stands for the space of possible weights, a machine learning model predicts the output using a non- lincar function f(x, w) and the optimal weights, w", to minimize the objective (loss) function L(y, f(x, w)):
w arg min [(x wEW
(1)
Due to the highly complex and non-linear nature of machine learning models, it is impossible to find a closed-form solution for the optimization problem given in Eq. (1) ada (2008). The stochastic gradient descent algorithm is rotuedd to avoid extensive computation and give an iterative method to estimate the optimal weights. The formula for SGD is given as:
w + 1) = w(k) AV L(y;, f(x;, w)),
(2)
where w(j) represents the weights at 5th step, V W is the gradient of the objective function calculated using a single training example yz), and X is determined by the step size and the learning rate.
Although SGD is a simple algorithu that can be applied to various tasks, it faces challenges related to tuming and scalability, which hinder its ability to converge quickly in deep learning algorithms. If the initial weights are not properly defined, or without preconditioned gradients that consider curvature information, the algorithm can get trapped in local minima Le et al. (2011); Hinton & Salakhutdinov (2006). To estimate the minimum of the objective function more effectively, deeper understanding of the error surface is required. In addition to using gradients, the exploitation of second-order derivatives can lead to faster convergence. However, this requires calculating the Hessian matrix of the objective function. Calculating the second derivative with respect to each weight is computationally expensive and can lead to memory issues in deep networks. The Hessian matrix and its approximations are also utilized in the Normalized Least Mean Squares (NLMS)-type methods, which will be discussed in the following subsection.
1.2 Normalized Least Mean Squares (NLMS)
As pointed out above, the NLMS is widely used to estimate the weights of adaptive filter, which is basic lincar neuron. In minimum mean square error filtering, assume u, the input to system, is a 1 M random vector with zero mean and positive-defiite covariance matrix R, and d, the desired output of the system, is a scalar random variable with zero mean and a finite variance o3 The linear estimation problem is defined as the solution of
min E ld - uw w
2

---

Published in Transactions on Machine Learning Research (08/2024)
where w is vector containing the filter coefficients to be optimized. The linear estimation problem declares the cost function as the mean-square error and it is defined as:
J(w = E |dd - uw = E(d - - ud-(d-
(4)
is straightforward to obtain the cost function J(w) in terms of the covariance and cross-covariance matrices:
where () denotes a transpose. If expand Eq.
J(w) - d R w www-wwnee R + wr Rau w,
(5)
where Rut.- E[du] is the cross-covariance matrix of and u. The closed-form solution to such problem in (3) can be found using the linear estimation theory as Rn w Raui however, may not be possible to obtain closed-form solution for problems with criteria other than the mean-squarc-croor criterion.
The Least Mean Squares (LMS) algorithm Widrow et al. 1960) computes the stochasticc gradient and updates the weight vector iteratively to find a solution for the problem in Eq. (3). The weight vector can be updated using the following iterative process:
w(1) w(j - 1) + Au Au²(()e(()) (j)e(i);
-
(6)
where u(j) is j-th observation of the random vector and e(j) d()) u² w(j 1) is the error vector at time j. The updating term is obtained as the negative of the stochastic gradient of the mean squared error function defined in Eq. 4 with respect to the weights.
The NLMS algorithm has been shown to achieve a better convergence rate compared to LMS by incorporating a different step-size parameter for each component u, of the vector [Saeed (2008). The LMS algorithm can encounter scalability issues when the input signal is large or when the step-size parameter too large, Since the LMS algorithm uses gradient-based approach to update the filter coefficients, and if the step-size parameter is too large, the filter coefficients can diverge. To address this, normalization is introduced to the update term:
w() w(j 1) 1)+
| U(),
and the NLMS converges to the Wiener filter solution of the optimization problem in (3 long as 0 < À < Theodoridis et al. (2110) Yamada et al. 2002).
Another interpretation of the NLMS algorithm is based on the fact that the error e(j) d(j) n w should be minimized by selecting an appropriate weight vector w. The equation d(j) = u" w is a hyperplane in the M dimensional weight space w E R.M. When the vector w(j - 1) is projected onto the hyperplane e(j)- = d())- - u w, we obtain the update equation:
c
w()) w(j 1)
10TT u())
(8)
As shown in Fig. 1 the error is minimized by selecting the next weight vector on the hyperplanc d(j) = u.w. The orthogonal projection operation described in Eq. (8) minimizes the Euclidean distance between the vector
w(j - 1) and the hyperplane d(j) = u w Combettes (1933 Trussell & Civanlar 1984); Cetinn et al. (1997) 2013) The weights converge to the intersection of the hyperplanes as shown in Fig. | provided that the
interscction of the hyperplancs is non-empty Combettes (1993); Cetin al. (2013)
Other distance measures lead to different update equations such as the fj-norm-based updates:
e())
w()) = w(j - 1) + ||u(t)]] u(j),
(9)
where |u(ji]| is the 4 norm of the vector uy Gunay et al. 2212 Sayin et al. [2014] Arikan et al. (1994 1995): Aydin et al. (1999). The fi-norm-based method is usually more robust to outliers in input.
This paper describes a new optimization algorithm inspired by Normalized LMS. It is called Input Normalized-SGD (INSGD) and utilizes the same approach as in NLMS. INSGD provides a better solu- tion to the variability issue that may cause divergence or inconsistent results and obtains better accuracy results on benchmark datasets. By adapting the concepts of NLMS deep learning, we can potentially improve the convergence behavior and overall performance of DNN models.
3

---

Published in Transactions on Machine Learning Research (08/2024)
w(f (V-1) 1)
d(j 1) uu'w'
w(j)
d()) = u'w"
w( 2)
w(j + 1)
d(j + 2) = u'w*
d(j + 1) =u' w'
Figure 1: Geometric description of the NLMS projection in R?.
2
Related Works
Machine learning models commonly utilize the backpropagation method for optimization Rumelhart el al. (1986); LeCun et al. (1888). Stochastic Gradient Descent (SGD) is a widely used optimization algorithm with various modifications in the machine learning community. While SGD can provide couvergence with proper initialization, researchers have identified both positive and negative aspects of SGD and have attempted enhance it according to their specifie objectives Ruder (2016). One issue with SGD is that it updates weights based solely the instantaneous gradient, which may lead to lack of global information and oscillations. Another challenge is using constant learning rate for all weights in the model. As training progresses, certain weights become more important than others, requiring different step sizes to ensure effective learning.
In recent years, the Adaptive Gradient (AdaGrad) algorithm aims to enhance optimization by adaptively adjusting the learning rate for each weight based on the cumulative sum of past. and current squared gradi- ents Duchi et al. (2011). This adaptive approach allows for finer adjustments of the learning rate, ensuring larger updates for weights with smaller gradicnts and vice versa. AdaGrad's formulas arc:
w(k: 11) +- w(k) Vo(k) +e V Pw(k) L,
(10)
m(k) + v(k 1) + [V w(k) T.)3
(11)
where v represents the weighted moving average of squared gradients, is the learning rate, and v(-1) = 0.
Later, RMSProp builds upon AdaGrad by incorporating momentum through an exponentially weighted moving average of squared gradients Hinton et al. (2012). This modification addresses AdaGrad's issue of diminishing learning rates, ensuring a smoother and more stable update process. RMSProp's formulas are:
w(k (1)+1)++ w(k)
Vo(k)+e W
7
(12)
v(k) Bu(k + 91 (V [w(c))2}²
(13)
where 3 represents the momentum parameter. RMSProp strikes balance between AdaGrad's adaptability and momentum's 's stability, leading to improved optimization performance.
Another widely used optimization algorithm is Adaptive Moment Estimation (Adam) Kingma & Ba (014)) It builds upon the concepts of momentum and the divisor factor used in RMSProp. In addition to maintaining an exponentially weighted moving average of the squared gradients like RMSProp, Adam also incorporates the notion of momentum by keeping track of an exponentially weighted moving average of the gradients themselves. This combination of momentum and the divisor factor makes Adam more adaptive and robust
4

---

Published in Transactions on Machine Learning Research (08/2024)
compared to RMSProp and AdaGrad. By considering both the first and second moments of the gradients, Adam adjusts the learning rate for each parameter individually, taking into account both the magnitude and direction of the gradients. This enables Adam to converge faster and handle a wider range of optimization scenarios. The algorithm is implemented as:
w(k + t (k) - Vu(k) + m(k),
(14)
o(k) + Bu(k 1) + (1 3) [V ww]
(15)
m(k) + 3m(k - 1) + (1 B)V wL)
(16)
where is the momentum and m(-1) -.
Layer-wise Adaptive Rate Scaling (LARS) is optimization algorithm designed to improve the training of deep neural networks by dynamically adjusting the learning rates for each layer based on the norm of the gradients and the norm of the weights You et al. 2017):
W (k:] w(k + 6 w(k) 7 V (N +
V Ww(K) L
(17)
where O, the parameter vector at time step 1, v7 is the base learning rale, (L(),) is the gradient of the loss function with respect 0,, e is small constant to prevent division by zero. The limitation is the performance of LARS compared to other adaptive methods can vary and is influenced by factors such as the dataset, model architecture, and hyperparaaeteers.
Another adaptive learning algorithm proposed by Singh et al. 2015 presented Layer-Specific Adaptive Learning Rate (LSALR), where the parameters in the same layer share similar gradients. Therefore, the learning rate the entire layer should be similar but different layers should have different learning rates. The work described adjust the learning rate to escape from the saddle points and it uses the & norm in gradients:
1) (k:) - (1 1 11+1og log (1 [ (
V waakyh.
(18)
Eq. 118 allows the learning rate increases when the gradients are small. The aim to correct the update term when the gradients are small in the high error low curvature saddle points. Therefore, the algorithm escapes from saddle points with large learning rate. Similarly, scales the learning rate stability if the gradients are too large. The use of the log function provides the scaling under different conditions.
In summary, Adam, AdaGrad, and RMSProp are optimization algorithms that address the limitations of standard stochastic gradient descent (SGD). These algorithms improve the convergence speed in various scenarios. While they incorporate normalization parameters, the update terms in these algorithms are still input-dependent and gradually decrease over iterations. On the other hand, our approach uses a normaliza- tion term based on the layer's input. its details will be presented in the following section.
3 Methodology
3.1 Input Normalized Stochastic Gradient Descent Algorithm
Input Normalized Stochastic Gradient Descent (INSGD) utilizes a similar approach as NLMS, We focus on enhancing the robustness to variability and real-time processing capabilities of the INSGD optimizer, addressing challenges related to various training settings.
In deep learning, we minimize the cost function:
N F(W) =
N Σ FW(W,,
5

---

Published in Transactions on Machine Learning Research (08/2024)
where W represents the parameters of the network, N is the number of training samples, and F& (W) is the loss due to the k-th training data. Let us first assume that there are linear neurons in the last layers of the network and d, is the desired value of the i-th neuron. Furthermore, let w1, be the initial weights of the i-th ncuron. We want the ncuron to satisfy
where x denotes the input vector to the neuron. During training, we have W1,0 Ia A de where x is the input vector due to the k-th training pattern. We select the new set of weights of the neuron by solving
arg min ||w.0 w/l?
(19)
s.1. X dІ-
One can easily obtain the solution using the Lagrange multiplier method, and the solution to the optimization problem is the orthogonal projection onto the hyperplane w xx d, Solving Eq. 19 gives us an update equation
Ci
W111 W0 +
+ xx112 XkI
(20)
where the error = d, - W1.0 -X,, the update parameter À = 1, and e is a small number to avoid the division by O. This selection of weights reduces (&(W) and it is the same as the gradient descent with a new step size determined by the length of the input vector. It is also the well-known NLMS algorithu used adaptive filtering and signal processing as shown in Sec 1.2 Eq. 7 The NLMS algorithm converges for < X < when the input is a wide-sense stationary random process. Inspired by the NLMS algorithm we can continue updating the neurons of the inner layers of the network in the same manner.
When the i-th neuron is not linear, we have
<(w x) di,
(21)
where (b)- is the activation function. In this case, solve the following problem to update the neuron weights:
arg min |r.0 w|1
(22)
s.t. (ww x²) = di,
or
arg min ||wc,0 - w/1
(23)
s.t. w X2 c((4).
where o(-) is the inverse of the vx-)) function. When v-(-) is the sigmoid, leaky-RELU, tanh, v((-) has well-defined inverse. If the activation function is ReLU, the negative values in the inverse are set to Os. Tn this case, the weight update equation will be
W = W3o0 + A (o(d)) Wi. xx)
Xk.
(24)
++ |1|12
By employing the solution described in Eq. 24 the NLMS algorithm can be adapted to optimize the weights in the final layer to minimize various cost functions. However, extending the INSGD algorithm to deeper networks with multiple layers poses a challenge in its derivation. We adopt similar assumptions to those used in the backpropagation algorithm to derive the INSGD algorithm for each weight in deep-learning model. These assumptions provide foundation for developing the INSGD algorithm, allowing effectively optimizing the weights across the layers of the deep learning model.
In addition to the final layer, we incorporate the input feature maps of each layer to apply the gradient term with normalization to the neurons using the backpropagation algorithm. This enables the optimizer to propagate the gradients and update the weights layer-wisely throughout the network. By leveraging the information from the input feature maps, we enhance the training process by ensuring that the gradients are

---

Published in Transactions on Machine Learning Research (08/2024)
appropriately scaled and normalized at each layer. This approach allows for effective gradient propagation and weight updates, ultimately contributing to improved optimization and performance of the deep learning model:
Vw L(e) c+ X 1
W Wk+1 WE
(25)
where Xş is the vector of inputs to the neuron and We are the weights of the neurons. Note that we drop in the weight notation that represents the neuron since the algorithm is applicable to every neuron. For convenience, we also change the notation for the learning rate from to μ.. description of how the INSGD optimizer algorithu works for any layer of a typical deep network shown in Fig. 2
CNN Feature Extractor
Fully Connected Label Predictor
aly
Bly iwF
Twy
loss by
W
Wiz
features
target outputy
Convolutional Layer
W
X,
Wia
Wyo
Wys
rupT Xo
Layar
gradient
giedient
4
aly JWi, e
aly
Updata algorithm
Update Slgorthhm
Bby awsy
Bby
Wi, (1 = wr, WG,(0
Wy. + = (1) +
116il awyo
Figure 2: INSGD algorithm for different layers. It utilizes the input to each layer to update the weights.
Power normalization in NLMS is used to introduce memory into the recursion so that the input power is estimated using the input data in the remote past. In the INSGD algorithm, we introduce an input momentum term to estimate the power of the dataset, enabling power normalization. By replacing the denominator term with the estimated input power, we emphasize the significance of power estimation in our algorithm. Furthermore, the utilization of input momentum allows capturing the norm of all the inputs. Denoted as P. the input momentum term accumulates the squared lg norm of the input instances:
PA = 3P + 0)
(26)
While estimating the input power is crucial, the normalization factor can grow excessively, resulting infinitesimally small updates. To address this, we draw inspiration from the Layer Specific Adaptive Learning Rate (LSALR) approach (Singh al. (2015) and employ the logarithm function and moving-average method to stabilize the normalization factor. However, the use of the logarithm function introduces the risk of negative values. If the power is too low, the function could yield negative value, reversing the direction of the update. To mitigate this, we employ a function with the rectified linear unit, which avoids the issue of negative values. Adding a regularizer may not be sufficient to resolve this problem, hence the choice of the rectified linear unit function. The function is designed as follows:
fe(u) { {
u iu if
(27)
where 6 regularizer to avoid the division by 0. After devising the function in Eq. (27 and the logarithm approach for Eq. (26) the optimization algorithm for any weight in any layer in network becomes
+++ = W fe(lo((P)) Pk)) Vw L(e),
(28)
7

---

Published in Transactions on Machine Learning Research (08/2024)
where Ps is defined in Eq. (26). and it is the estimate of the imput power that is updated with every instance of x and the proposed = 0.01. Therefore, it makes sure that the update term is always positive and stable. The iterative algorithm defined in Eq. (28) is the Input Normalized SGD algorithm.
One can explore different norms, such as the l or lee norm, as alternatives to the l2 norm. In our experiments, we also investigated using the I norm to assess impact on performance. NLMS algorithms based on the I norm are known to be more robust against outliers in the input, which suggests potential benefits in deep neural network training. In this study, we examined both the l2 and 4 norms and their implications. Since NLMS is based on the norm, the algorithm presented in Eq. (26) utilizes the l2 norm. However, for broader applicability, we can adapt the power estimation follows:
PR = 8P x)|[x11
(29)
where [1-IE is the p power of the p-norm. Extension of the INSGD to convolutional layers straightforward. The pseudocode algorithun of INSGD given Algorithm D
Algorithm Input Normalized Gradient Descent with Momentum for ł <- to do 9: + Vof((O)-1)
b Denote the gradient If input momentum not 0
if + then
if >> then
Pt + 3P (1- x1)
B Accumulate the power of input norm
else
P: + ||x,0112
end if
end if
It
< f(log(P.))
Ct
> Division by input norm
if + 0 then
Weight Decay
9t 9- 9t A0,-1
end if
if 7 / then
> Gradient with Momentum
if <>> then
by + rb: + (1 (1-T)
else
bi - 91
end if
9t + b;
end if
0g t 9,-1 pge
b Update the Weights
end for
3.2 Models Architecture
In this study, we conduct experiments using six different networks to evaluate the performance of the INSGD algorithm in the classification tasks of CIFAR-10, CIFAR-100, and ImageNet-1K, We make modifications to the network architectures and initialization settings to assess the impact of the INSGD algorithm. In this study, we employ several networks for the classification tasks. Specifically, we utilize ResNet-20 20 He ct al. (2016). MobileNetV3 Howard al. (2019) and Vision Transformer (ViT) Dosovitskiy al. (2020) for CIFAR-10 10, WResNet-118 Zagoruyko Komodakis (2016) for CIFAR-100, ResNet-50 and MobileNetV3s for Iet Additionally, design a custom CNN architecture specifically for CIFAR-10, which consists

---

Published in Transactions on Machine Learning Research (08/2024)
of four convolutional layers, each followed by batch normalization layer. These networks are chosen to provide diverse set of architectures and enable a comprehensive evaluation of the INSGD algorithm's performance across different datasets. The structures of ResNet-20 and custom-designed CNN in this study are shown in Tables and 2 respectively.In addition, we choose a patch size 4, a linear layer dimension of 512, depth of 6, and 8 heads designing the Vision Transformer (ViT). We train it using warm restarts with Cosine anncaling Loshchilov & Hutter (2016).
Table 1:
ResNet-20 Structure for CIFAR-10 classification task. Building blocks are shown in brackets, with the numbers of blocks stacked.
Layer
Output Shape
Implementation Details
Convl
16 x 32 x 32
3 x 3, 16
3 x 3, 16
Conv2 x
16 x 32 x 32
x
3 x 3,16
3 x 3,32
Conv3 X
32 x 16 x
x
3 3,32
3 3,32
Conv4
64 8 x
x
3 3,64
GAP
64
Global Average Pooling
Output
10
Linear
Table 2: Structure of the custom network with 4 conv layers for the CIFAR-10 classification task.
Layer
Output Shape Implementation Details
Conv1
8 x 32 32
3 3,8
Conv2
16 x 16 x 16
x 3,16, stride = 2
Conv3
32 x 8 x 8
3 x 3,, stride 2
Conv4
64 x x 4
x 3, stride =
Dropout
64 4
p= p= 0.2
Flatten
1024
Output
10
Linear
On large benchmark datasets, traditional optimization algorithms often struggle to find the optimum results if the learning rate not properly chosen. In such cases, these algorithms may diverge and fail converge to the desired solution. However, the INSGD algorithm offers solution by providing flexibility in learning rate selection, thereby improving the chances of reaching the global optimum. By adapting the learning rate dynamically based on the input and gradient information, INSGD euhances the optimization process and increases the likelihood of achieving superior results on large-scale datasets.
4 Experimental Results
Our experiments are carried out on a workstation with an NVIDIA GeForce GTX 1660 Ti GPU for the CIFAR-10 and workstation with NVIDIA RTX A6000 GPU for the CIFAR-100 and ImageNet-1K.
4.1 CIFAR-10 Classification
We conduct a series of experiments using the CIFAR-10 dataset which consists of 10 classes, initially em- ploying the custom-desiggned CNN and ResNet-20 models for training. In certain experiments, we make modifications the custom network explore the algorithm's capabilities. All the experiments are re- peated 7 times with different seeds verify the results. These experiments aim assess the algorithm's performance under various conditions.
9

---

Published in Transactions on Machine Learning Research (08/2024)
The base setting employed the SGD optimizer with weight decay of 0.0005 and momentum of 0.9. The models are trained using mini-batch size of 128 for 200 epochs, with an initial learning rate ranging from 0.5 to 0.01. The learning rate is reduced at multiple steps with varying rates. To augment the data, we perform padding of 4 pixels on the training images, followed by random crops to obtain 32x32 images. Random horizontal flips are also applied to the images with a probability of 0.5. Normalization is performed on the images using a mean of [0.4914, 0.4822, 0.4465] and standard deviation of [0.2023, 0.1994, 0.2010]. Throughout the training process, the best models are saved based on their accuracy on the CIFAR-10 test dataset. These settings are adopted from He et al. (0166).
In the initial experiment, we employ the ResNet-20 model as our bascline. The batch size is fixed at 128. We compare the accuracy results of our algorithm against those of other commonly used optimization algorithms, which are discussed in Section 2 The detailed accuracy results are presented in Table 3
Table 3: Accuracy results of ResNet-20 on the CIFAR-10 10 dataset with different initial learning rates using different optimization algorithms.
Optimizer Initial Learning Rate
Test Accuracy
SGD
0.1
92.57=0.11%
Adam 0.001
01.34+0.01%
Adagrad
0.1
89.41
Adadelta
0.1
89.24+0.01%%
INSGD-
0.1
92.67=0.13%
INSGD-E2
0.1
92.60+0.111%
In addition to showcasing the testing accuracy results, understanding the behavior of each optimizer through- out the training process is crucial. Figure | visually represents the progression of testing set errors for cach optimizer over 200 epochs. By training the ResNet-20 model with each optimizer, we can observe the corre- sponding testing set error depicted in the plot. This visualization offers valuable insights into the convergence speed and overall behavior of each optimizer, enabling a comprehensive analysis of their performance and effectiveness.
As depicted in Figure 4 the INSGD algorithm with both norms consistently exhibits lower error rates in the testing set, which indicates the superior performance and effectiveness of INSGD in optimizing the model's parameters and minimizing the testing set errors. The ability of INSGD to adaptively adjust the learning rates for each individual parameter contributes to its remarkable performance in achieving lower errors during the training process. We can also observe that the behavior of INSGD closcly aligns with that of SGD, indicating their compatibility and similarity optimization behavior.
As seen in Table 3 only SGD closely matches the performance of our optimizer in this training scenario. Therefore, we concentrate comparing INSGD with SGD for training ResNet-20 CIFAR-10. We explore the impact of varying batch sizes on the normalization factor to understand how input size affects the training process. Analyzing the results across different batch sizes is crucial due to the trade-off between time and memory usage. While larger datasets may benefit from larger batch sizes to expedite training time, it is important to consider the increased memory requirements. Table 4 presents the accuracy results of other algorithms and INSGD when training the model with different batch sizes. To accommodate the increased batch size, we adjust the learning rate similar to the linear scaling rule described in Goyal et al. (0117).
Table 4 demonstrates that the INSGD optimizer maintains high performance across different batch sizes. Similar to NLMS, we anticipate that INSGD will be effective for real-time processing (online learning), highlighting its potential for superior performance in online learning scenarios.
To enhance the diversity of models utilized in our experiments, we incorporate the MobileNetV3s model for comparative analysis. A batch size of 256 is used in MobileNet training. The mean and standard deviation of 7 experiments with different seeds are shown in Table 5 As depicted, the results clearly demonstrate that the INSGD algorithm outperforms other conventional optimization algorithms in terms of performance. This
10

---

Published in Transactions on Machine Learning Research (08/2024)
ResNet-20 training on CIFAR-10
Adagrad
Adam %92.57=0.11
1.75
INSGD_L1 Acc: #9.26670.13
INSGD L2 Acc: %92.622.11
Adadelta Acc: %69.2442
150
1.25
Test Loss
1.00
0.75
0.50
0.25
125
Epoch
Figure 3: Optimizer performance over 200 epochs: Comparing test loss convergence across different
optimizers.
Table 4: Accuracy results of the ResNet-20 the CIFAR-10 dataset with different batch sizes.
Optimizer
Batch
Learning Rate
Test Accuracy
SGD
128
0.1
92.57+0..1%%
INSGD-
128
0.1
92.67-10.13 13%
INSGD-62
128
0.1
92.60-10.119
SGD
256
0.1
92.43:10.233
INSGD-
256
0.1
92.46-10.244 20.2%%
INSGD-
256
0.1
92.44+0.220
SGD
512
0.2
92.25-10.266
INSGD-
512
0.2
92.38-0.2299
INSGD-(2
512
0.2
92.32-+0.26%
11

---

Published in Transactions on Machine Learning Research (08/2024)
finding highlights the superior capabilities of INSGD in achieving improved outcomes across the evaluated metrics.
Table 5: Accuracy results of MobileNetV3 on the CIFAR-10 dataset with different initial learning rates using different optimization algorithms.
Optimizer Initial Learning Rate
Test Accuracy
Adam Adagrad
0.001
91.75=t0.09%
0.05
87.24+0.01%
Adadelta
0.05
84.78=0.29%%
SGD
0.05
93.06++0.11%%
INSGD-
0.05
93.04+0.16%
INSGD-E2
0.05
93.10=0.01%%
We also conduct experiments using the custom network for the CIFAR-10 training to validate our algorithm. We employed similar scttings those used in ResNet-20.. The accuracy results of the custom network with different initial learning rates are presented in Table
Table 6: Accuracy results of the custom-designed CNN on the CIFAR-10 datasct with different initial learning rales and reduction rales.
Optimizer Initial Learning Rate
Test Accuracy
SGD INSGD-E1
0.1
79.24
0.1
79.41+0.37%%
INSGD-
0.1
79.30=0.38%%
SGD
0.25
65.92+1.32%%
INSGD-
0.25
73.24+1.17%
INSGD-f2
0.25
74.68+00.85%
SGD
0.01
79.08=t0.34%
INSGD-L1
0.01
08.83:
INSGD-
0.01
78.92.40.31%
The toy network, used as a simplified reprosentation of the model, plays a crucial role in evaluating the effectiveness of our algorithm. The results oblained from training the toy network confirmn the robust of INSGD to variability, as it shows consistent accuracy results regardless of the network architecture or the learning rate used. Given the overlap in the experiments conducted with the custom network and ResNet-20, opted not to replicate the ResNet-20 experiments using the toy network. This decision was made to avoid redundancy in our findings and to focus on exploring the direct impact of INSGD.
We conducted experiments on a modified network architecture to explore the impact of INSGD's input normalization. Given that INSGD normalizes the input, we hypothesized that the batch normalization layer might become redundant. To test this hypothesis, we implemented on a toy network, as smaller scale allows for easier amplification and verification of this claim and on ResNet-2O. Table shows the accuracy results of networks trained without the batch normalization layer.
From Table 7 it is evident that the absence of batch normalization layer can be managed effectively by the optimization offered by INSGD. In particular, SGD diverges during training of the toy network without batch normalization, whereas INSGD achieves performance levels close to optimal settings.
Normalization layers, such as Batch Normalization, are used during forward propagation and are trainable components of neural networks. While they help stabilize the learning process and improve convergence by normalizing the input to each layer, they do not directly influence the update terms of the convolutional
12

---

Published in Transactions on Machine Learning Research (08/2024)
Table Accuracy results of the custom-designed CNN on the CIFAR-10 dataset with no batch normalization layer.
Model
Optimizer
Initial Learning Rate
Test Accuracy
SGD
0.1
11.07:2.74%
Custom CNN
INSGD-
0.1
10.85%%
INSGD-E2
0.1
74.78:7..84%
SGD
0.1
90.14+0.18%
ResNet-20
INSGD-L1
0.1
90.55-10.26%
INSGD-I2
0.1
90.70=1..5%%
layers. Cousequently, the use of INSGD allows for normalization during backpropagation as well, further stabilizing the optimization process and potentially enhancing model performance and training efficiency.
Beyond CNNs, we trained a Vision Transformer (ViT) model CIFAR-10 using different optimizers and settings. We hypothesized that the partitioning of an image into patches could influence how the optimizer converges during training. Table g presents the results of the ViT under various training configuratioss. To facilitate a warm startup, we employed Cosine Annealing for learning rate scheduling and trained the model for 300 epochs. The base setting utilized a batch size of 256. Additionally, we evaluated the performance of optimizers without the layer normalization layer.
Table Accuracy results of the VIT on the CIFAR-10 dataset with different initial learning rates and reduction rates.
Optimizer LayerNorm Learning Rate
Test Accuracy
SGD Adam INSGD-&
0.05
82.79:00.00%
0.0005
85.3810.44%
0.05
84.52+00.43%
INSGD-&
0.05
84.49:10.53%
SGD
0.01
79.90+00.21%
Adam
0.0005
85.13::0.45%
INSGD-
x
0.05
83.50++0.18%
INSGD-
0.05
83.32++0.48%
We observed that the INSGD optimizer can achieve convergence and results close to optimal when training the Vision Transformer (ViT). However, when trained with a warm startup, Adam outperforms both INSGD and SGD. This suggests that while INSGD can effectively handle the training of ViT models and achieve competitive results, Adam may offer superior performance under certain conditions, particularly when the learning rate is scheduled with warm restart. Further investigation into the interplay between optimizer choice, warm startup strategies, and model archilecture could provide deeper insights into the optimal training procedures for VIT models.
4.2 CIFAR-100 Experiment
We further extend our research by conducting experiments the CIFAR-100 dataset. CIFAR-100 is more challenging datasct compared to CIFAR-10 as it contains 100 classes instead of 10, requiring models to have a higher level of discrimination and classification capability. The increased class diversity in CIFAR-100 poses additional difficulty in achieving high accuracy and generalization performance. To ensure adequate representation of cach class the training process, we opted incrcase the batch size to 256 for this particular experiment. Before our study, Wide ResNet-18 was recognized for its convergence capabilities and satisfactory results Zagoruyko & Komodakis (2016) In alignment with the settings outlined in the Wide
13

---

Published in Transactions on Machine Learning Research (08/2024)
ResNet paper, replaced the optimizer algorithm with INSGD. Similar to our CIFAR-10 experiment, the model was trained for 200 epochs, and we report the highest accuracy achieved on the testing data.
Table 9: Accuracy results of the Wide ResNet-18 on the CIFAR-100 dalaset.
Optimizer
LR
Batch
Top-1 Acc.
Top-5 Acc.
SGD
0.1
128
78.24+0.43%
94.31=0.09%
INSGD-&
0.1
128
78.08:00244%
94.34 1.7%
INSGD-&2
0.1
128
78.47=0.16%
94.39-10.2%%
SGD
0.1
256
77.22:10.43%
93.79-10.437
INSGD-&1
0.1
256
78.15=0.4335
94.54+0.433
INSGD-(2
0.1
256
77.89+00.43%
93.98++0.43%
The results presented in Table g provide compelling evidence of the effectiveness of the INSGD algorithm in achieving improved convergence on complex datasets. The superior performance of INSGD, as evidenced by higher Top-1 and Top-5 accuracy, establishes utility training sophisticatod models on challenging datasets. These findings underscore the algorithm's capability handle intricate data distributions and optimize model performance, thereby showcasing its potential for advancing the state-of-the-art in deep learning.
WResNet-18 training CIFAR-100
INSGD_L _100__1-- Acc: %78.15=2043
3.5
INSGD_L _2 Acc: %677.89+0.4
SGD Acc: %77.2220.43
3.0
Test Loss
2.0
1.5
25
50
75
100 Epoch
125
150
175
200
Figure 4: Optimizer performance over 200 epochs: Comparing test loss convergence across different
optimizers.
4.3 ImageNet- -K Results
In this section, we present the test accuracy results on the ImageNet- 1K dataset. We utilize the ResNet-50 model, discussed in Scction 3.2 The training process is conducted using the official Py Torch ImageNet- -K training code Ima (002)). Specifically, we employ the SGD and INSGD optimizers with a weight decay of 0.0001 and momentum of 0.9.
14

---

Published in Transactions on Machine Learning Research (08/2024)
The ImageNet-1K dataset consists of 1.2 million images and is known for its difficulty in training. Due to the image resolution and resource constraints, adopting larger batch sizes is not feasible in our environment. As result, we train the models with a mini-batch size of 256, an initial learning rate of 0.1 for 90 epochs, and a learning rate reduction of 1/10 after every 30 epochs. Due to resource availability, we repeated the experiment times with different seeds.
To augment the data, we perform random cropping and horizontal flipping with a probability of 0.5, resulting in 224 x 224 images. The images are then normalized using mean of [0.485, 0.456, 0.406] and standard deviation of [0.229, 0.224, 0.225].
The accuracy of the best models is presented in Table 10 based on the center-crop top-1 accuracy and top-5 accuracy on the ImageNet-IK validation dataset. These accuracies are obtained from the model with the highest center-crop top-1 accuracy, providing a comprehensive evaluation of the model's performance on the ImageNet-1K dataset.
Table 10: Accuracy results of ResNet-50 on the ImageNet- -K dataset.
Optimizer
Learning Rate
Top-1 Acc.
Top-5 Acc.
Adam
0.001
66.71++0.08% 75.60++0.14%
87.65=0.15%
SGD
0.1
92.67=0.12%
INSGD-
0.1
75.92:10.43%
92.75=0.10%
INSGD-
0.1
75.90:+0.14%
92.81=0.04%
The results presented in Table 10 highlight the improved top-1 accuracy achieved by the INSGD algorithm on the ImageNet-1KK dataset. This improvement is particularly significant considering the scale of the dataset, demonstrating the effectiveness of INSGD in handling large and complex datasets. By leveraging the input normalization factor, INSGD enables the model to converge more effectively by aligning the gradient direction and appropriate magnitude. Using different seeds not only confirms but also strengthens the evidence of improvement brought by the INSGD algorithm. It essential to know that Adam may perform better training ResNet-50 however, it looks like similar settings with the INSGD and SGD, it doesn't achieve the same performance.
The power estimation obtained through momentum in INSGD indicates that the optimization algorithm can benefit from considering the entire input sequence. It suggests that the algorithm can capture long- term dependencies and utilize them for better optimization performance. Furthermore, it is worth noting that the batch size used in our experiments is relatively small compared to the number of images in the dataset. Exploring the algorithm's behavior with larger batch sizes would be an interesting avenue for future investigation.
5 Conclusion
In this paper, we proposed a novel neural network training method called INSGD, which incorporates ideas from the widely used NLMS algorithm in adaptive filtering. INSGD introduces a normalization step to the weight update term that normalizes the update term using only the input vector the neurons. The normalization can be performed using both the and la norms.
To evaluate the effectiveness of INSGD, conducted experiments various datasets using different mod- cls. Notably, our algorithm consistently demonstrated improvements in testing accuracy across multiple datasets. For example, on the CIFAR-10 dataset, INSGD achieved a significant boost in accuracy compared to traditional stochastic gradient algorithms. We observed similar positive outcomes on other datasets, such as CIFAR-100 and ImageNet- 1K, when employing different models like ResNet-20 20 and ResNet-50.
Traditional optimization algorithms often lack flexibility when it comes to selecting hyperparameters, which limit their effectiveness. However, the INSGD (Input Normalized Stochastic Gradient. Descent) algorithm
15

---

Published in Transactions on Machine Learning Research (08/2024)
offers solutions to this limitation by leveraging imput normalization. By normalizing the imput data, INSGD enables greater flexibility in tuning hyperparameters, leading to more robust and stable performance.
The promising results obtained across diverse datasets and models validate the effectiveness of INSGD in enhancing the training process. By incorporating the normalization factor into the stochastic gradient algorithm, INSGD effectively leverages the benefits of the NLMS algorithm, leading to improved performance in various object recognition scenarios. The NLMS algorithm is frequently applied in real-time processing scenarios, where the input signal exhibits temporal variations. An area worthy of exploration involves investigating the online learning mechanisms of the INSGD algorithm in such dynamic environments.
Acknowledgments
A. E. Cetin was partially supported by NSF 2303700 and 2229659.
References
Sleepest Cleepest-Descenl Descent. Technique, chapter 8, pp. 138 147. John Wiley & Sons, Ltd, 2008. ISBN 9780470374122. 10. doi: http:///00000/000000//////001110 URL https://oulineeibrrayy.woo wtw.://rwrwwwilll.l.oooooo///ooa com com/doi/abs/ 10. 10.1002/9970447774122..o
Imagenct training in pytorch. http:///gtthtoooooo////////////a 2022. Ac- cessed: 2022-12-27.
O. Arikan, A. Enis Cetin, and E. Erxin. Adaptive filtering for non-gaussian stable processes. IEEE Signal Processing Letters, 1(11):163 -168,,65,--111 1994. doi: 10.1109/97.335000
O. Arikan, M. Belge, A.E. Cetin, and E. Erzin. Adaptive filtering approaches for non-gaussian stable processes. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 2, pp. 1400-1400) vol.2, 1995. doi: 10. 10.110//CAASPP.1995..0
G. Aydin, O. Arikan, and A.E. Cetin. Robust adaplive filtering algorithms for /spl alpha/-stable random processes. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 46(2): 198 202, 1999. doi: 111109982.75220
Lóon Bottou. Largo-scale machine learning with stochastic gradient descent. In Proceedings of COMP- 2TAT'2010: 19th International Conference on Computational StatisticsPoris France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177-186. Springer, 2010.
A Enis Cetin, Omer N Gerek, and Yasemin Yardimci. Equiripple fir filter design by the fft algorithm. 1EEE Signal Processing Magazine, 14(2):60 64, 1997.
A Enis Cetin, Alican Bozkurt, Osman Gunay, Yusuf Hakan Habiboglu, Kivane Kosc, Ibrahim Onaran, Mohammad Tofighi, and Rasim Akin Sevimli. Projections onto convex sets (pocs) based optimization by lifting. In 2013 IEFE Global Conference Signal and Information Processing, pp. 623 623. IEEF, 2013.
SC Chan and Yi Zhou. Convergence behavior of nlms algorithm for gaussian inputs: Solutions using gen- eralized abelian integral functions and step size selection. Journal of Signal Processing Systems, 59(3): 255-265, 2010.
Patrick L Combettes. The foundations of set theoretic estimation. Proceedings of the IEFE, 8122):182-208 1993.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1 11929, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint XrXi:: 026777,7 2017.
Osman Gunay, Behçet Ugur Toreyin, Kivane Kose, and A. Enis Cotin. Entropy-functionallannn online adaptive docision fusion framework with application wildfire detection video. IEEE Transactions on Image Processing, 21)5):2855 2865, 2012. doi: 20100//T.2001.2111111
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Geoffrey E Hinton and Ruslan R Salakhutdinor. Reducing the dimensionality of data with neural networks. science, 313(5786):504-507, 2006.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chi Shang-Chihh Chen, Bo Chen, Mingxing Tan; Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1314-1324, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint arXiv:1412.6900 2014.
Alex Krizhevsky, Πla Sutskever, and Geoffrey Hinton. Imagenct classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90) 2017.
Quoc V Te, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y Ng. On opti- mization methods for deep learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 263-272, 2011.
Yann LeCun, D Touresky, Hinton, and T Sejnowski. theoretical framework for back wack-propagation. Proceedings of the 1988 connectionisl models summer school, volume 1, pp. 21 21-28, 1988.
Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 (10):1995, 1995.
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics, pp. 983 992. PMLR, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0300 03983, 2016.
V John Mathows and Zhenhua Xic. A stochastic gradient adaptive filter with gradient adaptive step size. IEEE Imnsacttons on Signal Processing, (1(6))2075 2087, 1993.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXh::1609.04747. 2016.
David F. Rumelhart, Geoffrey E. Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6088)::33 536, 1986.
Ali H. Sayed. Normalized LMS Algorithm, chapter 11, pp. 178-182. John Wilcy & Sons, Ltd, 2008 URL https://onllindlieerry
ISBN 9780440874122. doi: https: http://www....p./////oooooooo//a wi htt.ppyyyooaaa/a//////ooooooooo.
17

---

Published in Transactions on Machine Learning Research (08/2024)
Muhammed O. Sayin, N. Denizcan Vanli, and Suleyman Serdar Kozat. A novel family of adaptive filtering
algorithms based on the logarithmic cost. IEEE Transactions on Signal Processing, 62(17):44114442, 2014. doi:
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXi:: 1556, 2014.
Bharat Singh, Soham De, Yangmuzi Zhang, Thomas Goldstein, and Gavin Taylor. Layer-specific adaptive learning rates for deep networks. In 2015 IEEE 14th International Conference on Machine Learning and Applicalions (ICMLA), pp. 364-368. IEEE, 2015.
Sergios Theodoridis, Konstantinos Slavakis, and Isao Yamada. Adaplive learning in a world of projections. IEEE Signal Processing Mogazine, 28(1):97-123, 2010.
H. Trussell and M Civanlar. The leasible solution in signal restoration. IEEE Transactions on Acoustics, Speech, and Signal Processing, 32(2):201 212, 1984.
Bernard Widrow, Marcian E Hoff, al. Adaptive switching circuits. In IRE WESCON convention record, volume 4, pp. 96-104. New York, 1960.
Isao Yamada, Konstantinos Slavakis, and Kenyu Yamada. An efficient robust adaptive filtering algorithm based on parallel subgradient projection techniques. IEEE Transactions on Signal Processing, 50(5): 1091-1101, 2002.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprinl arXiv:1708.0888 03888, 2017.
Sergey Zagoruyko and Nikos Komodakis, Wide residual networks. arv preprint arXtc: 07146, 2016.
18


Reviews:
[{"review_id": "Artdt14s7j", "reviewer": "Reviewer_xCFE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is supported by extensive numerical experiments and provides a comprehensive review of related works.\n\nWeakness: While the numerical experiments demonstrate some promise for the proposed method, the core concept of adaptively changing the learning rate has already been extensively explored in existing literature. I observed no significant advantages of the proposed algorithm and a noticeable lack of theoretical analysis, even in a basic two-layer neural network. Additionally, the paper does not provide clear guidelines on when to apply $\\ell_1$-SGD versus $\\ell_2$-SGD, or how to select $\\beta$ in INSGD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an adaptive gradient descent algorithm in which the learning rate linearly depends on the $\\ell_1$ or $\\ell_2$ normalization of inputs to the neuron weights for training deep learning models. Numerical results demonstrate that the corresponding $\\ell_1$-SGD/$\\ell_2$-SGD, or an advanced version that memorizes the past input norm (INSGD), slightly outperforms vanilla SGD across various neural network architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1) There appears to be a typo in Equation (2) and Figure 2; the \"+\" sign in the equations should be a \"-\".\n\n(2) The captions for tables should be placed above the tables.\n\n(3) To be honest, I am skeptical that INSGD performs universally better than vanilla SGD. It is crucial to characterize the conditions under which INSGD can perform well, rather than solely reporting numerical results. \n\n(4) Techniques such as Newton's method utilize approximations of the Hessian matrix to adaptively select the learning rate. I am interested in learning the authors' discussion of INSGD with these methods.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z7vqULt2JX", "reviewer": "Reviewer_a8QD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the logic is mostly clear. The idea is interesting. Most claims in the paper are verified. I think there would be audience interested in the proposed method. One drawback is that the experiments are constrained to medium-sized CV tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel optimization algorithm called Input Normalized Stochastic Gradient Descent (INSGD), which is inspired by the Normalized Least Mean Squares (NLMS) algorithm used in adaptive filtering. The proposed method show improvements over SGD and Adam in some medium-sized experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.How does INSGD help \"Robustness to variability\"? The authors mentioned this as a motivation but did not show how it is resolved. Please verify this claim.\n\n2. Why do need extra normalization in INSGD when we have BatchNorm in the architecture? Need discussion.\n\n2. How would INSGD perform on non-CNN tasks, like ViT? Or on non-CV tasks such as language tasks?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fxA0CaYyNu", "reviewer": "Reviewer_KDbw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors introduce a novel algorithm for training deep neural networks.\n\nWeaknesses: I feel that the paper's overall writing quality is subpar and lacks clarity. Several weaknesses can be identified:\n\nPresentation and Clarity: The presentation of the paper is found to be lacking. In Section 1.2, the explanation is notably poor. The problem is redundantly defined in equations (3), (4), and (5) without subsequent utilization in the text. The introduction of the NLMS could be more succinct since a substantial amount of space is dedicated to explaining a single scheme. Furthermore, the authors introduce the update equation (24) with excessive detail, yet it is not incorporated into Algorithm 1, resulting in an overall confusing presentation.\n\nTypographical Errors: The text contains various typographical errors. For instance, in equation (2), the minus sign ' - ' should be corrected to a plus sign ' + '. In equation (5), the variable 'w' should be represented as ${\\bf w}$. Within Algorithm 1, the symbol $\\tau$ should be replaced with $\\gamma$. Additionally, in equation (28), $e_k$ should be altered to $w_k$.\n\nContribution and Theory: From my perspective, the main contribution of the paper appears to be equation (28). The authors do not adequately present the theoretical underpinnings of the paper. Therefore, it is suggested that they focus more on the numerical results. However, I believe the numeric results are inadequate as  Transformer is not included for comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel optimization algorithm called Input Normalized Stochastic Gradient Descent (INSGD), inspired by the NLMS algorithm in adaptive filtering. The authors argue that INSGD enhances training performance for machine learning models by applying $L_1$ and  $L_2$  based normalizations to the learning rate. Experimental results show that the INSGD algorithm outperforms existing methods, improving accuracy levels on various benchmark datasets and models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see Strengths And Weaknesses", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There were many typos and the presentation suffered quite a lot in the initial submission. I understand the authors had to spend a lot of time on the extra experiments and couldn't have done a major writing revision during the discussion period. After a quick look, it appears there are still many small issues in the paper: the equation in the right side of Figure 2 uses $w_{\mathbf{y}_0}(i)$ ($w$ instead of $\mathbf{w}$); Figure 3, Figure 4, and Table 11 have missing punctuation; equation (28) wrote $log(P_k)$ instead of $\log(P_k)$; strange phrases like "by Singh et al. in Singh et al. (2015)" (why not just write "by Singh et al. (2015)"?); Table 11's top row is not bold whereas it is in the other tables; the definition of $e_k$ on page 6 is not bold, whereas everywhere else it's written as $\mathbf{e}_k$; $v(k)$ for some reason is not bold.  
I'm also a bit confused as to why the authors reported only a single random seed in Figure 4. It'd be much better to provide the average over multiple seeds with confidence intervals. I also think it'd be nice to add similar figures for at least two other experiments that are not on CIFAR-10.

I hope the authors will take their time to proofread the paper, make the necessary changes, and submit the final version.
Decision: Accept (poster)