# TD-DNN: A Time Decay-Based Deep Neural Network for Recommendation System
## 4 Proposed Methodology
### Constructing Deep Neural Network
#### 4.2.4 Training Process
The Adam optimizer [11] is an extension of stochastic gradient descent which leverages the power of adaptive learning rates methods to find individual learning rates for each parameter. Its name comes from the fact that it adapts the learning rate for each neural network weight using estimations of the first and second moments of gradient descent. Apart from these, some fractional calculus-based new optimizers have been introduced recently, i.e., Fractional Stochastic Gradient Descent, Moving Information-based Stochastic Gradient Descent, and Normalized Fractional Stochastic Gradient Descent. The Fractional Stochastic Gradient Descent (FSGD) [72] is implemented using fractional calculus ideas for efficient matrix factorization. A fast and accurate form of SGD called Moving Information-based Stochastic Gradient Descent (MISGD) [73] is introduced to successfully address the fuzzy nature of preferences by capturing the collective effect of ratings from previous update histories. The Normalized Fractional Stochastic Gradient Descent optimizer (NF-SGD) [74] is a normalized form of F-SGD with a time-varying learning rate. It adapts the learning rate and delivers quick recommendations based on users' preferences. Among the most popular optimizers, the Adam optimizer is [75] is simple to implement, computationally efficient, uses little memory, and can handle massive data sets and parameters; hence, it is used in developing the model. The Adam optimizer combinestwo gradient descent methodologies [76]: Momentum and RMSProp, with Momentum providing smoothing and RMSProp efficiently changing the learning rate. The weight and bias update in Momentum are given by:  
\[w=w-\alpha\cdot v_{dw}b=b-\alpha\cdot v_{db} \tag{8}\]  
The weight and bias update in RMSProp are given by:  
\[w=w-\alpha\cdot\left((\partial\mathrm{L}/\partial\mathrm{w})/\sqrt{s_{dw}+ \varepsilon}\,\right)b=b-\alpha\cdot\left((\partial\mathrm{L}/\partial \mathrm{b})/\sqrt{s_{db}+\varepsilon}\,\right) \tag{9}\]  
where \(w\) and \(b\) are the weights and bias, respectively, \(v_{dw}\) is the sum of gradients of weights at time \(t\), \(v_{db}\) is the sum of gradients of bias at time \(t\), \(s_{dw}\) is the sum of square of past gradients of weights, \(s_{db}\) is the sum of square of the past gradients of bias, \(\partial\mathrm{L}\) is the derivative of loss function, \(\partial\mathrm{w}\) is the derivatives of weight at time \(t\), and \(\partial\mathrm{w}\) is the derivative of bias. Initially, \(v_{dw}\), \(s_{dw}\), \(v_{db}\) and \(s_{db}\) all are set to zero. On iteration \(t\), they are computed as follows. \[ \begin{array}{l}v_{dw}=\beta_{1}\,v_{dw_{t-1}}+(1-\beta_{1})\,\frac{ \partial\mathrm{L}}{\partial\mathrm{w}}\\ v_{db}=\beta_{1}\,v_{dh_{t-1}}+(1-\beta_{1})\,\frac{\partial\mathrm{L}}{ \partial\mathrm{b}}\\ s_{dw}=\beta_{2}\,v_{dw_{t-1}}+(1-\beta_{2})\,\left(\frac{\partial\mathrm{L}}{ \partial\mathrm{w}}\right)^{2}\\ s_{db}=\beta_{2}\,v_{dh_{t-1}}+(1-\beta_{2})\,\left(\frac{\partial\mathrm{L}}{ \partial\mathrm{b}}\right)^{2}\end{array}  \tag{10}\]  
In Momentum and RMSProp, initially, \(v_{dw}\), \(s_{dw}\), \(v_{db}\) and \(s_{db}\) were initialized to 0, but it is observed that they become 0 when both \(\beta_{1}\) and \(\beta_{2}\) are near to 1. The Adam optimizer [76] fixed this issue by using the corrected formula as given below. This is also completed to keep the weights under control when they approach the global minimum. The corrected formulas for \(v_{dw},s_{dw}\), \(v_{db}\) and \(s_{db}\) are given below:  
\[ \begin{array}{l}v_{dw}^{corrected}=\frac{v_{dw}}{1-\beta_{1}^{\intercal}}\\ v_{dw}^{corrected}=\frac{v_{dw}}{1-\beta_{1}^{\intercal}}\\ s_{dw}^{corrected}=\frac{s_{dw}}{1-\beta_{2}^{\intercal}}\\ s_{db}^{corrected}=\frac{s_{db}}{1-\beta_{2}^{\intercal}}\end{array}  \tag{11}\]  
Put these correct values in Equation (12) to compute the weight and bias in Adam optimizer. \[