# Deep Convolutional Neural Networks Object Detector for Real-Time Waste Identification
## 2 Methods and Dataset
### Methods
#### 2.3.3 Determining the Appropriate Learning Rate for Training our CNN Model  
Learning rate is a key hyper-parameter for training neural networks. Finding the appropriate value for each application is very important in order to achieve high accuracy without overfitting the model. The weights are updated in relation to the learning rate after each batch in the direction of the minimum loss. In order to find the optimal higher and lower bounds of our learning rate, where the network is actually learning, we trained our model for several epochs, updating the learning rate after each batch starting from \(1\times 10^{-10}\) and increasing it until reaching \(1\times 10^{-1}\). We noticed that between \(1\times 10^{-10}\) and \(1\times 10^{-7}\), the loss is around the value of 40 and does not change too much, thus the learning rate is too small for the network to learn. Starting at approximately \(1\times 10^{-6}\), the loss starts to decrease. This value is the smallest learning rate where our model can actually learn. By the time we hit \(1\times 10^{-5}\), the loss drops rapidly, meaning that the network is learning at a fast pace and continues in the same way until the learning rate is \(1-10^{-1}\). After this threshold, the loss starts to converge until the training stops. One way of setting the appropriate learning rate is to use the cyclic learning rate [33]. Cyclical Learning Rates enable our learning rate to oscillate back and forth between a lower and upper bound. Thus, the network is able to come out from either saddle points or local minima, while low learning rates may not be sufficient to break out and descend into areas with lower losses. Furthermore, the model and optimizer may be very sensitive to the initial learning rate.  
In our case, it turned out that the model was learning between \(1\times 10^{-1}\) and \(1\times 10^{-6}\), with a good rate between \(1\times 10^{-2}\) and \(1\times 10^{-5}\). Taking this into consideration, an exponential learning rate has been used, with the initial learning rate of \(1\times 10^{-2}\). Moreover, a stepping learning rate has been added to boost the training.  
\[l_{r}=l_{r(0)}\cdot k^{t/t_{k}}, \tag{1}\]  
where the initial learning rate, \(l_{r(0)}\), is \(1\times 10^{-2}\), the decay, \(k\), is 0.95, and the decay steps, \(t_{k}\), is 10,000. The training stops when the global step, \(t\), reaches 300 k or 500 k depending on the model.  
Loss optimization can be obtained by applying adaptive learning rate using several methods, such as Adaptive Moment Optimization (Adam) [PERSON] [34], Root Mean Square Propagation (RMSprop) [35], Adaptive gradient algorithm (Adagrad) [36], Adadelta [37], or Nesterov Adaptive Moment Optimization (Nadam) [38]. In our approach, finding the optimal learning rate limits has been carried out using RMSProp. This optimization was further used in the paper when training the object detector as well. Weights computation using the learning rate and RMSProp is presented in the equations below:  
\[v_{t}=\beta v_{t-1}+(1-\beta)g_{t}^{2}, \tag{2}\]  
\[w_{t}=w_{t-1}-l_{r}\frac{g_{t}}{\sqrt{v_{t}+\varepsilon}} \tag{3}\]  
where \(v_{t}\) is the moving average of the squared gradient, \(g_{t}^{2}\) is the gradient of the cost function with respect to the weight, \(w_{t}\) is the weight, and \(\beta\) is the moving average parameter (the value is usually 0.9).