# Ensemble of Networks for Multilabel Classification
## 5 Optimizers
### DiffGrad Variants  
These are the variants of the diffGrad optimization method used in generating the ensembles:  
* DGrad [68] is based on the moving average of the element-wise squares of the parameter gradients;
* Cos#1 [68] is a minor variant of DGrad based on the application of a cyclic learning rate [70];
* Exp is a variant of a method proposed in [69] that is based on the application of an exponential function;
* Sto is a stochastic approach that stalls the optimizer on flat surfaces or small wells.  
The proposed approaches have different methods for defining \(\xi_{t}\), followed by the application of Equation (28) to calculate the final update for \(\theta_{t}\).  
The rationale for all these variants is to avoid optimizer stalling on flat surfaces due to the low value of \(\Delta g_{t}\).  
DGrad [68] differs from diffGrad by defining the absolute difference between two consecutive steps of the gradient as  
\[\Delta ag_{t}=\lvert g_{t}-avg_{t}\rvert, \tag{29}\]  
where \(avg_{t}\) contains the moving average of the element-wise squares of the parameter gradients; we then normalize \(\Delta ag_{t}\) by its maximum as  
\[\Delta a\hat{g}_{t}=\left(\frac{\Delta ag_{t}}{\max(\Delta ag_{t})}\right) \tag{30}\]  
With DGrad, \(\xi_{t}\) is defined as the Sigmoid of \(4\cdot\Delta a\hat{g}_{t}\). The rationale for multiplying by four is to exploit the range of the output of the sigmoid function [68]:  
\[\xi_{t}=Sig(4\cdot\Delta a\hat{g}_{t}) \tag{31}\]  
Cos#1 [68] is a variant of DGrad with a cyclic learning rate. The idea is to improve classification accuracy without tuning and with fewer iterations [17]. In this work, the \(cos()\) periodic function is selected to define a range of variation in the learning rate:  
\[lr_{t}=\left(2-\left\lvert\cos(\frac{\pi\cdot t}{steps})\right\rvert e^{-0.01 \cdot(mod(t,steps)+1)}\right), \tag{32}\]  
where \(mod()\) denotes the function modulo and where \(steps=30\) is the period [68]. The plot of \(lr_{t}\) for \(t\) in the range 1:2 \(\times\)\(steps\) is reported in Figure 2.  
With Cos#1, \(lr_{t}\) is used as a multiplier of \(\Delta ag_{t}\) in the definition of \(\xi_{t}\) (Equation (31)):  
\[\xi_{t}=Sig(4\cdot lr_{t}\cdot\Delta a\hat{g}_{t}) \tag{33}\]  
Exp [69] performs the element-wise operations of product and exponential. Exp is designed to reduce the impact of large variations in the gradient and help the function converge for small values. The function in Equation (34) decays slower than the negative exponential for larger values. It also provides less focus on gradient variations that tend to zero. A larger area of greater gain is thus produced [69]:  
\[lr_{t}=\Delta ag_{t}\cdot e^{(-k\cdot\Delta ag_{t})}, \tag{34}\]with \(k\) set to 2 in our experiments.  
The final learning rate (\(\zeta_{t}\)) is the normalized result multiplied by a correction factor (1.5), which moves the mean towards the unit as illustrated in Figure 3:  
\[\xi_{t}=1.5\frac{lr_{t}}{\max(lr_{t})} \tag{35}\]  
Sto (short for Stochastic), proposed here, adds noise to the learning rate to reduce the tendency to stall on flat surfaces or small wells. The noise is independent of the gradient direction.  
Let \(\mathcal{X}\) be a matrix of independent uniform random variables in range \([0,1]\) and \(J\) an all-ones matrix:  
\[\mathcal{X}=\begin{bmatrix}X_{1,1}&\cdots&X_{1,n}\\ \vdots&\ddots&\vdots\\ X_{m,1}&\cdots&X_{m,n}\end{bmatrix}\text{ and }J=\begin{bmatrix}1&\cdots&1\\ \vdots&\ddots&\vdots\\ 1&\cdots&1\end{bmatrix},\]  
where \(X_{i,j}\sim\mathcal{U}(0,1)\) are random variables with a uniform density function. The learning rate is defined as  
\[lr_{t}=\Delta qg_{t}\cdot e^{(-4\cdot\Delta qg_{t})}\cdot(\mathcal{X}+0.5 \cdot J) \tag{36}\]  
\[\xi_{t}=1.5\frac{lr_{t}}{\max(lr_{t})} \tag{37}\]  
In Equation (36), matrix \(J\) shifts the range of \(X\) by 0.5 to move the mean across \(1\).