# Biomedical Image Classification via Dynamically Early Stopped Artificial Neural Network
## 3 A New Dynamic Early Stopping Technique
### The Proposal
#### 3.1.4 Lines 17-29--Dynamic Early Stopping  
If the accuracy on the VS is not improved, a counter is increased (line 17). Subsequently (line 18), the value of the counter is compared to the prefixed value \(p\) of the patience. In standard early stopping strategies, if the counter is greater than the value of \(p\), then the training phase is stopped. On the contrary, in the suggested dynamic early stopping technique, the training phase is not immediately stopped, but it is allowed to continue with different hyperparameters. In particular, the LR is decreased by a factor \(c_{1}\in(0,1)\) (line 20) and/or the MBS is increased by a proper rule (line 22) depending on \(c_{2}\in[1,M]\) and \(dim\in\{0,\cdots,M\}\), where \(M\) is a constant related to hardware or memory limitations, for example, it can be the maximum number of samples which can be stored in the GPU. Finally (line 23), the value of the patience is divided by a factor \(\gamma>1\). Thanks to a smaller LR and/or a mini-batch of a larger size, the optimizer employed in the training phase should stabilize and provide new iterates closer to the minimum point. However, if the LR becomes too small and the MBS reaches the hardware limitations, then the ending of the training process is forced. In particular, if the patience is reduced more than a prefixed value \(\overline{p}\), then the training is stopped (lines 27-29).  
To summarize, different from the standard early stopping, the proposed one avoids a sharp ending of the training process and the difficult tuning of a fixed value of the patience.  
Moreover, it allows to really exploit the nature of the dynamic selection rules for the LR and MBS, thus ensuring a more efficient learning phase. Finally, we remark that the possibility to refine the values for the LR and the MBS along the iterative process allows to make their initial setting less crucial than in the static approaches where the hyperparameters are kept fixed during the training.  
```
1 Choose \(maxepochs\), \(x_{0}\), \(\eta_{0}\), \(\mathcal{S}_{0}\subset\mathcal{N}\) of cardinality \(n_{0}\), \(c_{1}\in(0,1)\), \(c_{2}\in[1,M]\), \(dim\in\{0,\dots,M\}\), \(\gamma\geq 1\), \(p\), \(\overline{p}\in\mathbb{N}\)
2 Initialize \(BestAcc=0\), \(p_{\text{rnd}}=0\)
3 for\(k=0,\dots,maxepochs\)do
4 while\(i\leq\frac{n}{n_{k}}\)do
5\(i=i+1\)
6 Select a mini-batch \(\mathcal{S}_{i}\) randomly and uniformly from \(\mathcal{N}\) of cardinality \(n_{k}\)
7 Compute a stochastic direction \(d_{i}\) on the mini-batch \(\mathcal{S}_{i}\)
8 Compute \(x_{i+1}=x_{i}-\eta_{i}\cdot d_{i}\)
9
10 end for
11 Save the parameters of the final model: \(x_{k}=x_{i+1}\) and evaluate the model on the VS: \(AccVal_{k}\)
12 if\(AccVal_{k}>BestAcc\)then
13\(counter=0\)
14\(p_{\text{rnd}}=0\)
15\(BestAcc=AccVal_{k}\)
16 Save the parameters of the best model: \(x_{best}=x_{k}\)
17 else
18\(counter=counter+1\)
19 if\(counter>p\)then
20\(counter=0\)
21 Learning rate decreasing: \(\eta_{k+1}=c_{1}\eta_{k}\)
22 and/or
23 Mini-batch size increasing: \(n_{k+1}=c_{2}n_{k}+dim\) or \(n_{k+1}=\max\{dim,c_{2}n_{k}\}\)
24\(p=\frac{p}{\gamma}\)
25\(p_{\text{rnd}}=p_{\text{rnd}}+1\)
26 end if
27
28 end for
29 if\(p_{\text{rnd}}>\overline{p}\)then
30 return
31 end if
32
33 end for
```  
**Algorithm 1**A stochastic gradient method with dynamical early stopping