# Deep Neural Networks Training by Stochastic Quasi-Newton Trust-Region Methods
## 3 Results
### Comparison with StORM  
We have performed a comparison between our sQN training algorithms and the algorithm STORM (Algorithm 5 in [47]). STORM relies on an adaptive batching strategy aimed at avoiding inaccurate stochastic function evaluations in the TR framework. Note that the real reduction of the objective function is not guaranteed in a stochastic-trust-region approach. In [32; 47], the authors claim that if the stochastic functions are sufficiently accurate, this will increase the number of true successful iterations. Therefore, they considered a progressive sampling strategy with sample size  
\[b_{k}=\min(N,\max(b_{0}\cdot k+b_{1},\lceil\frac{1}{\delta_{k}^{2}}\rceil)),\]  
where \(\delta_{k}\) is the trust region radius at iteration \(k\), \(N\) is the total number of samples, and \(b_{0},b_{1}\) are \(b_{0}=100\), \(b_{1}=32\times 32\times 3\) for CIFAR10, and \(b_{1}=28\times 28\times 1\) for Fashion-MNIST.  
We have applied STORM with both L-SR1 and L-BFGS updates. We have compared the performances of the sL-SR1-TR and sL-BFGS-TR algorithms employing different overlapping batch sizes running for 10 epochs with the performance provided by STORM with progressive batch size \(b_{k}\) running for 50 epochs. We allowed STORM to execute for more epochs, i.e., 50 epochs, since, due to its progressive sampling behavior it passed 10 epochs very rapidly. The largest batch size reached by STORM within this number of epochs is near \(b_{k}=25000\) (i.e., 50 percent of the total number of training samples).  
The results of this experiment are summarized in Figure 5. In both Fashion-MNIST and CIFAR10 problems, the algorithms with \(bs=500\) and \(bs=1000\) produce comparable or higher final accuracy than STORM at the end of their respective training phases. Even if we set a fixed budget of time corresponding to the one needed by STORM to perform 50 epochs, sL-QN-TR algorithms with \(bs=500\) and \(bs=1000\) provide comparable or higher accuracy. The results corresponding to the smallest and largest batch sizes need a separate discussion. When \(bs=100\), the stochastic QN algorithms are not better than STORM with any fixed budget of time; however, they provide higher final training accuracy and testing accuracy, except for the Fashion-MNIST problem on ResNet-20 trained by sL-BFGS-TR.  
By contrast, when \(bs=5000\), sL-BFGS-TR algorithms produce higher or comparable training accuracy but not a testing accuracy comparable to the one provided by STORM. This seems quite logical, as using such a large batch size causes the algorithms to perform a small number of iterations and then to update the parameter vector only a few times; allowing longer training time or more epochs can compensate for this lower accuracy. Finally, this experiment shows also that the sL-BFGS-TR algorithm with \(bs=5000\) yields higher accuracy within less time than that obtained when \(bs=100\) is used.  
Figure 5: The performance of sL-BFGS-TR and sL-SR1-TR with different fixed batch sizes (\(bs\)), in comparison to STORM. Left and right columns display the Training and Testing accuracies, respectively.