# Deep Neural Networks Training by Stochastic Quasi-Newton Trust-Region Methods
## 4 Conclusions  
We have studied stochastic QN methods for training deep neural networks. We have considered both L-SR1 and L-BFGS updates in a stochastic setting in a trust region framework. Extensive experimental work--including the effect of batch normalization (BN), the limited memory parameter, the sampling strategy, and batch size--has been reported and discussed. Our experiments show that BN is a key factor in the performance of stochastic QN algorithms and that sL-BFGS-TR behaves comparably to or slightly better than sL-SR1-TR when BN layers are used, while sL-SR1-TR performs better in networks without BN layers. This behavior is in accordance with the property of L-SR1 updates allowing for indefinite Hessian approximations in non-convex optimization. However, the exact reason for the different behavior of the two stochastic QN algorithms with networks not employing BN layers is not completely clear and would deserve further investigation. The reported experimental results have illustrated that employing larger batch sizes within a fixed number of epochs produces lower training accuracy, which can be recovered by longer training. Regarding training time, our results have also shown a slight superiority in the accuracy reached by both algorithms when larger batch sizes are used within a fixed budget of time. This suggests the use of large batch sizes also in view of the parallelization of the algorithms. The proposed sQN algorithms, with the overlapping fixed-size sampling strategy, revealed to be more efficient than the adaptive progressive batching algorithm STORM, which naturally incorporates a variance reduction technique. Finally, our results show that sQN methods are efficient in practice and, in some instances, outperform tuned Adam. We believe that this contribution fills a gap concerning the real performance of the SR1 and BFGS updates in realistic large-size DNNs, and it is expected to help steer the researchers in this field towards the option of the proper quasi-Newton method. The following supporting information can be downloaded at: [[https://www.mdpi.com/article/10.3390/a16100490/s1](https://www.mdpi.com/article/10.3390/a16100490/s1)]([https://www.mdpi.com/article/10.3390/a16100490/s1](https://www.mdpi.com/article/10.3390/a16100490/s1)). The following supporting information includes additional numerical results for different image classification problems, enhancing the comprehensive nature of our study. Figure S1: MNIST, LeNet-like: Accuracy and loss evolution vs. epoch. Figure S2: F-MIIST, LeNet-like: Accuracy and loss evolution vs. epoch. Figure S3: F-MIIST, ResNet-20: Accuracy and loss evolution vs. epoch. Figure S4: CIFAR10, ResNet-20: Accuracy and loss evolution vs. epoch. Figure S5: F-MNIST, ResNet-20 (No BN): Accuracy and loss evolution vs. epoch. Figure S6: CIFAR10, ResNet-20 (No BN): Accuracy and loss evolution vs. epoch. Figure S7: MNIST, ConvNet3 FC2: Accuracy and loss evolution vs. epoch. Figure S8: F-MNIST, ConvNet3 FC2: Accuracy and loss evolution vs. epoch. Figure S9: CIFAR10, ConvNet3 FC2: Accuracy and loss evolution vs. epoch. Figure S10: MNIST, ConvNet3 FC2 (No BN): Accuracy and loss evolution vs. epoch. Figure S11: F-MNIST, ConvNet3 FC2 (No BN): Accuracy and loss evolution vs. epoch. Figure S12: CIFAR10, ConvNet3 FC2 (No BN): Accuracy and loss evolution vs. epoch. Figure S13: MNIST and F-MNIST: Accuracy evolution vs. CPU time. Figure S14: CIFAR10: Accuracy evolution vs. CPU time. Figure S15: MNIST, LeNet-like: Comparison with tuned [PERSON]. Figure S16: F-MNIST, ResNet-20: Comparison with tuned [PERSON]. Figure S17: F-MNIST, ResNet-20 (No BN): ResNet-20: Comparison with tuned [PERSON]. Figure S18: CIFAR10, ConvNet3 FC2: Comparison with tuned [PERSON]. Figure S19: CIFAR10, ConvNet3 FC2 (No BN): Comparison with tuned [PERSON]. Conceptualization, A.M. and M.Y; methodology, A.M. and M.Y; software implementation, M.Y.