Question: How exactly was the memory of Hessian in L-BFGS and ESGD recorded?
Evidence: Observation on loss decrease. To study the convergence of Kernel SGD, we first illustrate the trajectories of training loss in Figure. As shown in the figure, L-BFGS gets stuck at the saddle points or local minimum in the early stage. ESGD helps escape the saddle points but fails to converge to a better loss on the tabular and text data. Kernel SGD reaches the stable point in a small number of epochs while producing the lowest loss in all types of tasks. This indicates that Kernel SGD has the ability to escape the saddle points and can find a better solution in the non-convex problems. Analysis on generalization. Here, we compare our method with second-order optimization to evaluate the generalization performance. The third and fourth columns of Table list the training and test accuracy achieved by the best models of each optimizer. The best model was selected according to the highest validation accuracy. The values of selected learning rates are available in the supplementary material. Our method shows a remarkable generalization performance and can achieve the highest test accuracy on all the tasks. Especially on IMDb, Kernel SGD improves the test accuracy by around 28%. Moreover, Kernel SGD can mitigate overfitting and achieves a stable accuracy, which is demonstrated by the relatively small gaps between the test accuracy and training accuracy and the small variances. Note that we did not adopt pre-processing techniques such as random flipping on the tested data for fair comparison and thus the accuracy in Table may be slightly different from those shown in other studies.
Analysis on convergence speed. We tested the convergence speed and recorded the convergence time in the fifth column of Table. Kernel SGD converges up to 30 times faster than second-order optimization baselines. For Cov-tw and IMDb, L-BFGS converges faster because it stops too early with a relatively large loss and underfits the problems as shown in Figure and Table. Kernel SGD takes more epochs to converge to a better loss and thus needs more convergence time.
Analysis on memory cost. The memory for the Hessian matrix is represented in the sixth column of Table. We show the size of the whole kernel matrix in Kernel SGD which is an n × m matrix.
In L-BFGS and ESGD, the Hessian matrix of neural networks are not explicitly computed. For L-BFGS, we recorded the memory consumption for storing the historical weights and gradients. In ESGD, we recorded the memory of the preconditioning matrix. Kernel SGD uses much smaller memory to store the kernel matrix in larger neural networks such as ResNet-18 and LSTM, for the computation is based on the training instances rather than the weights in the networks.
Sanity check. Although our main aim in this paper is to improve the second-order optimization methods, for a sanity check and for completeness, we further compare Kernel SGD with first-order optimizers which are mini-batch SGD and SGD with momentum (SGD+M). The parameter momentum was set as 0.9 for SGD+M. The results are shown on the last two columns of Table. Our method still outperforms the first-order optimizers in terms of generality on most tasks (i.e., 5 out of 6 tasks in total). Our method converges even faster than the first-order optimizers with image data.
Analysis on memory cost. The memory for the Hessian matrix is represented in the sixth column of Table 2. We show the size of the whole kernel matrix in Kernel SGD which is an n × m matrix. In L-BFGS and ESGD, the Hessian matrix of neural networks are not explicitly computed. For L-BFGS, we recorded the memory consumption for storing the historical weights and gradients. In ESGD, we recorded the memory of the preconditioning matrix. Kernel SGD uses much smaller memory to store the kernel matrix in larger neural networks such as ResNet-18 and LSTM, for the computation is based on the training instances rather than the weights in the networks.
Answer: For L-BFGS, we recorded the memory consumption for storing the historical weights and gradients used in L-BFGS. In ESGD, we recorded the memory of the preconditioning matrix. We have added this explanation in the fourth paragraph of Section 4.2.