- SGD/Adam/AdamW: For the NMT task, we report the results of SGD from AdaHessian [34], and search learning rate among  $\{5\mathrm{e} - 5,1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3\}$  and epsilon in  $\{1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8,1\mathrm{e} - 6,1\mathrm{e} - 4\}$  for Adam/AdamW, and for both optimizers the optimal learning rate/epsilon is  $5\mathrm{e} - 4 / 1\mathrm{e} - 12$  . For the LM task, we follow the settings from AdaBelief [39], setting learning rate to 30 for SGD and 0.001 for Adam/AdamW while epsilon to 1e-12 for Adam/AdamW when training 1-layer LSTM. For 2-layer LSTM, we conduct a similar search with learning rate among  $\{1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3,1\mathrm{e} - 2\}$  and epsilon within  $\{1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8,1\mathrm{e} - 6,1\mathrm{e} - 4\}$  , and the best parameters are learning rate  $= 1\mathrm{e} - 2$  and epsilon  $= 1\mathrm{e} - 8 / 1\mathrm{e} - 4$  for Adam/AdamW. For 3-layer LSTM, learning rate  $= 1\mathrm{e} - 2$  and epsilon  $= 1\mathrm{e} - 8$  are used for Adam/AdamW.
- AdaBelief: For the NMT task we use the recommended configuration from the latest implementation<sup>4</sup> for transformer. We search learning rate in  $\{5\mathrm{e} - 5,1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3\}$  and epsilon in  $\{1\mathrm{e} - 16,1\mathrm{e} - 14,1\mathrm{e} - 12,1\mathrm{e} - 10,$ $1\mathrm{e} - 8\}$ , and the best configuration is to set learning rate as  $5\mathrm{e} - 4$  and epsilon as  $1\mathrm{e} - 16$ . We adopt the same LSTM experimental setup for the LM task and reuse the optimal settings provided by AdaBelief [39], except for 2-layer LSTM, where we search for the optimal learning rate in  $\{1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3,1\mathrm{e} - 2\}$  and epsilon in  $\{1\mathrm{e} - 16,1\mathrm{e} - 14,1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8\}$ . However, the best configuration is identical to the recommended.
- AdaHessian: For the NMT task, we adopt the same experimental setup as in the official implementation. For LM task, we search the learning rate among  $\{1\mathrm{e} - 3,1\mathrm{e} - 2,0.1,1\}$  and hessian power among  $\{0.5,1,2\}$ . We finally select 0.1 for learning rate and 0.5 for hessian power for 1-layer LSTM, and 1.0 for learning rate and 0.5 for hessian power for 2,3-layer LSTM. Note that AdaHessian appears to overfit when using learning rate 1.0. Accordingly, we also try to decay its learning rate at the 50th/90th epoch, but it achieves a similar PPL.
- AGD: For the NMT task, we search learning rate among  $\{5\mathrm{e} - 5,1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3\}$  and  $\delta$  among  $\{1\mathrm{e} - 16,1\mathrm{e} - 14,$ $1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8\}$ . We report the best result with learning rate  $5\mathrm{e} - 5$  and  $\delta$  as  $1\mathrm{e} - 14$  for AGD. For the LM task, we search learning rate among  $\{1\mathrm{e} - 4,5\mathrm{e} - 4,1\mathrm{e} - 3,5\mathrm{e} - 3,1\mathrm{e} - 2\}$  and  $\delta$  from  $1\mathrm{e} - 16$  to  $1\mathrm{e} - 4$ , and the best settings for learning rate  $(\delta)$  is  $5\mathrm{e} - 4$  (1e-10) and  $1\mathrm{e} - 3$  (1e-5) for 1-layer LSTM (2,3-layer LSTM).