SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} \sqrt {\frac {\pmb {\nu} _ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}} \\ = \mathbb {E} \frac {\sigma_ {0} ^ {2} + \frac {\nu_ {0 , i}}{1 - \beta_ {2}}}{\sqrt {\frac {\nu_ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}}} + \sum_ {t = 1} ^ {T} \mathbb {E} \frac {| \boldsymbol {g} _ {t , i} | ^ {2}}{\sqrt {\frac {\nu_ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}}} \mathbf {1} _ {| \boldsymbol {G} _ {t, i} | <   \frac {\sigma_ {0}}{\sigma_ {1}}} \\ + \sum_ {t = 1} ^ {T} \mathbb {E} \frac {\left| \boldsymbol {g} _ {t , i} \right| ^ {2}}{\sqrt {\frac {\nu_ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} \left| g _ {s , i} \right| ^ {2} + \sigma_ {0} ^ {2}}} \mathbf {1} _ {\left| \boldsymbol {G} _ {t, i} \right| \geq \frac {\sigma_ {0}}{\sigma_ {1}}} \\ \leq \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 \sqrt {1 - \beta_ {2}} \sigma_ {1} ^ {2} \mathbb {E} \sum_ {t = 1} ^ {T} \frac {\left| \boldsymbol {G} _ {t , i} \right| ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} \mathbf {1} _ {| \boldsymbol {G} _ {t, i} | \geq \frac {\sigma_ {0}}{\sigma_ {1}}} \\ \leq \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 \sqrt {1 - \beta_ {2}} \sigma_ {1} ^ {2} \mathbb {E} \sum_ {t = 1} ^ {T} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}}. \\ \end{array}
$$
Secondly, as  $\beta_{2} \to 1$  as  $T \to \infty$ ,  $\beta_{1} \leq \sqrt{\beta_{2}} - 8\sigma_{1}^{2}(1 - \beta_{2})\beta_{2}^{-2}$  holds for large enough  $T$ , and thus Theorem 1 holds. Applying the value of  $\beta_{1}, \beta_{2}$ , and  $\eta$  to Eq. (16), we obtain that
$$
\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\| \frac {1}{\sqrt [ 4 ]{\tilde {\boldsymbol {\nu}} _ {t}}} \odot \boldsymbol {G} _ {t} \right\| ^ {2} \right] \leq D _ {2} \sqrt {T} + D _ {1} \sqrt {T} \sum_ {i = 1} ^ {d} \mathbb {E} \ln \nu_ {T, i} + \frac {6 4}{(1 - c) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}}. \tag {20}
$$
Summing Eq. (19) with respect to  $i$  then gives
$$
\begin{array}{l} \sum_ {i = 1} ^ {d} \mathbb {E} \sqrt {\frac {\nu_ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}} \\ \leq \sum_ {i = 1} ^ {d} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 \sqrt {1 - \beta_ {2}} \sigma_ {1} ^ {2} \sum_ {i = 1} ^ {d} \sum_ {t = 1} ^ {T} \frac {| \boldsymbol {G} _ {t , i} | ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {t , i}}} \\ \leq \sum_ {i = 1} ^ {d} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 D _ {2} \sigma_ {1} ^ {2} \sqrt {b} + 2 D _ {1} \sigma_ {1} ^ {2} \sqrt {b} \sum_ {i = 1} ^ {d} \mathbb {E} \ln \boldsymbol {\nu} _ {T, i} + \frac {1 2 8 \sigma_ {1} ^ {2} \sqrt {b}}{(1 - c) ^ {2} \sqrt {T}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}} \\ = \sum_ {i = 1} ^ {d} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 D _ {2} \sigma_ {1} ^ {2} \sqrt {b} + 4 D _ {1} \sigma_ {1} ^ {2} \sqrt {b} \sum_ {i = 1} ^ {d} \mathbb {E} \ln \sqrt {\boldsymbol {\nu} _ {T , i}} + \frac {1 2 8 \sigma_ {1} ^ {2} \sqrt {b}}{(1 - c) ^ {2} \sqrt {T}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}} \\ \leq \sum_ {i = 1} ^ {d} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 D _ {2} \sigma_ {1} ^ {2} \sqrt {b} + 4 D _ {1} \sigma_ {1} ^ {2} \sqrt {b} \sum_ {i = 1} ^ {d} \mathbb {E} \ln \left(\sum_ {i = 1} ^ {d} \sqrt {1 - \beta_ {2}} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}}\right) \\ + \frac {1 2 8 \sigma_ {1} ^ {2} \sqrt {b}}{(1 - c) ^ {2} \sqrt {T}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}} \\ \leq \sum_ {i = 1} ^ {d} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + 2 \sigma_ {0} ^ {2} T + \sigma_ {0} ^ {2}} + 2 D _ {2} \sigma_ {1} ^ {2} \sqrt {b} + 4 D _ {1} \sigma_ {1} ^ {2} \sqrt {b} \sum_ {i = 1} ^ {d} \ln \mathbb {E} \left(\sum_ {i = 1} ^ {d} \sqrt {1 - \beta_ {2}} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}}\right) \\ + \frac {1 2 8 \sigma_ {1} ^ {2} \sqrt {b}}{(1 - c) ^ {2} \sqrt {T}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}}, \\ \end{array}
$$
where the second inequality is due to Eq. (20), the second-to-last inequality is due to Eq. (18), and the last inequality is due to Jensen's inequality. Solving the above inequality with respect to
$$
\begin{array}{l} \sqrt {1 - \beta_ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \sqrt {\frac {\nu_ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}} \mathrm {t h e n g i v e s} \\ \sqrt {1 - \beta_ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \sqrt {\frac {\pmb {\nu} _ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}} \\ \leq 2 \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + 3 b \sigma_ {0} ^ {2}} + \frac {4 D _ {2} \sigma_ {1} ^ {2} b}{\sqrt {T}} + \frac {2 5 6 \sigma_ {1} ^ {2} b}{(1 - c) ^ {2} T} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}} + \frac {1 6 D _ {1} \sigma_ {1} ^ {2} b}{\sqrt {T}} \ln \left(e + \frac {4 \tilde {D} \sigma_ {1} ^ {2} b}{\sqrt {T}}\right). \tag {21} \\ \end{array}
$$
Therefore, by Cauchy's inequality, we have
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \| \boldsymbol {G} _ {t} \| _ {1} \right] ^ {2} \leq \left(\sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left\| \frac {1}{\sqrt [ 4 ]{\tilde {\nu} _ {t} ^ {1}}} \odot \boldsymbol {G} _ {t} \right\| ^ {2} \right]\right) \left(\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} \mathbb {E} \sqrt {\tilde {\nu} _ {t , i}}\right).
$$
$$
\sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} \sqrt {\widetilde {\boldsymbol {\nu}} _ {t , i}} \leq \sum_ {t = 1} ^ {T} \sum_ {i = 1} ^ {d} \sqrt {\beta_ {2} \boldsymbol {\nu} _ {t - 1 , i} + (1 - \beta_ {2}) \sigma_ {0} ^ {2}} \leq T \sum_ {i = 1} ^ {d} \sqrt {1 - \beta_ {2}} \sqrt {\frac {\boldsymbol {\nu} _ {0 , i}}{1 - \beta_ {2}} + \sum_ {s = 1} ^ {T} | g _ {s , i} | ^ {2} + \sigma_ {0} ^ {2}},
$$
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \| \pmb {G} _ {t} \| _ {1} \right] ^ {2} \\ \leq \left(2 T \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + 3 b \sigma_ {0} ^ {2}} + 4 D _ {2} \sigma_ {1} ^ {2} b \sqrt {T} + \frac {2 5 6 \sigma_ {1} ^ {2} b}{(1 - c) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}} + 1 6 D _ {1} \sigma_ {1} ^ {2} b \sqrt {T} \ln \left(e + \frac {4 \tilde {D} \sigma_ {1} ^ {2} b}{\sqrt {T}}\right)\right) \\ \times \left(D _ {2} \sqrt {T} + D _ {1} \sqrt {T} \sum_ {i = 1} ^ {d} \mathbb {E} \ln \nu_ {T, i} + \frac {6 4}{(1 - c) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}}\right) \\ \leq \left(2 T \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + 3 b \sigma_ {0} ^ {2}} + 4 D _ {2} \sigma_ {1} ^ {2} b \sqrt {T} + \frac {2 5 6 \sigma_ {1} ^ {2} b}{(1 - c) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}} + 1 6 D _ {1} \sigma_ {1} ^ {2} b \sqrt {T} \ln \left(e + \frac {4 \tilde {D} \sigma_ {1} ^ {2} b}{\sqrt {T}}\right)\right) \\ \times \left( \right.2 D _ {1} \sqrt {T} \sum_ {i = 1} ^ {d} \ln \left(2 \sum_ {i = 1} ^ {d} \sqrt {\nu_ {0 , i} + 3 b \sigma_ {0} ^ {2}} + \frac {4 D _ {2} \sigma_ {1} ^ {2} b}{\sqrt {T}} + \frac {2 5 6 \sigma_ {1} ^ {2} b}{(1 - c) ^ {2} T} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\nu} _ {1 , i}}} + \frac {1 6 D _ {1} \sigma_ {1} ^ {2} b}{\sqrt {T}} \ln \left(e + \frac {4 \tilde {D} \sigma_ {1} ^ {2} b}{\sqrt {T}}\right)\right) \\ \left. + \frac {6 4}{(1 - c) ^ {2}} \sum_ {i = 1} ^ {d} \mathbb {E} \frac {\boldsymbol {G} _ {1 , i} ^ {2}}{\sqrt {\tilde {\boldsymbol {\nu}} _ {1 , i}}} + D _ {2} \sqrt {T}\right). \\ \end{array}
$$
Table 1: Exploring effect of  $\beta_{1}$  of Adam. We explore the dataset of Cifar10 using VGG13[28] and ResNet18[17] and WiKiText2[24] using Transforemer[29]. We show the training loss after 50 epochs
β1	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	0.99	0.999
Cifar10 ResNet18	0.2268	0.2197	0.2158	0.2197	0.2182	0.2198	0.2217	0.2204	0.2218	0.2222	0.2351	0.3620
Cifar10 VGG13	0.1416	0.1605	0.1428	0.1453	0.1391	0.1421	0.1387	0.1457	0.1417	0.1419	0.1551	0.3497
WikiText2	3.3600	3.3589	3.3586	3.3573	3.3565	3.3599	3.3627	3.3634	3.3659	3.3749	3.4314	6.3274
As mentioned in Section 6, one of the limitations of our theory is that it can not provide better results when momentum is present. To complement such a limitation, we initialize a empirical study of the effect of momentum in Adam as follows.
Experimental setting. We use Adam training on Cifar 10 with ResNet18 [28] and VGG13 [17] and wikitext2 with two-layer Transformer [29] for 50 epoch and record its training loss at 50 epoch
as a measure for the optimization speed. Smaller loss indicates better optimization. The batch size is set 1024 for Cifar10 dataset and 100 for WikiText2 Dataset.
The results are given in Table 1. Our discoveries are:
- Momentum can benefit the optimization when the  $\beta$  is not too large.
- For all datasets, larger  $\beta_{1}$  (Setting  $\beta_{1}$  close to 1) will worse the optimization.
Connection with Theorem 1. One can easily observe that in Theorem 1 both  $C_1$  and  $C_2$  polynomials depend on  $\frac{1}{1 - \beta_1}$  and thus so does  $\mathbb{E}\sum_{t = 1}^{T}\| \nabla f(\boldsymbol {w}_t)\| = \tilde{\mathcal{O}} (\frac{1}{1 - \beta_1})$ . Therefore, Theorem 1 is aligned with the experimental results in the sense that both our theory and the experimental results indicate that the Adam will become worse when the  $\beta_{1}$  is close to 1. However, Theorem 1 cannot explain the benefit of using momentum (by setting  $\beta_{1}$  larger than 0). This may be due to that our Theorem 1 is a worst-case analysis. We conjecture that theoretically deriving the benefit of momentum requires restricting the underlying objective function to a more specific range, which we leave as a future work.