SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ResNet50 on ImageNet. Due to the constraints on computational resources, we report the averaged results over 2 independent runs. For this dataset, we randomly resize and crop all images to a resolution of  $224 \times 224$ , and apply random horizontal flip, normalization during training. The batch size is chosen as 128 with a cosine learning rate scheduling with an initial step size 0.05. The momentum and weight decay of base optimizer, SGD, are set as 0.9 and  $10^{-4}$ , respectively. We further tune  $\rho$  from  $\{0.05, 0.075, 0.1, 0.2\}$ , and chooses  $\rho = 0.075$  for SAM. VaSSO uses  $\theta = 0.99$ . VaSSO and ASAM adopt the same  $\rho = 0.075$ .
ViT-S/32 on ImageNet. We follow the implementation of (Du et al., 2022b), where we train the model for 300 epochs with a batch size of 4096. The baseline optimizer is chosen as AdamW with weight decay 0.3. SAM relies on  $\rho = 0.05$ . For the implementation of GSAM and V+G, we adopt the same implementation from (Zhuang et al., 2022).
D Missing proofs
Alg. 1 can be written as
$$
\mathbf {x} _ {t + \frac {1}{2}} = \mathbf {x} _ {t} + \boldsymbol {\epsilon} _ {t} \tag {7a}
$$
$$
\mathbf {x} _ {t + 1} = \mathbf {x} _ {t} - \eta_ {t} \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \tag {7b}
$$
where  $\| \epsilon_t\| = \rho$ . In SAM, we have  $\epsilon_{t} = \rho \frac{\mathbf{g}_{t}(\mathbf{x}_{t})}{\|\mathbf{g}_{t}(\mathbf{x}_{t})\|}$ , and in VaSSO we have  $\epsilon_{t} = \rho \frac{\mathbf{d}_{t}}{\|\mathbf{d}_{t}\|}$ .
This subsection presents useful lemmas to support our main results.
Lemma 1. Alg. 1 (or equivalently iteration (7)) ensures that
$$
\eta_ {t} \mathbb {E} \big [ \langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) \rangle \big ] \leq \frac {L \eta_ {t} ^ {2}}{2} \mathbb {E} \big [ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \big ] + \frac {L \rho^ {2}}{2}.
$$
$$
\left\langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \right\rangle = \left\langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) + \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \right\rangle . \tag {8}
$$
Taking expectation conditioned on  $\mathbf{x}_t$ , we arrive at
$$
\begin{array}{l} \mathbb {E} \left[ \left\langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \right\rangle | \mathbf {x} _ {t} \right] \\ = \mathbb {E} \left[ \langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} (\mathbf {x} _ {t}) \rangle | \mathbf {x} _ {t} \right] + \mathbb {E} \left[ \langle \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) \rangle | \mathbf {x} _ {t} \right] \\ = \mathbb {E} \left[ \langle \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \rangle | \mathbf {x} _ {t} \right] \\ \leq \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| \cdot \| \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \| | \mathbf {x} _ {t} \right] \\ \stackrel {(a)} {\leq} L \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| \cdot \| \mathbf {x} _ {t} - \mathbf {x} _ {t + \frac {1}{2}} \| | \mathbf {x} _ {t} \right] \\ \stackrel {(b)} {=} L \rho \| \nabla f (\mathbf {x} _ {t}) \| \\ \end{array}
$$
where (a) is because of Assumption 2; and (b) is because  $\mathbf{x}_t - \mathbf{x}_{t + \frac{1}{2}} = -\pmb{\epsilon}_t$  and its norm is  $\rho$ . This inequality ensures that
$$
\eta_ {t} \mathbb {E} \big [ \big \langle \nabla f (\mathbf {x} _ {t}), \nabla f (\mathbf {x} _ {t}) - \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) \big \rangle | \mathbf {x} _ {t} \big ] \leq L \rho \eta_ {t} \| \nabla f (\mathbf {x} _ {t}) \| \leq \frac {L \eta_ {t} ^ {2} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2}}{2} + \frac {L \rho^ {2}}{2}
$$
where the last inequality is because  $\rho \eta_t\| \nabla f(\mathbf{x}_t)\| \leq \frac{1}{2}\eta_t^2\| \nabla f(\mathbf{x}_t)\|^2 +\frac{1}{2}\rho^2$  . Taking expectation w.r.t.  $\mathbf{x}_t$  finishes the proof.
Lemma 2. Alg. 1 (or equivalently iteration (7)) ensures that
$$
\mathbb {E} \left[ \| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \| ^ {2} \right] \leq 2 L ^ {2} \rho^ {2} + 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] + 2 \sigma^ {2}.
$$
Proof. The proof starts with bounding  $\| \mathbf{g}_t(\mathbf{x}_{t + \frac{1}{2}})\|$  as
$$
\begin{array}{l} \left\| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \right\| ^ {2} = \left\| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) + \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) \right\| ^ {2} \\ \leq 2 \left\| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) - \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) \right\| ^ {2} + 2 \left\| \mathbf {g} _ {t} \left(\mathbf {x} _ {t}\right) \right\| ^ {2} \\ \stackrel {(a)} {\leq} 2 L ^ {2} \left\| \mathbf {x} _ {t} - \mathbf {x} _ {t + \frac {1}{2}} \right\| ^ {2} + 2 \left\| \mathbf {g} _ {t} (\mathbf {x} _ {t}) \right\| ^ {2} \\ \stackrel {(b)} {=} 2 L ^ {2} \rho^ {2} + 2 \| \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) + \nabla f (\mathbf {x} _ {t}) \| ^ {2} \\ \end{array}
$$
where (a) is the result of Assumption 2; and (b) is because  $\mathbf{x}_t - \mathbf{x}_{t + \frac{1}{2}} = -\pmb {\epsilon}_t$  and its norm is  $\rho$
Taking expectation conditioned on  $\mathbf{x}_t$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \| ^ {2} | \mathbf {x} _ {t} \right] \leq 2 L ^ {2} \rho^ {2} + 2 \mathbb {E} \left[ \| \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) + \nabla f (\mathbf {x} _ {t}) \| ^ {2} | \mathbf {x} _ {t} \right] \\ \leq 2 L ^ {2} \rho^ {2} + 2 \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} + 2 \sigma^ {2} \\ \end{array}
$$
where the last inequality is because of Assumption 3. Taking expectation w.r.t. the randomness in  $\mathbf{x}_t$  finishes the proof.
Lemma 3. Let  $A_{t + 1} = \alpha A_t + \beta$  with some  $\alpha \in (0,1)$ , then we have
$$
A _ {t + 1} \leq \alpha^ {t + 1} A _ {0} + \frac {\beta}{1 - \alpha}.
$$
Proof. The proof can be completed by simply unrolling  $A_{t+1}$  and using the fact  $1 + \alpha + \alpha^2 + \ldots + \alpha^t \leq \frac{1}{1 - \alpha}$ .
Proof. Using Assumption 2, we have that
$$
\begin{array}{l} f \left(\mathbf {x} _ {t + 1}\right) - f \left(\mathbf {x} _ {t}\right) \\ \leq \left\langle \nabla f (\mathbf {x} _ {t}), \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \right\rangle + \frac {L}{2} \left\| \mathbf {x} _ {t + 1} - \mathbf {x} _ {t} \right\| ^ {2} \\ = - \eta_ {t} \langle \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \rangle + \frac {L \eta_ {t} ^ {2}}{2} \| \mathbf {g} _ {t} \left(\mathbf {x} _ {t + \frac {1}{2}}\right) \| ^ {2} \\ = - \eta_ {t} \langle \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) - \nabla f (\mathbf {x} _ {t}) + \nabla f (\mathbf {x} _ {t}) \rangle + \frac {L \eta_ {t} ^ {2}}{2} \| \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) \| ^ {2} \\ = - \eta_ {t} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} - \eta_ {t} \langle \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) - \nabla f (\mathbf {x} _ {t}) \rangle + \frac {L \eta_ {t} ^ {2}}{2} \| \mathbf {g} _ {t} (\mathbf {x} _ {t + \frac {1}{2}}) \| ^ {2}. \\ \end{array}
$$
Taking expectation, then plugging Lemmas 1 and 2 in, we have
$$
\mathbb {E} \big [ f (\mathbf {x} _ {t + 1}) - f (\mathbf {x} _ {t}) \big ] \leq - \Big (\eta_ {t} - \frac {3 L \eta_ {t} ^ {2}}{2} \Big) \mathbb {E} \big [ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \big ] + \frac {L \rho^ {2}}{2} + L ^ {3} \eta_ {t} ^ {2} \rho^ {2} + L \eta_ {t} ^ {2} \sigma^ {2}.
$$
As the parameter selection ensures that  $\eta_t \equiv \eta = \frac{\eta_0}{\sqrt{T}} \leq \frac{2}{3L}$ , it is possible to divide both sides with  $\eta$  and rearrange the terms to arrive at
$$
\left(1 - \frac {3 L \eta}{2}\right) \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \leq \frac {\mathbb {E} \left[ f (\mathbf {x} _ {t}) - f (\mathbf {x} _ {t + 1}) \right]}{\eta} + \frac {L \rho^ {2}}{2 \eta} + L ^ {3} \eta \rho^ {2} + L \eta \sigma^ {2}.
$$
Summing over  $t$ , we have
$$
\begin{array}{l} \left(1 - \frac {3 L \eta}{2}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \leq \frac {\mathbb {E} \left[ f (\mathbf {x} _ {0}) - f (\mathbf {x} _ {T}) \right]}{\eta T} + \frac {L \rho^ {2}}{2 \eta} + L ^ {3} \eta \rho^ {2} + L \eta \sigma^ {2} \\ \stackrel {(a)} {\leq} \frac {f (\mathbf {x} _ {0}) - f ^ {*}}{\eta T} + \frac {L \rho^ {2}}{2 \eta} + L ^ {3} \eta \rho^ {2} + L \eta \sigma^ {2} \\ = \frac {f (\mathbf {x} _ {0}) - f ^ {*}}{\eta_ {0} \sqrt {T}} + \frac {L \rho_ {0} ^ {2}}{2 \eta_ {0} \sqrt {T}} + \frac {L ^ {3} \eta_ {0} \rho_ {0} ^ {2}}{T ^ {3 / 2}} + \frac {L \eta_ {0} \sigma^ {2}}{\sqrt {T}} \\ \end{array}
$$
where (a) uses Assumption 1, and the last equation is obtained by plugging in the value of  $\rho$  and  $\eta$ . This completes the proof to the first part.
For the second part of this theorem, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} + \boldsymbol {\epsilon} _ {t}) \| ^ {2} \right] = \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} + \boldsymbol {\epsilon} _ {t}) + \nabla f (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \\ \leq 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] + 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} + \boldsymbol {\epsilon} _ {t}) - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \\ \leq 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} \| ^ {2} \right] + 2 L ^ {2} \rho^ {2} \\ = 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t} \| ^ {2} \right] + \frac {2 L ^ {2} \rho_ {0} ^ {2}}{T}. \\ \end{array}
$$
Averaging over  $t$  completes the proof.
Proof. To bound the MSE, we first have that
$$
\begin{array}{l} \left\| \mathbf {d} _ {t} - \nabla f (\mathbf {x} _ {t}) \right\| ^ {2} \tag {9} \\ = \| (1 - \theta) \mathbf {d} _ {t - 1} + \theta \mathbf {g} _ {t} (\mathbf {x} _ {t}) - (1 - \theta) \nabla f (\mathbf {x} _ {t}) - \theta \nabla f (\mathbf {x} _ {t}) \| ^ {2} \\ = (1 - \theta) ^ {2} \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t}) \| ^ {2} + \theta^ {2} \| \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \\ + 2 \theta (1 - \theta) \langle \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) \rangle . \\ \end{array}
$$
Now we cope with three terms in the right hand of (9) separately.
The second term can be bounded directly using Assumption 2
$$
\mathbb {E} \left[ \| \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) \| ^ {2} | \mathbf {x} _ {t} \right] \leq \sigma^ {2}. \tag {10}
$$
For the third term, we have
$$
\mathbb {E} \left[ \left\langle \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t}), \mathbf {g} _ {t} (\mathbf {x} _ {t}) - \nabla f (\mathbf {x} _ {t}) \right\rangle | \mathbf {x} _ {t} \right] = 0. \tag {11}
$$
The first term is bounded through
$$
\begin{array}{l} \left\| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t}) \right\| ^ {2} = \left\| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) + \nabla f (\mathbf {x} _ {t - 1}) - \nabla f (\mathbf {x} _ {t}) \right\| ^ {2} \\ \stackrel {(a)} {\leq} (1 + \lambda) \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} + \left(1 + \frac {1}{\lambda}\right) \| \nabla f (\mathbf {x} _ {t - 1}) - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \\ \leq (1 + \lambda) \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} + \left(1 + \frac {1}{\lambda}\right) L ^ {2} \| \mathbf {x} _ {t - 1} - \mathbf {x} _ {t} \| ^ {2} \\ = (1 + \lambda) \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} + \left(1 + \frac {1}{\lambda}\right) \eta^ {2} L ^ {2} \| \mathbf {g} _ {t - 1} \left(\mathbf {x} _ {t - \frac {1}{2}}\right) \| ^ {2} \\ \end{array}
$$
where (a) is because of Young's inequality. Taking expectation and applying Lemma 2, we have that
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t}) \right\| ^ {2} \right] \tag {12} \\ \leq (1 + \lambda) \mathbb {E} \left[ \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} \right] + \left(1 + \frac {1}{\lambda}\right) \eta^ {2} L ^ {2} \left(2 L ^ {2} \rho^ {2} + 2 \mathbb {E} \left[ \| \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} \right] + 2 \sigma^ {2}\right) \\ \leq (1 + \lambda) \mathbb {E} \left[ \left\| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \right\| ^ {2} \right] + \left(1 + \frac {1}{\lambda}\right) \cdot \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {T}}\right). \\ \end{array}
$$
The last inequality uses the value of  $\eta = \frac{\eta_0}{\sqrt{T}}$  and  $\rho = \frac{\rho_0}{\sqrt{T}}$ . In particular, we have  $\eta^2\rho^2 L^4 = \mathcal{O}(1 / T^2)$  and  $\eta^2 L^2\sigma^2 = \mathcal{O}(\sigma^2 /T)$ , and
$$
\eta^ {2} L ^ {2} \mathbb {E} \big [ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \big ] = \frac {\eta_ {0} ^ {2} L ^ {2}}{T} \mathbb {E} \big [ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \big ] \leq \eta_ {0} ^ {2} L ^ {2} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \big [ \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \big ] = \mathcal {O} \bigg (\frac {\sigma^ {2}}{\sqrt {T}} \bigg)
$$
where the last equation is the result of Theorem 1.
Combining (9) with (12), (10) and (11), and choosing  $\lambda = \frac{\theta}{1 - \theta}$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \| \mathbf {d} _ {t} - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right] \leq (1 - \theta) \mathbb {E} \left[ \| \mathbf {d} _ {t - 1} - \nabla f (\mathbf {x} _ {t - 1}) \| ^ {2} \right] + \frac {(1 - \theta) ^ {2}}{\theta} \mathcal {O} \left(\frac {\sigma^ {2}}{\sqrt {T}}\right) + \theta^ {2} \sigma^ {2} \\ \leq \theta \sigma^ {2} + \mathcal {O} \left(\frac {(1 - \theta) ^ {2} \sigma^ {2}}{\theta^ {2} \sqrt {T}}\right) \\ \end{array}
$$
where the last inequality is the result of Lemma 3.
D.4 Proof of Theorem 3
Proof. We adopt a unified notation for simplicity. Let  $\mathbf{v}_t \coloneqq \mathbf{d}_t$  for VaSSO, and  $\mathbf{v}_t \coloneqq \mathbf{g}_t(\mathbf{x}_t)$  for SAM. Then for both VaSSO and SAM, we have that
$$
f \left(\mathbf {x} _ {t}\right) + \left\langle \mathbf {v} _ {t}, \boldsymbol {\epsilon} _ {t} \right\rangle = f \left(\mathbf {x} _ {t}\right) + \rho \| \mathbf {v} _ {t} \| = f \left(\mathbf {x} _ {t}\right) + \rho \| \mathbf {v} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right) + \nabla f \left(\mathbf {x} _ {t}\right) \|. \tag {13}
$$
For convenience, let  $\epsilon_t^* = \rho \nabla f(\mathbf{x}_t) / \| \nabla f(\mathbf{x}_t)\|$ . From (13), we have that
$$
\begin{array}{l} f \left(\mathbf {x} _ {t}\right) + \left\langle \mathbf {v} _ {t}, \boldsymbol {\epsilon} _ {t} \right\rangle = f \left(\mathbf {x} _ {t}\right) + \rho \| \mathbf {v} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right) + \nabla f \left(\mathbf {x} _ {t}\right) \| \tag {14} \\ \leq f \left(\mathbf {x} _ {t}\right) + \rho \| \nabla f (\mathbf {x} _ {t}) \| + \rho \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \| \\ = f \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f \left(\mathbf {x} _ {t}\right), \boldsymbol {\epsilon} _ {t} ^ {*} \right\rangle + \rho \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \|. \\ \end{array}
$$
Applying triangular inequality  $\left| \| \mathbf{a} \| - \| \mathbf{b} \| \right| \leq \| \mathbf{a} - \mathbf{b} \|$ , we arrive at
$$
\begin{array}{l} f \left(\mathbf {x} _ {t}\right) + \left\langle \mathbf {v} _ {t}, \boldsymbol {\epsilon} _ {t} \right\rangle = f \left(\mathbf {x} _ {t}\right) + \rho \| \nabla f (\mathbf {x} _ {t}) - (\nabla f (\mathbf {x} _ {t}) - \mathbf {v} _ {t}) \| \tag {15} \\ \geq f \left(\mathbf {x} _ {t}\right) + \rho \| \nabla f \left(\mathbf {x} _ {t}\right) \| - \rho \| \mathbf {v} _ {t} - \nabla f \left(\mathbf {x} _ {t}\right) \| \\ = f \left(\mathbf {x} _ {t}\right) + \left\langle \nabla f \left(\mathbf {x} _ {t}\right), \boldsymbol {\epsilon} _ {t} ^ {*} \right\rangle - \rho \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \|. \\ \end{array}
$$
Combining (14) with (15), we have
$$
\left| \mathcal {L} _ {t} (\mathbf {v} _ {t}) - \mathcal {L} _ {t} \left(\nabla f (\mathbf {x} _ {t})\right) \right| \leq \rho \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \|
$$
$$
\mathbb {E} \left[ | \mathcal {L} _ {t} (\mathbf {v} _ {t}) - \mathcal {L} _ {t} (\nabla f (\mathbf {x} _ {t})) | \right] \leq \rho \mathbb {E} \left[ \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \| \right] \leq \rho \sqrt {\mathbb {E} \left[ \| \mathbf {v} _ {t} - \nabla f (\mathbf {x} _ {t}) \| ^ {2} \right]}.
$$
The last inequality is because  $(\mathbb{E}[a])^2 \leq \mathbb{E}[a^2]$ . This theorem can be proved by applying Assumption 3 for SAM and Lemma 2 for VaSSO.