SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} [ \tilde {F} ^ {t + 1} - \tilde {F} ^ {t} ] \\ \geq \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \frac {C _ {2}}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ + \left(\frac {1}{8 \gamma} - 4 \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} N + C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2}\right) (1 - \beta) ^ {2} G _ {f} ^ {2}\right) \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} \\ - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2} + \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ \left. \right. - \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} + 4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) (1 - \beta) ^ {2} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right)\right\| ^ {2} \\ + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} - \left(8 C _ {1} \gamma^ {2} G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {2} \eta^ {2} G _ {g} ^ {2} \beta^ {2} + 2 \gamma G _ {g} ^ {2} + 8 \gamma G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {1} \gamma^ {2} G _ {g} ^ {2}\right) \frac {\sigma^ {2}}{B} \\ - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} \\ - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} + \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Now we plug in Lemma 2 and 3:
$$
\begin{array}{l} \mathbb {E} [ \tilde {F} ^ {t + 1} - \tilde {F} ^ {t} ] \\ \geq \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \frac {C _ {2}}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ + \left(\frac {1}{8 \gamma} - 4 \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} N + C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2} + N\right) (1 - \beta) ^ {2} G _ {f} ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {4 8}{\beta} G _ {f} ^ {2}\right) \\ \times \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} \\ + \left(1 - \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} + 4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2} + 1\right) (1 - \beta) ^ {2}\right) \\ \times \sum_ {j = 1} ^ {N} \mathbb {E} \left| \left| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right| \right| ^ {2} \\ + \left(\frac {1}{8 \gamma} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {2 4}{\beta} G _ {f} ^ {2}\right) \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ - \left(8 C _ {1} \gamma^ {2} G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {2} \eta^ {2} G _ {g} ^ {2} \beta^ {2} + 2 \gamma G _ {g} ^ {2} + 8 \gamma G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {1} \gamma^ {2} G _ {g} ^ {2} + 2 \beta^ {2} N\right) \frac {\sigma^ {2}}{B} \\ + (1 - \frac {\beta}{2}) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \underbrace {\left(\frac {1}{1 - (1 - \frac {\beta}{2}) ^ {2}} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} - 1\right)} _ {= 0} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
By our assumption in (15)-(17):
$$
\begin{array}{l} \frac {1}{8 \gamma} - 4 (4 C _ {1} \gamma^ {2} L _ {g} ^ {2} N + C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2} + N) (1 - \beta) ^ {2} G _ {f} ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {4 8}{\beta} G _ {f} ^ {2} \geq 0, \\ 1 - \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} + 4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2} + 1\right) (1 - \beta) ^ {2} \geq 0, \\ \frac {1}{8 \gamma} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {2 4}{\beta} G _ {f} ^ {2} \geq 0, \\ \end{array}
$$
then we can have the 'clean' bound:
A.3 Proof of theorem 2
Summing the inequality from  $t = 1$  to  $T$  yields:
$$
\begin{array}{l} \frac {\mathbb {E} \left[ \tilde {F} ^ {T} - \tilde {F} ^ {0} \right]}{T} + \left(2 C _ {1} \gamma^ {2} L _ {g} ^ {2} N \beta^ {2} + 2 C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2} \beta^ {2} + 4 \gamma G _ {g} ^ {2} + 2 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 \beta^ {2}\right) \frac {\sigma^ {2}}{B} \\ \geq \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \frac {C _ {2}}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ \geq \min  \left\{\frac {C _ {1}}{2} \gamma^ {2}, \frac {C _ {2}}{2} \eta^ {2} \right\} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ \end{array}
$$
We compute the upper and lower bound of  $C_1$  and  $C_2$ . For  $C_1$
$$
C _ {1} = \frac {1}{4 \gamma} - \frac {L}{2} - \frac {3}{2} \eta L ^ {2} - 1 6 L _ {g} ^ {2} G _ {f} ^ {4} \gamma - \frac {1 9 2 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} - \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}
$$
The upper bound  $C_1 \leq \frac{1}{4\gamma}$  holds trivially. For lower bound, since we choose
$$
\gamma = \min \left\{\frac {1}{2 0 L}, \frac {1}{6 0 \eta L ^ {2}}, \frac {1}{8 \sqrt {1 0} L _ {g} G _ {f} ^ {2}}, \frac {\mu^ {2} \eta \beta}{7 6 8 0 N G _ {g} ^ {2} G _ {f} ^ {2}} \right\}
$$
we know that  $C_1 \geq \frac{1}{8\gamma}$ .
For  $C_2$ :
The upper bound  $C_2 \leq \frac{1}{\eta} + \frac{3\mu}{2}$  holds trivially. For lower bound, since we choose:
$$
\eta = \min \left\{\frac {\sqrt {2}}{3 L}, \frac {\mu}{3 6 L ^ {2}} \right\}, \beta = 0. 1 \leq 1 - \frac {\sqrt {3}}{2}
$$
it holds that  $C_2 \geq \frac{1}{2\eta}$ .
Since  $\frac{1}{8\gamma} \leq C_1 \leq \frac{1}{4\gamma}$  and  $\frac{1}{2\eta} \leq C_2 \leq \frac{1}{\eta} + \frac{3\mu}{2} \leq \frac{2}{\eta}$ , we have:
$$
\begin{array}{l} \frac {\mathbb {E} \left[ \tilde {F} ^ {T} - \tilde {F} ^ {0} \right]}{T} + \left(\frac {1}{2} \gamma L _ {g} ^ {2} N \beta^ {2} + \frac {4}{\eta} \eta^ {2} G _ {g} ^ {2} N ^ {2} \beta^ {2} + 4 \gamma G _ {g} ^ {2} + 2 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 \beta^ {2}\right) \frac {\sigma^ {2}}{B} (18) \\ \geq \min  \left\{\frac {1}{1 6} \gamma , \frac {1}{4} \eta \right\} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) \right\| ^ {2} + \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) \right\| ^ {2} (19) \\ \end{array}
$$
We then need to verify the conditions (15), (16) and (17) in Lemma 9 can hold under our choice of  $\eta_{\mathbf{x}}, \eta_{\mathbf{y}}$  and  $\beta$ . To guarantee (15) holding, we need:
$$
\gamma \leq \min  \left\{\sqrt {\frac {1}{1 2 8 L _ {g} ^ {2} N}}, \frac {1}{5 2 G _ {f} ^ {2} (2 \eta G _ {g} ^ {2} N ^ {2} + N)}, \frac {\mu^ {2} \eta}{3 0 7 2 0 0 G _ {g} ^ {2} G _ {f} ^ {2}} \right\}.
$$
To guarantee (16) holding, we need:
$$
\gamma \leq \min \left\{\frac {1}{5 0 L _ {g} ^ {2}}, \frac {1}{2 0 0 L _ {g} ^ {2} G _ {f} ^ {2}} \right\}, \eta \leq \frac {1}{1 0 0 G _ {g} ^ {2}}.
$$
To guarantee (17) holding, we need:
$$
\gamma \leq \frac {\mu^ {2} \eta}{1 0 8 0 G _ {g} ^ {2} G _ {f} ^ {2}}
$$
Next we examine how large the  $\mathbb{E}[\tilde{F}^T -\tilde{F}^0 ]$  is. By definition of potential function, we have:
$$
\begin{array}{l} \mathbb {E} [ \tilde {F} ^ {T} - \tilde {F} ^ {0} ] \\ = \hat {F} ^ {T} - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {T} - (f _ {\hat {\tau}} (\mathbf {w} ^ {T - 1}) - f _ {j} (\mathbf {w} ^ {T - 1})) \right\| ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {T} - z _ {j} ^ {T - 1} \right\| ^ {2} \\ \left. \right. - \left(\hat {F} ^ {0} - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {0} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {- 1}\right) - f _ {j} \left(\mathbf {w} ^ {- 1}\right)\right)\right\| ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {0} - z _ {j} ^ {- 1} \right\| ^ {2}\right) \\ \leq \hat {F} ^ {T} - \hat {F} ^ {0} + \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {0} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {- 1}) - f _ {j} (\mathbf {w} ^ {- 1})) \right\| ^ {2} + \frac {(1 - \frac {\beta}{2}) ^ {2}}{\beta - \frac {\beta^ {2}}{4}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {0} - z _ {j} ^ {- 1} \right\| ^ {2}. \\ \end{array}
$$
By our choice  $z_{j}^{0} = z_{j}^{-1} = f_{\widehat{T}}(\mathbf{w}^{-1}) - f_{j}(\mathbf{w}^{-1}), \mathbf{w}^{0} = \mathbf{w}^{-1}$ , we have
$$
\mathbb {E} \left[ \tilde {F} ^ {T} - \tilde {F} ^ {0} \right] \leq \hat {F} ^ {T} - \hat {F} ^ {0}
$$
Next we examine how large the  $\mathbb{E}[\hat{F}^T -\hat{F}^0 ]$  is.
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {T} - \hat {F} ^ {0} ] \\ = F (\boldsymbol {\alpha} ^ {T}, \mathbf {w} ^ {T}) + s ^ {T} - \left(\frac {1}{4 \gamma} + 4 L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {\eta L ^ {2}}{2} + \frac {1 9 2 N G _ {g} ^ {2}}{\mu^ {2} \eta \beta} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \| \mathbf {w} ^ {T} - \mathbf {w} ^ {T - 1} \| ^ {2} \\ - \left(\frac {1}{8 \gamma} + \frac {9 6 N G _ {g} ^ {2}}{\mu^ {2} \eta \beta}\right) \left\| \mathbf {w} ^ {T - 1} - \mathbf {w} ^ {T - 2} \right\| ^ {2} + \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \left\| \boldsymbol {\alpha} ^ {T} - \boldsymbol {\alpha} ^ {T - 1} \right\| ^ {2} \\ \left. - F \left(\boldsymbol {\alpha} ^ {0}, \mathbf {w} ^ {0}\right) - s ^ {0} + \left(\frac {1}{4 \gamma} + 4 L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {\eta L ^ {2}}{2} + \frac {1 9 2 N G _ {g} ^ {2}}{\mu^ {2} \eta \beta} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \| \mathbf {w} ^ {0} - \mathbf {w} ^ {- 1} \| ^ {2} \right. \\ + \left(\frac {1}{8 \gamma} + \frac {9 6 N G _ {g} ^ {2}}{\mu^ {2} \eta \beta}\right) \left\| \mathbf {w} ^ {- 1} - \mathbf {w} ^ {- 2} \right\| ^ {2} - \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \left\| \boldsymbol {\alpha} ^ {0} - \boldsymbol {\alpha} ^ {- 1} \right\| ^ {2} \\ \leq F _ {\max } + \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \left\| \boldsymbol {\alpha} ^ {T} - \boldsymbol {\alpha} ^ {T - 1} \right\| ^ {2} \\ \end{array}
$$
Notice that  $\alpha^{-1} = \alpha^{0}$ ,  $\mathbf{w}^{0} = \mathbf{w}^{-1} = \mathbf{w}^{-2}$  we have
$$
\mathbb {E} \left[ \hat {F} ^ {T} - \hat {F} ^ {0} \right] \leq F _ {\max } + \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \| \boldsymbol {\alpha} ^ {T} - \boldsymbol {\alpha} ^ {T - 1} \| ^ {2}
$$
According to updating rule,
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {\alpha} ^ {T} - \boldsymbol {\alpha} ^ {T - 1} \right\| ^ {2} = \mathbb {E} \left\| \eta \mathbf {g} _ {\boldsymbol {\alpha}} ^ {T - 1} \right\| ^ {2} \leq 2 \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} F (\boldsymbol {\alpha} ^ {t - 1}, \mathbf {w} ^ {t - 1}) \right\| ^ {2} + 2 \eta^ {2} \sum_ {j = 1} ^ {N} G _ {g} ^ {2} \left\| z _ {j} ^ {t} - \left(f _ {\hat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ \leq 2 \eta^ {2} \left(2 N B _ {g} ^ {2} + 2 L ^ {2}\right) + 2 \eta^ {2} N G _ {g} ^ {2} \left(4 \frac {(1 - \beta) ^ {4}}{1 - (1 - \beta) ^ {2}} \gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + 2 \beta^ {2} \frac {(1 - \beta) ^ {2}}{1 - (1 - \beta) ^ {2}} \frac {\sigma^ {2}}{B}\right) \\ \leq 4 \eta^ {2} \left(N B _ {g} ^ {2} + L ^ {2}\right) + 2 \eta^ {2} N G _ {g} ^ {2} \left(1 4 \gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + 2 \frac {\sigma^ {2}}{B}\right) \\ \end{array}
$$
Putting pieces together we can have the upper bound of  $\mathbb{E}[\tilde{F}^{T + 1} - \tilde{F}^1 ]$
$$
\mathbb {E} [ \tilde {F} ^ {T + 1} - \tilde {F} ^ {1} ] \leq F _ {\max } + \frac {9}{2 \eta} \left(4 \eta^ {2} (N B _ {g} ^ {2} + L ^ {2}) + 2 \eta^ {2} N G _ {g} ^ {2} \left(1 4 \gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + 2 \frac {\sigma^ {2}}{B}\right)\right)
$$
Plugging above bound back to (19) yields:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \left\| \nabla G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ \leq \max  \left\{\frac {1 6}{\gamma}, \frac {4}{\eta} \right\} \cdot \frac {F _ {\max } + \frac {9}{2} \left(4 \eta \left(N B _ {g} ^ {2} + L ^ {2}\right) + 2 \eta N G _ {g} ^ {2} \left(1 4 \gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + 2 \frac {\sigma^ {2}}{B}\right)\right)}{T} \\ + \max  \left\{\frac {1 6}{\gamma}, \frac {4}{\eta} \right\} \cdot \left(\frac {1}{2} \gamma L _ {g} ^ {2} N \beta^ {2} + 4 \eta G _ {g} ^ {2} N ^ {2} \beta^ {2} + 4 \gamma G _ {g} ^ {2} + 2 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 \beta^ {2}\right) \frac {\sigma^ {2}}{B} \\ \leq O \left(\frac {1}{\eta} \cdot \frac {F _ {\max } + \left(\eta \left(N B _ {g} ^ {2} + L ^ {2}\right) + \eta N G _ {g} ^ {2} \left(\gamma^ {2} G _ {f} ^ {4} G _ {g} ^ {2} + \frac {\sigma^ {2}}{B}\right)\right)}{T}\right) \\ + O \left(\frac {1}{\eta} \cdot \left(\gamma L _ {g} ^ {2} N \beta^ {2} + \eta G _ {g} ^ {2} N ^ {2} \beta^ {2} + \gamma G _ {g} ^ {2} + \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 \beta^ {2}\right) \frac {\sigma^ {2}}{B}\right) \\ \end{array}
$$
Define  $\kappa = L / \mu$ . Recall that we choose:
$$
\eta = \Theta \left(\frac {1}{\kappa L}\right), \gamma = \Theta \left(\min \left\{\frac {1}{L}, \frac {1}{L _ {g} G _ {f} ^ {2}}, \frac {\mu^ {2}}{N G _ {g} ^ {2} G _ {f} ^ {2} \kappa L} \right\}\right)
$$
To guarantee RHS is less than  $\epsilon^2$ , we need:
$$
T = \Omega \left(\max \left\{\frac {F _ {\mathrm {m a x}}}{\eta \epsilon^ {2}}, \frac {N B _ {g} ^ {2}}{\epsilon^ {2}} \right\}\right), B = \Theta \left(\max \left\{\frac {\mu^ {2} L _ {g} ^ {2} \sigma^ {2}}{G _ {g} ^ {2} G _ {f} ^ {2} \epsilon^ {2}}, \frac {G _ {g} ^ {2} N \sigma^ {2}}{\epsilon^ {2}}, \frac {\mu^ {2}}{N G _ {f} ^ {2} \epsilon^ {2}}, \frac {\mu^ {2} L _ {g} ^ {2} \sigma^ {2}}{N G _ {g} ^ {2} \epsilon^ {2}}, \frac {\kappa L \sigma^ {2}}{\epsilon^ {2}} \right\}\right),
$$
which yields the total gradient complexity:
$$
O \left(\frac {\kappa L F _ {\mathrm {m a x}}}{\epsilon^ {2}} \cdot \max \left\{\frac {\kappa L \sigma^ {2}}{\epsilon^ {2}}, \frac {\mu^ {2} L _ {g} ^ {2} \sigma^ {2}}{G _ {g} ^ {2} G _ {f} ^ {2} \epsilon^ {2}}, \frac {G _ {g} ^ {2} N \sigma^ {2}}{\epsilon^ {2}}, 1 \right\}\right).
$$
B Proofs for  $\mathbf{w}^*$  Approximation Algorithm
Proof. First, we define  $\Phi (\alpha ,\mathbf{w}) = \sum_{j = 1}^{N}\alpha (j)\cdot f_{j}(\mathbf{w})$ . Indeed,  $\Phi (\cdot ,\mathbf{w})$  is  $\sqrt{N} G_{f}$  smooth for all  $\mathbf{w}\in \mathcal{W}$ . To see this, we consider
$$
\begin{array}{l} \| \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w}) - \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha} ^ {\prime}, \mathbf {w}) \| = \left\| \sum_ {i = 1} ^ {N} \alpha_ {i} \nabla f _ {i} (\mathbf {w}) - \sum_ {i = 1} ^ {N} \alpha_ {i} ^ {\prime} \nabla f _ {i} (\mathbf {w}) \right\| \\ = \left\| \sum_ {i = 1} ^ {N} \left(\alpha_ {i} - \alpha_ {i} ^ {\prime}\right) \nabla f _ {i} (\mathbf {w}) \right\| \\ \leq \sum_ {i = 1} ^ {N} \left| \alpha_ {i} - \alpha_ {i} ^ {\prime} \right| \max  _ {i \in [ N ]} \| \nabla f _ {i} (\mathbf {w}) \| \\ \leq \left\| \boldsymbol {\alpha} - \boldsymbol {\alpha} ^ {\prime} \right\| _ {1} G _ {f} \\ \leq \sqrt {N} G _ {f} \| \boldsymbol {\alpha} - \boldsymbol {\alpha} ^ {\prime} \|. \\ \end{array}
$$
According to optimality conditions we have:
$$
\langle \mathbf {w} - \mathbf {w} ^ {*} (\boldsymbol {\alpha}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha})) \rangle \geq 0,
$$
$$
\langle \mathbf {w} - \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha} ^ {\prime}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) \rangle \geq 0
$$
Substituting  $\mathbf{v}$  with  $\mathbf{w}^{*}(\alpha^{\prime})$  and  $\mathbf{w}^{*}(\alpha)$  in the above first and second inequalities respectively yields:
$$
\left\langle \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha})) \right\rangle \geq 0,
$$
$$
\langle \mathbf {w} ^ {*} (\boldsymbol {\alpha}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha} ^ {\prime}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) \rangle \geq 0
$$
$$
\langle \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha})) - \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha} ^ {\prime}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) \rangle \geq 0, \tag {20}
$$
$$
\langle \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) - \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha})) \geq \mu \| \mathbf {v} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \| ^ {2}. \tag {21}
$$
Adding up (20) and (21) yields:
$$
\left\langle \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}), \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) - \nabla_ {\mathbf {w}} \Phi (\boldsymbol {\alpha} ^ {\prime}, \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime})) \right\rangle \geq \mu \| \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \| ^ {2}
$$
Finally, using  $\sqrt{N} G_{f}$  smoothness of  $\Phi$  will conclude the proof:
$$
\begin{array}{l} \sqrt {N} G _ {f} \| \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \| \| \boldsymbol {\alpha} - \boldsymbol {\alpha} ^ {\prime} \| \geq \mu \| \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \| ^ {2} \\ \Longleftrightarrow \kappa^ {*} \| \boldsymbol {\alpha} - \boldsymbol {\alpha} ^ {\prime} \| \geq \| \mathbf {w} ^ {*} (\boldsymbol {\alpha} ^ {\prime}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha}) \|. \\ \end{array}
$$
B.2 Preliminaries of the proof of theorem 3
In this section, we introduce some notational preliminaries for the proof for Theorem 3. Following the framework in [21], we introduce three models, neural network predictor, random feature predictor and kernel least square predictor. In the following sections we will use  $w(j)$  to denote the  $j$ th coordinate of vector  $\mathbf{w}$ .
Two-layer ReLU neural network (NN) predictor Recall that in Section 3.1 we consider a two layer vector-valued neural network  $\mathbf{h}_{\theta}:\mathbb{R}^{N}\to \mathbb{R}^{d}$  ..
$$
\mathbf {h} _ {\boldsymbol {\theta}} (\mathbf {x}) = \left[ \mathbf {a} _ {1} ^ {\top} (\mathbf {U} ^ {1} \mathbf {x}) _ {+}, \dots , \mathbf {a} _ {d} ^ {\top} (\mathbf {U} ^ {d} \mathbf {x}) _ {+} \right],
$$
where parameters of the hidden layer are matrices  $\mathbf{U}^i\in \mathbb{R}^{m\times N}$ , collectively captured by the parameter vector  $\pmb {\theta} = (\mathrm{vec}(\mathbf{U}^1),\dots ,\mathrm{vec}(\mathbf{U}^d))\in \mathbb{R}^{dmN}$ , and we also define  $\pmb{\theta}^{j} = \mathrm{vec}(\mathbf{U}^{j})\in \mathbb{R}^{Nm}$ . Here  $\mathbf{a}_i\in \{\pm 1 / \sqrt{m}\} ^m$  are parameters of the output layer.
Next, induce the Neural Tangent Feature (NTF) operator, defined as
$$
\boldsymbol {\phi} _ {t} ^ {j} = \left( \begin{array}{c} a _ {j} (1) \mathbb {I} \{\mathbf {u} _ {t} ^ {j} ^ {\top} (1) \boldsymbol {\alpha} \} \boldsymbol {\alpha} \\ \vdots \\ a _ {j} (m) \mathbb {I} \{\mathbf {u} _ {t} ^ {j} ^ {\top} (m) \boldsymbol {\alpha} \} \boldsymbol {\alpha} \end{array} \right) \in \mathbb {R} ^ {d m} \qquad (j, t \in \mathbb {N})
$$
where  $\mathbf{u}_t^{j}^{\top}(r)$  is  $r$ th row of  $\mathbf{U}_t^j$ . Note that we dropped dependence on  $\alpha$  on the left hand side — it is always assumed that the operator is applied to variable  $\alpha$ . Also, note that the operator comes by differentiation of the neural network with respect to a  $j$ -th component of the hidden layer, that is  $\phi_t^j (\pmb {\alpha}) = \nabla_{\pmb{\theta}^j}\mathbf{h}_{\pmb{\theta}}(\pmb {\alpha})|_{\pmb{\theta}^j = \pmb{\theta}_t^j}$ . For convenience, we also introduce the following NTF matrix notation, given multiple input vectors  $\alpha_{1},\ldots ,\alpha_{n}$ :
$$
\boldsymbol {\Phi} _ {t} ^ {j} = \left[ \phi_ {t} ^ {j} (\boldsymbol {\alpha} _ {1}), \dots , \phi_ {t} ^ {j} (\boldsymbol {\alpha} _ {n}) \right] \in \mathbb {R} ^ {d m \times n}.
$$
Finally, it is not hard to see that the use of NTF operator recovers the neural network predictor:
$$
\mathbf {h} _ {t} (\boldsymbol {\alpha}) = \left[ \begin{array}{c} \phi_ {t} ^ {1} (\boldsymbol {\alpha}) ^ {\top} \boldsymbol {\theta} _ {t} ^ {1} \\ \vdots \\ \phi_ {t} ^ {d} (\boldsymbol {\alpha}) ^ {\top} \boldsymbol {\theta} _ {t} ^ {d} \end{array} \right]
$$
Using the above definitions, we can write least squares objective as
$$
\widehat {\mathcal {R}} (\boldsymbol {\theta} _ {t}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\mathbf {h} _ {t} (\boldsymbol {\alpha} _ {i}) - \mathbf {w} ^ {*} (\boldsymbol {\alpha} _ {i})\right) ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {d} \left(\boldsymbol {\phi} _ {t} ^ {j} ^ {\top} (\boldsymbol {\alpha} _ {i}) \boldsymbol {\theta} _ {t} ^ {j} - w ^ {*} (\boldsymbol {\alpha} _ {i}) (j)\right) ^ {2},
$$
and similarly for the pointwise loss function,
$$
\widehat {\mathcal {R}} ^ {j} (\pmb {\theta} _ {t} ^ {j}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(h _ {t} ^ {j} (\pmb {\alpha} _ {i}) - w ^ {*} (\pmb {\alpha} _ {i}) (j)\right) ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\phi_ {t} ^ {j} ^ {\top} (\pmb {\alpha} _ {i}) \pmb {\theta} _ {t} ^ {j} - w ^ {*} (\pmb {\alpha} _ {i}) (j)\right) ^ {2},
$$
and for the GD update rule:
$$
\boldsymbol {\theta} _ {t + 1} ^ {j} = \boldsymbol {\theta} _ {t} ^ {j} - \eta \mathbf {g} _ {t} ^ {j}, \text {w h e r e} \mathbf {g} _ {t} ^ {j} = \frac {1}{n} \sum_ {i = 1} ^ {n} \phi_ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) \left(\mathbf {h} _ {t} ^ {j} (\boldsymbol {\alpha} _ {i}) - w _ {i} ^ {t} (j)\right).
$$