SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \ell_ {i} ^ {\prime \prime} (t) - \frac {1}{4} \right| = \left| \frac {\exp (- y _ {i} ^ {e} \cdot f (\mathbf {W} , \mathbf {x} _ {i} , t))}{(1 + \exp (- y _ {i} ^ {e} \cdot f (\mathbf {W} , \mathbf {x} _ {i} , t))) ^ {2}} - \frac {1}{4} \right| \\ = \left| \frac {1}{\exp \left(y _ {i} ^ {e} \cdot f (\mathbf {W} , \mathbf {x} _ {i} , t)\right) + 2 + \exp \left(- y _ {i} ^ {e} \cdot f (\mathbf {W} , \mathbf {x} _ {i} , t)\right)} - \frac {1}{4} \right| \\ \leq \left| \frac {1}{4} - \frac {1}{2 + 2 \exp ((2 L R) ^ {2} / 2)} \right| = \Theta ((L R) ^ {2}). \\ \end{array}
$$
Similarly, we give the upper bound of  $I_4$ :
$$
\begin{array}{l} I _ {4} = \frac {2}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} (0) \left(\ell_ {i} ^ {\prime} (t) + \frac {1}{2}\right) \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} (0) \frac {1}{2} \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {2}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \beta \| \mathbf {w} _ {j, r} (0) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} (\ell_ {i} ^ {\prime} (t) + \frac {1}{2}) \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \beta \| \mathbf {w} _ {j, r} (0) \| _ {2} \| \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \frac {1}{2} \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {6 4 \beta^ {2} L R \gamma (\frac {3}{2} \sigma_ {0} d) ^ {2}}{m}. \\ \end{array}
$$
Together, we obtain the upper bound for  $\left|\mathbf{H}_{e,e^{\prime}}^{1}(t) - \mathbf{H}_{e,e^{\prime}}^{1,\infty}\right|$ :
$$
\left| \mathbf {H} _ {e, e ^ {\prime}} ^ {1} (t) - \mathbf {H} _ {e, e ^ {\prime}} ^ {1, \infty} \right| \leq \frac {6 4 \beta^ {2} R (R + \frac {3}{2} \sigma_ {0} d)}{m} + \frac {1 2 8 \beta^ {2} L R (\frac {3}{2} \sigma_ {0} d) ^ {2}}{m}.
$$
Then we calculate the upper bound for the residual terms:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {2} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) j y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e \top} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j y _ {i ^ {\prime}} ^ {e ^ {\prime}} \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ = \frac {2}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} (t) \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) j y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} (t) \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j y _ {i ^ {\prime}} ^ {e ^ {\prime}} \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \stackrel {(a)} {\leq} \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e} (t) \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \stackrel {(b)} {\leq} \frac {1 2 8 \beta^ {2} L ^ {2} R ^ {4}}{m}, \\ \end{array}
$$
where inequality (a) is by the smoothness property of the activation function and Cauchy-Schwarz inequality, and inequality (b) is by triangle inequality and the fact that  $\| \mathbf{v}_i^e\| _2\leq 2$  for all  $i\in n_e$  and  $e\in \mathcal{E}_{\mathrm{all}}$ , and  $|\ell_i^{\prime \prime}|\leq 1$  for all  $i\in [n]$  and Lemma D.8. Similarly, we further provide the upper bound of residual terms:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {3} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) j y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e \top} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime} (t) j \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ = \frac {2}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} (t) \ell_ {i} ^ {\prime \prime} (t) \dot {y} _ {i} ^ {e} (t) j y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e \top} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} (t) \ell_ {i ^ {\prime}} ^ {\prime} (t) j \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {6 4 \beta^ {2} L R ^ {3}}{m}. \\ \end{array}
$$
Similarly, we further have that:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {4} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime} (t) j \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ = \frac {2}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} (t) \ell_ {i} ^ {\prime} (t) j \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} (t) \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime} (t) \mathbf {v} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j \mathbf {v} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {6 4 \beta^ {2} L R ^ {3}}{m}. \\ \end{array}
$$
Keep going on, we provide the computation results further:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {5} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime} (t) j y _ {i} ^ {e} \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime} (t) j y _ {i ^ {\prime}} ^ {e ^ {\prime}} \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \stackrel {(a)} {\leq} \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime} (t) \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \stackrel {(b)} {\leq} \frac {2 \beta^ {2} R ^ {2} \sigma_ {p} ^ {2} d}{m}, \\ \end{array}
$$
where inequality (a) is by smoothness property of non-linear activation function and Cauchy inequality, inequality (b) is by Lemma D.6 and Lemma D.8. Next, we calculate the
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {6} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) j \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) j \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {8 \beta^ {2} d L ^ {4} R ^ {2}}{m}. \\ \end{array}
$$
Similarly, the next  $\mathbf{H}$  term can be calculated as follows:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {7} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} j \pmb {\xi} _ {i} ^ {e \top} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime} (t) j y _ {i ^ {\prime}} ^ {e ^ {\prime}} \pmb {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {4 \beta^ {2} \sigma_ {p} ^ {2} d L R ^ {3}}{m}. \\ \end{array}
$$
Finally, we have the upper for the last term:
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} ^ {8} (t) \right| = \left| \sum_ {j} \sum_ {r = 1} ^ {m} \left(\frac {1}{n _ {e} m}\right) \left(\frac {1}{n _ {e ^ {\prime}} m}\right) \sum_ {i = 1} ^ {n _ {e}} \psi^ {\prime} y _ {i} ^ {e} \ell_ {i} ^ {\prime} (t) j \pmb {\xi} _ {i} ^ {e \top} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \psi^ {\prime} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) j \pmb {\xi} _ {i ^ {\prime}} ^ {e} \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) \right| \\ \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e ^ {\prime}}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime} (t) \boldsymbol {\xi} _ {i} ^ {e ^ {\top}} \sum_ {i ^ {\prime} = 1} ^ {n _ {e ^ {\prime}}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime \prime} (t) \hat {y} _ {i ^ {\prime}} ^ {e ^ {\prime}} (t) \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right| \\ \leq \frac {4 \beta^ {2} \sigma_ {p} ^ {2} d L R ^ {3}}{m}. \\ \end{array}
$$
To summarize, we have that,
$$
\begin{array}{l} \left| \mathbf {H} _ {e, e ^ {\prime}} (t) - \mathbf {H} _ {e, e ^ {\prime}} ^ {\infty} \right| \leq \frac {3 2 \beta^ {2} R (R + \frac {3}{2} \sigma_ {0} d)}{m} + \frac {1 2 8 \beta^ {2} (R + \frac {3}{2} \sigma_ {0} ^ {2} d) ^ {2} L ^ {2} R ^ {2}}{m} + \frac {1 2 8 \beta^ {2} L R ^ {3}}{m} \\ + \frac {2 \beta^ {2} R ^ {2} \sigma_ {p} ^ {2} d}{m} + \frac {8 \beta^ {2} \sigma_ {p} ^ {2} d L ^ {2} R ^ {4}}{m} + \frac {4 \beta^ {2} \sigma_ {p} ^ {2} d L R ^ {3}}{m} \\ \leq O \left(\frac {\beta^ {2} L R}{m}\right). \\ \end{array}
$$
where we have used  $\sigma_{p} = O(d^{-2})$ ,  $R = o(1)$ , and  $\sigma_0 = O(\sqrt{R / d})$ . Furthermore, we show that the perturbation term in Equation (22) is bounded during training. In particular, we show the complete expression:
$$
\mathbf {g} _ {e} (t) = \sum_ {j = \pm 1} \sum_ {r = 1} ^ {m} \left\langle \frac {\partial C _ {\mathrm {I R M v 1}} ^ {e} (\mathbf {W} , t)}{\partial \mathbf {w} _ {j , r} (t)}, \frac {\partial L (\mathbf {W} , t)}{\partial \mathbf {w} _ {j , r} (t)} \right\rangle
$$
$$
\begin{array}{l} = \sum_ {j = \pm 1} \sum_ {r = 1} ^ {m} \left[ \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e} \rangle) \cdot j \mathbf {v} _ {i} ^ {e} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e} \rangle) \cdot j \mathbf {v} ^ {e} _ {i} \right] \\ + \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \hat {y} _ {i} ^ {e} \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e} \rangle) \cdot j y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), y _ {i} ^ {e} \mathbf {v} _ {i} ^ {e} \rangle) \cdot j \mathbf {v} _ {i} ^ {e} \\ + \frac {\eta}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), \pmb {\xi} _ {i} \rangle) \cdot j y _ {i} ^ {e} \pmb {\xi} _ {i} \frac {\eta}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), \pmb {\xi} _ {i} \rangle) \cdot j y _ {i} ^ {e} \pmb {\xi} _ {i} \\ \left. + \frac {\eta}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), \pmb {\xi} _ {i} \rangle) \cdot j y _ {i} ^ {e} \pmb {\xi} _ {i} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \hat {y} _ {i} ^ {e} \psi^ {\prime} (\langle \mathbf {w} _ {j, r} (t), \pmb {\xi} _ {i} \rangle) \cdot j \pmb {\xi} _ {i} \right] \\ \end{array}
$$
$$
\triangleq I _ {1} + I _ {2} + I _ {3} + I _ {4}.
$$
Similar to the computation process for matrix  $\mathbf{H}$ , we adopt a divide and conquer manner:
$$
\begin{array}{l} | I _ {1} | \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime} (t) \mathbf {v} _ {i} ^ {e \top} \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \mathbf {v} _ {i} ^ {e} \right| \\ \leq \frac {3 2 \beta^ {2} (R + \frac {3}{2} \sigma_ {0} ^ {2} d) ^ {2}}{m}. \\ \end{array}
$$
The techniques used are the same when deriving upper bound for matrix  $\mathbf{H}$ . Next, we have
$$
\begin{array}{l} \left| I _ {2} \right| \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) \mathbf {v} _ {i} ^ {e} ^ {\top} \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \mathbf {v} _ {i} ^ {e} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \mathbf {v} _ {i} ^ {e} \right| \\ \leq \frac {6 4 \beta^ {2} R ^ {2} L R}{m}. \\ \end{array}
$$
The last second term can be calculated as follows:
$$
\begin{array}{l} \left| I _ {3} \right| \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime} (t) \boldsymbol {\xi} _ {i} ^ {e \top} \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} ^ {e} \| _ {2} \ell_ {i} ^ {\prime} (t) \boldsymbol {\xi} _ {i} ^ {e} \right| \\ \leq \frac {2 \beta^ {2} R ^ {2} \sigma_ {p} ^ {2} d}{m}, \\ \end{array}
$$
Finally, we show the upper bound of last term:
$$
\begin{array}{l} \left| I _ {4} \right| \leq \frac {2 \beta^ {2}}{m n _ {e} n _ {e}} \left| \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} \| _ {2} \ell_ {i} ^ {\prime \prime} (t) \hat {y} _ {i} ^ {e} (t) \boldsymbol {\xi} _ {i} ^ {e \top} \sum_ {i = 1} ^ {n _ {e}} \| \mathbf {w} _ {j, r} (t) \| _ {2} \| \boldsymbol {\xi} _ {i} ^ {e} \| _ {2} \ell_ {i ^ {\prime}} ^ {\prime} (t) \boldsymbol {\xi} _ {i} ^ {e} \right| \\ \leq \frac {4 \beta^ {2} \sigma_ {p} ^ {2} d L R ^ {3}}{m}. \\ \end{array}
$$
In a summary, we have the following inequality:
$$
\begin{array}{l} | \mathbf {g} _ {e} (t) | \leq \frac {3 2 \beta^ {2} R ^ {2}}{m} + \frac {6 4 \beta^ {2} L R ^ {3}}{m} + \frac {2 \beta^ {2} R ^ {2} \sigma_ {p} ^ {2} d}{m} + \frac {4 \beta^ {2} R ^ {3} \sigma_ {p} ^ {2} d L}{m} \\ \leq O \left(\frac {\beta^ {2} L R ^ {2}}{m}\right), \\ \end{array}
$$
where we have used  $\sigma_{p} = O(d^{-2})$  and  $R = o(1)$ . With all the bounds at hand, we are ready to have the dynamics for  $\| \mathbf{c}(t)\| _2^2$
$$
\frac {d \| \mathbf {c} (t) \| _ {2} ^ {2}}{d t} = - 2 \lambda \mathbf {c} ^ {\top} (t) \mathbf {H} (t) \mathbf {c} (t) - \mathbf {c} (t) \mathbf {g} (t) \leq - \lambda_ {0} \lambda \| \mathbf {c} (t) \| _ {2} ^ {2}, \tag {23}
$$
which requires that  $\| \mathbf{H}(t) - \mathbf{H}^{\infty}\|_{2}\leq \lambda_{0}$ . This leads to the following inequality:
$$
\begin{array}{l} \| \mathbf {H} (t) - \mathbf {H} ^ {\infty} \| _ {2} \leq \| \mathbf {H} (t) - \mathbf {H} ^ {\infty} \| _ {F} \leq \sum_ {i, j} | \mathbf {H} _ {i j} (t) - \mathbf {H} _ {i j} ^ {\infty} | \\ \leq \frac {\left| \mathcal {E} _ {t r} \right| ^ {2} \beta^ {2} L R}{m} \leq \lambda_ {0}. \\ \end{array}
$$
which leads to the conclusion for  $R$  as follows:
$$
R \leq \frac {\lambda_ {0} m}{\left| \mathcal {E} _ {t r} \right| ^ {2} \beta^ {2} L}. \tag {24}
$$
Besides, we have the inequality that
$$
\left\| \mathbf {g} \right\| _ {2} \leq \frac {\sqrt {\left| \mathcal {E} _ {t r} \right|} \beta^ {2} L R}{m} \leq \lambda \lambda_ {0} \left\| \mathbf {c} (0) \right\| _ {2}. \tag {25}
$$
Combined with Equation (24), we obtain the condition for  $\lambda$  as follows:
$$
\lambda \geq 1 / \left(\sigma_ {0} \sqrt {\left| \mathcal {E} _ {t r} \right| ^ {3}}\right). \tag {26}
$$
By inequality (23), taking the convergence time  $T = \Omega \left(\frac{\log(\sigma_0 / \epsilon)}{\eta\lambda\lambda_0}\right)$  we have that:
$$
\| \mathbf {c} (T) \| _ {2} \leq \epsilon .
$$
According to the gradient descent for IRMV1 objective function, the evolution of coefficients can be expressed as:
$$
\begin{array}{l} \gamma_ {j, r} ^ {i n v} (t + 1) = \gamma_ {j, r} ^ {i n v} (t) - \frac {\eta}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} (1 + 2 \lambda C _ {\mathrm {I R M v 1}} ^ {e} (t)) \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi_ {i} ^ {\prime} (t) \operatorname {R a d} (\alpha) _ {i} \\ - \frac {\eta \lambda}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} 2 C _ {\mathrm {I R M v 1}} ^ {e} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \psi_ {i} ^ {\prime} (t) \hat {y} _ {i} ^ {e} \cdot y _ {i} ^ {e} \mathrm {R a d} (\alpha) _ {i}, \\ \end{array}
$$
$$
\begin{array}{l} \gamma_ {j, r} ^ {s p u} (t + 1) = \gamma_ {j, r} ^ {s p u} (t) - \frac {\eta}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} \left(1 + 2 \lambda C _ {\mathrm {I R M v 1}} ^ {e} (t)\right) \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \psi_ {i} ^ {\prime} (t) \mathrm {R a d} (\beta_ {e}) \\ - \frac {\eta \lambda}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} 2 C _ {\mathrm {I R M v 1}} ^ {e} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \psi_ {i} ^ {\prime} (t) \ell_ {i} ^ {\prime \prime} \hat {y} _ {i} ^ {e} \cdot y _ {i} ^ {e} \operatorname {R a d} \left(\beta_ {e}\right) _ {i}. \\ \end{array}
$$
$$
\begin{array}{l} | \gamma_ {j, r} ^ {i n v} (t + 1) | \leq | \gamma_ {j, r} ^ {i n v} (t) | + \left| \frac {\eta}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} (1 + 2 \lambda C _ {\mathrm {I R M v 1}} ^ {e} (t)) \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \psi_ {i} ^ {\prime} (t) \ell_ {i} ^ {\prime} (t) \mathrm {R a d} (\alpha) _ {i} \right| \\ + \left| \frac {\eta \lambda}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} 2 C _ {\mathrm {I R M v 1}} ^ {e} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \psi_ {i} ^ {\prime} (t) \hat {y} _ {i} ^ {e} \cdot y _ {i} ^ {e} \mathrm {R a d} (\alpha) _ {i} \right| \\ \leq | \gamma_ {j, r} ^ {i n v} (t) | + C \frac {\eta \sqrt {| \mathcal {E} _ {t r} |} \lambda \beta R ^ {2} L}{m} \| \mathbf {c} (t) \| _ {2}. \\ \end{array}
$$
$$
| \gamma_ {j, r} ^ {s p u} (t + 1) | \leq | \gamma_ {j, r, 2} ^ {s p u} (t) | + C \frac {\eta \sqrt {| \mathcal {E} _ {t r} |} \lambda \beta R ^ {2} L}{m} \| \mathbf {c} (t) \| _ {2}.
$$
At the time step  $T$ , the feature learning satisfies that:
$$
\gamma_ {j, r} ^ {i n v} (T) \leq C \frac {\eta \sqrt {| \mathcal {E} _ {t r} |} \lambda \beta R ^ {2} L T}{m} \| \mathbf {c} (0) \| _ {2}; \quad \gamma_ {j, r} ^ {s p u} (T) \leq C \frac {\eta \sqrt {| \mathcal {E} _ {t r} |} \lambda \beta R ^ {2} L T}{m} \| \mathbf {c} (0) \| _ {2}.
$$
To make sure that  $\gamma_{j,r}^{inv}(T) = o(1)$  and  $\gamma_{j,r}^{spu}(T) = o(1)$ , we need the following condition:
$$
C \frac {\eta \sqrt {\left| \mathcal {E} _ {t r} \right|} \lambda \beta R ^ {2} L T}{m} \| \mathbf {c} (0) \| _ {2} \leq d ^ {- \frac {1}{2}}, \tag {27}
$$
combined with inequality (24) and inequality (26), we have:
$$
\sigma_ {0} \leq \frac {\left| \mathcal {E} _ {t r} \right| ^ {7 / 2} \beta^ {3} L}{d ^ {1 / 2} m ^ {2} \lambda_ {0} ^ {2} \log (1 / \epsilon)}.
$$
D.4 Proof for Proposition 4.3
Proposition D.9 (Restatement of Proposition 4.3). Consider training the CNN model with the same data as Theorem 4.1, suppose that  $\psi(x) = x$ ,  $\gamma_{j,r,1}(t_1) = \gamma_{j,r,1}(t_1 - 1)$ , and  $\gamma_{j,r,2}(t_1) = \gamma_{j,r,2}(t_1 - 1)$  at the end of ERM pre-train  $t_1$  and  $\mathcal{E}_{tr} = \{(0.25,0.1), (0.25,0.2)\}$ . Suppose that  $\delta > 0$ , and  $n > C\log(1/\delta)$ , with  $C$  being a positive constant, then with a high probability at least  $1 - \delta$ , we have
-  $\sum_{e} C_{IRM\nu l}^{e}(t_{1}) = 0$ .
-  $\gamma_{j,r,1}(t_1 + 1) > \gamma_{j,r,1}(t_1)$ .
-  $\gamma_{j,r,2}(t_1 + 1) < \gamma_{j,r,2}(t_1)$ .
Proof of Proposition D.9. According to the gradient descent for IRMV1 objective function, the evolution of coefficients can be expressed as:
$$
\begin{array}{l} \gamma_ {j, r, 1} (t + 1) = \gamma_ {j, r, 1} (t) - \frac {\eta}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} (1 + 2 \lambda C _ {\mathrm {I R M v l}} ^ {e} (t)) \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \operatorname {R a d} (\alpha) _ {i} \\ - \frac {\eta \lambda}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} 2 C _ {\mathrm {I R M v l}} ^ {e} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \hat {y} _ {i} ^ {e} \cdot y _ {i} ^ {e} \operatorname {R a d} (\alpha) _ {i}, \\ \end{array}
$$
$$
\begin{array}{l} \gamma_ {j, r, 2} (t + 1) = \gamma_ {j, r, 2} (t) - \frac {\eta}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} (1 + 2 \lambda C _ {\mathrm {I R M v 1}} ^ {e} (t)) \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime} (t) \operatorname {R a d} (\beta_ {e}) \\ - \frac {\eta \lambda}{m} \cdot \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} 2 C _ {\mathrm {I R M v 1}} ^ {e} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime \prime} \hat {y} _ {i} ^ {e} \cdot y _ {i} ^ {e} \operatorname {R a d} \left(\beta_ {e}\right) _ {i}, \\ \end{array}
$$
where  $\ell''(y_i^e \cdot f(\mathbf{W}, \mathbf{x}_i^e)) = \frac{\exp(-y_i^e \cdot f(\mathbf{W}, \mathbf{x}_i))}{(1 + \exp(-y_i^e \cdot f(\mathbf{W}, \mathbf{x}_i)))^2}$ .
To simplify the notation, we further define