SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Xi_ {j, r, i ^ {\prime}} ^ {t + 1, e ^ {\prime}} = \Xi_ {j, r, i ^ {\prime}} ^ {t, e ^ {\prime}} + \frac {\eta}{m} \sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \frac {y _ {i} ^ {e} \cdot \left\langle \boldsymbol {\xi} _ {i} ^ {e} , \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right\rangle}{1 + \exp \left\{y _ {i} ^ {e} \hat {y} _ {i} ^ {e} \right\}}. \tag {10}
$$
Notice that the updates (8), (9) for  $\Lambda_{j,r},\Gamma_{j,r}$  are independent of  $j,r$  . Denoting
$$
\Delta_ {\Lambda} ^ {t} \triangleq \frac {1}{m} \sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \frac {\operatorname {R a d} (\alpha) _ {i}}{1 + \exp \{y _ {i} ^ {e} \hat {y} _ {i} ^ {e} \}},
$$
$$
\Delta_ {\Gamma} ^ {t} \triangleq \frac {1}{m} \sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \frac {\mathrm {R a d} (\beta_ {e}) _ {i}}{1 + \exp \{y _ {i} ^ {e} \hat {y} _ {i} ^ {e} \}},
$$
we can conclude that for any  $j\in \{+1, - 1\} ,r\in [m]$
$$
\Lambda_ {j, r} ^ {t + 1} = \Lambda_ {j, r} ^ {t} + \eta \cdot \Delta_ {\Lambda} ^ {t} = \eta \cdot \sum_ {k = 0} ^ {t} \Delta_ {\Lambda} ^ {k} + \Lambda_ {j, r} ^ {0}, \tag {11}
$$
$$
\Gamma_ {j, r} ^ {t + 1} = \Gamma_ {j, r} ^ {t} + \eta \cdot \Delta_ {\Gamma} ^ {t} = \eta \cdot \sum_ {k = 0} ^ {t} \Delta_ {\Gamma} ^ {k} + \Gamma_ {j, r} ^ {0}.
$$
Then, we write the logit  $\hat{y}_i^e$  as
$$
\begin{array}{l} \hat {y} _ {i} ^ {e} = \sum_ {j \in \{\pm 1 \}} \frac {j}{m} \sum_ {r = 1} ^ {m} \left[ \left\langle \mathbf {w} _ {j, r} ^ {t}, y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot \mathbf {v} _ {1} + y _ {i} ^ {e} \cdot \operatorname {R a d} (\beta_ {e}) _ {i} \cdot \mathbf {v} _ {2} + \mathbf {x} _ {i, 2} ^ {e} \right\rangle \right] \\ = \sum_ {j \in \{\pm 1 \}} \frac {j}{m} \sum_ {r = 1} ^ {m} \left[ j y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot \Lambda_ {j, r} ^ {t} + j y _ {i} ^ {e} \cdot \operatorname {R a d} (\beta_ {e}) _ {i} \cdot \Gamma_ {j, r} ^ {t} + j \cdot \Xi_ {j, r, i} ^ {t, e} \right] \\ = \sum_ {j \in \{\pm 1 \}} \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot \Lambda_ {j, r} ^ {t} + y _ {i} ^ {e} \cdot \operatorname {R a d} (\beta_ {e}) _ {i} \cdot \Gamma_ {j, r} ^ {t} + \Xi_ {j, r, i} ^ {t, e} \right] \\ = y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Lambda_ {j , r} ^ {t}}{m} + y _ {i} ^ {e} \cdot \operatorname {R a d} (\beta_ {e}) _ {i} \cdot \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Gamma_ {j , r} ^ {t}}{m} + \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Xi_ {j , r , i} ^ {t , e}}{m} \\ = y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} + y _ {i} ^ {e} \cdot \operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} \\ + y _ {i} ^ {e} \cdot \operatorname {R a d} (\alpha) _ {i} \cdot \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Lambda_ {j , r} ^ {0}}{m} + y _ {i} ^ {e} \cdot \operatorname {R a d} \left(\beta_ {e}\right) _ {i} \cdot \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Gamma_ {j , r} ^ {0}}{m} + \sum_ {j \in \{\pm 1 \}} \sum_ {r = 1} ^ {m} \frac {\Xi_ {j , r , i} ^ {t , e}}{m}. \\ \end{array}
$$
Denoting  $\mathbb{Q}_i^e\triangleq \operatorname {Rad}(\alpha)_i\sum_{j\in \{\pm 1\}}\sum_{r = 1}^m\frac{\Lambda_{j,r}^0}{m} +\operatorname {Rad}(\beta_e)_i\sum_{j\in \{\pm 1\}}\sum_{r = 1}^m\frac{\Gamma_{j,r}^0}{m} +y_i^e$ $\begin{array}{rl}\sum_{j\in \{\pm 1\}}\sum_{r = 1}^{m}\frac{\Xi_{j,r,i}^{t,e}}{m}, & \mathrm{we~have} \end{array}$
$$
\hat {y} _ {i} ^ {e} = y _ {i} ^ {e} \cdot \left(\operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} + \operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} + \mathbb {Q} _ {i} ^ {e}\right),
$$
We need the following concentration lemma to control the scale of  $\mathbb{Q}_i^e$ , whose proof is given in Appendix D.2.1.
Lemma D.2. Denote  $\underline{n} \triangleq \min_{e \in \mathcal{E}_{tr}} n_e, n \triangleq \sum_{e \in \mathcal{E}_{tr}} n_e$ . For  $\rho > 0$ , if
$$
\sigma_ {p} ^ {2} = O \left(\min  \left\{d ^ {- 1 / 2} \log^ {- 1 / 2} (n m / \rho), T ^ {- 1} \eta^ {- 1} m \left(d + n \sqrt {d \log (n ^ {2} / \rho)}\right) ^ {- 1} \right\}\right),
$$
then with probability at least  $1 - \rho$ , for any  $e \in \mathcal{E}_{tr}$ ,  $i \in [n_e]$ , it holds that  $|\mathbb{Q}_i^e| = O(\underline{n}^{-1})$ .
Then  $\Delta_{\Lambda}^{t}$  and  $\Delta_{\Gamma}^{t}$  can be explicitly written as
$$
\Delta_ {\Lambda} ^ {t} =
$$
$$
\sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \frac {\operatorname {R a d} (\alpha) _ {i}}{1 + \exp \left\{\operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} \right\} \cdot \exp \left\{\operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} \right\} \cdot \exp \left\{\mathbb {Q} _ {i} ^ {e} \right\}},
$$
$$
\Delta_ {\Gamma} ^ {t} =
$$
$$
\sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \frac {\operatorname {R a d} (\beta_ {e}) _ {i}}{1 + \exp \left\{\operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} \right\} \cdot \exp \left\{\operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} \right\} \cdot \exp \left\{\mathbb {Q} _ {i} ^ {e} \right\}}.
$$
We are going to analyze the convergences of two sequences  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$ . Notice that
$$
\Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t} =
$$
$$
\sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \frac {\operatorname {R a d} (\beta_ {e}) _ {i} + \operatorname {R a d} (\alpha) _ {i}}{1 + \exp \left\{\operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} \right\} \cdot \exp \left\{\operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} \right\} \cdot \exp \left\{\mathbb {Q} _ {i} ^ {e} \right\}},
$$
$$
\Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t} =
$$
$$
\sum_ {e \in \mathcal {E} _ {t r}} \frac {1}{n _ {e} m} \sum_ {i = 1} ^ {n _ {e}} \frac {\operatorname {R a d} (\beta_ {e}) _ {i} - \operatorname {R a d} (\alpha) _ {i}}{1 + \exp \left\{\operatorname {R a d} (\alpha) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Lambda} ^ {k} \right\} \cdot \exp \left\{\operatorname {R a d} (\beta_ {e}) _ {i} \cdot 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \Delta_ {\Gamma} ^ {k} \right\} \cdot \exp \left\{\mathbb {Q} _ {i} ^ {e} \right\}}.
$$
We can further write these two terms as
$$
\begin{array}{l} \Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t} = \sum_{e\in \mathcal{E}_{tr}}\frac{2}{n_{e}m}\sum_{\substack{i\in [n_{e}]\\ \operatorname {Rad}(\beta_{e})_{i} = +1\\ \operatorname {Rad}(\alpha)_{i} = +1}}\frac{1}{1 + \exp\left\{2\eta\cdot\sum_{k = 0}^{t - 1}(\Delta_{\Gamma}^{k} + \Delta_{\Lambda}^{k})\right\} \cdot \exp\left\{\mathbb{Q}_{i}^{e}\right\}} \\ -\sum_{e\in \mathcal{E}_{tr}}\frac{2}{n_{e}m}\sum_{\substack{i\in [n_{e}]\\\operatorname {Rad}(\beta_{e})_{i} = -1\\\operatorname {Rad}(\alpha)_{i} = -1}}\frac{1}{1 + \exp\left\{-2\eta\cdot\sum_{k = 0}^{t - 1}(\Delta_{\Gamma}^{k} + \Delta_{\Lambda}^{k})\right\} \cdot \exp\left\{\mathbb{Q}_{i}^{e}\}}, \\ \end{array}
$$
$$
\begin{array}{l} \Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t} = \sum_{e\in \mathcal{E}_{tr}}\frac{2}{n_{e}m}\sum_{\substack{i\in [n_{e}]\\ \operatorname {Rad}(\beta_{e})_{i} = +1\\ \operatorname {Rad}(\alpha)_{i} = -1}}\frac{1}{1 + \exp\left\{2\eta\cdot\sum_{k = 0}^{t - 1}(\Delta_{\Gamma}^{k} - \Delta_{\Lambda}^{k})\right\} \cdot \exp\left\{\mathbb{Q}_{i}^{e}\right\}} \\ -\sum_{e\in \mathcal{E}_{tr}}\frac{2}{n_{e}m}\sum_{\substack{i\in [n_{e}]\\ \operatorname {Rad}(\beta_{e})_{i} = -1\\ \operatorname {Rad}(\alpha)_{i} = +1}}\frac{1}{1 + \exp\left\{-2\eta\cdot\sum_{k = 0}^{t - 1}(\Delta_{\Gamma}^{k} - \Delta_{\Lambda}^{k})\right\} \cdot \exp\left\{\mathbb{Q}_{i}^{e}\right\}}. \\ \end{array}
$$
According to Lemma D.2, for all  $e \in \mathcal{E}_{tr}, i \in [n_e], \rho > 0$ , letting  $\delta \triangleq \exp \{O(\underline{n}^{-1})\} - 1$ , we have  $1 + \delta \geq \exp \{\mathbb{Q}_i^e\} \geq (1 + \delta)^{-1}$  with probability at least  $1 - \rho$ . Let  $C_{j\ell}^{e} \triangleq |\{i | \operatorname{Rad}(\alpha)_i = j, \operatorname{Rad}(\beta_e)_i = \ell, i \in \mathcal{E}_e\}|$  for any  $j \in \{\pm 1\}, \ell \in \{\pm 1\}, e \in \mathcal{E}_{tr}$ , and then define  $\overline{C}_{j\ell} \triangleq \sum_{e \in \mathcal{E}_{tr}} \frac{C_{j\ell}^e}{n_e}$ .
We can upper bound and formulate  $\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}$  and  $\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}$  as
$$
\begin{array}{l} \Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t} \leq \\ \frac {2}{m} \left(\frac {\overline {{C}} _ {+ 1 + 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta) ^ {- 1}} - \frac {\overline {{C}} _ {- 1 - 1}}{1 + \exp \left\{- 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)}\right) \\ = \frac {2}{m} \cdot \frac {\bar {C} _ {+ 1 + 1} (1 + \delta) - \bar {C} _ {- 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\}}{1 + \delta + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\}}, (12) \\ \Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t} \leq \\ \frac {2}{m} \left(\frac {\overline {{C}} _ {- 1 + 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta) ^ {- 1}} - \frac {\overline {{C}} _ {+ 1 - 1}}{1 + \exp \left\{- 2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)}\right) \\ = \frac {2}{m} \cdot \frac {\bar {C} _ {- 1 + 1} (1 + \delta) - \bar {C} _ {+ 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\}}{1 + \delta + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\}}. (13) \\ \end{array}
$$
Based on similar arguments, we can also establish lower bounds for these two terms,
$$
\Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t} \geq \frac {2}{m} \cdot \frac {\overline {{C}} _ {+ 1 + 1} - \overline {{C}} _ {- 1 - 1} (1 + \delta) \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)}, \tag {14}
$$
$$
\Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t} \geq \frac {2}{m} \cdot \frac {\overline {{C}} _ {- 1 + 1} - \overline {{C}} _ {+ 1 - 1} (1 + \delta) \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)}. \tag {15}
$$
The upper and lower bounds (12), (13), (14) and (15) imply that the convergences of  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  are determined by recursive equations of the form  $\mathcal{Q}^t = \frac{C_1 - C_2 \cdot \exp\{\eta \sum_{k=0}^{t-1} \mathcal{Q}^k\}}{1 + C_3 \cdot \exp\{\eta \sum_{k=0}^{t-1} \mathcal{Q}^k\}}$ . We first establish that with suitably chosen  $\eta$ , the sequences  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  are guaranteed to be positive. Observed that for the  $\mathcal{Q}^t$ -type recursive equation, the sign of  $\mathcal{Q}^0$  is independent of  $\eta$ , and only determined by the constants  $C_1, C_2, C_3$ . At iteration 0, (14) and (15) give
$$
\Delta_ {\Gamma} ^ {0} + \Delta_ {\Lambda} ^ {0} \geq \frac {2}{m} \cdot \frac {\bar {C} _ {+ 1 + 1} - \bar {C} _ {- 1 - 1} (1 + \delta)}{2 + \delta}, \tag {16}
$$
$$
\Delta_ {\Gamma} ^ {0} - \Delta_ {\Lambda} ^ {0} \geq \frac {2}{m} \cdot \frac {\bar {C} _ {- 1 + 1} - \bar {C} _ {+ 1 - 1} (1 + \delta)}{2 + \delta}. \tag {17}
$$
To proceed, we need the following concentration lemma to control the deviations of the constants  $\overline{C}_{+1 + 1}$ ,  $\overline{C}_{+1 - 1}$ ,  $\overline{C}_{-1 + 1}$  and  $\overline{C}_{-1 - 1}$  from their expectations, whose proof is given in Appendix D.2.2.
Lemma D.3. For  $\rho > 0$ , considering two environments and denoting  $\epsilon_C \triangleq \sqrt{\frac{2\log(16 / \rho)}{\underline{n}}}$ , with probability at least  $1 - \rho$ , we have
$$
\left| \overline {{C}} _ {+ 1 + 1} - (1 - \alpha) (2 - \beta_ {1} - \beta_ {2}) \right| \leq \epsilon_ {C},
$$
$$
\left| \overline {{C}} _ {+ 1 - 1} - (1 - \alpha) \left(\beta_ {1} + \beta_ {2}\right) \right| \leq \epsilon_ {C}, \tag {18}
$$
$$
\left| \overline {{C}} _ {- 1 + 1} - \alpha (2 - \beta_ {1} - \beta_ {2}) \right| \leq \epsilon_ {C},
$$
$$
\left| \bar {C} _ {- 1 - 1} - \alpha \left(\beta_ {1} + \beta_ {2}\right) \right| \leq \epsilon_ {C}.
$$
Using Lemma D.3, with probability at least  $1 - \rho$ , the constants  $\overline{C}_{+1 + 1}$ ,  $\overline{C}_{+1 - 1}$ ,  $\overline{C}_{-1 + 1}$  and  $\overline{C}_{-1 - 1}$  are close to their expectations.
Based on our assumptions that
$$
\alpha , \beta_ {1}, \beta_ {2} <   \frac {1 - \epsilon_ {C} - \delta (\frac {1}{4} + \frac {\epsilon_ {C}}{2})}{2} \quad (\alpha , \beta_ {1}, \beta_ {2} \text {a r e s u f f i c i e n t l y s m a l l e r t h a n} \frac {1}{2}),
$$
$$
\alpha > \frac {\beta_ {1} + \beta_ {2}}{2} + \epsilon_ {C} + \frac {\delta (1 + \epsilon_ {C})}{2} \quad (\alpha \text {i s s u f f i c i e n t l y l a r g e r t h a n} \frac {\beta_ {1} + \beta_ {2}}{2}),
$$
it can be verified that with probability at least  $1 - 2\rho$ ,  $\Delta_{\Gamma}^{0} + \Delta_{\Lambda}^{0} > 0$ ,  $\Delta_{\Gamma}^{0} - \Delta_{\Lambda}^{0} > 0$ .
Then, at iteration 1, from (14) and (15), we see that as long as we require
$$
\eta <   \min \left\{\frac {1}{2 (\Delta_ {\Gamma} ^ {0} + \Delta_ {\Lambda} ^ {0})} \log \frac {\overline {{C}} _ {+ 1 + 1}}{\overline {{C}} _ {- 1 - 1} (1 + \delta)}, \frac {1}{2 (\Delta_ {\Gamma} ^ {0} - \Delta_ {\Lambda} ^ {0})} \log \frac {\overline {{C}} _ {- 1 + 1}}{\overline {{C}} _ {+ 1 - 1} (1 + \delta)} \right\},
$$
it holds that  $\Delta_{\Gamma}^{1} + \Delta_{\Lambda}^{1} > 0, \Delta_{\Gamma}^{1} - \Delta_{\Lambda}^{1} > 0$ . By recursively applying this argument, we see the requirement for  $\eta$  to ensure that  $\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t} > 0$  and  $\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t} > 0$  for any  $t \in \{0, \dots, T\}$  is
$$
\eta <   \min  \left\{\frac {1}{2 \sum_ {k = 0} ^ {T - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right)} \log \frac {\overline {{C}} _ {+ 1 + 1}}{\overline {{C}} _ {- 1 - 1} (1 + \delta)}, \frac {1}{2 \sum_ {k = 0} ^ {T - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right)} \log \frac {\overline {{C}} _ {- 1 + 1}}{\overline {{C}} _ {+ 1 - 1} (1 + \delta)} \right\}. \tag {19}
$$
In other words, for the  $\mathcal{Q}^t$ -type recursive equation, as long as  $\mathcal{Q}^0 \geq 0$ , there always exists a sufficiently small  $\eta$  to guarantee that the whole sequence  $\{\mathcal{Q}^t\}$  is positive. From now on, we will focus on the case where the two sequences  $\{\Delta_{\Gamma}^t + \Delta_{\Lambda}^t\}$  and  $\{\Delta_{\Gamma}^t - \Delta_{\Lambda}^t\}$  decrease to an  $\epsilon_{\Delta} > 0$  error, i.e.,  $\min_{t \in \{0, \dots, T\}} \{\Delta_{\Gamma}^t + \Delta_{\Lambda}^t, \Delta_{\Gamma}^t - \Delta_{\Lambda}^t\} = \epsilon_{\Delta}$ .
Then, we show that the two sequences  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  decrease monotonically, which thus leads to a more refined upper bound for  $\eta$  at (19). Inspect the upper bounds (12), (13) at iteration  $t + 1$ , which can be written as
$$
\Delta_ {\Gamma} ^ {t + 1} + \Delta_ {\Lambda} ^ {t + 1} \leq
$$
$$
\frac {2}{m} \cdot \frac {\overline {{C}} _ {+ 1 + 1} - \overline {{C}} _ {- 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t}\right) \right\} (1 + \delta) ^ {- 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t}\right) \right\} (  1 + \delta) ^ {- 1}} \triangleq \spadesuit^ {t + 1},
$$
$$
\Delta_ {\Gamma} ^ {t + 1} - \Delta_ {\Lambda} ^ {t + 1} \leq
$$
$$
\frac {2}{m} \cdot \frac {\overline {{C}} _ {- 1 + 1} - \overline {{C}} _ {+ 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t}\right) \right\} (1 + \delta) ^ {- 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t}\right) \right\} (  1 + \delta) ^ {- 1}} \triangleq \clubsuit^ {t + 1}.
$$
Requiring that  $\eta >\max \left\{\frac{1}{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}}\log (1 + \delta),\frac{1}{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}}\log (1 + \delta)\right\},\forall t\in \{0,\ldots ,T\} \Rightarrow \eta>$ $\epsilon_{\Delta}^{-1}\log (1 + \delta)$ , we have
$$
\begin{array}{l} \spadesuit^ {t + 1} <   \frac {2}{m} \cdot \frac {\overline {{C}} _ {+ 1 + 1} - \overline {{C}} _ {- 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t}\right) \right\} (1 + \delta) ^ {- 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} + \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)} \\ <   \Delta_ {\Gamma} ^ {t} + \Delta_ {\Lambda} ^ {t}, \\ \end{array}
$$
$$
\begin{array}{l} \clubsuit^ {t + 1} <   \frac {2}{m} \cdot \frac {\overline {{C}} _ {- 1 + 1} - \overline {{C}} _ {+ 1 - 1} \cdot \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot \exp \left\{2 \eta \cdot \left(\Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t}\right) \right\} (1 + \delta) ^ {- 1}}{1 + \exp \left\{2 \eta \cdot \sum_ {k = 0} ^ {t - 1} \left(\Delta_ {\Gamma} ^ {k} - \Delta_ {\Lambda} ^ {k}\right) \right\} \cdot (1 + \delta)} \\ <   \Delta_ {\Gamma} ^ {t} - \Delta_ {\Lambda} ^ {t}, \\ \end{array}
$$
where the last inequalities use the lower bounds (14) and (15).
Based on the above discussion and (19), we can now clarify the requirements of  $\eta$  for the sequences  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  to be positive and monotonically decreasing:
$$
\begin{array}{l} \epsilon_ {\Delta} ^ {- 1} \log (1 + \delta) <   \eta <   \min  \left\{\frac {m (2 + \delta)}{4 T (\bar {C} _ {+ 1 + 1} (1 + \delta) - \bar {C} _ {- 1 - 1})} \log \frac {\bar {C} _ {+ 1 + 1}}{\bar {C} _ {- 1 - 1} (1 + \delta)}, \right. \tag {20} \\ \left. \frac {m (2 + \delta)}{4 T (\overline {{C}} _ {- 1 + 1} (1 + \delta) - \overline {{C}} _ {+ 1 - 1})} \log \frac {\overline {{C}} _ {- 1 + 1}}{\overline {{C}} _ {+ 1 - 1} (1 + \delta)} \right\}, \\ \end{array}
$$
which uses the upper bounds (12) and (13) at iteration 0. The constants  $\overline{C}_{+1 + 1}$ ,  $\overline{C}_{+1 - 1}$ ,  $\overline{C}_{-1 + 1}$  and  $\overline{C}_{-1 - 1}$  can be substituted using the concentration bounds at (18) to generate an upper bound for  $\eta$  that only involves  $\alpha, \beta_{1}, \beta_{2}, m, \delta, T, \epsilon_{C}$ . Here we omit the precise upper bound for clarity. Note that
the left hand side of (20) approaches 0 if  $\delta \to 0$ , which means that there exists a constant choice of  $\eta$  in (20) if  $\underline{n}$  is sufficiently large in Lemma D.2 and D.3.
To conclude, in view of (11), the convergences of the sequences  $\{\Delta_{\Gamma}^{t} + \Delta_{\Lambda}^{t}\}$  and  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  imply that  $\Lambda_{j,r}^{t}$  and  $\Gamma_{j,r}^{t}$  are converging, and the positive sequence  $\{\Delta_{\Gamma}^{t} - \Delta_{\Lambda}^{t}\}$  indicates that the increment of the spurious feature  $\Gamma_{j,r}^{t+1} - \Gamma_{j,r}^{t}$  is larger than that of the invariant feature  $\Lambda_{j,r}^{t+1} - \Lambda_{j,r}^{t}$  at any iteration  $t \in \{0, \dots, T-1\}$ .
D.2.1 Proof of Lemma D.2
First, we recall some concentration inequalities for sub-Gaussian random variables. Since  $\pmb{\xi}_i^e\sim \mathcal{N}(0,\sigma_p^2\cdot (\mathbf{I}_d - \mathbf{v}_1\mathbf{v}_1^\top -\mathbf{v}_2\mathbf{v}_2^\top))$  , for  $(i^{\prime},e^{\prime})\neq (i,e)$  , using Bernstein's inequality for sub-exponential random variables, we have for sufficiently small  $a\geq 0$
$$
\Pr \left\{\left| \left\langle \boldsymbol {\xi} _ {i} ^ {e}, \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \right\rangle \right| \geq a \right\} \leq 2 \exp \left\{- \frac {a ^ {2}}{4 \sigma_ {p} ^ {4} (d - 2)} \right\},
$$
$$
\Pr \left\{\left| \| \boldsymbol {\xi} _ {i} ^ {e} \| _ {2} ^ {2} - \sigma_ {p} ^ {2} (d - 2) \right| \geq a \right\} \leq 2 \exp \left\{- \frac {a ^ {2}}{5 1 2 \sigma_ {p} ^ {4} (d - 2)} \right\}.
$$
Moreover, for  $\xi_r \sim \mathcal{N}(0, \sigma_0^2)$  (indicating the initial weights  $\mathbf{w}_{j,r}^0$ ), the standard Gaussian tail gives
$$
\Pr \left\{\left| \frac {1}{m} \sum_ {r = 1} ^ {m} \xi_ {r} \right| \geq a \right\} \leq 2 \exp \left\{- \frac {m a ^ {2}}{2 \sigma_ {0} ^ {2}} \right\}.
$$
Denote  $n \triangleq \sum_{e \in \mathcal{E}_{tr}} n_e, \underline{n} \triangleq \min_{e \in \mathcal{E}_{tr}} n_e$ , by properly choosing  $a$  for each tail bound and applying a union bound, we can conclude that for  $\rho > 0$ , with probability at least  $1 - \rho$ , it holds that  $\forall i, e, i', e', r$ ,
$$
| \langle \boldsymbol {\xi} _ {i} ^ {e}, \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \rangle | \leq 2 \sigma_ {p} ^ {2} \sqrt {(d - 2) \log \frac {8 n ^ {2}}{\rho}}, \| \boldsymbol {\xi} _ {i} ^ {e} \| _ {2} ^ {2} \leq \sigma_ {p} ^ {2} (d - 2) + 1 6 \sigma_ {p} ^ {2} \sqrt {2 (d - 2) \log \frac {8 n}{\rho}},
$$
$$
\left| \frac {1}{m} \sum_ {r = 1} ^ {m} \xi_ {r} \right| \leq \sigma_ {0} \sqrt {\frac {2}{m} \log \frac {3 2 m}{\rho}}, \quad | \langle \boldsymbol {\xi} _ {r}, \boldsymbol {\xi} _ {i ^ {\prime}} ^ {e ^ {\prime}} \rangle | \leq 2 \sigma_ {p} \sigma_ {0} \sqrt {(d - 2) \log \frac {1 6 n m}{\rho}}.
$$
We start with bound the growth of  $\Xi_{j,r,i}^{t,e}$ . By bounding the update rule (10), with probability at least  $1 - \rho$ , we have