SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} y _ {i} f (t, \mathbf {x} _ {i}) = \sum_ {j \sim i} \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) - \sum_ {j \not \sim i} \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) \\ \geq \sum_ {j \sim i} \left(\phi \left(\left\langle \mathbf {w} _ {j} ^ {(t _ {0})}, \mathbf {x} _ {i} \right\rangle\right) + \eta T _ {i j} (t _ {0}, t) - \eta (\gamma + \rho) B _ {j} (t _ {0}, t) - \eta \phi (\rho - \gamma) G _ {j} ^ {(i)} (t _ {0}, t)\right) \\ - \sum_ {j \neq i} \left(\phi \left(\left\langle \mathbf {w} _ {j} ^ {\left(t _ {0}\right)}, \mathbf {x} _ {i} \right\rangle\right) - \eta T _ {i j} (t _ {0}, t) + \eta (\gamma + \rho) B _ {j} (t _ {0}, t) + \eta \phi (\rho - \gamma) G _ {j} ^ {(i)} (t _ {0}, t) + \eta\right) \\ = y _ {i} f \left(t _ {0}, \mathbf {x} _ {i}\right) + \eta \left(T _ {i} \left(t _ {0}, t\right) - (\gamma + \rho) B \left(t _ {0}, t\right) - \phi (\rho - \gamma) G ^ {(i)} \left(t _ {0}, t\right) - m\right). \\ \end{array}
$$
For Statement 2
$$
\begin{array}{l} y _ {i} f (t, \mathbf {x} _ {i}) = \sum_ {j \not \sim i} \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) - \sum_ {j \not \sim i} \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) \\ \geq \sum_ {j \not \sim i} \left(\phi (\langle \mathbf {w} _ {j} ^ {(t _ {0})}, \mathbf {x} _ {i} \rangle) + \eta T _ {i j} (t _ {0}, t) - \eta (\gamma + \rho) G _ {j} (t _ {0}, t) - \eta \phi (\rho - \gamma) B _ {j} ^ {(i)} (t _ {0}, t)\right) \\ - \sum_ {j \not \sim i} \left(\phi \left(\left\langle \mathbf {w} _ {j} ^ {\left(t _ {0}\right)}, \mathbf {x} _ {i} \right\rangle\right) - \eta T _ {i j} \left(t _ {0}, t\right) + \eta (\gamma + \rho) G _ {j} \left(t _ {0}, t\right) + \eta \phi (\rho - \gamma) B _ {j} ^ {(i)} \left(t _ {0}, t\right) + \eta\right) \\ \geq y _ {i} f (t _ {0}, \mathbf {x} _ {i}) + \eta (T _ {i} (t _ {0}, t) - (\gamma + \rho) G (t _ {0}, t) - \phi (\rho - \gamma) B ^ {(i)} (t _ {0}, t) - m). \\ \end{array}
$$
The following lemma bounds the number of updates of corrupt and clean points in an interval of iterations in terms of their hinge loss at the beginning of the interval as well as the number of clean and corrupt updates.
Lemma B.5. Let  $t \geq t_0$ . For  $i \in S_T$ ,
$$
T _ {i} (t _ {0}, t) \leq \frac {\ell (t _ {0} , \mathbf {x} _ {i})}{\eta} + (\gamma + \rho) B (t _ {0}, t) + \phi (\rho - \gamma) G ^ {(i)} (t _ {0}, t) + 3 m.
$$
For  $i\in S_F$
$$
T _ {i} (t _ {0}, t) \leq \frac {\ell (t _ {0} , \mathbf {x} _ {i})}{\eta} + (\gamma + \rho) G (t _ {0}, t) + \phi (\rho - \gamma) B ^ {(i)} (t _ {0}, t) + 3 m.
$$
Proof. We will show this for  $i \in S_T$ ; the  $i \in S_F$  case is analogous but with the roles of corrupt and clean points reversed. We proceed by induction on  $t$  and assume  $\ell(t_0, \mathbf{x}_i) \leq a$ . If  $t = t_0$  this holds trivially because the left-hand side is zero and the right-hand side is positive. Otherwise, assume the inequality holds at iteration  $t$ . By Lemma B.4 and our assumption on  $\ell(t_0, \mathbf{x}_i)$ ,
$$
\begin{array}{l} y _ {i} f (t, \mathbf {x} _ {i}) \geq y _ {i} f (t _ {0}, \mathbf {x} _ {i}) + \eta (T _ {i} (t _ {0}, t) - (\gamma + \rho) B (t _ {0}, t) - \phi (\rho - \gamma) G ^ {(i)} (t _ {0}, t) - m) \\ \geq (1 - a) + \eta (T _ {i} (t _ {0}, t) - (\gamma + \rho) B (t _ {0}, t) - \phi (\rho - \gamma) G ^ {(i)} (t _ {0}, t) - m). \\ \end{array}
$$
We consider two cases:
1. If  $\eta (T_i(t_0,t) - (\gamma +\rho)B(t_0,t) - \phi (\rho -\gamma)G^{(i)}(t_0,t) - m)\geq a$  then we see that  $\ell (t,\mathbf{x}_i) = 0$  Therefore,
$$
\begin{array}{l} T _ {i} \left(t _ {0}, t + 1\right) = T _ {i} \left(t _ {0}, t\right) \\ \leq \frac {a}{\eta} + (\gamma + \rho) B (t _ {0}, t) + 3 m \\ \leq \frac {a}{\eta} + (\gamma + \rho) B (t _ {0}, t + 1) + 3 m. \\ \end{array}
$$
2. Otherwise,  $T_{i}(t_{0}, t) \leq \frac{a}{\eta} + (\gamma + \rho) B(t_{0}, t) + \phi(\rho - \gamma) G^{(i)}(t_{0}, t) + m$ . Since there are only  $2m$  neurons, we bound
$$
\begin{array}{l} T _ {i} (t _ {0}, t + 1) = T _ {i} (t _ {0}, t) + 2 m \\ \leq \left(\frac {a}{\eta} + (\gamma + \rho) B (t _ {0}, t) + \phi (\rho - \gamma) G ^ {(i)} (t _ {0}, t) + m\right) + 2 m \\ \leq \frac {a}{\eta} + (\gamma + \rho) B (t _ {0}, t + 1) + \phi (\rho - \gamma) G ^ {(i)} (t _ {0}, t) + 3 m. \\ \end{array}
$$
Appendix C Benign overfitting
Assumption 2. With  $\delta, \rho \in (0,1)$  and  $C$  a generic, positive constant, then we assume the following conditions on the data and model hyperparameters.
1.  $n\geq C\log (\frac{3}{\delta})$
2.  $m\geq C\log \left(\frac{6k}{\delta}\right)$
3.  $d \geq \max \left\{3, 3\rho^{-2}\ln \left(\frac{9n^2}{\delta}\right)\right\}.$
4.  $k < \frac{n}{100}$
5.  $5\sqrt{\frac{\ln(9n^2 / \delta)}{d}} \leq \gamma \leq \frac{4}{5n}$ ,
6.  $\lambda_w < \eta$
In addition to the assumptions detailed in Assumption 2, in our analysis we use three further conditions.
Assumption 3. Let  $\rho \in (0,1)$  satisfy  $\gamma \geq 5\rho$ . In addition to the assumptions detailed in Assumption 2, assume that the following conditions hold.
1.  $|\Gamma_p| > 0.99m$  for  $p \in \{-1, 1\}$ .
2. For all  $i\in S_F$  there is  $j\in \Gamma_{y_i}$  such that  $i\in \mathcal{A}_j^{(0)}$
3. For all  $i, l \in [2n]$ ,  $i \neq l$  then  $|\langle \mathbf{n}_i, \mathbf{n}_l \rangle| \leq \frac{\rho}{1 - \gamma}$ .
We remark that under these conditions then for sufficiently large  $n$  the inequalities  $\rho \leq \min \left\{\frac{n - 3k}{n + k}\gamma, \frac{1}{6(n - k)}\right\}$  and  $\gamma + \rho < \min \left\{\sqrt{\frac{1}{4(n - k)}k}, \frac{1}{n - k}, \frac{1}{99k}, \frac{1}{100}\right\}$  are satisfied. As shown in the following lemma, these three additional conditions hold with high probability over the randomness of the initialization and training sample.
Lemma C.1. There exists a positive constant  $C$  such that for any  $\delta \in (0,1)$  if  $n\geq C\log (\frac{3}{\delta})$  then the extra conditions of Assumption 3 hold with probability at least  $1 - \delta$
Proof. Using Lemma A.3, under the Assumption 3 for sufficiently large  $n$  there exists a positive constant  $c$  such that the probability the first condition does not hold is at most  $\exp(-cn)$ . Alternatively,
setting  $\delta \geq 3\exp (-cn)$  and rearranging, as long as  $n\geq C\log \left(\frac{3}{\delta}\right)$  then the probability the first condition does not hold is at most  $\frac{\delta}{3}$ . Conditioned on the first event, using Lemma A.6 then if  $n\geq C\log \left(\frac{3}{\delta}\right)$  and  $m\geq C\log \left(\frac{6k}{\delta}\right)$  then the probability condition two does not hold is also at most  $\frac{\delta}{3}$ . Therefore the probability that the first two events hold is at least  $(1 - \frac{\delta}{3})^2\geq 1 - \frac{2\delta}{3}$ . For the third condition, noting  $\frac{\rho}{1 - \gamma} >\rho$  and  $\rho \leq \gamma /5$ , then by Lemma A.1 the probability the third condition does not hold is also at most  $\frac{\delta}{3}$ . Therefore, we conclude that all three properties hold with probability at least  $1 - \delta$ .
C.1 Proof of Lemma 3.2
The following lemma characterizes an iteration independent upper bound on the number of clean and corrupt updates. This result will prove significant for proving the termination of GD.
Lemma C.2 (Lemma 3.2). Assume Assumption 3 holds. Suppose further that at some epoch  $t_0$  the loss of every clean point is bounded above by  $a \in \mathbb{R}_{\geq 0}$ , while the loss of every corrupted point is bounded above by  $b \in \mathbb{R}_{\geq 0}$ . Then the total number of updates which occurs after this epoch is upper bounded as follows,
$$
G (t _ {0}, t) \leq \frac {2 (n - k)}{1 - 4 k (n - k) (\gamma + \rho) ^ {2}} \left(\frac {a}{\eta} + 3 m + 2 k (\gamma + \rho) \left(\frac {b}{\eta} + 3 m\right)\right),
$$
$$
B (t _ {0}, t) \leq \frac {2 k}{1 - 4 k (n - k) (\gamma + \rho) ^ {2}} \left(\frac {b}{\eta} + 3 m + 2 (n - k) (\gamma + \rho) \left(\frac {a}{\eta} + 3 m\right)\right)
$$
for all  $t \geq t_0$ .
Proof. From Lemma B.5,  $\rho \leq \gamma$ , and the assumption on  $a$  and  $b$ ,
$$
G (t _ {0}, t) = \sum_ {i \in \mathcal {S} _ {T}} T _ {i} (t _ {0}, t) \leq 2 (n - k) \left(\frac {a}{\eta} + (\gamma + \rho) B (t _ {0}, t) + 3 m\right),
$$
$$
B (t _ {0}, t) = \sum_ {i \in \mathcal {S} _ {F}} T _ {i} (t _ {0}, t) \leq 2 k \left(\frac {b}{\eta} + (\gamma + \rho) G (t _ {0}, t) + 3 m\right).
$$
Substituting these bounds into each other, and as  $\gamma +\rho < (4(n - k)k)^{-1 / 2}$  under Assumption 3, we arrive at the iteration independent bound on the number of updates as claimed in the statement of the theorem.
C.2 Early training and proof of Lemma 3.3
Lemma C.3. Under Assumption 3, then for  $i \in S_T$  and  $j \in \Gamma$  it follows that  $\langle \mathbf{w}_j^{(1)}, \mathbf{x}_i \rangle > 0$  iff  $i \sim j$ . For  $i \in S_F$ ,  $j \in \Theta_p$ , and  $i \not\sim j$  it follows that  $\langle \mathbf{w}_j^{(1)}, \mathbf{x}_i \rangle > 0$  if  $i \in \mathcal{A}_j^{(0)}$ .
Proof. Suppose  $j \in \Gamma$ ,  $i \sim j$ ,  $i \in S_T$ . Recall from definition of  $\Gamma_p$  that  $G_j^{(i)}(0,1)(\gamma - \rho) - B_j^{(i)}(0,1)(\gamma + \rho) \geq \frac{2\lambda_w}{\eta}$ . Using Lemma B.2
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \rangle \geq \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle + \eta \left(T _ {i j} (0, 1) + G _ {j} ^ {(i)} (0, 1) (\gamma - \rho) - B _ {j} ^ {(i)} (0, 1) (\gamma + \rho)\right) \\ > \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle + \eta (G _ {j} (0, 1) (\gamma - \rho) - B _ {j} (0, 1) (\gamma + \rho)) \\ \geq \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle + 2 \lambda_ {w} \\ > \lambda_ {w}. \\ \end{array}
$$
On the other hand, if  $i \neq j$  then again from Lemma B.2
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle - \eta \left(T _ {i j} (0, 1) + G _ {j} ^ {(i)} (0, 1) (\gamma - \rho) - B _ {j} ^ {(i)} (0, 1) (\gamma + \rho)\right) \\ \leq \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle - \eta (G _ {j} (0, 1) (\gamma - \rho) - B _ {j} (0, 1) (\gamma + \rho)) \\ \leq - \lambda_ {w}. \\ \end{array}
$$
Now consider  $i \in S_F, i \not\sim j$ , and  $i \in \mathcal{A}_j^{(0)}$ . By Lemma B.2 and the definition of  $\Theta$
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \rangle \geq \langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle + \eta (T _ {i j} (0, 1) + B _ {j} ^ {(i)} (0, 1) (\gamma - \rho) - G _ {j} ^ {(i)} (0, 1) (\gamma + \rho)) \\ > \eta (1 - \gamma + \rho) + \eta \left(B _ {j} (0, 1) (\gamma - \rho) - G _ {j} (0, 1) (\gamma + \rho)\right) \\ \geq 0. \\ \end{array}
$$
Lemma C.4 (Lemma 3.3). Suppose Assumption 3 holds. Let  $j \in \Gamma_p$ . Let  $0 < t < \mathcal{T}_0$ . A point  $i \in \mathcal{A}_j^{(t)}$  if one of the following conditions hold:
1.  $i\in S_T$  and  $i\sim j$
2.  $i\in \mathcal{S}_F,i\not\sim j,$  and  $i\in \mathcal{A}_j^{(1)}$
Furthermore, if one of the following conditions hold, then  $i \notin \mathcal{A}_j^{(t)}$ :
1.  $i\in S_T$  and  $i\not\sim j$
2.  $i\in \mathcal{S}_F,i\not\sim j,$  and  $i\not\in\mathcal{A}_j^{(1)}$
Proof. We proceed by induction. For  $t = 1$ , the  $i \in S_T$  case was shown in Lemma C.3 and the  $i \in S_F$ ,  $i \not\sim j$  case is clear. Now, suppose the lemma holds for iteration  $t$  and consider iteration  $t + 1$ . First let  $i \in S_F$ ,  $i \not\sim j$ . If  $i \in A_j^{(1)}$  then
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \geq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle + \eta \left(T _ {i j} (t, t + 1) - G _ {j} ^ {(i)} (t, t + 1) (\gamma + \rho) + B _ {j} ^ {(i)} (t, t + 1) (\gamma - \rho)\right) \\ > \eta (1 - (n - k) (\gamma + \rho)) \\ \geq 0. \\ \end{array}
$$
Here the first line is Lemma B.2, the second line comes from the inductive hypothesis, and the third line comes from  $(\gamma + \rho) < \frac{1}{n - k}$  (Assumption 3). If  $i \notin \mathcal{A}_j^{(1)}$  then
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle + \eta (T _ {i j} (t, t + 1) - G _ {j} ^ {(i)} (t, t + 1) (\gamma - \rho) + B _ {j} ^ {(i)} (t, t + 1) (\gamma + \rho)) \\ <   \eta (- (n - k) (\gamma - \rho) + 2 k (\gamma + \rho)) \\ = - \eta ((n - 3 k) \gamma - (n + k) \rho) \\ <   0. \\ \end{array}
$$
Again, the first line is Lemma B.2, the second line uses the inductive hypothesis, and the fourth line uses  $\rho \leq \frac{n - 3k}{n + k}\gamma$  (Assumption 3).
Now, let  $i \in S_T$ . We again use, in order, Lemma B.2, the inductive hypothesis, and  $\rho \leq \frac{n - 3k}{n + k}\gamma$ . If  $i \sim j$  then
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \geq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle + \eta (T _ {i j} (t, t + 1) + G _ {j} ^ {(i)} (t, t + 1) (\gamma - \rho) - B _ {j} ^ {(i)} (t, t + 1) (\gamma + \rho)) \\ > \eta (1 + \rho - \gamma) + \eta ((n - k) (\gamma - \rho) - 2 k (\gamma + \rho)) \\ > 0. \\ \end{array}
$$
If  $i \neq j$  then
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle - \eta (T _ {i j} (t, t + 1) + G _ {j} ^ {(i)} (t, t + 1) (\gamma - \rho) - B _ {j} ^ {(i)} (t, t + 1) (\gamma + \rho)) \\ <   - \eta ((n - k) (\gamma - \rho) - 2 k (\gamma + \rho)) \\ = - \eta ((n - 3 k) \gamma - (n + k) \rho) \\ \leq 0. \\ \end{array}
$$
Lemma C.5. Suppose Assumption 3 holds. For all  $t_0 \leq t_1 < T_0$ ,
$$
G _ {j} (t _ {0}, t _ {1}) \leq (n - k) (t _ {1} - t _ {0} + 2) + \frac {1}{\gamma - \rho}
$$
Proof. First we claim that for all  $i \in S_T$ ,  $j \not\sim i$ , and  $t < \mathcal{T}_0$
$$
\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq \lambda_ {w} + 2 \eta k (\gamma + \rho).
$$
We prove the claim by induction. The base case  $t = 0$  follows because  $\langle \mathbf{w}_j^{(0)},\mathbf{x}_i\rangle \leq \lambda_w$ . Now suppose it is true at iteration  $t$ . If  $\langle \mathbf{w}_j^{(t)},\mathbf{x}_i\rangle >0$  then by Lemma B.2,
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle - \eta T _ {i j} (t, t + 1) + \eta (\gamma + \rho) B _ {j} (t, t + 1) \\ \leq \left\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \right\rangle - \eta (1 - 2 k (\gamma + \rho)) \\ \leq \left\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \right\rangle \\ \end{array}
$$
using  $\gamma +\rho <  \frac{1}{2k}$  . From this, the claim follows. Otherwise,
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle - \eta T _ {i j} (t, t + 1) + \eta (\gamma + \rho) B _ {j} (t, t + 1) \\ \leq 2 \eta k (\gamma + \rho). \\ \end{array}
$$
We now turn to the statement of the lemma, again proceeding by induction. The base case  $t_1 = t_0$  is clear. Otherwise, we consider two cases:
1. If  $G_{j}(t_{0}, t_{1}) > \frac{1 + 2k(t_{1} - t_{0} + 1)(\gamma + \rho)}{\gamma - \rho}$  then for all  $i \in S_{T}$  and  $j \not\sim i$ , by Lemma B.2,
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t _ {1})}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t _ {0})}, \mathbf {x} _ {i} \rangle - \eta (T _ {i j} (t _ {0}, t _ {1}) + G _ {j} ^ {(i)} (t _ {0}, t _ {1}) (\gamma - \rho) - B _ {j} ^ {(i)} (t _ {0}, t _ {1}) (\gamma + \rho)) \\ \leq \left\langle \mathbf {w} _ {j} ^ {(t _ {0})}, \mathbf {x} _ {i} \right\rangle - \eta \left(G _ {j} (t _ {0}, t _ {1}) (\gamma - \rho) - B _ {j} (t _ {0}, t _ {1}) (\gamma + \rho)\right) \\ <   (\lambda_ {w} + 2 \eta k (\gamma + \rho)) + 2 \eta k (t _ {1} - t _ {0}) (\gamma + \rho) - \eta G _ {j} (t _ {0}, t _ {1}) (\gamma - \rho) \\ \leq 0 \\ \end{array}
$$
by the claim and  $\lambda_w < \eta$  (Assumption 3). Therefore,  $G_j(t_0, t_1 + 1) \leq G_j(t_0, t_1) + (n - k)$ .
2. If  $G_{j}(t_{0}, t_{1}) \leq \frac{1 + 2k(t_{1} - t_{0} + 1)(\gamma + \rho)}{\gamma - \rho}$ , then
$$
\begin{array}{l} G _ {j} \left(t _ {0}, t _ {1} + 1\right) \leq \frac {1 + 2 k \left(t _ {1} - t _ {0} + 1\right) (\gamma + \rho)}{\gamma - \rho} + 2 (n - k) \\ \leq \frac {1}{\gamma - \rho} + \frac {2 k (t _ {1} - t _ {0} + 1) (n - k)}{2 k} + 2 (n - k) \\ \leq \frac {1}{\gamma - \rho} + (t _ {1} - t _ {0} + 3) (n - k). \\ \end{array}
$$
Lemma C.6. Suppose Assumption 3 holds. For all  $t < \mathcal{T}_0$ ,  $i \in S_F$ , and  $i \sim j$
$$
\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq (\lambda_ {w} + 2 \eta k (\gamma - \rho)) + 2 \eta (\gamma + \rho) (n - k) + \frac {\eta (\gamma + \rho)}{\gamma - \rho}
$$
Proof. Consider  $t < T_0$ . We consider three cases
1. If  $t = 0$  then
$$
\langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle \leq \lambda_ {w}.
$$
2. If  $\langle \mathbf{w}_j^{(t - 1)},\mathbf{x}_i\rangle \leq 0$  then by Lemma B.2
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t - 1)}, \mathbf {x} _ {i} \rangle \\ - \eta \left(T _ {i j} (t - 1, t) - G _ {j} ^ {(i)} (t - 1, t) (\gamma + \rho) + B _ {j} ^ {(i)} (t - 1, t) (\gamma - \rho)\right) \\ \leq 2 \eta k (\gamma - \rho). \\ \end{array}
$$
3. If  $\langle \mathbf{w}_j^{(t - 1)},\mathbf{x}_i\rangle >0$  then let  $t^\prime < t$  be the smallest iteration such that  $\langle \mathbf{w}_j^{(\tau)},\mathbf{x}_i\rangle >0$  for all  $t^\prime \leq \tau < t$ . By Lemma B.2, Lemma C.5, and the previous two cases above,
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq \langle \mathbf {w} _ {j} ^ {(t ^ {\prime})}, \mathbf {x} _ {i} \rangle - \eta \left(T _ {i j} (t ^ {\prime}, t) - G _ {j} ^ {(i)} (t ^ {\prime}, t) (\gamma + \rho) + B _ {j} ^ {(i)} (t ^ {\prime}, t) (\gamma - \rho)\right) \\ \leq \left(\lambda_ {w} + 2 \eta k (\gamma - \rho)\right) \\ - \eta \left(\left[ 1 - (\gamma + \rho) (n - k) \right] \left(t - t ^ {\prime}\right) - 2 (\gamma + \rho) (n - k) - \frac {\gamma + \rho}{\gamma - \rho}\right). \\ \end{array}
$$
By  $\gamma +\rho <  \frac{1}{n - k}$  (Assumption 3) we conclude
$$
\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle \leq (\lambda_ {w} + 2 \eta k (\gamma - \rho)) + 2 \eta (\gamma + \rho) (n - k) + \frac {\eta (\gamma + \rho)}{\gamma - \rho}.
$$
Lemma C.7 (Lemma 3.4). Suppose Assumption 3 holds. There is an iteration  $\mathcal{T}_1 < \mathcal{T}_0$  during training and expressions  $C_1$ ,  $C_2$ , and  $C_3$  where the following hold:
1. For all  $p\in \{-1,1\}$ $j\in \Gamma_{p},i\sim j$  , and  $i\in S_T$
$$
\left\langle \mathbf {w} _ {j} ^ {\left(\mathcal {T} _ {1}\right)}, \mathbf {x} _ {i} \right\rangle \geq \frac {3}{4 m}.
$$
2. For all  $p\in \{-1,1\}$ $j\in \Gamma_p,i\not\sim j$  , and  $i\in S_T$
$$
\left\langle \mathbf {w} _ {j} ^ {\left(\mathcal {T} _ {1}\right)}, \mathbf {x} _ {i} \right\rangle \leq - \frac {3 n \gamma}{4 m}.
$$
3. For all  $i\in S_T$
$$
\ell \left(\mathcal {T} _ {1}, \mathbf {x} _ {i}\right) \leq \frac {1}{3}.
$$
$$
\mathcal {T} _ {1} = \frac {1}{1 . 0 3 \eta m (1 + (\gamma + \rho) (n - k))} + O (1)
$$
Proof. Fix  $i \in S_T$ . At every iteration  $1 \leq t < \mathcal{T}_0$ , we bound
$$
\begin{array}{l} \ell (t, \mathbf {x} _ {i}) - \ell (t + 1, \mathbf {x} _ {i}) = y _ {i} [ f (t + 1, \mathbf {x} _ {i}) - f (t, \mathbf {x} _ {i}) ] \\ = \sum_ {j = 1} ^ {2 m} (- 1) ^ {i + j} [ \phi (\langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \rangle) - \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) ] \\ = \sum_ {j \notin \Gamma_ {(- 1) ^ {i + 1}}} (- 1) ^ {i + j} \left[ \phi \left(\left\langle \mathbf {w} _ {j} ^ {(t + 1)}, \mathbf {x} _ {i} \right\rangle\right) - \phi \left(\left\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \right\rangle\right) \right] \\ \leq \eta \sum_ {j \notin \Gamma_ {(- 1) ^ {i + 1}}} T _ {i j} (t, t + 1) + (\gamma + \rho) G _ {j} ^ {(i)} (t, t + 1) \\ = \eta \left(\sum_ {j \in \Gamma_ {(- 1) ^ {i}}} T _ {i j} (t, t + 1) + (\gamma + \rho) G _ {j} ^ {(i)} (t, t + 1)\right) \\ + \eta \left(\sum_ {j \notin \Gamma} T _ {i j} (t, t + 1) + (\gamma + \rho) G _ {j} ^ {(i)} (t, t + 1)\right) \\ \leq \eta 0. 9 9 m [ 1 + (\gamma + \rho) (n - k) ] + 0. 0 2 \eta m [ 1 + 2 (\gamma + \rho) (n - k) ], \\ \end{array}
$$
where we use in order:  $t < \mathcal{T}_0$ , the definition of  $f(t,\mathbf{x})$ , Lemma C.4, Lemma B.3,  $\Gamma_{-1}\cap \Gamma_1 = \emptyset$  and Lemma C.4 again. We also use  $|\Gamma_p|\geq 0.99m$  (Assumption 3). We further simplify this bound to conclude
$$
\ell (t + 1, \mathbf {x} _ {i}) - \ell (t, \mathbf {x} _ {i}) \leq 1. 0 3 \eta m [ 1 + (\gamma + \rho) (n - k) ]
$$
Additionally, we bound
$$
\begin{array}{l} \ell (1, \mathbf {x} _ {i}) = 1 - y _ {i} f (1, \mathbf {x} _ {i}) \\ = 1 - \sum_ {j = 1} ^ {2 m} (- 1) ^ {i + j} \phi \left(\left\langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \right\rangle\right) \\ \geq 1 - \sum_ {i \sim j} \phi (\langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \rangle) \\ \geq 1 - \sum_ {i \sim j} [ \phi (\langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle) + \eta T _ {i j} (0, 1) + \eta (\gamma + \rho) G _ {j} ^ {(i)} (0, 1) ] \\ \geq 1 - m \left[ \lambda_ {w} + \eta + 2 \eta (\gamma + \rho) (n - k) \right]. \\ \end{array}
$$
Therefore as long as
$$
t <   \frac {1 - m [ \lambda_ {w} + \eta + 2 \eta (\gamma + \rho) (n - k) ]}{1 . 0 3 \eta m [ 1 + (\gamma + \rho) (n - k) ]} + 1 = \frac {1}{1 . 0 3 \eta m [ 1 + (\gamma + \rho) (n - k) ]} + O (1),
$$
$$
\begin{array}{l} \ell (t, \mathbf {x} _ {i}) = \ell (1, \mathbf {x} _ {i}) + \sum_ {t ^ {\prime} = 1} ^ {t} \ell \left(t ^ {\prime} + 1, \mathbf {x} _ {i}\right) - \ell \left(t ^ {\prime}, \mathbf {x} _ {i}\right) \\ \geq 1 - m \left[ \lambda_ {w} + \eta + 2 \eta (\gamma + \rho) (n - k) \right] - 1. 0 3 \left(t ^ {\prime} - 1\right) \eta m \left[ 1 + (\gamma + \rho) (n - k) \right] \\ > 0. \\ \end{array}
$$
Notice that this does not depend on  $i$  or  $p$ . Therefore we can let  $\mathcal{T}_1$  be the largest integer satisfying this bound for  $t$  and bound  $\ell(\mathcal{T}_1, \mathbf{x}_\ell) > 0$  for all  $l \in S_T$ . To verify that  $\mathcal{T}_1 < \mathcal{T}_0$ , consider  $i \in S_F$ :
$$
\begin{array}{l} y _ {i} f (t, \mathbf {x} _ {i}) = \sum_ {j = 1} ^ {2 m} - (- 1) ^ {i + j} \phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) \\ = \sum_ {i \not \sim j} [ (\phi (\langle \mathbf {w} _ {j} ^ {(t)}, \mathbf {x} _ {i} \rangle) - \phi (\langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle)) + \phi (\langle \mathbf {w} _ {j} ^ {(0)}, \mathbf {x} _ {i} \rangle) ] \\ \leq \sum_ {i \not \sim j} \left(T _ {i j} (0, t) + (\gamma + \rho) B _ {j} ^ {(i)} (0, t) + \lambda_ {w}\right) \\ \leq \eta m t [ 1 + 2 (\gamma + \rho) k ] + m \lambda_ {w} \\ \end{array}
$$
This is less than 1 for all  $t < \mathcal{T}_1$  since  $k \leq \frac{n}{3}$  (Assumption 3).
Now, fix  $i \in S_T$  again. For  $i \sim j$ , we then can use Lemma B.2 and Lemma C.4:
$$
\begin{array}{l} \langle \mathbf {w} _ {j} ^ {\left(\mathcal {T} _ {1}\right)}, \mathbf {x} _ {i} \rangle \geq \langle \mathbf {w} _ {j} ^ {(1)}, \mathbf {x} _ {i} \rangle + \eta (T _ {i j} (1, \mathcal {T} _ {1}) + (\gamma - \rho) G _ {j} ^ {(i)} (1, \mathcal {T} _ {1}) - (\gamma + \rho) B _ {j} ^ {(i)} (1, \mathcal {T} _ {1})) \\ \geq 0 + \eta (\mathcal {T} _ {1} - 2) (1 + (n - k - 1) (\gamma - \rho) - 2 k (\gamma + \rho)) \\ = \frac {1 + (n - k - 1) (\gamma - \rho) - 2 k (\gamma + \rho)}{1 . 0 3 m [ 1 + (\gamma + \rho) (n - k) ]} + O (\eta) \\ \geq \frac {1}{m} - \frac {2 \rho (n - k) - 2 k (\gamma + \rho) - (\gamma + \rho) - 2 k (\gamma + \rho)}{m (1 + (\gamma + \rho) (n - k))} + O (\eta) \\ \geq \frac {1}{m} - \frac {1 / 6 + 4 / 9 9 + 1 / 1 0 0}{m} + O (\eta) \\ \geq \frac {3}{4 m}. \\ \end{array}
$$
using  $\rho \leq \frac{1}{6(n - k)}$ ,  $\eta$  is sufficiently small, and  $\gamma + \rho < \min \left\{\frac{1}{99k}, \frac{1}{100}\right\}$  (Assumption 3). Now assume  $i \not\sim j$ . Using Lemma B.2 and Lemma C.4 we can bound