SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
b _ {n + 1} \leq \left(1 - \frac {d}{n}\right) b _ {n} + \frac {c}{n ^ {2}},
$$
where  $c > 1, c_1 > 0$ . Then
$$
b _ {n} \leq \frac {c}{d - 1} \frac {1}{n} + \mathcal {O} \left(\frac {1}{n ^ {2}} + \frac {1}{n ^ {d}}\right).
$$
We set  $x_{t + \eta} \coloneqq \mathbb{E}\left(f(\theta_t) - f(\theta^*)\right)$  for  $t \geq 1$  and  $x_k = \mathbb{E}\left(f(\theta_1) - f(\theta^*)\right)$  for  $k \leq \eta$ . Using 31 and  $k = t + \eta$  we get
$$
x _ {k + 1} \leq (1 - \frac {4}{k}) x _ {k} + \frac {8 B (M + \epsilon^ {2})}{N m ^ {2} k ^ {2}}
$$
Now using  $d = 4$  and  $c = \frac{8B(M + \epsilon^2)}{Nm^2}$ , we have  $x_{k+1} \leq (1 - \frac{d}{k})x_k + \frac{c}{k^2}$  such that E.2 yields
$$
x _ {t} \leq \frac {8 B (M + \epsilon^ {2})}{3 N m ^ {2} t} + \mathcal {O} \left(\frac {1}{t ^ {2}} + \frac {1}{t ^ {d}}\right)
$$
$$
\mathbb {E} \left(f \left(\theta_ {t}\right) - f \left(\theta^ {*}\right)\right) \leq \frac {8 B \left(M + \epsilon^ {2}\right)}{3 N m ^ {2} (t + \eta)} + \mathcal {O} \left(\frac {1}{t ^ {2}} + \frac {1}{t ^ {d}}\right) \leq \frac {8 B \left(M + \epsilon^ {2}\right)}{3 N m ^ {2} t} + \mathcal {O} \left(\frac {1}{t ^ {2}} + \frac {1}{t ^ {4}}\right)
$$
Now, if there is a  $t$  with  $\Pi_W(\theta_t^s -\gamma_t\bar{m}_t)\neq \theta_t^s -\gamma_t\bar{m}_t$ , we can still bound  $\mathbb{E}\left(f(\theta_t) - f(\theta^*)\right)$  by some constant because  $W$  is bounded and  $f$  is Lipschitz. Correspondingly, as  $P(\exists t\leq T:\Pi_W(\theta_t^s -\gamma_t\bar{m}_t)\neq \theta_t^s -\gamma_t\bar{m}_t)\in O(\frac{1}{NT})$ , the total expectation for both cases combined is bounded by  $\frac{8B(M + \epsilon^2)}{3m^2NT} +\mathcal{O}\left(\frac{1}{NT}\right) + \mathcal{O}\left(\frac{1}{T^2}\right)$ .
To prove theorem 6.1, we use a non-asymptotic version of Chung's Lemma Chung (1954) similar to the one used in the proof of Lemma 1 in Rakhlin et al. (2012):
Lemma E.3. For constants  $c > 0$  and  $d > 1$ , whenever  $t + 1 \geq d$  and the recursive inequality
$$
x _ {t + 1} \leq (1 - \frac {d}{t + 1}) x _ {t} + \frac {c}{(t + 1) ^ {2}},
$$
holds we get that if
$$
x _ {t} \leq \frac {2 d ^ {2} c}{t (d ^ {3} - d ^ {2})}
$$
for  $t = k$  the same is true for  $t = k + 1$ .
Proof. Using the condition on  $x_{t}$ , we obtain
$$
\begin{array}{l} x _ {t + 1} \leq (1 - \frac {d}{t + 1}) x _ {t} + \frac {c}{(t + 1) ^ {2}} \\ \leq \frac {2 d ^ {2} c}{t \left(d ^ {3} - d ^ {2}\right)} - \frac {2 d ^ {3} c}{t (t + 1) \left(d ^ {3} - d ^ {2}\right)} + \frac {c}{(t + 1) ^ {2}} \\ \end{array}
$$
as by assumption  $\frac{d}{t + 1} \leq 1$ . As  $d > 1$ ,  $\frac{2d^2c}{(t + 1)(d^3 - d^2)}$  is positive and we can divide the equation above by it to obtain
$$
\frac {x _ {t + 1} (t + 1) \left(d ^ {3} - d ^ {2}\right)}{2 d ^ {2} c} \leq \frac {t + 1}{t} - \frac {d}{t} + \frac {\left(d ^ {3} - d ^ {2}\right)}{\left(t + 1\right) 2 d ^ {2}}
$$
for which we want to show that it is bounded above by 1. Multiplying the equation
$$
\frac {t + 1}{t} - \frac {d}{t} + \frac {(d ^ {3} - d ^ {2})}{(t + 1) 2 d ^ {2}} \leq 1
$$
by  $t(t + 1)$  for  $t\geq 1$  we get
$$
t ^ {2} + 2 t + 1 - d t - d + t \frac {(d ^ {3} - d ^ {2})}{2 d ^ {2}} \leq t ^ {2} + t
$$
$$
t + 1 - d t - d + \frac {t}{2} (d - 1) \leq 0,
$$
$$
\frac {1}{2} (1 - d) t + 1 - d \leq 0,
$$
$$
(1 - d) t \leq 2 (d - 1),
$$
which is true whenever
$$
t \geq - 2.
$$
Theorem E.1. In the settings of E.2, assume that there exist scalars  $M \geq 0$  and  $M_V \geq 0$ , such that for all  $t$ :
$$
\mathbb {E} _ {s _ {i}} \left(\| \left(e _ {t} ^ {i} \left(\theta_ {t} ^ {s}\right) \| ^ {2}\right) \leq M + M _ {V} \| \nabla f \left(\theta_ {t} ^ {s}\right)\right) \| _ {2} ^ {2}. \tag {32}
$$
Assume that for some integer constant  $\eta > 1$ , such that  $\frac{4}{\eta m + m} \leq \frac{1}{B(M_V / N + 1)}$ , the learning rate is set as  $\gamma_t = \frac{4}{\eta m + tm}$ .
Then any player's best response strategy fulfills  $\alpha_{i}^{t}\leq \frac{8L}{C^{t}(N - 2)m\sqrt{T1 + \eta}}$  independent of other players' strategies, such that for any given  $\epsilon >0$ ,  $C^t\geq \frac{8L}{\epsilon(N - 2)m\sqrt{T + \eta}}$  yields  $\alpha_{i}^{t}\leq \epsilon$  for rational players. In that case, as long as  $W$  is bounded and we have that  $P(\exists t\leq T:\Pi_W(\theta_t^s -\gamma_t\bar{m}_t)\neq \theta_t^s -\gamma_t\bar{m}_t)\in O(\frac{1}{NT})$  we get  $\mathbb{E}\left(f(\theta_t) - f(\theta^*)\right)\in O(\frac{1 + M + \epsilon^2}{Nt}) + O(\frac{1}{t^2})$ .
Proof. We make use of inequality 25 from the proof of E.1 to analyse the difference between two trajectories that are identical except for the actions of player  $i$ .
$$
\begin{array}{l} \mathbb {E} \| \theta_ {t + 1} - \theta_ {t + 1} ^ {\prime} \| ^ {2} \\ \leq \left(1 + \gamma_ {t} ^ {2} (B ^ {2} + B ^ {\prime 2}) - 2 \gamma_ {t} m\right) \mathbb {E} \| \theta_ {t} - \theta_ {t} ^ {\prime} \| ^ {2} + \frac {\gamma_ {t} ^ {2}}{N ^ {2}} \delta_ {i} ^ {t} \\ = (1 + \frac {1 6}{m ^ {2} (\eta + t) ^ {2}} (B ^ {2} + B ^ {\prime 2}) - \frac {8}{\eta + t}) \mathbb {E} \| \theta_ {t} - \theta_ {t} ^ {\prime} \| ^ {2} + \frac {1 6}{(\eta + t) ^ {2} N ^ {2} m ^ {2}} \delta_ {i} ^ {t}. \\ \leq (1 + \frac {1 6}{m ^ {2} (\eta + t) ^ {2}} (B ^ {2} + B ^ {\prime 2}) - \frac {8}{\eta + t}) \mathbb {E} \| \theta_ {t} - \theta_ {t} ^ {\prime} \| ^ {2} + \frac {1 6}{(\eta + t) ^ {2} N ^ {2} m ^ {2}} \delta_ {i} ^ {t}. \\ \end{array}
$$
For  $t\geq 0$  and  $\eta \geq \max \{\frac{32(B^2 + B'^2)}{13m^2},1\}$  we get
$$
1 6 \frac {B ^ {2} + B ^ {\prime 2}}{m ^ {2} (t + \eta) ^ {2}} - \frac {8}{t + \eta} \leq - \frac {1 . 5}{t + \eta} \tag {33}
$$
by calculating
$$
\begin{array}{l} \eta \geq \frac {3 2 \left(B ^ {2} + B ^ {\prime 2}\right)}{1 3 m ^ {2}} \\ \Longrightarrow 6. 5 \eta + 6. 5 t \geq \frac {1 6 \left(B ^ {2} + B ^ {\prime 2}\right)}{m ^ {2}} \\ \Longrightarrow 8 \eta + 8 t - \frac {1 6 (B ^ {2} + B ^ {\prime 2})}{m ^ {2}} \geq 1. 5 \eta + 1. 5 t \\ \end{array}
$$
$$
\begin{array}{l} \Longrightarrow 8 \eta + 8 t - \frac {1 6 \left(B ^ {2} + B ^ {\prime 2}\right)}{m ^ {2}} \geq 1. 5 \frac {(t + \eta) ^ {2}}{t + \eta} \\ \Longrightarrow \frac {8}{\eta + t} - \frac {1 6 \left(B ^ {2} + B ^ {\prime 2}\right)}{\left(\eta + t\right) ^ {2} m ^ {2}} \geq \frac {1 . 5}{t + \eta} \\ \Rightarrow \frac {1 6 \left(B ^ {2} + B ^ {\prime 2}\right)}{(\eta + t) ^ {2} m ^ {2}} - \frac {8}{\eta + t} \leq - \frac {1 . 5}{t + \eta} \\ \end{array}
$$
We now set  $x_{t + \eta} \coloneqq \mathbb{E}\| \theta_{t + 1} - \theta_{t + 1}'\| ^2$  for  $t \geq 0$  and  $x_{k} = 0$  for  $k \leq \eta$ . These definitions are consistent for  $t = 0$  because  $\mathbb{E}\| \theta_1 - \theta_1'\| ^2 = 0$ .
Using 33 and  $k = t - 1 + \eta$ , we get
$$
x _ {k + 1} \leq (1 - \frac {1 . 5}{k + 1}) x _ {k} + \frac {1 6}{(k + 1) ^ {2} N ^ {2} m ^ {2}} \delta_ {i} ^ {t}.
$$
At the same time,  $x_{\eta} = \mathbb{E}\| \theta_1 - \theta_1'\|^2 = 0 \leq \frac{4c}{\eta - 1}$  for any  $c > 0$ . Correspondingly, E.3 with  $d = 1.5$  and  $c = \frac{16}{N^2m^2} \max_t\{\delta_i^t\}$  implies that for  $k \geq \eta$  and  $k + 1 \geq 1.5$  we get
$$
x _ {k} \leq \frac {4 c}{k}
$$
$$
\mathbb {E} \| \theta_ {t} - \theta_ {t} ^ {\prime} \| ^ {2} \leq \frac {4 c}{t - 1 + \eta}
$$
for  $t\geq 1$  . This yields
$$
\mathbb {E} \| \theta_ {T + 1} - \theta_ {T + 1} ^ {\prime} \| ^ {2} \leq \frac {6 4 \max  _ {t} \left\{\delta_ {i} ^ {t} \right\}}{(T + \eta) N ^ {2} m ^ {2}}.
$$
$$
\mathbb {E} \left| f \left(\theta_ {T + 1}\right) - f \left(\theta_ {T + 1} ^ {\prime}\right) \right| \leq L \sqrt {\frac {6 4 \operatorname* {m a x} _ {t} \left\{\delta_ {i} ^ {t} \right\}}{(T + \eta) N ^ {2} m ^ {2}}}.
$$
Again, we can bound the difference between expected penalized rewards for two trajectories with all  $\alpha_{k}^{i}$  fixed but two different values  $\alpha_{t}^{i}$  and  $(\alpha_{t}^{i})'$  varying for  $k = t$  by considering  $\delta_{k} = 0$  for all  $k \neq t$ . This yields
$$
\mathbb {E} | f (\theta_ {T + 1}) - f (\theta_ {T + 1} ^ {\prime}) | \leq L \sqrt {\frac {6 4 \delta_ {i} ^ {t}}{(T + \eta) N ^ {2} m ^ {2}}} = L \sqrt {\frac {6 4 | \alpha_ {t} ^ {i} - (\alpha_ {t} ^ {i}) ^ {\prime} | ^ {2}}{(T + \eta) N ^ {2} m ^ {2}}}.
$$
As in E.2, this allows us to upper bound the gains in penalized reward from changing  $\alpha_{t}^{i}$  to  $(\alpha_{t}^{i})^{\prime}$  by
$$
\frac {8 L}{N m \sqrt {T + \eta}} | \alpha_ {t} ^ {i} - (\alpha_ {t} ^ {i}) ^ {\prime} | + C _ {t} (\frac {N - 2}{N} ((\alpha_ {t} ^ {i}) ^ {\prime}) ^ {2} - \frac {N - 2}{N} (\alpha_ {t} ^ {i}) ^ {2}).
$$
In particular, for  $(\alpha_{t}^{i})^{\prime} = 0$  this bound becomes
$$
\frac {8 L}{N m \sqrt {T + \eta}} \alpha_ {t} ^ {i} - C _ {t} (\frac {N - 2}{N} (\alpha_ {t} ^ {i}) ^ {2})
$$
which is zero at  $\alpha_{t}^{i} = 0$  and at
$$
\alpha_ {t} ^ {i} = \frac {- 2 \frac {8 L}{N m \sqrt {T + \eta}}}{- 2 C _ {t} \frac {N - 2}{N}} = \frac {8 L}{C _ {t} (N - 2) m \sqrt {T + \eta}} \tag {34}
$$
and negative for  $\alpha_{t}^{i}$  larger than that as long as  $N > 2$
Again the noise  $\alpha_{t}^{i}$  added by any rational player  $i$  at a given time step  $t$  can therefore be limited to any fixed constant  $\epsilon > 0$  by choosing  $C_{t}$  such that  $\frac{8L}{C_t(N - 2)m\sqrt{T + \eta}} \leq \epsilon$ , i.e.  $C_{t} \geq \frac{8L}{\epsilon(N - 2)m\sqrt{T + \eta}}$ .
We conclude by using E.1 to obtain the convergence rate.
As a final step, we show that a version of E.1 also holds for more general reward functions, thus proving 6.1.
Theorem E.2. Up to constants, theorems E.2 and E.1 also hold for the reward  $\mathcal{R}_{U_p}^i (\theta_{T + 1}^1,\dots ,\theta_{T + 1}^N,f) = U^i (f(\theta_{T + 1}^i),\dots ,f(\theta_{T + 1}^N)) - \sum_{t = 1}^T C_t^U\| m_t^i -\frac{1}{N}\sum_jm_t^j\| ^2 +$ $\frac{1}{N - 1}\sum_{k\neq i}\sum_{t = 1}^{T}C_{t}^{U}\| m_{t}^{k} - \frac{1}{N}\sum_{j}m_{t}^{j}\|^{2}$  for arbitrary  $l_{1}$ -Lipschitz  $U^i$  with common Lipschitz constant  $L_{U}$ . Any bound on  $\alpha_{t}^{i}$  can be achieved by setting  $C_t^U = L_UNC_t$  for the  $C_t$  achieving the same bound in E.2 or E.1 respectively.
Proof. We first note that for any pointwise bound  $\mathbb{E}|f(\theta_{T + 1}^j) - f((\theta_{T + 1}^j))'| \leq K$
$$
\begin{array}{l} \mathbb {E} \left| U ^ {i} \left(f \left(\theta_ {T + 1} ^ {i}\right), \dots , f \left(\theta_ {T + 1} ^ {N}\right)\right) - U ^ {i} \left(f \left(\left(\theta_ {T + 1} ^ {i}\right) ^ {\prime}\right), \dots , f \left(\left(\theta_ {T + 1} ^ {N}\right) ^ {\prime}\right)\right) \right| \\ \leq \mathbb {E} L _ {U} \sum_ {j} | f (\theta_ {T + 1} ^ {j}) - f ((\theta_ {T + 1} ^ {j})) ^ {\prime} | \\ = L _ {U} \sum_ {j} \mathbb {E} | f (\theta_ {T + 1} ^ {j}) - f ((\theta_ {T + 1} ^ {j})) ^ {\prime} | \\ \leq L _ {U} N K. \\ \end{array}
$$
This means that the gains  $|U^{i}(f(\theta_{T + 1}^{i}),\ldots ,f(\theta_{T + 1}^{N})) - U^{i}(f((\theta_{T + 1}^{i})^{\prime}),\ldots ,f((\theta_{T + 1}^{N})^{\prime}))|$  in unpen-ralized reward player  $i$  can achieve by using a given  $\alpha_{t}^{i}$  at time  $t$  instead of  $\alpha_{t}^{i} = 0$  is multiplied by  $L_{U}N$  compared to 28 and 34. Thus, for a given  $C_t^U$ , the bound on  $\alpha_{t}^{i}$  for rational players is multiplied by  $NL_{U}$  as well, as the quadratic formula solution for  $\alpha_{t}^{i}$  is linear in the linear term. Correspondingly, we need to set  $C_t^U = L_UNC_t$  to achieve the same bounds on  $\alpha_{t}^{i}$  as in E.2 or E.1.
Now, for  $\alpha_{t}^{i}\leq \epsilon$ , the bound on the expected absolute change in  $U^{i}(f(\theta_{T + 1}^{i}),\ldots ,f(\theta_{T + 1}^{N}))$  is  $L_{U}N$  times higher than for  $f(\theta_{T + 1}^{j})$  using the bounds above. Because these bounds are linear in  $\epsilon$  for E.2, we can achieve a bound of  $\delta$  for  $U^{i}(f(\theta_{T + 1}^{i}),\ldots ,f(\theta_{T + 1}^{N}))$  by ensuring  $\alpha_{t}^{i}\leq \frac{\epsilon}{NL_{U}}$  for  $\epsilon$  achieving a bound of  $\delta$  for  $f(\theta_T^i)$ . In total, we thus need to multiply the corresponding  $C_t$  by  $NL_{U}$  twice to achieve a given bound on the gains in unpenalized reward from cheating: Once because we need a smaller bound on  $\alpha_{t}^{i}$  to achieve the same bound on the unpenalized reward, and once to ensure that rational players are incentivized to use that smaller bound.
Discussion on the projection assumptions We note that the assumption  $P(\exists t \leq T : \Pi_W(\theta_t^s - \gamma_t \bar{m}_t) \neq \theta_t^s - \gamma_t \bar{m}_t) \in O\left(\frac{1}{NT}\right)$  in particular holds if  $W$  is chosen such that  $\| w - \theta_0^s \| \in \Omega(T)$  for all  $w$  in the boundary of  $W$  while  $\| \bar{m}_t \| \in O\left(\| \theta_t^s - \theta_0^s \|\right)$  with probability one for all  $t \leq T$ . In that case, the linearly decaying learning rate ensures that  $(\| \theta_t^s - \theta_0^s \|)$  stays in  $O(T)$  and thus in  $W$  (for appropriately chosen constants) with probability one.
Similarly, Lemma 5 in Rakhlin et al. (2012) states that for a probability one bound on the gradient norm  $\| \tilde{m}_t\| \leq G,\forall t$  , we have that  $\| \theta_t^s -\theta^\star \| <  \frac{2G}{m}$  with probability one, such that the iterates stay in  $W$  without the need for any projections, as long as  $G$  grows slower in  $T$  than inf  $_{w\in \delta (W)}\| w - \theta^{\star}\|$  In particular, if  $f$  grows quadratically in  $\theta -\theta^{\star}$  and noise is proportional to the gradient norm,  $G$  grows linearly in the distance between  $\theta^{\star}$  and the boundary of  $W$  , such that the condition holds for the right proportionality constants.