SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[14] Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199-3210. PMLR, 2020.
[15] Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487, 2021.
[16] David A Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-118, 1975.
[17] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[18] Adam Tauman Kalai and Santosh Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research, pages 423-440, 2002.
[19] Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM), 45(6):983-1006, 1998.
[20] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[21] Gene Li, Pritish Kamath, Dylan J Foster, and Nathan Srebro. Eluder dimension and generalized rank. arXiv preprint arXiv:2104.06970, 2021.
[22] Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems, 27, 2014.
[23] Lev Reyzin. Statistical queries and statistical algorithms: Foundations and applications. arXiv preprint arXiv:2004.00557, 2020.
[24] Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
[25] Balázs Szö rényi. Characterizing statistical query learning: simplified notions and proofs. In International Conference on Algorithmic Learning Theory, pages 186-200. Springer, 2009.
[26] P Turán. On an extremal problem in graph theory (in Hungarian). Mat. Fiz. Lapok, 48:436-452, 1941.
[27] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[28] Vladimir Vovk. A game of prediction with expert advice. Journal of Computer and System Sciences, 56(2):153-173, 1998.
[29] Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems, 33:6123-6135, 2020.
[30] Zheng Wen and Benjamin Van Roy. Efficient exploration and value function generalization in deterministic systems. Advances in Neural Information Processing Systems, 26, 2013.
[31] Ke Yang. New lower bounds for statistical query learning. Journal of Computer and System Sciences, 70(4):485-509, 2005.
A Missing proofs of Section 3
A.1 Analysis of Least Squares Algorithms (Algorithms 1 and 3)
Our analysis relies on the following variant of Freedman's inequality [16] (see Agarwal et al. [1, Lemma 9] and Beygelzimer et al. [6, Theorem 1]).
Lemma 16 (Simplified Freedman's inequality). Let  $R > 0$  and let  $X_{1},\ldots ,X_{n}$  be a sequence of real-valued random variables, such that for all  $i\in [n]$  it holds that  $X_{i}\leq R$  and  $\mathbb{E}[X_i|X_1,\dots ,X_{i - 1}] = 0$ . For any  $\delta \in (0,1)$ , and  $\eta \in (0,1 / R)$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {i = 1} ^ {n} X _ {i} \leq \eta \sum_ {i = 1} ^ {n} \mathbb {E} \left[ X _ {i} ^ {2} \mid X _ {1}, \dots , X _ {i - 1} \right] + \frac {\ln (1 / \delta)}{\eta}. \tag {5}
$$
Next we define an  $\epsilon$ -cover of a set  $\mathcal{Z}$ , that will be used in the bound of Theorem 17.
Definition 7 (ε-cover). Let  $\psi$  be the pseudometric over the set  $\mathcal{Z}$  defined, for any  $z_{1}, z_{2} \in \mathcal{Z}$ , as
$$
\psi \left(z _ {1}, z _ {2}\right) = \sup  _ {z \in \mathcal {Z}} \left| \rho (z \mid z _ {1}) - \rho (z \mid z _ {2}) \right|. \tag {6}
$$
We say a set  $N \subseteq \mathcal{Z}$  is an  $\epsilon$ -cover of  $\mathcal{Z}$  with respect to  $\psi$  if for every  $z \in \mathcal{Z}$  there exists some  $z' \in N$  such that  $\psi(z, z') \leq \epsilon$ . We denote by  $\mathcal{N}(\mathcal{Z}, \epsilon)$  the minimum cardinality of any  $\epsilon$ -cover of  $\mathcal{Z}$ .
For example, in the case of linear bandits (see Section 5.2) when  $\mathcal{Z} = \mathcal{Z}^{lb}$  and  $\rho = \rho_{\mathrm{bandits}}$ , it can be shown that  $\mathcal{N}(\mathcal{Z}^{lb},\epsilon)$  is upper bounded by the  $\ell_2$ -covering number of the  $n$ -dimensional unit ball. This is because for any  $z,z_{1},z_{2}\in \Theta \times \mathcal{A}$
$$
\left| \rho (z \mid z _ {1}) - \rho (z \mid z _ {2}) \right| = \left| \langle \boldsymbol {\theta} _ {1}, \mathbf {a} \rangle - \langle \boldsymbol {\theta} _ {2}, \mathbf {a} \rangle \right| \leq \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| \| \mathbf {a} \| \leq \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \|.
$$
The bound on  $\mathcal{N}(\mathcal{Z}^{lb},\epsilon)$  now follows because the  $\ell_2$ -covering number of the unit ball with radius  $\epsilon$  is  $O\big((3 / \epsilon)^n\big)$  (see, for example, Lemma D.1 of Du et al. [11]).
We next show that Algorithm 1 satisfies the decaying estimation error property with  $C_{T,\delta}$  that scales logarithmically with the covering number with respect to  $\psi$ .
Theorem 17 (LS guarantee). Consider the setting from Section 2, where the learner sequentially issues the queries  $z_{1}, \ldots, z_{T}$  and receives responses  $r_{1}, \ldots, r_{T}$ . Assume there is  $\beta \geq 0$  such that  $|r_{t} - \mathbb{E}[r_{t} \mid z_{t}]| \leq \beta$  for all  $t$ , and  $\beta' \geq 2\beta$  such that for all  $z, z'$ ,  $|\rho(z \mid z') - \rho(z \mid z^{*})| \leq \beta'$ . Let  $\tilde{\mathcal{Z}}$  be a set of alternatives such that  $z^{*} \in \tilde{\mathcal{Z}}$  and let  $\hat{z}_{t}$  be defined as the least squares optimizer,
$$
\hat {z} _ {t} = \underset {z \in \tilde {\mathcal {Z}}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {t - 1} \Big (\rho (z _ {i} \mid z) - r _ {i} \Big) ^ {2}.
$$
Then, for any sequence of queries  $z_{1},\ldots ,z_{T}\in \tilde{\mathcal{Z}}$  (possibly equal to  $\hat{z}_1,\dots ,\hat{z}_T$ ), we have with probability  $1 - \delta$ , for all  $t\in [T]$  simultaneously,
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq C _ {T, \delta}, a n d
$$
$$
z ^ {*} \in \left\{z \in \tilde {\mathcal {Z}}: \sum_ {i = 1} ^ {t - 1} \left(\rho (z _ {i} \mid z) - \rho (z _ {i} \mid \hat {z} _ {t})\right) ^ {2} \leq C _ {T, \delta} \right\},
$$
where  $C_{T,\delta} = 16\beta \beta^{\prime}\ln \bigl (2TN(\tilde{\mathcal{Z}},\beta^{\prime} / T)\bigr /\delta \bigr)$
Proof. For  $i = 1, \dots, T$ , let  $h_i = (z_1, r_1, \dots, z_{i-1}, r_{i-1}, z_i)$  denote the history of interaction up to the query  $z_i$ , but excluding the response  $r_i$ , and let  $\xi_i = r_i - \rho(z_i \mid z^*)$ . In the interactive estimation setting, we then have  $\mathbb{E}[\xi_i \mid h_i] = 0$ , and by the lemma assumption,  $\mathbb{E}[\xi_i^2 \mid h_i] \leq \beta^2$ .
Since  $\hat{z}_t$  is the minimizer of the least squares loss up to time  $t$ , we have
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - r _ {i}\right) ^ {2} \leq \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - r _ {i}\right) ^ {2},
$$
which can be rewritten, substituting  $r_i = \rho (z_i\mid z^*) + \xi_i$  , as
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right) - \xi_ {i}\right) ^ {2} \leq \sum_ {i = 1} ^ {t - 1} \xi_ {i} ^ {2}.
$$
Therefore, by re-arranging terms, we get
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 2 \sum_ {i = 1} ^ {t - 1} \xi_ {i} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right). \tag {7}
$$
Set  $\epsilon_{1} = \beta^{\prime} / T$ , and let  $N$  be a minimal  $\epsilon_{1}$ -cover of  $\tilde{\mathcal{Z}}$  with respect to the pseudometric  $\psi$  (see Eq. 6). Furthermore, let  $\hat{z}_t^\epsilon \in N$  be an element of this cover that is  $\epsilon_{1}$ -close to  $\hat{z}_t$  (with respect to  $\psi$ ). Then,
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 2 \sum_ {i = 1} ^ {t - 1} \xi_ {i} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) \\ = 2 \sum_ {i = 1} ^ {t - 1} \xi_ {i} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}\right) \right. \\ \left. + \rho \left(z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) \\ \leq 2 t \beta \epsilon_ {1} + 2 \sum_ {i = 1} ^ {t - 1} \xi_ {i} \left(\rho \left(z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right), \tag {8} \\ \end{array}
$$
where the first inequality follows from Eq. (7), and the last inequality follows because  $|\xi_i| \leq \beta$  and  $\hat{z}_t^\epsilon$  is  $\epsilon_1$ -close to  $\hat{z}_t$ .
Now, for any  $z \in N$  and  $i \in [T]$ , define
$$
K _ {i} ^ {z} = \xi_ {i} \left(\rho \left(z _ {i} \mid z\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right).
$$
Since  $\mathbb{E}[\xi_i\mid h_i] = 0$ , we have, for any fixed  $z\in N$ ,  $\mathbb{E}[K_i^z\mid h_i] = 0$ . This means that for any fixed  $z\in N$ ,  $K_{1}^{z},\ldots ,K_{T}^{z}$  is a martingale difference sequence. By the lemma assumptions,  $|K_i^z |\leq \beta \beta '$ . Also,
$$
\mathbb {E} \left[ \left(K _ {i} ^ {z}\right) ^ {2} \mid h _ {i} \right] \leq \beta^ {2} \mathbb {E} \left[ \left(\rho \left(z _ {i} \mid z\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \mid h _ {i} \right] = \beta^ {2} \left(\rho \left(z _ {i} \mid z\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2}. \tag {9}
$$
Thus, by Freedman's inequality (Lemma 16) with  $\eta = 1 / (4\beta \beta^{\prime})$  and  $\delta^{\prime} = \frac{\delta}{T|N|}$ , we obtain that for any fixed  $z\in N$  and  $t\in [T]$ , with probability at least  $1 - \delta^\prime$ ,
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - 1} \xi_ {i} \Big (\rho (z _ {i} \mid z) - \rho (z _ {i} \mid z ^ {*}) \Big) \leq \frac {1}{4 \beta \beta^ {\prime}} \sum_ {i = 1} ^ {t - 1} \beta^ {2} \Big (\rho (z _ {i} \mid z) - \rho (z _ {i} \mid z ^ {*}) \Big) + 4 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right) \\ = \frac {\beta}{4 \beta^ {\prime}} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid z\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} + 4 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right). \tag {10} \\ \end{array}
$$
Taking a union bound over all  $z \in N$  and  $t \in [T]$ , we obtain that Eq. (10) holds with probability at least  $1 - \delta$  simultaneously for all  $z \in N$  and  $t \in [T]$ . Henceforth, we assume that we are in the event when Eq. (10) holds for all  $z \in N$  and  $t \in [T]$ .
Applying the bound of Eq. (10) with  $z = \hat{z}_t^\epsilon$  to the sum on the right-hand side of Eq. (8) then yields
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 2 t \beta \epsilon_ {1} + \frac {\beta}{2 \beta^ {\prime}} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \tag {11} \\ + 8 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right). \\ \end{array}
$$
Using the inequality  $(a + b)^2 \leq 2a^2 + 2b^2$ , which holds for any  $a, b \in \mathbb{R}$ , and the fact that  $\hat{z}_t^\epsilon$  and  $\hat{z}_t$  are  $\epsilon_1$ -close, we obtain, for every  $i = 1, \dots, t - 1$ ,
$$
\left(\rho (z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}) - \rho (z _ {i} \mid z ^ {*})\right) ^ {2} = \left(\left[ \rho (z _ {i} \mid \hat {z} _ {t} ^ {\epsilon}) - \rho (z _ {i} \mid \hat {z} _ {t}) \right] + \left[ \rho (z _ {i} \mid \hat {z} _ {t}) - \rho (z _ {i} \mid z ^ {*}) \right]\right) ^ {2}
$$
$$
\leq 2 \epsilon_ {1} ^ {2} + 2 \left(\rho \left(z _ {i} \mid z ^ {*}\right) - \rho \left(z _ {i} \mid \hat {z} _ {t}\right)\right) ^ {2}.
$$
Plugging this into the right-hand side of Eq. (11) yields
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 2 t \beta \epsilon_ {1} + \frac {\beta}{\beta^ {\prime}} t \epsilon_ {1} ^ {2} + \frac {\beta}{\beta^ {\prime}} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \\ + 8 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right) ^ {= 1} \\ \leq 2 t \beta \epsilon_ {1} + \frac {t \beta \epsilon_ {1} ^ {2}}{\beta^ {\prime}} + \frac {1}{2} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \\ + 8 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right), \\ \end{array}
$$
where the last inequality follows by the assumption that  $\beta' \geq 2\beta$ . Then, by re-arranging terms and multiplying by 2, we get
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 4 t \beta \epsilon_ {1} + \frac {2 t \beta \epsilon_ {1} ^ {2}}{\beta^ {\prime}} + 1 6 \beta \beta^ {\prime} \ln \left(\frac {T | N |}{\delta}\right).
$$
Recall that we set  $\epsilon_1 = \beta'/T$ , and  $N$  is a minimal  $\epsilon_1$ -cover of  $\tilde{\mathcal{Z}}$ , so  $|N| = \mathcal{N}(\tilde{\mathcal{Z}}, \beta'/T)$ . Plugging these values in the previous equation, we thus obtain that with probability at least  $1 - \delta$ , for all  $t \in [T]$
$$
\begin{array}{l} \sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid \hat {z} _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq 4 \beta \beta^ {\prime} + \frac {2 \beta \beta^ {\prime}}{T} + 1 6 \beta \beta^ {\prime} \ln \left(\frac {T \mathcal {N} (\tilde {\mathcal {Z}} , \beta^ {\prime} / T)}{\delta}\right) \\ \leq 1 6 \beta \beta^ {\prime} \ln \left(\frac {2 T \mathcal {N} (\tilde {\mathcal {Z}} , \beta^ {\prime} / T)}{\delta}\right), \tag {12} \\ \end{array}
$$
where the last inequality follows because  $4 + 2 / T \leq 16 \ln 2$  for  $T \geq 1$ . Finally, when Eq. (12) holds, we also have
$$
z ^ {*} \in \left\{z \in \tilde {\mathcal {Z}}: \sum_ {i = 1} ^ {t - 1} \Big (\rho (z _ {i} \mid z) - \rho (z _ {i} \mid \hat {z} _ {t}) \Big) ^ {2} \leq 1 6 \beta \beta^ {\prime} \ln \left(\frac {2 T \mathcal {N} (\tilde {\mathcal {Z}} , \beta^ {\prime} / T)}{\delta}\right) \right\}.
$$
Considering Algorithm 1 and using Theorem 17 with  $\tilde{\mathcal{Z}} = \mathcal{Z}_{\alpha}$ ,  $z_{t} = \hat{z}_{t}$ ,  $\beta = 2$  and  $\beta' = 4$  then immediately yields the following corollary ( $\alpha$ -large self-evaluations follow because  $\hat{z}_t \in \mathcal{Z}_\alpha$ ):
Corollary 18. Consider the setting from Section 2 with a set of alternatives  $\mathcal{Z}$  and an evaluation function  $\rho$ . Let  $\alpha$  be an optimality level such that  $\mathcal{N}(\mathcal{Z}_{\alpha},4 / T) = e^{o(T)}$ . Then Algorithm 1 has  $\alpha$ -large self-evaluations and satisfies the decaying error property with  $C_{T,\delta} = 128\ln \left(2TN(\mathcal{Z}_{\alpha},4 / T) / \delta\right)$ .
Similarly, Theorem 17 also implies that Algorithm 3 satisfies the decaying error property as well as  $\alpha^{*}$ -large self-evaluations, although  $\alpha^{*}$  is not known:
Corollary 19. Consider the setting from Section 2 with a set of alternatives  $\mathcal{Z}$  and an evaluation function  $\rho$ , and assume that  $\mathcal{N}(\mathcal{Z},4 / T) = e^{o(T)}$ . Then Algorithm 3 with  $R = 128\ln \left(2TN(\mathcal{Z},4 / T) / \delta\right)$  has  $\alpha^*$ -large self-evaluations and satisfies the decaying error property with  $C_{T,\delta} = 4R = 512\ln \left(2TN(\mathcal{Z},4 / T) / \delta\right)$ .
Proof. We apply Theorem 17 with  $\tilde{\mathcal{Z}} = \mathcal{Z}$ ,  $\beta = 2$  and  $\beta' = 4$ . Our choice of  $R$  in Algorithm 3 coincides with the value of  $C_{T,\delta}$  appearing in Theorem 17, and therefore the theorem implies that  $z^{*} \in \mathcal{Z}_{t}$  with probability at least  $1 - \delta$  for all  $t \in [T]$ . In that case the queries  $z_{t}$  issued by the algorithm satisfy  $\rho(z_{t} \mid z_{t}) \geq \rho(z^{*} \mid z^{*}) = \alpha^{*}$  and thus the algorithm has  $\alpha^{*}$ -large self-evaluations.
For the second part, the triangle inequality implies that with probability at least  $1 - \delta$  for all  $t\in [T]$
$$
\begin{array}{l} \sqrt {\sum_ {i = 1} ^ {t - 1} \left(\rho (z _ {i} \mid z) - \rho (z _ {i} \mid z _ {t})\right) ^ {2}} \leq \sqrt {\sum_ {i = 1} ^ {t - 1} \left(\rho (z _ {i} \mid z) - \rho (z _ {i} \mid \hat {z} _ {t})\right) ^ {2}} + \sqrt {\sum_ {i = 1} ^ {t - 1} \left(\rho (z _ {i} \mid \hat {z} _ {t}) - \rho (z _ {i} \mid z _ {t})\right) ^ {2}} \\ \leq \sqrt {R} + \sqrt {R}, \\ \end{array}
$$
where the bound on the first term on the right-hand side follows by Theorem 17 and the bound on the second term by the fact that  $z_{t} \in \mathcal{Z}_{t}$ .
A.2 Proof of Theorem 4
The theorem follows immediately from Corollary 18, because  $\mathcal{N}(\mathcal{Z}_{\alpha},\epsilon)\leq |\mathcal{Z}_{\alpha}|\leq |\mathcal{Z}| < \infty$  for any  $\alpha$  and  $\epsilon$ .
A.3 Online Regression Oracles
We assume access to an online regression oracle  $\mathcal{R}$  , which solves a regression problem over a function class  $\Phi = \{\phi_z:z\in \mathcal{Z}\}$  indexed by  $z\in \mathcal{Z}$  , where  $\phi_z:\mathcal{Z}\to \mathbb{R}$  is defined as  $\phi_z(z^{\prime}) = \rho (z^{\prime}\mid z)$  for all  $z^{\prime}\in \mathcal{Z}$  ; that is, functions  $\phi_z$  evaluate  $\rho$  in its first argument.
The oracle operates in the following protocol: In each time step, the oracle receives an observation  $z_{t}$ , produces a prediction  $\hat{\rho}_{t} \in \mathbb{R}$ , and finally receives a response  $r_{t}$  and incurs square loss  $(\hat{\rho}_{t} - r_{t})^{2}$ . We assume that for any  $T$  and sequence of observations and responses (even if generated adaptively), the oracle satisfies the following regret bound:
$$
\sum_ {t = 1} ^ {T} \left(\widehat {\rho} _ {t} - r _ {t}\right) ^ {2} - \inf  _ {z \in \mathcal {Z}} \sum_ {t = 1} ^ {T} \left(\rho \left(z _ {t} \mid z\right) - r _ {t}\right) ^ {2} \leq \operatorname {R e g r e t} _ {\mathcal {R} e g} (T), \tag {13}
$$
where  $\mathrm{Regret}_{\mathcal{R}eg}(\cdot)$  is a non-decreasing sublinear function (that typically also depends on various properties of  $\rho, \mathcal{Z}$ , and the range of responses  $r_t$ ). For many function classes, there are well-known constructions of online regression oracles that satisfy Eq. (13) [9, 28, 18]. For example, if  $\Phi$  is finite, there are oracles with  $\mathrm{Regret}_{\mathcal{R}eg}(T) = O(\ln |\Phi|)$  and for parametric classes, such as linear functions, there are oracles with  $\mathrm{Regret}_{\mathcal{R}eg}(T) = O(d\log(T / d))$ . More examples can be found in Section 2.2 of Foster and Rakhlin [14].
Algorithm 4 Interactive Estimation via Least Squares
1: Input: online regression oracle  $\mathcal{R}\mathrm{eg}$ , optimality level  $\alpha$ .
2: Initialize  $z_{1}$  to an arbitrary element of  $\tilde{\mathcal{Z}}_{\alpha}$ .
3: for  $t = 1 \dots T$  do
4: Use  $\mathcal{R}eg$  to predict  $\widehat{\rho_t}$  given the observation  $z_{t}$
5: Observe reward  $r_{t+1}$  and pass it to  $\mathcal{R}eg$ .
$$
6: \quad \text {S e t} z _ {t + 1} = \underset {z \in \mathcal {Z} _ {\alpha}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {t} \left(\rho (z _ {i} \mid z) - \widehat {\rho} _ {i}\right) ^ {2}.
$$
7: end for
We now analyze Algorithm 4 under the assumption of access to an online regression oracle. This algorithm takes as input an online regression oracle  $\mathcal{R}$  reg. Algorithm 4 can also be modified using the optimistic least squares template of Russo and Van Roy [24] to handle the case when  $\alpha^{*}$  is unknown. This is done by replacing step 6 of Algorithm 4 with the following two steps:
$$
\mathcal {Z} _ {t + 1} = \left\{z \in \mathcal {Z}: \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z\right) - \widehat {\rho} _ {i}\right) ^ {2} \leq R \right\},
$$
$$
z_{t + 1} = \operatorname *{argmax}_{z\in \mathcal{Z}_{t + 1}}\rho (z\mid z),
$$
where  $R = 8\mathrm{Regret}_{\mathcal{R}eg}(T) + 64\beta \max \{\beta, \beta'\} \ln \left(\frac{T}{\delta}\right)$  and  $\beta, \beta'$  are defined as in Lemma 20. The results of Lemma 20 justify the validity of these choices (using a similar reasoning as in Corollary 19) and imply that Algorithm 4 satisfies the decaying estimation error property (Definition 3), provided that the regression oracle  $\mathcal{R}eg$  satisfies the regret bound of Eq. (13).
Lemma 20. Consider the setting defined in Section 2 with  $\alpha \leq \alpha^{*}$ , and assume there are  $\beta, \beta' \geq 0$  such that  $|r_t - \mathbb{E}[r_t \mid z_t]| \leq \beta$  for all  $t$  and there is  $\beta' \geq 2\beta$  s.t. for all  $z \in \mathcal{Z}$  and  $\hat{\rho} \in \Gamma_z$ ,  $|\hat{\rho} - \rho(z \mid z^*)| \leq \beta'$  where  $\Gamma_z \subset \mathbb{R}$  is the space of plausible responses of  $\mathcal{R}$ g for input  $z$ . The sequence of queries  $z_1, \ldots, z_T$  as defined in Algorithm 4 satisfies with probability at least  $1 - \delta$  for all  $t \in [T]$  simultaneously,
$$
\sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z _ {t + 1}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq C _ {T, \delta} \quad a n d \quad z ^ {*} \in \left\{z \in \mathcal {Z} _ {\alpha}: \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z\right) - \widehat {\rho} _ {i}\right) ^ {2} \leq C _ {T, \delta} \right\}
$$
where  $C_{T,\delta} = 8\mathrm{Regret}_{\mathcal{R}eg}(T) + 64\beta \max \{\beta ,\beta '\} \ln \left(\frac{T}{\delta}\right)$ .
Proof. Let  $\xi_{i} = r_{i} - \rho (z_{i}\mid z^{*})$ . Recall that  $\mathbb{E}[r_i\mid z_i] = \rho (z_i\mid z^*)$  and therefore, by assumption,  $|\xi_i|\leq \beta$  for all  $i$ . By definition, the online regression oracle satisfies
$$
\begin{array}{l} \sum_ {i = 1} ^ {t} \left(\widehat {\rho} _ {i} - r _ {i}\right) ^ {2} \leq \inf  _ {z \in \mathcal {Z}} \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z\right) - r _ {i}\right) ^ {2} + \operatorname {R e g r e t} _ {\mathcal {R} e g} (t) \\ \stackrel {(i)} {\leq} \sum_ {i = 1} ^ {t} \left(\rho \left(z _ {i} \mid z ^ {*}\right) - r _ {i}\right) ^ {2} + \operatorname {R e g r e t} _ {\mathcal {R} e g} (t) \\ = \sum_ {i = 1} ^ {t} \xi_ {i} ^ {2} + \operatorname {R e g r e t} _ {\mathcal {R} e g} (t). \tag {14} \\ \end{array}
$$
Inequality  $(i)$  holds because  $z^{*}\in \mathcal{Z}$ . Expanding the LHS,
$$
\sum_ {i = 1} ^ {t} (\widehat {\rho_ {i}} - r _ {i}) ^ {2} = \sum_ {i = 1} ^ {t} \Big [ (\widehat {\rho_ {i}} - \rho (z _ {i} | z ^ {*})) ^ {2} - 2 \xi_ {i} (\widehat {\rho_ {i}} - \rho (z _ {i} | z ^ {*})) + \xi_ {i} ^ {2} \Big ].
$$
Plugging this back into Eq. (14) and rearranging, we obtain
$$
\sum_ {i = 1} ^ {t} \left(\widehat {\rho} _ {i} - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq \sum_ {i = 1} ^ {t} 2 \xi_ {i} \left(\widehat {\rho} _ {i} - \rho \left(z _ {i} \mid z ^ {*}\right)\right) + \operatorname {R e g r e t} _ {\mathcal {R} e g} (t). \tag {15}
$$
For any  $i \in [T]$  we define:
$$
K _ {i} = \xi_ {i} (\widehat {\rho} _ {i} - \rho (z _ {i} \mid z ^ {*}))
$$
$$
\mathbb {E} \left[ K _ {i} \mid \left\{z _ {\ell}, \widehat {\rho_ {\ell}} \right\} _ {\ell = 1} ^ {i} \right] = 0.
$$
Thus  $K_{1},\ldots ,K_{T}$  is a martingale difference sequence. Notice that  $|K_i|\leq \beta \beta '$  and that