SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Nataly Brukhim
nbrukhim@princeton.edu
Miro Dudik
mdudik@microsoft.com
Aldo Pacchiano
Broad Institute of MIT and Harvard & Boston University
apacchia@broadinstitute.org
Robert Schapire
schapire@microsoft.com
We study an abstract framework for interactive learning called **interactive estimation** in which the goal is to estimate a target from its "similarity" to points queried by the learner. We introduce a combinatorial measure called "dissimilarity dimension" which is used to derive learnability bounds in our model. We present a simple, general, and broadly-applicable algorithm, for which we obtain both regret and PAC generalization bounds that are polynomial in the new dimension. We show that our framework subsumes and thereby unifies two classic learning models: statistical-query learning and structured bandits. We also delineate how the dissimilarity dimension is related to well-known parameters for both frameworks, in some cases yielding significantly improved analyses.
We study a general interactive learning protocol called interactive estimation. In this model, the learner repeatedly queries the environment with an element from a set of alternatives, and observes a stochastic reward whose expectation is given by an arbitrary measure of the "similarity" between the queried alternative and the unknown ground truth. Thus, in rough terms, the goal is to estimate a target from its similarity to queried alternatives. By studying such a general abstraction of interactive learning, we are able to reason about the properties of a very broad family of learning settings, and to make connections across a variety of contexts.
Our results are based on a combinatorial complexity measure we introduce called the dissimilarity dimension, which is used to derive learnability bounds in our model. Intuitively, this measure corresponds to the length of the longest sequence of alternatives in which each one has a similar suboptimal value of similarity to all its predecessors. We then use the measure to analyze the performance of a simple, broadly-applicable class of algorithms which repeatedly make new queries that best fit the preceding observations. We prove both regret bounds and PAC generalization bounds that are all polynomial in the dissimilarity dimension.
We show that our learning framework subsumes two classic learning models that were seemingly unrelated prior to this work:
First, our model subsumes the statistical query (SQ) model, introduced by Kearns [19] for designing noise-tolerant learning algorithms. In the SQ model, the learner can sequentially ask certain queries of an oracle, who responds with answers that are only approximately correct, with the goal of correctly estimating a target. Despite its simplicity, it has been proven to be a powerful model. Indeed, a wide range of algorithmic techniques in machine learning are implementable using SQ learning. Thus, it has been proven useful, not only for designing noise-tolerant algorithms, but also for its connections
to other noise models, and as an explanatory tool to prove hardness of many problems (see the survey of Reyzin [23]). We show that our framework subsumes the SQ model, and furthermore that the dissimilarity dimension generalizes well-known parameters that characterize SQ learnability.
Second, our model captures structured bandits, in which the learner repeatedly chooses actions which yield stochastic rewards, with the goal of minimizing regret relative to the best action in hindsight. Over more than a decade, the eluder dimension [24] has been a central technique for analyzing regret for contextual bandits and reinforcement learning (RL) with function approximation [30, 22, 29, 10]. We will see that the dissimilarity dimension is upper-bounded by the eluder dimension, and that there can in fact be a large gap between the two. This sometimes leads to an improved analysis when relying on the proposed dissimilarity measure rather than the eluder dimension.
Because SQ and bandits are both subsumed by our framework, all the results mentioned above directly apply to those settings as well, including the applicability of our general-purpose algorithms.
To summarize, our main contributions are as follows:
- Unified framework. We derive a general framework which captures various interactive learning settings, including specifically SQ and bandits.
- Novel dimension, performance bounds. We introduce the dissimilarity dimension that is used to derive learnability bounds in our model. We study a general, simple algorithm, and give a novel analysis that results in both regret and PAC generalization bounds that are polynomial in the new dimension. We also give lower bounds in the SQ and bandit settings.
- Improved analysis. We show instances in which the standard analysis of a certain class of algorithms using the eluder dimension yields bounds that are arbitrarily large, but in which an analysis using our dimension yields low regret bounds.
Related work. The interactive estimation model we consider in this work is defined with respect to an evaluation function that can be thought of as an arbitrary measure of the "similarity" between the queried alternative and the target. Previously, Balcan and Blum [3] developed a theory of similarity-based learning that generalizes kernel methods, providing sufficient conditions for a similarity function to be useful for learning. Chen et al. [8] review several approaches to classification based on similarity between examples, including, for instance, kernels and nearest neighbors. Ben-David et al. [5] studied a learning-by-distances model that resembles ours using a metric as a measure of similarity. In comparison to these works, our model admits an arbitrary similarity measure for which we derive a general dimension, algorithm, and bounds.
In the context of bandit and reinforcement learning, a parameter called the decision-estimation coefficient (DEC) has recently been proposed by Foster et al. [15] to characterize learnability in interactive decision making. Unlike DEC, our dimension is combinatorial in nature, and applies to settings like SQ, which are not captured by DEC.
As discussed above, our model subsumes SQ and bandits, both of which have been extensively studied (see the references above as well as various surveys [20, 23]).
In this paper, we study an interactive learning protocol called interactive estimation. In this protocol, the learner is provided with a set  $\mathcal{Z}$  of alternatives, and an evaluation function  $\rho: \mathcal{Z} \times \mathcal{Z} \to [-1, 1]$ . Intuitively,  $\rho$  can be viewed as a measure of "similarity," though it need not be symmetric. There is also a distinguished alternative  $z^* \in \mathcal{Z}$  called the target, fixed throughout the interaction, and unknown to the learner. In each of a sequence of steps  $t = 1, \ldots, T$ , the learner selects one alternative  $z_t \in \mathcal{Z}$  and receives a stochastic reward  $r_t \in [-1, 1]$  drawn independently, conditioned on  $z_t$ , with expectation satisfying  $\mathbb{E}[r_t | z_t] = \rho(z_t | z^*)$ . Informally, by choosing alternatives and observing their similarity to  $z^*$ , the learner aims to get close to the target. The special case when  $r_t = \rho(z_t | z^*)$ , that is, when rewards are deterministic functions of the queried alternatives, is referred to as the deterministic setting.
We generally assume  $\rho(z^{*} \mid z^{*}) \geq \rho(z \mid z^{*})$  for all  $z \in \mathcal{Z}$  and denote this optimal value as  $\alpha^{*} \coloneqq \rho(z^{*} \mid z^{*})$ . We will assume that the value of  $\alpha^{*}$  is known to the learner or that we are provided with an alternate optimality level  $\alpha \leq \alpha^{*}$  such that the task is to identify  $z$  with  $\rho(z \mid z^{*}) \geq \alpha$ . At the end of Section 3 we discuss how this assumption can be relaxed.
We consider two alternative goals for a learner in this model: sublinear regret and PAC generalization. A learner achieves sublinear regret relative to an optimality level  $\alpha \leq \alpha^{*}$  if  $\mathrm{Regret}(T,\alpha) = o(T)$ , where
$$
\operatorname {R e g r e t} (T, \alpha) = \sum_ {t = 1} ^ {T} \left(\alpha - \rho \left(z _ {t} \mid z ^ {*}\right)\right).
$$
We say that a learner achieves PAC generalization if for any  $\epsilon, \delta > 0$  and  $\alpha \leq \alpha^{*}$ , with probability at least  $1 - \delta$  (over the randomness of the query responses and the learner's own randomization), after  $m(\epsilon, \delta, \alpha)$  interactions in the protocol above, the learner outputs  $\hat{z}$  such that  $\rho(\hat{z} \mid z^{*}) \geq \alpha - \epsilon$ . The function  $m(\epsilon, \delta, \alpha)$  is referred to as sample complexity. We recover standard notions of regret and PAC generalization by setting  $\alpha = \alpha^{*}$ .
Example 1 (Point on a sphere). Let  $\|\cdot\|$  denote the standard Euclidean norm in  $\mathbb{R}^n$  and let  $\mathcal{Z} = S_{n-1} = \{\mathbf{z} \in \mathbb{R}^n : \|\mathbf{z}\| = 1\}$  be the unit sphere in  $\mathbb{R}^n$ . The goal is to estimate an unknown point  $\mathbf{z}^* \in S_{n-1}$  based on rewards equal to the inner product between the queries and the target, that is,  $r_t = \rho_{\mathrm{sphere}}(\mathbf{z}_t \mid \mathbf{z}^*) := \langle \mathbf{z}_t, \mathbf{z}^* \rangle$ .
We now introduce the two main examples corresponding to classic learning models that are subsumed by the interactive estimation model.
Example 2 (Structured bandits). Let  $\mathcal{A}$  be an action set,  $\mathcal{F}$  a space of reward functions  $f:\mathcal{A}\to [-1,1]$ , and  $f^{*}\in \mathcal{F}$  the target reward function. In step  $t$ , the learner chooses an action  $a_{t}\in \mathcal{A}$  and receives reward  $r_t\in [-1,1]$  with  $\mathbb{E}[r_t\mid a_t] = f^* (a_t)$ . The goal is to maximize the sum of rewards. Let  $a^* = \operatorname{argmax}_{a\in \mathcal{A}}f^* (a)$  be an optimal action. To represent bandits in our formalism, we let  $\mathcal{Z} = \mathcal{F}\times \mathcal{A},z^{*} = (f^{*},a^{*})$ , and  $\rho_{\mathrm{bandits}}((f,a)\mid (f^{*},a^{*})) = f^{*}(a)$ .
The structured bandit problem has been extensively studied, and Example 2 captures its expressiveness within our framework. For example, it recovers the possibly simplest case of  $K$ -armed bandits, by considering  $\mathcal{A} = \{1, \dots, K\}$  and  $\mathcal{F} = [0,1]^K$ . At each round, the learner chooses arm  $a_t \in \mathcal{A}$  and observes a reward  $r_t$  which is drawn from a distribution with mean  $f^*(a_t)$ . See Appendix D for a more concrete example of  $K$ -armed bandits instantiated within our framework. In Section 5 we also give concrete bounds for other example classes including linear bandits and GLM bandits.
We remark that although the protocol in which the learner submits pairs  $(f, a)$  in each round rather than an action  $a$ , may seem more complicated than the standard bandits protocol, it is in fact equivalent as the function  $f$  is ignored in the evaluation. Moreover, this formulation naturally captures any algorithms for realizable bandits. Such algorithms often keep track of a version space (set of functions  $f$  consistent with the data), and so at each point of interaction, there is an implicit  $f_t$  that is associated with  $a_t$  produced at that time. The above protocol simply makes the choice of  $f_t$  explicit.
Example 3 (SQ learning). Given a domain  $\mathcal{X}$ , the goal is to learn a binary classifier  $h^*:\mathcal{X}\to \{\pm 1\}$  from some hypothesis class  $\mathcal{H}\subseteq \{\pm 1\}^{\mathcal{X}}$ , based on training examples  $(x,y)$  drawn from some distribution  $D$  such that  $y = h^{*}(x)$ . In step  $t$ , the learner produces a hypothesis  $h_t$  and observes the accuracy of  $h_t$  on a fresh finite sample. In this case,  $\mathcal{Z} = \mathcal{H}$ , the evaluation function is equal to the expected accuracy  $\rho_{\mathrm{SQ}}(h|h^*) = \mathbb{E}_{x\sim D}[h(x)h^* (x)]$ , and the reward is the empirical accuracy on a fresh sample.
The SQ learning model considered in this work (Example 3) differs from the original model of Kearns [19] because it is restricted, as in previous works [7, 13, 31], to so-called correlational queries (called CSQs) and assumes stochastic responses, as opposed to allowing arbitrary queries and adversarial responses. We discuss relationships between various SQ variants in Appendix B.
We finish this section by introducing a central concept of this paper, a new combinatorial complexity measure called the dissimilarity dimension which, as we will see, allows us to derive learnability bounds for the interactive estimation protocol.
Definition 1 (Dissimilarity dimension). For a set  $\mathcal{Z}$ , scalars  $\alpha \in \mathbb{R}$ ,  $\epsilon > 0$ , and evaluation function  $\rho: \mathcal{Z} \times \mathcal{Z} \to [-1,1]$ , the dissimilarity dimension  $\mathrm{d}_{\rho}(\mathcal{Z}, \alpha, \epsilon)$  is the largest integer  $d$  for which there exist  $z_1, \ldots, z_d \in \mathcal{Z}$  with  $\rho(z_i \mid z_i) \geq \alpha$ , and a scalar  $c \leq \alpha - \epsilon$ , such that for all  $i < j$ ,
$$
\left| \rho (z _ {i} \mid z _ {j}) - c \right| \leq \frac {\epsilon}{\sqrt {d}}.
$$
Figure 1: An illustration of the dissimilarity dimension  $\mathsf{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon) = 4$  . The figure shows  $\rho$  values on the [0, 1] interval for a sequence of elements  $z_{1},z_{2},z_{3},z_{4}$  . The four green points represent self-evaluation values  $\rho (z_i\mid z_i)$  for  $i = 1,\ldots ,4$  ; all are greater than or equal to  $\alpha$  The six blue points represent values  $\rho (z_i\mid z_j)$  for  $i <   j$  ; all are within the distance  $\epsilon /\sqrt{d}$  from the value  $c\leq \alpha -\epsilon$
Furthermore, denote the monotonic dissimilarity dimension as  $\overline{\mathrm{d}}_{\rho}(\mathcal{Z},\alpha ,\epsilon)\coloneqq \max_{\epsilon^{\prime}\geq \epsilon}\mathrm{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon^{\prime})$
Note that  $\mathsf{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon) = 0$  if there is no  $z$  such that  $\rho (z\mid z)\geq \alpha$ , and otherwise  $\mathsf{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon)\geq 1$ . In particular, if  $\alpha \leq \alpha^{*}$  then  $\mathsf{d}_{\rho}(\mathcal{Z},\alpha ,\epsilon)\geq 1$ .
In rough terms, this dimension corresponds to the longest sequence of points with  $\alpha$ -large self-evaluation, such that the evaluation  $\rho(z_i \mid z_j)$  of each point  $z_i$  relative to every successive point  $z_j$  is "small" (significantly less than  $\alpha$ ), and also tightly clustered around some value  $c$ . Thus, each point is similar to itself, but dissimilar from all successive points to about the same degree. The idea is illustrated in Figure 1. The monotonic dissimilarity dimension is the tightest upper bound on the dissimilarity dimension that is non-increasing in  $\epsilon$ .
Various concrete examples where the dissimilarity dimension can be bounded are provided in Section 5. For instance, using a general bound for linear bandits from Section 5, we can show that for the task of finding a point on a sphere based on inner products (Example 1), the dimension  $\overline{\mathsf{d}}_{\rho_{\mathrm{sphere}}}(\mathcal{Z},\alpha ,\epsilon)\leq 4n + 3$ , a bound that is independent of both  $\alpha$  and  $\epsilon$ .
3 Algorithms and upper bounds
In this section we analyze algorithms for the interactive estimation protocol, which we call interactive estimation algorithms. We show that when an interactive estimation algorithm satisfies two properties, large self-evaluations and decaying estimation error, then its regret can be bounded using the dissimilarity dimension. We introduce a simple algorithm (Algorithm 1), which satisfies these properties for many standard classes of alternatives. The first property requires that the algorithm only select alternatives that would achieve the expected reward of at least  $\alpha$  if they were the target:
Definition 2 ( $\alpha$ -large self-evaluations). An interactive estimation algorithm has  $\alpha$ -large self-evaluations if at every time step  $t = 1, \ldots, T$ , it selects a query  $z_t$  such that  $z_t \in \mathcal{Z}_{\alpha}$ , where
$$
\mathcal {Z} _ {\alpha} = \{z \in \mathcal {Z}: \rho (z \mid z) \geq \alpha \}. \tag {1}
$$
Algorithm 1 satisfies this property, with the optimality level  $\alpha$  provided as input. At the end of the section, we discuss the case when  $\alpha$  is not provided and  $\alpha^{*}$  is unknown. We derive an optimistic version of Algorithm 1 that achieves  $\alpha^{*}$ -large self-evaluations with high probability.
The second property states that the queries produced by the algorithm provide increasingly good estimates of the expected rewards in the previous rounds (that is they are good estimators with the benefit of hindsight), as quantified by the square loss.
Definition 3 (Decaying estimation error). An interactive estimation algorithm has decaying estimation error if there exists  $C_{T,\delta} \geq 0$  growing sublinearly in  $T$ , that is,  $C_{T,\delta} = o(T)$ , such that with
<sup>1</sup>We remark that the term  $\sqrt{d}$  in Definition 1 can be generalized to any function  $g(d)$  and state our main results in terms of  $g$ . However, for ease of presentation we pick  $g(d) = \sqrt{d}$  as it simplifies the comparison between the dissimilarity dimension and eluder dimension (e.g. Thm. 11 and Prop. 12).
Algorithm 1 Interactive Estimation via Least Squares
1: Input: set of alternatives  $\mathcal{Z}$ , evaluation function  $\rho$ , optimality level  $\alpha$ , number of steps  $T$ .
2: for  $t = 1, \dots, T$  do
3: Submit the query  $z_{t} = \operatorname*{argmin}_{z\in \mathcal{Z}_{\alpha}}\sum_{i = 1}^{t - 1}\Bigl (\rho (z_{i}\mid z) - r_{i}\Bigr)^{2}$ .
4: Observe reward  $r_t$ .
5: end for
probability at least  $1 - \delta$  the sequence of queries  $z_{1},\ldots ,z_{T}$  produced by the algorithm satisfies
$$
\sum_ {i = 1} ^ {t - 1} \left(\rho \left(z _ {i} \mid z _ {t}\right) - \rho \left(z _ {i} \mid z ^ {*}\right)\right) ^ {2} \leq C _ {T, \delta} \tag {2}
$$
for all  $t\in \{1,\ldots ,T\}$  simultaneously.
Algorithm 1 optimizes an empirical version of Eq. (2), with the observed rewards  $r_i$  in place of the expectations  $\rho(z_i \mid z^*)$ . Thus, in the deterministic setting, with  $r_i = \rho(z_i \mid z^*)$ , Algorithm 1 satisfies this property with  $C_{T,\delta} = 0$ . It can also be shown that it satisfies this property when the set of alternatives is finite:
Theorem 4. Assume that  $|\mathcal{Z}| < \infty$ . Then Algorithm 1 satisfies the decaying estimation error property with  $C_{T,\delta} = O\big(\ln (T|\mathcal{Z}| / \delta)\big)$ .
In the general case, when  $\mathcal{Z}$  is infinite, we show that Algorithm 1 satisfies the decaying estimation error property with  $C_{T,\delta} = O\bigl (\log (TN / \delta)\bigr)$  where  $N$  is a suitable covering number of  $\mathcal{Z}$  (see Corollary 18 in Appendix A.1). For example, for linear bandits, which is an instance of Example 2 in which the action set and function class correspond to a subset of the unit ball in  $\mathbb{R}^n$ , we obtain  $C_{T,\delta} = O\big(n\log (1 / \epsilon) + \log (T / \delta)\big)$ .
In Appendix A.3, we discuss an approach in which we have access to an online regression oracle for the least squares problem in step 3. We show that a suitably modified version of Algorithm 1 has a decaying estimation error as long as the online regression oracle achieves a sublinear regret (but without further dependence on a covering number).
To develop some intuition how Algorithm 1 works, we can again consider the  $K$ -armed bandit problem (a special case of Example 2), and suppose that  $\alpha = 0.75$ . In each step, the algorithm picks a pair  $(f_t, a_t)$ , where  $f_t \in [0,1]^K$  is the vector of mean reward estimates and  $a_t$  is the arm with the largest mean estimate. The estimates  $f_t(a)$ ,  $a = 1, \ldots, K$ , are formed by optimizing the least squares error of the observed rewards, under the constraint that at least one of the mean estimates must be above 0.75. As a result, the algorithm pulls the arm with the largest average reward as long as that average is above 0.75 (arms that have not been pulled are assumed to have averages above 0.75). If all the averages are below 0.75 then the algorithm selects the arm  $a$  with the smallest value  $n_a(0.75 - \hat{\mu}_a)^2$ , where  $n_a$  is how many times the arm has been pulled so far and  $\hat{\mu}_a$  is its average reward; it can be verified that this solves the least squares problem subject to the constraint that at least one of the mean estimates is above 0.75.
We next state our main results: a regret bound and a PAC generalization guarantee. They are both based on bounding how many "bad" queries any algorithm with large self-evaluations and decaying estimation error can make. Concretely, we say that a query  $z \in \mathcal{Z}$  is  $\epsilon$ -bad if its suboptimality gap is greater than  $\epsilon$ , that is, if
$$
\rho (z \mid z ^ {*}) <   \alpha - \epsilon .
$$
The next lemma shows that the number of  $\epsilon$ -bad queries is upper bounded polynomially in the dissimilarity dimension.
Lemma 5 (Few bad queries). Let  $\epsilon, \delta > 0$ , and let  $d = \overline{\mathrm{d}}_{\rho}(\mathcal{Z}, \alpha, \epsilon) < \infty$  for some set  $\mathcal{Z}$ , evaluation function  $\rho$  and  $\alpha \leq \alpha^{*}$ . Let  $\mathsf{Alg}$  be an interactive estimation algorithm with  $\alpha$ -large self-evaluations and a decaying estimation error with some  $C_{T,\delta}$ . Then, with probability at least  $1 - \delta$ , the number of  $\epsilon$ -bad queries that  $\mathsf{Alg}$  makes in  $T$  steps is at most  $2d^{1.5}\ln(4/\epsilon) + 12d^{2.5}C_{T,\delta}/\epsilon^2$ . Consequently, if  $C_{T,\delta} \geq \ln(2T)$ , then with probability at least  $1 - \delta$ , the number of  $\epsilon$ -bad queries is at most  $36d^{2.5}C_{T,\delta}/\epsilon^2$ , and if  $C_{T,\delta} = 0$  then it is at most  $2d^{1.5}\ln(4/\epsilon)$ .
Algorithm 2 PAC Interactive Estimation
1: Input: set of alternatives  $\mathcal{Z}$ , evaluation function  $\rho$ , optimality level  $\alpha$ , base interactive estimation algorithm Alg, parameters  $T$ ,  $n_1$ ,  $n_2$ .
2: Run algorithm Alg with the provided  $\mathcal{Z},\rho ,\alpha ,T$
3: Sample  $n_1$  indices  $t_1, \ldots, t_{n_1}$  uniformly at random from  $\{1, \ldots, T\}$ .
4: For each  $\ell = 1,\dots ,n_{1}$  , submit query  $z_{t_\ell}$  for  $n_2$  times; denote the average response  $\bar{r}_{t_\ell}$
5: Let  $\ell = \operatorname{argmax}_{\ell \in \{1, \dots, n_1\}} \bar{r}_{t_\ell}$ .
6: Output  $\hat{z} = z_{t_{\hat{\varepsilon}}}$
The above result is the core component of our main theorems. The proof is given in Appendix A.4; here we sketch the main ideas. The goal is to show that the "bad" interval  $[-1, \alpha - \epsilon]$  cannot contain too many queries made by Alg. The proof starts by partitioning this interval into disjoint subintervals and then bounds the number of queries in each subinterval. It does so by constructing a graph with nodes corresponding to queries, which are connected by an edge if they satisfy the dimension conditions. The decaying errors that imply a certain minimum number of edges (as a function of number of queries). On the other hand, the dissimilarity dimension bounds the size of the largest clique, which implies an upper bound on the number of edges (using Turán's Theorem [26], a standard result from extremal graph theory). Combining the bounds yields an upper bound on the number of queries in the subinterval. Summing across subintervals proves the lemma.
The following theorems use Lemma 5 to bound both the regret and PAC sample complexity. The proofs are deferred to Appendices A.6 and A.7.