Title: A Unified Model and Dimension for Interactive Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interactive learning

bandits

statistical queries
Abstract: We study an abstract framework for interactive learning called interactive estimation in which the goal is to estimate a target from its ``similarity'' to points queried by the learner.
We introduce a combinatorial measure called Dissimilarity dimension which largely captures learnability in our model.
We present a simple, general, and broadly-applicable algorithm, for which we obtain both regret and PAC generalization bounds that are polynomial in the new dimension. We show that our framework subsumes and thereby unifies two classic learning models:
statistical-query learning and structured bandits. We also delineate how the Dissimilarity dimension is related to well-known parameters for both frameworks, in some cases yielding significantly improved analyses.

A Unified Model and Dimension for Interactive Estimation

Abstract

We study an abstract framework for interactive learning called interactive estimation in which the goal is to estimate a target from its 'similarity' to points queried by the learner. We introduce a combinatorial measure called dissimilarity dimension which is used to derive learnability bounds in our model. We present a simple, general, and broadly-applicable algorithm, for which we obtain both regret and PAC generalization bounds that are polynomial in the new dimension. We show that our framework subsumes and thereby unifies two classic learning models: statistical-query learning and structured bandits. We also delineate how the dissimilarity dimension is related to well-known parameters for both frameworks, in some cases yielding significantly improved analyses.

1 Introduction

We study a general interactive learning protocol called interactive estimation . In this model, the learner repeatedly queries the environment with an element from a set of alternatives , and observes a stochastic reward whose expectation is given by an arbitrary measure of the 'similarity' between the queried alternative and the unknown ground truth. Thus, in rough terms, the goal is to estimate a target from its similarity to queried alternatives. By studying such a general abstraction of interactive learning, we are able to reason about the properties of a very broad family of learning settings, and to make connections across a variety of contexts.

Our results are based on a combinatorial complexity measure we introduce called the dissimilarity dimension , which is used to derive learnability bounds in our model. Intuitively, this measure corresponds to the length of the longest sequence of alternatives in which each one has a similar suboptimal value of similarity to all its predecessors. We then use the measure to analyze the performance of a simple, broadly-applicable class of algorithms which repeatedly make new queries that best fit the preceding observations. We prove both regret bounds and PAC generalization bounds that are all polynomial in the dissimilarity dimension.

We show that our learning framework subsumes two classic learning models that were seemingly unrelated prior to this work:

First, our model subsumes the statistical query (SQ) model, introduced by Kearns [19] for designing noise-tolerant learning algorithms. In the SQ model, the learner can sequentially ask certain queries of an oracle, who responds with answers that are only approximately correct, with the goal of correctly estimating a target. Despite its simplicity, it has been proven to be a powerful model. Indeed, a wide range of algorithmic techniques in machine learning are implementable using SQ learning. Thus, it has been proven useful, not only for designing noise-tolerant algorithms, but also for its connections

to other noise models, and as an explanatory tool to prove hardness of many problems (see the survey of Reyzin [23]). We show that our framework subsumes the SQ model, and furthermore that the dissimilarity dimension generalizes well-known parameters that characterize SQ learnability.

Second, our model captures structured bandits, in which the learner repeatedly chooses actions which yield stochastic rewards, with the goal of minimizing regret relative to the best action in hindsight. Over more than a decade, the eluder dimension [24] has been a central technique for analyzing regret for contextual bandits and reinforcement learning (RL) with function approximation [30, 22, 29, 10]. We will see that the dissimilarity dimension is upper-bounded by the eluder dimension, and that there can in fact be a large gap between the two. This sometimes leads to an improved analysis when relying on the proposed dissimilarity measure rather than the eluder dimension.

Because SQ and bandits are both subsumed by our framework, all the results mentioned above directly apply to those settings as well, including the applicability of our general-purpose algorithms.

To summarize, our main contributions are as follows:

Unified framework. We derive a general framework which captures various interactive learning settings, including specifically SQ and bandits.

Novel dimension, performance bounds. We introduce the dissimilarity dimension that is used to derive learnability bounds in our model. We study a general, simple algorithm, and give a novel analysis that results in both regret and PAC generalization bounds that are polynomial in the new dimension. We also give lower bounds in the SQ and bandit settings.

Improved analysis. We show instances in which the standard analysis of a certain class of algorithms using the eluder dimension yields bounds that are arbitrarily large, but in which an analysis using our dimension yields low regret bounds.

Related work. The interactive estimation model we consider in this work is defined with respect to an evaluation function that can be thought of as an arbitrary measure of the 'similarity' between the queried alternative and the target. Previously, Balcan and Blum [3] developed a theory of similaritybased learning that generalizes kernel methods, providing sufficient conditions for a similarity function to be useful for learning. Chen et al. [8] review several approaches to classification based on similarity between examples, including, for instance, kernels and nearest neighbors. Ben-David et al. [5] studied a learning-by-distances model that resembles ours using a metric as a measure of similarity. In comparison to these works, our model admits an arbitrary similarity measure for which we derive a general dimension, algorithm, and bounds.

In the context of bandit and reinforcement learning, a parameter called the decision-estimation coefficient (DEC) has recently been proposed by Foster et al. [15] to characterize learnability in interactive decision making. Unlike DEC, our dimension is combinatorial in nature, and applies to settings like SQ, which are not captured by DEC.

As discussed above, our model subsumes SQ and bandits, both of which have been extensively studied (see the references above as well as various surveys [20, 23]).

2 Setting

In this paper, we study an interactive learning protocol called interactive estimation . In this protocol, the learner is provided with a set Z of alternatives , and an evaluation function ρ : Z × Z → [ -1 , 1] . Intuitively, ρ can be viewed as a measure of 'similarity,' though it need not be symmetric. There is also a distinguished alternative z ∗ ∈ Z called the target , fixed throughout the interaction, and unknown to the learner. In each of a sequence of steps t = 1 , . . . , T , the learner selects one alternative z t ∈ Z and receives a stochastic reward r t ∈ [ -1 , 1] drawn independently, conditioned on z t , with expectation satisfying E [ r t | z t ] = ρ ( z t | z ∗ ) . Informally, by choosing alternatives and observing their similarity to z ∗ , the learner aims to get close to the target. The special case when r t = ρ ( z t | z ∗ ) , that is, when rewards are deterministic functions of the queried alternatives, is referred to as the deterministic setting .

We generally assume ρ ( z ∗ | z ∗ ) ≥ ρ ( z | z ∗ ) for all z ∈ Z and denote this optimal value as α ∗ := ρ ( z ∗ | z ∗ ) . We will assume that the value of α ∗ is known to the learner or that we are provided with an alternate optimality level α ≤ α ∗ such that the task is to identify z with ρ ( z | z ∗ ) ≥ α . At the end of Section 3 we discuss how this assumption can be relaxed.

We consider two alternative goals for a learner in this model: sublinear regret and PAC generalization . A learner achieves sublinear regret relative to an optimality level α ≤ α ∗ if Regret ( T, α ) = o ( T ) , where

We say that a learner achieves PAC generalization if for any ϵ, δ > 0 and α ≤ α ∗ , with probability at least 1 -δ (over the randomness of the query responses and the learner's own randomization), after m ( ϵ, δ, α ) interactions in the protocol above, the learner outputs ˆ z such that ρ (ˆ z | z ∗ ) ≥ α -ϵ . The function m ( ϵ, δ, α ) is referred to as sample complexity. We recover standard notions of regret and PAC generalization by setting α = α ∗ .

Example 1 (Point on a sphere) . Let ∥·∥ denote the standard Euclidean norm in R n and let Z = S n -1 = { z ∈ R n : ∥ z ∥ = 1 } be the unit sphere in R n . The goal is to estimate an unknown point z ∗ ∈ S n -1 based on rewards equal to the inner product between the queries and the target, that is, r t = ρ sphere ( z t | z ∗ ) := ⟨ z t , z ∗ ⟩ .

We now introduce the two main examples corresponding to classic learning models that are subsumed by the interactive estimation model.

Example 2 (Structured bandits) . Let A be an action set, F a space of reward functions f : A → [ -1 , 1] , and f ∗ ∈ F the target reward function. In step t , the learner chooses an action a t ∈ A and receives reward r t ∈ [ -1 , 1] with E [ r t | a t ] = f ∗ ( a t ) . The goal is to maximize the sum of rewards. Let a ∗ = argmax a ∈A f ∗ ( a ) be an optimal action. To represent bandits in our formalism, we let Z = F × A , z ∗ = ( f ∗ , a ∗ ) , and ρ bandits ( ( f, a ) ∣ ∣ ( f ∗ , a ∗ ) ) = f ∗ ( a ) .

The structured bandit problem has been extensively studied, and Example 2 captures its expressiveness within our framework. For example, it recovers the possibly simplest case of K -armed bandits, by considering A = { 1 , ..., K } and F = [0 , 1] K . At each round, the learner chooses arm a t ∈ A and observes a reward r t which is drawn from a distribution with mean f ∗ ( a t ) . See Appendix D for a more concrete example of K -armed bandits instantiated within our framework. In Section 5 we also give concrete bounds for other example classes including linear bandits and GLM bandits.

We remark that although the protocol in which the learner submits pairs ( f, a ) in each round rather than an action a , may seem more complicated than the standard bandits protocol, it is in fact equivalent as the function f is ignored in the evaluation. Moreover, this formulation naturally captures any algorithms for realizable bandits. Such algorithms often keep track of a version space (set of functions f consistent with the data), and so at each point of interaction, there is an implicit f t that is associated with a t produced at that time. The above protocol simply makes the choice of f t explicit.

Example 3 (SQ learning) . Given a domain X , the goal is to learn a binary classifier h ∗ : X → {± 1 } from some hypothesis class H ⊆ {± 1 } X , based on training examples ( x, y ) drawn from some distribution D such that y = h ∗ ( x ) . In step t , the learner produces a hypothesis h t and observes the accuracy of h t on a fresh finite sample. In this case, Z = H , the evaluation function is equal to the expected accuracy ρ SQ ( h | h ∗ ) = E x ∼ D [ h ( x ) h ∗ ( x )] , and the reward is the empirical accuracy on a fresh sample.

The SQ learning model considered in this work (Example 3) differs from the original model of Kearns [19] because it is restricted, as in previous works [7, 13, 31], to so-called correlational queries (called CSQs) and assumes stochastic responses, as opposed to allowing arbitrary queries and adversarial responses. We discuss relationships between various SQ variants in Appendix B.

We finish this section by introducing a central concept of this paper, a new combinatorial complexity measure called the dissimilarity dimension which, as we will see, allows us to derive learnability bounds for the interactive estimation protocol.

Definition 1 (Dissimilarity dimension) . For a set Z , scalars α ∈ R , ϵ > 0 , and evaluation function ρ : Z × Z → [ -1 , 1] , the dissimilarity dimension d ρ ( Z , α, ϵ ) is the largest integer d for which there exist z 1 , . . . , z d ∈ Z with ρ ( z i | z i ) ≥ α , and a scalar c ≤ α -ϵ , such that for all i < j ,

(

(

Furthermore, denote the monotonic dissimilarity dimension as d ρ ( Z , α, ϵ ) := max ϵ ′ ≥ ϵ d ρ ( Z , α, ϵ ′ ) . 1

Note that d ρ ( Z , α, ϵ ) = 0 if there is no z such that ρ ( z | z ) ≥ α , and otherwise d ρ ( Z , α, ϵ ) ≥ 1 . In particular, if α ≤ α ∗ then d ρ ( Z , α, ϵ ) ≥ 1 .

In rough terms, this dimension corresponds to the longest sequence of points with α -large selfevaluation, such that the evaluation ρ ( z i | z j ) of each point z i relative to every successive point z j is 'small' (significantly less than α ), and also tightly clustered around some value c . Thus, each point is similar to itself, but dissimilar from all successive points to about the same degree. The idea is illustrated in Figure 1. The monotonic dissimilarity dimension is the tightest upper bound on the dissimilarity dimension that is non-increasing in ϵ .

Various concrete examples where the dissimilarity dimension can be bounded are provided in Section 5. For instance, using a general bound for linear bandits from Section 5, we can show that for the task of finding a point on a sphere based on inner products (Example 1), the dimension d ρ sphere ( Z , α, ϵ ) ≤ 4 n +3 , a bound that is independent of both α and ϵ .

3 Algorithms and upper bounds

In this section we analyze algorithms for the interactive estimation protocol, which we call interactive estimation algorithms . We show that when an interactive estimation algorithm satisfies two properties, large self-evaluations and decaying estimation error , then its regret can be bounded using the dissimilarity dimension. We introduce a simple algorithm (Algorithm 1), which satisfies these properties for many standard classes of alternatives. The first property requires that the algorithm only select alternatives that would achieve the expected reward of at least α if they were the target:

Definition 2 ( α -large self-evaluations) . An interactive estimation algorithm has α -large selfevaluations if at every time step t = 1 , . . . , T , it selects a query z t such that z t ∈ Z α , where

Algorithm 1 satisfies this property, with the optimality level α provided as input. At the end of the section, we discuss the case when α is not provided and α ∗ is unknown. We derive an optimistic version of Algorithm 1 that achieves α ∗ -large self-evaluations with high probability.

The second property states that the queries produced by the algorithm provide increasingly good estimates of the expected rewards in the previous rounds (that is they are good estimators with the benefit of hindsight ), as quantified by the square loss.

Definition 3 (Decaying estimation error) . An interactive estimation algorithm has decaying estimation error if there exists C T,δ ≥ 0 growing sublinearly in T , that is, C T,δ = o ( T ) , such that with

√

Algorithm 1 Interactive Estimation via Least Squares

1: Input: set of alternatives Z , evaluation function ρ , optimality level α , number of steps T .

2: for t = 1 , . . . , T do

3: Submit the query z t = argmin z ∈Z α t -1 ∑ i =1 ( ρ ( z i | z ) -r i ) 2 .

4: Observe reward r t .

5: end for

probability at least 1 -δ the sequence of queries z 1 , . . . , z T produced by the algorithm satisfies

for all t ∈ { 1 , . . . , T } simultaneously.

Algorithm 1 optimizes an empirical version of Eq. (2), with the observed rewards r i in place of the expectations ρ ( z i | z ∗ ) . Thus, in the deterministic setting, with r i = ρ ( z i | z ∗ ) , Algorithm 1 satisfies this property with C T,δ = 0 . It can also be shown that it satisfies this property when the set of alternatives is finite:

Theorem 4. Assume that |Z| < ∞ . Then Algorithm 1 satisfies the decaying estimation error property with C T,δ = O ( ln( T |Z| /δ ) ) .

In the general case, when Z is infinite, we show that Algorithm 1 satisfies the decaying estimation error property with C T,δ = O ( log( TN/δ ) ) where N is a suitable covering number of Z (see Corollary 18 in Appendix A.1). For example, for linear bandits , which is an instance of Example 2 in which the action set and function class correspond to a subset of the unit ball in R n , we obtain C T,δ = O ( n log(1 /ϵ ) + log( T/δ ) ) .

In Appendix A.3, we discuss an approach in which we have access to an online regression oracle for the least squares problem in step 3. We show that a suitably modified version of Algorithm 1 has a decaying estimation error as long as the online regression oracle achieves a sublinear regret (but without further dependence on a covering number).

To develop some intuition how Algorithm 1 works, we can again consider the K -armed bandit problem (a special case of Example 2), and suppose that α = 0 . 75 . In each step, the algorithm picks a pair ( f t , a t ) , where f t ∈ [0 , 1] K is the vector of mean reward estimates and a t is the arm with the largest mean estimate. The estimates f t ( a ) , a = 1 , . . . , K , are formed by optimizing the least squares error of the observed rewards, under the constraint that at least one of the mean estimates must be above 0 . 75 . As a result, the algorithm pulls the arm with the largest average reward as long as that average is above 0 . 75 (arms that have not been pulled are assumed to have averages above 0 . 75 ). If all the averages are below 0 . 75 then the algorithm selects the arm a with the smallest value n a (0 . 75 -ˆ µ a ) 2 , where n a is how many times the arm has been pulled so far and ˆ µ a is its average reward; it can be verified that this solves the least squares problem subject to the constraint that at least one of the mean estimates is above 0 . 75 .

We next state our main results: a regret bound and a PAC generalization guarantee. They are both based on bounding how many 'bad' queries any algorithm with large self-evaluations and decaying estimation error can make. Concretely, we say that a query z ∈ Z is ϵ -bad if its suboptimalty gap is greater than ϵ , that is, if

The next lemma shows that the number of ϵ -bad queries is upper bounded polynomially in the dissimilarity dimension.

Lemma 5 (Few bad queries) . Let ϵ, δ > 0 , and let d = d ρ ( Z , α, ϵ ) < ∞ for some set Z , evaluation function ρ and α ≤ α ∗ . Let Alg be an interactive estimation algorithm with α -large self-evaluations and a decaying estimation error with some C T,δ . Then, with probability at least 1 -δ , the number of ϵ -bad queries that Alg makes in T steps is at most 2 d 1 . 5 ln(4 /ϵ ) + 12 d 2 . 5 C T,δ /ϵ 2 . Consequently, if C T,δ ≥ ln(2 T ) , then with probability at least 1 -δ , the number of ϵ -bad queries is at most 36 d 2 . 5 C T,δ /ϵ 2 , and if C T,δ = 0 then it is at most 2 d 1 . 5 ln(4 /ϵ ) .

Algorithm 2 PAC Interactive Estimation

1: Input: set of alternatives Z , evaluation function ρ , optimality level α ,

base interactive estimation algorithm Alg , parameters T , n 1 , n 2 .

2: Run algorithm Alg with the provided Z , ρ, α, T .

3: Sample n 1 indices t 1 , . . . , t n 1 uniformly at random from { 1 , . . . , T } .

4: For each ℓ = 1 , . . . , n 1 , submit query z t ℓ for n 2 times; denote the average response ¯ r t ℓ .

5: Let ˆ ℓ = argmax ℓ ∈{ 1 ,...,n 1 } ¯ r t ℓ .

6: Output ˆ z = z t ˆ ℓ .

The above result is the core component of our main theorems. The proof is given in Appendix A.4; here we sketch the main ideas. The goal is to show that the 'bad' interval [ -1 , α -ϵ ] cannot contain too many queries made by Alg . The proof starts by partitioning this interval into disjoint subintervals and then bounds the number of queries in each subinterval. It does so by constructing a graph with nodes corresponding to queries, which are connected by an edge if they satisfy the dimension conditions. The decaying errors that imply a certain minimum number of edges (as a function of number of queries). On the other hand, the dissimilarity dimension bounds the size of the largest clique, which implies an upper bound on the number of edges (using Turán's Theorem [26], a standard result from extremal graph theory). Combining the bounds yields an upper bound on the number of queries in the subinterval. Summing across subintervals proves the lemma.

The following theorems use Lemma 5 to bound both the regret and PAC sample complexity. The proofs are deferred to Appendices A.6 and A.7.

Theorem 6 (Regret) . Let δ, T > 0 , and let d = d ρ ( Z , α, 1 /T ) for some set Z , evaluation function ρ and α ≤ α ∗ . Let Alg be an interactive estimation algorithm with α -large self-evaluations and a decaying estimation error with some C T,δ . If C T,δ ≥ ln(2 T ) then with probability at least 1 -δ , the regret of Alg satisfies

In the deterministic setting, Regret( T, α ) ≤ 1 + 12 d 1 . 5 .

For an algorithm with a decaying estimation error, the term C T,δ is sublinear in T , implying a sublinear regret in Theorem 6. For example, Algorithm 1 has a decaying estimation error with C T,δ that scales logarithmically with T/δ for many standard function classes, and so the overall regret scales as O ( √ T log T ) (see Corollary 18 in Appendix A.1).

To derive PAC generalization guarantees, we apply a variant of online-to-batch reduction to any algorithm with large self-evaluations and a decaying estimation error. The resulting approach, shown in Algorithm 2, satisfies the following guarantee (proved in Appendix A.7):

Theorem 7 (PAC generalization) . Let ϵ, δ > 0 , and let d = d ρ ( Z , α, ϵ ) for some set Z , evaluation function ρ and α ≤ α ∗ . Let Alg be an interactive estimation algorithm with α -large self-evaluations and a decaying estimation error with C T,δ ≥ ln(2 T ) , and suppose that we run Algorithm 2 with Alg as the base algorithm, T ≥ 64 d 2 . 5 ( C T,δ/ 2 ) /ϵ 2 , n 1 = ⌈ log 2 (4 /δ ) ⌉ , and n 2 = ⌈ 128 ln(8 n 1 /δ ) /ϵ 2 ⌉ . Then, with probability at least 1 -δ , the output ˆ z ∈ Z satisfies

and the overall number of issued queries is O ( d ( C T,δ/ 2 )+ln (1 /δ ) ϵ 2 ) .

In the deterministic setting, it suffices to run Alg with T > 2 d 1 . 5 ln(4 /ϵ ) and return ˆ z = z ˆ t where ˆ t = argmax t ∈{ 1 ,...,T } r t is the index of the largest observed reward. Then, with probability 1, we obtain ρ (ˆ z | z ∗ ) ≥ α -ϵ and issue at most O ( d 1 . 5 ln(4 /ϵ )) queries.

Unknown α ∗ and optimism. Algorithms 1 and 2 achieve performance guarantees with respect to a provided optimality level α ≤ α ∗ . When it is not easy to provide a non-trivial α (for example, when α ∗ is unknown and cannot be non-trivially bounded), Algorithm 3 uses the optimistic least squares algorithmic template (see, e.g., [24]) to ensure α ∗ -large self-evaluations with high probability and to achieve a sublinear Regret ( T, α ∗ ) . Algorithm 3 takes as input a confidence radius parameter R of the same order as the decaying estimation error parameter C T,δ for Algorithm 1. We can then show that z ∗ ∈ Z t with high probability for all t ∈ { 1 , . . . , T } . Therefore, z t must satisfy

Algorithm 3 Optimistic Interactive Estimation via Least Squares

1: Input: set of alternatives Z , evaluation function ρ , number of steps T , confidence-set radius R .

2: for t = 1 , . . . , T do

3: Compute confidence set

4: Submit the query z t = argmax z ∈Z t ρ ( z | z ) .

5: Observe reward r t .

6: end for

ρ ( z t | z t ) ≥ ρ ( z ∗ | z ∗ ) = α ∗ . In Appendix A.1 we show this modified version of the algorithm satisfies the decaying estimation error property. This technique allows us to achieve a sublinear Regret ( T, α ∗ ) without knowing α ∗ beforehand. Similar to the case of fixed α , it is possible to derive a version of Algorithm 3 that leverages an online regression oracle. (See Appendix A.3 for details.)

4 Statistical queries

In this section we consider the statistical query (SQ) model, as defined in Example 3. In particular, we study the connection between our generalized framework and SQ learning, showing specifically that the dissimilarity dimension can be used to recover generalization bounds based on a known combinatorial parameter that characterizes SQ learning, called the strong SQ dimension . There are several notions of such a dimension [12, 25]. Here we focus on the one due to Szörényi [25]:

Definition 4 (Strong SQ dimension, [25]) . For a fixed distribution D over X , the strong SQ dimension of a hypothesis class H ⊆ {± 1 } X with respect to some ϵ > 0 , denoted dim SQ ( H , ϵ ) , is the largest number d for which there exist h 1 , . . . , h d ∈ H such that:

(a) |⟨ h i , h j ⟩| ≤ 1 -ϵ for all 1 ≤ i < j ≤ d , and

The dissimilarity and strong SQ dimensions are closely related to one another in the sense of each providing a kind of polynomial bound on the other, as stated in the next proposition (see Appendix B.1 for the proof).

Proposition 8. Let D be a fixed distribution over X , and let H ⊆ {± 1 } X be a hypotheses class. For ϵ > 0 , let d SQ ( ϵ ) = dim SQ ( H , ϵ ) , and let d ρ ( ϵ ) = d ρ SQ ( H , 1 , ϵ ) .

Similarly, if d ρ (4 ϵ ) ≥ 2 then

We next give a lower bound based on the strong SQ dimension, which together with Proposition 8 will allow us to lower bound sample complexity of any interactive estimation algorithm in the SQ setting in terms of the dissimilarity dimension.

Theorem 9 (SQ lower bound) . Let ϵ > 0 , and let H ⊆ {± 1 } X be a hypothesis class with strong SQ dimension d SQ = dim SQ ( H , 2 ϵ ) ≥ 11 . Let Alg be any interactive estimation algorithm with the property that for any target h ∗ ∈ H , Alg outputs an ϵ -approximation to h ∗ with probability at least 2 / 3 using at most m queries. Then m> 3 √ d SQ / 12 .

The proof relies on a reduction to a lower bound of Szörényi [25]. However, the lower bound of Szörényi [25] holds within an SQ model that differs from ours, in that it allows adversarial query responses. Therefore, we first need to show how to obtain a learning algorithm Alg ′ that can be used with an adversarial oracle from an interactive estimation algorithm Alg that uses an unbiased stochastic query oracle (as we assume in this work). To do this, we apply the reduction technique developed by Feldman et al. [13]. (See Appendix B.2 for the full proof and additional details.)

Combining Theorem 9 and Proposition 8 yields a lower bound on the sample complexity of interactive estimation in the SQ setting, for a sufficiently small ϵ , in terms of the dissimilarity dimension:

Corollary 10. Let ϵ > 0 , and let H ⊆ {± 1 } X be a hypothesis class with strong SQ dimension dim SQ ( H , 2 ϵ ) ≥ 11 . Let d ρ ( ϵ ) = d ρ SQ ( H , 1 , ϵ ) . Assume ϵ ≤ 1 / ( 2 √ d ρ ( ϵ ) ) . Let Alg be any interactive estimation algorithm with the property that for any target h ∗ ∈ H , Alg outputs an ϵ -approximation to h ∗ with probability at least 2 / 3 using at most m queries. Then m> 3 √ d ρ ( ϵ ) / 12 .

5 Bandits

In this section we focus on the bandits setting described in Example 2. We study the relationship between the dissimilarity dimension and the eluder dimension [24], a common combinatorial dimension for bounding regret of bandit algorithms. We show that eluder dimension can be used to upper bound the dissimilarity dimension, and we also highlight the cases when dissimilarity dimension leads to a tighter analysis.

Throughout this section we follow the setup introduced in Example 2. We consider an action set A , a class F of reward functions f : A → [ -1 , 1] , and a target reward function f ∗ ∈ F . We map this to our setting by considering the set of alternatives Z = F × A , evaluation function ρ bandits ( ( f, a ) ∣ ∣ ( f ′ , a ′ ) ) = f ′ ( a ) and the target ( f ∗ , a ∗ ) , where a ∗ = argmax a ∈A f ∗ ( a ) .

5.1 Comparison with eluder dimension

We start by describing the relationship between our dimension and the eluder dimension. Following Russo and Van Roy [24], we define ϵ -dependence and ϵ -eluder dimension as follows:

Definition 5 ( ϵ -dependence) . An action a ∈ A is ϵ -dependent on actions { a 1 , . . . , a n } ⊆ A with respect to F if any pair of functions f, f ′ ∈ F satisfying √∑ n i =1 ( f ( a i ) -f ′ ( a i )) 2 ≤ ϵ also satisfies | f ( a ) -f ′ ( a ) | ≤ ϵ . Furthermore, an action a is ϵ -independent of { a 1 , . . . , a n } with respect to F if it is not ϵ -dependent on { a 1 , . . . , a n } .

Definition 6 ( ϵ -eluder dimension) . The ϵ -eluder dimension dim E ( F , ϵ ) is the length d of the longest sequence of elements in A such that every element is ϵ -independent of its predecessors. Moreover, the monotone eluder dimension is defined as dim E ( F , ϵ ) := max ϵ ′ ≥ ϵ dim E ( F , ϵ ′ ) .

The next theorem shows that the dissimilarity dimension is upper bounded by the eluder dimension (see Appendix C.1 for a proof):

Nevertheless, as the next example shows, the eluder dimension can be arbitrarily large, while the dissimilarity dimension remains constant. In this example, the action set is a circle in R 2 , that is, A = C := { v ∈ R 2 : ∥ v ∥ = 1 } . We fix two open semicircles U 0 , U 1 ⊆ C with positive x and y coordinates, respectively, and for any N ∈ N and ϵ > 0 , construct a function class F N,ϵ with all the functions f : A → [ -1 , 1] obtained by the following process. First, pick one of the semicircles U j and any N points from U j . On each of these points, f can equal either + ϵ or -ϵ . Everywhere else in U j , f equals zero, and everywhere outside U j , it equals the linear function ⟨ v , a ⟩ parameterized by some v ∈ C \ U j . Thus, the functions are constructed to be 'simple' (namely, linear) near the optimal action v , but complex far from it. The eluder dimension is large to capture overall complexity, whereas the dissimilarity dimension is small to capture the simplicity near the optimum. (See Appendix C.5 for the formal construction of F N,ϵ and the proof of Proposition 12.)

Proposition 12. Let ϵ ∈ (0 , 1 / 2) , N ∈ N and consider the action set A = C . Then, there is a function class F N,ϵ ⊆ [ -1 , 1] A , such that for Z N,ϵ := F N,ϵ × A , ρ = ρ bandits , it holds that d ρ ( Z N,ϵ , 1 , ϵ ) ≤ 16 , but the eluder dimension is lower bounded as dim E ( F N,ϵ , ϵ ) ≥ N .

Thus, our regret bound based on the dissimilarity dimension implies that (optimistic) least squares algorithms have a regret independent of N . The same analysis with the eluder dimension [24] yields

a regret bound scaling polynomially with N . This shows that in the cases when the function classes are simple near the optimum, but complex far from it, the dissimilarity dimension can better capture the statistical complexity of bandit optimization than the eluder dimension.

5.2 Dissimilarity dimension bounds

We next derive dissimilarity dimension bounds for several standard bandit classes. Existing bounds on eluder dimension can be used to immediately bound the dissimilarity dimension, but in several cases we are able to obtain tighter bounds.

We first consider linear bandits . Let B n = { v ∈ R n : ∥ v ∥ ≤ 1 } be the unit ball in R n . Actions are chosen from a set A ⊆ B n ; the reward function class is F lb = { f θ : θ ∈ Θ } , where Θ ⊆ B n and f θ ( a ) = ⟨ θ , a ⟩ . The corresponding set of alternatives is denoted Z lb = F lb ×A . In this case we obtain the following bound (see Appendix C.2 for a proof):

Theorem 13 (Linear bandits) . Let Z lb be as defined above, let ρ = ρ bandits , and let ϵ > 0 , α ≤ α ∗ . Then d ρ ( Z lb , α, ϵ ) ≤ 4 n +3 . Moreover, when α = 1 , then d ρ ( Z lb , α, ϵ ) ≤ 2 n +1 .

The proof proceeds by deriving an upper bound as well as a lower bound on the rank of the matrix M with entries M ij = ρ ( z i | z j ) -c obtained from elements z 1 , . . . , z d that satisfy the dimension condition for d = d ρ ( Z , α, ϵ ) with a scalar c . The upper bound on the rank is n +1 , and the lower bound is d/ 4 (which can be tightened to d/ 2 when M is symmetric). The upper bound is obtained by basic linear algebra and the lower bound from a standard result on ranks of perturbed identity matrices [2, Lemma 2.2]. Combining these bounds then yields the claim of Theorem 13. Similar to existing bounds on eluder dimension [24, Proposition 6], our bound in Theorem 13 is linear in n . However, the eluder dimension bound has an additional dependence on 1 /ϵ , while our bound does not.

Next, we consider generalized linear model (GLM) bandits. Similar to linear bandits, the action set is A ⊆ B n , but the function class includes a nonlinearity. Specifically, we are provided with a function g : R → R that is differentiable and strictly increasing, and consider the function class F glm = { f θ : θ ∈ Θ } where Θ ⊆ B n and f θ ( a ) = g ( ⟨ θ , a ⟩ ) . Furthermore, we assume that there are h, h > 0 such that for all a ∈ A , θ ∈ Θ , we have h ≤ g ′ ( ⟨ θ , a ⟩ ) ≤ h . Define r = h/h . We again denote Z glm = F glm ×A . Using an existing bound on the eluder dimension for GLM bandits ([24], Proposition 7) and the fact that our dimension is bounded by the eluder dimension (Theorem 11), we obtain the following bound (see Appendix C.3 for a proof):

Theorem 14 (GLM bandits) . Let Z glm be as defined above, let ρ = ρ bandits , and let ϵ > 0 , α ≤ α ∗ . Then d ρ ( Z glm , α, ϵ ) ≤ O ( nr 2 log( h/ϵ )) .

By considering a different proof technique, along the lines of Theorem 13, it might be possible to tighten this bound. We leave this extension for future work.

Next, we consider a bandit setting that is similar to GLMs, but in this case the non-linearity is provided by the non-differentiable rectified linear unit (ReLU) activation function relu ( x ) = max { x, 0 } . We consider the action set A = B n , and the set of reward functions F relu consisting of all functions of the form f θ ,b ( a ) = relu ( ⟨ θ , a ⟩ -b ) for some θ ∈ B n and b ∈ [0 , 1) . The subset of F relu with a fixed value of b is denoted F relu b , and we consider the set of alternatives Z relu b = F relu b ×B n .

Unlike the classes considered above, this setting can be shown to be challenging to learn in the general case. Indeed, it turns out that eluder dimension (as well as a related measure called star dimension) is growing at least exponentially with n [21, 10]. The same lower bound can be shown for the dissimilarity dimension by a similar proof technique. The following theorem also provides an exponential upper bound, showing that in certain regimes the exponential dependence is tight (see Appendix C.4 for a proof):

Theorem 15 (ReLU bandits) . Let Z relu be as defined above, let ρ = ρ bandits , and let ϵ, b > 0 such that b ≤ 1 -ϵ . Then d ρ ( Z relu b , 1 -b, ϵ ) = O ( ϵ -n/ 2 ) , and d ρ ( Z relu 1 -ϵ , ϵ, ϵ ) = Ω ( ϵ -n/ 2 ) .

We note that previous work ([10], Theorem 5.1) has shown that for a function class of one-layer neural networks with ReLU activations, obtaining sublinear regret requires T = Ω( ϵ -( n -2) ) .

6 Conclusion

In this paper, we have introduced a new model for interactive estimation and proposed a new combinatorial dimension, called dissimilarity dimension, to study the hardness of learning in this

model. In (stochastic, correlational) statistical query learning, our dimension is polynomially related to the strong SQ dimension. In bandits, our dimension is upper bounded by the eluder dimension, and there are examples where the dissimilarity dimension leads to much tighter regret bounds.

While this work provides an initial investigation of the dissimilarity dimension, many open questions remain. For example, our regret bound for the general setting scales as d 1 . 25 . Is it possible to tighten this to linear dependence, as is the case, for example, for eluder dimension? On the algorithmic side, we currently require solving a least squares problem of size t in iteration t . Although we also introduce an algorithm that leverages an online regression oracle (see Appendix A.3), the oracle-based approach still requires solving a least squares problem (on the data smoothed by the oracle). Is it possible to derive dissimilarity-dimension-based regret bounds directly for the predictions produced by the oracle? Ultimately, we hope investigations of relationships between dissimilarity dimension and related notions may help us understand the hardness of learning in interactive settings.

Acknowledgments and Disclosure of Funding

Aldo Pacchiano would like to thank the support of the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard. This work was supported in part by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard.

References

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Noga Alon. Perturbed identity matrices have high rank: Proof and applications. Combinatorics, Probability and Computing , 18(1-2):3-15, 2009.

Maria-Florina Balcan and Avrim Blum. On a theory of learning with similarity functions. In Proceedings of the 23rd international conference on Machine learning , pages 73-80, 2006.

Shai Ben-David and Eli Dichterman. Learning with restricted focus of attention. Journal of Computer and System Sciences , 56(3):277-298, 1998.

Shai Ben-David, Alon Itai, and Eyal Kushilevitz. Learning by distances. Information and Computation , 117(2):240-250, March 1995.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

Nader H Bshouty and Vitaly Feldman. On using extended statistical queries to avoid membership queries. Journal of Machine Learning Research , 2(Feb):359-395, 2002.

Yihua Chen, Eric K Garcia, Maya R Gupta, Ali Rahimi, and Luca Cazzanti. Similarity-based classification: Concepts and algorithms. Journal of Machine Learning Research , 10(3), 2009.

Thomas M Cover. Universal portfolios. Mathematical finance , 1(1):1-29, 1991.

Kefan Dong, Jiaqi Yang, and Tengyu Ma. Provable model-based nonlinear bandit and reinforcement learning: Shelve optimism, embrace virtual curvature. Advances in Neural Information Processing Systems , 34:26168-26182, 2021.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

Vitaly Feldman. A complete characterization of statistical query learning with applications to evolvability. Journal of Computer and System Sciences , 78(5):1444-1459, 2012.

Vitaly Feldman, Elena Grigorescu, Lev Reyzin, Santosh S Vempala, and Ying Xiao. Statistical algorithms and a lower bound for detecting planted cliques. Journal of the ACM (JACM) , 64(2): 1-37, 2017.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

David A Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Adam Tauman Kalai and Santosh Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research , pages 423-440, 2002.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Gene Li, Pritish Kamath, Dylan J Foster, and Nathan Srebro. Eluder dimension and generalized rank. arXiv preprint arXiv:2104.06970 , 2021.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Lev Reyzin. Statistical queries and statistical algorithms: Foundations and applications. arXiv preprint arXiv:2004.00557 , 2020.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Balázs Szörényi. Characterizing statistical query learning: simplified notions and proofs. In International Conference on Algorithmic Learning Theory , pages 186-200. Springer, 2009.

P Turán. On an extremal problem in graph theory (in Hungarian). Mat. Fiz. Lapok , 48:436-452, 1941.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Vladimir Vovk. A game of prediction with expert advice. Journal of Computer and System Sciences , 56(2):153-173, 1998.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020.

Zheng Wen and Benjamin Van Roy. Efficient exploration and value function generalization in deterministic systems. Advances in Neural Information Processing Systems , 26, 2013.

Ke Yang. New lower bounds for statistical query learning. Journal of Computer and System Sciences , 70(4):485-509, 2005.

A Missing proofs of Section 3

A.1 Analysis of Least Squares Algorithms (Algorithms 1 and 3)

Our analysis relies on the following variant of Freedman's inequality [16] (see Agarwal et al. [1, Lemma 9] and Beygelzimer et al. [6, Theorem 1]).

Lemma 16 (Simplified Freedman's inequality) . Let R > 0 and let X 1 , . . . , X n be a sequence of realvalued random variables, such that for all i ∈ [ n ] it holds that X i ≤ R and E [ X i | X 1 , . . . , X i -1 ] = 0 . For any δ ∈ (0 , 1) , and η ∈ (0 , 1 /R ) , with probability at least 1 -δ ,

Next we define an ϵ -cover of a set Z , that will be used in the bound of Theorem 17.

Definition 7 ( ϵ -cover) . Let ψ be the pseudometric over the set Z defined, for any z 1 , z 2 ∈ Z , as

We say a set N ⊆ Z is an ϵ -cover of Z with respect to ψ if for every z ∈ Z there exists some z ′ ∈ N such that ψ ( z, z ′ ) ≤ ϵ . We denote by N ( Z , ϵ ) the minimum cardinality of any ϵ -cover of Z .

For example, in the case of linear bandits (see Section 5.2) when Z = Z lb and ρ = ρ bandits, it can be shown that N ( Z lb , ϵ ) is upper bounded by the ℓ 2 -covering number of the n -dimensional unit ball. This is because for any z, z 1 , z 2 ∈ Θ ×A ,

The bound on N ( Z lb , ϵ ) now follows because the ℓ 2 -covering number of the unit ball with radius ϵ is O ( (3 /ϵ ) n ) (see, for example, Lemma D.1 of Du et al. [11]).

We next show that Algorithm 1 satisfies the decaying estimation error property with C T,δ that scales logarithmically with the covering number with respect to ψ .

Theorem 17 (LS guarantee) . Consider the setting from Section 2, where the learner sequentially issues the queries z 1 , . . . , z T and receives responses r 1 , . . . , r T . Assume there is β ≥ 0 such that ∣ ∣ r t -E [ r t | z t ] ∣ ∣ ≤ β for all t , and β ′ ≥ 2 β such that for all z, z ′ , ∣ ∣ ρ ( z | z ′ ) -ρ ( z | z ∗ ) ∣ ∣ ≤ β ′ . Let ˜ Z be a set of alternatives such that z ∗ ∈ ˜ Z and let ˆ z t be defined as the least squares optimizer,

Then, for any sequence of queries z 1 , . . . , z T ∈ ˜ Z (possibly equal to ˆ z 1 , . . . , ˆ z T ), we have with probability 1 -δ , for all t ∈ [ T ] simultaneously,

where C T,δ = 16 ββ ′ ln ( 2 T N ( ˜ Z , β ′ /T ) / δ ) .

Proof. For i = 1 , . . . , T , let h i = ( z 1 , r 1 , . . . , z i -1 , r i -1 , z i ) denote the history of interaction up to the query z i , but excluding the response r i , and let ξ i = r i -ρ ( z i | z ∗ ) . In the interactive estimation setting, we then have E [ ξ i | h i ] = 0 , and by the lemma assumption, E [ ξ 2 i | h i ] ≤ β 2 .

Since ˆ z t is the minimizer of the least squares loss up to time t , we have

which can be rewritten, substituting r i = ρ ( z i | z ∗ ) + ξ i , as

Therefore, by re-arranging terms, we get

Set ϵ 1 = β ′ /T , and let N be a minimal ϵ 1 -cover of ˜ Z with respect to the pseudometric ψ (see Eq. 6). Furthermore, let ˆ z ϵ t ∈ N be an element of this cover that is ϵ 1 -close to ˆ z t (with respect to ψ ). Then,

where the first inequality follows from Eq. (7), and the last inequality follows because | ξ i | ≤ β and ˆ z ϵ t is ϵ 1 -close to ˆ z t .

Now, for any z ∈ N and i ∈ [ T ] , define

Since E [ ξ i | h i ] = 0 , we have, for any fixed z ∈ N , E [ K z i | h i ] = 0 . This means that for any fixed z ∈ N , K z 1 , . . . , K z T is a martingale difference sequence. By the lemma assumptions, | K z i | ≤ ββ ′ . Also,

Thus, by Freedman's inequality (Lemma 16) with η = 1 / (4 ββ ′ ) and δ ′ = δ T | N | , we obtain that for any fixed z ∈ N and t ∈ [ T ] , with probability at least 1 -δ ′ ,

Taking a union bound over all z ∈ N and t ∈ [ T ] , we obtain that Eq. (10) holds with probability at least 1 -δ simultaneously for all z ∈ N and t ∈ [ T ] . Henceforth, we assume that we are in the event when Eq. (10) holds for all z ∈ N and t ∈ [ T ] .

Applying the bound of Eq. (10) with z = ˆ z ϵ t to the sum on the right-hand side of Eq. (8) then yields

Using the inequality ( a + b ) 2 ≤ 2 a 2 +2 b 2 , which holds for any a, b ∈ R , and the fact that ˆ z ϵ t and ˆ z t are ϵ 1 -close, we obtain, for every i = 1 , . . . , t -1 ,

where the last inequality follows by the assumption that β ′ ≥ 2 β . Then, by re-arranging terms and multiplying by 2 , we get

Recall that we set ϵ 1 = β ′ /T , and N is a minimal ϵ 1 -cover of ˜ Z , so | N | = N ( ˜ Z , β ′ /T ) . Plugging these values in the previous equation, we thus obtain that with probability at least 1 -δ , for all t ∈ [ T ] ,

where the last inequality follows because 4 + 2 /T ≤ 16 ln 2 for T ≥ 1 . Finally, when Eq. (12) holds, we also have

Considering Algorithm 1 and using Theorem 17 with ˜ Z = Z α , z t = ˆ z t , β = 2 and β ′ = 4 then immediately yields the following corollary ( α -large self-evaluations follow because ˆ z t ∈ Z α ):

Corollary 18. Consider the setting from Section 2 with a set of alternatives Z and an evaluation function ρ . Let α be an optimality level such that N ( Z α , 4 /T ) = e o ( T ) . Then Algorithm 1 has α -large self-evaluations and satisfies the decaying error property with C T,δ = 128ln ( 2 T N ( Z α , 4 /T ) / δ ) .

Similarly, Theorem 17 also implies that Algorithm 3 satisfies the decaying error property as well as α ∗ -large self-evaluations, although α ∗ is not known:

Corollary 19. Consider the setting from Section 2 with a set of alternatives Z and an evaluation function ρ , and assume that N ( Z , 4 /T ) = e o ( T ) . Then Algorithm 3 with R = 128ln ( 2 T N ( Z , 4 /T ) / δ ) has α ∗ -large self-evaluations and satisfies the decaying error property with C T,δ = 4 R = 512 ln ( 2 T N ( Z , 4 /T ) / δ ) .

Proof. We apply Theorem 17 with ˜ Z = Z , β = 2 and β ′ = 4 . Our choice of R in Algorithm 3 coincides with the value of C T,δ appearing in Theorem 17, and therefore the theorem implies that z ∗ ∈ Z t with probability at least 1 -δ for all t ∈ [ T ] . In that case the queries z t issued by the algorithm satisfy ρ ( z t | z t ) ≥ ρ ( z ∗ | z ∗ ) = α ∗ and thus the algorithm has α ∗ -large self-evaluations.

For the second part, the triangle inequality implies that with probability at least 1 -δ for all t ∈ [ T ] ,

where the bound on the first term on the right-hand side follows by Theorem 17 and the bound on the second term by the fact that z t ∈ Z t .

A.2 Proof of Theorem 4

The theorem follows immediately from Corollary 18, because N ( Z α , ϵ ) ≤ |Z α | ≤ |Z| < ∞ for any α and ϵ .

A.3 Online Regression Oracles

We assume access to an online regression oracle R eg , which solves a regression problem over a function class Φ = { ϕ z : z ∈ Z} indexed by z ∈ Z , where ϕ z : Z → R is defined as ϕ z ( z ′ ) = ρ ( z ′ | z ) for all z ′ ∈ Z ; that is, functions ϕ z evaluate ρ in its first argument.

The oracle operates in the following protocol: In each time step, the oracle receives an observation z t , produces a prediction ̂ ρ t ∈ R , and finally receives a response r t and incurs square loss ( ̂ ρ t -r t ) 2 . We assume that for any T and sequence of observations and responses (even if generated adaptively), the oracle satisfies the following regret bound:

where Regret R eg ( · ) is a non-decreasing sublinear function (that typically also depends on various properties of ρ , Z , and the range of responses r t ). For many function classes, there are well-known constructions of online regression oracles that satisfy Eq. (13) [9, 28, 18]. For example, if Φ is finite, there are oracles with Regret R eg ( T ) = O (ln | Φ | ) and for parametric classes, such as linear functions, there are oracles with Regret R eg ( T ) = O ( d log( T/d )) . More examples can be found in Section 2.2 of Foster and Rakhlin [14].

Algorithm 4 Interactive Estimation via Least Squares

1: Input: online regression oracle R eg , optimality level α .

2: Initialize z 1 to an arbitrary element of Z α .

3: for t = 1 . . . T do

4: Use R eg to predict ̂ ρ t given the observation z t .

5: Observe reward r t +1 and pass it to R eg .

7: end for

We now analyze Algorithm 4 under the assumption of access to an online regression oracle. This algorithm takes as input an online regression oracle R eg . Algorithm 4 can also be modified using the optimistic least squares template of Russo and Van Roy [24] to handle the case when α ∗ is unknown. This is done by replacing step 6 of Algorithm 4 with the following two steps:

where R = 8Regret R eg ( T ) + 64 β max { β, β ′ } ln ( T δ ) and β, β ′ are defined as in Lemma 20. The results of Lemma 20 justify the validity of these choices (using a similar reasoning as in Corollary 19) and imply that Algorithm 4 satisfies the decaying estimation error property (Definition 3), provided that the regression oracle R eg satisfies the regret bound of Eq. (13).

Lemma 20. Consider the setting defined in Section 2 with α ≤ α ∗ , and assume there are β, β ′ ≥ 0 such that | r t -E [ r t | z t ] | ≤ β for all t and there is β ′ ≥ 2 β s.t. for all z ∈ Z and ̂ ρ ∈ Γ z , ∣ ∣ ̂ ρ -ρ ( z | z ∗ ) ∣ ∣ ≤ β ′ where Γ z ⊂ R is the space of plausible responses of R eg for input z . The sequence of queries z 1 , . . . , z T as defined in Algorithm 4 satisfies with probability at least 1 -δ for all t ∈ [ T ] simultaneously,

Proof. Let ξ i = r i -ρ ( z i | z ∗ ) . Recall that E [ r i | z i ] = ρ ( z i | z ∗ ) and therefore, by assumption, | ξ i | ≤ β for all i . By definition, the online regression oracle satisfies

Inequality ( i ) holds because z ∗ ∈ Z . Expanding the LHS,

Plugging this back into Eq. (14) and rearranging, we obtain

For any i ∈ [ T ] we define:

Observe that

Thus K 1 , . . . , K T is a martingale difference sequence. Notice that | K i | ≤ ββ ′ and that

Then, by plugging this into Freedman's inequality (Lemma 16) with η := 1 4 min(1 /β 2 , 1 /ββ ′ ) and δ ′ := δ/T , we get that for any fixed t ∈ [ T ] , with probability at least 1 -δ ,

Plugging Eq. (16) back into Eq. (15) and rearranging terms yields

with probability at least 1 -δ for all t ∈ [ T ] . Thus we conclude that with probability at least 1 -δ , for all t ∈ [ T ] ,

By the triangle inequality,

Since by definition z t +1 = argmin z ∈Z α ∑ t i =1 ( ρ ( z i | z ) -̂ ρ i ) 2 , we have

Substituting back into the triangle inequality above,

implying

Plugging Eq. (17) on the right-hand side yields

The result follows by using the monotonicity of Regret R eg ( t ) .

A.4 Proof of Lemma 5

The proof uses Turán's Theorem [26], a standard result from extremal graph theory that bounds the number of edges of a graph that does not contain a clique of a given size:

Theorem 21 (Turán's Theorem) . Let G = ( V, E ) be an undirected graph without self-loops and whose largest clique is of size at most d . Then

We now turn to the proof of Lemma 5. First note that if ϵ ≥ 1 + α then no query is ϵ -bad, because α -ϵ ≤ -1 ≤ ρ ( z | z ∗ ) for every z ∈ Z , and therefore the lemma holds. In the remainder of the proof, we assume that 0 < ϵ < 1 + α .

Consider the queries z 1 , . . . , z T and their corresponding values relative to z ∗ , denoted as v t = ρ ( z t | z ∗ ) for t ∈ [ T ] . Aquery z t is ϵ -bad if its corresponding value v t is in the interval I = [ -1 , α -ϵ ) . The proof proceeds by partitioning the interval I into subintervals and separately bounding the number of values v t in each subinterval.

To define these subintervals, let q = 1 + 1 √ d , and consider the sequence of suboptimality gaps ϵ i = q i -1 ϵ for i = 1 , . . . , n +1 , where

The gaps ϵ i form an increasing sequence ϵ, qϵ, q 2 ϵ, . . . such that the last element satisfies

Using these gaps we define intervals I i = [ α -ϵ i +1 , α -ϵ i ) for i = 1 , . . . , n . Since ϵ n +1 ≥ 1 + α , the union I 1 ∪··· ∪ I n = [ α -ϵ n +1 , α -ϵ ) covers the interval I . We bound the number of values v t in each interval I i .

Let S i be the set of query indices with values in I i , that is S i = { t ∈ [ T ] : ρ ( z t | z ∗ ) ∈ I i } , let m i = | S i | , and assume that m i ≥ 2 (the case m i ≤ 1 will be dealt with later). Furthermore, let c i = α -( ϵ i +1 + ϵ i ) / 2 be the midpoint of the interval I i . Since the width of the interval I i is ϵ i +1 -ϵ i = ( q -1) ϵ i = ϵ i / √ d , we obtain

for all t ∈ S i .

Let d i = d ρ ( Z , α, ϵ i ) be the (non-monotonic) dissimilarity dimension with respect to the suboptimality gap ϵ i . Since α ≤ α ∗ and ϵ i ≥ ϵ , we have 1 ≤ d i ≤ d . We construct an upper bound on m i , exploiting the fact that d i is the dissimilarity dimension with respect to ϵ i .

In the rest of the proof we refer to a pair of queries with indices s, t ∈ S i such that s < t as dissimilar if

This is exactly the property appearing in the definition of the dissimilarity dimension with respect to ϵ i , and so there cannot be more than d i queries such that every pair is dissimilar (note that all the queries z t satisfy ρ ( z t | z t ) ≥ α thanks to α -large self-evaluations).

The derivation of the bound on m i proceeds in several steps. First, we identify pairs of dissimilar queries and construct a graph where each edge corresponds to a dissimilar pair. Second, we use Turán's Theorem (Theorem 21) to upper bound the number of such pairs, using the fact that the graph cannot contain a clique of size greater than d i . Finally, using the bound on the number of dissimilar pairs, we bound m i .

To start, let t 1 < t 2 < . . . < t m i be the query indices included in S i , and let k ∈ { 2 , . . . , m i } . Consider a uniform distribution over ℓ ∈ [ k -1] . Then by Markov's inequality and the decaying estimation error property, we obtain

Multiplying by k -1 , we therefore obtain

and summing across all k ∈ { 2 , . . . , m i } then yields

We next construct an undirected graph without self-loops, G i = ( V i , E i ) . The vertex set of the graph is V i = S i . The edge set is defined to be

By comparing with Eq. (19), we obtain

Note that any pair of vertices s < t connected by an edge corresponds to a dissimilar pair of queries:

where the first inequality is the triangular inequality, the second inequality follows by combining the definition of E i and Eq. (18), and the final one is from the fact that d i ≤ d . From the definition of the dissimilarity coefficient, the largest clique in G i is of size at most d i . Using Turán's Theorem, we thus must have

Combining with the lower bound on | E i | from Eq. (20), we obtain

Dividing by m i , multiplying by 2 d , and rearranging then yields

We have originally assumed that m i ≥ 2 , but the bound that we have just derived also holds when m i ≤ 1 (because d ≥ 1 ).

To complete the proof it suffices to sum up the upper bounds on m i across i = 1 , . . . , n :

To bound n , we use the fact that α ≤ 1 , the inequality ln(1 + x ) ≥ x 1+ x (which holds for x ≥ 0 ), and the fact that d ≥ 1 :

Also,

Plugging these back in Eq. (21) yields

completing the proof of the main claim of the lemma.

The second claim holds vacuously when T = 0 , so assume that T ≥ 1 . If C T,δ ≥ ln(2 T ) , and using the fact that (ln x ) ≤ x and 2 ≤ 3 ln 2 , we can write

which yields the first part of the second claim. The second part is immediate by plugging in C T,δ = 0 in the main claim.

A.5 Useful lemmas

In this subsection we prove two lemmas that will be needed for the proofs of the main results (Theorems 6 and 7) in Appendices A.6 and A.7. They both rely on a standard technique of bounding a sum by a definite integral:

Proposition 22. Let f : R → R be a non-increasing function and T ≥ 1 . Then

Proof. The proof is immediate by noting that f ( t ) ≤ ∫ t t -1 f ( t )d t .

In the lemmas below we write R + to denote [0 , + ∞ ) .

Lemma 23. Let q 1 , . . . , q T be a sequence in R + , and let κ : R + → R + be a non-increasing function such that for all ϵ > 0 ,

Proof. First, since we are only concerned with bounding the sum ∑ t q t , we assume without loss of generality that the sequence is in descending order, i.e., q 1 ≥ · · · ≥ q T . Then, for any τ ≥ 0 ,

Consider any k such that q k > τ . Since the sequence q 1 , . . . , q T is non-increasing, we have

where the last inequality follows by the monotonicity of κ . This in turn implies that q k ≤ √ κ ( τ ) k . Therefore,

Combining Eqs. (22), (23) and (24), we get

Then, for any τ ≥ 0 ,

By Proposition 22, which concludes the proof.

Lemma 24. Let a > 0 , let q 1 , . . . , q T be a sequence of reals in [0 , a ] , and let κ : R + → R + be a non-increasing function such that for all ϵ ∈ (0 , a ] ,

Proof. We follow a similar proof strategy as in Lemma 23 and start by bounding the sum ∑ t q t . We assume without loss of generality that the sequence is in descending order, i.e., q 1 ≥ · · · ≥ q T . Then, for any τ ≥ 0 ,

Then, for any τ ≥ 0 ,

Consider any k such that q k > τ . Then

where the last inequality follows by the monotonicity of κ and the fact that ln( a/q k ) ≥ 0 . This in turn implies that q k ≤ a exp ( -k κ ( τ ) ) . Therefore,

By Proposition 22,

Combining Eqs. (25), (26) and (27), we get

which concludes the proof.

A.6 Proof of Theorem 6

Throughout the proof we use the shorthand d ϵ = d ρ ( Z , α, ϵ ) , so d = d 1 /T . The proof proceeds by applying Lemmas 23 and 24 to the bounds on the number of bad queries from Lemma 5. Specifically, let q t = [ α -ρ ( z t | z ∗ )] + denote the suboptimality of each query z t made by the algorithm. Then, for any ϵ > 0 , the number of ϵ -bad queries can be written as ∑ T t =1 1 ( q t ≥ ϵ ) .

First consider the case C T,δ ≥ ln(2 T ) . By Lemma 5, with probability at least 1 -δ , the number of ϵ -bad queries is at most 36 d 2 . 5 ϵ C T,δ /ϵ 2 . Setting κ ( ϵ ) = 36 d 2 . 5 ϵ C T,δ , we apply Lemma 23, with τ = 1 /T , to obtain that with probability at least 1 -δ ,

If C T,δ = 0 , then by Lemma 5, the number of ϵ -bad queries is at most 2 d 1 . 5 ϵ ln(4 /ϵ ) . Setting a = 4 and κ ( ϵ ) = 2 d 1 . 5 ϵ , we apply Lemma 24, with τ = 1 /T , to obtain

completing the proof.

A.7 Proof of Theorem 7

First, we consider the deterministic setting. By Lemma 5, at most 2 d 1 . 5 ln(4 /ϵ ) of queries issued by Alg are ϵ -bad. Setting T > 2 d 1 . 5 ln(4 /ϵ ) implies that at least one query is not ϵ -bad. Thus, returning ˆ z for which the observed reward is the largest guarantees that ρ (ˆ z | z ∗ ) ≥ α -ϵ , as needed.

Next, we prove the result for the case C T,δ ≥ ln(2 T ) . By Lemma 5, with probability at least 1 -δ/ 2 , there are at most 16 9 ϵ 2 · 36 d 2 . 5 ( C T,δ/ 2 ) queries that are 3 ϵ/ 4 -bad. Setting T ≥ 64 d 2 . 5 ( C T,δ/ 2 ) /ϵ 2 , implies that at least half of the queries are not 3 ϵ/ 4 -bad. In the remainder of the proof, we only consider the high-probability event in which this is the case.

For n 1 = ⌈ log 2 (4 /δ ) ⌉ the probability that all n 1 samples are 3 ϵ/ 4 -bad is at most (1 / 2) n 1 ≤ δ/ 4 .

For n 2 = ⌈ 128 ln(8 n 1 /δ ) /ϵ 2 ⌉ , by applying Hoeffding's inequality and union bound over each of the n 1 rounds we get that with probability at most δ/ 4 there is some index ℓ ≤ n 1 for which | ¯ r t ℓ -ρ ( z t ℓ | z ∗ ) | > ϵ/ 8 .

Overall, with probability at least 1 -δ we get that there is at least one index j of the n 1 sampled indices that is not 3 ϵ/ 4 -bad, and that | ¯ r t ℓ -ρ ( z t ℓ | z ∗ ) | ≤ ϵ/ 8 for all ℓ = 1 , . . . , n 1 . Therefore,

For all indices k that are ϵ -bad we have

Thus, for all of the ϵ -bad queries we have ¯ r t k < ¯ r t j , and so Algorithm 2 will not return any of the ϵ -bad queries, because it is choosing the index with maximum value of ¯ r t ℓ . In other words, the returned query z t ˆ ℓ satisfies

B Missing proofs of Section 4

First we discuss the connection between our SQ setting and the SQ model of Kearns [19]. We focus on two aspects in which they appear to differ and explain why these models are equivalent.

Correlational vs general statistical queries. The restriction of the SQ model in which the oracle may only output the approximate correlation between a query and the target function, termed correlational statistical query (CSQ), was studied by Bshouty and Feldman [7]. The CSQ oracle can be viewed as providing something akin to a negative distance between the query and the target. This is equivalent to the learning by distances framework of Ben-David et al. [5], who defined their model independently of Kearns [19]. Bshouty and Feldman [7] showed that an arbitrary statistical query can be answered by asking two SQs that are independent of the target and two CSQs. That is, in the distribution-dependent learning model (i.e., when the learner has access to the distribution over X ), correlational queries can simulate general queries.

Adversarial vs statistical noise. The setting we consider in this work assumes stochastic query responses, similar to several previous works [13, 31, 4]. On the other hand, the original SQ model [19] assumed that the query oracle can respond with an adversarial (rather than statistical) noise, up to a pre-specified tolerance parameter τ > 0 . The previous works have shown that the two noise models are equivalent [13, 31, 4]

B.1 Proof of Proposition 8

We first prove the first inequality of Eq. (3). Let ϵ > 0 and let d = d SQ ( ϵ ) . Then there exists a sequence h 1 , . . . , h d ∈ H satisfying both conditions of Definition 4. Let d ′ be equal to the leftmost expression of Eq. (3). We aim to show d ρ ( ϵ ) ≥ d ′ . Note that d ′ ≤ d .

̸

Let c be the midpoint between c min = min i<j ⟨ h i , h j ⟩ and c max = max i<j ⟨ h i , h j ⟩ . Then c ≤ 1 -ϵ . Moreover, for all i = j ,

where the last inequality follows from our choice of d ′ (which ensures d ′ ≤ 4( dϵ ) 2 ). Thus, h 1 , . . . , h d ′ , the first d ′ elements of the original sequence of hypotheses, satisfy Definition 1, proving the claim.

̸

We prove the first inequality of Eq. (4) in a similar way. Let us re-define d = d ρ (4 ϵ ) and let d ′ be equal to the leftmost expression of Eq. (4). As before, d ′ ≤ d . Then there exists a sequence h 1 , . . . , h d ∈ H satisfying the conditions of Definition 1 for some c ≤ 1 -4 ϵ . Then for all i = j , ⟨ h i , h j ⟩ ≤ c + 4 ϵ √ d ≤ 1 -ϵ , since d ≥ 2 . Moreover, for all i = j and i ′ = j ′ ,

̸

̸

with the last inequality following from our choice of d ′ . Thus, h 1 , . . . , h d ′ , the first d ′ hypotheses in the original sequence, satisfy Definition 4.

The second inequality of Eq. (4), now follows from the first inequality of Eq. (3), since if the second inequality of Eq. (4) does not hold then the leftmost expression of Eq. (3) must be at least d ρ ( ϵ ) , a contradiction. Likewise, the second inequality of Eq. (3), now follows from the first inequality of Eq. (4).

B.2 Lower bound setting

Definition 8 (SQ oracle (adversarial)) . Let D be the input distribution over the domain X . For a tolerance parameter τ > 0 , O adv ( τ ) := O adv D,h ∗ ( τ ) oracle is the oracle that for any query function h ∈ H , returns a value v ∈ [ µ -τ, µ + τ ] , where µ = E x ∼ D [ h ( x ) h ∗ ( x )] .

Definition 9 (Sample oracle (statistical)) . Let D be the input distribution over the domain X . The Sample oracle O := O D,h ∗ oracle is the oracle that given any function h ∈ H , takes an independent random sample x from D and returns the value v = h ( x ) h ∗ ( x ) .

We will need the following results for our proof. The first is a reduction from an adversarial noise oracle to a statistical one. Specifically, consider the learning setting defined in Section 2, for a sample oracle O . Let Alg be a (possibly randomized) algorithm for that setting. The following theorem shows a simulation of O via O adv the SQ oracle.

Theorem 25 ([13], Theorem 3.13) . Assume that Alg outputs a ϵ -approximation to h ∗ with probability at least δ , using m samples from O . Then, for any δ ′ ∈ (0 , 1 / 4] , there exists a SQ algorithm Alg ′ that uses at most m queries to O adv ( δ ′ 2 /m ) and outputs an ϵ -approximation to h ∗ with probability at least δ -δ ′ .

Their result is obtained by simulating Alg using O adv as follows: for any query of Alg to O , the response of O adv to that query is used as bias for a coin flip, which is then given to the learner as the simulated outcome of O . They then prove that the true m samples of O and the simulated coin flips are statistically close by bounding their distributional distance. This implies that the success probability of Alg ′ , the simulated algorithm, is not much worse than that of Alg , the original algorithm.

We note that the result originally stated in [13] differs from Theorem 25 above in two ways. First, it reduces to a variant of O adv ( τ ) with a tolerance τ ′ ∈ [ τ, √ τ ] . Thus, it holds for O adv ( τ ) as well. Second, it is phrased in a more general setting of search problems over distributions, which captures the SQ model, as detailed in [13], Section 6.

The second result that is needed for our proof is the following lower bound due to [25].

Theorem 26 ([25], Theorem 8) . Let ϵ > 0 , and let H ⊆ {± 1 } X be a hypothesis space with strong SQ dimension d SQ := dim SQ ( H , 2 ϵ ) ≥ 3 (see Definition 4). Then for any SQ algorithm Alg using m queries to O adv ( τ ) with tolerance τ ≥ 2 / √ d SQ , there exist h ∗ ∈ H such that if Alg outputs an ϵ -approximation to h ∗ , then m>d SQ τ 2 / 3 .

B.2.1 Proof of Theorem 9

Set δ = 2 / 3 . Let D be a distribution over X . Assume towards contradiction that there exists a learning algorithm Alg such that for any h ∗ ∈ H , given oracle access to O := O D,h ∗ and using m ≤ 3 √ d SQ / 12 samples from O , the algorithm Alg outputs an ϵ -approximation to h ∗ with probability at least δ .

We then apply Theorem 25 for δ ′ = δ/ 2 to simulate the algorithm using O adv := O adv D,h ∗ . The resulting algorithm uses m ≤ 3 √ d SQ / 12 queries to O adv ( τ ) for τ = δ ′ 2 /m > 4 / (3 3 √ d SQ ) ≥ 2 / √ d SQ and has success probability of at least δ -δ ′ = δ/ 2 > 1 / 3 . By Theorem 26 we obtain a contradiction, as m>d SQ τ 2 / 3 > 3 √ d SQ / 2 .

C Missing proofs of Section 5

C.1 Proof of Theorem 11

Let d = d ρ ( Z , α, 3 ϵ/ 2) . Note that the eluder dimension is always at least 1 , so the theorem trivially holds if d ≤ 9 . In the remainder of the proof assume that d ≥ 10 .

From the definition of the monotonic dissimilarity dimension, there exists τ ≥ 3 ϵ/ 2 such that d = d ( Z , α, τ ) . Let ( f 1 , a 1 ) , . . . , ( f d , a d ) be a sequence satisfying the dimension conditions for τ .

We will show that the first ⌈ d/ 9 ⌉ elements of this sequence also satisfy the conditions of the eluder dimension for some ϵ ′ ≥ ϵ . Specifically, we will show that there is some ϵ ′ ≥ ϵ such that every element a j with j ≤ ⌈ d/ 9 ⌉ in the sequence above is ϵ ′ -independent of its predecessors. That is, we will show that for every such element a j , there exists a pair of functions f, f ′ ∈ F that satisfy

yet it also holds that f ( a j ) -f ′ ( a j ) > ϵ ′ .

By definition of the dissimilarity dimension, there exists c ≤ α -τ such that for all i < j ,

Then, by the triangle inequality,

Therefore,

and so for all j ≤ ⌈ d/ 9 ⌉ it holds that,

Next, recall that for all j ≤ d we have f j ( a j ) ≥ α ≥ c + τ and f j +1 ( a j ) ≤ c + τ √ d . Thus,

where the last inequality holds for d ≥ 10 . Overall, Eqs. (31) and (32) then demonstrate that for ϵ ′ = 2 τ/ 3 ≥ ϵ , the element a j is ϵ ′ -independent of its predecessors, finishing the proof.

C.2 Proof of Theorem 13

Our proof uses the following result on ranks of perturbed identity matrices (see [2, Lemma 2.2]):

̸

Lemma 27. Let A ∈ R d × d be a symmetric matrix such that A ii = 1 for all i and | A ij | ≤ 1 / √ d for all i = j . Then rank( A ) > d/ 2 .

The proof begins by constructing a matrix M whose entries are derived from the evaluation values of elements that satisfy the dimension condition. Then we bound the rank of M from above as well as from below. The lower bound is expressed in terms of the dimension d = d ρ ( Z , α, ϵ ) while the upper bound is expressed in terms of n . Combining the bounds then yields the result of the theorem.

Construction of M . Let ( f θ 1 , a 1 ) , . . . , ( f θ d , a d ) denote the alternatives that satisfy the dimension conditions, with respect to some value c such that c ≤ α -ϵ (see Definition 1). Define M to be the d × d matrix with entries M ij = ⟨ θ i , a j ⟩ -c for i, j ≤ d . Note that all diagonal entries of M are at least α -c ≥ ϵ , and all other entries are in [ -ϵ √ d , ϵ √ d ] .

Upper bound on rank( M ) . Let K ∈ R d × d be the matrix of inner products, K ij = ⟨ θ i , a j ⟩ , and let U be the Gram matrix for the set of vectors θ 1 , . . . , θ d , a 1 , . . . , a d . Then U is a 2 d × 2 d matrix of the rank at most n , because the vectors are of the dimension n (see, e.g., [17, Theorem 7.2.10]), and K is a submatrix of U , so rank( K ) ≤ rank( U ) ≤ n . Moreover, M = K -c 11 ⊤ , where 1 is the all-ones vector in R d . Therefore, by subadditivity of rank,

Lower bound on rank( M ) . Let D ∈ R d × d be the diagonal matrix with entries D ii = 1 / √ M ii . Since M ii ≥ ϵ > 0 , we have 0 < D ii ≤ 1 / √ ϵ . Consider the matrix M ′ = DMD . Then

because the matrix D is non-singular (see [17, Section 0.4.6(b)]). Furthermore, matrix M ′ satisfies M ′ ii = 1 for all i and

̸

for all i = j . Consider the symmetric matrix S = ( M ′ +( M ′ ) ⊤ ) / 2 . Then, we also have S ii = 1 for all i , and | S ij | ≤ 1 / √ d for all i = j . Thus, by Lemma 27, rank( S ) > d/ 2 . Moreover, by the subadditivity of the rank

Combining Eqs. (35), (34) and (33), we therefore obtain

and so d < 4 n +4 . Since d is an integer, we must have d ≤ 4 n +3 .

In the special case that α = 1 , we have ⟨ θ i , a i ⟩ ≥ 1 for all i ≤ d , which is only possible when θ i = a i for all i ≤ d . As a result, the matrices M and M ′ are both symmetric, and thus S = M ′ and

implying that d ≤ 2 n +1 .

C.3 Proof of Theorem 14

Using an existing bound on the eluder dimension for GLM bandits ([24], Proposition 7), and the fact that our dimension is bounded by the eluder dimension (Theorem 11) the result follows.

C.4 Proof of Theorem 15

Denote d = d ρ ( Z relu b , 1 -b, ϵ ) . Notice that since b < 1 , for any θ , a ∈ B n such that θ = a it holds that f θ ,b ( a ) < f θ ,b ( θ ) = 1 -b . Let ( f θ 1 ,b , θ 1 ) , . . . , ( f θ d ,b , θ d ) be a sequence of elements satisfying the dimension definition, with respect to a corresponding scalar c ≤ 1 -b -ϵ . Since the evaluation is symmetric for ReLU functions, we can view this sequence as a set, and denote U = { θ 1 , . . . θ d } . In addition, note that by the dimension definition, for all θ ∈ U , ∥ θ ∥ = 1 .

̸

̸

We start by proving an upper bound on d . Assume d ≥ 9 . First, consider the case c ≤ ϵ/ 3 . Let U 0 be any subset of the unit sphere such that for all θ = θ ′ in U 0 it holds that ⟨ θ , θ ′ ⟩ ≤ b +2 ϵ/ 3 . Observe that for all such θ = θ ′ we have f θ ,b ( u ′ ) = f θ ′ ,b ( θ ) ∈ [0 , 2 ϵ/ 3] . Thus, we get that d ≤ | U 0 | .

̸

̸

A standard sphere covering argument shows that the size of such a set is upper bounded as follows. The δ -covering number of the unit sphere is at most (3 /δ ) n ([27], Cor. 4.2.13). Thus, there are at most (3 /δ ) n points such that each pair θ = θ ′ satisfies ∥ θ -θ ′ ∥ ≥ δ , or equivalently ⟨ θ , θ ′ ⟩ ≤ 1 -δ 2 / 2 . By setting δ = √ 2(1 -b -2 ϵ/ 3) we get that | U 0 | ≤ ( 3 δ ) n ≤ ( 3 2 √ 2( ϵ -2 ϵ/ 3) ) n ≤ (4 / √ ϵ ) n , which yields the desired bound.

̸

Now, consider the case c > ϵ/ 3 . In this case, for all i = j , we have that f θ j ,b ( θ i ) ≥ c -ϵ √ d ≥ c -ϵ/ 3 > 0 , and so ⟨ θ i , θ j ⟩ > b . Let c ′ = c + b . Thus, it must also hold that, |⟨ θ i , θ j ⟩ -c ′ | ≤ ϵ √ d for all i = j . Note that c ′ < 1 -ϵ . Then, by applying Lemma 13 we get that d is upper bounded by 2 n +4 . Overall, the bound in the claim holds.

̸

Next, we show a lower bound on d = d ρ ( Z relu 1 -ϵ , ϵ, ϵ ) , by lower bounding the size of the set U defined above. We now apply a sphere packing argument, which shows that there exists such a set U with size | U | ≥ (1 / 2 ϵ ) n/ 2 . We follow a similar argument as above. Specifically, the δ -packing number of the unit sphere is at most (1 /δ ) n ([27], Cor. 4.2.13). By plugging in δ = √ 2 ϵ , yields the desired bound.

̸

C.5 Proof of Proposition 12

We start with an auxiliary lemma:

Lemma 28 (dissimilarity subadditivity) . Let Z 1 and Z 2 be two sets, and let α ∈ R and ϵ > 0 . Denote Z = Z 1 ∪ Z 2 and let ρ : Z × Z → R be an evaluation function. Then,

Proof. Let z 1 , . . . , z d ⊆ Z such that there exists c ≤ α -ϵ with | ρ ( z i | z j ) -c | ≤ ϵ √ d for all i < j , and ρ ( z i | z i ) ≥ α . Let I 1 , I 2 ⊆ [ d ] be disjoint sets of indices with { z i } i ∈ I 1 ⊆ Z 1 and I 2 = [ d ] \ I 1 . Consider the sub-sequence z ℓ 1 , . . . , z ℓ | I 1 | ordered by appearance in z 1 , . . . , z d of elements in I 1 . By definition for all i < j ,

Therefore the sequence z ℓ 1 , . . . , z ℓ | I 1 | satisfies | ρ ( z ℓ i | z ℓ j ) -c | ≤ ϵ √ | I 1 | for all i < j and therefore | I 1 | ≤ d ρ ( Z 1 , α, ϵ ) . The same logic implies | I 2 | ≤ d ρ ( Z 2 , α, ϵ ) .

To get the monotonic version, note that if ϵ ∗ = arg max ϵ ′ ≥ ϵ d ρ ( Z , α, ϵ ′ ) ,

where the first inequality is by the first statement of the lemma which was proved above, the next inequality is by definition of the monotonic dimension, and the last inequality follows by ϵ ≤ ϵ ∗ .

We can now construct the classes that demonstrate the separation of the eluder and dissimilarity dimension.

We consider two overlapping semicircles, indexed by j ∈ { 0 , 1 } , and defined as

For each j ∈ { 0 , 1 } , and any N ∈ N and ϵ > 0 , we define the function class

containing functions and

In words, the functions in the class F j,N,ϵ are linear outside of the semicircle U j , and zero in the semicircle U j , except for a set of size N , where they can take any combination of values + ϵ and -ϵ . For any N ∈ N and ϵ > 0 , we define the class F N,ϵ := ⋃ j ∈{ 0 , 1 } F j,N,ϵ and show that this class has a constant dissimilarity dimension but its eluder dimension is at least N .

Finally, consider the action set A = C and the function class F N,ϵ as defined above. Let Z N,ϵ = F N,ϵ ×A , ρ = ρ bandits and ϵ ∈ (0 , 1 / 2) . We how show that dim E ( F N,ϵ , ϵ ) ≥ N , but d ρ ( Z N,ϵ , 1 , ϵ ) ≤ 16 . First we prove the following lower bound on the eluder dimension of F j,N,ϵ .

Lemma 29. The eluder dimension of F j,N,ϵ satisfies dim E ( F j,N,ϵ , ϵ ) ≥ N for all j ∈ { 0 , 1 } .

Proof. Let a 1 , . . . , a N be an arbitrary set of points in U j and consider the functions { f i } N +1 i =1 ⊆ F j,N,ϵ,S, v that for all i, i ′ ≤ N satisfy:

̸

and f N +1 ( a i ) = ϵ for all i ≤ N . We now show that for all i ≤ N , the action a i is ϵ -independent of a 1 , . . . , a i -1 with respect to F j,N,ϵ,S, v . This holds since √ ∑ i -1 j =1 ( f i ( a j ) -f N +1 ( a j )) 2 = 0 while | f i ( a i ) -f N +1 ( a i ) | = 2 ϵ > ϵ . This finalizes the proof.

Lemma 30. Denote Z N,ϵ = F N,ϵ ×A and let ϵ ∈ (0 , 1 / 2) . Then, d ρ ( Z N,ϵ , 1 , ϵ ) ≤ 16 .

Proof. Denote by Z j,N,ϵ = F j,N,ϵ ×A for all j ∈ { 0 , 1 } . We start by showing that for all j ∈ { 0 , 1 } , d ρ ( Z j,N,ϵ , 1 , ϵ ) ≤ 8 . Let z 1 , . . . , z d be a maximal sequence certifying the dissimilarity dimension d ρ ( Z N,ϵ , 1 , ϵ ) ≥ d , i.e., it holds that,

Since ρ ( z i | z i ) ≥ 1 , it must be the case that z i = ( f v i ,S i ,σ i , v i ) with v i ̸∈ U j (since otherwise the self evaluations would be strictly less than 1 ). This implies that ρ ( z i | z j ) = ⟨ v i , v j ⟩ for all i < j . Consequently, the score evaluations of all z 1 , . . . , z d are equivalent to the score evaluations of the linear problem defined by v 1 , . . . , v d . Thus Theorem 13 implies the maximum length of such a sequence can be of size at most 2 × 2+4 = 8 . Finally the sub-additivity of the dissimilarity dimension (see Lemma 28) implies,

Combining the results of Lemmas 29 and 30 finalizes the proof of Proposition 12.

D Multi-Armed Bandits

In this section we explore the dissimilarity dimension of the K -armed bandit problem. In this setting the learner interacts with a set of K arms and at every step of a sequential interaction pulls an arm a t ∈ [ K ] and receives a reward r t such that E [ r t ] = µ i t where µ a t is the mean reward of arm a t . For simplicity we will assume µ a ∈ [0 , 1] for all a ∈ [ K ] and that | r t | ≤ 1 .

The K -armed bandit problem is an instance of structured bandits where A = [ K ] and F = [0 , 1] K . The dissimilarity dimension of the K -armed bandit problem satisfies,

Proposition 31. Consider the action set A = [ K ] and the function class F = [0 , 1] K as defined above. Let α ∈ [0 , 1] and Z = F × A , ρ = ρ bandits and ϵ ∈ (0 , 1 / 2) . Then d ρ ( Z , α, ϵ ) ≤ K .

Proof. Let c ≤ α -ϵ and z 1 , . . . , z d ∈ Z with z i = ( f i , a i ) be a maximal sequence such that, ρ ( z i | z i ) ≥ α while

For i < j . Substituting the definition of ρ , this implies f i ( a i ) ≥ α for all i ∈ [ K ] while | f j ( a i ) -c | ≤ ϵ √ d for all i < j . By definition of c , if d ≥ 2

̸

Let I i = { a ℓ } i ℓ =1 be the set of actions up to index i in the tuple sequence z 1 , . . . , z i . Equation 37 implies that f j ( a ) ≤ α -ϵ 4 for all j > i . Since a j satisfies f j ( a j ) ≥ α > α -ϵ 4 this implies a j ̸∈ I i . We conclude that a i = a j for all i < j . Since there are at most K different arm values, this implies d ≤ K .

D.1 Structured Bandits

We will now explain in detail how what Algorithm 1 reduces to in the structured bandits setting from Example 2. We write z i = ( f i , a i ) for all i ∈ [ T ] . The large evaluation set Z α can be reduced to the following set of functions,

Algorithm 5 Interactive Estimation for Structured Bandits

1: Input: action set A , function class F , optimality level α , number of steps T . Compute largeevaluation function-action set F α = { f ∈ F s.t. max a ∈A f ( a ) ≥ α } .

2: for t = 1 , . . . , T do

3: Compute regression function

4: Submit the query a t = argmax a ∈A f t ( a ) .

5: Observe reward r t .

6: end for

Since the ρ bandits function ρ bandits ( z i | z ) is independent on a for z = ( f, a ) , the least squares equation argmin z ∈Z α ∑ t -1 i =1 ( ρ ( z i | z ) -r i ) 2 reduces to,

finally, to ensure the action query has a self evaluation of at least α , we output a t = argmax a ∈A f t ( a ) .

We will now explain in detail what the Optimistic Interactive Estimation Algorithm 3 reduces to in the structured bandits setting from Example 2. We write z i = ( f i , a i ) for all i ∈ [ T ] .

The least squares objective ( z t = argmin z ∈Z α ∑ t -1 i =1 ( ρ ( z i | z ) -r i ) 2 ) can be written as,

The action component of the z element in this objective can be ignored since the ρ bandits evaluation function does not depend on it. The confidence ball Z t = { z ∈ Z : ∑ t -1 i =1 ( ρ ( z i | z ) -ρ ( z i | ˆ z t ) ) 2 ≤ R } reduces to,

The query can be reduced to the action component of z t ,

Algorithm 6 summarizes this reduction and corresponds to the standard optimistic least squares for sturctured bandit problems from [24].

Algorithm 6 Optimistic Interactive Estimation for Structured Bandits

1: Input: action set A , function class F , confidence-set radius R , number of steps T .

2: for t = 1 , . . . , T do

3: Compute confidence set

4: Submit the query a t = argmax f,a ∈F t ×A f ( a ) .

5: Observe reward r t .

6: end for

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a general interactive learning setting, which is termed \"interactive estimation\". In this model, we are given a set of alternatives, which contains the target. At each step, the learner can choose an alternative and receive the similarity of this alternative with the target. Based on this input, the goal is to get as close as possible to the target. The authors then demonstrate that many well studied settings are special cases of this general framework, including structured bandits and SQ learning. Then, they introduce a novel combinatorial parameter which they call \"dissimilarity dimension\". They prove that a general algorithm which, under reasonable assumptions, has regret that is bounded by a polynomial of the dissimilarity dimension. Furthermore, they show that the dissimilarity dimension is polynomially related to the strong SQ dimension, which automatically gives lower bounds in the sample complexity in terms of the dissimilarity dimension. Lastly, they prove that the dissimilarity dimension is upper bounded by the Eluder dimension and give concrete examples where the gap between the two is quite wide, which yields improved regret analysis for some bandit settings.\n\nSTRENGTHS:\nOverall, the contributions of this paper are novel and highly relevant for the learning theory community.\nIn terms of results, both definitions of the interactive estimation setting and the dissimilarity dimension are valuable contributions of this work. It is interesting that one can define such a general setting that encompasses various seemingly unrelated problems and still provide a meaningful combinatorial measure of complexity that characterizes learnability under general conditions. The fact that studying the dissimilarity dimension results in tightening various existing results about specific bandit settings is also an important feature of this work, and it seems to open up unexplored directions of research in this well-established field. Another important aspect of this work is that it provides an algorithmic template, which can be instantiated using various algorithms that exist in the literature.\nOn the technical side, the main proofs contain novel and non-trivial ideas, such as the idea of using Turan's Theorem to bound the number of bad queries in terms of the dissimilarity dimension in Lemma 5. \nIn terms of presentation, the paper is very well written, with all key ideas explained clearly and succinctly. It is also nice that the authors try to give intuition for the important ideas that are involved in proving the Theorems, such as the intuitive reason for why the dissimilarity dimension is \"tighter\" than the Eluder dimension (before Proposition 12).\n\nWEAKNESSES:\nThe main weakness in terms of results is the decaying estimation assumption, which is crucial in order for Algorithm 1 to work. The authors provide several alternative assumptions which imply the decaying estimation, such as the existence of an online regression oracle with sublinear regret, or bounded covering numbers for the set of alternatives. While these are reasonable assumptions that have been used in many settings in prior work, it would be interesting to see whether they can be relaxed, or whether they are somehow related to the assumption of boundedness of the dissimilarity dimension. Right now there is no discussion in the paper about that relation.\n\nQUESTIONS:\n-One question, which is also briefly discussed by the authors, involves the computational complexity of the proposed algorithms. In particular, are there some interesting conditions for the similarity measure $\\rho$, under which the least squares optimization of Algorithm 1 can be implemented in polynomial time? \n-It is implicitly assumed that for any given $z_1,z_2 \\in \\mathcal{Z}$, the algorithm can compute $\\rho(z_1|z_2)$. This is certainly the case in the specific settings of SQ learning and structured bandits, which are studied in the paper, but one could imagine situations where that computation is not possible. Would it be possible to impose some conditions on $\\rho$ which still enable learning of the target, without having access to an evaluation oracle for $\\rho$?\n\nLIMITATIONS:\nYes, the limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors introduce a framework for interactive estimation, where the learner interacts with some oracle that provides information in the following way: the learner submits a query $z$ and the oracle returns a stochastic reward whose mean is given by a measure of dissimilarity between the query and the ground truth. The main benchmark they care about is the regret guarantees that the learner can achieve. The authors propose a dimension which they call the dissimilarity dimension whose finiteness provides a sufficient condition for learnability in this model. They also propose an algorithm based on the least-squares method that achieves sublinear regret when the dissimilarity dimension is finite. Then, they also describe an online-to-batch conversion that can be used to establish learning guarantees in the offline setting. Moreover, they show how their framework can be instantiated to capture the SQ learning setting and the structured bandit setting. Finally, they illustrate learning tasks for which their approach yields sharper bounds than the state of the art.\n\nSTRENGTHS:\n* The authors are studying fundamental learning problems and any new perspectives which lead to better results in this line of work should be welcome.\n* The dissimilarity dimension can be viewed as a more localized version of the eluder dimension, which as the authors show, can lead to better bounds than the latter.\n* It is always interesting to try to unify settings, but I have some doubts about this unification (see Weaknesses/Questions).\n* Even though the techniques build upon the prior work, they are not trivial.\n* This dimension could have applications in other settings as well.\n* The paper is well-written and easy to follow, I particularly enjoyed the comparison with the eluder dimension and the explanation about the improvement that the dissimilarity dimension yields.\n\nWEAKNESSES:\n* My main concern is that the unification that the authors propose under this framework feels a bit forced and not very natural. In particular, in the model the authors propose the learner queries a point $z_t$ in every round $t$ and gets a stochastic reward whose mean is $\\rho(z_t|z^\\star)$, where $z^\\star$ is the ground truth and $\\rho$ can be thought of as a dissimilarity measure. In order to get the results in the bandit setting, they let $z = (a, f)$, where $a$ is some arm and $f$ is some reward function that maps arms to rewards. This does not feel very natural because the traditional way to think of this setting is that the learner queries arms, and not arms and functions. Moreover, the way the distance function is defined is $\\rho_{bandits}((f,a)|(f^\\star,a^\\star)) = f^\\star(a)$, so we can see that both $a^\\star, f$ do not affect its value. It seems as if it was defined this way in order to make the definition of the dissimilarity measure work, but it does not exactly convey the message it was supposed to. Am I missing something?\n* Line 50: \"Largely characterizes learnability in our setting\". This is a bit vague and inaccurate. I don't understand what it means for a dimension to \"largely characterize\" learnability, since it either characterizes it or it doesn't. What you have shown is that finite dissimilarity dimension implies learnability, and that there are instances with infinite such dimension that are not learnable. However, there could be  problems whose dissimilarity dimension is infinite, by they are still learnable.\n\nQUESTIONS:\n* What is the intuition behind the scaling of the distance in definition 1 by $\\sqrt{d}$?\n* The example in Proposition 12 is certainly interesting, but are you aware of some more \"natural\" settings where the dissimilarity dimension can be finite whereas the eluder dimension is infinite?\n* Can you derive logarithmic regret bounds that depend on some notion of suboptimality gap? For instance, in the context of bandits that would be the gap between the reward of the best arm and the second best arm.\n* Can the results be extended to contextual bandits and RL? A long line of work has established strong connections between the eluder dimension and regret bounds in various RL settings.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a learning model where an unknown target exists, and in each round, the learner selects a choice and receives a stochastic feedback indicating the similarity between the choice and the target. The objectives examined involve minimizing regret or providing a PAC-style guarantee, ensuring that a choice with approximately maximum similarity to the goal is returned with high probability. This model encompasses both bandits and correlational statistical query algorithms.\n\nThe paper introduces the concept of \"dissimilarity dimension\", a combinatorial measure that captures the learnability within the model. It proves that the dissimilarity dimension implies learnability and can be achieved by any algorithm satisfying two properties: (i) selecting choices with high similarity to themselves and (ii) improving the quality of reward estimates over time. The paper presents a simple algorithm that satisfies these properties, demonstrating that it achieves bounded regret and a PAC generalization guarantee.\n\nIn terms of the connection to the statistical query (SQ) model, the paper establishes that the dissimilarity dimension is polynomially related (both in upper and lower bounds) to the “strong SQ dimension”. Additionally, regarding the connection to bandits, the paper demonstrates that the dissimilarity dimension is upper bounded by the eluder dimension, which is commonly used in bandit problem literature. The paper provides concrete examples of bounding the dissimilarity dimension for bandit problems and illustrates cases where the dissimilarity dimension is significantly smaller than the eluder dimension.\n\nSTRENGTHS:\nThe paper introduces a novel learning protocol and presents a combinatorial dimension that captures learnability. Furthermore, it establishes connections between this dimension and two extensively studied learning models. The fact that the proposed dimension can be much smaller than the existing eluder dimension is particularly interesting and perhaps surprising result. The work is original and novel, and I would be happy to recommend acceptance.\n\nWEAKNESSES:\nI do not see any strong weaknesses.\n\nQUESTIONS:\nI was under the impression that the results in Szörényi 2009 hold for real-valued hypotheses as well. Is there a particular reason why Section 4 restricts to binary classifiers?\n\nLIMITATIONS:\nAddressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel unified framework for interactive estimation. The framework involves a learner making repeated queries to the environment and observing rewards that correspond to the similarity between the query and an unknown target. The algorithm then uses these observed signals to estimate the target. This general framework for interactive learning is connected to two instantiations of learning models: the SQ model and structured bandits, which are studied in this paper. Additionally, a new combinatorial dimension is proposed to measure the difficulty of learning, and it is used to offer standard analysis to achieve tighter regret bounds.\n\nSTRENGTHS:\nThe paper has several notable strengths. It proposes a new unified framework for interactive estimation and a novel dimension called dissimilarity dimension to measure hardness. It connects two classical frameworks and offers the analysis with them, then finds with their proposed dimension, tighter regret bound can be achieved. Furthermore, it shows another possible to rely on the proposed dissimilarity measure.\n\nWEAKNESSES:\nHere are some potential improvements to the original content:\n\n1. The writing sometimes is too abstract to understand the concept (Introduction) and hard to follow (Section 3 Algorithm). It can be beneficial if more explanation and practical settings are provided.\n\n2. The completeness of the framework can be further enhanced through additional empirical experiments.\n\nQUESTIONS:\n1. Why do we need a new dissimilarity measure and the intuition behind it? I am curious about the difference between it and typical approach as dissimilarity measurement.\n\n2. Since this framework claims to be unified to many settings, is there any experiment we can validate its universal feature?\n\nLIMITATIONS:\nThe paper introduces a unified framework for interactive learning that has the potential to be applied in various scenarios. It also establishes connections with two classical models to demonstrate its impact and applicability. The completeness of the framework can be further enhanced through additional empirical experiments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}