Title: Adaptive Experimentation When You Can't Experiment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Experimental Design

multi-armed bandit
Abstract: This paper introduces the confounded pure exploration transductive linear bandit (CPET-LB) problem. 
As a motivating example,  often online services cannot directly assign users to specific control or treatment experiences either for business or practical reasons. In these settings, naively comparing treatment and control groups that may result from self-selection can lead to biased estimates of underlying treatment effects. 
Instead, online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment. 
Our methodology provides online services with an adaptive experimental design approach for learning the best-performing treatment for such encouragement designs. 
We consider a more general underlying model captured by a linear structural equation and formulate pure exploration linear bandits in this setting. Though pure exploration has been extensively studied in standard adaptive experimental design settings, we believe this is the first work considering a setting where noise is confounded.  Elimination-style algorithms using experimental design methods in combination with a novel finite-time confidence interval on an instrumental variable style estimator are presented with sample complexity upper bounds nearly matching a minimax lower bound. Finally, experiments are conducted that demonstrate the efficacy of our approach.

Adaptive Experimentation When You Can't Experiment

Abstract

This paper introduces the confounded pure exploration transductive linear bandit ( CPET-LB ) problem. As a motivating example, often online services cannot directly assign users to specific control or treatment experiences either for business or practical reasons. In these settings, naively comparing treatment and control groups that may result from self-selection can lead to biased estimates of underlying treatment effects. Instead, online services can employ a properly randomized encouragement that incentivizes users toward a specific treatment. Our methodology provides online services with an adaptive experimental design approach for learning the best-performing treatment for such encouragement designs . We consider a more general underlying model captured by a linear structural equation and formulate pure exploration linear bandits in this setting. Though pure exploration has been extensively studied in standard adaptive experimental design settings, we believe this is the first work considering a setting where noise is confounded. Elimination-style algorithms using experimental design methods in combination with a novel finitetime confidence interval on an instrumental variable style estimator are presented with sample complexity upper bounds nearly matching a minimax lower bound. Finally, experiments are conducted that demonstrate the efficacy of our approach.

1 Introduction

In this study, we present a methodology for adaptive experimentation in scenarios characterized by potential confounding. Online services routinely conduct thousands of A/B tests annually [23]. In most online A/B/N experimentation, meticulous user-level randomization is essential to ensure unbiased estimates of treatment effects at the population level , commonly known as average treatment effects (ATE). In this setting, firms are often interested in understanding the treatment with the highest average outcome if presented to all members of the population. However, in many settings firms may not be able to randomize, for example if a feature must be rolled out to all users for various business reasons [30]. In such instances, users may choose to engage with a feature or not based on potentially unobservable preferences. Thus the resulting measured outcome may be correlated with the decision to engage in a specific feature. I.e. the underlying characteristics of the user confound the relationship between the decision to use the feature being evaluated and the effect of the feature. Thus, naively comparing the average outcome for users who engage with a feature with those who do not suffers from a (selection) bias. This setting is captured in Figure 1.

A potential solution is for services to employ encouragement designs where users are presented with incentives that encourage users to engage with a specific feature [4, 7, 12]. As a concrete example, many online services have introduced membership levels with different offerings and 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Tanner Fiez Amazon fieztann@amazon.com prices available to all users. Given a set of membership level options, the service is interested in knowing the counterfactual of which level has the optimal outcome (e.g., total revenue) if every user chooses to join that membership level. In this setting, encouragements could be coupons or trials for corresponding membership levels. In these settings, the firm can use intent-to-treat (ITT) estimates for the treatment effect which naively compare the average outcomes between the groups given different encouragements. In practice, given an encouragement a user may not engage with the corresponding feature choosing either the control or a different feature. Hence, the resulting ITT estimate may be a diluted estimate of the ATE [3]. However, all is not lost: if the encouragement presented to a user is properly randomized, and the service guarantees that the encouragement only affects the outcome through the choice of user treatment, then the encouragement acts as an instrumental variable . Standard analysis from the econometrics and compliance literature show that two-stage least squares ( 2SLS ) estimators can then be used to provide consistent estimates of treatment effects.

At the same time, firms are also increasingly utilizing adaptive experimentation techniques, often known as pure exploration multi-armed bandits ( MAB ) algorithms [26, 15], to accelerate traditional A/B/N testing. Pure exploration MAB techniques promise to deliver accurate inferences in a fraction of the time and cost as traditional methods. Similar to A/B testing, bandit methods assume users are properly randomized and can fail to learn the optimal treatment if naively used and may be sample inefficient if they fail to take the confounded structure into account.

Contributions. In this work, we provide a methodology for experimenters seeking to use adaptive experimentation in settings with confounding where encouragements are available. We formulate this work in the more general and novel setting of confounded pure exploration transductive linear bandits ( CPET-LB ) (Section 1.1). We present algorithms using experimental design for the CPET-LB problem and analyze the resulting sample complexity. As we demonstrate, even in the simple multi-armed bandit setting described above, computing an effective sampling pattern requires using the machinery of linear bandits. Without knowledge of the underlying structural model, existing linear bandit approaches could lead to suboptimal sampling. The main technical challenge we face is simultaneously improving our estimate of the structural model while designing with inaccurate estimates (Section 3). This approach crucially relies on our development of novel finite-time confidence bounds for two-stage least squares ( 2SLS ) style estimators that may be of independent interest (Section 2.2). Moreover, we provide worst-case sample complexity lower bounds that are nearly matched by our sample complexity upper bounds (Appendix D). Though the goal of this work is primarily theoretical, we empirically show the efficacy of our method over existing solutions (Appendix E).

1.1 General Problem Formulation

A confounded pure exploration transductive linear bandits ( CPET-LB ) instance consists of finite collections of measurement vectors Z ⊂ R d and evaluation vectors W ⊂ R d . At each time t ∈ N , the learner selects z t ∈ Z and observes a pair of noisy responses x t ∈ X ⊆ R d and y t ∈ R generated via the structural equation model

where Γ ∈ R d × d and θ ∈ R d are model parameters. 1 We define the history H t -1 = { ( z s , x s , y s ) } s<t and E t -1 [ · ] = E [ ·|H t -1 ] denoting the conditional expectation under the filtration generated by H t -1 . The noise { η t } ∞ t =1 and { ε t } ∞ t =1 satisfy the following set of assumptions unless otherwise noted.

Assumption 1. We assume ε t |H t -1 is 1-sub-Gaussian (and thus E [ ε t |H t -1 ] = 0 ). Furthermore, η t |H t -1 is σ 2 η -sub-Gaussian vectors (and thus E [ η t |H t -1 ] = 0 ), i.e.,

X

Y

Outcome

Z

Encouragement

U

Confounder

Goal. The objective is to identify w ∗ := arg max w ∈W w ⊤ θ with probability at least 1 -δ for δ ∈ (0 , 1) while taking a minimum number of measurements.

̸

In the setting where Γ = I, η = 0 and E t -1 [ ε t | x t ] = 0 , our setting reduces to the standard pure exploration transductive linear bandit problem [33, 15]. In general, the joint noise process [ η t , ε t ] may be dependent across the entries. In particular, we are allowing for the data generating process to be endogenous , meaning that E t -1 [ ε t | x t ] = 0 . That is, ε t can affect not just y t , but also x t given a choice of z t . The presence of endogeneity is a key challenge in the CPET-LB problem.

Assumption 2 (Exclusion Restriction) . We assume that E t -1 [ z t ε t ] = 0 , or alternatively that z t is uncorrelated with ε t .

The variable z t is commonly referred to as an instrumental variable [3]. We consider algorithms for the CPET-LB problem that stop at a H t -measurable time τ ∈ N , and produce a recommendation ̂ w ∈ W . The goal is δ -PAC algorithms with efficient sample complexity guarantees.

̸

Definition 1.1 ( δ -PAC) . We say an algorithm is δ -PAC for a CPET-LB problem with W , Z ⊂ R d if for all θ ∈ R d and Γ ∈ R d × d , it holds that P θ, Γ ( ̂ w = w ∗ ) ≤ δ for δ ∈ (0 , 1) .

1.2 Encouragement Designs

The CPET-LB feedback model generalizes the classical compliance setting.

̸

Compliance as a Special Case. In compliance problems, a decision-maker has access to a set of treatment that can be offered to users, while the users themselves have the option to accept the treatment they are presented or instead opt-in to a different treatment. The goal is to identify the treatment with the optimal average outcome if all users were to accept it. Specifically, given a finite set A = { 1 , 2 , . . . , d } , a decision-maker presents user t ∈ N with an encouragement for a treatment i ∈ A , the user then selects treatment j ∈ A where potentially j = i , and an outcome y t results. To capture compliance with the CPET-LB framework, we set Z = X = W = { e 1 , · · · , e d } and the parameter Γ captures the probability of accepting a treatment given an encouragement for a potentially different treatment. Specifically, Γ( i, j ) = P ( x t = e i | z t = e j ) , and a straightforward computation shows that x t = Γ ⊤ z t + η t where E [ η t | z t ] = 0 with

Moreover, y t = x ⊤ t θ + ε t gives the resulting reward, which is clearly correlated with the user choice so that cov ( η t , ε t ) = 0 . Finally, e ⊤ i θ = θ i gives the expected value of treatment i over the population and our goal is to identify w ∗ = arg max e i ∈W e ⊤ i θ . 2 Note that when Γ = I , we automatically have that η t = 0 and there is no confounding. This reduces to the standard MAB setting.

̸

Motivating Compliance Example. As a motivating compliance example representing the membership level discussion from the introduction, consider a location model that assumes each user t ∈ N arriving online has an underlying unobserved one-dimensional preference u t ∼ N (0 , σ 2 u ) . If an algorithm presents the user with encouragement z I t = e I t for I t ∈ A , then the user selects into the membership level given by J t = min j ∈A | I t + u t -j | so that x t = e J t . This process captures a user being more likely to opt-in to membership levels that are closer to the encouragement that they were presented. The outcome is then given by y t = x ⊤ t θ + u t .

We conduct an experiment with this problem instance (see Fig. 2 and Appendix B for more details). Specifically, d = 6 , θ = [ 1 -0 . 95 0 0 . 45 0 . 95 0 . 99 ] , and σ 2 u = 0 . 35 . Observe that w ∗ = e 1 = arg max w ∈W w ⊤ θ . An upper confidence bound (UCB) selection strategy is simulated that maintains estimates of the mean reward of each encouragement i ∈ A , namely ̂ µ i,t = ∑ t s =1 1 { z t = e i } y t , and then pulls the one with the highest UCB. The UCB selection strategy is combined with a pair of recommendation strategies. The UCB-OLS algorithm estimates the mean reward of each treatment using an OLS estimator, namely ̂ θ i,t LS = ∑ t s =1 1 { x t = e i } y t / ∑ t s =1 1 { x t = e i } , and recommends arg max a ∈A ̂ θ i,t LS . Moreover, the UCB-IV algorithm uses an instrumental variableestimator (see the next section) that incorporates knowledge of Γ similar to 2SLS to deconfound

(a)

w

⊤

θ

and

w

⊤

Γ

θ

for

w

∈ {

e

1

, . . . , e

d

}

(b)

P

(

w

t

=

w

∗

)

for the instance

eTre

UCB-OLS

UCB-IV

CPEG

1.0

1.00

Identification Probability

0.75

0.8

0.50

alue

0.25

0.6

0.00

0.4

0.25

0.50

0.2

0.75

1.00

0.0

1

3

4

5

0

20000

40000

60000

80000

Evaluation Index

TimeIndex

̂

estimates of the mean rewards of each treatment and recommends the treatment with the maximum estimate. The results over 100 simulations are shown in Fig. 2. UCB-OLS completely fails to identify θ 1 = arg max i ∈ d θ i due to a biased estimate, whereas UCB-IV does better. However, UCB-IV methods seem to have a constant probability of error. To see why, note that the expected reward from pulling z = e i is e ⊤ i Γ θ . These values are plotted in orange in Figure 2a. In particular, with some constant probability, UCB zeroes in on arm 6 becauses of the mean estimates on the z 's, and as a result fails to give enough samples to learn that arm 1 is indeed the best. In contrast, our proposed method CPEG, Algorithm 1 manages to find the best arm with significantly higher probability.

Notation. Let ∆( Z ) = { λ ∈ R |Z| : λ ≥ 0 , ∑ z ∈Z λ z = 1 } denote the set of probability distributions over the set Z . Given a distribution λ ∈ ∆( Z ) and matrix Γ ∈ R d × d , define the operator A ( λ, Γ) := ∑ z ∈Z λ z Γ ⊤ zz ⊤ Γ . Given Z ∈ R T × d and Γ ∈ R d × d , define the operator ¯ A ( Z, Γ) := ∑ T t =1 Γ ⊤ z t z ⊤ t Γ = Γ ⊤ Z ⊤ Z Γ where z t ∈ R d denotes row t of Z . Given a vector x ∈ R d and a symmetric positive-definite matrix A ∈ R d × d we let ∥ x ∥ 2 A = x ⊤ Ax . We adopt the standard notation that ( a ∨ b ) ≡ max { a, b } and ( a ∧ b ) ≡ min { a, b } for a, b ∈ R . σ min ( A ) , σ max ( A ) denote the minimum and maximum singular value of a matrix A . We denote by polylog ( x 1 , . . . , x n ) any polylogarithmic factors of x 1 , . . . , x n .

1.3 Related Works

Our work is at the intersection of several parallel tracks of literature, pure exploration linear bandits, causal bandits, and econometrics. The most relevant work on pure exploration in linear bandits is the RAGE algorithm of [15, 33]. RAGE is nearly instance optimal for linear bandits in the non-confounded setting. Extensions of RAGE to various noise models including logistic and heteroskedastic noise have been considered [35, 20]. Other algorithms for pure exploration linear bandits have been proposed and we leave it for future work to extend the ideas of this paper to those settings [27, 10].

Confounding in bandits was first considered in the regret minimization setting by [5]. They introduces the Multi-armed bandit with unobserved confounders (MABUC) problem. They empirically demonstrate traditional bandit algorithms can have linear regret in this setting and provide an algorithm that effectively employs observed intuition. The early work of [21] also assumes there is an additional unobserved latent class at each time that determines confounding in a compliance setting. They provide novel notions of regret, relative to the instrument with the highest reward ( arg max Z ⊤ Γ ⊤ θ in our notation), the highest treatment ( arg max w w ⊤ θ ), regret relative to the best latent class at each time, and regret on the set of 'compliers'. They discuss the suitability of these various notions of regret, and discuss when sublinear regret is possible. We remark that their approach is similar to ours in the sense that they assume a form of homogeneous effects across the population, and use an estimate of Γ . Recently [24] also consider the problem of compliance, however they don't take explicit non-confounding into account and assume an explicit parametric model that determines the non-compliance. This is analogous to the Heckman selection model considered in econometrics [18].

The recent works of [11, 36, 17] considered an online setting where at each time they observe a set { ( x t , z t ) } where x t is the action of interest and z t is an associated instrument. If action I t is selected, the reward observed is y t = x ⊤ I t θ t + ε t , where x t may be endogeneous. Similar to the standard linear bandit setting [1, 26], the goal is to minimize regret relative to the best action at each time. We remark that this setting is very different from ours. Effectively, we are choosing which instrument to select at each time to learn the best-performing treatment - in particular we can't choose a particular intervention. In their setting, they are choosing an intervention at each time and using the instrument purely for de-confounding the result. Experimental design for instruments to have more effective estimation has been considered by [8].

In the causal bandit problem, an underlying causal graph between a set of interventions and a reward value is assumed. Actions correspond to intervening (i.e. a 'do' operation [31]) at one or more specific nodes in the causal graph and then observing the corresponding value at the reward node. Causal bandits have been studied extensively in the regret setting [25, 28, 6] and the pure exploration setting [32]. Though past works have allowed for unobserved confounders in the graph e.g. [29], their goal is to learn the best performing intervention, which in our setting would be arg max z ∈Z z ⊤ Γ θ instead of w ∗ .

Encouragement designs have been considered in many applications in online and offline settings. One of the earliest works on encouragement designs is [7], which considers the problem of using encouragements to determine the impact of coupons at a grocery store. More recent applications include [4, 30, 13] all in the context of online services and treatments that are required to be served to all users. Most of these works consider a small number of treatments and a heterogeneous treatment effect - hence are interested in LATE estimator. As far as we are aware, we are the only work that considers adaptive encouragement design in the context of the model given in Equation 1 and for multiple treatments.

2 Estimators and Inference

We now present estimators for the unknown parameter θ and prove the associated statistical properties. The estimators discussed in this section are critical to our algorithmic solution outlined in Section 3.

2.1 Estimators

Before describing our solution concept, we quickly review potential options for estimating θ based on a dataset Z T = [ z 1 , · · · , z T ] ⊤ ∈ R T × d , X T = [ x 1 , · · · , x T ] ⊤ ∈ R T × d , Y T = [ y 1 , · · · , y T ] ⊤ ∈ R T , assumed to be generated according to the model in Eq. (1). Recall that the ordinary-least-squares ( OLS ) estimator for θ is given by

̸

Observe that ̂ θ LS is potentially a biased and inconsistent estimator for θ in the presence of endogenous noise since E [ ε t | x t ] = 0 . To remediate this problem, we define a general class of estimators that includes several standard estimators. Given an invertible matrix Ψ ∈ R d × d , let ¯ X T := Z T Ψ , and consider corresponding estimators termed Ψ -IV estimators of the form

When Ψ = I we recover the OLS estimator. In the rest of the paper, we will focus on two different potential options for Ψ .

Case 1: Oracle . Ψ = Γ . To begin, observe that the structural equation model from Eq. (1) can be combined by substituting the second equation into the first to obtain the reduced form

Since z t is independent of the i.i.d. process η ⊤ t θ + ε t , the least squares estimator which regresses y t onto z ⊤ t Γ is unbiased for estimation of θ and given by

This estimator will be used to design our general solution concept presented in Section 3. Of course in practice we cannot expect to know Γ , but we may be able to estimate it.

Case 2: P-2SLS . Ψ = ̂ Γ . We consider a setting where ̂ Γ is an (unbiased) estimator of Γ , learned using least-squares from an independent dataset Z T 1 = [ z ′ 1 , · · · , z ′ T 1 ] , X T 1 = [ z ′ 1 , · · · , z ′ T 1 ] collected non-adaptively. 3 That is, ̂ Γ = ( Z ⊤ T 1 Z T 1 ) -1 Z ⊤ T 1 X T 1 and:

We refer to the resulting estimator as a pseudo two stage least squares ( P-2SLS ) estimator. The main advantage of the P-2SLS estimator over standard 2SLS (given in Appendix C) is easier inference since now { ε t } t ≤ T of our dataset is independent of the measurements of the first dataset Z T 1 , X T 1 . In the econometrics literature, such an estimator is referred to as a two-sample 2SLS estimator [19].

2.2 Confidence Intervals

In the section that follows, we develop a general algorithmic approach that relies on experimental design aimed at reducing the uncertainty in our estimates of the optimal treatment. To this end, we first develop finite-time confidence intervals for estimators presented in the previous section given data generated according to the model in Eq. (1) and collected from non-adaptive designs.

We begin by characterizing the properties of the noise structure in the combined model of Eq. (5) with the following set of results.

Lemma 2.1. Under Assumption 1, the noise process ν := η ⊤ θ + ε is σ 2 ν -sub-Gaussian where σ 2 ν = 2( σ 2 η ∥ θ ∥ 2 2 +1) , specifically when the instance is compliance, σ 2 ν = 2(4 ∥ θ ∥ 2 2 +1) .

Oracle Confidence Interval . As in the last section, we assume that we have access to a dataset ( Z T , X T , Y T ) generated according to Eq. 1 and collected non-adaptively. Given Lemma 2.1, it can be shown that w ⊤ ̂ θ oracle is a sub-Gaussian random variable satisfying the following.

Lemma 2.2. With probability at least 1 -δ for δ ∈ (0 , 1) and w ∈ R d ,

where σ 2 ν is the sub-Gaussian parameter of the noise ν := η ⊤ θ + ε characterized in Lemma 2.1.

The proof of this result is in Appendix G.2.

P-2SLS Confidence Interval . We now present a novel finite-time confidence interval for the P-2SLS estimator. As discussed in the previous section with respect to this estimator, we assume access a set of data ( Z T 1 , X T 1 ) generated according to Eq. (1) and collected non-adaptively for the purpose of estimating Γ . Moreover, assume access to a separate set of data ( Z T 2 , X T 2 , Y T 2 ) generated according to Eq. (1) and collected non-adaptively for the purpose of estimating θ .

Theorem 2.3. Suppose that ̂ Γ = ( Z ⊤ T 1 Z T 1 ) -1 Z ⊤ T 1 X T 1 and ̂ θ P-2SLS = ( ̂ Γ ⊤ Z T 2 Z T 2 ̂ Γ) -1 ̂ Γ Z ⊤ T 2 Y T 2 . Then, for any w ∈ W , with probability at least 1 -δ for δ ∈ (0 , 1) ,

where

The proof is presented in Appendix G.3. Observe that the first term in the P-2SLS estimator confidence interval given by √ 2 σ 2 ν ∥ w ∥ 2 ¯ A ( Z T 2 , ̂ Γ) -1 log ( 4 /δ ) matches the Oracle estimator confidence interval in Lemma 2.2 when ̂ Γ = Γ . The second term scaling like O ( ∥ w ∥ ¯ A ( Z T 2 , ̂ Γ) -1 ∥ θ ∥ 2 σ η √ d +log ( 1 /δ ) ) ,

is an upper bound on the approximation error w ⊤ ( ̂ Γ -1 Γ -I ) θ for any w ∈ R d , assuming that ̂ Γ is learned from an OLS estimator (see Theorem G.3 for details).

We will see that the form of this confidence interval is particularly convenient for our algorithmic approach given in Section 3. In particular, the form of the variance ∥ w ∥ 2 ¯ A ( Z T 2 , Γ) -1 on the first term only depends on a design over instruments. Thus, we can choose an experimental design over Z 's which reduces this variance optimally.

Remark 2.4 . In practice we expect the first stage of samples, ( Z T 1 , X T 1 ) to be collected from either a burn-in period or from existing historical data. We remark that assuming two stages of samples is common in the orthogonal and double machine learning for estimating nuisance parameters in the data generating process (e.g. Γ ) [9]. Our result matches the existing literature on the asymptotic variance of two sample 2SLS estimators (e.g., Theorem 1 of [19]).

Remark 2.5 . The asymptotic variance of standard 2SLS is known to involve a factor σ 2 ε , instead of σ 2 ν as we have [18]. Recent work by [11] shows a variance involving dσ 2 ε . However, it's unclear how to use the form of their confidence interval directly for experimental design. In addition, their work is not sufficiently general to handle the general forms of noise that we consider in Lemma 2.1.

3 Adaptive Experimental Design Algorithms

We now present adaptive experimental design algorithms for the CPET-LB problem. Our main insight utilizes Eq. 1 by plugging the model for x into the top equation resulting in the relationship

When Γ is known, by Eq. 5, we see that CPET-LB reduces to a standard pure exploration transductive linear bandit problem where the measurement set is given by { Γ ⊤ z } z ∈Z ⊂ R d , the evaluation set is W ⊂ R d , and the feedback model is given by y = v ⊤ θ + ν where the noise ν = θ ⊤ η + ε is sub-Gaussian and as before the goal is to identify arg max w ∈W w ⊤ θ . An existing approach to this problem is given by the RAGE algorithm [15], which we use as the basis of our approach. Addressing the case of unknown Γ is our major algorithmic contribution, where we develop solutions to improve our estimate of Γ and learn w ∗ simultaneously. As a warm-up to this approach, we first consider the setting when Γ is known.

3.1 Warm-Up: Known Structural Model

Algorithm 1 assumes a parameter L ν , which acts as an upper bound on the sub-Gaussian constant of the noise ν = θ ⊤ η + ε . In each round k , an active set of potentially optimal vectors ̂ W k ⊂ W is maintained. CPEG aims to sample in such a way that reduces the uncertainty of the estimates on the gaps ( w -w ′ ) ⊤ θ for each pair w,w ′ ∈ ̂ W k maximally each round. In any given round the algorithm takes N k samples Z N k , the confidence interval of Lemma 2.2 shows that the error in estimating ( w -w ′ ) ⊤ θ scales with ∥ w -w ′ ∥ 2 (Γ ⊤ Z ⊤ Nk Z Nk Γ) -1 . This motivates utilizing an experimental design approach where we choose a distribution λ k ∈ ∆( Z ) to minimize max w,w ′ ∈ ̂ W k ∥ w -w ′ ∥ ( ∑ z ∈Z λ z Γ ⊤ zz ⊤ Γ) -1 . The number of resulting samples taken from this design N k is chosen to guarantee that the confidence interval of Lemma 2.2 is less than 2 -k . Then, the elimination step in Line 8 guarantees that all w ∈ W such that ( w ∗ -w ) ⊤ θ > 2 · 2 -k are then eliminated from the active set by round k +1 of the procedure. To actually choose our samples, as is common in this literature [15], we use an efficient rounding procedure, ROUND that requires a minimum number of samples r ( ω ) .

Sample Complexity Guarantee. The sample complexity of Algorithm 1 depends on the following problem-dependent quantity ρ ∗ ( γ ) that captures the underlying hardness of a problem instance in terms of ( W , Z , Γ , θ ) , when γ = 0 , we abbreviate ρ ∗ (0) = ρ ∗ ,

Theorem 3.1. Algorithm 1 is δ -PAC and terminates in at most c (1 + ω ) L ν ρ ∗ log ( 1 /δ ) + cr ( ω ) samples, where c hides logarithmic factors of ∆ := min w ⟨ w ∗ -w,θ ⟩ and |W| , as well as constants.

The proof of this result is in Appendix H.1. In the unconfounded case when Γ = I and η = 0 , E t -1 [ ε t | x t ] = 0 this matches the sample complexity of [15]. In particular, for the case where Z = X = W , the problem further reduces to a standard multi-armed bandit, and if ε is 1-subGaussian noise, [33] shows that ρ ∗ = O ( ∑ d i =2 ( θ 1 -θ i ) -2 )) , which is the optimal sample complexity

of best-arm identification for multi-armed bandits. The following lemma shows that the conditioning of Γ can have a strong impact on the resulting sample complexity.

To further illustrate the impact of Γ , imagine an extreme setting where Γ = (1 -ε ) /d 11 ⊤ + εI and ε ≈ 0 , i.e. Γ is a perturbation of 1 /d 11 ⊤ . It's straightforward to show that the upper bound in the first display of Lemma 3.2 is of the order O ( dε -2 ) (this is also a lower bound - see Appendix K.1). In particular, the upper bound on the sample complexity is of the form dε -2 / ∆ 2 min . This is in sharp contrast to the linear bandit case, when Γ = I and we are guaranteed a sample complexity of no more than d/ ∆ 2 min samples. To gain some intuition, regardless of the choice of λ , ∑ i ≤ d λ i Γ ⊤ e i e ⊤ i Γ ≈ Γ . As a result,

Algorithm 1 CPEG:Confounded pure exploration with Γ

1: Input Z , W , Ψ = Γ , δ, L ν ≥ σ 2 ν , ω ,

2: Initialize: k = 1 , W 1 = W , ζ 1 = 1

3: Set f ( w,w ′ , Γ , λ ) := ∥ w -w ′ ∥ 2 ( ∑ z ∈Z λ z Γ ⊤ zz ⊤ Γ) -1

4: while |W k | > 1 do

6: ρ ( W k ) = min λ ∈ ∆( Z ) max w,w ′ ∈W k f ( w,w ′ , Γ , λ )

5: λ k = arg min λ ∈ ∆( Z ) max w,w ′ ∈W k f ( w,w ′ , Γ , λ ) .

7: N k := ⌈ 2(1 + ω )2 2 k ρ ( W k ) L ν log ( 4 k 2 |W| /δ ) ⌉ ∨ r ( ω )

8: Pull arms in Z N k = ROUND ( λ k , N k ) and observe Y N k .

9: Compute ̂ θ k Γ = ( Z ⊤ N k Z N k Γ ) -1 Z ⊤ N k Y N k

10: W k +1 = W k \{ w ∈ W k |∃ w ′ ∈ W k , ⟨ w ′ -w, ̂ θ k Γ ⟩ > 2 -k } , k ← k +1

11: end while

12: Output: w ∈ W k

ρ ∗ →∞ as ε → 0 . Intuitively in the limit, regardless of which instrument i ≤ d is being pulled, the resulting distribution on the treatments is uniform (the instruments are weak ). Thus, it is impossible to deconfound the measurement noise, and recover an estimate of θ . This is a phenomenon which does not arise in the standard multi-armed bandit case with unconfounding.

Remark 3.3 . We also consider a setting where instead of given Γ directly, we are given an estimate ̂ Γ of Γ based on offline data. We discuss such an adaptation of Algorithm 1 to this setting in Appendix I and provide a sample complexity which reflects the error in Γ (scaling with ρ ∗ ( γ ) for γ > 0 ). We remark that this result is subsumed by the approach of Section 3.2 and so we omit it in the main text.

Lower bound. Due to the noise model from confounding and the dependence of the noise θ ⊤ η + ε , the instance-dependent lower bounds of [15] do not immediately apply. We develop a lower bound tailored for the confounding setting that nearly match the upper bounds of our algorithms. What's more, our lower bound illustrates the additional difficulty that arises from confounding by an additional factor of d 2 compared to the standard transductive linear bandit problem in the most general setting where entries of η are sub-Gaussian, but not necessarily independent nor bounded. Due to space limit, we defer it to Appendix D.

3.2 Fully Unknown Structural Model

We now consider the setting where Γ is fully unknown. The difficulty of this setting is that the data collection process needs to support both estimation of Γ and θ simultaneously. Our algorithm, built upon Algorithm 1, is summarized in Algorithm 3. At its core, each phase of the algorithm is divided into two sub-phases, for estimating Γ and θ respectively. Specifically, the second sub-phase is essentially same as Algorithm 1 with ̂ Γ k in place of Γ where ̂ Γ k is estimated from the first sub-phase. The main novelty of our algorithmic design lies in the first sub-phase, which resolves the challenge of performing the optimal design for estimating Γ . To explain this challenge, the confidence interval for P-2SLS estimators of Theorem 2.3 indicates that one should pull arms so that we control both D 2 := max w,w ′ ∥ w -w ′ ∥ 2 ¯ A ( Z , Γ ) (error from ˆ θ P -2SLS ) and D 1 := max w,w ′ ∥ w -w ′ ∥ 2 ¯ A ( Z , Γ )

(error from ̂ Γ k ) to be below the target error O ( ζ 2 k ) at each phase (ignoring unimportant factors for discussion). Controlling D 2 is trivial, which is done in the second sub-phase as we described above.

However, for D 1 , a similar strategy cannot be done because the estimate ̂ Γ k is computed directly by sampling arms in Z T 1 . That is, the ideal design, based on which we will collect data points z 1 , . . . , z n , requires access to the random matrix ̂ Γ k that can only be computed after sampling z 1 , . . . , z n ,. This

Algorithm 3 CPEUG: Confounded pure exploration with with unknown Γ

creates a cycle that seems impossible to resolve. Such an issue, to our knowledge, has not been seen in existing work on pure exploration, and thus resolving it is our key technical contribution.

Our solution is to compute the design based on ̂ Γ k from the previous phase. We then perform a doubling trick where we double the sample size (while following the computed design) until D 1 becomes smaller than the target error O ( ζ 2 k ) . The intuition is that in later phases the estimate ̂ Γ k from the previous phase will be accurate enough to ensure that the design is efficient. Note that this novel algorithm induces extra randomness in how many samples we end up collecting in the first sub-phase, which remains random even after conditioning on the history, unlike the second sub-phase. This makes the analysis challenging, which we describe after the main result.

Our algorithm additionally employs the so-called E-optimal design to ensure that the covariance matrix of the collected data used to estimate Γ is well-conditioned. This conditioning is required to ensure that ̂ Γ k concentrates fast enough to Γ as shown in the analysis. The E-optimal design is a well-known design objective in experimental design that aims to maximize the smallest singular value: λ ∗ E := arg min λ ∈ ∆( Z ) σ max ( V -1 ( λ )) , where V = ∑ z ∈Z λ z zz ⊤ . We denote κ -1 0 := σ max ( V -1 ( λ ∗ E )) = -1 ∗

Algorithm 2 θ -estimator Input W , δ, ζ, ̂ Γ , ω, L ν ̂ λ = arg min λ ∈ ∆( Z ) max w,w ′ ∈W f ( w,w ′ , ̂ Γ , λ ) ρ ( W ) = min λ ∈ ∆( Z ) max w,w ′ ∈W f ( w,w ′ , ̂ Γ , λ ) N 2 = ⌈ 2(1 + ω ) ζ -2 ρ ( W ) L ν log ( 4 |W| δ ) ⌉ ∨ r ( ω ) get N 2 samples per design ̂ λ denoted as { Z 2 , X 2 , Y 2 } ▷ via ROUND update ̂ θ P-2SLS = ( ̂ Γ ⊤ Z ⊤ 2 Z 2 ̂ Γ) -1 ̂ Γ ⊤ Z ⊤ 2 Y 2 Output: ̂ θ P-2SLS

σ min ( V ( λ E )) as the smallest singular value achieved by the E-optimal design.

We present our analysis result Theorem 3.4 where we show that, even without knowledge of Γ , the sample complexity scales with the key problem difficulty ρ ∗ almost matching the sample complexity of Algorithm 1 which relies on knowledge of Γ .

Theorem 3.4. Algorithm 3 is δ -PAC and terminates in at most

pulls, ignoring both of the additive and multiplicative logarithms of ∆ , |W| , ρ ∗ , ρ 0 , M , where

Note that ρ 0 does not get hurt by ⟨ w ∗ -w,θ ⟩ , ( ρ ∗ does). It comes from the fact that in the first phase, we initialize that algorithm with E-optimal design.

The challenge of the analysis can be summarized in two-fold. First, since the concentration result in Theorem 2.3 is w.r.t. ̂ Γ k , we need to analyze how the random matrix ̂ Γ k concentrates around Γ and how this impacts the sample complexity. For this, we develop a novel concentration inequality

that relates the confidence width involving ˆ Γ from Theorem 2.3 with the same quantity involving Γ in place of ˆ Γ . Second, our algorithm creates a long-range error propagation, which is highly nontrivial to analyze. To see this, the quality of ˆ Γ k is affected by the design objective function max w,w ′ f ( w,w ′ , ˆ Γ k -1 , λ ) , which depends on the error of the estimate ˆ Γ k -1 from the previous phase. This error is, in turn, affected by the error of ˆ Γ k -2 by the same mechanism. This is repeated all the way back to the first phase. Thus, any abnormal behavior from the first iteration will have a cumulative impact to even the end. In our analysis, we successfully analyze how the error is propagated from the previous iterations, which forms a complicated recursion. Resolving this recursion is our key novelty in the analysis.

Remark 3.5 . Our algorithm requires knowledge of a lower bound γ min of λ min (Γ) . The knowledge of γ min is for simplicity only as one can obtain such a lower bound that is at least half of the true value λ min (Γ) via an efficient sampling procedure that we describe in Appendix K.

Experiments. We provide experiments for the instance of Section 1.2 in the Appendix E. The experiments show that our approach is more sample efficient than natural passive baselines (e.g. A/B testing), or naively applying existing Pure-Exploration linear bandit methods and performs similarly to the oracle complexity.

4 Conclusion

This work introduces the CPET-LB problem in which the learning protocol is characterized by a linear structural equation model governed by parameters Γ and θ . We provide a general solution that simultaneously estimates the structural model while optimally designing to learn the best-arm. The key ideas behind our approach are based on linear experimental design techniques, an instrumental variable estimator whose variance can be controlled by the design, and novel finite-time confidence intervals on this estimator. This paper presents a number of directions for future work including considering situations where the d z = d x , analysis to improve the dependence on the underlying noise variance, and the pursuit of a tight information-theoretic instance-dependent lower-bound. We hope that this line of work motivates increased discussion of the real impact of confounding on applicability of adaptive experimentation.

̸

References

Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Allen-Zhu, Z., Li, Y., Singh, A., and Wang, Y. Near-optimal discrete optimization for experimental design: A regret minimization approach. Mathematical Programming , 186:439-478, 2021.

Angrist, J. D., Imbens, G. W., and Rubin, D. B. Identification of causal effects using instrumental variables. Journal of the American Statistical Association , 91(434):444-455, 1996.

Bakshy, E., Eckles, D., and Bernstein, M. S. Designing and deploying online field experiments. In International Conference on World Wide Web , pages 283-292, 2014.

Bareinboim, E., Forney, A., and Pearl, J. Bandits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems , 28, 2015.

Bilodeau, B., Wang, L., and Roy, D. Adaptively exploiting d-separators with causal bandits. Advances in Neural Information Processing Systems , 35:20381-20392, 2022.

Bradlow, E. Encouragement designs: an approach to self-selected samples in an experimental design. Marketing Letters , 9:383-391, 1998.

Chandak, Y., Shankar, S., Syrgkanis, V., and Brunskill, E. Adaptive instrument design for indirect experiments. arXiv preprint arXiv:2312.02438 , 2023.

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. Double/debiased machine learning for treatment and structural parameters, 2018.

Degenne, R., Ménard, P., Shang, X., and Valko, M. Gamification of pure exploration for linear bandits. In International Conference on Machine Learning , pages 2432-2442. PMLR, 2020.

Della Vecchia, R. and Basu, D. Online instrumental variable regression: Regret analysis and bandit feedback. arXiv preprint arXiv:2302.09357 , 2023.

Elbers, B. Encouragement designs and instrumental variables for a/b testing, August 2023. URL https://engineering.atspotify.com/2023/08/ encouragement-designs-and-instrumental-variables-for-a-b-testing/ . Accessed: 2024-05-19.

Engineering, S. Encouragement designs and instrumental variables for a/b testing, 2023. Accessed: December 26, 2023.

Even-Dar, E., Mannor, S., Mansour, Y., and Mahadevan, S. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

Fiez, T., Jain, L., Jamieson, K. G., and Ratliff, L. Sequential experimental design for transductive linear bandits. Advances in Neural Information Processing Systems , 32, 2019.

Gales, S. B., Sethuraman, S., and Jun, K.-S. Norm-agnostic linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 73-91. PMLR, 2022.

Gong, X. and Zhang, J. Dual instrumental method for confounded kernelized bandits. arXiv preprint arXiv:2209.03224 , 2022.

Greene, W. H. Econometric analysis . Pearson Education India, 2003.

Inoue, A. and Solon, G. Two-sample instrumental variables estimators. The Review of Economics and Statistics , 92(3):557-561, 2010.

Jun, K.-S., Jain, L., Mason, B., and Nassif, H. Improved confidence bounds for the linear logistic model and applications to bandits. In International Conference on Machine Learning , pages 5148-5157. PMLR, 2021.

Kallus, N. Instrument-armed bandits. In Algorithmic Learning Theory , pages 529-546. PMLR, 2018.

Katz-Samuels, J., Jain, L., Karnin, Z., and Jamieson, K. G. An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits. In Advances in Neural Information Processing Systems , volume 33, pages 10371-10382, 2020.

Kohavi, R., Tang, D., and Xu, Y. Trustworthy online controlled experiments: A practical guide to a/b testing . Cambridge University Press, 2020.

Kveton, B., Liu, Y., Kruijssen, J. M., and Nie, Y. Non-compliant bandits. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 1138-1147, 2023.

Lattimore, F., Lattimore, T., and Reid, M. D. Causal bandits: Learning good interventions via causal inference. Advances in Neural Information Processing Systems , 29, 2016.

Lattimore, T. and Szepesvári, C. Bandit algorithms . Cambridge University Press, 2020.

Li, Z., Jamieson, K., and Jain, L. Optimal exploration is no harder than thompson sampling. arXiv preprint arXiv:2310.06069 , 2023.

Lu, Y., Meisami, A., Tewari, A., and Yan, W. Regret analysis of bandit problems with causal background knowledge. In Conference on Uncertainty in Artificial Intelligence , pages 141-150. PMLR, 2020.

Malek, A., Aglietti, V., and Chiappa, S. Additive causal bandits with unknown graph. In International Conference on Machine Learning , 2023.

Mummalaneni, S., Yoganarasimhan, H., and Pathak, V. V. Producer and consumer engagement on social media platforms. SSRN 4173537 , 2022.

Pearl, J. Causality . Cambridge university press, 2009.

Sen, R., Shanmugam, K., Dimakis, A. G., and Shakkottai, S. Identifying best interventions through online importance sampling. In International Conference on Machine Learning , pages 3057-3066. PMLR, 2017.

Soare, M., Lazaric, A., and Munos, R. Best-arm identification in linear bandits. Advances in Neural Information Processing Systems , 27, 2014.

Vershynin, R. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Weltz, J., Fiez, T., V olfovsky, A., Laber, E., Mason, B., Nassif, H., and Jain, L. Experimental designs for heteroskedastic variance. arXiv preprint arXiv:2310.04390 , 2023.

Zhang, J., Chen, Y., and Singh, A. Causal bandits: Online decision-making in endogenous settings. arXiv preprint arXiv:2211.08649 , 2022.

Appendix

Table of Contents

A Broader Impacts

This work is algorithmic and not tied to a particular application that would have immediate negative impact.

B Illustrative Example

We now present an illustrative experiment that highlights the challenges of endogenous noise and the insufficiency of standard experimentation approaches used in the absence of confounding.

Instance Definition. Toward connecting back to membership example in Section 1, consider that a service has d membership options given by the set A = { 1 , . . . , d } . Let the set Z = { e 1 , · · · , e d } represent encouragements (incentives or advertisements) for the corresponding membership options given by W = X = { e 1 , · · · , e d } . We consider a location model that assumes each user t ∈ N arriving online has an underlying unobserved one-dimensional preference u t ∼ N (0 , σ 2 u ) . If an algorithm presents the user with encouragement z I t = e I t for I t ∈ A , then the user selects into the

membership level given by J t = min j ∈A | I t + u t -j | so that x t = e J t . For a visual depiction, see Figure 3a. This process captures a user being more likely to opt-in to membership levels that are closer to the encouragement that they were presented. The outcome is then given by y t = x ⊤ t θ + u t . This problem instance is a specific compliance instance. For this experiment, we take d = 6 , let θ = [ 1 -0 . 95 0 0 . 45 0 . 95 0 . 99 ] and σ 2 u = 0 . 35 . Observe that the optimal evaluation vector is w ∗ = e 1 = arg max w ∈W w ⊤ θ .

We simulate a UCB strategy which maintains estimates of the average reward of each of the possible d incentives, namely ̂ µ i,t = ∑ t s =1 1 { z t = e i } y t and then pulls the one with the highest upper confidence bound. This models current practice of using a bandit algorithm to select which incentive to show a user. Our results averaged over 100 simulations are in Figure 3d. At each round we estimate the average reward of each level using an OLS estimator, i.e. ̂ θ OLS i,t = ∑ t s =1 1 { x t = e i } y t / ∑ t s =1 1 { x t = e i } , and check whether it matches the true value (denoted as UCB-OLS). We also consider an instrumental variable-estimator (see the next Section) which incorporates knowledge of Γ similar to 2SLS to deconfound our estimate (UCB-IV). As the plot demonstrates, UCB-OLS completely fails to identify θ 1 = arg max i ∈ d θ i (this line is hard to see it is at 0) due to a biased estimate, whereas UCB-IV does better. However, UCB-IV methods seem to have a constant probability of error. To see why, note that the expected reward from pulling z = e i is e ⊤ i Γ θ . These values are plotted in orange in Figure 3c. In particular, with some constant probability, UCB runs on the empirical rewards from pulling z 's zeroes in on arm 6, and as a result fails to give enough samples to learn that arm 1 is indeed the best. In contrast, our proposed method CPEG, Algorithm 1 manages to find the best arm with significantly higher probability (the algorithm was run with δ = . 1 ) in the given time horizon.

C Standard 2SLS estimator

Consider Ψ = ̂ Γ 2 SLS = ( Z ⊤ T Z T ) -1 Z ⊤ T X T . In this setting, we recover the standard two-stage-leastsquares ( 2SLS ) estimator,

Note that the 2SLS estimator is a biased, but consistent estimator of the parameter θ [3, 18].

Note that in particular, the asymptotic variance of 2SLS is known to be σ 2 ε ∥ w ∥ ( ̂ Γ ⊤ 2SLS Z ⊤ T Z T ̂ Γ 2SLS ) -1 [18]. Recent work by [11] provides a confidence interval of the form | w ⊤ ( ̂ θ 2SLS -θ ) | ≤ O ( dσ 2 ε ∥ w ∥ ( ̂ Γ ⊤ 2SLS Z ⊤ T Z T ̂ Γ 2SLS ) -1 √ log ( T/δ ) ) . However, it's unclear how to use the form of their confidence interval directly for experimental design due to the dependence of ˆ Γ 2 SLS on the random quantity X . In addition, their work is not sufficiently general to handle the general forms of noise that we consider in Lemma 2.1.

D A non-interactive lower bound

Due to the noise model from confounding and the dependence of the noise θ ⊤ η + ε , the instancedependent lower bounds of [15] do not immediately apply. In this section, we develop a lower bound tailored for the confounding setting.

̸

Toward characterizing the optimal sample complexity, we develop a lower bound for a specific non-adaptive algorithm A that has access to the matrix Γ governing the structural equation model. In particular, suppose that the non-adaptive algorithm A is allowed to select a sequence of T measurements { z I 1 , . . . z I t . . . , z I T } to query prior to collecting any observations, where I t represents the index of the vector z ∈ Z chosen at time t ∈ { 1 , . . . , T } . Then, given the observations { y 1 , . . . , . . . y t , . . . , y T } generated by the environment, a candidate optimal vector ̂ w ∈ W is returned by the algorithm. We are interested in the necessary number of observations T that must be collected in order to ensure P ( ̂ w = w ∗ ) ≤ δ for some δ ∈ (0 , 1) . Thus, it is natural that the optimal non-adaptive algorithm A using the estimator ̂ θ oracle forms a recommendation rule such that ̂ w = arg max w ∈W w ⊤ ̂ θ oracle . We now state our lower bound result with respect to the non-adaptive oracle algorithm.

1

2

3

4

5

6

I

t

J

t

I

t

+

u

t

u

t

(a) Depiction of the problem instance.

(b) Heat-map of

Γ

.

(c)

w

⊤

θ

and

w

⊤

Γ

θ

for

w

∈ {

e

1

, . . . , e

d

}

(d)

P

(

w

t

=

w

∗

)

for the instance

P(x=e;|z=ei)

0.93

0.07

0.00

0.00

0.00

0.00

1

2

0.07

0.85

0.08

0.00

0.00

0.00

-1

3

0.00

0.07

0.85

0.08

0.00

0.00

4

0.00

0.00

0.08

0.85

0.08

0.00

5

0.00

0.00

0.00

0.07

0.85

0.08

-0

6

0.00

0.00

0.00

0.00

0.07

0.93

1

2

3

4

5

6

Z=ej

eTre

UCB-OLS

UCB-IV

CPEG

1.0

1.00

Identification Probability

0.75

0.8

0.50

alue

0.25

0.6

0.00

0.4

0.25

0.50

0.2

0.75

1.00

0.0

1

2

3

4

0

20000

40000

60000

80000

EvaluationIndex

Time Index

̂

̸

Theorem D.1 (Non-Adaptive Oracle Lower Bound) . Consider a problem instance characterized by W ⊂ R d , Z ⊂ R d , Γ ∈ R d × d , and θ ∈ R d . Assume Γ is known, θ is unknown, and the noise process is jointly Gaussian and defined by γ := [ η ε ] ∼ N (0 , Σ) where Σ ∈ R ( d +1) × ( d +1) is an arbitrary correlation matrix. For δ ∈ (0 , 0 . 05] , if the non-adaptive oracle algorithm acquires T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 samples on the problem instance where σ 2 := v ⊤ Σ v and v := [ θ 1 ] ∈ R d +1 , then P ( ̂ w = w ∗ ) ≥ δ .

̸

Corollary D.2. There exists a problem instance characterized by W ⊂ R d , Z ⊂ R d , Γ ∈ R d × d , and θ ∈ R d with a noise process satisfying Assumption 1 such that if the non-adaptive oracle algorithm acquires T ≤ max { d ∥ θ ∥ 2 2 , √ d ∥ θ ∥ 2 } ρ ∗ log ( 1 /δ ) / 2 samples, then P ( ̂ w = w ∗ ) ≥ δ for δ ∈ (0 , 0 . 05] .

The proof of Theorem D.1 is in Appendix F.1. Notably, the result is reminiscent of lower bounds for the standard pure exploration transductive linear bandit problem without confounding [15, 22] when given the measurement set { Γ ⊤ z } z ∈W , evaluation set Z , and parameter θ .

Notably, the upper bounds for our algorithms nearly match the lower bound of Theorem D.1. However, it is interesting to observe that the sample complexity incurs an additional factor of d 2 relative to the standard transductive linear bandit problem in the most general setting where entries of η are sub-Gaussian, but not necessarily independent nor bounded. This illustrates the additional difficulty that arises from confounding. We point out that this is not likely to be a tight lower bound. In

particular, it is a lower bound with respect to a non-adaptive algorithm that uses the particular choice of estimator. We leave improved lower bounds to future work.

E Experiments

We now present experiments a collection of experiments on CPET-LB problem instances. The experiments demonstrate that our approach produces efficient designs for inference and estimation.

E.1 Comparison Algorithms

The baselines that our approaches are compared with are discussed below. We run experiments both when Γ is known and when Γ is fully unknown.

E.1.1 Known Γ .

To standardize the experiments, the baselines considered run in rounds mirroring the structure of Algorithm 1. Specifically, in round k ∈ N a sampling algorithm selects a design λ k ∈ ∆( Z ) , collects N k samples from the design, and forms a Ψ -IV estimate of θ with Ψ = Γ that is combined with a confidence interval (Lemma 2.2 to either eliminate evaluation vectors or validate a stopping condition. The number of samples N k taken in round k ∈ N by any of the algorithms is given by N k = ⌈ 2(1+ ω ) ζ -2 k ρ ( W k ) L ν log ( 4 k 2 |W| /δ ) ⌉∨ r ( ω ) where ρ ( W k ) = max w,w ′ ∈W k f ( w,w ′ , Γ , λ ) for design λ k ∈ ∆( Z ) and an active set of evaluation vectors W k . The round sample count guarantees that given any experimental design, all vectors w ∈ W such that ( w ∗ -w ) ⊤ θ > 2 · 2 -k can be determined to be suboptimal by the end of round k . The sampling methods we consider are now described.

Static Oracle. This design selects λ k = arg min λ ∈ ∆( Z ) max w ∈W\{ w ∗ } f ( w ∗ , w ′ , Γ , λ ) .

Static XY-Optimal. This design selects λ k = arg min λ ∈ ∆( Z ) max w,w ′ ∈W f ( w,w ′ , Γ , λ ) .

Static Uniform. This design selects λ k,z = 1 / |Z| ∀ z ∈ Z .

Adaptive Uniform (SE). This design selects λ k,w = 1 / |W k | ∀ w ∈ W k . Note that this algorithm is effectively an adaption of action-elimination [14] .

The static designs are independent of the round and simply terminate when all evaluation vectors can be eliminated except for a recommended optimal vector ̂ w .

E.1.2 Unknown Γ .

For this set of experiments, we compare Algorithm 3 against a collection of variations of the sampling procedures. Specifically, we compare against methods that either replace only the experimental design for estimating Γ , or only the experimental design for estimating θ , or both with uniform sampling. We label the approaches as N -N , where N represents the sampling approaches (XY or uniform) for Γ and θ respectively. Moreover, to make our approach more practical, we modify the algorithm so that log( Z T , δ ) = 4 d +log ( 1 /δ ) . The step of incrementally adding more E-optimal design samples is also removed, so we collect E-optimal design samples only once in the beginning of Algorithm 4. We find that even with these modifications to the algorithm, correctness is maintained empirically.

E.2 Experiment 1: Jump-Around Instance

We first return back to the location model of Section B. Recall that Z = W = X = { e 1 , · · · , e d } . For this experiment, we take d = 6 , let θ = [ 1 -0 . 95 0 . 45 0 . 45 0 . 95 0 . 45 ] and σ 2 u = 0 . 275 . The results of the experiment are shown in Figure 4a for the case of known Γ . We see that Algorithm 1 performs much better than the baselines and nearly matches the oracle design. Delving into the approach, it is able to quickly eliminate all but w 1 and w d -1 and then puts more mass on z 1 and z d -1 to reduce the uncertainty on w 1 and w d -1 . For the case of unknown Γ , the results are shown in Figure 4d, where θ 5 is reduced to 0 . 9 so that all approaches could finish.

(a) Experiment 1: Known

Γ

(b) Experiment 2: Known

Γ

, iden-

tity

(c) Experiment 2: Known

Γ

, per-

mutation

(d) Experiment 1: Unknown

Γ

(e) Experiment 2:

Unknown

Γ

,

identity

(f) Experiment 2:

Unknown

Γ

,

permutation

Oracle

CPEG

SE

Uniform

Static-XY

1.2

1.4

1.2

Sample Complexity

1.0

Sample Complexity

10

Sample Complexity

0.8

0.6

.

2

0.4

0.4

0.2

0.2

0.0

Algorithm

1.0

0.9

80

0.7

1.0

0.9

0.8

0.7

XY-XY

Uniform-XY

XY-Uniform

Uniform-Uniform

1.2

3.0

3.0

Sample Complexity

1.0

Sample Complexity

2.5

Sample Complexity

2.5

0.8

0.6

1.5

1.5

0.4

1.0

1.0

0.2

0.5

0.5

0.0

Algorithm

0.0

Algorithm

0.0

Algorithm

E.3 Experiment 2: Interpolation Instance

Let Z = W = X = { e 1 , · · · , e d } define the measurement, evaluation, and observation sets. We first consider that Γ := (1 -ε ) d 1 d 1 ⊤ d + εI d for a parameter ε ∈ (0 , 1) where 1 d is a d -dimensional vector of 1's and I d is the d -dimensional identity matrix. For this experiment, we take d = 4 and let θ = [ 0 . 5 0 . 583 0 . 67 0 . 75 ] . As in all compliance instances, η t = x t -Γ ⊤ z t , and in this simulation η t = 0 . 4 η ⊤ t v t , where v t = ¯ v t / ∥ ¯ v t ∥ 2 and ¯ v t ∼ N (0 , I d ) . The results of the experiment are shown in Figure 4b for ε ∈ { 1 , 0 . 9 , 0 . 8 , 0 . 7 } with Γ known. Note that Static-XY and Uniform overlap, and SE and CPEG overlap. We see that Algorithm 1 and the adaptive uniform strategy perform similarly and near optimally. This is to be expected since the most efficient way to gather observations for treatments is to encourage that treatment, given that if the encouragement is not followed each of the alternatives is equally likely and provides no additional information of interest. Moreover, as discussed earlier, the problem gets more challenging as Γ → 1 d 1 ⊤ d /d . Note that the identity matrix could be replaced with a permutation matrix, in which case uniform sampling with elimination becomes highly suboptimal. The results for the case of Γ unknown are shown in Figure 4e with ε = 0 . 99 . This shows the value that comes from the experimental design for estimating both Γ and θ .

To demonstrate the superiority of our algorithm over SE, we also consider that Γ := (1 -ε ) d 1 d 1 ⊤ d + εI p d , where I p d is a permutation matrix as follows,

All other settings remain the same as in the previous interpolation instance. The results for the known Γ case, shown in Figure 4c, indicate that SE exhibits significant underperformance due to its sampling rule not accounting for the permutation effect in Γ . In contrast, CPEG consistently achieves near-optimal performance. Note that Static-XY and Uniform still overlap. Figure 4f presents the results for the unknown Γ case, where we can notice that, comparing with Figure 4e, estimating different permutation matrices (with the identity matrix as a special case) does not affect problem difficulty.

F Proofs of the lower bound

F.1 Proof of Theorem D.1

Theorem D.1. Consider a problem instance characterized by W ⊂ R d , Z ⊂ R d , Γ ∈ R d × d , and θ ∈ R d . Assume Γ is known, θ is unknown, and the noise process is jointly Gaussian and defined by γ := [ η ε ] ∼ N (0 , Σ) where Σ ∈ R ( d +1) × ( d +1) is an arbitrary correlation matrix. For δ ∈ (0 , 0 . 05] , if the non-adaptive oracle algorithm acquires T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 samples on the problem instance where σ 2 := v ⊤ Σ v and v := [ θ 1 ] ∈ R d +1 , then P ( ̂ w = w ∗ ) ≥ δ .

̸

Proof. We begin by recalling the framework of the non-adaptive oracle algorithm and discussing the properties of its estimator for the noise structure described in the statement of the result.

Non-Adaptive Oracle and Instance Definition. The non-adaptive oracle algorithm A selects T measurements to query prior to collecting any data. Let I t represent the index of the vector z ∈ Z chosen at time t ∈ { 1 , . . . , T } . The noise process for the instance under consideration is assumed to be jointly Gaussian and defined by γ t := [ η t ε t ] ∼ N (0 , Σ) where Σ ∈ R ( d +1) × ( d +1) is an arbitrary positive semidefinite matrix. Defining ¯ x I t := Γ ⊤ z I t , v := [ θ 1 ] ∈ R d +1 and ν t := v ⊤ γ t , the feedback model can be described as follows:

Observe that the noise is independent and identically distributed as ν t ∼ N (0 , σ 2 ) where σ 2 := v ⊤ Σ v since γ t ∼ N (0 , Σ) . Moreover, the noise process is exogeneous with E [ ν t | ¯ x I t ] = 0 since ¯ x I t is deterministic given the index choice I t .

Let { z I t } T t =1 , { ¯ x I t } T t =1 , and { y t } T t =1 denote the observations collected by the non-adaptive oracle algorithm A and define Z T ∈ R T × d , ¯ X T ∈ R T × d , and Y T ∈ R T to contain the respective stacked observations. Algorithm A obtains an estimate ̂ θ oracle by minimizing the sum of squares as follows:

Given ̂ θ oracle , the non-adaptive oracle algorithms A returns a recommendation defined by ̂ w = arg max w ∈W w ⊤ ̂ θ oracle . Note that since ̂ θ oracle is obtained by least squares with exogeneous, independent and identically distributed mean-zero Gaussian noise, it is straightforward to verify the estimator is distributed as

where

Proof by Contradiction. To begin, recall that

̸

Suppose for the sake of contradiction that the number of samples collected by the non-adaptive oracle algorithm A is T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 and P ( ̂ w = w ∗ ) < δ for δ ∈ (0 , 0 . 05] . To reach a contradiction, we analyze the distribution of ( w -w ∗ ) ⊤ ̂ θ for some w = w ∗ and show that with probability at least δ it is positive. We remark that this proof follows similar techniques to that of the proof of Theorem 3 of [22].

̸

Let λ ∈ ∆( Z ) represent the empirical sampling distribution of the algorithm A , which is defined such that λ z = 1 T ∑ T t =1 { z I t = z } for each z ∈ Z . Moreover, define

Note that ρ ∗ ( λ ) ≥ ρ ∗ and observe by definition,

Thus, by Eq. (8), and

Furthermore, by the definition of ρ ∗ ( λ ) , the assumption T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 , and the fact ρ ∗ ( λ ) ≥ ρ ∗ , we obtain

Now, consider a random variable W ∼ N (0 , 1) . Proposition 2.1.2 of Vershynin [34] gives an anti-concentration result showing that for all ζ > 0 ,

We apply this result to the quantity ( ˜ w -w ∗ ) ⊤ ( ̂ θ Oracle -θ ) / ⟨ w ∗ -˜ w,θ ⟩ to conclude that ( ˜ w -w ∗ ) ⊤ ̂ θ Oracle > 0 with probability at least δ . Toward doing so, let c ∈ (1 , 1 . 15] be a constant and define

Observe that W ∼ N (0 , 1) . The following analysis holds for δ ∈ (0 , 0 . 05] given that c ∈ (1 , 1 . 15] as assumed:

The final inequality can be verified computationally. Thus, with probability at least δ for δ ∈ (0 , 0 . 05] , we obtain

Observe that the final inequality holds since c > 1 and ( w ∗ -˜ w ) ⊤ θ > 0 by definition.

̸

This result directly implies that with probability at least δ for δ ∈ (0 , 0 . 05] , the vector ̂ w returned by algorithm A is not w ∗ . This is a contradiction, so we conclude that if the non-adaptive oracle algorithm A acquires T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 samples, then P ( ̂ w = w ∗ ) ≥ δ for δ ∈ (0 , 0 . 05] .

F.2 Proof of Corollary D.2

Corollary D.2. There exists a problem instance characterized by W ⊂ R d , Z ⊂ R d , Γ ∈ R d × d , and θ ∈ R d with a noise process satisfying Assumption 1 such that if the non-adaptive oracle algorithm acquires T ≤ max { d ∥ θ ∥ 2 2 , √ d ∥ θ ∥ 2 } ρ ∗ log ( 1 /δ ) / 2 samples, then P ( ̂ w = w ∗ ) ≥ δ for δ ∈ (0 , 0 . 05] .

̸

̸

Proof. To begin, consider the specifications of Theorem D.1 and its result. That is, a problem an arbitrary instance characterized by W ⊂ R d , Z ⊂ R d , Γ ∈ R d × d , and θ ∈ R d where the noise process is jointly Gaussian and defined by γ := [ η ε ] ∼ N (0 , Σ) where Σ ∈ R ( d +1) × ( d +1) is an arbitrary correlation matrix. Observe that the noise process defined by γ satisfies Assumption 1. The result states that if the non-adaptive oracle algorithm acquires T ≤ σ 2 ρ ∗ log ( 1 /δ ) / 2 samples on the problem instance where σ 2 := v ⊤ Σ v and v := [ θ 1 ] ∈ R d +1 , then P ( ̂ w = w ∗ ) ≥ δ for δ ∈ (0 , 0 . 05] . From this point, we show that there exists a parameter θ and correlation matrix Σ such that σ 2 := v ⊤ Σ v ≥ max { d ∥ θ ∥ 2 2 , √ d ∥ θ ∥ 2 } in order to reach the stated conclusion.

̸

Notation. Let ζ η i ,ε ∈ [ -1 , 1] denote the correlation between η i and ε for i ∈ { 1 , . . . , d } . Similarly, let ζ η i ,η j = ζ η j ,η i ∈ [ -1 , 1] denote the correlation between η i and η j for i = j ∈ { 1 , . . . , d } . Note that the correlation of η i with itself for i ∈ { 1 , . . . , d } is σ 2 η i = ζ η i ,η i = 1 and similarly the correlation of ε with itself is σ 2 ε = ζ ε,ε = 1 . The correlation matrix Σ is then given by

where

Moreover, we use the notation Σ ⊤ θ,i ∈ R d to denote the i -th row of Σ ⊤ θ , or equivalently the i -th column of Σ θ , for any i ∈ { 1 , . . . , d } .

Lower Bounding Noise Variance. Given the above notation, we now work toward lower bounding σ 2 := v ⊤ Σ v . Observe that by algebraic manipulations,

Since Σ θ is a real symmetric matrix, an eigendecomposition exists such that Σ θ = Q Λ Q ⊤ where Λ = diag ( λ 1 , . . . , λ d ) ∈ R d × d is a diagonal matrix containing the eigenvalues of Σ θ and Q ∈ R d × d

is an orthogonal matrix with columns corresponding to the eigenvectors of Σ θ . Let q i := Q ⊤ i denote column i of the matrix Q for i = { 1 , . . . , d } , which is equivalently eigenvector i of Σ θ for i = { 1 , . . . , d } . Without loss of generality, assume that the eigenvectors are of unit length so that ∥ q i ∥ 2 = 1 for all i = { 1 , . . . , d } .

Given this information, suppose that the parameter θ in the instance is equal to a scalar multiple of the eigenvector of Σ θ corresponding to the maximum eigenvalue. Note this is equivalent to the statement that θ is equal to some scalar multiple of the column q ∗ ∈ R d of the matrix Q where q ∗ is the eigenvector of Σ θ corresponding to the maximum eigenvalue λ ∗ . Thus, we take θ = c · q ∗ for some c ∈ R and observe that ∥ θ ∥ 2 = c . Toward quantifying the value of v ⊤ Σ v for the problem instance, we begin by characterizing θ ⊤ Σ θ θ for the choice of θ . Consider the following analysis:

̸

Thus, in general for this choice of θ ,

To conclude, take Σ θ := 1 d 1 ⊤ d where 1 d represents the d -dimensional vector of all ones. Since this is a rank-1 matrix, the maximum eigenvalue is λ ∗ = 1 ⊤ d 1 d = d and the remainder of the eigenvalues are zero. Observe that q ∗ = 1 d / √ d is an eigenvector corresponding to the maximum eigenvalue since 1 d 1 ⊤ d q ∗ = dq ∗ . Thus,

This completes the proof since we have shown that there exists a parameter θ and correlation matrix Σ such that σ 2 := v ⊤ Σ v ≥ max { d ∥ θ ∥ 2 2 , √ d ∥ θ ∥ 2 } , which by Theorem D.1 allows us to make the stated conclusion.

G Proofs of the confidence interval

G.1 Proof of Lemma 2.1

The first statement is an immediate consequence of Lemma G.2 and the second statement is proven in Lemma G.1.

Lemma G.1. In the compliance model, the noise η ⊤ θ + ε follows a ( 8 ∥ θ ∥ 2 2 +2 ) -sub-Gaussian distribution.

Proof. In compliance, we have z, x ∈ { e 1 , · · · , e d } , and

Let us figure out the sub-Gaussian parameter of the random vector η . Fix any unit vector a . First, we have E [ ⟨ η, a ⟩ ] = 0 . Second, we have

Thus, η ⊤ a is bounded and zero-mean and thus 2 2 -sub-Gaussian. This implies that

and thus η is a 2 2 -sub-Gaussan random vector. Then, η ⊤ θ is (2 ∥ θ ∥ ) 2 -sub-Gaussian.

Using Lemma G.2, we have that η ⊤ θ + ε is 2(4 ∥ θ ∥ 2 +1) -sub-Gaussian.

Lemma G.2. Let A and B random variables that are each σ 2 A - and σ 2 B -sub-Gaussian but are correlated. Then, A + B is 2( σ 2 A + σ 2 B ) -sub-Gaussian.

Proof. By definition of sub-Gaussian, we have for any γ ∈ R ,

G.2 Proof of Lemma 2.2

Lemma 2.2. Suppose that T observations are collected non-adaptively from the structural equation model in Eqs. (1) and Γ ∈ R d × d is known. Then, with probability at least 1 -δ for δ ∈ (0 , 1) and w ∈ R d ,

where σ 2 ν is the sub-Gaussian parameter of the noise process ν := η ⊤ θ + ε as characterized in Lemma 2.1.

Proof. Given the knowledge of Γ , we have the oracle 2SLS estimator

Note that

Denote ν t := η ⊤ θ + ε t . For any w ∈ W , we have t 〈 ̂ θ oracle -θ, w 〉 = 〈   T ∑ t =1 z s z ⊤ s Γ   -1 T ∑ t =1 z s y t -θ, w 〉 = 〈   T ∑ t =1 z s z ⊤ s Γ   -1 T ∑ t =1 z s ( z ⊤ s Γ θ + ν t ) -θ, w 〉 = 〈   T ∑ t =1 z s z ⊤ s Γ   -1   T ∑ t =1 z s z ⊤ s Γ θ + T ∑ t =1 z s ν t   -θ, w 〉 = 〈   T ∑ t =1 z s z ⊤ s Γ   -1 T ∑ t =1 z s ν t , w 〉 = t ∑ q =1 〈   T ∑ t =1 z s z ⊤ s Γ   -1 z q , w 〉 ν q . By Lemma G.1, the noise ν t is σ 2 ν -sub-Gaussian, we have 〈   T ∑ t =1 z s z ⊤ s Γ   -1 z q , w 〉 ν q is 〈 ( ∑ T t =1 z s z ⊤ s Γ ) -1 z q , w 〉 2 σ 2 ν -sub-Gaussian. Thus P      〈 ̂ θ oracle -θ, w 〉 ≥ √ √ √ √ √ 2 t ∑ q =1 〈   T ∑ t =1 z s z ⊤ s Γ   -1 z q , w 〉 2 σ 2 ν log ( 1 δ )      ≤ δ. Concisely, t ∑ q =1 〈   T ∑ t =1 z s z ⊤ s Γ   -1 z q , w 〉 2 = w ⊤   T ∑ t =1 z s z ⊤ s Γ   -1   t ∑ q =1 z q z ⊤ q        T ∑ t =1 z s z ⊤ s Γ   -1    ⊤ w = w ⊤ Γ -1      T ∑ t =1 z s z ⊤ s Γ   -1    ⊤ w = w ⊤    Γ ⊤   T ∑ t =1 z s z ⊤ s   Γ    -1 w. (11) Thus, P      〈 ̂ θ oracle -θ, w 〉 ≥ √ √ √ √ √ √ 2 w ⊤    Γ ⊤   T ∑ t =1 z s z ⊤ s   Γ    -1 wσ 2 ν log ( 1 δ )      ≤ δ. We can further write it as P   〈 ̂ θ oracle -θ, w 〉 ≥ √ 2 ∥ w ∥ 2 ( Γ ⊤ ( ∑ T t =1 z s z ⊤ s ) Γ ) -1 σ 2 ν log ( 1 δ )   ≤ δ.

By taking a union bound over, we have the confidence interval for the absolute value as in the statement of the lemma.

G.3 Proof of Theorem 2.3

Theorem 2.3. Suppose that ̂ Γ is estimated through a design matrix Z T 1 ∈ R T 1 × d and ̂ θ P-2SLS is estimated through a design matrix Z T 2 ∈ R T 2 × d . Then, for any w ∈ W , with probability at least 1 -δ ,

where σ 2 ν is the sub-Gaussian parameter of the noise ν := η ⊤ θ + ε , σ 2 η is the sub-Gaussian parameter of the noise η , and

Proof. For the pseudo 2SLS estimator, we have

For any w ∈ W , we have

We upper bound the first term and the second term separately. For the first term, by Lemma 2.1, we know ν t is σ 2 ν -subGaussian, we have ∑ T 2 q =1 〈 ( ∑ T 2 t =1 z I t z ⊤ I t ̂ Γ ) -1 z I q , w 〉 ν q is

∑ T 2 q =1 〈 ( ∑ T 2 t =1 z I t z ⊤ I t ̂ Γ ) -1 z I q , w 〉 2 σ 2 ν -subGaussian. Thus by the concentration inequality of sub-

Gaussian random variables, we have

By similar calculation as (11), we have with probability at least 1 -δ 2 ,

Thus with probability at least 1 -δ 2 ,

By Theorem G.3, we have with probability at least 1 -δ 2 ,

Combining (13) and (14), we have with probability at least 1 -δ , for any w ∈ W ,

By a union bound, we have the confidence interval for the absolute value of the inner product,

Theorem G.3. Suppose that the least square estimator ̂ Γ is estimated through a design matrix Z T 1 ∈ R T 1 × d , then it satisfies, with probability at least 1 -δ ,

for any w ∈ W .

Proof. Define V = Z ⊤ T 1 Z T 1 , S ∈ R T 1 × d as the matrix with i -th row being η ⊤ i , the stacked noise, i.e., the data collection process of the design matrix Z T 1 is X = Z T 1 Γ + S . Then we have

where the last inequality is due to Lemma G.4.

Lemma G.4. Suppose we have z 1 , . . . , z T ∈ R d and η 1 , . . . , η T ∈ R d such that η T | z 1 , η 1 , . . . , z T -1 , η T -1 , z T is σ 2 η -sub-Gaussian vector (defined in Assumption 1). Let Z, S ∈ R T × d be matrices whose t -th row is z ⊤ t and η ⊤ t respectively. Suppose ∥ z t ∥ ≤ L z , ∀ t . Let V = Z ⊤ Z . Then, ∀ δ ∈ (0 , 1) , we have, with probability at least 1 -δ ,

Proof. By the definition of operator norm, we have

Considering a fixed w ∈ R d , by Lemma G.5, we have with probability at least 1 -δ ,

By Lemma G.6 and a union bound, for the ε -covering C ε , we have the following event happens with probability no more than δ :

When E

does not happen, we have

Thus,

By choosing ε = 1 2 and Lemma G.6, we have

Lemma G.5. [Self-Normalized Bound for Vector-Valued Martingales] Suppose we have z 1 , . . . , z t ∈ R d and η 1 , . . . , η t ∈ R d such that η t | z 1 , η 1 , . . . , z t -1 , η t -1 , z t is σ 2 η -sub-Gaussian vector (defined in Assumption 1). Let Z, S ∈ R t × d be matrices whose s -th row is z ⊤ s and η ⊤ s respectively. Suppose ∥ z s ∥ ≤ L z , ∀ s . Let V t = Z ⊤ Z . Then, ∀ b ∈ R t , δ ∈ (0 , 1) , we have, with probability at least 1 -δ ,

Proof. Since each row of S is a σ 2 η -subGaussian vector, we have that ( Sb ) i / ∥ b ∥ is σ 2 η -subGaussian. Using Lemma G.7 with ε s = ( Sb ) i / ∥ b ∥ completes the proof.

Lemma G.6. [26][Lemma 20.1] There exists a set C ε ⊂ R d with |C ε | ≤ ( 3 ε ) d such that for any x ∈ { x | x ∈ R d , ∥ x ∥ 2 = 1 } , there exists a y ∈ C ε such that ∥ x -y ∥ 2 ≤ ε .

Lemma G.7. Let z 1 , z 2 , . . . ∈ { z ∈ R d : ∥ z ∥ 2 ≤ L z } and ε 1 , ε 2 , . . . ∈ R be random variables such that ε k | z 1 , ε 1 , . . . , z t -1 , ε t -1 , z t is σ 2 ε -sub-Gaussian. Let V t = ∑ t s =1 z s z ⊤ s . Then,

Proof. Let Z t ∈ R t × d be the design matrix and define ε t := ( ε 1 , . . . , ε t ) ⊤ ∈ R t . Let us omit the subscript t from Z t , ε t and V t . Note that

It remains to bound ∥ Z ⊤ ε ∥ ( V + σ min ( V ) I ) -1 . We use union bound with the standard self-normalized inequality of Lattimore and Szepesvári [26][Theorem 20.4 and Note 20.2]. Specifically, let λ k = 2 -k +1 for k ≥ 1 . Then,

Under the event E , we have two cases:

Altogether, we have

We conclude the proof by simplifying the RHS.

H Proofs of sample complexity when given Γ

H.1 Proof of Theorem 3.1

Algorithm 5 Confounded pure exploration with known Γ

Theorem 3.1. Algorithm 1 is δ -PAC for the CPET-LB problem and terminates in at most c (1 + ω ) L ν ρ ∗ log ( 1 /δ ) + cr ( ω ) samples, where c hides logarithmic factors of ∆ and |W| .

Proof. Part 1 δ -PAC

By the confidence interval in Lemma 2.2, we have, with probability at least 1 -δ 2 k 2 |W| ,

Define a good event E k,w for each k and W k as

We claim that with probability at least 1 -δ , the event ⋂ ∞ k =1 ⋂ w ∈W k E k,w holds. It can be proved by a union bound.

where the last step is by the fact that ∑ ∞ k =1 1 k 2 = π 2 6 < 2 . Under the event ⋂ ∞ k =1 ⋂ w ∈W k E w,k , to show that the best arm is never eliminated, it suffices to show that for any sub-optimal arm w ∈ W k ,

Thus the best arm w ∗ never satisfies the elimination condition. Next we show that at the end of stage k , any suboptimal arm w that satisfies

is eliminated. To show this, we need to show that w satisfies the elimination condition,

This implies that with probability at least 1 -δ , w ∗ always survives.

Part 2 sample complexity

Define S k = { w ∈ W | ⟨ w ∗ -w,θ ⟩ ≤ 4 ζ k } . Thus with probability at least 1 -δ , we have ⋂ k {W k ⊆ S k } . This implies the following is true with probability at least 1 -δ for all k

Define ∆ to be the minimum gap between w ∗ and any other w ∈ W , i.e., ∆ := min w ∈W\{ w ∗ } ⟨ w ∗ -w,θ ⟩ . Then for k ≥ ⌈ log ( 4∆ -1 ) ⌉ , we have S k = { w ∗ } with probability

at least 1 -δ . The total sample complexity is the summation of the number of samples in each round,

On the other hand, we have

where the last inequality is by the triangle inequality, i.e.,

Combining (16) and (17), we have

I Proofs of sample complexity when given ̂ Γ

Algorithm 6 Confounded pure exploration with known ̂ Γ

Theorem I.1. Suppose that we have ̂ Γ that is an OLS estimate from an offline dataset { Z T , X T } collected non-adaptively through a fixed design ξ and the efficient rounding procedure ROUND,as well as T ≥ 4 σ 2 η σ min ( A ( ξ, Γ) ) log ( Z T , δ/ 2 ) . Algorithm 6 guarantees that with probability at least 1 -δ , a 6 γ -good arm is returned, where

Also, the algorithm terminates in at most

samples, where c hides logarithmic factors of ∆ and |W| and γ .

Proof. The proof can be divided into four steps:

The best arm w ∗ is never eliminated.

At the end of stage k , any suboptimal arm w that satisfies ⟨ θ, w ∗ -w ⟩ ≥ 2 ζ k + 2 γ is eliminated.

The stopping condition is met in finite time.

When the stopping condition is met, there are only 6 γ -good arms left.

and

The upper bound of the sample complexity.

Step 1: The best arm w ∗ is never eliminated.

By the confidence interval in Theorem 2.3, we have, with probability at least 1 -δ 2 k 2 |W| , for any k and w ∈ W k ,

Define a good event E k,w for each k and w ∈ W k as

By the same calculation as (15), we claim that with probability at least 1 -δ , the event ⋂ ∞ k =1 ⋂ w ∈W k E k,w holds. Under the event ⋂ ∞ k =1 ⋂ w ∈W k E w,k , to show that the best arm is never eliminated, it suffices to show that for any sub-optimal arm w ∈ W k ,

Thus the best arm w ∗ never satisfies the elimination condition.

Step 2: At the end of stage k , any suboptimal arm w that satisfies ⟨ θ, w ∗ -w ⟩ ≥ 2 ζ k +2 γ is eliminated.

To prove this, we show that such arm w must satisfy the elimination condition,

Thus the arm w is eliminated.

Step 3: The stopping condition is met in finite time.

Given the result in Step 2 and the fact that ζ k is an exponentially decreasing sequence, we know that all of the arms w satisfying ⟨ θ, w ∗ -w ⟩ > 2 γ will be eliminated in finite time. This means that only the arms w satisfying ⟨ θ, w ∗ -w ⟩ ≤ 2 γ will remain. We need to show that ∀ w,w ′ ∈ W k , 〈 w ′ -w, ̂ θ k P-2SLS 〉 ≤ 4 γ can be achieved in finite time. When ζ k ≤ γ , (which will happen in

finite time),

Step 4: When the stopping condition is met, there are only 6 γ -good arms left.

For any w ∈ W k , we have

Step 5: The upper bound of the sample complexity.

Define W (2 γ ) as the set of 2 γ -good arms, i.e., W (2 γ ) := { w ∈ W | ⟨ θ, w ∗ -w ⟩ ≤ 2 γ } . Then the best arm in the set W\W (2 γ ) has a suboptimality gap min w ∈W\W (2 γ ) ⟨ θ, w ∗ -w ⟩ . We define ∆ min (2 γ ) := min w ∈W\W (2 γ ) ⟨ θ, w ∗ -w ⟩ -2 γ . By the result in Step 2, we know that after k ∗ := ⌈ log ( 4∆ min (2 γ ) -1 ) ⌉ stages, all of the arms in W\W (2 γ ) are eliminated. Define S k = { w ∈ W | ⟨ w ∗ -w,θ ⟩ ≤ 4 ζ k +2 γ } . Thus with probability at least 1 -δ , we have ⋂ k {W k ⊆ S k } .

The sample complexity of the algorithm is the total number of samples pulled, which is

Note that the factor of min λ ∈ ∆( Z ) max w,w ′ ∈ S k ∥ w -w ′ ∥ 2 A ( λ, Γ) -1 has the underlying true Γ in it, while the algorithm uses the plugged-in ̂ Γ . We need to relate it to ρ ( S k ) := min λ ∈ ∆( Z ) max w,w ′ ∈ S k ∥ w -w ′ ∥ 2 A ( λ, ̂ Γ) -1 . By Lemma J.9, and defining λ ∗ z ( S k ) as the optimal design for S k , i.e.,

Define λ ′ k = 1 2 λ ∗ k + 1 2 ξ , we have

where the last inequality is due to

as well as, by Lemma J.11, when T ≥ 4 σ 2 η σ min ( A ( ξ, Γ) ) log ( Z T , δ/ 2 ) , we have, with probability at least 1 -δ/ 2 , we have

We can lower bound ρ ∗ ( γ ) by

Given (19), (20) and (21), we have

where c hides logarithmic factors of ∆ and |W| and γ .

J Proofs of sample complexity with unknown Γ

Algorithm 7 Optimal design with unknown Γ

The algorithms of Γ -estimator and θ -estimator we present below are slightly different from the one in the main text. In the main text, we omit the phase index k in the algorithm for simplicity.

Algorithm 8 Γ -

Algorithm 9 θ -estimator

Input W , δ, ζ , Γ , ω, L , k

Lemma J.1. Algorithm 3 and N k, 1 ,ℓ , N k, 0 ,ℓ guarantees three properties:

Property 1: N k, 1 ,ℓ ≥ N k, 0 ,ℓ .

Property 2: 1 2 ( N k, 0 ,ℓ + N k, 1 ,ℓ ) ≤ N k, 0 ,ℓ -1 + N k, 1 ,ℓ -1 .

Proof. For Property 1, recall that N k, 0 ,ℓ and N k, 1 ,ℓ are defined as

and

We prove the result by induction. For the result for ℓ = 1 , note that N k, 1 , 1 is of the form N k, 1 , 1 = 2 a +2 b ln(1 + 2 c +2 bd ) and N k, 0 , 1 is of the form N k, 0 , 1 = a + b ln( c + dr ) , where

By the contraposition of Lemma K.5, we have

Thus we have N k, 1 , 1 ≥ N k, 0 , 1 . Now we assume the result holds for ℓ , i.e., N k, 1 ,ℓ ≥ N k, 0 ,ℓ and prove that it holds for ℓ +1 . We have

It suffices to prove that

We have

For Property 2, it suffices to prove that

This is equivalent to prove that

The last inequality holds because c = M > 1 . For Property 3, we have,

We have

Theorem J.2. Algorithm 3 guarantees that with probability at least 1 -δ , the best arm is returned, and the algorithm terminates in at most

pulls, ignoring both of the additive and multiplicative logarithms of ∆ , |W| , ρ ∗ , ρ 0 , M , where

and and

Note that ρ 0 does not get hurt by ⟨ w ∗ -w,θ ⟩ . It comes from the fact that in the first phase, we initialize that algorithm with E-optimal design.

Proof. Part 1: correctness of the algorithm

The idea of the proof is similar to the proof of Theorem 3.1.

Recall that the confidence interval of P-2SLS can be break down into two terms (12).

Given ̂ Γ , for a w ∈ W , with probability at least 1 -δ 2 the first term satisfies

For any w ∈ W , with probability at least 1 -δ 2 , the second term satisfies

Note that by Lemma G.4, the above inequality holds for all w ∈ W , and the RHS is essentially a result of

In the vanilla form of the confidence of P-2SLS , we can define good events as

for the first term, for any w ∈ W ,

for the second term, ∥ ∥ ∥ V -1 / 2 Z ⊤ T S ∥ ∥ ∥ op ≤ √ σ 2 η log ( Z T , δ/ 4 ) .

For our algorithm design, since we use the doubling trick for the first sub-phase, we need to define the good event for the first sub-phase as the samples from each doubling trick iteration satisfies the self-normalized concentration inequality of Lemma G.4.

We define the good event for ℓ -th doubling trick iteration in the first sub-phase of phase k as

where Z k,ℓ = Z k, 0 ,ℓ ∪ Z k, 1 ,ℓ , V k,ℓ = Z ⊤ k,ℓ Z k,ℓ and S k,ℓ is stacked noise matrix during collecting samples Z k,ℓ per the model X = Z Γ + S . By a union bound, we have

For the second sub-phase in phase k , we define the good event for the second sub-phase in phase k and w ∈ W as

By a union bound, we have

Under the good event ( ⋂ ∞ k =1 ⋂ ∞ ℓ =1 E 1 k,ℓ ) ⋂ ( ⋂ ∞ k =1 ⋂ w ∈W k E 2 k,w ) , we have with probability at least 1 -δ , for all k and w ∈ W ,

The rest of proof is same as the proof of Theorem 3.1.

Part 2: sample complexity of algorithm

Sample complexity for first sub-phase

Recall that λ E = arg max λ ∈ ∆( Z ) σ min ( ∑ z ∈Z λ z zz ⊤ ) is the E-optimal design to maximize the minimum singular value of ∑ z ∈Z λ z zz ⊤ and κ 0 = max λ σ min ( ∑ z ∈Z λ z zz ⊤ ) is the maximum minimum singular value of ∑ z ∈Z λ z zz ⊤ . At the beginning of first sub-phase in phase k , the algorithm first samples N k, 0 , 0 arms according to λ E .

Before we proceed to the main proof of the sample complexity, we first address a minor technique issue to avoid cumbersomeness. For the logarithmic term that appears in the algorithm and confidence interval,

when 2 ∧ σ min ( V ) = 2 , it is equivalent to

Our algorithm design guarantees that 2 ∧ σ min ( V ) = 2 is always true whenever we need to use the logarithmic term log( Z T , δ ) , given that the samples of our interest always includes the E-optimal

design samples Z k, 0 ,ℓ and the number of samples from the E-optimal design ∣ ∣ Z k, 0 ,ℓ ∣ ∣ is always larger than 2 κ 0 . So for the remaining part of the proof, we will use ˜ log( N,δ ) instead of log( Z T , δ ) .

Denote the samples of E-optimal design that mixed into the samples from ℓ -th doubling trick iteration in the first sub-phase of phase k as Z k, 0 ,ℓ and ∣ ∣ Z k, 0 ,ℓ ∣ ∣ = N k, 0 ,ℓ . By Lemma J.3, our choice of N k, 0 ,ℓ

is a sufficient condition to guarantee

Multiply both sides of (22) by N k, 0 ,ℓ + N k, 1 ,ℓ N k, 0 ,ℓ and we have

By Property 1 of Lemma J.1, we have α k,ℓ := N k, 0 ,ℓ N k, 0 ,ℓ + N k, 1 ,ℓ < 1 / 2 , then

These condition on N k, 0 ,ℓ and N k, 0 ,ℓ + N k, 1 ,ℓ are needed for the proof.

Denote the total number of doubling trick iterations as L k for phase k . In the case of L k = 1 , the samples from the first doubling trick iteration satisfies stopping condition of the first sub-phase already, and the algorithm will not enter the second doubling trick iteration. Thus the total number of samples for the first sub-phase is

In the case of L k > 1 , for ℓ ∈ { 1 , · · · , L k } , denote ̂ Γ ℓ as the estimate of Γ at the end of the ℓ -th doubling trick iteration. With these notations, we have

At the end of L k -th doubling trick iteration, the stopping condition is satisfied, i.e.,

We short ˜ log ( N k, 0 ,ℓ + N k, 1 ,ℓ , δ k,ℓ ) = ˜ log k,ℓ .

At the end of ( L k -1) -th doubling trick iteration, the stopping condition is not satisfied, i.e.,

Denote ξ L k as the empirical distribution of Z k, 0 ,L k ∪ Z k, 1 ,L k . Then above two conditions imply that the number of samples for L k -th and ( L k -1) -th doubling trick iterations respectively satisfy

and

Note that by Property 2 of Lemma J.1,

Thus

For any ℓ ∈ { 1 , . . . , L k } , by Lemma J.7, the factor of max w,w ′ ∈W k ∥ ∥ w -w ′ ∥ ∥ 2 A ( ξ ℓ , ̂ Γ ℓ ) -1 can be upper bounded by

We will upper bound the three terms in the RHS of (29) separately.

For the first term , by Lemma J.6,

where ˜ λ k is the optimal design for W k in the first sub-phase of phase k -1 (based on the last doubling trick iteration), i.e.,

Then for any λ ,

where for ( b 1 ) , we plug in λ ′ k := α ∗ k λ E +(1 -α ∗ k ) λ ∗ k , with α ∗ k ≤ 1 / 2 will be defined later and λ ∗ k is the optimal design for W k given Γ , i.e.,

( b 2 ) is due to the fact that α ∗ k ≤ 1 / 2 . For the second term in the RHS of (31), given the condition (23), i.e.,

where for ( b 1 ) , we use the condition (22) on N k -1 , 0 ,L k -1 , and ( b 2 ) is due to Lemma J.5. Plug (32) into (31), we have

Plug (33) into (30), we have the first term in the RHS of (29) can be upper bounded by

where for the last inequality we use the fact that the stopping condition is satisfied at the end of ( L k -1 ) -th doubling trick iteration for phase k -1 per (26).

For the second term in the RHS of (29), by Lemma J.10, when the condition (23) is satisfied, i.e., when

we have

where the last inequality is due to Lemma J.5.

For the third term in the RHS of (29), by Lemma J.11, when the condition (23) is satisfied, i.e., when

we have

where ( b 1 ) is due to (26), ( b 2 ) is due to (22). Plug (34), (35) and (36) into (29), we have

With ℓ = L k -1 and the fact that g > 24 whose exact value will be set later, we can rearrange (37) as

Note that the LHS of above can be lower bounded by (28),

Rearrange the terms in (39) and (38) and setting g to be larger enough and using the fact that ζ k = ζ k -1 / 2 , we have

Note that by definition ˜ log k,L k -1 < ˜ log k,L k , thus

Denote D k := N k, 0 ,L k + N k, 1 ,L k and divide both sides of above by ˜ log k,L k , we have

In the case of L k = 1 , by Property 3 of Lemma J.1, we have

Thereby we have

Taking a summation over k on both sides of (40), we have

Thus by setting g = 72 × 2 and rearranging the terms, we have

For D 1 , which corresponds to the first sub-phase where we use E-optimal design with doubling trick, we have the stopping condition,

This implies that when the stopping condition is met

and

Since we use E-optimal design for the first phase, the factor of ∥ ∥ w -w ′ ∥ ∥ 2 A ( ξ L 1 -1 , ̂ Γ L 1 -1 ) -1 can be upper bounded by

Plug (48) into (47), and use the fact that 2 N 1 , 0 ,L 1 -1 = N 1 , 0 ,L 1 , ζ 1 = 1 , we have N 1 , 0 ,L 1 ≤ 24 ∥ θ ∥ 2 2 L η ˜ log 1 ,L 1 -1 max w,w ′ ∈W 1 ∥ ∥ w -w ′ ∥ ∥ 2 A ( λ E , Γ) -1 ≤ 24 ∥ θ ∥ 2 2 L η ˜ log 1 ,L 1 max w,w ′ ∈W 1 ∥ ∥ w -w ′ ∥ ∥ 2 A ( λ E , Γ) -1 . (49)

Thus we have,

By the same calculation as (17) and (19), we have

where c is an absolute constant. Next we lower bound the left hand side of (45). To do this, we first upper bound ˜ log k,L k as

where the inequality above uses the fact that L k is the index of the last doubling trick iteration for phase k , by the design of the doubling trick, we have L k ≤ log 2 ( D k d ) .

Denote ρ 3 := K ∗ M ln( dM ) , looking back at (44), we have

Sample complexity for second sub-phase

The design for the second sub-phase of phase k is based on ̂ Γ L k , the estimate of Γ at the end of the L k -th doubling trick iteration in the first sub-phase of phase k ,

Then for any λ , by Lemma J.7 we have

We plug in λ ′′ k := α ∗ k λ E +(1 -α ∗ k ) λ ∗ k , with α ∗ k ≤ 1 / 2 will be defined later and λ ∗ k is the optimal design for W k ,

where the last inequality is due to the fact that α ∗ k ≤ 1 / 2 . For the second term in the RHS of above, by Lemma J.11 we have,

where for ( b 1 ) , we have N k, 0 ,L k ≥ 4 gL η ˜ log k,L k σ min ( A ( λ E , Γ) ) , and ( b 2 ) is due to Lemma J.5. Plug (53) into (52), we have

where the last inequality is due to (26). According to the algorithm design, the number of samples in the second sub-phase of phase k is defined as

with ρ ( W k ) = min λ ∈ ∆( Z ) max w,w ′ ∈W k ∥ w -w ′ ∥ 2 A ( λ, ̂ Γ L k ) -1 . Then we have, by setting g ≥ 4 ,

where ' ⪅ ' hides logarithmic factors of |W| for the second term and constants for simplicity. Plug (54) into the above inequality. Also note that by Lemma 2.1, we can always set L ν = 2( ∥ θ ∥ 2 2 L η +1) . Thus,

This essentially means that the sample complexity for the second sub-phase ∑ N k, 2 can be upper bounded by summation of the sample complexity we pay for Algorithm 1 and the sample complexity of the first sub-phase ∑ K ∗ k =1 ( N k, 0 ,L k + N k, 1 ,L k ) . Combine (51) and (55), we conclude the result.

Lemma J.3. Denote

A sufficient condition for to hold is

Proof. Given

By Lemma J.4, for the formula we have

Thus a sufficient condition for the inequality to hold is

Lemma J.4. Let X ≥ 1 , A, B ≥ 0 , then a sufficient condition for X ≥ A ln ( D + BX ) + C is X ≥ 2 A ln( AD + AB ) + 2 C.

Proof. The proof is motivated by Gales et al. [16]. Let f ∈ (0 , 1) , then

Set f = 1 / 2 and by the fact X ≥ 1 , we have

Lemma J.5. Suppose that we have a data set { Z T , X T } . Denote the empirical distribution of Z T as ξ . The number of samples satisfies

̂ Γ is the OLS estimate of Γ based on { Z T , X T } . Then

where the last inequality is due to Lemma J.10 with g = 2 . Rearranging the terms, we have

Lemma J.6. Suppose that we use ROUND to sample N 0 arms according to λ E denoted as Z 0 , and N 1 arms according to λ 1 denoted as Z 1 , with N 1 ≥ N 0 . Denote the empirical distribution of all the collected samples as ξ , then

Proof. Denote the empirical distribution of Z 0 as ξ 0 , and the empirical distribution of Z 1 as ξ 1 , then we have

The result follows by noting that

Lemma J.7. Suppose that we have an estimate ̂ Γ that is invertible, for any x ∈ R d and covariance matrix V , we have

Proof. Suppose we have an estimate ̂ Γ . Then,

Note that

Thus,

This implies that

Lemma J.8. Suppose λ f = arg min f ( λ ) and λ g = arg min g ( λ ) and f ( λ ) ≤ g ( λ ) + h ( λ ) , then f ( λ f ) ≤ g ( λ g ) + h ( λ g ) .

Proof.

and

Lemma J.9. Define λ ∗ z and ˜ λ z

as the optimal design regarding Γ and that regarding its estimate ̂ Γ respectively. Then, we have

Proof. By Lemma J.7, for any w,w ′ ∈ W and λ z ∈ ∆ Z ,

Thus

Lemma J.10. Suppose that we have ̂ Γ that is an OLS estimate from an offline dataset { Z T 1 , X T 1 } collected non-adaptively through a fixed design λ and the efficient rounding procedure ROUND. Let V = Z ⊤ T 1 Z T 1 . Then, for any x ∈ R d and g ≥ 1 , we have, with probability 1 -δ ,

where p is the cardinality of support of λ .

Proof. We first show that

We can upper bound the the term ∥ ∥ ∥ ∥ V -1 / 2 Γ -⊤ ( V -1 Z ⊤ T 1 S ) ⊤ ( Γ + V -1 Z ⊤ T 1 S ) -⊤ x ∥ ∥ ∥ ∥ 2 2 =: V by noticing that it appears in both of a 1 , a 2 above. Thus we have the inequality

By rearranging the terms, we have

By Lemma G.4, with probability 1 -δ , we have

Thus, with probability 1 -δ , we have

when

To further upper bound V , we first find a sufficient condition on T 1 such that ∥ ∥ Γ -1 V -1 Γ -⊤ ∥ ∥ op σ 2 η log( Z T 1 , δ ) ≤ 1 2 g , g ≥ 1 . By Lemma J.14, when T 1 ≥ 2 p ,

To upper bound the right hand side by 1 2 g , we need

With this condition (58), we have with probability 1 -δ ,

Lemma J.11. Suppose that we have ̂ Γ that is an OLS estimate from an offline dataset { Z T 1 , X T 1 } collected non-adaptively through a fixed design λ and the efficient rounding procedure ROUND. Let ˙ V be any positive definite matrix. Then, for any x ∈ R d , we have, with probability 1 -δ , when

where p is the cardinality of support of λ .

Proof.

Given the condition (58) holds, we have with probability 1 -δ ,

where ( a 1 ) is due to (59) and setting g = 1 .

Lemma J.12. For a least square estimate ̂ Γ that is estimated through a design matrix Z and ̂ Γ is invertible, we have

Proof. Since ̂ Γ is a least square estimator, we have

Lemma J.13. For two invertible matrixes A,B ∈ R d × d , we have

Proof. We have

Lemma J.14. Suppose that we have a design matrix Z T that is sampled from a distribution λ ∈ ∆ Z , with the efficient rounding procedure ROUND. Let p represent the cardinality of support of λ . We have, if T ≥ 2 p ,

where σ min ( · ) is the smallest singular value of a matrix.

Proof. Suppose that each arm z ∈ Z is sampled t z times, the empirical distribution of Z T is ξ := ( t z T ) z ∈Z . Thus, we have

By Fiez et al. [15, Proposition 2], we have

where α ∈ [ T T +2 p , 1] when T ≥ 2 p . Given the fact that both of ∑ z ∈Z ξ z Γ ⊤ zz ⊤ Γ and ∑ z ∈Z λ z Γ ⊤ zz ⊤ Γ are positive definite, we have σ min ( ∑ z ∈Z ξ z Γ ⊤ zz ⊤ Γ ) ≥ ασ min ( ∑ z ∈Z λ z Γ ⊤ zz ⊤ Γ ) . Thus, we have

When T ≥ 2 p , we have α ≥ 1 / 2 , which implies the result.

K Estimating λ min (Γ)

In this section, we introduce a simple adaptive procedure that finds a high probability lower bound on γ ∗ min := λ min (Γ) that is sufficiently accurate (i.e., within a constant factor of γ ∗ min ). For simplicity, we assume ∥ z t ∥ ≤ 1 , ∀ t in this section.

Our algorithm leverages confidence bounds to adaptively determine how many samples we like to take. Let ˆ Γ t := ( Z ⊤ Z ) -1 Z ⊤ X be the least square estimate of Γ after sampling t times to obtain { ( z s , x s ) } t s =1 where Z ∈ R t × d and X ∈ R t × d are the design matrices. Let V t := ∑ t s =1 z s z ⊤ s . We define the lower and upper confidence bound for γ ∗ min as follows:

where

and LCB (0) := -∞ and UCB (0) := ∞ .

The following lemma shows that LCB ( t ) and UCB ( t ) form a valid anytime confidence bound for γ ∗ min .

Lemma K.1. (Correctness of the confidence bounds)

Equipped with the confidence bounds, we are now ready to describe our algorithm for learning γ ∗ min (see Algorithm 10). Since the tightness of the confidence bounds depends on the smallest eigenvalue of V t , it is natural to use the E-optimal design as defined in Section 3.2. Recall that the solution of the E-optimal design is λ ∗ E and κ 0 is the smallest singular value achieved by λ ∗ E . We take in a rounding procedure for the E-optimal design ROUND E ( λ, t ) that takes in t samples and design λ and outputs integer sample count assignments { N z } z ∈Z so that if we sample according to these counts then we have

After determining the base sample counts { m z } z ∈Z by ROUND ( λ ∗ E , ⌈ r ( ω ) ⌉ , ω ) , we start doubling the sample size until we satisfy the condition LCB ( t ) > 1 2 UCB ( t ) . Note that the sampling scheme in the while loop is designed such that the total number of samples collected up to (and including) j -th iteration is 2 j -1 ⌈ r ( ω ) ⌉ . Once the loop stops, we return LCB ( t ) as the claimed lower approximation of the γ ∗ min .

Let N w be the total number of samples we used in Algorithm 10. Then, the next theorem shows that the estimate returned by our algorithm is both a valid lower bound to γ ∗ min and sufficiently accurate.

Algorithm 10 Learning λ min (Γ)

Input: Arm set Z , rounding procedure ROUND E for the E-optimal design, rounding accuracy ω Initialize: j = 1 , t = 0 .

Compute the E-optimal design λ ∗ E for Z .

while LCB ( t ) ≤ 1 2 UCB ( t ) do

Compute { m z } z ∈Z by ROUND E ( λ ∗ E , ⌈ r ( ω ) ⌉ ) .

For each arm z ∈ Z , sample 2 j -1 m z -1 { j = 1 } 2 j -2 m z times.

̸

Estimate ˆ Γ t using all samples collected so far (total t data points)

j

j

+1

.

end while

Output : LCB ( t )

Theorem K.2. (Correctness of Algorithm 10) The total number of samples denoted by N w used in Algorithm 10 satisfies that, with probability at least 1 -δ ,

We next analyze the sample complexity of the algorithm, which essentially shows the scaling of 1 ( γ ∗ min ) 2 κ 0 even if the algorithm does not need knowledge of γ ∗ min .

Theorem K.3. (Sample complexity of Algorithm 10) Then, with ω = 1 , we have, with probability at least 1 -δ ,

We remark that Allen-Zhu et al. [2] provides a rounding procedure with r ( ε ) = O ( d/ε 2 ) .

Proof of Lemma K.1 . Note that ˆ Γ = Γ + V -1 t Z ⊤ S where Z, S ∈ R t × d is the design matrices with s -th row being z ⊤ s and η ⊤ s respectively. Using Lemma G.4, we have, with probability at least 1 -δ ,

The well-known Weyl's theorem implies that max k | λ k ( ˆ Γ t ) -λ k (Γ) | ≤ ∥ ˆ Γ t -Γ ∥ op where λ k ( A ) is the k -th largest singular value of the matrix A . Choosing k = d and combining it with the display above conclude the proof.

Proof of Theorem K.2 . We assume ∀ t ≥ 1 , LCB ( t ) ≤ γ ∗ min ≤ UCB ( t ) , which happens with probability at least 1 -δ . Then, it is trivial to see that LCB ( N w ) ≤ γ ∗ min .

For the other inequality, we use the fact that the stopping condition was satisfied with N w :

This concludes the proof.

Proof of Theorem K.3 . We assume ∀ t ≥ 1 , LCB ( t ) ≤ λ min (Γ) ≤ UCB ( t ) , which happens with probability at least 1 -δ . In this proof, we let ˆ γ min := λ min ( ˆ Γ N w ) and γ ∗ min := λ min (Γ) for brevity.

If N w = ⌈ r (1) ⌉ , then there is nothing to prove. Otherwise, the loop in the algorithm was iterated more than once. Then, since the stopping condition was satisfied with N w , we have that in the previous iteration where t = N w / 2 the stopping condition was not satisfied. Thus,

Using the following two inequalities:

we have

On the other hand, with the rounding procedure, we have

since ω = 1 . Using this and the fact that N w ≥ d , it is easy to see that there exists an absolute constant c 1 such that

Then, there exists an absolute constant c 2 such that

We have N w on both sides. We invoke Lemma K.5 with r = 1 + N w to obtain

Lemma K.4. Let A, Γ ∈ R d × d where A is symmetric positive semi-definite. Then,

Proof.

Taking the inverse on both sides concludes the proof.

K.1 Proof of Lemma 3.2

Define Λ = ∑ d i =1 λ i e i e ⊤ i , i.e. the diagonal matrix with λ on the diagonal. Note that

So for an upper bound, we consider λ = 1 /d 1 ,

The result about ρ ∗ follows immediately.

When Γ = (1 -ε ) /d 11 ⊤ + εI , a computation using Sherman-Morrison shows that Γ -1 = 1 /ε [ I -(1 -ε ) /d 11 ⊤ ] . Thus

where the last line follows from the Kiefer-Wolfowitz Theorem [26].

K.2 lemma for solving x less than ln(x)

Lemma K.5. Let a, b, c, d > 0 . Then, for every r ,

Proof. If dr ≤ c , then

If dr > c , then,

Either case, we have

Lemma K.6. Let α, β > 0 . Then, for any r ,

Proof.

If r ≤ 2 α , then there is nothing to prove. If r > 2 α , then r ≤ 2 α ln(1 + α ) + 2 β . Either case, we have r ≤ 2 α ln( e + α ) + 2 β .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and introduction summarize our main theorems.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: For example, we explain our limitations on lower bound.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are given is the beginning of the paper and referred later when necessary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Details on the experiment setup are given. It is mainly a theoretical paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: It is mainly a theoretical paper. We attach the code used for the simulation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: It is mainly a theoretical paper with some simulations in appendix. No deep learning or other complicated training involved. But we attach our code for reference.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bar is reported, e.g., Fig 2b.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: It is mainly a theoretical paper. Some simulations are given in the appendix. No need for heavy computation resource.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper conforms all the code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: we discuss it in the appendix. It is mainly a theoretical and algorithmic paper, without any sensitive data, very unlikely to have negative impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks, no such data or model involved.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use any existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not produce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not use anything like crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not use anything like crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This article studies the pure exploration transductive linear bandit problem in the presence of instrumental variables. The authors assume a linear structural equation model on the instrument, treatment and outcome. The proposed method attempts to estimate the parameters in the structural model while optimally designing to learn the best arm (i.e. treatment).  The paper is mainly theoretical, but some simulations are provided to demonstrate the method.

STRENGTHS:
1. The combination of instrument variables and pure exploration appears novel.
2. The proposed method is guided by finite-time confidence bounds for two-stage least squares. The lower and upper bounds of sample complexity  are provided in the paper.
3. The method outperforms the standard methods UCB-OLS and UCB-IV.

WEAKNESSES:
1. Section 3 is very dense. The authors can consider shortening the content before Section 2.2 and use more room to explain Section 3.
2. No real-data demonstration of the proposed method.

QUESTIONS:
N.A.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The paper introduces the confounded pure exploration transductive linear bandit (CPET-LB) problem, which addresses the challenges of conducting adaptive experimentation in environments where direct randomization is not possible. From my understanding, the paper studies the best arm identification problem under linear structural model with non-compliance. The paper proposes the algorithms with nearly optimal sample complexity guarantee.

STRENGTHS:
1.	The paper provides a thorough theoretical analysis, including proofs of finite-time confidence intervals for the estimators and sample complexity bounds. The theoretical contributions are significant and appear to be solid.
2.	The problem is practical-relevant and important.

WEAKNESSES:
1.	The connections with two streams of the literature should be more clearly stated. The basic problem set up is very classical econometric setting where the non-compliance exists. The analysis framework and some of the tools are very standard in transductive linear bandit problem. 
2.	The presentation of Section 1 is not easy to follow. I feel the authors try to manage the terminology from causal inference and pure exploration. For example, in Line 78, “measurement” and ”evaluation” are new terminology of the paper and a bit hard to connect with the example proposed in introduction. 
3.	The authors might want to reconsider the title. The title, from my perspective, is a bit confusing and misleading.

QUESTIONS:
1.	Could you please elaborate a bit more on why the confidence interval in Section 2.2 is novel? I know the traditional 2SLS always using asymptotic normality to construct CI. Is non-asymptotic or asymptotic the key difference?

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper addresses the issue of adaptive experimentation using "encouragement" rather than "compulsion instruction," a scenario commonly encountered in industrial applications. The proposed solution integrates linear bandit algorithms with instrumental variables regressions. The authors provide rigorous theoretical guarantees for their method and demonstrate its superior performance compared to traditional A/B testing and conventional linear bandit algorithms.

STRENGTHS:
The scenarios examined by the authors are well-motivated. In practice, numerous situations exist where only encouragement can be employed to influence user decisions. Considering the shift in industry from traditional A/B testing to adaptive experimentation, the study presented here has the potential for significant industry impact.

In addition, the proposed algorithm is intuitive, and the theoretical guarantees provided are robust.

WEAKNESSES:
The authors could enhance their study by conducting additional experiments to verify the robustness of their results.

Furthermore, there should be a more detailed discussion of the p-values and confidence intervals to strengthen the statistical analysis.

QUESTIONS:
How to think of p-values/confidence intervals as it is important in an experimentation setting.

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of pure exploration bandits in the setting of encouragement designs. The authors describe the problem in terms of online instrumental variable regression. Toward this end, the authors derive a finite time confidence interval. Using this as the main tool, the authors then describe a pure exploration transductive bandit. Algorithms are provided for optimizing the pure exploration problem. A number of theoretical results are provided describing the entailed sample complexity of the proposed approach. Empirical results are provided which show strong performance against other baselines.\n\nSTRENGTHS:\nThis is a problem that has practical relevance in both industrial and social scientific settings. The task, to my knowledge, is novel in its formulation, and the authors do a nice job of delineating this work from prior art. The authors do a nice job of presenting algorithms and analysis in both the settings of a known and unknown structural models. Further, there is thorough analysis of each of the proposed procedures' properties.\n\nWEAKNESSES:\nMy concerns are largely around two things: \n1. There is a fairly limited experimental evaluation here. It would be helpful if the authors provided a more thorough evaluation of the proposed approaches' behavior across a wider range of settings. \n\n2. The text is a little meandering at times and as a result was a little hard to follow on first read. I would suggest a round of editing in order to improve the narrative and organization of the paper.\n\nQUESTIONS:\nThroughout an E-optimal design is also employed. My main question is whether the choice of E-optimality is done as a matter of convenience, or if it is motivated by aspects of the problem that would favor this criterion over other choices.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}