SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our assumption that  $T^{\alpha} \geq 4$  aligns with our intuition that a minimum of two samples per arm is required to identify a linear structure. Theorem 1 demonstrates that the bound  $T^{-\frac{\alpha}{2}}$  decreases as  $\alpha$  increases, indicating that a longer exploration period leads to better estimation of the dynamic treatment effect. When  $\alpha$  is set to 1, we conduct pure exploration throughout the entire experimental horizon, similar to the well-known switchback experiments [21]. In this case, the estimation error can be controlled to the order of  $\mathcal{O}(1 / \sqrt{T})$ , which aligns with traditional statistical results of linear regression [79]. Moreover, our analysis can be extended to any  $l_{p}$ -norm, with only slight variations
Algorithm 1: Linear Exploration and OFU (L-E0FU)
1 Input: Controlling parameter  $\alpha$ , instance parameters:  $\sigma_{a}$  for all  $a \in \{1,2\}, T, S, \delta$   
2 Initialization:  $\lambda = 1$ ,  $\mathcal{D}_{1,0} = \mathcal{D}_{2,0} = \mathcal{D}_{1,T} = \mathcal{D}_{1,T} = \emptyset$ ,  $\phi(\cdot) = (1,\cdot)^{\top}$   
3 for  $t = 1,2,\dots, [T^{\alpha}]$  do // Phase 1: Exploration  
4 Select  $A_{t} = 1$  if  $t$  is odd; otherwise select  $A_{t} = 2$   
5 Observe reward  $r_t$ ;  
6  $\mathcal{D}_{A_t,0} \gets \mathcal{D}_{A_t,0} \cup \{(\phi(t), r_t)\}$  and  $\mathcal{D}_{A_t,T} \gets \mathcal{D}_{A_t,T} \cup \{(\phi(t), r_t)\}$ ;  
7 end for  
8 for  $t = [T^{\alpha}] + 1$ ,  $[T^{\alpha}] + 2,\dots, T$  do // Phase 2: OFU  
9  $V_{a,t} = \sum_{(\phi(i), r_i) \in \mathcal{D}_{a,T}} \phi(i) \phi(i)^{\top} + \lambda I$  for  $a \in \{1,2\}$ ;  
10  $\gamma_{a,t} = \sigma_a \sqrt{2 \log(\frac{2 \det(V_{a,t})^{1/2} \det(\lambda I)^{-1/2}}{\delta})} + \lambda^{1/2} S$  for  $a \in \{1,2\}$ ;  
11  $\tilde{\theta}_{a,t} = V_{a,t}^{-1} \sum_{(\phi(i), r_i) \in \mathcal{D}_{a,T}} r_i \phi(i)$  for  $a \in \{1,2\}$ ;  
12  $\tilde{R}_t(a) = \tilde{\theta}_{a,t}^\top \phi(t) + \gamma_{a,t} \| \phi(t) \|_{V_{a,t}^{-1}}$  for  $a \in \{1,2\}$ ;  
13 Select  $A_t = \arg \max_{a \in \{1,2\}} \tilde{R}_t(a)$ ;  
14 Observe reward  $r_t$ ;  
15  $\mathcal{D}_{A_t,T} \gets \mathcal{D}_{A_t,T} \cup \{(\phi(t), r_t)\}$ ;  
16 end for  
17  $\hat{\theta}_{a,T}^{\text{L - EOFU}(\alpha)} = (\sum_{(\phi(i), r_i) \in \mathcal{D}_{a,0}} \phi(i) \phi(i)^{\top})^{-1} \sum_{(\phi(i), r_i) \in \mathcal{D}_{a,0}} r_i \phi(i)$  for  $a \in \{1,2\}$ ;  
18 Output:  $\hat{\theta}_T^{\text{L - EOFU}(\alpha)} = \hat{\theta}_{1,T}^{\text{L - EOFU}(\alpha)} - \hat{\theta}_{2,T}^{\text{L - EOFU}(\alpha)}$ ;
in the constant  $c_{1}$ . Additionally, note that the high probability bound can be used for constructing confidence intervals and conduct hypothesis testing.
The second phase of our L-E0FU follows the OFU principle. Specifically, at each time period  $t$ , L-E0FU calculates an optimistic estimate of the rewards for each arm, denoted as  $\tilde{R}_t(a)$  in line 12. This estimate is referred to as optimistic because, with probability at least  $1 - \delta$  for all  $t \in [T]$  and  $a \in \{1,2\}$ ,  $\tilde{R}_t(a) = \tilde{\theta}_{a,t}^\top \phi(t) + \gamma_{a,t} \| \phi(t) \|_{V_{a,t}^{-1}} \geq \theta_a^\top \phi(t) = \mathbb{E}[R_t(a)]$ , where  $\gamma_{a,t} \leq \sigma_a (\sqrt{12 \log(T) - 2 \log(4.5 \lambda \delta)} + \lambda^{1/2} S)$  and by default,  $\lambda$  is set to be 1. The proof of this claim follows the standard techniques outlined in Theorem 2 of [2]. Based on the optimistic estimation, L-E0FU selects the arm with the highest optimistic estimated value. The OFU principle has been shown to strike a delicate bal
Figure 2: An illustrative example on OFU.
ance between exploration and exploitation in the bandit literature [2][55]. As mentioned earlier, one of the challenges in our problem is the possibility of the optimal arm changing over time. The OFU principle has an additional advantage in that it can automatically and implicitly detect when the optimal arm switches from one to another. To illustrate this, Figure 2 presents an example of how OFU behaves when the optimal arm switches. Initially (before  $t_3$ ), arm 1 outperforms arm 2. According to L-E0FU, we continue to play arm 1, resulting in a relatively fast shrinkage of the confidence interval for the estimated values of arm 1. Conversely, for arm 2, the confidence interval even expands as  $\phi(t)$  increases in magnitude. As time approaches  $t_3$ , the difference between arm 1 and arm 2 gradually diminishes. At time period  $t_3$ , the optimistic estimation of arm 2 surpasses that of arm 1, prompting L-E0FU to start playing arm 2. This addresses the challenge mentioned earlier. The following theorem presents the regret upper bound.
Theorem 2 For any  $\nu \in \mathcal{E}_1$  and  $0 < \alpha \leq 1$ , with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathcal {R} _ {\nu} ^ {L - E O F U (\alpha)} (T) \leq 4 S T ^ {2 \alpha} + 4 (\sigma_ {1} \vee \sigma_ {2}) (\sqrt {1 2 \log (T) - 2 \log (4 . 5 \lambda \delta)} + \lambda^ {1 / 2} S) \sqrt {\frac {4 ^ {1 + \frac {2}{\alpha}}}{\ln 2} \ln (2 T) T}, \\ \text {w h e r e} \lambda \text {i s i n i t i a l i z e d a s 1 . F u r t h e r m o r e ,} \delta = 1 / T \text {l e a d s t o} \mathbb {E} [ \mathcal {R} _ {\nu} ^ {L - E O F U (\alpha)} (T) ] = \widetilde {\mathcal {O}} (T ^ {2 \alpha} \vee \sqrt {T}). \end{array}
$$
Equation (2) can be decomposed into two terms: a  $T^{2\alpha}$  term and a  $\sqrt{T\ln(T)}$  term. The first term arises from the exploration stage, as  $\sum_{t=1}^{T^{\alpha}} t = \mathcal{O}(T^{2\alpha})$  intuitively. The second term is a result of the
OFU phase. Only when  $\alpha \geq \frac{1}{4}$  does the first term dominate the second term. In fact, there is little motivation for a decision maker to choose  $\alpha < \frac{1}{4}$ , as we will discuss in detail in the next subsection. For pure theoretical interest, the regret behavior in the region around  $\alpha = 0$  could be studied by choosing  $\alpha = 1 / \ln \ln (T)$ , given that there is an  $\alpha$  in the denominator. For those familiar with the bandit literature, a similar  $\mathcal{O}(\sqrt{T\ln(T)})$  bound can be obtained in the stochastic contextual bandit setting [2]. As we mentioned before, one possible approach to reduce our problem to the one in [2] is by treating  $\phi (t)$  as a special kind of context. However, a key difference is that the context in [2] is assumed to be bounded by a constant, whereas in our setting, the bound should be  $\sqrt{1 + T^2}$  if we know  $T$  in advance. Consequently, to apply all the techniques from [2],  $\lambda$  in Equation (2) should be chosen as  $\Theta (T)$  instead of 1. However, such a choice of  $\lambda$  would yield a regret bound of  $\mathcal{O}(T)$ , which is significantly weaker than the  $\mathcal{O}(\sqrt{T})$  regret we derived. Furthermore, in the stochastic contextual bandit setting, where everything is assumed to be bounded, the regret can always be upper bounded by  $\mathcal{O}(T)$  for any policy, and thus only sublinear regrets are meaningful. However, here, with the linear trend, the largest possible regret for a policy without careful design can be on the order of  $\mathcal{O}(T^2)$ . Thus, the  $\mathcal{O}(T^{2\alpha})$  regret bound is still informative even if  $1 / 2 < \alpha < 1$ .
Another important point to emphasize is that although the regret can be decomposed into two terms, it does not mean that we can analyze each phase independently of the other. The  $T^{\alpha}$  samples we collect from the first stage play a crucial role in our analysis for the second phase. This is evidenced by the fact that  $\alpha$  not only appears in the first term, but also takes effect in the second term by a non-trivial way. Specifically, the regret of the second phase can be upper bounded by the sum of the length of the confidence interval at each time, i.e., the regret at time  $t$  can be bounded by  $2\gamma \| \phi (t)\|_{V_{1,t}^{-1}}\mathbb{I}_{A_t = 1} + 2\gamma \| \phi (t)\|_{V_{2,t}^{-1}}\mathbb{I}_{A_t = 2}$ , where  $V_{a,t}$  is defined to be  $\sum_{i = 1}^{t}\phi (i)\phi (i)^{\top}\mathbb{I}_{A_i = a} + \lambda I$ . The  $T^{\alpha}$  samples collected in the first stage are efficient in controlling the minimum eigenvalue of  $V_{a,t}$ , and thus be useful in controlling the magnitude of the regret.
In addition, we would like to compare our two-stage design with the traditional exploration-then-exploitation design [55]. The similarity between the two lies in the fact that the first exploration phase provides valuable information for future decision-making, as we discussed earlier. However, the key difference is that exploration-then-exploitation typically only exploits the information gathered during the first exploration phase and does not continue to learn new information during the second phase. In contrast, our L-EOFU algorithm continues to and has to learn throughout the entire learning period, since the environment may rapidly evolve over time. Moreover, the exploration phase in L-EOFU also plays an important role in conducting causal inference. Therefore, when designing the length of the first phase, we must consider not only the regret but also the quality of causal inference. One reason why we move away from the exploration-then-exploitation method is that it may not always guarantee the optimal regret rate. Another important motivation for conducting pure exploration first is that the cost of exploration may increase linearly with time.
2.3 Optimality of The Design
In this subsection, we will examine the optimality of our L-EOFU design from three different perspectives. Firstly, we will consider two extremes where the decision maker is solely concerned about either the estimation error of the dynamic treatment effect or the regret. Subsequently, when jointly considering the two objectives, we will introduce the concept of Pareto optimality, which describes a scenario where neither the statistical power of estimating the treatment effect nor the regret can be improved without compromising the other. First, we focus on the best achievable quality of the inference of  $\theta_{\nu}$ , regardless of the regret incurred during the experiment. The following lemma describes the intrinsic difficulty of conducting inference for the dynamic treatment effect.
Lemma 1 For any constant  $c_2 > 0$ ,  $\inf_{\hat{\theta}} \sup_{\nu \in \mathcal{E}_1} \mathbb{P}_{\nu}\left(\left\|\hat{\theta} - \theta_{\nu}\right\|_1 > \frac{2\sqrt{2}c_2\sigma_0}{\sqrt{T}}\right) \geq \frac{1}{2} - c_2$ . Moreover,  $\inf_{\hat{\theta}} \sup_{\nu \in \mathcal{E}_1} \mathbb{E}_{\nu}\left[\left\|\hat{\theta} - \theta_{\nu}\right\|_1\right] \geq \frac{\sqrt{2}\sigma_0}{8\sqrt{T}}$ .
This lemma establishes an information-theoretic lower bound on the best achievable quality of inference for  $\theta_{\nu}$ . It indicates that, for any estimator  $\hat{\theta}$ , there will always exist a challenging instance that results in the estimator being at least  $\Theta(1/\sqrt{T})$  away from the true value. Our proof idea is to reduce the two-dimensional estimation problem to a one-dimensional problem. Specifically, we consider a subset of all the problem instances  $\mathcal{E}_0 := \{\nu \in \mathcal{E}_1 : \theta_{a,1} = 0, \forall a \in \{1,2\}\}$ , which includes all the stationary instances (i.e., without time trends), and this provides a natural lower
bound that  $\inf_{\hat{\theta}}\sup_{\nu \in \mathcal{E}_1}\mathbb{P}_{\nu}\left(\left\| \hat{\theta} -\theta_{\nu}\right\| _1 > \frac{2\sqrt{2}c_2\sigma_0}{\sqrt{T}}\right)\geq \inf_{\hat{\theta}}\sup_{\nu \in \mathcal{E}_0}\mathbb{P}_{\nu}\left(\left\| \hat{\theta} -\theta_{\nu}\right\| _1 > \frac{2\sqrt{2}c_2\sigma_0}{\sqrt{T}}\right)$ . For the instance class  $\mathcal{E}_0$ , the problem is reduced to estimating only the intercept terms. By choosing  $\alpha = 1$ , Theorem 1 indicates  $\| \hat{\theta}_T^{\mathrm{L - E0FU(1)}} - \theta_\nu \| _1$  can be upper bounded by  $\mathcal{O}(1 / \sqrt{T})$ , which matches with the lower bound we derive in Lemma 1 in terms of the dependence of the order of  $T$ . Therefore, we can state the following theorem.
Theorem 3 (Optimal Error Rate Design)  $\hat{\theta}_T^{L - E0FU(1)}$  achieves the optimal  $1 / \sqrt{T}$  error rate, which can not be improved up to a constant factor. Moreover,  $\inf_{\hat{\theta}}\sup_{\nu \in \mathcal{E}_1}\mathbb{E}_{\nu}\left[\left\| \hat{\theta} -\theta_{\nu}\right\| _1\right] = \Theta (1 / \sqrt{T})$
Theorem 3 further establishes that the existence of  $\hat{\theta}_{T}^{\mathrm{L - E0FU(1)}}$  confirms the tightness of the  $1 / \sqrt{T}$  lower bound derived in Lemma 1. As mentioned earlier, setting  $\alpha = 1$  corresponds to the simple switchback experimental design. Theorem 3 implies that even in the presence of linear trends, switchback experiments can still maintain strong statistical power. However, there is no free lunch. Choosing  $\alpha = 1$  results in a regret upper bound of  $\mathcal{O}(T^2)$ , which can be highly undesirable if minimizing regret is also one of the objectives. Now, we turn to discuss about the other extreme that the regret minimization is the only objective. We present the minimax regret lower bound as follows.
Lemma 2 (Regret Lower Bound)  $\inf_{\pi} \sup_{\nu \in \mathcal{E}_1} \mathbb{E}\left[\mathcal{R}_{\nu}^{\pi}(T)\right] \geq \frac{\sigma_0 \sqrt{T}}{32e}$ .
This lemma describes that any admissible policy  $\pi$  will face some hard instance such that the expected regret could be no less than  $\Omega(\sqrt{T})$ . Similar as before, we can bridge our problem with the tradition stochastic bandit problem by noticing that  $\mathcal{E}_0 \subset \mathcal{E}_1$ . It is widely recognized that the stochastic bandit problem has a well-established minimax regret lower bound of  $\Omega(\sqrt{T})$  [55]. Intuitively, given that the traditional stochastic bandit problem is a special case within our framework, this lower bound naturally extends to our setting. Formally, by Theorem 2, we have the following theorem, which states that the  $\sqrt{T}$  lower bound is indeed possible to achieve and is optimal.
Theorem 4 (Optimal Regret Rate Design) For any  $0 < \alpha \leq \frac{1}{4}$ ,  $L\text{-} E0FU(\alpha)$  can guarantee a regret of  $\widetilde{\mathcal{O}} (\sqrt{T})$ , which can not be further improved up to logarithmic factors. Furthermore,  $\inf_{\pi}\sup_{\nu \in \mathcal{E}_1}\mathbb{E}\left[\mathcal{R}_{\nu}^{\pi}(T)\right] = \widetilde{\Theta} (\sqrt{T})$ .
Theorem 4 reveals that any  $\alpha \in (0, \frac{1}{4}]$  yields a homogeneous regret rate dependence on  $T$ . However, the estimation error, as shown in Theorem 1, decreases with increasing  $\alpha$ . Consequently, increasing  $\alpha$  from nearly 0 to  $\frac{1}{4}$  does not significantly increase the regret rate, which remains at  $\widetilde{\mathcal{O}}(\sqrt{T})$ . However, it significantly reduces the estimation error bound. Therefore, there is limited motivation for decision makers to select  $\alpha < \frac{1}{4}$ . Nevertheless, there is no free lunch. Even if we choose  $\alpha = \frac{1}{4}$ , the estimation error can only be bounded by  $\mathcal{O}(T^{-\frac{1}{8}})$ , which is much larger than the optimal bound of  $\frac{1}{\sqrt{T}}$ . While the optimal regret rate in this case aligns with that of the traditional stochastic bandit, the optimal design for linear trends faces additional challenges. Intuitively, a regret rate optimal design for linear trends needs to improve from a naive design's  $T^2$  regret to the  $\sqrt{T}$  order. In contrast, the regret rate optimal design for traditional multi-armed bandits only needs to improve from  $T$  regret to  $\sqrt{T}$ .
From the two cases discussed above, we have uncovered a fundamental trade-off between regret and the statistical power. The design that achieves the optimal regret rate suffers from a large estimation error, while the design that achieves the optimal error rate incurs a large regret. To address this trade-off, we now employ the concept of Pareto optimality from multi-objective optimization theory. This concept allows us to characterize the circumstances where neither regret nor estimation error of the treatment effect can be improved without worsening the other. In other words, we aim to statistically determine the type of optimality that can be achieved between these two extremes.
Formally, we first define the policy class  $\Pi_{\beta}$  as  $\Pi_{\beta} := \{\pi : \exists c_0 > 0, \forall \nu \in \mathcal{E}_1$  and  $T, \mathbb{E}[\mathcal{R}_{\nu}^{\pi}(T)] \leq c_0 T^{\beta}\}$ , where  $c_0$  in this context can depend on universal constants or logarithmic terms of  $T$ , but it cannot include any polynomial terms of  $T$ . The set  $\Pi_{\beta}$  encompasses all policies that can achieve a regret of  $\widetilde{\mathcal{O}}(T^{\beta})$ . It is evident that  $\mathrm{L - E0FU}(\alpha)$  belongs to  $\Pi_{\frac{1}{2} \vee (2\alpha)}$ . Our aim is to investigate the best achievable estimation error when the decision maker incurs a regret of at most  $\widetilde{\mathcal{O}}(T^{\beta})$ . The following lemma reveals the inherent difficulty of conducting inference under limited regret budget.
Lemma 3 For any  $\pi \in \Pi_{\beta}$ , there exists a constant  $c^{\pi}$  only dependent of numerical constants and logarithm terms of  $T$ , such that for any  $\xi > 0$ ,  $\inf_{\hat{\theta}} \max_{\nu \in \mathcal{E}_1} \mathbb{P}_{\nu}^{\pi}\left(\left\|\hat{\theta} - \theta_{\nu}\right\|_1 \geq c^{\pi} \sigma_0 \xi T^{-\frac{\beta}{4}}\right) \geq \frac{1}{2} - \xi$ . Moreover,  $\inf_{\hat{\theta}} \max_{\nu \in \mathcal{E}_1} \mathbb{E}_{\nu}^{\pi}\left[\left\|\hat{\theta} - \theta_{\nu}\right\|_1\right] \geq \frac{c^{\pi} \sigma_0}{16} T^{-\frac{\beta}{4}}$ .
Lemma 3 demonstrates that when employing an assignment policy  $\pi$  with a regret of  $\widetilde{\mathcal{O}}(T^{\beta})$ , any estimator, regardless of its sophistication, will encounter challenging instances where the estimation error is at least  $\widetilde{\Omega}(T^{-\frac{\beta}{4}})$ . This finding distinguishes our study, which incorporates linear trends, from the work of [74] that investigates multi-armed bandit experimental design without linear trends. In the absence of linear trends, [74] reveals that under an assignment policy  $\pi$  with a regret of  $\widetilde{\mathcal{O}}(T^{\beta})$ , the estimation error has a minimax lower bound of  $\widetilde{\Omega}(T^{-\frac{\beta}{2}})$ . The key distinction arises from the fact that linear trends can significantly increase the cost of exploration, resulting in higher regret during the exploration phase. Consequently, with the same regret budget, the information collected in our case may be substantially less than in the absence of linear trends. The proof follows this intuition by constructing two similar instances with the same optimal arm but slightly different suboptimal arms. The only way to obtain useful information to distinguish the two is by playing the suboptimal arm. Given a regret budget of  $T^{\beta}$ , at most approximately  $T^{\frac{\beta}{2}}$  samples can be obtained from the suboptimal arm, leading to a lower bound of  $T^{-\frac{\beta}{4}}$ .
Theorem 5 (Pareto Optimal Design) If  $\alpha \geq \frac{1}{4}$ ,  $L\text{-} E0FU(\alpha)\in \Pi_{2\alpha}$ , and thus, the  $T^{-\frac{\alpha}{2}}$  error rate in Theorem 1 can not be further improved in terms of the dependence on  $T$ . Furthermore, for a fixed  $\beta \geq \frac{1}{2}$ ,  $\inf_{\hat{\theta}_T,\pi \in \Pi_\beta}\max_{\nu \in \mathcal{E}_1}\mathbb{E}_\nu^\pi \left[\left\| \hat{\theta}_T - \theta_\nu \right\| _1\right] = \widetilde{\Theta}\left(T^{-\frac{\beta}{4}}\right)$ .
The theorem establishes that the regret bound of  $T^{-\frac{\alpha}{2}}$  achieved by L-E0FU(α) cannot be improved by a constant factor, given that the decision-making policy belongs to  $\Pi_{2\alpha}$ . This implies that L-E0FU(α) is Pareto optimal for any  $\alpha \geq \frac{1}{4}$ . The reasons are as follows: on the one hand, if one desires a better guarantee on the estimation error than  $T^{-\frac{\alpha}{2}}$ , the only option is to accept a higher regret rate by adopting a larger decision-making policy class  $\Pi_{2\alpha + \epsilon}$  for some  $\epsilon > 0$ . On the other hand, if a regret of  $\widetilde{\mathcal{O}}(T^{2\alpha - \epsilon})$  is aimed for, the estimation error will inevitably increase to the order of  $T^{-2\alpha + \epsilon}$ . In other words, improving either the statistical power of causal inference or the efficiency of online decision-making necessitates a degradation in the other. Intuitively, Theorem 5 captures a statistical Pareto frontier for the trade-off between conducting causal inference and minimizing regret, which can be expressed as Estimation Error  $= \Theta(\text{Regret}^{-\frac{1}{4}})$ . In Theorem 3, we reveal that  $\alpha = 1$  leads to the optimal design in terms of error rate. By combining Theorem 5, we further conclude that the large  $T^2$  regret incurred by  $\alpha = 1$  is necessary for any estimator achieving the optimal error rate of  $\frac{1}{\sqrt{T}}$ . Moreover, the  $T^{-\frac{1}{8}}$  error rate achieved by  $\alpha = \frac{1}{4}$ , although relatively large, is the best that any regret-optimal policy can achieve. Figure 3 illustrates the Pareto frontier under linear trends, alongside the Pareto frontier for standard multi-armed bandit experimental design studied in [74].
From Figure 3, we can observe the significant impact of the non-stationary structure on optimal experimental design. First, in standard bandit experiments, the maximum possible regret is of the order  $T$  since all parameters are assumed to be bounded. However, with the presence of linear trends, traditional designs such as switchback experiments are likely to incur regret of the order  $T^2$ . Furthermore, under the same regret budget, the best achievable estimation error bounds differ substantially. For instance, given assignment policies with regret of the order  $T$ , the
Figure 3: The Pareto Frontier.
optimal design for standard bandit experiments can attain an error bound of  $T^{-1/2}$ , whereas in the presence of linear trends, the best achievable bound is  $T^{-1/4}$ . Intuitively, this is because linear trends make gathering information from the suboptimal arm increasingly costly in terms of regret. Similarly,
when linear trends exist, the decision maker will always have to pay additional regret to obtain an estimator with the same level of precision. Additionally, there is a worst-case infeasible region as indicated by Lemma 2 that no policy can achieve a regret smaller than  $\sqrt{T}$  order in the worst case.