SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\operatorname {d e c} _ {\epsilon} ^ {a c} (f) \leq \min  _ {\lambda > 0} \left\{\lambda^ {\frac {1}{1 - \alpha}} \alpha^ {\frac {\alpha}{1 - \alpha}} (\alpha - 1) \Psi_ {\alpha} (f) ^ {\frac {1}{\alpha - 1}} + \lambda \epsilon^ {2} \right\} \tag {27}
$$
for  $\alpha > 1$ .
Proof. We start by noting that for  $x_{1}, \ldots, x_{\alpha} \geq 0$ , from AM-GM we have that  $\alpha (x_{1} \cdot x_{2} \cdots x_{\alpha})^{1 / \alpha} \leq x_{1} + \cdots + x_{\alpha}$ . Substituting  $x_{2} = x_{3} = \cdots x_{\alpha}$ , we get  $\alpha \cdot x_{1}^{\frac{1}{\alpha}} \cdot x_{2}^{\frac{\alpha - 1}{\alpha}} - x_{1} \leq (\alpha - 1)x_{2}$ . Writing  $x_{1} = \lambda \mu I_{f}\nu$  and  $x_{2} = \alpha^{\frac{\alpha}{1 - \alpha}}\left(\frac{(\mu\Delta\nu)^{\alpha}}{\lambda\mu I_{f}\nu}\right)^{\frac{1}{\alpha - 1}}$  and using the previous inequality with the  $\mathrm{dec}_{\epsilon}^{ac}$  program gives the result.
The information ratio  $\Psi_{\alpha ,f}(\mu ,\nu)$  can be thought of as the Bayesian information ratio in [Russo and Van Roy, 2014] where the expectation is taken over the distribution  $\nu$  of possible environments. However, for information gain  $I_{f}$ , [Russo and Van Roy, 2014] use entropy difference in the posterior distribution of  $\pi_f^*$  before and after the observation is revealed.
Proof of Lemma 5.
$$
\begin{array}{l} \mathsf {d e c} _ {\epsilon} ^ {a c, f} (\hat {f} _ {t}) = \min _ {\mu \in \mathcal {P} (\Pi)} \min _ {\lambda \geq 0} \max _ {b \in \Pi} \max _ {g \in \mathcal {M}} \langle \phi_ {b}, g \rangle - \langle \phi_ {\mu}, \hat {f} _ {t} \rangle - \lambda \| g - \hat {f} _ {t} \| _ {V (\mu)} ^ {2} + \lambda \epsilon^ {2} \\ = \min  _ {\lambda \geq 0} \max  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {b \in \Pi} \left\langle \phi_ {b} - \phi_ {\mu}, \hat {f} _ {t} \right\rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} + \lambda \epsilon^ {2}. \tag {28} \\ \end{array}
$$
The first equality is by definition, and the second equality follows from solving the quadratic maximization over  $g \in \mathcal{M} = \mathbb{R}^d$ . To show that the problem is convex in  $\mu$ , note that taking inverses of positive semi-definite matrices  $X, Y$  is a convex function, i.e.  $((1 - \eta)X + \eta Y)^{-1} \preceq (1 - \eta)X^{-1} + \eta Y^{-1}$ . In particular,  $V((1 - \eta)\mu_1 + \eta \mu_2)^{-1} \preceq (1 - \eta)V(\mu_1)^{-1} + \eta V(\mu_2)^{-1}$ . With this the claim follows.
D Convex Program for Fixed  $\lambda$
Take Eq. (28) and fix  $\lambda > 0$ . Then we have the following saddle-point problem:
$$
\begin{array}{l} \min _ {\mu \in \mathcal {P} (\Pi)} \max _ {b \in \Pi} \langle \phi_ {b} - \phi_ {\mu}, \hat {f} _ {t} \rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} + \lambda \epsilon^ {2} \\ = \lambda \epsilon^ {2} + \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {b \in \Pi} \langle \phi_ {b} - \phi_ {\mu}, \hat {f} _ {t} \rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} \\ \end{array}
$$
Up to the constant additive term, this saddle point problem is equivalent to the following convex program
$$
\min  _ {y \in \mathbb {R}, \mu \in \mathbb {R} ^ {\Pi}} y \quad \text {s . t .} \quad y \geq \left\langle \phi_ {b} - \phi_ {\mu}, \hat {f} _ {t} \right\rangle + \frac {1}{4 \lambda} \| \phi_ {b} \| _ {V (\mu) ^ {- 1}} ^ {2} \quad \forall b \in \Pi
$$
$$
\mathbf {1} \mu = 1
$$
$$
\mu_ {\pi} \geq 0 \quad \forall \pi
$$
All experiments below were run on a semi-bandit problem with a "revealing action", as alluded to in the paragraph below Lemma 4. Specifically, we assume a semi-bandit model where  $\mathcal{M} = \mathbb{R}^d$  and the features are  $\phi_{\pi}\in \mathbb{R}^{d}$ . For an instance  $f^{*}\in \mathcal{M}$ , the reward function is  $r_{f^{*}} = \langle \phi_{\pi},f^{*}\rangle$  for all  $\pi \in \Pi$ . There is one revealing (sub-optimal) action  $\hat{\pi}\neq \pi_{f^{*}}^{*}$ . The observation for any action  $\pi \neq \hat{\pi}$  is
$$
M _ {f ^ {*}} (\pi) = \mathcal {N} \left(\left\langle \phi_ {\pi}, f ^ {*} \right\rangle , 1\right) \tag {29}
$$
Define  $M_{\hat{\pi}} = [\phi_{\pi_1}, \dots, \phi_{\pi_{|\Pi|}}]^\top$ . Then the observation for action  $\hat{\pi}$  is
$$
M _ {f ^ {*}} (\hat {\pi}) = \mathcal {N} \left(M _ {\hat {\pi}} f ^ {*}, \mathbf {1} _ {d}\right) \tag {30}
$$
Thus, the information for any action  $\pi \neq \hat{\pi}$  is
$$
I _ {f} (g, \pi) = \frac {\sigma^ {2}}{2} \left\langle \phi_ {\pi}, g - f \right\rangle^ {2} \tag {31}
$$
while the information for action  $\hat{\pi}$  is
$$
I _ {f} (g, \hat {\pi}) = \frac {\sigma^ {2}}{2} \| M _ {\hat {\pi}} (g - f) \| ^ {2} = \frac {\sigma^ {2}}{2} \sum_ {\pi} \left\langle \phi_ {\pi}, g - f \right\rangle^ {2} \tag {32}
$$
For this setting  $\operatorname{Est}_n \leq \mathcal{O}(d\log(n))$  (see Appendix A.1).
E.1 Experimental Setup
Our main objective is to compare our algorithm ANYTIME-E2D to the fixed-horizon E2D algorithm by Foster et al. [2021]. ANYTIME-E2D and E2D were implemented by using the procedure described in Section 3.4. Both ANYTIME-E2D and E2D need to solve the inner convex problem in Lemma 5. To do so we use Frank-Wolfe [Frank and Wolfe, 1956, Dunn and Harshbarger, 1978, Jaggi, 2013] for 100 steps and warm-starting the optimization at the solution from the previous round,  $\mu_{t-1}$ . For ANYTIME-E2D we further perform a grid search over  $\lambda \in [0, \max_{g \in \mathcal{M}} \epsilon^{-2} \mathrm{dec}^{ac,f}(g)]$  (with a discretization of 50 points) to optimize over lambda within each iteration of Frank-Wolfe. For both the E2D and ANYTIME-E2D algorithm we used the version with the gaps  $\Delta$  replaced with  $\Delta_f$ , since we noticed that both algorithms performed better with  $\Delta_f$ . For E2D, the scale hyperparameter  $\lambda$  was set using  $\lambda = \sqrt{\frac{n}{4 \log(n)}}$  as mentioned in Foster et al. [2021, Section 6.1.1]. While for ANYTIME-E2D we set the hyper-parameter  $\epsilon_t^2 = d / t$ . Further, we compare to standard bandit algorithms: Upper Confidence Bound (UCB) and Thompson Sampling (TS) [Lattimore and Szepesvári, 2020a].
E.2 Experiment 1
In this experiment, we aim to demonstrate the advantage of having an anytime algorithm. Specifically, we tune  $\lambda$  in the E2D algorithm for different horizons  $n = 200, 500, 1000, 2000$ , but run it for a fixed horizon of  $n = 2000$ . As such, we expect our algorithm ANYTIME-E2D to perform better than E2D when  $\lambda$  was tuned for the incorrect horizons (i.e.  $n = 200, 500, 1000$ ). The feature dimension is  $d = 3$ . The number of decisions is  $|\Pi| = 10$ . We generated the features  $\phi_{\pi}$  for each  $\pi \in \Pi$  and parameter  $f^{*} \in \mathbb{R}^{d}$  randomly at the beginning and then kept them fixed throughout the experimentation. 100 independent runs were performed for each algorithm.
The results of the experiment can be seen as the left plot in Fig. 1. As expected, our algorithm ANYTIME-E2D performs better than E2D (for  $n = 200, 500, 1000$ ). This indicates that the E2D algorithm is sensitive to different settings of  $\lambda$ , which is problematic when the horizon is not known beforehand. Whereas our ANYTIME-E2D algorithm performs well even when the horizon is not known.
E.3 Experiment 2
In this experiment, we investigate the case when  $n < d^4$ . As pointed out below Lemma 3, we expect improvement in this regime as the regret bound of our algorithm is  $R_{n} \leq \min \{d\sqrt{n}, d^{1/3}n^{2/3}\}$ ,
Figure 1: Running ANYTIME-E2D, TS, UCB, and E2D optimized for different horizons  $n \in \{200,500,1000,2000\}$ . Left: The result for horizon  $n = 2000$ , and the feature space dimension  $d = 3$ . Right: The result for horizon  $n = 1000$ , and the feature space dimension  $d = 30$ .
while the default, fixed-horizon E2D algorithm cannot achieve these bounds simultaneously and one has to pick one of  $d\sqrt{n}$  or  $d^{1/3}n^{2/3}$  beforehand for setting the scale hyperparameter  $\lambda$ . It is standard that the choice of  $\lambda$  is made according to the  $d\sqrt{n}$  regret bound for E2D Foster et al. [2021](which is not optimal when  $n \ll d^4$ ), especially, if the horizon is not known beforehand. Thus, we set the horizon to  $n = 1000$  and the dimension of the feature space to  $d = 30$ , which gives us that  $n = 1000 \ll 810000 = d^4$ . The rest of the setup and parameters are the same as in the previous experiment except for the features  $\phi_{\pi}$  and  $f^*$  which are again chosen randomly in the beginning and then kept fixed throughout the experiment.
The results of the experiment can be seen as the right plot in Fig. 1. As expected, our algorithm ANYTIME-E2D performs better than E2D, UCB, and TS. This indicates that indeed, ANYTIME-E2D is likely setting  $\lambda$  appropriately to achieve the preferred  $d^{1/3}n^{2/3}$  regret rate for small horizons. The poor performance of the other algorithms can be justified, since E2D is optimized based on the worse  $d\sqrt{n}$  regret rate (for small horizons), while the UCB and TS algorithms are not known to get regret better than  $d\sqrt{n}$ .