SlowGuess/NeurIPS_2023_OCR huggingface dataset:
These simulations explore the empirical behaviour of Algorithm 1 and Algorithm 2 when instantiated with  $\epsilon(d) = 2^d$  and curved boundary oracles  $\Lambda$  and  $\Xi$ . To save space, precise details on the experiments as well additional figures are elided to Appendix F. Reference implementations which reproduce the figures are available at https://github.com/microsoft/csrobust.
4.1 The i.i.d. setting
These simulations exhibit our techniques on i.i.d. data. Although the i.i.d. setting does not fully exercise the technique, it is convenient for visualizing convergence to the unique true CDF. In this setting the DKW inequality applies, so to build intuition about our statistical efficiency, we compare our bounds with a naive time-uniform version of DKW resulting from a  $(6 / \pi^2 t^2)$  union bound over time.
Beta distribution In this case the data is smooth wrt the uniform distribution on  $[0,1]$  so we can directly apply Algorithm 1 and Algorithm 2. Figure 2 shows the bounds converging to the true CDF as  $t$  increases for an i.i.d.  $\mathrm{Beta}(6,3)$  realization. Figure 8 compares the bound width to time-uniform DKW at  $t = 10000$  for Beta distributions that are increasingly less smooth with respect to the uniform distribution. The DKW bound is identical for all, but our bound width increases as the smoothness decreases.
The additional figures in Appendix F clearly indicate tighter bounds at extreme quantiles, in correspondence with Theorem 3.3.
Beyond the unit interval In Figure 7 (main text) and Appendix F.1 we present further simulations of i.i.d. lognormal and Gaussian random variables, ranging over  $\mathbb{R}^+$  and  $\mathbb{R}$  respectively, and using Algorithm 3. The logarithmic dependence of the bound width upon the probe value is evident.
An Exhibition of Failure Figure 4 shows the (empirical) relative convergence when the data is simulated i.i.d. uniform over  $[0, \epsilon]$  for decreasing  $\epsilon$  (hence decreasing smoothness). The reference width is the maximum bound width obtained with Algorithm 1 and Algorithm 2 at  $t_{\mathrm{ref}} = 10000$  and  $\epsilon = 1/16$ , and shown is the multiplicative factor of time required for the maximum bound width to match the reference width as smoothness varies. The trend is consistent with arbitrarily poor convergence with arbitrarily small  $\epsilon$ . Because this is i.i.d. data, DKW applies and a uniform bound (independent of  $\epsilon$ ) is available. Thus while our instance-dependent guarantees are valuable in practice, they can be dominated by stronger guarantees leveraging additional assumptions. On a positive note, a logarithmic dependence on smoothness is evident over many orders of magnitude, confirming the analysis of Theorem 3.3.
Importance-Weighted In these simulations, in addition to being i.i.d.,  $X_{t}$  and  $W_{t}$  are drawn independently of each other, so the importance weights merely increase the difficulty of ultimately estimating the same quantity.
In the importance-weighted case, an additional aspect is whether the importance-weights have finite or infinite variance. Figures 5 and 13 demonstrate convergence in both conditions when using DDRM for pointwise bounds. Figures 14 and 15 show the results using empirical Bernstein pointwise bounds. In theory, with enough samples and infinite precision, the infinite variance Pareto simulation would eventually cause the empirical Bernstein variant to reset to trivial bounds, but in practice this is not observed. Instead, DDRM is consistently tighter but also consistently more expensive to compute, as exemplified in Table 2. Thus either choice is potentially preferable.
Table 2: Comparison of DDRM and Empirical Bernstein on i.i.d.  $X_{t} \sim \mathrm{Beta}(6,3)$ , for different  $W_{t}$ . Width denotes the maximum bound width  $\sup_v U_t(v) - L_t(v)$ . Time is for computing the bound at 1000 equally spaced points.
Wt	WHAT	WIDTH	TIME (SEC)
EXP(1)	DDRM	0.09	24.8
EMP. BERN	0.10	1.0
PARETO(3/2)	DDRM	0.052	59.4
EMP. BERN	0.125	2.4
4.2 Nonstationary
Continuous Polya Urn In this case
$$
X _ {t} \sim \operatorname {B e t a} \left(2 + \gamma_ {t} \sum_ {s <   t} 1 _ {X _ {s} > 1 / 2}, 2 + \gamma_ {t} \sum_ {s <   t} 1 _ {X _ {s} \leqslant 1 / 2}\right),
$$
i.e.,  $X_{t}$  is Beta distributed with parameters becoming more extreme over time: each realization will increasingly concentrate either towards 0 or 1. Suppose  $\gamma_{t} = t^{q}$ . In the most extreme case that  $\left(t = \sum_{s \leqslant t} 1_{X_{s} > 1/2}\right)$ , the conditional distribution at time  $t$  is Beta  $(x; 2 + t\gamma_{t}, 2) = O(t^{1 + q})$ , hence  $d\mathbb{P}_{t} / dU = O(t^{1 + q})$ , which is smooth enough for our bounds to converge. Figure 3 shows the bounds covering the true CDF for two realizations with different limits. Figure 12 shows (for one realization) the maximum bound width, scaled by  $\sqrt{t / \log(t)}$  to remove the primary trend, as a function of  $t$  for different  $\gamma_{t}$  schedules.
Importance-Weighted Continuous Polya Urn In this case  $W_{t}$  is drawn i.i.d. either  $W_{t} = 0$  or  $W_{t} = w_{\max}$ , such as might occur during off-policy evaluation with an epsilon-greedy logging policy. Given  $W_{t}$ , the distribution of  $X_{t}$  is given by
$$
\begin{array}{l} X _ {t} \mid W _ {t} \sim \operatorname {B e t a} \left(2 + \gamma_ {t} \sum_ {s <   t} 1 _ {X _ {s} > 1 / 2} 1 _ {W _ {s} = W _ {t}}, \right. \\ \left. 2 + \gamma_ {t} \sum_ {s <   t} 1 _ {X _ {s} <   1 / 2} 1 _ {W _ {s} = W _ {t}}\right), \\ \end{array}
$$
i.e., each importance weight runs an independent Continuous Polya Urn. Because of this, it is possible for the unweighted CDF to mostly concentrate at one limit (e.g., 1) but the weighted CDF to concentrate at another limit (e.g., 0). Figure 6 exhibits this phenomenon.
Constructing nonasymptotic confidence bands for the cumulative distribution function of i.i.d. random variables is a classical problem of statistical inference dating back to Dvoretzky et al. [1956] and Massart [1990]. While these bounds are quantile-uniform, they are ultimately fixed-time bounds
(i.e. not time-uniform). In other words, given a sample of i.i.d. random variables  $X_{1},\ldots ,X_{n}\sim F$  these fixed time bounds  $[\dot{L}_n(x),\dot{U}_n(x)]_{x\in \mathbb{R}}$  satisfy a guarantee of the form:
$$
\mathbb {P} (\forall x \in \mathbb {R}, \dot {L} _ {n} (x) \leqslant F (x) \leqslant \dot {U} _ {n} (x)) \geqslant 1 - \alpha , \tag {8}
$$
for any desired error level  $\alpha \in (0,1)$ . Howard and Ramdas [2022] developed confidence bands  $[\bar{L}_t(x),\bar{U}_t(x)]_{x\in \mathbb{R},t\in \mathbb{N}}$  that are both quantile- and time-uniform, meaning that they satisfy the stronger guarantee:
$$
\mathbb {P} (\forall x \in \mathbb {R}, t \in \mathbb {N}, \bar {L} _ {t} (x) \leqslant F (x) \leqslant \bar {U} _ {t} (x)) \geqslant 1 - \alpha . \tag {9}
$$
However, the bounds presented in Howard and Ramdas [2022] ultimately focused on the classical i.i.d. on-policy setup, meaning the CDF for which confidence bands are derived is the same CDF as those of the observations  $(X_{t})_{t = 1}^{\infty}$ . This is in contrast to off-policy evaluation problems such as in randomized controlled trials, adaptive A/B tests, or contextual bandits, where the goal is to estimate a distribution different from that which was collected (e.g. collecting data based on a Bernoulli experiment with the goal of estimating the counterfactual distribution under treatment or control). Chandak et al. [2021] and Huang et al. [2021] both introduced fixed-time (i.e. non-time-uniform) confidence bands for the off-policy CDF in contextual bandit problems, though their procedures are quite different, rely on different proof techniques, and have different properties from one another. Waudby-Smith et al. [2022, Section 4] later developed time-uniform confidence bands in the off-policy setting, using a technique akin to Howard and Ramdas [2022, Theorem 5] and has several desirable properties in comparison to Chandak et al. [2021] and Huang et al. [2021] as outlined in Waudby-Smith et al. [2022, Table 2].
Nevertheless, regardless of time-uniformity or on/off-policy estimation, all of the aforementioned prior works assume that the distribution to be estimated is fixed and unchanging over time. The present paper takes a significant departure from the existing literature by deriving confidence bands that allow the distribution to change over time in a data-dependent manner, all while remaining time-uniform and applicable to off-policy problems in contextual bandits. Moreover, we achieve this by way of a novel stitching technique which is closely related to those of Howard and Ramdas [2022] and Waudby-Smith et al. [2022].
This work constructs bounds by tracking specific values, in contrast with i.i.d. techniques which track specific quantiles. The value-based approach is amenable to proving correctness qua Theorem 3.1, but has the disadvantage of sensitivity to monotonic transformations. We speculate it is possible to be covariant to a fixed (wrt time) but unknown monotonic transformation without violating known impossibility results. A technique with this property would have increased practical utility.
The authors thank Ian Waudby-Smith for insightful discussion and review.
Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. arXiv preprint arXiv:2202.04690, 2022.
Francesco Paolo Cantelli. Sulla determinazione empirica delle leggi di probabilita. Giorn. Ist. Ital. Attuari, 4(421-424), 1933.
Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems, 34: 27475-27490, 2021.
Ioannis Chatzigeorgiou. Bounds on the lambert function and their application to the outage analysis of user cooperation. IEEE Communications Letters, 17(8):1505-1508, 2013.
Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, pages 642-669, 1956.
Xiequan Fan, Ion Grama, and Quansheng Liu. Exponential inequalities for martingales with applications. Electronic Journal of Probability, 20:1-22, 2015.
William Feller. An introduction to probability theory and its applications, 3rd edition. Wiley series in probability and mathematical statistics, 1958.
Valery Glivenko. Sulla determinazione empirica delle leggi di probabilita. Gion. Ist. Ital. Attauri., 4: 92-99, 1933.
Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis of Online and Differentially Private Learning. In Advances in Neural Information Processing Systems, volume 33, pages 9203-9215, 2020.
Nika Haghtalab, Yanjun Han, Abhishek Shetty, and Kunhe Yang. Oracle-Efficient Online Learning for Beyond Worst-Case Adversaries, November 2022a. arXiv:2202.08549 [cs, stat].
Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis with Adaptive Adversaries. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 942-953, 2022b.
Steven R Howard and Aaditya Ramdas. Sequential estimation of quantiles with applications to A/B testing and best-arm identification. Bernoulli, 28(3):1704-1728, 2022.
Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055-1080, 2021.
Audrey Huang, Liu Leqi, Zachary Lipton, and Kamyar Azizzadenesheli. Off-policy risk assessment in contextual bandits. Advances in Neural Information Processing Systems, 34:23714-23726, 2021.
Michael Kearns and Lawrence Saul. Large deviation methods for approximate probabilistic inference. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, pages 311-319, 1998.
Michael Lindon, Chris Sanden, and Vaché Shirikian. Rapid regression detection in software deployments through sequential testing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3336-3346, 2022.
Wentao Liu. Risk-Aware Financial Portfolio Management with Distributional Deep Deterministic Policy Gradient. PhD thesis, University of Toronto (Canada), 2021.
Pascal Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The annals of Probability, pages 1269-1283, 1990.
Paul Mineiro. A lower confidence sequence for the changing mean of non-negative right heavy-tailed observations with bounded mean. arXiv preprint arXiv:2210.11133, 2022.
Frank WJ Olver, Daniel W Lozier, Ronald F Boisvert, and Charles W Clark. NIST handbook of mathematical functions hardback and CD-ROM. Cambridge university press, 2010.
Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359, 2010.
Iosif Pinelis. Exact lower and upper bounds on the incomplete gamma function. arXiv preprint arXiv:2005.06384, 2020.
Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online Learning: Stochastic, Constrained, and Smoothed Adversaries. In Advances in Neural Information Processing Systems, volume 24, 2011.
Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability theory and related fields, 161(1):111-153, 2015.
Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. Statistical Science, 26(1):84-101, 2011.
Yi Shen, Jessilyn Dunn, and Michael M Zavlanos. Risk-averse multi-armed bandits with unobserved confounders: A case study in emotion regulation in mobile health. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 144–149. IEEE, 2022.
Yuheng Wang and Margaret P Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control. Artificial Intelligence, page 103743, 2022.
Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytime-valid off-policy inference for contextual bandits. arXiv preprint arXiv:2210.10768, 2022.
A Confidence Sequences for Fixed  $v$
Since our algorithm operates via reduction to pointwise confidence sequences, we provide a brief self-contained review here. We refer the interested reader to Howard et al. [2021] for a more thorough treatment.
A confidence sequence for a random process  $X_{t}$  is a time-indexed collection of confidence sets  $\mathrm{CI}_t$  with a time-uniform coverage property  $\mathbb{P}\left(\forall t\in \mathbb{N}:X_t\in \mathrm{CI}_t\right)\geqslant 1 - \alpha$ . For real random variables, the concept of a lower confidence sequence can be defined via  $\mathbb{P}\left(\forall t\in \mathbb{N}:X_t\geqslant L_t\right)\geqslant 1 - \alpha$ , and analogously for upper confidence sequences; and a lower and upper confidence sequence can be combined to form a confidence sequence  $\mathrm{CI}_t\doteq \{x|L_t\leqslant x\leqslant U_t\}$  with coverage  $(1 - 2\alpha)$  via a union bound.
One method for constructing a lower confidence sequence for a real valued parameter  $z$  is to exhibit a real-valued random process  $E_{t}(z)$  which, when evaluated at the true value  $z^{*}$  of the parameter of interest, is a non-negative supermartingale with initial value of 1, in which case Ville's inequality ensures  $\mathbb{P}\left(\forall t\in \mathbb{N}:E_t(z^*)\leqslant \alpha^{-1}\right)\geqslant 1 - \alpha$ . If the process  $E_{t}(z)$  is monotonically increasing in  $z$ , then the supremum of the lower contour set  $L_{t}\doteq \sup_{z}\left\{z|E_{t}(z)\leqslant \alpha^{-1}\right\}$  is suitable as a lower confidence sequence; an upper confidence sequence can be analogously defined.
We use the above strategy as follows. We bound these deviations using the following nonnegative martingale,
$$
E _ {t} (\lambda) \doteq \exp \left(\lambda S _ {t} - \sum_ {s \leqslant t} \log \left(h \left(\lambda , \theta_ {s}\right)\right)\right), \tag {10}
$$
where  $\lambda \in \mathbb{R}$  is fixed and  $h(\lambda, z) \doteq (1 - z)e^{-\lambda z} + ze^{\lambda(1 - z)}$ , the moment-generating function of a centered Bernoulli(z) random variable. Equation (10) is a test martingale qua Shafer et al. [2011], i.e., it can be used to construct time-uniform bounds on  $\hat{q}_t - q_t$  via Ville's inequality.
Next we lower bound Equation (10),
and eliminate the explicit dependence upon  $\theta_{s}$ , by noting  $\log h(\lambda, \cdot)$  is concave and therefore
$$
E _ {t} (\lambda) \geqslant \exp \left(\lambda t \left(q _ {t} - \hat {q} _ {t}\right) - t \log h (\lambda , q _ {t})\right), \tag {11}
$$
because  $\left(tf(q) = \max_{\theta|1^{\top}\theta = tq} \sum_{s \leqslant t} f(\theta_s)\right)$  for any concave  $f$ . Equation (11) is monotonically increasing in  $q_t$  and therefore defines a lower confidence sequence. For an upper confidence sequence we use  $q_t = 1 - (1 - q_t)$  and a lower confidence sequence on  $(1 - q_t)$ .
Regarding the choice of  $\lambda$ , in practice many  $\lambda$  are (implicitly) used via stitching (i.e., using different  $\lambda$  in different time epochs and majorizing the resulting bound in closed form) or mixing (i.e., using a particular fixed mixture of Equation (11) via a discrete sum or continuous integral over  $\lambda$ ); our choices will depend upon whether we are designing for tight asymptotic rates or low computational footprint. We provide specific details associated with each theorem or experiment.
Note Equation (11) is invariant to permutations of  $X_{1:t}$  and hence the empirical CDF at time  $t$  is a sufficient statistic for calculating Equation (11) at any  $v$ .
A.1 Challenge with quantile space
In this section assume all CDFs are invertible for ease of exposition.
In the i.i.d. setting, Equation (10) can be evaluated at the (unknown) fixed  $v(q)$  which corresponds to quantile  $q$ . Without knowledge of the values, one can assert the existence of such values for a countably infinite collection of quantiles and a careful union bound of Ville's inequality on a particular discretization can yield an LIL rate: this is the approach of Howard and Ramdas [2022]. A key advantage of this approach is covariance to monotonic transformations.
Beyond the i.i.d. setting, one might hope to analogously evaluate Equation (10) at an unknown fixed value  $v_{t}(q)$  which for each  $t$  corresponds to quantile  $q$ . Unfortunately,  $v_{t}(q)$  is not just unknown,
but also unpredictable with respect to the initial filtration, and the derivation that Equation (10) is a martingale depends upon  $v$  being predictable. In the case that  $X_{t}$  is independent but not identically distributed,  $v_{t}(q)$  is initially predictable and therefore this approach could work, but would only be valid under this assumption.
The above argument does not completely foreclose the possibility of a quantile space approach, but merely serves to explain why the authors pursued a value space approach in this work. We encourage the interested reader to innovate.
B Unit Interval Bounds
B.1 Worked Example
Our (synthetic) data set consists of five values, each of which has occurred 1000 times:
$$
D = \left\{\left(1 0 0 0, 0\right), \left(1 0 0 0, \frac {1}{7}\right), \left(1 0 0 0, \frac {2}{7}\right), \left(1 0 0 0, \frac {3}{7}\right), \left(1 0 0 0, \frac {6}{7}\right) \right\}.
$$
We use resolution  $\epsilon(d) = 2^d$  and coverage error  $\alpha = 1/20$ .
Upper bound for  $v = 4/7$  The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point  $\rho_1 = 2^{-1}[2^1 v] = 1$  again yields the trivial bound of 1. There are still empirical counts between the probe value  $(v = 4/7)$  and the bound value  $(\rho_1 = 1)$  so the algorithm continues. The second evaluated point  $\rho_2 = 2^{-2}[2^2 v] = 3/4$ , for which there are 4000 empirical counts below  $\rho_2$  out of 5000 total. The pointwise confidence sequence is evaluated with counts (4000, 5000) and coverage error  $\delta_2 = \alpha/24$ , resulting in improved bound  $\approx 0.825$ . Now, there are no empirical counts between the probe value  $(v = 4/7)$  and the bound value  $(\rho_2 = 3/4)$  so the algorithm terminates. To see all subsequent bounds are dominated, note that a tighter upper bound  $\rho_{d>2}$  will result in the same empirical counts (4000 out of 5000) but a looser coverage error and hence worse bound.
Upper bound for  $v = 13/28$  The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point  $\rho_1 = 2^{-1}[2^1 v] = 1/2$ , for which there are 4000 empirical counts below  $\rho_2$  out of 5000 total. The pointwise confidence sequence is evaluated with counts (4000, 5000) and coverage error  $\delta_1 = \alpha/4$ , resulting in improved bound  $\approx 0.822$ . There are no empirical counts between the probe value  $(v = 13/28)$  and the bound value  $(\rho_1 = 1/2)$  so the algorithm terminates. Relative to the previous example, the bound is slightly tighter as the discretization worked better for this  $v$ .
Upper bound for  $v = 5/14$  The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point  $\rho_1 = 2^{-1}[2^1 v] = 1/2$ , for which there are 4000 empirical counts below  $\rho_2$  out of 5000 total. The pointwise confidence sequence is evaluated with counts (4000, 5000) and coverage error  $\delta_1 = \alpha/4$ , resulting in improved bound  $\approx 0.822$ . There are empirical counts between the probe value  $(v = 5/14)$  and the bound value  $(\rho_1 = 1/2)$  so the algorithm continues. The second evaluated point  $\rho_2 = 2^{-2}[2^2 v] = 1/2$ , which has the same empirical counts but worse coverage error at this level, and hence does not improve the bound. There are empirical counts between the probe value  $(v = 5/14)$  and the bound value  $(\rho_2 = 1/2)$  so the algorithm continues. The third evaluated point  $\rho_3 = 2^{-3}[2^3 v] = 3/8$ , for which there are 3000 empirical counts below  $\rho_3$  out of 5000 total. The pointwise confidence sequence is evaluated with counts (3000, 5000) and coverage error  $\delta_3 = \alpha/96$ , resulting in improved bound  $\approx 0.633$ . There are no empirical counts between the probe value  $(v = 5/14)$  and the bound value  $(\rho_3 = 3/8)$  so the algorithm terminates.
B.2 Lower Bound
Algorithm 2 is extremely similar to Algorithm 1: the differences are indicated in comments. Careful inspection reveals the output of Algorithm 1,  $U_{t}(v)$ , can be obtained from the output of Algorithm 2,  $L_{t}(v)$ , via  $U_{t}(v) = 1 - L_{t}(1 - v)$ ; but only if the sufficient statistics are adjusted such that  $\Xi_t(\rho_d; \delta, \Psi_t) = 1 - \Lambda_t(1 - \rho_d; \delta, \Psi_t')$ . The reference implementation uses this strategy.
We prove the results for the upper bound Algorithm 1; the argument for the lower bound Algorithm 2 is similar.
Algorithm 2 Unit Interval Lower Bound.  $\epsilon (d)$  is an increasing function specifying the resolution of discretization at level  $d$ .  $\Lambda_t(\rho ;\delta ,d,\Psi_t)$  is a lower confidence sequence for fixed value  $\rho$  with coverage at least  $(1 - \delta)$ .
Input: value  $v$ ; confidence  $\alpha$ ; sufficient statistic  $\Psi_t$ . // comments below indicate differences from upper bound  
//  $\Psi_t \doteq X_{1:t}$  or  $\Psi_t \doteq (W_{1:t}, X_{1:t})$   
Output:  $L_t(v)$  satisfying Equation (2).  
if  $v < 0$  then return 0 end if // check for underflow of range rather than overflow  
 $l \gets 0$  // initialize with 0 instead of 1  
 $v \gets \min(1, v)$  // project onto  $[0, 1]$  using min instead of max  
for  $d = 1$  to  $\infty$  do  
 $\rho_d \gets \epsilon(d)^{-1}[\epsilon(d)v]$  // use floor instead of ceiling  
 $\delta_d \gets \alpha/2^d\epsilon(d)$ $l \gets \max(l, \Lambda_t(\rho_d; \delta, \Psi_t))$  // use lower bound instead of upper bound  
if  $0 = \sum_{s \leqslant t} 1_{X_s \in [\rho_d, v)}$  then  
return  $l$   
end if  
end for