SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Voleon Group
steve@stevehoward.org
Estimation of a complete univariate distribution from a sequence of observations is a useful primitive for both manual and automated decision making. This problem has received extensive attention in the i.i.d. setting, but the arbitrary data dependent setting remains largely unaddressed. We present computationally felicitous time-uniform and value-uniform bounds on the CDF of the running averaged conditional distribution of a sequence of real-valued random variables. Consistent with known impossibility results, our CDF bounds are always valid but sometimes trivial when the instance is too hard, and we give an instance-dependent convergence guarantee. The importance-weighted extension is appropriate for estimating complete counterfactual distributions of rewards given data from a randomized experiment, e.g., from an A/B test or a contextual bandit.
What would have happened if I had acted differently? Although as old as time itself, successful companies have recently embraced this question via offline estimation of counterfactual outcomes using data from existing randomized experiments or contextual bandits. The problem is important in diverse domains such as software testing [Lindon et al., 2022, Wang and Chapman, 2022], portfolio management [Liu, 2021], and medicine [Shen et al., 2022]. These experiments are run in the real (digital) world, which is rich enough to demand non-asymptotic statistical techniques under non-parametric and non-stationary (i.e., not i.i.d.) models. Although existing methods apply for estimating average outcomes in this general setting (either under the observed distribution or counterfactual ones), estimating a complete distribution of outcomes is heretofore only possible with additional assumptions: see Table 1 for a summary and Section 5 for complete discussion of related work.
To fix ideas, we briefly describe an application from Lindon et al. [2022] in the context of canary testing: rolling out changes in an online service to a small, random subset of users in order to detect accidental performance regressions while minimizing effect on overall user experience. The metric of interest measures latency for fetching content from the service. It is common to look beyond the mean of the latency distribution and especially to check for regressions in upper quantiles. As such, the authors choose to estimate bounds on the entire CDF of this latency metric under both the control and treatment arms and check for a statistically significant differences at any point in the CDF. The hope is to detect regressions as soon as possible, often within seconds or minutes, so the authors employ a sequential method which allows an automated system to continuously update the CDF bounds as data accumulates and to stop as soon as a significant regression is detected. Statistically, this translates into the requirement of confidence bands for the CDF which are both uniform over time (valid after every update) and uniform over values (so we can check for regressions at any quantile). We seek such bounds whose statistical validity is guaranteed under a minimum of assumptions.
Intriguingly, this problem is provably impossible in the general data dependent setting [Rakhlin et al., 2015]. Consequently, our bounds always achieve non-asymptotic coverage, but may converge to zero width slowly or not at all, depending on the hardness of the instance. We call this design principle AVAST (Always Valid And Sometimes Trivial).
Table 1: Comparison to prior art for quantile-uniform CDF estimation. Time-uniform: a sequence of confidence bands whose coverage holds uniformly over time with high probability. Nonstationary: does not require an i.i.d. assumption. Non-asymptotic: guarantees hold at all sample sizes. Non-parametric: guarantees apply over an infinite-dimensional class of distributions. Counterfactual: method applies to estimating a distribution other than the one which generated the observed data, via importance weighting.  $w_{\mathrm{max}}$ -free: guarantee applies without a bound on the maximum importance weight. See Section 5 for details.
REFERENCE	TIME-UNIFORM?	NON-STATIONARY?	NON-ASYMPTOTIC?	NON-PARAMETRIC?	COUNTER-FACTUAL?	wmax-FREE?
HR22	✓	✓	✓	N/A
HLLA21	✓	✓	✓
UNO21, [IID]	✓	✓	✓	✓
UNO21, [NS]	✓	✓	✓
WS22, [§4]	✓	✓	✓	✓	✓
THIS PAPER	✓	✓	✓	✓	✓	✓
1. In Section 3.2 we provide a time- and value-uniform upper bound on the CDF of the averaged historical conditional distribution of a discrete-time real-valued random process. Consistent with the lack of sequential uniform convergence of linear threshold functions [Rakhlin et al., 2015], the bounds are Always Valid (see Theorem 3.1) And Sometimes Trivial, i.e., the width guarantee is instance-dependent (see Theorem 3.3) and may not converge to zero width in the infinite data limit. When the data generating process is smooth with respect to the uniform distribution on the unit interval, the bound width adapts to the unknown smoothness parameter, following the framework of smoothed online learning [Rakhlin et al., 2011, Haghtalab et al., 2020, 2022b,a, Block et al., 2022].
2. In Section 3.3 we extend the previous technique to distributions with support over the entire real line, and further to distributions with a known countably infinite or unknown nowhere dense set of discrete jumps; with analogous instance-dependent guarantees.
3. In Section 3.4 we extend the previous techniques to importance-weighted random variables, achieving our ultimate goal of estimating a complete counterfactual distribution of outcomes.
We exhibit our techniques in various simulations in Section 4. Computationally our procedures have comparable cost to point estimation of the empirical CDF, as the empirical CDF is a sufficient statistic.
Let  $(\Omega, \mathcal{F}, \{\mathcal{F}_t\}_{t \in \mathbb{N}}, \mathbb{P})$  be a probability space equipped with a discrete-time filtration, on which let  $X_t$  be an adapted, real-valued random process. Let  $\mathbb{E}_t[\cdot] \doteq \mathbb{E}[\cdot | \mathcal{F}_t]$ . The quantity of interest is the (random) map  $\overline{\mathrm{CDF}}_t : \mathbb{R} \to [0,1]$ , the CDF of the averaged historical conditional distribution at time  $t$ :
$$
\overline {{\mathrm {C D F}}} _ {t} (v) \doteq \frac {1}{t} \sum_ {s \leqslant t} \mathbb {E} _ {s - 1} \left[ 1 _ {X _ {s} \leqslant v} \right]. \tag {1}
$$
We desire simultaneously time- and value-uniform bounds which hold with high probability, i.e., adapted sequences of maps  $L_{t}, U_{t} : \mathbb{R} \to [0,1]$  satisfying
$$
\mathbb {P} \left(\forall t \in \mathbb {N}: L _ {t} (v) \leqslant \overline {{\mathrm {C D F}}} _ {t} (v) \leqslant U _ {t} (v)\right) \geqslant 1 - 2 \alpha . \tag {2}
$$
Note that the estimand  $\overline{\mathrm{CDF}}_t$  is changing at each time step as we incorporate the conditional distribution of the latest observation into our historical average. The maps  $L_{t},U_{t}$  provide a sequence of confidence bands which contain this sequence of changing CDFs uniformly over time with high probability.
In the i.i.d. setting, Equation (1) is deterministic and independent of  $t$ , reducing to the CDF of the (unknown) generating distribution. In this setting, the classic results of Glivenko [1933] and
Figure 1: Visualization of Algorithm 1. The values of interest are uncountably infinite; the algorithm allocates probability to maintain upper bounds on a countably infinite set of points  $\rho$  at different resolution levels via the monotonicity of  $\overline{\mathrm{CDF}}_t(v)$ . As resolution increases, the value  $\rho$  better approximates  $v$ , but the allocated probability decreases; the algorithm chooses the tightest of available bounds. Shaded nodes would be consulted for an upper bound for  $v = 5/17$ .
Algorithm 1 Unit Interval Upper Bound.  $\epsilon (d)$  is an increasing function specifying the resolution of discretization at level  $d$ .  $\Xi_t$  ( $\rho$ ;  $\delta$ ,  $d$ ,  $\Psi_t$ ) is an upper confidence sequence for fixed value  $\rho$  with coverage at least  $(1 - \delta)$ .
Input: value  $v$ ; confidence  $\alpha$ ; sufficient statistic  $\Psi_t$ . // e.g.  $\Psi_t \doteq X_{1:t}$  or  $\Psi_t \doteq (W_{1:t}, X_{1:t})$   
Output:  $U_t(v)$  satisfying Equation (2).  
if  $v > 1$  then return 1 end if  
 $u \gets 1$ $v \gets \max(0, v)$   
for  $d = 1$  to  $\infty$  do  
 $\rho_d \gets \epsilon(d)^{-1} [\epsilon(d)v]$ $\delta_d \gets \alpha / (d(d+1)\epsilon(d))$ $u \gets \min(u, \Xi_t (\rho_d; \delta_d, \Psi_t))$   
if  $0 = \sum_{s \leqslant t} 1_{X_s \in (v, \rho_d]}$  then  
return  $u$   
end if  
end for
Cantelli [1933] established uniform convergence of linear threshold functions; subsequently the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality characterized fixed-time and value-uniform convergence rates [Dvoretzky et al., 1956, Massart, 1990]; extended later to simultaneously time- and value-uniform bounds [Howard and Ramdas, 2022]. The latter result guarantees an  $O(t^{-1}\log (\log (t)))$  confidence interval width, matching the limit imposed by the Law of the Iterated Logarithm.
AVAST principle In contrast, under arbitrary data dependence, linear threshold functions are not sequentially uniformly convergent, i.e., the averaged historical empirical CDF does not necessarily converge uniformly to the CDF of the averaged historical conditional distribution [Rakhlin et al., 2015]. Consequently, additional assumptions are required to provide a guarantee that the confidence width decays to zero. In this paper we design bounds that are Always Valid And Sometimes Trivial, i.e., under worst-case data generation,  $\sup_v|U_t(v) - L_t(v)| = O(1)$  as  $t\to \infty$ . Fortunately our bounds are also equipped with an instance-dependent width guarantee based upon the smoothness of the distribution to a reference measure qua Definition 3.2.
Additional Notation Let  $X_{a:b} = \{X_s\}_{s=a}^b$  denote a contiguous subsequence of a random process. Let  $\mathbb{P}_t$  denote the average historical conditional distribution, defined as a (random) distribution over the sample space  $\mathbb{R}$  by  $\mathbb{P}_t(A) \doteq t^{-1} \sum_{s \leqslant t} \mathbb{E}_{s-1} \left[ 1_{X_s \in A} \right]$  for a Borel subset  $A$  (note  $\mathbb{P}_t$  represents the entire historical average while  $\mathbb{E}_t$  corresponds to a single conditional distribution).
3 Derivations
3.1 High Level Design
Our approaches work as reductions, achieving the value- and time-uniform guarantee of Equation (2) by combining bounds  $\Lambda_t, \Xi_t$  that satisfy a time-uniform guarantee at any fixed value  $\rho$ ,
$$
\mathbb {P} \left(\forall t \in \mathbb {N}: \Lambda_ {t} (\rho) \leqslant \overline {{\mathrm {C D F}}} _ {t} (\rho) \leqslant \Xi_ {t} (\rho)\right) \geqslant 1 - \delta (\rho). \tag {3}
$$
The bounds  $\Lambda_t, \Xi_t$  are tools for estimating a sequence of scalars, in this case  $(\overline{\mathrm{CDF}}_t(\rho))_{t=1}^{\infty}$  for a fixed value  $\rho$ . We show how to extend such tools to the more difficult problem of estimating a sequence of (cumulative distribution) functions.
There are multiple existing approaches to obtaining the guarantee of Equation (3): we provide a self-contained introduction in Appendix A. For ease of exposition, we will only discuss how to construct
a time- and value-uniform upper bound by combining fixed-value, time-uniform upper bounds, and defer the analogous lower bound construction to Appendix B.2. Our approach is to compose these fixed-value bounds into a value-uniform bound by taking a union bound over a particular collection of values, leveraging monotonicity of the CDF.
Quantile vs Value Space In the i.i.d. setting, a value-uniform guarantee can be obtained by taking a careful union bound over the unique value associated with each quantile [Howard and Ramdas, 2022]. This "quantile space" approach has advantages, e.g., variance based discretization and covariance to monotonic transformations. However, under arbitrary data dependence, the value associated with each quantile can change. Therefore we proceed in "value space". See Appendix A.1 for more details.
3.2 On the Unit Interval
Algorithm 1, visualized in Figure 1, constructs an upper bound on Equation (1) which, while valid for all values, is designed for random variables ranging over the unit interval. For a given value  $v$ , it searches over upper bounds on the CDF evaluated at a decreasing sequence of values  $\rho_1 \geqslant \rho_2 \geqslant \dots \geqslant v$  and exploits monotonicity of  $\overline{\mathrm{CDF}}_t(v)$ . That is, at each level  $d = 1, 2, \ldots$ , we construct a discretizing grid of size  $\epsilon(d)$  over the unit interval, and construct a time-uniform upper bound on  $\overline{\mathrm{CDF}}_t(\rho)$  for each grid point  $\rho$  using the fixed-value confidence sequence oracle  $\Xi_t$ . Then, for a given value  $v$ , at each level  $d$  we make use of the fixed-value confidence sequence for smallest grid point  $\rho_d \geqslant v$ , and we search for the level  $d$  which yields the minimal upper confidence bound. A union bound over the (countably infinite) possible choices for  $\rho_d$  controls the coverage of the overall procedure. Because the error probability  $\delta_d$  decreases with  $d$  (and the fixed-value confidence radius  $\Xi_t$  increases as  $\delta$  decreases), the procedure can terminate whenever no observations remain between the desired value  $v$  and the current upper bound  $\rho_d$ , as all subsequent bounds are dominated.
The lower bound is derived analogously in Algorithm 2 (which we have left to Appendix B.2 for the sake of brevity) and leverages a lower confidence sequence  $\Lambda_t(\rho; \delta, \Psi_t)$  (instead of an upper confidence sequence) evaluated at an increasingly refined lower bound on the value  $\rho \gets \epsilon(d)^{-1}[\epsilon(d)v]$ .
Theorem 3.1. If  $\epsilon(d) \uparrow \infty$  as  $d \uparrow \infty$ , then Algorithms 1 and 2 terminate with probability one. Furthermore, if for all  $\rho, \delta$ , and  $d$  the algorithms  $\Lambda_t(\rho; \delta, \Psi_t)$  and  $\Xi_t(\rho; \delta, \Psi_t)$  satisfy
$$
P \left(\forall t: \overline {{\mathrm {C D F}}} _ {t} (\rho) \geqslant \Lambda_ {t} (\rho ; \delta , \Psi_ {t})\right) \geqslant 1 - \delta , \tag {4}
$$
$$
P \left(\forall t: \overline {{\mathrm {C D F}}} _ {t} (\rho) \leqslant \Xi_ {t} (\rho ; \delta , \Psi_ {t})\right) \geqslant 1 - \delta , \tag {5}
$$
then guarantee (2) holds with  $U_{t}, L_{t}$  given by the outputs of Algorithms 1 and 2, respectively.
Proof. See Appendix B.3.
Theorem 3.1 ensures Algorithms 1 and 2 yield the desired time- and value-uniform coverage, essentially due to the union bound and the coverage guarantees of the oracles  $\Xi_t, \Lambda_t$ . However, coverage is also guaranteed by the trivial bounds  $0 \leqslant \overline{\mathrm{CDF}}_t(v) \leqslant 1$ . The critical question is: what is the bound width?
Smooed Regret Guarantee Even assuming  $X$  is entirely supported on the unit interval, on what distributions will Algorithm 1 provide a non-trivial bound? Because each  $\left[\Lambda_t(\rho; \delta, \Psi_t), \Xi_t(\rho; \delta, \Psi_t)\right]$  is a confidence sequence for the mean of the bounded random variable  $1_{X_s \leqslant \rho}$ , we enjoy width guarantees at each of the (countably infinite)  $\rho$  which are covered by the union bound, but the guarantee degrades as the depth  $d$  increases. If the data generating process focuses on an increasingly small part of the unit interval over time, the width guarantees on our discretization will be insufficient to determine the distribution. Indeed, explicit constructions demonstrating the lack of sequential uniform convergence of linear threshold functions increasingly focus in this manner [Block et al., 2022].
Conversely, if  $\forall t:\overline{\mathrm{CDF}}_t(v)$  was Lipschitz continuous in  $v$ , then our increasingly granular discretization would eventually overwhelm any fixed Lipschitz constant and guarantee uniform convergence. Theorem 3.3 expresses this intuition, but using the concept of smoothness rather than Lipschitz, as smoothness will allow us to generalize further [Rakhlin et al., 2011, Haghtalab et al., 2020, 2022b,a, Block et al., 2022].
Figure 2: CDF bounds approaching the true CDF when sampling i.i.d. from a  $\mathrm{Beta}(6,3)$  distribution. Note these bounds are simultaneously valid for all times and values.
Figure 3: Nonstationary Polya simulation for two seeds approaching different average conditional CDFs. Bounds successfully track the true CDFs in both cases. See Section 4.2.
Definition 3.2. A distribution  $D$  is  $\xi$ -smooth wrt reference measure  $M$  if  $D \ll M$  and  $\operatorname{ess}\sup_{M} \left( \frac{dD}{dM} \right) \leqslant \xi^{-1}$ .
When the reference measure is the uniform distribution on the unit interval,  $\xi$ -smoothness implies an  $\xi^{-1}$ -Lipschitz CDF. However, when the reference measure has its own curvature, or charges points, the concepts diverge. When reading Theorem 3.3, note  $\xi \leqslant 1$  (since the reference measure is a probability distribution) and as  $\xi \to 0$  the smoothness constraint is increasingly relaxed. Thus Theorem 3.3 states "for less smooth distributions, convergence is slowed."
Theorem 3.3. Let  $U_{t}(v)$  and  $L_{t}(v)$  be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively, when evaluated with  $\epsilon (d) = 2^{d}$  and the confidence sequences  $\Lambda_t$  and  $\Xi_t$  of Equation (15). If  $\forall t: \mathbb{P}_t$  is  $\xi_{t}$ -smooth wrt the uniform distribution on the unit interval then
$$
\forall t, \forall v: U _ {t} (v) - L _ {t} (v) \leqslant
$$
$$
\sqrt {\frac {V _ {t}}{t}} + \tilde {O} \left(\sqrt {\frac {V _ {t}}{t} \log \left(\xi_ {t} ^ {- 2} \alpha^ {- 1} t ^ {3 / 2}\right)}\right), \tag {6}
$$
where  $q_{t} \doteq \overline{\mathrm{CDF}}_{t}(v)$ ;  $V_{t} \doteq \frac{1}{t} + \frac{(q_{t} - 1 / 2)}{\log(q_{t} / 1 - q_{t})}$ ; and  $\tilde{O}()$  elides polylog  $V_{t}$  factors.
Theorem 3.3 matches our empirical results in two important aspects: (i) logarithmic dependence upon smoothness (e.g., Figure 4); (ii) tighter intervals for more extreme quantiles (e.g., Figure 2). Note the choice  $\epsilon(d) = 2^d$  ensures the loop in Algorithm 1 terminates after at most  $\log_2(\Delta)$  iterations, where  $\Delta$  is the minimum difference between two distinct realized values.
Worked Example To build intuition, in Appendix B.1 we explicitly calculate Algorithm 1 for a synthetic data set.
Arbitrary Support In Appendix D.1 we describe a variant of Algorithm 1 which uses a countable dense subset of the entire real line. It enjoys a similar guarantee to Theorem 3.3, but with an additional
width which is logarithmic in the probe value  $v$ :  $\tilde{O}\left(\sqrt{\frac{V_t}{t}\log\left(\left(2 + \xi_t|v|t^{-1 / 2}\right)^2\xi_t^{-2}\alpha^{-1}t^{3 / 2}\right)}\right)$ .
Note in this case  $\xi_{t}$  is defined relative to (unnormized) Lebesgue measure and can therefore exceed 1.
Discrete Jumps If  $\mathbb{P}_t$  is smooth wrt a reference measure which charges a countably infinite number of known discrete points, we can explicitly union bound over these additional points proportional to their density in the reference measure. In this case we preserve the above value-uniform guarantees. See Appendix D.2 for more details.
For distributions which charge unknown discrete points, we note the proof of Theorem 3.3 only exploits smoothness local to  $v$ . Therefore if the set of discrete points is nowhere dense, we eventually recover the guarantee of Equation (6) after a "burn-in" time  $t$  which is logarithmic in the minimum distance from  $v$  to a charged discrete point.
Figure 4: As smoothness decreases, we require more time to reach the same maximum confidence width. For low smoothness, DKW dominates our method. The logarithmic dependence matches our theory. See Section 4.1.
Figure 5: CDF bounds approaching the true counterfactual CDF when sampling i.i.d. from a  $\mathrm{Beta}(6,3)$  with infinite-variance importance weights, using DDRM for the oracle confidence sequence.
3.4 Importance-Weighted Variant
An important use case is estimating a distribution based upon observations produced from another distribution with a known shift, e.g., arising in transfer learning [Pan and Yang, 2010] or off-policy evaluation [Waudby-Smith et al., 2022]. In this case the observations are tuples  $(W_{t},X_{t})$ , where the importance weight  $W_{t}$  is a Radon-Nikodym derivative, implying  $\forall t:\mathbb{E}_t[W_t] = 1$  and a.s.  $W_{t}\geqslant 0$ ; and the goal is to estimate  $\overline{\mathrm{CDF}}_t(v) = t^{-1}\sum_{s\leqslant t}\mathbb{E}_{s - 1}[W_s1_{X_s\leqslant v}]$ . The basic approach in Algorithm 1 and Algorithm 2 is still applicable in this setting, but different  $\Lambda_t$  and  $\Xi_t$  are required. In Appendix E we present details on two possible choices for  $\Lambda_t$  and  $\Xi_t$ : the first is based upon the empirical Bernstein construction of Howard et al. [2021], and the second based upon the DDRM construction of Mineiro [2022]. Both constructions leverage the  $L^{*}$  Adagrad bound of Orabona [2019] to enable lazy evaluation. The empirical Bernstein version is amenable to analysis and computationally lightweight, but requires finite importance weight variance to converge (the variance bound need not be known, as the construction adapts to the unknown variance). The DDRM version requires more computation but produces tighter intervals. See Section 4.1 for a comparison.
Inspired by the empirical Bernstein variant, the following analog of Theorem 3.3 holds. Note  $\mathbb{P}_t$  is the target (importance-weighted) distribution, not the observation (non-importance-weighted) distribution.
Theorem 3.4. Let  $U_{t}(v)$  and  $L_{t}(v)$  be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively with  $\epsilon(d) = 2^{d}$  and the confidence sequences  $\Lambda_{t}$  and  $\Xi_{t}$  of Equation (18). If  $\forall t: \mathbb{P}_{t}$  is  $\xi_{t}$ -smooth wrt the uniform distribution on the unit interval then
$$
\begin{array}{l} \forall t, \forall v: U _ {t} (v) - L _ {t} (v) \leqslant \\ B _ {t} + \sqrt {\frac {(\tau + V _ {t}) / t}{t}} \\ + \tilde {O} \left(\sqrt {\frac {(\tau + V _ {t}) / t}{t} \log \left(\xi_ {t} ^ {- 2} \alpha^ {- 1}\right)}\right) \tag {7} \\ + \tilde {O} (t ^ {- 1} \log (\xi_ {t} ^ {- 2} \alpha^ {- 1})), \\ \end{array}
$$
where  $q_{t} \doteq \overline{\mathrm{CDF}}_{t}(v)$ ,  $K(q_{t}) \doteq (q_{t - 1 / 2}) / \log (q_{t / 1 - q_{t}})$ ;  $V_{t} = O\left(K(q_{t})\sum_{s\leqslant t}W_{s}^{2}\right)$ ,  $B_{t} \doteq t^{-1}\sum_{s\leqslant t}(W_{s} - 1)$ , and  $\tilde{O}()$  elides polylog  $V_{t}$  factors.
Proof. See Appendix E.2.
Theorem 3.4 exhibits the following key properties: (i) logarithmic dependence upon smoothness; (ii) tighter intervals for extreme quantiles and importance weights with smaller quadratic variation; (iii) no explicit dependence upon importance weight range; (iv) asymptotic zero width for importance weights with sub-linear quadratic variation.
Additional Remarks First, the importance-weighted average CDF is a well-defined mathematical quantity, but the interpretation as a counterfactual distribution of outcomes given different actions in the controlled experimentation setting involves subtleties: we refer the interested reader to Waudby-Smith et al. [2022] for a complete discussion. Second, the need for nonstationarity techniques for
Figure 6: A nonstationary, importance-weighted simulation in which the factual distribution (red) diverges dramatically from the counterfactual distribution (green). The bound correctly covers the counterfactual CDF.
Figure 7: Demonstration of the variant described in Section 3.3 and Appendix D.1 for distributions with arbitrary support, based on i.i.d. sampling from a variety of lognormal distributions. Logarithmic range dependence is evident.
estimating the importance-weighted CDF is driven by the outcomes  $(X_{t})$  and not the importance-weights  $(W_{t})$ . For example with off-policy contextual bandits, a changing historical policy does not induce nonstationarity, but a changing conditional reward distribution does.