Title: Time-uniform confidence bands for the CDF under nonstationarity
Venue: NeurIPS 2023
Accepted: yes
Keywords: off-policy evaluation

anytime-valid
Abstract: Estimation of a complete univariate distribution from a sequence of observations is a useful primitive for both manual and automated decision making. This problem has received extensive attention in the i.i.d. setting, but the arbitrary data dependent setting remains largely unaddressed. We present computationally felicitous time-uniform and value-uniform bounds on the CDF of the running averaged conditional distribution of a sequence of real-valued random variables. Consistent with known impossibility results, our CDF bounds are always valid but sometimes trivial when the instance is too hard, and we give an instance-dependent convergence guarantee.  The importance-weighted extension is appropriate for estimating complete counterfactual distributions of rewards given data from a randomized experiment, e.g., from an A/B test or a contextual bandit.

Time-uniform confidence bands for the CDF under nonstationarity

Abstract

Estimation of a complete univariate distribution from a sequence of observations is a useful primitive for both manual and automated decision making. This problem has received extensive attention in the i.i.d. setting, but the arbitrary data dependent setting remains largely unaddressed. We present computationally felicitous timeuniform and value-uniform bounds on the CDF of the running averaged conditional distribution of a sequence of real-valued random variables. Consistent with known impossibility results, our CDF bounds are always valid but sometimes trivial when the instance is too hard, and we give an instance-dependent convergence guarantee. The importance-weighted extension is appropriate for estimating complete counterfactual distributions of rewards given data from a randomized experiment, e.g., from an A/B test or a contextual bandit.

1 Introduction

What would have happened if I had acted differently? Although as old as time itself, successful companies have recently embraced this question via offline estimation of counterfactual outcomes using data from existing randomized experiments or contextual bandits. The problem is important in diverse domains such as software testing [Lindon et al., 2022, Wang and Chapman, 2022], portfolio management [Liu, 2021], and medicine [Shen et al., 2022]. These experiments are run in the real (digital) world, which is rich enough to demand non-asymptotic statistical techniques under nonparametric and non-stationary (i.e., not i.i.d.) models. Although existing methods apply for estimating average outcomes in this general setting (either under the observed distribution or counterfactual ones), estimating a complete distribution of outcomes is heretofore only possible with additional assumptions: see Table 1 for a summary and Section 5 for complete discussion of related work.

To fix ideas, we briefly describe an application from Lindon et al. [2022] in the context of canary testing: rolling out changes in an online service to a small, random subset of users in order to detect accidental performance regressions while minimizing effect on overall user experience. The metric of interest measures latency for fetching content from the service. It is common to look beyond the mean of the latency distribution and especially to check for regressions in upper quantiles. As such, the authors choose to estimate bounds on the entire CDF of this latency metric under both the control and treatment arms and check for a statistically significant differences at any point in the CDF. The hope is to detect regressions as soon as possible, often within seconds or minutes, so the authors employ a sequential method which allows an automated system to continuously update the CDF bounds as data accumulates and to stop as soon as a significant regression is detected. Statistically, this translates into the requirement of confidence bands for the CDF which are both uniform over time (valid after every update) and uniform over values (so we can check for regressions at any quantile). We seek such bounds whose statistical validity is guaranteed under a minimum of assumptions.

Intriguingly, this problem is provably impossible in the general data dependent setting [Rakhlin et al., 2015]. Consequently, our bounds always achieve non-asymptotic coverage, but may converge to zero width slowly or not at all, depending on the hardness of the instance. We call this design principle AVAST (Always Valid And Sometimes Trivial).

Contributions

In Section 3.2 we provide a time- and value-uniform upper bound on the CDF of the averaged historical conditional distribution of a discrete-time real-valued random process. Consistent with the lack of sequential uniform convergence of linear threshold functions [Rakhlin et al., 2015], the bounds are Always Valid (see Theorem 3.1) And Sometimes Trivial, i.e., the width guarantee is instance-dependent (see Theorem 3.3) and may not converge to zero width in the infinite data limit. When the data generating process is smooth with respect to the uniform distribution on the unit interval, the bound width adapts to the unknown smoothness parameter, following the framework of smoothed online learning [Rakhlin et al., 2011, Haghtalab et al., 2020, 2022b,a, Block et al., 2022].

In Section 3.3 we extend the previous technique to distributions with support over the entire real line, and further to distributions with a known countably infinite or unknown nowhere dense set of discrete jumps; with analogous instance-dependent guarantees.

In Section 3.4 we extend the previous techniques to importance-weighted random variables, achieving our ultimate goal of estimating a complete counterfactual distribution of outcomes.

We exhibit our techniques in various simulations in Section 4. Computationally our procedures have comparable cost to point estimation of the empirical CDF, as the empirical CDF is a sufficient statistic.

2 Problem Setting

Let p Ω , F , t F t u t P N , P q be a probability space equipped with a discrete-time filtration, on which let X t be an adapted, real-valued random process. Let E t r¨s . ' E r¨| F t s . The quantity of interest is the (random) map CDF t : R Ñr 0 , 1 s , the CDF of the averaged historical conditional distribution at time t :

We desire simultaneously time- and value-uniform bounds which hold with high probability, i.e., adapted sequences of maps L t , U t : R Ñr 0 , 1 s satisfying

Note that the estimand CDF t is changing at each time step as we incorporate the conditional distribution of the latest observation into our historical average. The maps L t , U t provide a sequence of confidence bands which contain this sequence of changing CDFs uniformly over time with high probability.

In the i.i.d. setting, Equation (1) is deterministic and independent of t , reducing to the CDF of the (unknown) generating distribution. In this setting, the classic results of Glivenko [1933] and

α

spend per level

α

2

α

2

α

4

α

4

α

8

α

8

Testing thresholds

𝜌

4

3

4

3

4

1

4

1

8

1

8

1

8

5

8

5

8

7

8

7

2

1

2

1

8

3

8

3

...

Algorithm 1 Unit Interval Upper Bound. ϵ p d q is an increasing function specifying the resolution of discretization at level d . Ξ t p ρ ; δ, d, Ψ t q is an upper confidence sequence for fixed value ρ with coverage at least p 1 ´ δ q .

Cantelli [1933] established uniform convergence of linear threshold functions; subsequently the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality characterized fixed-time and value-uniform convergence rates [Dvoretzky et al., 1956, Massart, 1990]; extended later to simultaneously time- and valueuniform bounds [Howard and Ramdas, 2022]. The latter result guarantees an O p t ´ 1 log p log p t qqq confidence interval width, matching the limit imposed by the Law of the Iterated Logarithm.

AVAST principle In contrast, under arbitrary data dependence, linear threshold functions are not sequentially uniformly convergent, i.e., the averaged historical empirical CDF does not necessarily converge uniformly to the CDF of the averaged historical conditional distribution [Rakhlin et al., 2015]. Consequently, additional assumptions are required to provide a guarantee that the confidence width decays to zero. In this paper we design bounds that are Always Valid And Sometimes Trivial, i.e., under worst-case data generation, sup v | U t p v q ´ L t p v q | ' O p 1 q as t Ñ 8 . Fortunately our bounds are also equipped with an instance-dependent width guarantee based upon the smoothness of the distribution to a reference measure qua Definition 3.2.

Additional Notation Let X a : b ' t X s u b s ' a denote a contiguous subsequence of a random process. Let P t denote the average historical conditional distribution, defined as a (random) distribution over the sample space R by P t p A q . ' t ´ 1 ř s ď t E s ´ 1 r 1 X s P A s for a Borel subset A (note P t represents the entire historical average while E t corresponds to a single conditional distribution).

3 Derivations

3.1 High Level Design

Our approaches work as reductions, achieving the value- and time-uniform guarantee of Equation (2) by combining bounds Λ t , Ξ t that satisfy a time-uniform guarantee at any fixed value ρ ,

The bounds Λ t , Ξ t are tools for estimating a sequence of scalars , in this case p CDF t p ρ qq 8 t ' 1 for a fixed value ρ . We show how to extend such tools to the more difficult problem of estimating a sequence of (cumulative distribution) functions .

There are multiple existing approaches to obtaining the guarantee of Equation (3): we provide a selfcontained introduction in Appendix A. For ease of exposition, we will only discuss how to construct

a time- and value-uniform upper bound by combining fixed-value, time-uniform upper bounds, and defer the analogous lower bound construction to Appendix B.2. Our approach is to compose these fixed-value bounds into a value-uniform bound by taking a union bound over a particular collection of values, leveraging monotonicity of the CDF.

Quantile vs Value Space In the i.i.d. setting, a value-uniform guarantee can be obtained by taking a careful union bound over the unique value associated with each quantile [Howard and Ramdas, 2022]. This 'quantile space' approach has advantages, e.g., variance based discretization and covariance to monotonic transformations. However, under arbitrary data dependence, the value associated with each quantile can change. Therefore we proceed in 'value space'. See Appendix A.1 for more details.

3.2 On the Unit Interval

Algorithm 1, visualized in Figure 1, constructs an upper bound on Equation (1) which, while valid for all values, is designed for random variables ranging over the unit interval. For a given value v , it searches over upper bounds on the CDF evaluated at a decreasing sequence of values ρ 1 ě ρ 2 ě ¨ ¨ ¨ ě v and exploits monotonicity of CDF t p v q . That is, at each level d ' 1 , 2 , . . . , we construct a discretizing grid of size ϵ p d q over the unit interval, and construct a time-uniform upper bound on CDF t p ρ q for each grid point ρ using the fixed-value confidence sequence oracle Ξ t . Then, for a given value v , at each level d we make use of the fixed-value confidence sequence for smallest grid point ρ d ě v , and we search for the level d which yields the minimal upper confidence bound. A union bound over the (countably infinite) possible choices for ρ d controls the coverage of the overall procedure. Because the error probability δ d decreases with d (and the fixed-value confidence radius Ξ t increases as δ decreases), the procedure can terminate whenever no observations remain between the desired value v and the current upper bound ρ d , as all subsequent bounds are dominated.

The lower bound is derived analogously in Algorithm 2 (which we have left to Appendix B.2 for the sake of brevity) and leverages a lower confidence sequence Λ t p ρ ; δ, Ψ t q (instead of an upper confidence sequence) evaluated at an increasingly refined lower bound on the value ρ Ð ϵ p d q ´ 1 t ϵ p d q v u .

Theorem 3.1. If ϵ p d q Ò 8 as d Ò 8 , then Algorithms 1 and 2 terminate with probability one. Furthermore, if for all ρ , δ , and d the algorithms Λ t p ρ ; δ, Ψ t q and Ξ t p ρ ; δ, Ψ t q satisfy

then guarantee (2) holds with U t , L t given by the outputs of Algorithms 1 and 2, respectively.

Theorem 3.1 ensures Algorithms 1 and 2 yield the desired time- and value-uniform coverage, essentially due to the union bound and the coverage guarantees of the oracles Ξ t , Λ t . However, coverage is also guaranteed by the trivial bounds 0 ď CDF t p v q ď 1 . The critical question is: what is the bound width?

Smoothed Regret Guarantee Even assuming X is entirely supported on the unit interval, on what distributions will Algorithm 1 provide a non-trivial bound? Because each r Λ t p ρ ; δ, Ψ t q , Ξ t p ρ ; δ, Ψ t qs is a confidence sequence for the mean of the bounded random variable 1 X s ď ρ , we enjoy width guarantees at each of the (countably infinite) ρ which are covered by the union bound, but the guarantee degrades as the depth d increases. If the data generating process focuses on an increasingly small part of the unit interval over time, the width guarantees on our discretization will be insufficient to determine the distribution. Indeed, explicit constructions demonstrating the lack of sequential uniform convergence of linear threshold functions increasingly focus in this manner [Block et al., 2022].

Conversely, if @ t : CDF t p v q was Lipschitz continuous in v , then our increasingly granular discretization would eventually overwhelm any fixed Lipschitz constant and guarantee uniform convergence. Theorem 3.3 expresses this intuition, but using the concept of smoothness rather than Lipschitz, as smoothness will allow us to generalize further [Rakhlin et al., 2011, Haghtalab et al., 2020, 2022b,a, Block et al., 2022].

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10

t

=100

t

=1000

t

=10000

i.i.d. Beta(6, 3)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true

CDF

1000,

X

0

bounds at

t

=1000,

X

0

true

CDF

1000,

X

1

bounds at

t

=1000,

X

1

t

1/2

Definition 3.2. A distribution D is ξ -smooth wrt reference measure M if D ! M and ess sup M p dD { dM q ď ξ ´ 1 .

When the reference measure is the uniform distribution on the unit interval, ξ -smoothness implies an ξ ´ 1 -Lipschitz CDF. However, when the reference measure has its own curvature, or charges points, the concepts diverge. When reading Theorem 3.3, note ξ ď 1 (since the reference measure is a probability distribution) and as ξ Ñ 0 the smoothness constraint is increasingly relaxed. Thus Theorem 3.3 states 'for less smooth distributions, convergence is slowed.'

Theorem 3.3. Let U t p v q and L t p v q be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively, when evaluated with ϵ p d q ' 2 d and the confidence sequences Λ t and Ξ t of Equation (15) . If @ t : P t is ξ t -smooth wrt the uniform distribution on the unit interval then

where q t . ' CDF t p v q ; V t . ' 1 { t ` p q t ´ 1 { 2 q { log p q t { 1 ´ q t q ; and ˜ O pq elides polylog V t factors.

Theorem 3.3 matches our empirical results in two important aspects: (i) logarithmic dependence upon smoothness (e.g.' Figure 4); (ii) tighter intervals for more extreme quantiles (e.g., Figure 2). Note the choice ϵ p d q ' 2 d ensures the loop in Algorithm 1 terminates after at most log 2 p ∆ q iterations, where ∆ is the minimum difference between two distinct realized values.

Worked Example To build intiution, in Appendix B.1 we explicitly calculate Algorithm 1 for a synthetic data set.

3.3 Extensions

Arbitrary Support In Appendix D.1 we describe a variant of Algorithm 1 which uses a countable dense subset of the entire real line. It enjoys a similar guarantee to Theorem 3.3, but with an additional width which is logarithmic in the probe value v :

Note in this case ξ t is defined relative to (unnormalized) Lebesgue measure and can therefore exceed 1.

Discrete Jumps If P t is smooth wrt a reference measure which charges a countably infinite number of known discrete points, we can explicitly union bound over these additional points proportional to their density in the reference measure. In this case we preserve the above value-uniform guarantees. See Appendix D.2 for more details.

For distributions which charge unknown discrete points, we note the proof of Theorem 3.3 only exploits smoothness local to v . Therefore if the set of discrete points is nowhere dense, we eventually recover the guarantee of Equation (6) after a 'burn-in' time t which is logarithmic in the minimum distance from v to a charged discrete point.

10

15

10

13

10

11

10

9

10

7

10

5

10

3

10

1

1.0

1.5

2.0

2.5

3.0

3.5

inf

{

t

t

ref

:

||

Ut

Lt

||

ref

}

Time Uniform DKW

3.4 Importance-Weighted Variant

An important use case is estimating a distribution based upon observations produced from another distribution with a known shift, e.g., arising in transfer learning [Pan and Yang, 2010] or off-policy evaluation [Waudby-Smith et al., 2022]. In this case the observations are tuples p W t , X t q , where the importance weight W t is a Radon-Nikodym derivative, implying @ t : E t r W t s ' 1 and a.s. W t ě 0 ; and the goal is to estimate CDF t p v q ' t ´ 1 ř s ď t E s ´ 1 r W s 1 X s ď v s . The basic approach in Algorithm 1 and Algorithm 2 is still applicable in this setting, but different Λ t and Ξ t are required. In Appendix E we present details on two possible choices for Λ t and Ξ t : the first is based upon the empirical Bernstein construction of Howard et al. [2021], and the second based upon the DDRM construction of Mineiro [2022]. Both constructions leverage the L ˚ Adagrad bound of Orabona [2019] to enable lazy evaluation. The empirical Bernstein version is amenable to analysis and computationally lightweight, but requires finite importance weight variance to converge (the variance bound need not be known, as the construction adapts to the unknown variance). The DDRM version requires more computation but produces tighter intervals. See Section 4.1 for a comparison.

Inspired by the empirical Bernstein variant, the following analog of Theorem 3.3 holds. Note P t is the target (importance-weighted) distribution, not the observation (non-importance-weighted) distribution.

Theorem 3.4. Let U t p v q and L t p v q be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively with ϵ p d q ' 2 d and the confidence sequences Λ t and Ξ t of Equation (18) . If @ t : P t is ξ t -smooth wrt the uniform distribution on the unit interval then

where q t . ' CDF t p v q , K p q t q . ' p q t ´ 1 { 2 q { log p q t { 1 ´ q t q ; V t ' O ` K p q t q ř s ď t W 2 s ˘ , B t . ' t ´ 1 ř s ď t p W s ´ 1 q , and ˜ O pq elides polylog V t factors.

Theorem 3.4 exhibits the following key properties: (i) logarithmic dependence upon smoothness; (ii) tighter intervals for extreme quantiles and importance weights with smaller quadratic variation; (iii) no explicit dependence upon importance weight range; (iv) asymptotic zero width for importance weights with sub-linear quadratic variation.

Additional Remarks First, the importance-weighted average CDF is a well-defined mathematical quantity, but the interpretation as a counterfactual distribution of outcomes given different actions in the controlled experimentation setting involves subtleties: we refer the interested reader to WaudbySmith et al. [2022] for a complete discussion. Second, the need for nonstationarity techniques for

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10 1

t

=10 2

t

=10 4

t

=10 6

i.i.d.

X

Beta(6, 3),

W

Pareto(1.5)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true weighted

CDF

10000

bounds at

t

=10000

true unweighted

CDF

10000

Importance-Weighted Continuous Polya Urn

=

t

1/2

,

w

10 0

10 2

10 4

10 6

10 8

10 10

0.00

0.02

0.04

0.06

0.08

0.10

Ut

(

v

)

Lt

(

v

)

i.i.d. LogNormal(0, 1)

LogNormal(10, 1)

Time uniform DKW

t

=10000

estimating the importance-weighted CDF is driven by the outcomes p X t q and not the importanceweights p W t q . For example with off-policy contextual bandits, a changing historical policy does not induce nonstationarity, but a changing conditional reward distribution does.

4 Simulations

These simulations explore the empirical behaviour of Algorithm 1 and Algorithm 2 when instantiated with ϵ p d q ' 2 d and curved boundary oracles Λ and Ξ . To save space, precise details on the experiments as well additional figures are elided to Appendix F. Reference implementations which reproduce the figures are available at https://github.com/microsoft/csrobust .

4.1 The i.i.d. setting

These simulations exhibit our techniques on i.i.d. data. Although the i.i.d. setting does not fully exercise the technique, it is convenient for visualizing convergence to the unique true CDF. In this setting the DKW inequality applies, so to build intuition about our statistical efficiency, we compare our bounds with a naive time-uniform version of DKW resulting from a p 6 { π 2 t 2 q union bound over time.

Beta distribution In this case the data is smooth wrt the uniform distribution on r 0 , 1 s so we can directly apply Algorithm 1 and Algorithm 2. Figure 2 shows the bounds converging to the true CDF as t increases for an i.i.d. Beta p 6 , 3 q realization. Figure 8 compares the bound width to time-uniform DKWat t ' 10000 for Beta distributions that are increasingly less smooth with respect to the uniform distribution. The DKW bound is identical for all, but our bound width increases as the smoothness decreases.

The additional figures in Appendix F clearly indicate tighter bounds at extreme quantiles, in correspondence with Theorem 3.3.

Beyond the unit interval In Figure 7 (main text) and Appendix F.1 we present further simulations of i.i.d. lognormal and Gaussian random variables, ranging over R ` and R respectively, and using Algorithm 3. The logarithmic dependence of the bound width upon the probe value is evident.

An Exhibition of Failure Figure 4 shows the (empirical) relative convergence when the data is simulated i.i.d. uniform over r 0 , ϵ s for decreasing ϵ (hence decreasing smoothness). The reference width is the maximum bound width obtained with Algorithm 1 and Algorithm 2 at t ref ' 10000 and ϵ ' 1 { 16 , and shown is the multiplicative factor of time required for the maximum bound width to match the reference width as smoothness varies. The trend is consistent with arbitrarily poor convergence with arbitrarily small ϵ . Because this is i.i.d. data, DKW applies and a uniform bound (independent of ϵ ) is available. Thus while our instance-dependent guarantees are valuable in practice, they can be dominated by stronger guarantees leveraging additional assumptions. On a positive note, a logarithmic dependence on smoothness is evident over many orders of magnitude, confirming the analysis of Theorem 3.3.

Importance-Weighted In these simulations, in addition to being i.i.d., X t and W t are drawn independently of each other, so the importance weights merely increase the difficulty of ultimately estimating the same quantity.

In the importance-weighted case, an additional aspect is whether the importance-weights have finite or infinite variance. Figures 5 and 13 demonstrate convergence in both conditions when using DDRM for pointwise bounds. Figures 14 and 15 show the results using empirical Bernstein pointwise bounds. In theory, with enough samples and infinite precision, the infinite variance Pareto simulation would eventually cause the empirical Bernstein variant to reset to trivial bounds, but in practice this is not observed. Instead, DDRM is consistently tighter but also consistently more expensive to compute, as exemplified in Table 2. Thus either choice is potentially preferable.

4.2 Nonstationary

i.e., X t is Beta distributed with parameters becoming more extreme over time: each realization will increasingly concentrate either towards 0 or 1 . Suppose γ t ' t q . In the most extreme case that ` t ' ř s ď t 1 X s ą 1 { 2 ˘ , the conditional distribution at time t is Beta p x ; 2 ` tγ t , 2 q ' O p t 1 ` q q , hence d P t { dU ' O p t 1 ` q q , which is smooth enough for our bounds to converge. Figure 3 shows the bounds covering the true CDF for two realizations with different limits. Figure 12 shows (for one realization) the maximum bound width, scaled by a t { log p t q to remove the primary trend, as a function of t for different γ t schedules.

Importance-Weighted Continuous Polya Urn In this case W t is drawn i.i.d. either W t ' 0 or W t ' w max , such as might occur during off-policy evaluation with an epsilon-greedy logging policy. Given W t , the distribution of X t is given by

i.e., each importance weight runs an independent Continuous Polya Urn. Because of this, it is possible for the unweighted CDF to mostly concentrate at one limit (e.g., 1) but the weighted CDF to concentrate at another limit (e.g., 0). Figure 6 exhibits this phenomenon.

5 Related Work

Constructing nonasymptotic confidence bands for the cumulative distribution function of i.i.d. random variables is a classical problem of statistical inference dating back to Dvoretzky et al. [1956] and Massart [1990]. While these bounds are quantile-uniform, they are ultimately fixed-time bounds

(i.e. not time-uniform). In other words, given a sample of i.i.d. random variables X 1 , . . . , X n ' F , these fixed time bounds r 9 L n p x q , 9 U n p x qs x P R satisfy a guarantee of the form:

for any desired error level α P p 0 , 1 q . Howard and Ramdas [2022] developed confidence bands r s L t p x q , s U t p x qs x P R ,t P N that are both quantileand time-uniform, meaning that they satisfy the stronger guarantee:

However, the bounds presented in Howard and Ramdas [2022] ultimately focused on the classical i.i.d. on-policy setup, meaning the CDF for which confidence bands are derived is the same CDF as those of the observations p X t q 8 t ' 1 . This is in contrast to off-policy evaluation problems such as in randomized controlled trials, adaptive A/B tests, or contextual bandits, where the goal is to estimate a distribution different from that which was collected (e.g. collecting data based on a Bernoulli experiment with the goal of estimating the counterfactual distribution under treatment or control). Chandak et al. [2021] and Huang et al. [2021] both introduced fixed-time (i.e. non-time-uniform) confidence bands for the off-policy CDF in contextual bandit problems, though their procedures are quite different, rely on different proof techniques, and have different properties from one another. Waudby-Smith et al. [2022, Section 4] later developed time-uniform confidence bands in the off-policy setting, using a technique akin to Howard and Ramdas [2022, Theorem 5] and has several desirable properties in comparison to Chandak et al. [2021] and Huang et al. [2021] as outlined in Waudby-Smith et al. [2022, Table 2].

Nevertheless, regardless of time-uniformity or on/off-policy estimation, all of the aforementioned prior works assume that the distribution to be estimated is fixed and unchanging over time . The present paper takes a significant departure from the existing literature by deriving confidence bands that allow the distribution to change over time in a data-dependent manner, all while remaining time-uniform and applicable to off-policy problems in contextual bandits. Moreover, we achieve this by way of a novel stitching technique which is closely related to those of Howard and Ramdas [2022] and Waudby-Smith et al. [2022].

6 Discussion

This work constructs bounds by tracking specific values, in contrast with i.i.d. techniques which track specific quantiles. The value-based approach is amenable to proving correctness qua Theorem 3.1, but has the disadvantage of sensitivity to monotonic transformations. We speculate it is possible to be covariant to a fixed (wrt time) but unknown monotonic transformation without violating known impossibility results. A technique with this property would have increased practical utility.

Acknowledgments and Disclosure of Funding

The authors thank Ian Waudby-Smith for insightful discussion and review.

References

Adam Block, Yuval Dagan, Noah Golowich, and Alexander Rakhlin. Smoothed online learning is as easy as statistical learning. arXiv preprint arXiv:2202.04690 , 2022.

Francesco Paolo Cantelli. Sulla determinazione empirica delle leggi di probabilita. Giorn. Ist. Ital. Attuari , 4(421-424), 1933.

Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. Advances in Neural Information Processing Systems , 34: 27475-27490, 2021.

Ioannis Chatzigeorgiou. Bounds on the lambert function and their application to the outage analysis of user cooperation. IEEE Communications Letters , 17(8):1505-1508, 2013.

Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics , pages 642-669, 1956.

Xiequan Fan, Ion Grama, and Quansheng Liu. Exponential inequalities for martingales with applications. Electronic Journal of Probability , 20:1-22, 2015.

William Feller. An introduction to probability theory and its applications, 3rd edition . Wiley series in probability and mathematical statistics, 1958.

Valery Glivenko. Sulla determinazione empirica delle leggi di probabilita. Gion. Ist. Ital. Attauri. , 4: 92-99, 1933.

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis of Online and Differentially Private Learning. In Advances in Neural Information Processing Systems , volume 33, pages 9203-9215, 2020.

Nika Haghtalab, Yanjun Han, Abhishek Shetty, and Kunhe Yang. Oracle-Efficient Online Learning for Beyond Worst-Case Adversaries, November 2022a. arXiv:2202.08549 [cs, stat].

Nika Haghtalab, Tim Roughgarden, and Abhishek Shetty. Smoothed Analysis with Adaptive Adversaries. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages 942-953, 2022b.

Steven R Howard and Aaditya Ramdas. Sequential estimation of quantiles with applications to A/B testing and best-arm identification. Bernoulli , 28(3):1704-1728, 2022.

Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics , 49(2):1055-1080, 2021.

Audrey Huang, Liu Leqi, Zachary Lipton, and Kamyar Azizzadenesheli. Off-policy risk assessment in contextual bandits. Advances in Neural Information Processing Systems , 34:23714-23726, 2021.

Michael Kearns and Lawrence Saul. Large deviation methods for approximate probabilistic inference. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence , pages 311319, 1998.

Michael Lindon, Chris Sanden, and Vaché Shirikian. Rapid regression detection in software deployments through sequential testing. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3336-3346, 2022.

Wentao Liu. Risk-Aware Financial Portfolio Management with Distributional Deep Deterministic Policy Gradient . PhD thesis, University of Toronto (Canada), 2021.

Pascal Massart. The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. The annals of Probability , pages 1269-1283, 1990.

Paul Mineiro. A lower confidence sequence for the changing mean of non-negative right heavy-tailed observations with bounded mean. arXiv preprint arXiv:2210.11133 , 2022.

Frank WJ Olver, Daniel W Lozier, Ronald F Boisvert, and Charles W Clark. NIST handbook of mathematical functions hardback and CD-ROM . Cambridge university press, 2010.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering , 22(10):1345-1359, 2010.

Iosif Pinelis. Exact lower and upper bounds on the incomplete gamma function. arXiv preprint arXiv:2005.06384 , 2020.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online Learning: Stochastic, Constrained, and Smoothed Adversaries. In Advances in Neural Information Processing Systems , volume 24, 2011.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Sequential complexities and uniform martingale laws of large numbers. Probability theory and related fields , 161(1):111-153, 2015.

Glenn Shafer, Alexander Shen, Nikolai Vereshchagin, and Vladimir Vovk. Test martingales, Bayes factors and p-values. Statistical Science , 26(1):84-101, 2011.

Yi Shen, Jessilyn Dunn, and Michael M Zavlanos. Risk-averse multi-armed bandits with unobserved confounders: A case study in emotion regulation in mobile health. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 144-149. IEEE, 2022.

Yuheng Wang and Margaret P Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control. Artificial Intelligence , page 103743, 2022.

Ian Waudby-Smith, Lili Wu, Aaditya Ramdas, Nikos Karampatziakis, and Paul Mineiro. Anytimevalid off-policy inference for contextual bandits. arXiv preprint arXiv:2210.10768 , 2022.

A Confidence Sequences for Fixed v

Since our algorithm operates via reduction to pointwise confidence sequences, we provide a brief self-contained review here. We refer the interested reader to Howard et al. [2021] for a more thorough treatment.

A confidence sequence for a random process X t is a time-indexed collection of confidence sets CI t with a time-uniform coverage property P p@ t P N : X t P CI t q ě 1 ´ α . For real random variables, the concept of a lower confidence sequence can be defined via P p@ t P N : X t ě L t q ě 1 ´ α , and analogously for upper confidence sequences; and a lower and upper confidence sequence can be combined to form a confidence sequence CI t . ' t x | L t ď x ď U t u with coverage p 1 ´ 2 α q via a union bound.

One method for constructing a lower confidence sequence for a real valued parameter z is to exhibit a real-valued random process E t p z q which, when evaluated at the true value z ˚ of the parameter of interest, is a non-negative supermartingale with initial value of 1, in which case Ville's inequality ensures P ` @ t P N : E t p z ˚ q ď α ´ 1 ˘ ě 1 ´ α . If the process E t p z q is monotonically increasing in z , then the supremum of the lower contour set L t . ' sup z ␣ z | E t p z q ď α ´ 1 ( is suitable as a lower confidence sequence; an upper confidence sequence can be analogously defined.

We use the above strategy as follows. We bound these deviations using the following nonnegative martingale,

where λ P R is fixed and h p λ, z q . ' p 1 ´ z q e ´ λz ` ze λ p 1 ´ z q , the moment-generating function of a centered Bernoulli( z q random variable. Equation (10) is a test martingale qua Shafer et al. [2011], i.e., it can be used to construct time-uniform bounds on ˆ q t ´ q t via Ville's inequality.

Next we lower bound Equation (10),

and eliminate the explicit dependence upon θ s , by noting log h p λ, ¨q is concave and therefore

because ˆ tf p q q ' max θ ˇ ˇ 1 J θ ' tq ř s ď t f p θ s q ˙ for any concave f . Equation (11) is monotonically increasing in q t and therefore defines a lower confidence sequence. For an upper confidence sequence we use q t ' 1 ´p 1 ´ q t q and a lower confidence sequence on p 1 ´ q t q .

Regarding the choice of λ , in practice many λ are (implicitly) used via stitching (i.e., using different λ in different time epochs and majorizing the resulting bound in closed form) or mixing (i.e., using a particular fixed mixture of Equation (11) via a discrete sum or continuous integral over λ ); our choices will depend upon whether we are designing for tight asymptotic rates or low computational footprint. We provide specific details associated with each theorem or experiment.

Note Equation (11) is invariant to permutations of X 1: t and hence the empirical CDF at time t is a sufficient statistic for calculating Equation (11) at any v .

A.1 Challenge with quantile space

In this section assume all CDFs are invertible for ease of exposition.

In the i.i.d. setting, Equation (10) can be evaluated at the (unknown) fixed v p q q which corresponds to quantile q . Without knowledge of the values, one can assert the existence of such values for a countably infinite collection of quantiles and a careful union bound of Ville's inequality on a particular discretization can yield an LIL rate: this is the approach of Howard and Ramdas [2022]. A key advantage of this approach is covariance to monotonic transformations.

Beyond the i.i.d. setting, one might hope to analogously evaluate Equation (10) at an unknown fixed value v t p q q which for each t corresponds to quantile q . Unfortunately, v t p q q is not just unknown,

but also unpredictable with respect to the initial filtration, and the derivation that Equation (10) is a martingale depends upon v being predictable. In the case that X t is independent but not identically distributed, v t p q q is initially predictable and therefore this approach could work, but would only be valid under this assumption.

The above argument does not completely foreclose the possibility of a quantile space approach, but merely serves to explain why the authors pursued a value space approach in this work. We encourage the interested reader to innovate.

B Unit Interval Bounds

B.1 Worked Example

Our (synthetic) data set consists of five values, each of which has occurred 1000 times:

We use resolution ϵ p d q ' 2 d and coverage error α ' 1 { 20 .

Upper bound for v ' 4 { 7 The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point ρ 1 ' 2 ´ 1 r 2 1 v s ' 1 again yields the trivial bound of 1. There are still empirical counts between the probe value ( v ' 4 { 7 ) and the bound value ( ρ 1 ' 1 ) so the algorithm continues. The second evaluated point ρ 2 ' 2 ´ 2 r 2 2 v s ' 3 { 4 , for which there are 4000 empirical counts below ρ 2 out of 5000 total. The pointwise confidence sequence is evaluated with counts p 4000 , 5000 q and coverage error δ 2 ' α { 24 , resulting in improved bound « 0 . 825 . Now, there are no empirical counts between the probe value ( v ' 4 { 7 q and the bound value ( ρ 2 ' 3 { 4 ) so the algorithm terminates. To see all subsequent bounds are dominated, note that a tighter upper bound ρ d ą 2 will result in the same empirical counts (4000 out of 5000) but a looser coverage error and hence worse bound.

Upper bound for v ' 13 { 28 The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point ρ 1 ' 2 ´ 1 r 2 1 v s ' 1 { 2 , for which there are 4000 empirical counts below ρ 2 out of 5000 total. The pointwise confidence sequence is evaluated with counts p 4000 , 5000 q and coverage error δ 1 ' α { 4 , resulting in improved bound « 0 . 822 . There are no empirical counts between the probe value p v ' 13 { 28 q and the bound value p ρ 1 ' 1 { 2 q so the algorithm terminates. Relative to the previous example, the bound is slightly tighter as the discretization worked better for this v .

Upper bound for v ' 5 { 14 The upper bound algorithm starts with the trivial upper bound of 1. The first evaluated point ρ 1 ' 2 ´ 1 r 2 1 v s ' 1 { 2 , for which there are 4000 empirical counts below ρ 2 out of 5000 total. The pointwise confidence sequence is evaluated with counts p 4000 , 5000 q and coverage error δ 1 ' α { 4 , resulting in improved bound « 0 . 822 . There are empirical counts between the probe value p v ' 5 { 14 q and the bound value p ρ 1 ' 1 { 2 q so the algorithm continues. The second evaluated point ρ 2 ' 2 ´ 2 r 2 2 v s ' 1 { 2 , which has the same empirical counts but worse coverage error at this level, and hence does not improve the bound. There are empirical counts between the probe value p v ' 5 { 14 q and the bound value p ρ 2 ' 1 { 2 q so the algorithm continues. The third evaluated point ρ 3 ' 2 ´ 3 r 2 3 v s ' 3 { 8 , for which there are 3000 empirical counts below ρ 3 out of 5000 total. The pointwise confidence sequence is evaluated with counts p 3000 , 5000 q and coverage error δ 3 ' α { 96 , resulting in improved bound « 0 . 633 . There are no empirical counts between the probe value p v ' 5 { 14 q and the bound value p ρ 3 ' 3 { 8 q so the algorithm terminates.

B.2 Lower Bound

Algorithm 2 is extremely similar to Algorithm 1: the differences are indicated in comments. Careful inspection reveals the output of Algorithm 1, U t p v q , can be obtained from the output of Algorithm 2, L t p v q , via U t p v q ' 1 ´ L t p 1 ´ v q ; but only if the sufficient statistics are adjusted such that Ξ t p ρ d ; δ, Ψ t q ' 1 ´ Λ t p 1 ´ ρ d ; δ, Ψ 1 t q . The reference implementation uses this strategy.

B.3 Proof of Theorem 3.1

We prove the results for the upper bound Algorithm 1; the argument for the lower bound Algorithm 2 is similar.

Algorithm 2 Unit Interval Lower Bound. ϵ p d q is an increasing function specifying the resolution of discretization at level d . Λ t p ρ ; δ, d, Ψ t q is a lower confidence sequence for fixed value ρ with coverage at least p 1 ´ δ q .

The algorithm terminates when we find a d such that 0 ' ř s ď t 1 X s Pp v,ρ d s . Since ϵ p d q Ò 8 as d Ò 8 , we have ρ d ' ϵ p d q r ϵ p d q ´ 1 v s Ó v , so that ř s ď t 1 X s Pp v,ρ d s Ó 0 . So the algorithm must terminate.

At level d , we have ϵ p d q confidence sequences. The i th confidence sequence at level d satisfies

Taking a union bound over all confidence sequences at all levels, we have

Thus we are assured that, for any v P R ,

Algorithm 1 will return Ξ t p ρ d ; δ d , d, Ψ t q for some d unless all such values are larger than one, in which case it returns the trivial upper bound of one. This proves the upper-bound half of guarantee (2). A similar argument proves the lower-bound half, and union bound over the upper and lower bounds finishes the argument.

C Proof of Theorem 3.3

Theorem 3.3. Let U t p v q and L t p v q be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively, when evaluated with ϵ p d q ' 2 d and the confidence sequences Λ t and Ξ t of Equation (15) . If @ t : P t is ξ t -smooth wrt the uniform distribution on the unit interval then

where q t . ' CDF t p v q ; V t . ' 1 { t ` p q t ´ 1 { 2 q { log p q t { 1 ´ q t q ; and ˜ O pq elides polylog V t factors.

Note v is fixed for the entire argument below, and ξ t denotes the unknown smoothness parameter at time t .

We will argue that the upper confidence radius U t p v q ´ t ´ 1 ř s ď t 1 X s ď v has the desired rate. An analogous argument applies to the lower confidence radius t ´ 1 ř s ď t 1 X s ď v ´ L t p v q , and the confidence width U t p v q ´ L t p v q is the sum of these two.

For the proof we introduce an integer parameter η ě 2 which controls both the grid spacing ( ϵ p d q ' η d ) and the allocation of error probabilities to levels ( δ d ' α {p η d ϵ p d qq ). In the main paper we set η ' 2 .

At level d we construct η d confidence sequences on an evenly-spaced grid of values 1 { η d , 2 { η d , . . . , 1 . We divide total error probability α { η d at level d among these η d confidence sequences, so that each individual confidence sequence has error probability α { η 2 d .

For a fixed bet λ and value ρ , S t defined in Section 3.2 is sub-Bernoulli qua Howard et al. [2021, Definition 1] and therefore sub-Gaussian with variance process V t . ' tK p q t q , where K p p q . ' p 2 p ´ 1 q { 2 log p p { 1 ´ p q is from Kearns and Saul [1998]; from Howard et al. [2021, Proposition 5] it follows that there exists an explicit mixture distribution over λ such that

is a (curved) uniform crossing boundary, i.e., satisfies

where S t . ' CDF t p ρ q ´ t ´ 1 ř s ď t 1 X s ď ρ is from Equation (10), and τ is a hyperparameter to be determined further below.

Because the values at level d are 1 { η d apart, the worst-case discretization error in the estimated average CDF value is

and the total worst-case confidence radius including discretization error is

Now evaluate at d such that ? ψ t ă ξ t η d ď η ? ψ t where ψ t . ' t p K p q t q ` τ { t q ´ 1 ,

The final result is not very sensitive to the choice of τ , and we use τ ' 1 in practice.

D Extensions

D.1 Arbitrary Support

Algorithm 3 is a variation on Algorithm 1 which does not assume a bounded range, and instead uses a countably discrete dense subset of the entire real line. Using the same argument of Theorem 3.3 with the modified probability from the modified union bound, we have

demonstrating a logarithmic penalty in the probe value v (e.g., Figure 7).

Algorithm 3 Entire Real Line Upper Bound. ϵ p d q is an increasing function specifying the resolution of discretization at level d . Ξ t p ρ ; δ, d, Ψ t q is an upper confidence sequence for fixed value ρ with coverage at least p 1 ´ δ q .

Sub-optimality of k d The choice of k d in Algorithm 3 is amenable to analysis, but unlike in Algorithm 1, it is not optimal. In Algorithm 1 the probability is allocated uniformly at each depth, and therefore the closest grid point provides the tightest estimate. However in Algorithm 3, the probability budget decreases with | k d | and because k d can be negative, it is possible that a different k d can produce a tighter upper bound. Since every k d is covered by the union bound, in principle we could optimize over all k d but it is unclear how to do this efficiently. In our implementation we do not search over all k d , but we do adjust k d to be closest to the origin with the same empirical counts.

D.2 Discrete Jumps

Known Countably Infinite Suppose D is smooth wrt a reference measure M , where M is of the form

with I a countable index set, 1 ě ř i P I ζ i and ˘ M a sub-probability measure normalizing to p 1 ´ ř i P I ζ i q . Then we can allocate p 1 ´ ř i P I ζ i q of our overall coverage probability to bounding ˘ M using Algorithm 1 and Algorithm 2. For the remaining t v i u i P I we can run explicit pointwise bounds each with coverage probability fraction ζ i .

Computationally, early termination of the infinite search over the discrete bounds is possible. Suppose (wlog) I indexes ζ in non-increasing order, i.e., i ď j ùñ ζ i ď ζ j : then as soon as there are no remaining empirical counts between the desired value v and the most recent discrete value v i , the search over discrete bounds can terminate.

E Importance-Weighted Variant

E.1 Modified Bounds

Algorithm 1 and Algorithm 2 are unmodified, with the caveat that the oracles Λ t and Ξ t must now operate on an importance-weighted realization p W 1: t , X 1: t q , rather then directly on the realization X 1: t .

E.1.1 DDRMVariant

For simplicity we describe the lower bound Λ t only. The upper bound is derived analogously via the equality Y s ' W s ´p W s ´ Y s q and a lower bound on p W s ´ Y s q : see Waudby-Smith et al. [2022, Remark 3] for more details.

This is the Heavy NSM from Mineiro [2022] combined with the L ˚ bound of Orabona [2019, §4.2.3]. The Heavy NSM allow us to handle importance weights with unbounded variance, while the Adagrad L ˚ bound facilitates lazy evaluation.

For fixed v , let Y t ' W t 1 X t ě v be a non-negative real-valued discrete-time random process, let ˆ Y t P r 0 , 1 s be a predictable sequence, and let λ P r 0 , 1 q be a fixed scalar bet. Then

is a test supermartingale [Mineiro, 2022, §3]. Manipulating,

where for p:q we use a no-regret learner on h pq with regret Reg p t q to any constant prediction ˆ Y ˚ t P r 0 , 1 s . The function h pq is M -smooth with M ' λ 2 p 1 ´ λ q 2 so we can get an L ˚ bound [Orabona, 2019, §4.2.3] of

thus essentially our variance process is inflated by a square-root. In exchange we do not have to actually run the no-regret algorithm, which eases the computational burden. We can compete with any in-hindsight prediction: if we choose to compete with the clipped running mean Y t then we end up with

which is implemented in the reference implementation as LogApprox:getLowerBoundWithRegret(lam) . The λ -s are mixed using DDRM from Mineiro [2022, Thm. 4], implemented via the DDRM class and the getDDRMCSLowerBound method in the reference implementation. getDDRMCSLowerBound provably correctly early terminates the infinite sum by leveraging

as seen in the termination criterion of the inner method logwealth(mu) .

To minimize computational overhead, we can lower bound log p a ` b q for b ě 0 using strong concavity qua Mineiro [2022, Thm. 3], resulting in the following geometrically spaced collection of sufficient statistics:

along with distinct statistics for z ' 0 . k is a hyperparameter controlling the granularity of the discretization (tighter lower bound vs. more space overhead): we use k ' 1 { 4 exclusively in our

experiments. Note the coverage guarantee is preserved for any choice of k since we are lower bounding the wealth.

Given these statistics, the wealth can be lower bounded given any bet λ and any in-hindsight prediction ˆ Y ˚ t via

Thus when accumulating the statistics, for each Y s ' W s 1 X s ě v , a value of α must be accumulated at key f p z l q , a value of p 1 ´ α q accumulated at key f p z u q , and a value of α p 1 ´ α q accumulated at key m p z l q . The LogApprox::update method from the reference implementation implements this.

Because these sufficient statistics are data linear, a further computational trick is to accumulate the sufficient statistics with equality only, i.e., for Y s ' W s 1 X s ' v ; and when the CDF curve is desired, combine these point statistics into cumulative statistics. In this manner only O p 1 q incremental work is done per datapoint; while an additional O p t log p t qq work is done to accumulate all the sufficient statistics only when the bounds need be computed. The method StreamingDDRMECDF::Frozen::__init__ from the reference implementation contains this logic.

E.1.2 Empirical Bernstein Variant

For simplicity we describe the lower bound Λ t only. The upper bound is derived analogously via the equality Y s ' W s ´p W s ´ Y s q and a lower bound on p W s ´ Y s q : see Waudby-Smith et al. [2022, Remark 3] for more details.

This is the empirical Bernstein NSM from Howard et al. [2021] combined with the L ˚ bound of Orabona [2019, §4.2.3]. Relative to DDRM it is faster to compute, has a more concise sufficient statistic, and is easier to analyze; but it is wider empirically, and theoretically requires finite importance weight variance to converge.

For fixed v , let Y t ' W t 1 X t ě v be a non-negative real-valued discrete-time random process, let ˆ Y t P r 0 , 1 s be a predictable sequence, and let λ P r 0 , 1 q be a fixed scalar bet. Then

is a test supermartingale [Mineiro, 2022, §3]. Manipulating,

¨

˛

where S t ' ř s ď t Y s ´ E s ´ 1 r Y s s and for p:q we use a no-regret learner on squared loss on feasible set r 0 , 1 s with regret Reg p t q to any constant in-hindsight prediction ˆ Y ˚ t P r 0 , 1 s . Since Y s is unbounded

above, the loss is not Lipschitz and we can't get fast rates for squared loss, but we can run Adagrad and get an L ˚ bound,

Thus basically our variance process is inflated by an additive square root.

We will compete with Y ˚ t ' min ` 1 , 1 t ř s Y s ˘ .

Akey advantage of the empirical Bernstein over DDRM is the availability of both a conjugate (closedform) mixture over λ and a closed-form majorized stitched boundary. This yields both computational speedup and analytical tractability.

For a conjugate mixture, we use the truncated gamma prior from Waudby-Smith et al. [2022, Theorem 2] which yields mixture wealth

where 1 F 1 p . . . q is Kummer's confluent hypergeometric function and Γ p¨ , ¨q is the upper incomplete gamma function. For the hyperparameter, we use τ ' 1 .

E.2 Proof of Theorem 3.4

Theorem 3.4. Let U t p v q and L t p v q be the upper and lower bounds returned by Algorithm 1 and Algorithm 2 respectively with ϵ p d q ' 2 d and the confidence sequences Λ t and Ξ t of Equation (18) . If @ t : P t is ξ t -smooth wrt the uniform distribution on the unit interval then

where q t . ' CDF t p v q , K p q t q . ' p q t ´ 1 { 2 q { log p q t { 1 ´ q t q ; V t ' O ` K p q t q ř s ď t W 2 s ˘ , B t . ' t ´ 1 ř s ď t p W s ´ 1 q , and ˜ O pq elides polylog V t factors.

Note v is fixed for the entire argument below, and ξ t denotes the unknown smoothness parameter at time t .

We will argue that the upper confidence radius U t p v q ´ t ´ 1 ř s ď t W s 1 X s ď v has the desired rate. An analogous argument applies to the lower confidence radius. One difference from the nonimportance-weighted case is that, to be sub-exponential, the lower bound is constructed from an upper bound on U 1 t p v q ' W s p 1 ´ 1 X s ď v q via L t p v q ´ 1 ´ U 1 t p v q , which introduces an additional B t ' t ´ 1 ř s ď t p W s ´ 1 q term to the width. (Note, because @ t : E t r W t ´ 1 s ' 0 , this term will concentrate, but we will simply use the realized value here.)

For the proof we introduce an integer parameter η ě 2 which controls both the grid spacing ( ϵ p d q ' η d ) and the allocation of error probabilities to levels ( δ d ' α {p η d ϵ p d qq ). In the main paper we set η ' 2 .

At level d we construct η d confidence sequences on an evenly-spaced grid of values 1 { η d , 2 { η d , . . . , 1 . We divide total error probability α { η d at level d among these η d confidence sequences, so that each individual confidence sequence has error probability α { η 2 d .

For a fixed bet λ and value ρ , S t defined in Appendix E.1.2 is sub-exponential qua Howard et al. [2021, Definition 1] and therefore from Lemma E.1 there exists an explicit mixture distribution over λ inducing (curved) boundary

where S t . ' CDF t p ρ q ´ t ´ 1 ř s ď t W s 1 X s ď ρ , and τ is a hyperparameter to be determined further below.

Because the values at level d are 1 { η d apart, the worst-case discretization error in the estimated average CDF value is

and the total worst-case confidence radius including discretization error is

Now evaluate at d such that ? ψ t ă ξ t η d ď η ? ψ t where ψ t . ' t pp τ ` V t q{ t q ´ 1 ,

where ˜ O pq elides polylog V t factors. The final result is not very sensitive to the choice of τ , and we use τ ' 1 in practice.

Lemma E.1. Suppose

is subψ e qua Howard et al. [2021, Definition 1]; then there exists an explicit mixture distribution over λ with hyperparameter τ ą 0 such that

is a (curved) uniform crossing boundary.

Proof. We can form the conjugate mixture using a truncated gamma prior from Howard et al. [2021, Proposition 9], in the form from Waudby-Smith et al. [2022, Theorem 2], which is our Equation (17).

where 1 F 1 p . . . q is Kummer's confluent hypergeometric function. Using Olver et al. [2010, identity 13.6.5],

where Γ p a, x q is the (unregularized) upper incomplete gamma function. From Pinelis [2020, Theorem 1.2] we have

Applying this to the mixture yields

where p:q follows from the self-imposed constraint S t ě C p τ q . This yields crossing boundary

Chatzigeorgiou [2013, Theorem 1] states

Substituting yields

From Feller [1958, Equation (9.8)] we have

and

Combining Equations (19) to (21) yields the crossing boundary

F Simulations

F.1 i.i.d. setting

For non-importance-weighted simulations, we use the Beta-Binomial boundary of Howard et al. [2021] for Λ t and Ξ t . The curved boundary is induced by the test NSM

with prior parameter b ' 1 . Further documentation and details are in the reference implementation csnsquantile.ipynb .

The importance-weighted simulations use the constructions from Appendix E: the reference implementation is in csnsopquantile.ipynb for the DDRM variant and csnsopquantile-ebern.ipynb for the empirical Bernstein variant.

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10

t

=100

t

=1000

t

=10000

i.i.d. Beta(6, 3)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10

t

=100

t

=1000

t

=10000

i.i.d. LogNormal(0, 1)

3

2

1

0

1

2

3

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10

t

=100

t

=1000

t

=10000

i.i.d. Gaussian(0, 1)

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true

CDF

1000,

X

0

bounds at

t

=1000,

X

0

true

CDF

1000,

X

1

bounds at

t

=1000,

X

1

t

1/2

10

4

10

3

10

2

10

1

10 0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Ut

(

v

)

Lt

(

v

)

i.i.d. Beta(6, 3)

i.i.d. Beta(6, 300)

i.i.d. Beta(6, 30000)

Time uniform DKW

t

=10000

10 0

10 2

10 4

10 6

10 8

10 10

0.00

0.02

0.04

0.06

0.08

0.10

Ut

(

v

)

Lt

(

v

)

i.i.d. LogNormal(0, 1)

LogNormal(10, 1)

Time uniform DKW

t

=10000

10 4

10 3

10 2

10 1

10 0

0

10 0

10 1

10 2

10 3

10 4

10 5

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Ut

(

v

)

Lt

(

v

)

i.i.d. Gaussian(0, 1)

Gaussian(100, 100)

Gaussian(10000, 10000)

Time uniform DKW

t

=10000

1.9

2.0

2.1

2.2

2.3

2.4

2.5

sup

v

(

Ut

(

v

)

Lt

(

v

)) ×

(

t

log(

t

)

)

t

=

t

1

t

=

t

1/2

t

=

t

0

10 2

10 3

10 4

10 5

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10 1

t

=10 2

t

=10 3

t

=10 4

i.i.d.

X

Beta(6, 3),

W

Exp(1)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10 1

t

=10 2

t

=10 3

t

=10 4

i.i.d.

X

Beta(6, 3),

W

Exp(1)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10 1

t

=10 2

t

=10 4

t

=10 6

i.i.d.

X

Beta(6, 3),

W

Pareto(1.5)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0

0.2

0.4

0.6

0.8

1.0

CDFt

(

v

)

true cdf

t

=10 1

t

=10 2

t

=10 4

t

=10 6

i.i.d.

X

Beta(6, 3)

W

Pareto(1.5)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for constructing time-uniform and value-uniform bounds on the CDF of the averaged historical distribution of a real valued random process. They propose a design principle AVAST (always valid and sometimes trivial), implying that their bounds always have coverage, but may converge to 0 slowly. The authors build upon the work of [Howard and Ramdas 2022], extending those results to non-stationary processes, and allowing for counterfactual estimation.\n\nSTRENGTHS:\n1. The authors provide a new algorithm for constructing time and value uniform bounds on the CDF of the averaged historical distribution of a real valued random process in the non-stationary case. The theoretical guarantees are cleanly provided, and the proofs appear to be correct.\n2. The authors extend these results to importance-weighted random variables.\n3. The authors simulate their algorithm on several synthetic examples, showcasing its adaptive performance.\n\nWEAKNESSES:\nThe one point that is not clear to me is the instance-adaptivity of Theorem 3.3. Is this a constant bound $\\xi_t$ constant for all $t$, i.e. $\\xi_t \\le \\xi$ for all $t$? If not, how is it that the bound is time-uniform while having no dependence on previous $\\xi_t$? It seems as though if $X_1,\\dots,X_{100}$ were i.i.d. $U[0,\\epsilon]$, and $X_{101}$ is independent $U[0,1]$, then the confidence interval width $U_{101}(\\epsilon/2)-L_{101}(\\epsilon/2)$ would depend critically on $\\epsilon$. Simulations would also be helpful to clarify this point.\n\nQUESTIONS:\nSee weaknesses, the instance-adaptivity of Theorem 3.3 is unclear. This work seems polished and well-written aside from this, but this result seems like a critical part of the paper. I am leaving a score of weak reject, but if this point is clarified I would be happy to see this paper accepted.\n\nLIMITATIONS:\nLine 79: leveraging *the* monotonicity\nline 137: two commas after e.g.\nLine 427: For a fixed \"bet\" $\\lambda$. It is not explained why $\\lambda$ should be treated as a bet.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the problem of estimating the CDF of a sequence of random variables obtained sequentially according to some distributions.\nMany classical results are known for this problem when the random variable sequence is obtained in an i.i.d. manner following a certain distribution, along with its confidence bands.\nOn the other hand, some impossibility results are known for the case where the random variables are dependent on each other, and their confidence bands are not sufficiently known.\nTo address this issue, the paper constructs confidence bands for CDFs with running averaged conditional distributions that are uniform in time and value.\nThe extension to importance-weighted random variables is also discussed.\n\nSTRENGTHS:\n- The paper shows how to construct confidence bands in the setting where there is dependence in the sequence of random variables obtained, an important problem in practical applications. This result is further extended to the importance-weighted setting, which is frequently encountered in many online decision making problems.\n- Through a number of experimental results, the paper shows that results that are consistent with the theoretical results are obtained.\n\nWEAKNESSES:\n- The contents of the Introduction seem to be insufficient, because it is not clear what kind of problem is being solved just by following line 12 to line 27, and then the explanation of contirbutions is given immediately after that.\nSection 5 (Related Work) seems to be useful for understanding the problem the authors are trying to solve, so it would be desirable to include it earlier. Even after adding it, the explanation of background knowledge still seems insufficient, so it seems desirable to use the remaining pages to provide more explanation in the current Introduction.\n\n- Although the comparison with existing results is clearly stated, the reviewer could not tell from the text what the major technical difficulties of the problems are. Many of the proofs and contributions are included in the appendix, and it would be desirable to have an explanation of what technical difficulties were solved to obtain the main results, even if they are only sketches.\n\nMinor issues and typos:\n- The horizontal line labels in Figures 1 to 7 are too small to see.\n\nQUESTIONS:\n- As noted in Weaknesses, there are few specific descriptions of technical difficulties, and reviewers expect authors to address them.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new construction of time-uniform confidence bands for CDF, where the standard toolkit such as Glivenko—Cantelli cannot be applied due to nonstationarity.\nAt a high level, the proposed method combines confidence sequence for a certain subset of values via a union bound.\nDespite the impossibility result from the adversarial online learning scenario, the proposed construction of confidence bands is shown to have vanishing width for a class of smooth distributions (Theorem 3.3).\nThis result holds for unbounded random variables (Section 3.3) as well as under a known distribution shift case (Section 3.4).\nThe simulations demonstrate that the proposed algorithms can be effective.\n\nSTRENGTHS:\nThe considered problem of constructing confidence bands for CDFs under nonstationarity is of great practical importance.\nThe proposed technique seems to be new and interesting. The theoretical guarantee in Theorem 3.3 is insightful as it characterizes the adaptivity.\n\nWEAKNESSES:\n- While I believe that the proposed method is new and novel, the writing is hard to follow, which can be much improved.\n- The paper is not very self-contained. For example, Table 1 needs to be elaborated further. For example, there is no description on “$w_{\\max}$-free” in the main text or in the caption.\n- The figures can be revised further. Legends and labels are too small and not very visible. And it might be better to put Figures 2-5 after Section 4 as these are never referred before the experimental section.\n\nQUESTIONS:\nMost of my questions are around Algorithm 1, which seems to contain the key idea of the proposed method. The authors should revise the manuscript so that the key ideas become clear.\n- Is i.i.d.ness is assumed throughout Section 3? It is never explicit except in line 80 before Section 3.4.\n- In Algorithm 1, aren’t the functions $\\epsilon(d)$ and $\\Xi_t$ also “inputs” (or “hyperparameters”) to the algorithm? If so, I think it’d better to put it in the algorithm, not in the caption. Further, I think that making Algorithm 1 illustrative by giving a concrete example of $\\epsilon$ and $\\Xi_t$ may improve readability. Since the paper is extremely notation-heavy, it is not very clear how to parse the algorithm at that abstract level.\n- In Algorithm 1, most of the readers will be confused with $W_{1:t}$, as it hasn’t been defined by then. I can search over the text and find it in Section 3.4 that it means the importance weight, but it only distracts a reader if it’s put in Algorithm 1. \n- In the caption of Figure 1, it is stated that “The algorithm searches over all d to optimize the overall bound via a provably correct early termination criterion”. What do you mean by the “provably correct early termination criterion”? In Algorithm 1, where is this corresponded? And where do you provide a provably correctness?\n\nAs the resulting estimator seems new and interesting, I am inclined to vote for accept, but only provided that the writing is greatly improved in the revision.\n\nLIMITATIONS:\nLimitations are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the time and value uniform bounds on the CDF of the running averaged conditional distribution of a real valued random variable. The new bounds do not require iid setting and always achieve non-asymptotic coverage. The converge speed depends on the smoothness of distribution against the reference distribution, which it is the uniform distribution. Following Howard et al (2021), a confidence sequence of a discrete time random process was employed, and a time uniform coverage property was a key to construct the uniform bounds.\nAuthors established the bounds for unit interval random variables, extended to real random variables then extended to importance weighted random variables. Algorithms for lower/upper bounds and theoretical properties are presented and simulation studies are provided.\n\nSTRENGTHS:\nReliable bounds on the CDF for dependent data setting is valuable as iid setting does not always hold. In order to deal with dependent data, the smoothness of distribution to a reference measure is assumed and the time-value uniform bounds are presented. It is examined numerically for both iid samples and nonstationary data.\n\nWEAKNESSES:\nThis is not my research area and, I found the paper was hard to read and follow. I have an impression that this paper is written assuming that readers have some knowledge and, detailed explanation would be helpful.\n\n\ni) Confidence sequences Lambda and Xi have very important roles but I can’t grab what they are from the Algorithms 1 and 2 and the justification. Although the confidence sequence review is given in Appendix A, it is not straight forward to make connections to Alg 1 and the rest of paper.\n\nii) Consequently, it was difficult to follow proofs without fully understanding the confidence sequences and their theoretical properties. Where are the Eqn 12 from? Could authors derive this from the assumptions and properties of the confidence sequence? \n\niii) Authors did extensive simulation studies with iid samples although the highlight of result is dropping iid assumption. \n\niv) Some abbreviations were not defined. For example, NSM, DKW and DDRM.\n\nv) Careful edit seems to be needed. For example, there are Eqn (10)s, S_t is not defined in Appendix A.\n\nQUESTIONS:\nPlease see the weakness section.\n\nLIMITATIONS:\nAuthors addressed the limitation of the new bounds.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}