Bernstein–von Mises for Adaptively Collected Data
Authors: Kevin Du
Yash Nair
Lucas Janson
Venue: NeurIPS 2025 poster ()
Keywords: Bernstein–von Mises
Uncertainty Quantification
Reinforcement Learning
Multi-armed Bandits
Asymptotic Normality
Frequentist Coverage of Credible Intervals
Primary area: theory

Abstract: Uncertainty quantification (UQ) for adaptively collected data, such as that coming from adaptive experiments, bandits, or reinforcement learning, is necessary for critical elements of data collection such as ensuring safety and conducting after-study inference. The data's adaptivity creates significant challenges for frequentist UQ, yet Bayesian UQ remains the same as if the data were independent and identically distributed (i.i.d.), making it an appealing and commonly used approach. Bayesian UQ requires the (correct) specification of a prior distribution while frequentist UQ does not, but for i.i.d. data the celebrated Bernstein–von Mises theorem shows that as the sample size grows, the prior `washes out' and Bayesian UQ becomes frequentist-valid, implying that the choice of prior need not be a major impediment to Bayesian UQ as it makes no difference asymptotically. This paper for the first time extends the Bernstein–von Mises theorem to adaptively collected data, proving asymptotic equivalence between Bayesian UQ and Wald-type frequentist UQ in this challenging setting. Our results do not require the standard stability condition for validity of Wald-type frequentist UQ, and thus provide positive results on frequentist validity of Bayesian UQ under stability. Counterintuitively however, they also provide a negative result that Bayesian UQ is not asymptotically frequentist valid when stability fails, despite the fact that the prior washes out and Bayesian UQ asymptotically matches standard Wald-type frequentist UQ. We empirically validate our theory (positive and negative) via a range of simulations.

Bernstein-von Mises for Adaptively Collected Data
Kevin Du Department of Statistics
Yash Nair
Department Statistics Stanford University
Harvard University
kevindu@college. www.alleeeeeaaaaeeeo.oomm hawvard. edu
yashnair@stanforde edu
Lucas Janson
Department of Statistics
Harvard University
1j janonnfas. www.aaa...dee.eoe edu
Abstract
Uncertainty quantification (UQ) for adaptively collected data, such as that coming from adaptive experiments, bandits, or reinforcement learning, is necessary for critical elements of data collection such as ensuring safety and conducting after- study inference. The data's adaptivity creates significant challenges for frequentist UQ. yet Bayesian UQ remains the same as if the data were independent and identically distributed (i.i.d.,, making it an appealing and commonly used approach. Bayesian UQ requires the (correct) specification of prior distribution while frequentist UQ does not, but for i.i.d. data the celebrated Bernstein-von Mises theorem shows that as the sample size grows, the prior 'washes out' and Bayesian UQ becomes frequentist-valid implying that the choice of prior need not be a major impediment to Bayesian UQ as it makes no difference asymptotically. This paper for the first time extends the Bernstein-von Mises theorem to adaptively collected data, proving asymptotic equivalence between Bayesian UQ and Wald- type frequentist UQ in this challenging setting. Our result showing this asymptotic agreement does not require the standard stability condition required by works studying validity of Wald-type frequentist UQ: in cases where stability is satisfied, results combined with these prior studies of frequentist UQ imply frequentist validity of Bayesian UQ. Counterintuitively however, they also provide a negative result that Bayesian UQ is not asymptotically frequentist valid when stability fails, despite the fact that the prior washes out and Bayesian UQ asymptotically matches standard Wald-type frequentist UQ We empirically validate theory (positive and negative) via range of simulations.
1 Introduction
Data in applications such as robotics 19] healthcare clinical trials [5], online education (7]. mobile health [2] 31), and online advertising [22] 10 is routinely being collected adaptively. This means that the data is collected sequentially, with decisions about the data collection itself being made online based on all the data observed up to that time point. In particular, the online decisions often try to focus on actions or interventions that prior data indicates will produce large values of some reward function, and this type of data collection includes adaptive experiments, multi-armed bandits, and reinforcement learning.
Two important elements of adaptive data collection are (1) assessing at the end of the data collection what has been learned (e.g., to assess confidence putatively optimal clinical treatment inform a future online advertising campaign) B3 14 and (2) making sure to avoid certain bad outcomes
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

during the data collection process (e.g., avoiding crashing a robot while it is reinforcement learning) [.. Both of these elements, as well as others, rely critically on uncertainty quantification (UQ) of the parameters of the data-generating environment, yet standard frequentist UQ, such as Wald-type inference based on the maximum likelihood estimator (MLF), is made far more complicated by the adaptivity of the data collection 21 which results in data which is not i..dd., Markovian, or even stationary.
On the other hand, the Bayesian statistical paradigm translates seamlessly to the adaptive setting, so that the vast wealth of Bayesian UQ methodology designed for i.i.d. data can be directly applied to adaptively collected data to provide strong Bayesian probabilistic guarantees. However, the validity of those guarantees predicated on correctly specifying a prior distribution for the parameters, which can be challenging to do or justify in practice. For i.i.d. data, the celebrated Bernstein-von Mises (BvM) theorem [29 Theorem 10.1] proves that for any reasonable choice of prior, Bayesian UQ asymptotically matches that of prior-free Wald-type frequentist UQ, providing strong Bayesian and frequentist justification for Bayesian UQ in large samples without the analyst having to worry too much about the choice of prior. Although BvM has been extended beyond i.idd data in number of ways (see Section 2 for more details), it has not been extended to the setting of adaptively collected data.
Contributions This paper for the first time proves BvM results for adaptively collected data, showing that Bayesian UQ asymptotically matches (asymptotically normal) Wald-type frequentist UQ under certain mild conditions. Our first result, in Section 33] applics to a very general class of adaptive linear Gaussian settings that include Gaussian multi-armed bandits, adaptive Gaussian linear bandits (a form of contextual bandit), and the linear quadratic regulator (a form of reinforcement learning on Markov decision process). We then show in Section 4 that in the special case of multi-armed bandits, the conditions of our first result can be weakened, and the Gaussian assumption can also be generalized any exponential family. Finally, we show in Section 5htht the conditions of the first result can also be weakened for linear (contextual) bandits. our theoretical results. A surprising aspect of our BvM results is that they do not require the key stability condition used in frequentist validity results; however, it is known that Wald-type frequentist inference may fail to be asymptotically valid in cases when the stability condition does not hold. The counterintuitive implication is that in some adaptive settings, Bayesian UQ asymptotically matches frequentist UQ (and the prior becomes irrelevant), yet Bayesian UQ is not asymptotically frequentist-valid.
validates
Notation and Terminology. Throughout this paper, we refer to a confidence set as asymptotically frequentist-valid at level a if for any parameter value, the limit inferior of the frequentist coverage at least a. Similarly, we refer to a credible set as asymptotically Bayesian-valid the limit inferior of the Bayesian coverage is at least 1 - a. We will use the term frequentist (resp. Bayesian) UQ generically to refer to any of the many forms of frequentist (resp. Bayesian) statistical inference such as hypothesis tests, confidence intervals, or prediction intervals (resp. such as Bayesian hypothesis tests, credible intervals, or posterior predictive intervals). By Wald-type frequentist UQ, we refer to standard asymptotic hypothesis testing or confidence interval construction using the MLE and its asymptotic Normality- for instance, as demonstrated in Example 15.6 of (29. Let (min and Amax (A) denote the minimum and maximum cigenvalues of the matrix A respectively.
level
2 Background
As mentioned in the previous section, Bayesian statistical inference requires no adjustment for the adaptivity of the data. This both makes the Bayesian approach an appealing and commonly used tool for UQ in adaptively collected data and means there is no need for (and thus a lack of) prior work extending it to the adaptive setting. Thus, prior work on UQ for adaptively collected data has focused on frequentist UQ, with the primary approach being to make assumptions on both the data and the adaptive assignment algorithm by which it is collected. These assumptions either (1) enable the use of martingale central limit theorems to establish asymptotic normality of common frequentist or (2) allow for conservative finite-sample
estimators like the MLE (see, e.g., 21 inference via martingale-based concentration bounds 15 16 I. These conditions are often highly technical and hard check, and reduce to or are related to a stability condition introduced in (1]] see Section 3 for more details. Another line of work uses randomization testing for frequentist UQ
33 14 12 23

---

for adaptively collected data, providing non non-asy asymptotic and non-conservative guarantees but relying on sampling procedures that can be computationally prohibitive use (25]
The classical BvM theorem [29. Theorem 10.1) proves that under minimal conditions, for i.i.d. data drawn from parametric model, the posterior distribution is asymptotically normal centered the MLE with variance equal to the inverse Fisher information, and hence agrees asymptotically with (and inherits the asymptotic frequentist validity of) Wald-style frequentist inference. Prior works generalizing BvM to non-i.i.d. data only apply in settings where the ratio of the maximum and minimum eigenvalues of the Fisher information matrix is bounded [18]
20 9.. This condition fails to hold in most adaptive settings where, over time, certain actions are learned to be better than others and are asymptotically sampled infinitely more often than the worse actions, resulting in an unbounded ratio (e.g.. regret- ootrial multi-arme, bandit algorithms sample suboptimal arms logarithmically often, leading to an eigenvalue ratio that grows like n/ log(n)
ByM on Adaptive Linear Gaussian Data
This paper will primarily consider the adaptive data collection setting, laid out in Algorithm I that at time step j allows the data collector to choose via an arbitrary function covariate vector ry based on all data observed so far (H,-1 ((x1, yı),-.., (Iz-1, V)-1))), and then observe Vj [xx] 8, We assume the only unknown in this sampling procedure is B, and hence the task at hand is to perform UQ for 33 based the full data trajectory Hn (Xn, yn), where X,, is the p matrix with jth row given by 2 and yn is the n-vector with jth entry given by y/-
Algorithm Adaptive Linear Gaussian Sampling Procedure
Input Covariate sampling rule A (Ro x R)* + A(R"), coefficient vector 3 E RP, variance
o² E +
Output Sampling trajectory H,
Ho + 0
for j = 1..,,,,7
Sample zg A(- 11)
Sample yj ~ NN 02)
H1 +- ((I1, (1))1)))
end for
Here, A denotes placeholder that represent any sampling algorithm including UCB, Thompson sampling, autoregressive models, etc. We now present our first result, which states that the posterior distribution is asymptotically normal, centered the MLE with variance equal the inverse empirical Fisher information.
Theorem Suppose (3) is the prior distribution for and T Hw) is the posterior after observing yn be the MLE 3 and let Bo be the true value of B. Assume the sampling procedure Algorithm [ satisfies the following conditions:
the trajectory Hn. Let 3n =
(i) Amin X X,) Bs so and log Xmax [ [ x..) = (x, (X.XX,)))
(ii) T(-) is continuous with positive density at Bo-
Then, the posterior distribution T ()) satisfies
(3 - N (8 Xx, 2 0.
Theorem Iss proof, given in Appendix [... first upper bounds the key total variation (TV) distance in terms of the L' distances of unnormalized densities, allowing to ignore normalizing constants in our analysis. Our proof then uses the fact that the posterior distribution not affected by the adaptivity of the algorithm, since terms in the likelihood involving the covariate sampling process A do not depend on the parameter and can thus be absorbed into the normalizing constant. We also use common trick in ByM-style proofs [18 29 of truncating the posterior to local ellipsoids defined as
3

---

{3: ( from those used in previous BvM-style proofs, as here the empirical Fisher information X,, v x does not necessarily have bounded condition number, implying that the resulting ellipsoids may be highly anisotropic, stretching much wider in some directions than in others. This step of our proof relies critically on the second part of condition (i) of Theorem I which allows us to show that the density of the representative normal distribution outside of local ellipsoids converges to zero. We furthermore show that the posterior can be written as proportional to the prior multiplied by the density of the representative normal, meaning we can truncate the distribution to local ellipsoid even if the prior is unbounded.
- Bo)1l S }, for some local radius Mn. This truncation, however, differs
Our proof requires condition (i) for a couple of reasons. the maximum eigenvalue of the observation matrix grows exponentially faster than the minimum eigenvalue, there two potential pathological consequences. First, Lai and Wei showed that condition (i) of Theorem i nearly necessary for MLE consistency; in particular, if condition (i) of Theorem is removed, the MLE not necessarily consistent [2T] Example 1]. Second, even if Bn is consistent but condition (i) fails, it may no longer be the case that the density of the distribution V (x,,,, x at some other point (3' # 8o converges to O. One can see this by writing the density as
8(8': o² xxx))
1 exp( (2To²))
X
(3' - x X, - -Ba)),
20
where ((xx; ft, o²) denotes the density value at x of the N(,, o²) distribution. While the exponential term in the above expression necessarily converges to zero if 3 * BB. Pn is consistent, and Amin (X,] xx) - oc, it is not necessarily true that the entire expression converges even bounded. This is because the quadratic term inside the exponential may only be of order O(Nmin (X,,X,,)) X,)) whereas the term [X,, X,, grows at least quickly Xmax (X,,, x.,). Thus, although the measure of a neighborhood of 83' under the probability measure N(3... (x,,,, must converge to zero, without condition (i), it is possible that the density of this distribution diverges at B'. We show in the proof of Theorem I that the likelihood function is proportional to the density of (8,,,, (x, n. x,,)- meaning that it might be possible that the density of both the prior and the normalized likelihood function diverge at B'. This could potentially cause the posterior to have asymptotically non- non- negleggle measure in a neighborhood of 8', which would violate the BvM statement as the measure of the normal distribution (N "X Xx,,)))))) converges to zero in a neighborhood of B8'.
is
Notably, Theorem [ does not require the key stability condition, originally from 21 but used in many works since then for frequentist UQ for adaptively collected data, which assumes the existence of deterministic sequence B, for which B (X,,, X,, 1/2 & Loxy- Without this condition, 8n may fail to be asymptotically normal [21, Example 3J. In Proposition 2 in Appendix D we show that their example does, however, satisfy the conditions of our Theorem This leads to the rather surprising result that there are settings where Bayesian UQ is asymptotically equivalent to standard Wald-style frequentist UQ, yet the latter (and therefore also the former) is asymptotically frequentist-invalle We will describe another such example in the next section, which requires the triangular array version of BvM we prove there.
The previous paragraph notes that Theorem T can indicate either asymptotic frequentist validity or invalidity of Bayesian UQ depending on the situation. However, in terms of Bayesian validity, Theorem I (and indeed all the theorems in the paper) provide a strong positive result regarding misspecified priors: they say that the prior distribution is 'washed out' as + oc. Thus, from Bayesian perspective, a credible interval asymptotically has the correct coverage even if the prior misspecified, as long as both the correct prior and the misspecified prior used for the inference are continuous and bounded, with the misspecified prior's support containing that of the correct prior. Note, however, that the rate at which the "wash out" effect occurs depends on the rate at which the posterior distribution converges in TV distance to normal distribution, which may be logarithmically slow for optimal bandit algorithms. Thus, it may not be accurate finite samples to treat misspecified prior having "washed out".
4

---

4 BvM on Multi-armed Bandits
notable special case of Algorithmiiis the multi-armed bandit setting, which corresponds the case when A's output is supported only on the basis vectors. The multi-armed bandit setting can model experiments involving adaptively chosen treatment arms atmiccyp by some optimization algorithm like UCB [3] or Thompson sampling [[]]-herr where we want to estimate the mean outcome of each arm. For the bandit setting, we will use the notation No, p!!)) denote the count and sample mean respectively of the pulls of arm i in the first n steps.
As discussed in the prior section, our BvM proof requires the assumption that the maximum eigenvalue of the data matrix grows subexponentially with respect to the minimum eigenvalue. However, this assumption is violated by many popular bandit algorithms such UCB, where suboptimal arms pulled only (log n)) times. Thus, we require a modification of the above proof to specifically the case of bandits to include these existing algorithms. We also show that we can generalize our result to triangular arrays data that is, we now allow the adaptive decision rule to depend on the sequence length n and superscript it by n to make this dependence explicit: A However, we assume that the parameter So. variance o², and prior и remain the same throughout the array. The triangular array formulation enables to extend our results to sampling procedures which have policies depending on the length of the overall experiment such as is the case in the batched bandit setting [33], First, we show a consistency result for the case of triangular array bandits.
Lemma 1. Suppose independent length-m., trajectories Hm, = =((aT, =(=T))yp)) ..)..... (f Vin..) )) drawn via Algorithm| 7 Jssnng sampling rules N for each n, where B and 0² remain the same for all trajectories. Let X., = (xx
Tmma) and ya = =( array version of Algorithm satisfies the following conditions:
vmn)) T. Assume the triangular
(i) Each 23 is basis vector, i.e. x² eep.
(ii) Amin (X,, Xn) 3 Dc.
Then, the MLE B, XX,,,,, yn is consistent, i.e. 3n B Bo-
The proof of Lemma] 1 appears in Appendix C In addition to this consistency result, we also need a condition that allows to truncate the posterior distribution. Note that the theorem below assumes that the prior density æ is bounded which we did not need in the proof of Theorem []] this condition guarantees that the posterior can be asymptotically approximated as proportional the likelihood when the regularity condition on the maximum eigenvalue is not met. With these changes, we no longer need the condition that log Amax (X,, X.,) (p(min (X,, X,.).. making Theorem 2 very broadly applicable to standard bandit algorithms.
Theorem 2. Assume the triangular array version of Algorithm 7 satisfies the following conditions:
(i) Each 2² is a basis vector, i.e. x" E (el
(ii) Amink (X, Xn) B sc.
(ili) T(-) has continuous and bounded density on IP which is positive at So-
Then, the posterior distribution ( m..) satisfies
T (3) - -V o² (X(,,, X 2 0.
Remark 1.
This result assumes homoskedasticity of the bandits, but also holds if each arm has 0, as shown in Theorem in the Appendix D
different (but still known) variance of. >
The proof of Theorem 2 appears Appendix B.2 Note that the rate of convergence of the TV distance
in Theorem 2 depends on the growth rate of Auuin X X,), which can be logarithmically slow for
optimal bandit algorithms as seen in Section 6 Theorem 2 ggives the following corollary (proven in Appendix D) in the setting of non-triangular-arras statement is uniform over sampling rules as long as Amin (X). I x,) grows arbitrarily large among these sampling rules.
bandits, showing that the BvM convergence
5

---

Corollary 1. For any sequences (Tn, En) for which In - oc and en + O, let Pr. be the sequence of sets of distributions of trajectories induced by sampling rules such that for all n and for all P Pn.
(i) Each 23 is a basis vector, i.e. x² ep).
(ii) P(Amin X,) > ra) 1 6n.
Then, we have for any >>
limsup sup (() - (xx > c) =0
n-000 PEP
Theorem 2 implies that the interval [p] + 21 51 in
1/21 is an asymptotically valid Bayesian credible interval. Note that this matches the Wald-type frequentist interval which one uses in i.i.d. settings where the MLE is indeed normal. But [33 shows that for Thompson sampling in the two-arm batched bandit setting, the distribution of the sample means is not asymptotically normal in the case when the true arm means are equal. Thus, Theorem 2 implies that in this setting the credible interval will fail to be asymptotically frequentist-vali despite asymptotically matching the usual Wald-type frequentist confidence interval.
We can generalize this result beyond Gaussian bandits to exponential family bandits, i.e. where we still constrain æy in Algorithm 1 to be a basis vector, but now instead of yj being sampled from Gaussian with mean = [3, it is sampled from exponential family (with density exp(ny,-bb(p)
with parameter 17 2 8. Exponential family models include many common distribution types such Berouuli,, Poisson, Gamma, Beta, and Geometric distributions. For the remainder of our results in this paper, we return to the non-triangular array setting where A is not allowed to depend on n.
Theorem 3. Let 80 E I2 be the true parameter value and Ba,s be the i-th coordinate of 3o. Let No, be the number of times arm was pulled and Yn,i = - N.. クト= [ i Let the local MLE
be Bas = Su,s + " and the empirical Fisher information be I, = diag (B,,))) Suppose the exponential family version of Algorithm 7 satisfies the following properties:
((
(i) Bo.1, -o,1,.--,Po,p Bo.p are in the interior of the natural parameter space.
(ii) min; Nn.i 2 co.
(ili) T(-) has continuous and bounded density on IP which is positive at Bo-
Then
T (3)H,,) -
2
The proof of Theorem 3 appears in Appendix B.3 The local MLE defined above is the maximizer of the second order Taylor expansion of the likeliilloood around 3g, and is asymptotically equivalent to the true MLE by the asymptotic equivalence of the log-likelihood to its second-order Taylor expansion in a local neighborhood of the true parameter value. Note that the above theorem requires the true parameters to be in the interior of the natural parameter space, meaning for extremes of common models such as Poisson with rate near 0 or Bernoulli with probability near 1, the conclusion of the theorem may be a bad approximation for finite n.
The proof of this theorem is similar that of Theorem [ but here we require a Taylor expansion to express the likelihood as a function proportional to Gaussian density. The bandit setting is particularly nice for performing this expansion since the likelihood factors into the product of the likelihoods for each individual arm. Thus, even if the eigenvalues of I, grow at asymptotically different rates, the likelihood still appr-oappo by the second-order expansion. This argument is harder when generalizing beyond bandits, as argued in Appendix [F) Additionally, to argue that we can truncate the posterior distribution to a local neighborhood, we use the convexity b(.) to bound the tails of the likelihood function. Our proof does not necessarily generalize to triangular arrays as it relies on uniform bounds on a serialized sequence of samples, such as the one provided by the law of iterated logarithms.

---

BvM on Gaussian Linear Bandits
Another special case for our BvM theorem the case of Gaussian linear bandits, in which context zj is observed for each arm pull 0j. impacting the resulting reward distribution through linear transformation, i.e. Djj ~ [ar, 0 where oi...., Om are parameter vectors. To model contextual bandits with our adaptive linear Gaussian data process in Algorithm [] we will let the parameter be / E IR where the (id d - ()-(id)th indices of /3 represent 0;; in other words, we stack the parameter vectors 0j..... On vertically. Then, when we observe context x' E IR and action i {l..... m), we sample the outcome from ( x o²) where the (id d + (id(tt index of Rnnd represents x'. Then,
BvM in the contextual bandit setting follows.
Theorem 4. Suppose p md and we decompose the covariate space
X = RR (0a) x ...... {023) ([Oa) R x... ((aa) U... ufo.) x {02} x R.))
Assume the sampling procedure in Algorithm [satisfies the following conditions:
(i) Each action 3 falls in X.
(ii) For all i = -,.... m, let Tj X - TRd be projection onto the (id - d (id(id)th coondinates. Letting Ln,i = -ラ= j=1 (z)))
idddd, for all we have
Amiu (In,)) 3g so and log Amax (I,,)) = Op(Amin (In,)))
(ii) T(-) continuous with bounded density on Rp and positive density at Bq-
Then, the posterior distribution (n) satisfies
[T] NT [X Xx, lry 3.
The proof of Theorem 4 appears in Appendix B.4 Note that condition (ii) in Theorem 4 different from condition (i) in Theorem because it concerns the distribution of contexts conditioned on a particular arm, rather than the distribution of arm pulls. Thus, it is possible for contextual bandit algorithm to satisfy condition (ii) when it pulls different arms at exponentially different rates. This result does not immediately generalize to triangular arrays, since we rely on consistency which may not hold in the triangular array setting for contextual bandits. However, the non-triangular array version of Theorem is special case of Theorem 4
Numerical Experiments
We discuss the empirical validity of the statement of Theorem T in the setting of multi-armed ban- dits, contextual bandits, and the linear quadratic regulator (LQQR.. ) As seen in the previous section, the posterior distribution is asymptotically equivalent in TV distance the normal distribution (B3,, of ox,, which we call the representative normal distribution. Although this equiva- lence holds asymptotically, the convergence rate to this representative normal distribution may be quite slow depending on the rate of growth of Amin (X, X,). We empirically perform posterior inference in three common adaptive settings and show that the posterior does empirically converge to the representative normal.
max (0, - - Q(X) to approximate P(X)
We use Monte Carlo and the relation 1P-V
the TV distance between the posterior distribution and the representative normal distribution. As seen in Figures 2 and 3] the convergence rate of the TV distance depends on the configuration of the true parameters. Note that in the multi-armed bandit setting, the convergence seems fastest when the arm means equal and becomes slower the margin increases. One explanation for this is that the result of Theorem 2 requires the condition Amiu (X,] X,,) L co, suggesting that the rate at which this minimum eigenvalue grows determines the rate of convergence of the TV distance, More
specifically, we see that in our proof of Theorem? we use the expression E
as an ((36))

---

UCB Gaussian Bandit
Lin-UCB Contextual Bandit
10a
10c
[0, 0]
undorminated dominated duplicctee
[0, 0.1]
BvM TV distance
-0-1
[0, 0.21
BvM TV distance
[0, 0.3]
[0, 0.4]
10-2
10-1
10-3
2000
4000
6000
8000
10000
0
200
400
600
800
1000
Time
Time
(Left) Average TV distance measured in the BvM statement for UCB in two-arm Gaussian bandits over horizon T = 10 using 10 replicates under five different true parameter configurations labelled by [un [ua] where f #22 are the true means. (Right) Average TV distance measured BvM statement for lin-UCB on three-arm Gaussian linear contextual bandits with context distribution (0, 22x2) under three different true parameter configurations. Standard Gaussian priors are used for all arms. TV estimates shown have standard error at most 0.1 times the TV estimate.
Figure 1:
UCB on Bernoulli Bandit
UCB on Poisson Bandit
[0.3, 0.5]
[0.2. 1]
[0.4, 0.5]
[0.5, 1]
BvM TV distance
-1
[0.5, 0.5]
BvM TV distance
[1. 1]
[0.5, 0.6]
[1, 2]
[0.5, 0.7]
[1, 5]
10
10-3
10-2
2000
4000
6000
10000
4000
6000
8000
10000
Time
Time
Figure 2: Average BvM TV distance for UCB Bernoulli bandits and Poisson bandits, under the same configurations as Figure Beta(1, priors are used for the Bernoulli bandit and Gamma(1, 1) priors for the Poisson bandit. The representative normal is centered the true MIE, which is asymptotically equivalent to the local MLE used in Theorem error most 0.1 times the TV estimate.
TV estimates shown have standard
upper bound on the TV distance where Kin H.
-²)) Thus, we see that the rate which Amiu [ X,, diverges affects the convergence rate of ii P Su. which in turn affects
=(x..) 2 This explains why the TV distance converges fastest for the zero the convergence x(3)
margin setting (even though that setting does not satisfy stability).
We simulate the algorithm Lin-UCB in the setting of contextual bandits under three different parameter configurations [22]. The "undominated" configuration represents a case where each arm optimal for some choice of context. In the "dominated" case, arm is never optimal for any choice of context, and in the "duplicate" case, two arms share the same parameters. In Figure T we empirically that the convergence of the posterior the representative normal is fastest for the undominated configuration, possibly because in this case, the optimal policy samples each arm with probability bounded away from zero. In the configuration where two are duplicate, there may be substantial bias in the estimation of the arm parameters [26). leading the "duplicate" case to have the slowest convergence
in Figure
can also model the Linear Quadratic Regulator (LQQR) 24 common setting used in control theory, using adaptive linear Gaussian framework. In LQR, control state transition of the

---

form 2j+1 = Az; + Bu, + Ej where A E R E Bixd, . bj are adaptively chosen actions, of ~ (0)) for = 1,....,,, n, and we aim to estimate the transition matrices A, B. To represent this as an instance of Algorithm [T we serialize the observed state vectors 2j into the sequence of observed outcome variables- that is, we use a trajectory of length ñt = nkk where Dii 9(i-1)6+1 Dikk represents Iz. We let the parameter value 3 be (A B) in row major order. Then,
the ((i- - (): + d) + (i(( + d)) indices of the covariate T 1)4+4 E (R((++))) represent
Zj Uy
and all other entries are zero.
We simulate the Noisy Certainty Equivalent Control (NCEC) algorithm on LQR under three different configurations [30]. The "determined" configuration represents a case where the action space has the same dimensionality as the state space and the action transition matrix full rank. The "stabilizable" case is underdetermined with fewer action dimensions than state dimensions but where the optimal policy allows the system to stable. The "unstabilizable" case is underdetermined where no policy makes the system stable. In all settings, we empirically see that the BvM TV distance decreases over time, with the convergence rate being the fastest for the unstabilizable case as has the highest growth rate of min (X,, X,). Finally, as we show in Proposition 3 in Appendix D the NCEC algorithm satisfies condition (i) in Theorem so long as a certain stability condition holds (see Assumption | in the same appendix section for precise definition); both our "determined" and "stabilizable" simulation settings satisfy this condition.
We also simulate Thompson Sampling in the two-stage Gaussian batched bandit with two arms, which is the setting analyzed in Theorem 2 of [33]. We compute the TV distance between the posterior and the representative normal in the same way for different values of the margin. As shown in Figure 3 we again see that the average BvM distance smallest when the margin is zero. We also plot the empirical coverage of the 95% credible interval for the margin, which matches the coverage level except near the zero-margin case, as suggested by 33 This is an example of a fairly well-behaved sampling process where our BvM result applies but Bayesian credible intervals are still not asymptotically frequentist-valid. Note that this setting satisfies the stability condition if and only if the margin is nonzero. This suggests that the asymptotic frequentist invalidity of Bayesian credible intervals is a local phenomenon around the zero- margin case. Figure 3 supports this analysis and also reveals that in this local region, some parameter values lead to overcoverage and some to undercoverage from a frequentist perspective. Our BvM result may provide an explanation for this, as the Bayesian coverage under a prior containing this local zero-margin region must be correct, meaning the coverage probability aggregated over the erroneous local region should match the coverage level.
Appendix Coontinss demonstration of our main results on a simple real-world dataset. Coverage plots for the other simulations are shown in Appendix H
7 Discussion and Future Work
This paper has shown that, for number of important classes of adaptively collected data, a Bernstein- von Mises theorem applies, linking Bayesian UQ and Wald-type frequentist UQ This ensures that under extremely mild conditions, Bayesian UQ is asymptotically Bayesian- -vlid even when the prior is misspecified, and when the stability condition of [2I holds, it also ensures Bayesian UQ is asymptotically frequentist-valid,
One of the surprising takeaways from our work is that when the stability condition of (1 fails, BvM holds but Bayesian UQ is asymptotically frequentist invalid. We note that our work, however, only considers the case when a fixed, nonrandom prior is used for Bayesian UQ This raises the question of whether there is a data- data-dependent way to set the prior so that Bayesian UQ is always asymptotically frequentist valid; we could think of such a method as empirical Bayesian UQ
Another direction of inquiry would be to extend the BvM result to general parametric models beyond just the Gaussian and exponential family cases. However, there are several obstacles to showing the adaptive BvM result in more general parametric models which we discuss in Appendix [F]
Note that this paper does not suggest a method of asymptotically frequentist-valid Bayesian inference; in fact, we would like to warn practitioners against using either Wald-type frequentist UQ or Bayesian UQ as if it were asymptotically frequentist-valid.

---

Thompson Sampling Batched Bandit
NCEC on LQR
10c
distance
determined
10-3
0.952
stabilizable
BvM TV distance
BvM TV distance
unstabilizable
Coverage
10
0.948
10-2
0.946
www.
10
Coverage
0.944
-0.2
0.0
0.2
0 200
400
600
800
1000
Margin
Time
Figure (Left) Average BvM TV distance and empirical coverage of the 95% credible interval for the margin for Thompson Sampling in the two-batch two-arm Gaussian bandit setting with 10 samples per batch. Error bars are 95% confidence intervals over 10 replicates. Blacked dotted line is the correct coverage level. (O, 1) priors are used. (Right) Average TV distance for Noisy Certainty Equivalent Control on LQR [30 under three different parameter configurations. Standard Gaussian priors used for all arms. TV estimates shown have standard error at most 0.2 times the TV estimate.
Declaration of L.MM usage:
ChatGPT-40 was used to create code templates for Python implementation of the lin-UCB and Stepwise Noisy Certainty Equivalent Control algorithms. The authors revised the templates to ensure correct implementation and modified them to verify the BvM statement.
Broader Impact:
The results presented here have the potential to guide the design safer systems and more reliable hypothesis testing in adaptive experiments. However, it should be noted that the Bernstein- Mises theorem does not immediately imply frequentist-valid inference as discussed. Thus, Bayesian UQ should not in general be treated frequentist-valid, Instead, result contributes understanding the differences between frequentist and Bayesian approaches in adaptively collected data.
complete
References
[1] Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24, 2011.
[2] Adrian Aguilera, Caroline A Figueroa, Rosa Hernandez-Ramen Urmimala Sarkar, Anupama Cemballi, Laura Gomez-Pathak, Jose Miramontes, Elad Yom-Tov, Bibhas Chakraborty, Xiaoxi Yan, et al. mhealth app using machine lcarning to increase physical activity in diabetes and depression:
clinical trial protocol for the diamante study. BMJ open, 10(8):c034723, 2020.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
[4] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
[5] Deepak L Bhatt and Cyrus Mehta. Adaptive designs for clinical trials. New England Journal of Medicine, (75(1):65-44 2016.
[6] Aurélien Bibaut, Maria Dimakopoulou, Nathan Kallus, Antoine Chambaz, and Mark van Der Laan. Post-contextual-andde inference. Advances neural information processing systems, 14:25548-28550 2021.
10

---

[7] Natalia Bochkina. Bernstein mises theorem and misspecified models: A review. Founda- tions of modern statisties, pages 355-380, 2019.
[8] Natalia A Bochkina and Peter Green. The bernstein-von mises theorem and nonregular models. 2014.
[9] Ismaël Castillo and Judith Rousseau. A bemstein-von mises theorem for smooth functionals in semiparametric models. 2015.
[10] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling, Advances in neural information processing systems, 24, 2011.
[11] Benjamin Connault. A weakly dependent bernstein-von mises theorem. Technical report, Working Paper, 2014.
[12] Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive linear models. In International Conference on Machine Learning, pages 1194-1203. PMLR, 2018.
[13] Allan Gut. Stopped random walks. Springer, 2009.
[14] Vitor Hadad, David A Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the national academy of sciences, (18(15):c201460))
2021.
[15] Steven R Howard, Aaditya Ramdas, Jon McAuliffe, and Jasject Sckhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 19(2):1055-1080, 2021.
[16] Emilie Kaufmann and Wouter M Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research, 22(246): 1-44, 2021.
[17] Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In International conference on algorithmic learning theory, pages 199-213. Springer, 2012.
[18] Bas JK Kleijn and Aad W Van der Vaart. The bernstein- ceatteiin-ooo-missee mines theorem under misspecifica- tion. 2012.
[19] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey, The International Journal of Robotics Research, 32111): 2013.
[20] Geerten Koers, Botond Szabó, and Aad van der Vaart. Misspecified bernstein-von mises theorem for hierarchical models. arXiv preprint arXiv:2.388.07888 2023.
[21] Tze Leung Lai and Ching Zong Wei. Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. The Annals of Statistics, pages 154-166, 1982.
[22] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670, 2010.
[23] Alexander R Luedtke and Mark Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics, 44(2):713, 2016.
[24] Volker Ludwig Mehrmann. The autonomous linear quadratic control problem: theory and numerical solution. Springer, 1991.
[25] Yash Nair and Lueas Janson. Randomization tests for adaptively collected data, arXiv preprint 27X.v:230..55365, 2023.

---

[26] Xinkun Nie, Xiaoying Tian, Jonathan Taylor, and James Zou. Why adaptively collected data have negative bias and how correct it. International Conference on Artificial Intelligence and Statistics, pages 1261-1269. PMLR, 2018.
[27] Anna Rafferty, Hujji Ying, Joseph Williams, et al. Statistical consequences of using multi-armed bandits to conduct adaptive educational experiments. Journal of Educational Data Mining, 11(1):47-7,, 2019.
[28] Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. Large-scale open dataset, pipeline, and benchmark for bandit algorithms. arXiv preprint 4rXiv:2008.07146. 2020.
[29] Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[30] Feicheng Wang and Lucas Janson. Exact asymptotics for linear quadratic adaptive control.
Journal of Machine Learning Research, 22(265):1-112, 2021.
[31] Elad Yom-Tov, Guy Feraru, Mark Kozdoba, Shie Mannor, Moshe Tennenholtz, and Irit Hochberg. Encouraging physical activity in patients with diabetes: intervention using rein- forcement learning system. Journal of medical Internet research, 19(10)3338, 2017.
[32] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
[33] Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. Advances in neural information processing systems, 33:9818-9829 18-9829, 2020.
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and motivation, we claim that the BvM theorem generalizes to adaptive data without assuming stability assumption on the data, This claim is shown in the proof of Theorem 1 and intuition for this result is given in section 3 of the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section of the text, discuss the phenomenon that Bayesian eredible intervals may fail to be frequentist-valid and how our work does not explain how to charac- terize this discrepancy. We also discuss how our BvM result requires the data be generated from an exponential family model which restricts the generality of the result.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our main results, Theorems 22 are all proven in Appendix B, with the supporting lemmas also proven in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The configurations for experiments are given in the figure captions of Figures] I 3 with the algorithms (lin-UCH and NCEC) cited. We also describe how the TV distance was estimated in the text.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
14

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Access the source code is provided in Appendix A.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
wwblic/guides/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. wtt.cccpppbllccc httpppulblluieseeeeppeppppppepee guides/ (oweeeumisssiino ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details for the experiment parameters are included in the captions of Figures [ 3 with full details included in the source code in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
The caption of figures describes the standard error of estimates provided in the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention in Appendix A that the data in each figure require under hours in CPU time to generate.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. h iet.eiiieeeeitcccciieeeeeeeeee
Answer: [Yes]
Our work does not involve human subjects and our data do not contain any private or personal information. We hope that the statistical inference methods provided in this paper guides more reliable experimentation methods in adaptive design.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We briefly discuss the broader impacts of this work at the end of the main text.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper does not contain any results that can be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
This paper does not existing code or data. The existing algorithms used in the paper are implemented independently.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects was done in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing or research with human subjects was done in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage of LLMs for this project is described under the "Declaration of LLM usage" section the end of the main text.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuril) httt::://eerrps.ccc/oooofereeeee /tt//////ooeeeeeee/22/////.... for what should or should not be described.
19

---

A
Source Code
The code for this project can be accessed at httppss//giuuu..mmm//iiia www.et/eeeeeeeeneieeenieeeeeeeee Each figure's data takes under three hours of CPU time to generate.
B
Main Theorems
B.1 Proof of Theorem 1
We first write an explicit expression for the posterior. Note that the likelihood can be written as, after removing constant factors that don't depend 8,
Hm)
II A(ry)1l--1))
P(yjlaz B)
(1/ x 3)2
II P(uyleg, (.g.)) () j=1
II
exp
20
j=1
7 II cxp j=l
(y/ 8 x2)²
(3) 2jj 3 xz)2
20
20²
x exp(
1
(Bn - 9) 9))
X,, 3)).
20
Thus, by Bayes' rule, the posterior can be written as
20 _ (3n - T)xx, x (3n -B))
((3HHm) & @(B)exp(
Let d(H,) = ||T(81H) { /TVV. By assumption, x(3)) > 0, so using Lemma
we have
d(Hn) $ (7((5))
T(3)
exp(
(3 3a) X,, (BB)))
((6)
(2TT22) X X
11
20
(8 - 3n) x X,, BBn))) d3
exp(
(2To2) X X,
202
-/ (33
1
1
exp(
X, Xn Bn)) (B Tx
(33)
V(2
X x
20
+
( (Hin)
-1) -1 Hn
= E
(BO)
where Kn is random variable marginally distributed as Kn H, ~ N(nn, x2x (,,, )1)) Let Cr. be sequence of random variables satisfying log Amav (X, X.)
P 0, which must exist by condition (i), i.e. we can let G, = Amiin X X,, log Amnax X X,, Then, to show
Pg and
Xmnt (X X..
d(H,,) 2 0, we use common trick when showing BvM-style results (29
18 that is, we will
partition the expectation into parts inside and outside a local ellipsoid, We will show the following two statements.
20

---

E
π (Kin)
-1 1)
1[|(X,, ( - -nn)1 S cal H -0
(1)
T (19)
T (Haa) (Bo)
E
-1)
x,, Bn)1 > cal Hn
30 0
(2)
To show T. note that we have
x. (Ki))
1 1111 X,) - BB)ll s cul Hh
E
π (B3)
+
1
Gn E (m((k)) T (Bo)) 1l1((rn Bn) S Amin (X, Xmn) Hn
t (3b)
sup
(T((K)) - +(Bo))+
t (Bo)
1/2
Amin X 172
sup 1/2
(T(Mm) +(()))+
T (36)
[Nn Bulls
1/2 110 -
Amin
Lai and Wei showed that 3n is consistent if condition (i) is satisfied 21 Thus, we have 2//2 Auin X 1/2 + [| -oll E, O, meaning the above expression does indeed converge to zero in probability if ग is continuous at 3g-
To show 2 note that by the triangle inequality, we have
T (1o))
) 1II(XX,,,,) x,,, 1/2 BB)]1 > cn] Hn
E
T (B6)
(( ) x..) 1/2 1/2 - Bm)112 1 > cnl Hm
π (Bo)
+ E XII - 3w]11 > enl Hn
The first term can then be bounded as
E
T
Bn)]1 > cn] Hn
(30)
/
T(Kn)) x,,
1/2 3w)l2 > cnl P(Ki Hn dks
(Bo)
/
x,,
1/2 Ba)l > cnl
(2TT02)0) X,) x, x,, expl-
On
dxn
((8)
20
(2 -p/2
Co Mmax X Xx, p0/2 /
exp(
1/2 1/2 10mn-Bm) Bn) cnldkin X,,
T(36)
20
(2/02)-P/22 exp(
Co
p log Amax (X,, X,u) / T (Kn)dRkn
+
T(30)
202
2
(2TO²)-PP)
Cn
p log Amax ( Xn))
exp(
T(8))
2
2
21

---

If log Amax (X, x
D the above expression converges zero in probability. To compute the
second term, note that (X,, xx) )/ ( (Ki -3n) Hn distributed as (N,, ((0,0_1)) 1p). Thus, Cp. 4 oc. ) (ni,
then E [X, )1/2
32] > cml nn 2 O. We have shown that each term (1) and (2) in
the decomposition of the upper bound for d(H,)) converges zero in probability, meaning dl (H.) converges to zero in probability as well.
B.2 Proof of Theorem 2
As in the proof of Theorem the posterior distribution can be expressed as
T T(3 x π (3) exp(
20 (Bn 3) ")xx x Xn, (3,, B)).
Letting d(Hlm..) (3( Nin,, 0² Xx, 1|] we also have
[T((m)) ((33) -1 ) Hm
d d(H S EE
where Kn mn N(3n, ( (x,)) By Lemma we have that 3n is consistent. If
Amin (X), 1 xn) 2 oc, then Kin 2g Bn
write Pin Bo (3x (80 Zn where Zn H of (X,,,,,,) and both the terms 3, 1h--ap Bu and Z, converge to zero prob- ability. If i is continuous and bounded, by Vitali's convergence theorem,
((in)
(BO) -1) -1)_] 0,
Eid(H )] SE
meaning d(H,..) 3
B.3 Proof of Theorem 3
Suppose (1) Yi.())... are the serialized arm pulls from arm i. Let Ye,N) = 2/2 Σソ= Vi,()) be the sequence sample means. By Lemma we have
PUNY r 11/4) almost surely as N oc.
(3)
for some small say 0.01. Since Yn.; = Y,,(Ns.), we have that Yn,; 14 O (N
Thus, Bm.: Bo. Op (N meaning 3r., is consistent for Bo,
We prove the result in three steps by truncating both the posterior and normal distribution to the set
On = { Vi, |3,- Bu.sl I en,i) } for some sequence random variables Cn,i such that N 1/3 n,t Co. B
and N 1/2-
n, Cn,i Ls oo as n + co. We then show that
|| = C8 TV 2
(4)
|| Cm -N || 2
(5)
(.. 1 - -N 2 0.
(6)
We first show 5 The likelihood function can be calculated as
L(B; Hn) x I
- -(3.))
1=1
22

---

By Taylor's Theorem, for any 8 E On, we have
T(33; Hn) x exp Σ Nu.i VV,e3, ENniiVen ((B,,) (B0,))) Bo,s)
2 (Bo.) Bo,,) V" (o,))9)
for some F & Cn However, b is bounded on a neighborhood of B., for each i and N2/3 On 2 0, we have that SupSEC., Na, (3,,,,)3 - Bo,s) 2 0. Thus, we can write
L(3; H.) & exp
N Ph. - b(8o,) - - (30, - Bo,,) 1 (300)))33s-Bo0. - +
+z((1)
1 Nn,1l bb" + (P,, (30,)) + 6" (80,))))0.))811 (1 (2)))(1))
exp
2
1 b((,,,)83 + (8..).)) (1+ op(1)
exp
Nn.il
2
1 P
-nni)² (1 o1)
exp
2
i=1
= eepp
(B 3.) (n +1))(1++
Let the density of the truncated normal distribution Cr (Bn,is 1 be expressed
An expl (8- (B L Bn.i)).
where An chosen such that the density is suitably normalized. Let d, (H₂) = [[THH(((HH))
NC.. (Bo: In Mlrv. By L we have
dn()) = / AnTSEC. TB)L(BB Hm,
A,1sEC, exp( 1 ( In) -Ba)) d3
[(Bm) H.)
T(B)
< An AnLsEC,
exp( (3-),) I (1))(1+
(T3))
exp( - (B (B- 3m) 1x Bu))) d3
2
where we say a sequence of random variables W,, (3) indexed by 3 is (1(1) if supgec. Wn(3) P 0. Thus, we have
dn(H)) S /
T(9)
+ tzp(1)) An Anlsec, exp(-7 B.)TT.(3 B..)) d3.
(3a.)
(11m)
= EE E Non
-1 1 02(1) -1+00(1)
(3p))
/supsec, T(3)
+ 01((1)
infgec. (33)
23

---

Then, since Cn.i & for each and is continuous, we know supsec, x(2) 4 meaning the above
infeec, [[B]
expression indeed converges to 0 in probability, Next, we show 44. Note that
(B)HH, (3)HH)) Pr(3 H.
Seg (B) H,)
Jg (B) H d3
1 π max Scc L(3; Hn
T Sc. L(3; M
where m= supa (33 and min inf (B3). Note that if is continuous and positive at So. then m-inn converges some positive constant. Then, since the ancillary function of the exponential family is convex, the likelihood function is log-come log-concave in B, meaning we have
Sog L(3; H.)dB Jc. L(B: d3)
L L(3 + (o (.) d
sup vEOC,
J L(3 + (u So)t: Hn dt
S exp((n(B0) + (ln(w) (no())) dt
< sup wEAC=
Io eexp(en( (36) + (e) - L, (80))t)tp-ld
exp( - dx
(1) (v) vEaC, 5o (b) tn (0) exp( - --)x0 dz
sup
Sa exp( -2 1z)x dx
So exp( -- 1 1 dr
where an infveao, (3v) (v). Note that the function a + L axp(-=)j=² da
Jo exp(- de goes zero as
co. Thus, it suffices to show that on 2 xc. To do this, note that by expression 7 have
1 P
an inf BEBC,
Σ
(B0,,, Br.c)² (vs +1.,))++
2 i=1
Since [Us - Briil 2 l|u:
Bo.il
130 - Bn,ill = -Ico,i - 1B0,; Br.ell, we have
an 2 inf vedOn
Nn, (Bo,:)11B0. 3 - -BB,. - -mn,
(Co,
i=1
inf vEOC,,
Σ
N b"..b"(Bo.il) (Bo,:) Bmnil - -2, + 22nnsllo.. - Bnol - 130.s - -1..13++ (1)
2
1=1
= inf
-
1
Σ
Nn.
2|30, Banel
+ (1)
vEdCn 2
Cn.i
i=1
By the definition of Cn as well as Equation indeed have an & oc. Lastly, show
(3. we know 2, Nn. 2 oo and
Aau- Ps 0. Thus, G
We have
(((B,,15)) 1
No (Bo, 1- )ll = :las=Pxxxxm (3, Xx - Bo.sl > Cn.i)
-Px,N [X,] Cn N,. (Bo,))
1/2 P, c0, then the above expression indeed coverges to zero in probability. If 1//2
24

---

B.4 Proof of Theorem 4
We only need to establish the consistency of In. after which the result follows by similar argument to the proof of Theorem Consistency can be shown from condition (ii) by Theorem 1 of [21].
C Technical Lemmas
Proof of Lemma We translate this directly to the bandit setting, after which this result follows from similar version of Theorem 2.2 of [1]. For each i 1,.... .......,. P, let 14 3g be the i-th coordinate of 3 which represents the true mean of arm i. Let Ni,, = (X,) be the number of times arm i is pulled in the trajectory H We serialize all pulls from each arm i-tha is, suppose
(Mu,, o²), we look up the next unused sample the serialized sequence (n.. and use that as our observation.
(1) "......... ind (1,,, Then, when drawing yf
Under this formulation of the data-generating process, the sample mean In,i is simply the mean of the first N samples from the serialized sequence V'...)..... Note that the serialized sample
means Ti, = 1/ Σ}= () satisfy K. 3 μ4 as + oo for each i, n by the strong law of large numbers. Then, Proposition iimlies that for each there exists some function (-) satisfying p c (B) = 0 such that for any B,
PN2
B - PlURAN) - mal S (B) 21 1
Crucially, note that the functions (-) can be chosen such that (-) does not depend on n. This is because the distribution of serialized samples (t...))))).... for each trajectory is identical, and in Proposition] [ the bound translation function is chosen solely based on the distribution of the almost-surely convergent sequence. Finally, by condition (ii), we indeed have that Ni.. & co for each i, meaning the condition P PN.... 2 B.] 2 B,
1 is indeed satisfied for some sequence
Bn 0o to see this, note that letting B, Bu PP 2 B) 1
}. we must have
constant C where IP S C & for infinitely many n which would contradict the statement that N diverges in probability. Thus, we have Pl 141 S < (B.)) 2 ef (Bx) for each meaning the sample mean Br,tt (N....)) is indeed consistent for 142-
Bn 0o as otherwise there would be
Proposition 1. (Bound translation) Suppose Yn is a sequence of random variables such that Y, " Y for some real number Y. Then there exists function cBB) with c(B) -> 0 as B such that for any random variable N and positive integer B, we have
PIN z
B
PJYN - S 21
B
Proof. Note that it sufficient to show that for any fixed 7), there is some function e, (B) such that g- 6, (B) 0 and
1
P[N 2 B
PI - Yl < C,(
B
To show this, we simply let ,, = PP > B, Yr Y > 7) If Y, converges almost surely to Y, then lim en = 0. Also, we indeed have
P[ Y S v/ P B and Vn 2 Y, - Y < 1- GB))
Since we can do this for any 7,, the statement of the result is indeed true.
Lemma Let P(-) and Q(-) be continuous distributions and c E be any constant. Then, |P- Q1 1.4 (P(x) d( & [(P((z) ddx)
25

---

Proof. Let - fx P(x) > Q(z)). Let = SA -A P(x) - Q(x)dx and d2 = SAC Q(x) - P(x)dx.
Then, note that
|P - 1 = 2 (P(()) -P(x) Q(x))+ 2 2
1
and
0 / P(x) Q(x)dr 01 02
Solving for 01 and 02, we get
41 = 02 = || Q1lvv.
To show the desired result, if 2 1, we have
|P - Q1TV = 01 V -P(o)-Q(z)dae ((P()))((s))+)))
Similarly, i then
|| Qllvv a2 5 / Q(x) - cP(x)de & ((cP(t) Q((=))
Lemma 3. Let X1, X2,.. be i...d. samples from a distribution P with mean μ and finite variance 0². Let Pm 1 Ei=1 X,. Then,
n 12- 11) 2 0.
for any e > 0.
Proof. By the law of iterated logarithms, we have
Vnlūn - nl|
lim sup 10+00
20 log log n
with probability 1. Then, we have
Vnljmn - ml 20 log log n
v 2202 2 log log
lim //p2 1l < R-+00
limsup
lim n=+00
0
n1a
with probability 1.
D Auxilliary results
Proposition 2. Let 1 be deterministic and take any real value. Let the adaptive sampling procedure simply deterministically set = yi-1 for > 1. Then, 3 = 1, the MLE is asymptotically non-Normal. However. condition (i) of Theorem 7 is satisfied by this sampling design.
Proof. The asymptotic non-Normality is established by Lai and Wei in 22 Example 3). The second part of condition (i) is satisfied because the covariates are 1-dimensional and the first part is satisfied because Er= x Pg oc, which is direct consequence of [2] Equation (4.9)).
26

---

We now recall an assumption from 30
Assumption 1 (Stability, (30 Assumption 1J). There exists a matrix Ko for which the maximum absolute eigenvalue of A + BKo is less than 1.
Proposition 3. Suppose of LQR satisfying Assumption using any configuration of the hyperparameters 0, B [1/2, and a > 0 permitted therein. Then, this satisfies condition (i) of Theorem
the stepwise NCEC algorithu (Algorithm 1 in (301) instance
Proof. First, observe that our Gram matrix
X X is more simply expressed as Jaxk @ G,, where
denotes the Gram matrix. Because the largest and smallest eigenvalues of Lack Gn are the same as that of G, it suffices to study the spectrum of the latter. Theorem of 30 shows, in the stabilizable regime, that
X denotes Kronecker product and G, : -i=1
Zi U4 U1
D Gr Gn (Dz')²
2 Iktd.
where
I 0 K
Dn = n log / (n)
(8)
La
V
20
72
00
On = -n log (n)
Σ p=0
(A+ (())((((BBB))))))))) 0²
(A+
BK) ((A+BK)9)
B
q=0
(9)
and K is matrix that does not depend on (see equation (3) of 30 for precise definition).
from which it follows that
G, - D.D = (1)
(10)
Below, we show that Amax (D, DT S (n) and Amin (Dn 2 2(n log" (n)). Because Equa- tion (10) implies both that Amia (G,) Amin (D,DD,) = (1) and Amin (G.) Amiu (D,D,T) =
(p(1). ), we indeed have that condition (i) of Theorem is met: log (Amax (Ga) = Op () (G,)) and
Amin (G,) 2 oc.
Upper bound on Amax (D.D., By submultiplicatiiity of the operator norm, we have, using equation (8. that the largest eigenvalue of DT is at most
1x
2
C
1/2
3 m))
log"
K
La
O
/
The middle term is of constant order and therefore it suffices upper-bound the last term. Because
1/2
12//1a S max (ICH/"I. 11.
and since |l / is again of constant I1 (I
0
order it suffices to simply bound ||C|/3| Using equation 9 and triangle inequality, this is at most aa/ (1n(1-8)/2 1og- (n)). Consequently, the maximum eigenvalue of D, is O (n).
Lower bound on Amiu (D, To lower bound the minimum eigenvalue of DT, we apply the same argument the maximum eigenvalue of - The largest eigenvalue of (D,.) - at most
n 8 log (n)
4 K
0 Ta
-11
C1/2 0
0 |
V+2
Then, /2/2
T2 < max(1 CT 11 VB/T2) max Laxx(Amin (C,) -1/2 3/
Again using equation (9) and the triangle inequality, this is at most O(1), meaning the minimum eigenvalue of D,, TT is at least (nn (n)).
27

---

Proof of Corollary 1: Suppose by contradiction that there is some is where
sup ((1]=(3 - -N o2 x 'llvv > c) > k PEPa
for infinitely many n. Then, for all such n, there is some sampling rule A" whose corresponding distribution trajectory satisfies
((1 - -N o² (x) > 2
However, this contradicts the statement of Theorem 2
E
Heteroskedastic Bandits
Algorithm 2 Heteroskcdastic Gaussian bandits
Input Action set A = -1,,.... (I,...,p), Arm sampling rule A (Ax R)" A(A), Reward distributions
B. = (3,,, ) Output Sampling trajectory Hn,
Ho <
for j = , n do Sample aj A(-1 A(-1H,-1)
Sample yj ~ (V 00)
Hj <- {(111 v1).......... (x), U3}}
end for
Theorem 5. Suppose generate length-m,, trajectory H ((aft v"),.., ...., (ah vii) for each n according to decision rules A in the case of Heteroskedastic Gaussian bandits. Assume the sampling procedure satisfies the following conditions:
(i) min; Ni.. 4 zo where Ni. = E = i].
(ii) (+) is continuous and bounded on T&P with positive density at Bo-
Then, the posterior distribution T (B) satisfies
[|)(3)))_)) (V diag(o, Ni. 0.
Proof. We rescale this problem to an instance of the homoskedastic bandit problem and apply Theorem Let the covariate sampling rule be Á" ("(H,-1) CAm(H)-1) and the rescaled parameters 1. Let #(3) x T * 3) where * represents
be 3 with the homoskedastic variance element-wine wise multiplication where a is the vector of (a Then, by Theorem 2
(3) - (X, - &
Rescaling the posterior back to the original parameters gives the desired result.
F Parametric BvM
There are considerable challenges when extending the BvM result to parametric models satisfying only weak regularity conditions such as differentiabilityy in quadratic mean. Consider replacing the normal distribution in Algorithm|] [ with general parametric model Po(-lay) with 0 = 3 -TTxy. xry. We may want to show that in this setting, the posterior distribution is asymptotically normal similar to the BvM statement, i.c.
28

---

[0,0HH,) - JT'0 '|rr 3 0
where In = = -j== Eo. (3) is the empirical Fisher Information and fg = log Pa is the log- likelihood. Suppose follow the steps of the classical BvM proof. Letting x (0) H,,) and NC. (O, JT denote truncations of these distributions to the set C., the classical proof proceeds in the following three steps-we show each of the following convergences:
|m(0|) (00 2
(0|0 - NC- J/ |/|TV [ NV |||VV & 0.
The first step requires a method of truncating the posterior distribution, which is in general quite difficult without strong assumptions on the behavior of the adaptive system similar to the results of (18) and [ΣI] With independent data, we could truncate the posterior due to Hoeffding bounds on the likelihood, but these bounds do not apply in adaptively collected data. The second step was implied by second-order Taylor expansion of the log- ligeliil in the classical proof. but in adaptive settings, the Fisher information matrix Jn may grow at different rates in different directions, i.c. (Ja) may grow at different rate than Amax (Jn). This would make argument based Taylor expansion more complicated and a valid proof seems to require further assumptions on the growth rate of Jn. Finally, the third step
implied in the i.i.d. setting the local consistency of 6n i.c / do) = Op (1) but this condition may be violated in adaptive experiments. In fact, as Lai and Wei showed, require some nontrivial assumptions the empirical Fisher Information to guarentee consistency, even the case of Gaussian regression.
G Real-world example
We compute the BvM total variation distance on a real-world instance of Bernoulli Thompson sampling provided in (28). The dataset consists of the interaction of a fashion recommendation algorithm with users, where the algorithm is optimizing for the number of clicks on fashion items. Although the original dataset is modeled with contextual bandit setting with batch size of 3, we simplify their environment to an 80-armed bandit problem by disregarding the context and flattening the batch size. These simplifications were made solely for computational convenience. The dataset consists of 12m impressions or time steps, during which we update the posterior distribution with Beta(1, 1) prior. The BvM total variation distance throughout this rollout is shown below.
1.0
TV dist
0.8
0.6-
0
10
Steps (millions)
Figure 4: BvM TV distance for the Zozo dataset with prior Beta(1, 1). TV estimates computed with 10* samples and have standard error at most 0.004.
As seen in Figure the convergence of the total variation distance is quite slow. This is perhaps due to the sparse success rates in the Bernoulli bandit and the larger dimensionality of the parameter
29

---

space. It should be noted that the original dataset contained missing data within batches, i.c. not all batches had size 3.
H
Plots
Lin-UCB on Contextual Bandit
NCEC on LQR
1.00
1.00
convex
determined stabilizable
dominated
0.98
duplicate
0.98
unstabilizable
Coverage
Coverage
0.96
0.96
0.94
0.94
0.92
0.92
0
200
400
600
800
1000
200
400
600
800
1000
Time
Time
Figure 5: Frequentist coverage of the Bayesian credible interval the contextual bandit and LQR, under the same configuration as Section Coverage estimates shown have standard error at most 0.004.
UCB on Gaussian Bandit
UCB on Bernoulli Bandit
1.00
1.00
[0.
0.5]
[0, 0.1)
[0.4, 0.5]
0.98
[0, 0.2]
0.98
[0.5, 0.5]
Coverage
[0, 0.3]
Coverage
[0.5, 0.6]
0.96
[0, 0.4]
[0.5, 0.7]
0.96
0.94
0.94
0.92
2000
4000
6000
8000
10000
2000
4000
6000 Time
8000
10000
Time
UCB Poisson Bandit
1.000
0.975
Coverage
0.950
0.925
[0.2, 1]
T0.5, 11
0.900
[1, 1]
[1. 2]
0.875
[1, 5]
0
2000
4000
6000 Time
8000
10000
Figure 6: Frequentist coverage of the Bayesian credible interval for UCB on Gaussian bandits, Bernoulli bandits and Poisson bandits, under the same configurations Section 6 estimates shown have standard error at most 0.004.
Coverage
30


Reviews:
[{"review_id": "Z9Dm1AVd0w", "reviewer": "Reviewer_iYuq", "summary": "This paper addresses the challenge of uncertainty quantification (UQ) for adaptively collected data, which arises in settings such as adaptive experiments, multi-armed bandits (MABs), and reinforcement learning. The core contribution is an extension of the classical Bernstein-von Mises (BvM) theorem to adaptively collected data, establishing asymptotic equivalence between Bayesian UQ and Wald-type frequentist UQ in these complex scenarios. One key takeaway is that Bayesian UQ can be asymptotically valid in a Bayesian sense even in adaptive settings, but asymptotic frequentist validity is now always assured — especially when the stability condition fails. In other words, posterior credible intervals can be misleading from a frequentist perspective, even though they are valid in a Bayesian sense (Bayesian validity).", "questions": "No further questions.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "1. Strengths:\n\n(S1) The proposed BvM theorem for adaptively collected data rigorously establishes when Bayesian UQ aligns with Wald-type frequentist UQ in adaptive settings, addressing a major gap in theoretical statistics.\n\n(S2) The main results are proven under mild conditions, notably not requiring the stability condition often assumed in prior works (e.g., [1]), thereby broadening the range of applicable scenarios.\n\n(S3) The paper provides a clear distinction between Bayesian validity vs. frequentist validity: while Bayesian UQ is always asymptotically valid in a Bayesian sense as long as the prior “washes out”, the asymptotic frequentist validity is not always guaranteed — especially when the stability condition fails. This distinction is crucial for practitioners.\n\n\n2. Weaknesses:\n\n(W1) Even though the BvM theorem is shown to hold without the stability condition, statistical inference with frequentist validity is not ensured in these cases. The paper demonstrates that posterior credible intervals can fail to have correct frequentist coverage without the stability condition.\n\n(W2) The author didn’t discuss the exact meaning of some key terminologies including asymptotic frequentist validity and Bayesian validity in the paper. While statisticians may be familiar with these terms, many researchers in ML or artificial intelligence might not be, so I believe it would be helpful to provide a direct explanation.\n\n\n[1] Tze Leung Lai and Ching Zong Wei, “Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems”, The Annals of Statistics, pp. 154-166, 1982.", "was_revised": false, "final_justification": "", "rebuttal": "(W1) It is correct that our results imply that, in some cases, Bayesian credible intervals will fail to be asymptotically frequentist-valid when frequentist MLE-based inference is invalid (e.g., possibly because the stability condition is not satisfied). However, instead of a weakness, we view this as an interesting consequence of our work. In particular, our main contribution is not a new method of implementing frequentist-valid inference, but rather to explain the behavior of Bayesian inference from a frequentist perspective. We note in our “Discussion and Future Work” section that asymptotic frequentist validity may fail even when the BvM statement holds, and we discuss potential ways a practitioner could mitigate this issue. For the sake of clarity, we will add the sentence “Note that this paper does not suggest a method of asymptotically frequentist-valid Bayesian inference; in fact, we would like to warn practitioners against using either Wald-type frequentist UQ or Bayesian UQ as if it were asymptotically frequentist-valid.” to the “Discussion and Future Work” section.\n\n(W2) We will give explicit definitions for asymptotic frequentist validity and asymptotic Bayesian validity in the introduction by adding the following sentences to a “Notation” paragraph after the “Contributions” paragraph in section 1: “Throughout this paper, we refer to a confidence set as asymptotically frequentist-valid at level alpha if the limit inferior of the frequentist coverage is at least 1-alpha for any parameter value. Similarly, we refer to a credible set as asymptotically Bayesian-valid at level alpha if the limit inferior of the Bayesian coverage is at least 1-alpha.”"}, {"review_id": "nhsRMN6g8O", "reviewer": "Reviewer_VNGk", "summary": "The authors derive a BvM theorem in a parametric setting for data that is sequentially observed in a linear model. Specifically, a data stream $H_i = ((y_1,x_1),\\dots,(y_i,x_i))$ where $x_{i+1}$ can be chosen on the basis of $H_i$, after which $Y_{i+1} = \\beta^\\top x_{i+1} + noise$ is observed, with iid Gaussian noise with known variance. \n\nThe authors show a BvM theorem for $\\beta$, comparing the posterior to a normal distribution centered at the MLE and with (empirical) Fisher information as its variance.", "questions": "Is there a reason why the proof technique would not work in the typical generality of parametric BvM's? I.e. likelihoods that differentiable in quadratic mean?", "limitations": "The main limitation in my eyes is mentioned in the strength/weaknesses section of the review.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper is well written.\n* The topic is interesting.\n* I like how the link to bandits is fleshed out in the paper.\n* The proofs appear sound.\n\nWeaknesses:\n* The setting is very limited; I see no reason why the proof technique would not work in the typical generality of parametric BvM's and there is a lack of discussion concerning this in the paper.\n* There is a lack of literature overview / comparison of BvM's with non-i.i.d. data. How does the result relate to investigations such as [1]? Does general (non-iid) machinery as developed by [2] not apply? \n\n[1] J. D. Borwanker, G. Kallianpur & B. L. S. Prakasa Rao. The Bernstein–von Mises Theorem for Markov Processes. Annals of Mathematical Statistics\n[2] Ismaël Castillo & Judith Rousseau. A Bernstein–von Mises theorem for smooth functionals in semiparametric models. Annals of Statistics\n[2]", "was_revised": false, "final_justification": "", "rebuttal": "We have spent considerable effort thinking of ways to extend our results to more general parametric models. We discuss in Appendix F reasons why our proof technique does not extend to the setting of general parametric models. Briefly, the main reason is that in general parametric models, we require an accurate second-order Taylor expansion of the log likelihood, but the unbounded condition numbers of the Fisher information matrix make it difficult to give a sufficiently strong error bound (even with the assumption of differentiability in quadratic mean). Please let us know if our discussion in Appendix F fails to clarify the reason that our proof technique does not generalize. Also, we’ll be sure to explicitly state the goal of generalizing to parametric models that are differentiable in quadratic mean, as we currently do not mention q.m.d. in Appendix F (though this class of models was exactly what we had in mind in our attempts to generalize).\n\nWe compared our results with previous BvM results in adaptively collected data in the second paragraph of our “Background” section. Prior results assume that the ratio of the maximum and minimum eigenvalues of the Fisher information matrix are bounded, and in fact most assume the stronger condition that there exists some consistent norming rate $\\delta_n \\to 0$ where $\\delta_n I_n$ converges in probability to some limiting Fisher information $I$. The results established by [2] also make this assumption where this norming rate is $1/n$, as the posterior distribution is shown to be asymptotically normal with variance $V_0/n$ for some fixed $V_0$. The assumption of a bounded condition number is not satisfied by optimal bandit algorithms as discussed in section 2 of our paper, and therefore both our theorem statement and proof technique do not require this assumption. To our knowledge, our result is the first non-i.i.d. BvM result to accomplish this. For the sake of completeness, we will add [2] to our block of citations on line 82. The result given in [1] provides an asymptotic error bound on the BvM distance, providing a more quantitative analysis of the convergence rate, similar to other prior works such as [3] which gives a nonasymptotic error bound. Their argument, like ours, requires truncating the posterior distribution, which they argue by proving bounds on the observed likelihood and empirical Fisher information. However, their argument relies critically on Doeblin’s condition which guarantees a fixed rate of convergence of the Markov chain to a stationary distribution, so it is difficult to generalize their result to non-Markovian settings such as bandits.\n\n[3] Lee, Jeyong, Junhyeok Choi, and Minwoo Chae. \"Online Bernstein-von Mises theorem.\" arXiv preprint arXiv:2504.05661 (2025)."}, {"review_id": "McAjcqIvJc", "reviewer": "Reviewer_RwF3", "summary": "This paper extends the Bernstein-von Mises (BvM) theorem to the adaptive data collection setting, in which the data is non-i.i.d. and the ratio between maximum and minimum eigenvalues of the Fisher information matrix is unbounded. It establishes that under such settings, with mild eigenvalue growth conditions, BvM-analogous convergences hold, implying Bayesian credible intervals retain frequentist validity asymptotically, even under adaptive data collection. The authors prove BvM results for adaptive linear Gaussian models, multi-armed bandits, and contextual bandits, supported by simulations validating convergence and coverage properties.", "questions": "Bayesian UQ can match frequentist UQ asymptotically, yet be frequentist-invalid when stability fails\n\n1. The bounded $b^{\\prime\\prime\\prime}$ requirement excludes some exponential families (e.g., Poisson near zero). Is this necessary? Can the proof be adapted without boundedness?\n2. Can results be validated on a public adaptive dataset? If not, please discuss challenges.\n3. Theorem 1 claims Bayesian validity under prior misspecification, but how quickly does this \"washout\" occur? Simulations demonstrate slow TV convergence for large margins. Please discuss the implications for finite-sample safety.\n4. The relaxation of the usual stability condition doesn't seem to be as significant as presented, since we still need it to guarantee frequentist-validity, or am I missing something?", "limitations": "Suggestions:\n1. Discuss mitigations for instability-induced invalidity.\n2. Address finite-sample prior sensitivity observed in simulations.\n3. Expand societal impact: Could invalid UQ in adaptive trials lead to harmful decisions?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- First BvM extension to adaptive data, addressing a critical gap in uncertainty quantification for bandits/RL.\n- Broad applicability to Gaussian bandits, exponential families, and contextual settings.\n- Theorems are rigorously proven with careful handling of non-i.i.d. likelihoods and anisotropic Fisher information.\n- Simulations validate positive/negative results across diverse adaptive settings (MAB, contextual bandits, LQR).\n\n**Weaknesses**\n1. Clarity issues:\n- The umbrella term *uncertainty quantification (UQ)* is used throughout. The author should use explicit terms of *Bayesian credible intervals* and *frequentist confidence intervals*.\n- Jargons like \"Wald-type\", \"frequentist-valid\" are used without first establishing precise definitions.\n- Using $H_n^n$ for triangular arrays (Section 4) alongside $H_n$ for fixed-$n$ trajectories is confusing\n2. Technical concerns:\n- Proof sketches (Section 3) omit key steps (e.g., why truncation works without bounded condition numbers)\n- Convergence in TV implies pointwise agreement but may not ensure uniform frequentist validity. Authors should discuss implications for confidence sets.\n3. Experimental limitations:\n- All experiments use synthetic data. No validation on real adaptive datasets.\n- Coverage plots are only shown for MAB. For contextual bandits/LQR, only TV distance is reported, leaving coverage unexplored.", "was_revised": true, "final_justification": "The authors duly addressed my concerns during the rebuttal and enhanced the clarity. The paper is solid with potential impact on future research. I raise my score from 3 to 5.", "rebuttal": "**Clarity**:\n\nWe chose to use the term “uncertainty quantification” in the paper because we wanted to include a variety of different use cases, not just constructing confidence/credible intervals (although these are what our empirical evaluations focus on). For instance, frequentist asymptotic normality is often used for forms of UQ other than confidence interval construction such as (multiple) hypothesis testing or prediction intervals (i.e., a high probability region for the response given a new test X). In the Bayesian paradigm, posterior distributions (and approximations thereof) are also often used for hypothesis testing and constructing posterior predictive intervals. To clarify our terminology, we will add the sentence “We will use the term frequentist (resp. Bayesian) UQ generically to refer to any of the many forms of frequentist (resp. Bayesian) statistical inference such as hypothesis tests, confidence intervals, or prediction intervals (resp. such as Bayesian hypothesis tests, credible intervals, or posterior predictive intervals).” to the introduction of the paper. We would prefer not to directly replace the phrase “UQ” as suggested, as that would restrict the use cases to frequentist confidence intervals and Bayesian credible intervals.\n\nWe will add precise definitions of “Wald-type” and “frequentist-valid” in the introduction section of our paper after the “Contributions” paragraph: “By Wald-type frequentist UQ, we refer to standard asymptotic hypothesis testing or confidence interval construction using the MLE and its asymptotic Normality—for instance, as demonstrated in Example 15.6 of [1].” and “Throughout this paper, we refer to a confidence set as asymptotically frequentist-valid at level alpha if the limit inferior of the frequentist coverage is at least 1-alpha for any parameter value. Similarly, we refer to a credible set as asymptotically Bayesian-valid at level alpha if the limit inferior of the Bayesian coverage is at least 1-alpha.”\n\nRegarding the notations $H_n^n$ and $H_n$, could the reviewer please elaborate on what specifically they found confusing? We would be happy to change our notation accordingly, but weren’t sure what aspect the reviewer was referring to. In case it helps, the reason we chose these notations was as follows: we use $H_j$ (e.g., in Algorithm 1) to denote the history (hence the letter $H$) up to time $j$. In the non-triangular array setting this history extends as the subscript increments, i.e., $H_{j+1}$ is simply $H_j$ with one more time step concatenated, so there is only one sequence of histories. But in the triangular array setting, the sequence of histories can be completely different for different $n$, i.e., $H_j^n$ and $H_j^{n+1}$ are not the same, so we need notation to distinguish them, hence the superscript so that $H_j^n$ and $H_j^{n+1}$ have distinct notations. Then, when we look at the entire history up to time $n$, which is an important quantity, the subscript also takes the value $j=n$, and we get the notation $H_n^n$, meaning the history up to time step $n$ of the $n$th trajectory in the triangular array.\n\n**Technical Concerns**:\n\nWe will add a more detailed explanation of why truncation works without bounded condition numbers to the extent that space allows in our main paper, akin to: “Our proof for why we can truncate the posterior relies critically on the second part of condition (i) of Theorem 1 which allows us to show that the density of the representative normal distribution outside of local ellipsoids converges to zero. We furthermore show that the posterior can be written as proportional to the prior multiplied by the density of the representative normal, meaning we can truncate the distribution to a local ellipsoid even if the prior is unbounded.”\n\nIt is correct that our results are not uniform in the underlying parameter. However, we have not come across any uniform BvM results in the existing literature (even in the standard i.i.d. setting). Additionally, it is generally not the case that the asymptotic normality of the MLE is uniform in the underlying parameter. For example, the (square-root-Fisher-information-rescaled) MLE is not asymptotically Normal in a triangular array setup with p=1/n for IID Bernoulli data. Consequently, the utility of a parameter-uniform BvM statement is unclear as, in most applications, it would not imply uniform frequentist validity of Bayes’ procedures.\n\nYour point about uniformity does raise an interesting question, however, which we had not considered before, which is whether or not our result gives uniform convergence of the TV distance over suitable sets of sampling rules. More formally, the triangular array result Theorem 2 of our paper can be interpreted as the following non-triangular-array uniform convergence result for multi-armed bandits.\n\nCorollary 1. For any sequences $(r_n, \\epsilon_n)$ for which $r_n \\to \\infty$ and $\\epsilon_n \\to 0$, let $\\mathcal{P}\\_n$ be the sequence of sets of distributions $P$ on $H_n$ induced by sampling rules $\\Lambda$ satisfying $P(\\lambda_{\\text{min}}(X_n^\\top X_n) > r_n) > 1-\\epsilon_n$. Then, we have for any $c>0$,\n\n$$\\limsup_{n \\to \\infty} \\sup_{P \\in \\mathcal{P}\\_n} P(||\\pi(\\beta|H_n) − \\mathcal{N}(\\hat{\\beta}\\_n, \\sigma^2(X_n^\\top X_n)^{−1})||_{TV} > c) = 0$$\n\nThis shows that we do have uniform convergence of TV distance over sets of sampling rules where $\\lambda_{\\text{min}}(X_n^\\top X_n)$ grows at some uniform rate. We will add this to our revision as a remark after Theorem 2.\n\n**Experimental Limitations**:\n\nOur validation procedure consists of running multiple iterations of an experiment on synthetic data and comparing confidence intervals to the ground truth to estimate coverage and other quantities of interest. This process would be difficult to execute in real world applications where we often don’t have access to the ground truth (perhaps only estimates of it). We can, however, include examples of real-world data and report the TV distance between the Bayesian posterior and the MLE-centered Gaussian distribution (with variance-covariance matrix using the empirical Fisher information matrix, rather than the true one, as the latter is unknown to us). However, it should be noted that, as we established, Bayesian credible intervals may fail to be asymptotically frequentist-valid (in the same settings in which Wald-type intervals are asymptotically valid).\n\nShowing coverage plots for the other non-MAB environments is a great suggestion - we’ll be sure to add these plots to the Appendix (due to space limitations) in the revision.\n\n**Questions**:\n\nWe apologize that we did not specify this more clearly earlier, but the bounded $b’’’$ condition is actually satisfied for all natural exponential family models (provided that the $\\beta_{0,i}$ parameters are in the interior of the parameter space). This is because the log-partition function $b(\\eta)$ is smooth on the interior of the natural parameter space (see, e.g., Theorem 2.2 of [2]) and consequently it is bounded on any sufficiently small neighborhood of any interior point.\n\nIn the case of Poisson, for example, letting $\\eta=\\log \\lambda$ be the natural parameter of the Poisson distribution, we have $b(\\eta)=\\exp(\\eta)$ meaning $b’’’(\\eta) = \\exp(\\eta)$, which is bounded on a union of neighborhoods around a finite number of true parameters. We will replace condition (i) with just “$\\beta_{0,1},\\ldots,\\beta_{0,p}$ are in the interior of the natural parameter space”.\n\nThe “washout” effect can indeed be logarithmically slow if the minimum eigenvalue of the Fisher Information grows logarithmically, which is certainly a problem for finite-sample validity which we agree should be better acknowledged. We will add some text discussing this to the last paragraph of Section 3: “Note, however, that the rate at which the ‘wash out’ effect occurs depends on the rate at which the posterior distribution converges in TV distance to a normal distribution, which may be logarithmically slow for optimal bandit algorithms. Thus, it may not be accurate in finite samples to treat a misspecified prior as having `washed out’.”\n\nYes, if we want to show asymptotic frequentist-validity using our theory, we do need the MLE to be asymptotically normal, which can be guaranteed by regularity conditions such as stability. What’s surprising is that the step that allows us to conclude asymptotic agreement between Bayesian inference and Wald-type frequentist inference, i.e. the BvM statement, still works without the stability condition. In this sense, our result is not a relaxation of the stability condition on frequentist results but instead a new theorem about the behavior of Bayesian inference from a frequentist perspective.\n\n**Suggestions**:\n\nOur work does not provide a mitigation to instability-induced instability, and as far as we know, there is no Bayesian mitigation for this issue. We will revise our Societal Impact section by addressing the fact that Bayesian UQ may be used as if it were frequentist valid. We will include a warning that in general, Bayesian UQ should not be treated as frequentist valid in adaptive settings.\n\n[1] Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.\n[2] Brown, Lawrence D. \"Fundamentals of statistical exponential families: with applications in statistical decision theory.\" Ims, 1986."}, {"review_id": "coHhvWzdNZ", "reviewer": "Reviewer_DzMC", "summary": "This study investigates the Bernstein–von Mises (BvM) theorem for adaptively collected data. Uncertainty quantification is a central issue in statistical analysis, and the authors aim to use the BvM theorem to bridge Bayesian and frequentist perspectives in this setting.", "questions": "See above.", "limitations": "N/A.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I recognize that the authors tackle ambitious tasks in adaptive experimental design. I also agree that uncertainty quantification for adaptively collected data is important, and the manuscript seeks to provide a general and powerful tool for this purpose.\n\nNevertheless, I encountered several difficulties while reading the draft, listed below.\n1. What are the precise definitions of $\\Lambda$, $\\lambda_{\\min}$, and $\\lambda_{\\max}$?\n2. What is the formal definition of the sampling rule?\n3. Do the assumptions required for each theorem hold?\n(If I have overlooked these definitions, I apologize to the authors.)\n\nAmong these points, I believe the third is the most critical. The authors remark that suboptimal arms are pulled only $O_p(\\log n)$ times. I agree that this assumption is central to the field, and, in practice, it renders many methods unrealistic. I would like to understand how the authors address this issue.\n\nFor example, Hadad et al. (2021) and subsequent work by the same group show that asymptotically normal estimators can be obtained under appropriate assumptions in bandit problems. However, when I examine their theorems, I find that the convergence rate becomes $1/\\sqrt{\\log(n)}$ for optimal algorithms, implying that, relative to the usual $1/\\sqrt{n}$ rate, we would need $\\exp(n)$ samples to achieve comparable performance for normal approximation. I consider this requirement unrealistic.\n\nAlthough I cannot completely follow the notation, I suspect that a similar convergence rate is implicitly required through $\\lambda_{\\min}$. If so, I ask the authors to clarify and justify the assumptions. If not, I would appreciate an explanation of how they avoid the problem described above.", "was_revised": true, "final_justification": "Personally, I am a bit skeptical about the results of the paper. However, I do not have any concrete evidence to support this view. My intuition is that the uncertainty could be assessed using a martingale central limit theorem or a concentration inequality, and when these do not hold, obtaining a BvM-type guarantee—while supported from a Bayesian perspective—may not yield practically useful bounds. That said, this is merely my personal intuition, so please do not put much weight on it. Since there does not appear to be any fatal error in the proofs, I will stick to my original score and vote for a borderline accept.", "rebuttal": "To clarify our notation, we will add the sentence “Let $\\lambda_{\\text{min}}(A)$ and $\\lambda_{\\text{max}}(A)$ denote the minimum and maximum eigenvalues of the matrix $A$ respectively” in the introduction of our paper. We will also clarify at the beginning of section 3 that $\\Lambda$, our sampling rule, is allowed to be an arbitrary function of the history up to the current data point; i.e. it is meant to denote a placeholder that can represent any sampling algorithm including UCB, Thompson sampling, autoregressive models, etc.\n\nWe included Theorem 2 as a separate theorem specifically to address the setting of multi-armed bandits, which does not satisfy the assumptions of our main theorem. Even though the assumptions of Theorem 2 are satisfied in the bandit setting, the convergence provided by our theorem may be logarithmically slow if suboptimal arms are pulled at logarithmic rates. In other words, we don’t avoid the problem you identified, and we will clarify in our paper after our main theorem that the convergence can be very slow."}, {"review_id": "w7T6T50ip1", "reviewer": "Reviewer_7V5X", "summary": "This paper presents Bernstein-von Mises (BvM) results for settings in which data collection is adaptive. Traditional BvM results will often operate under assumptions that are not entirely aligned with adaptive contexts, like assuming iid data, so the results presented here are a kind of extension, with some particular emphasis on applicability to specific types of Linear Gaussian and Exponential Family bandit settings. Further, they claim to have achieved these results without assuming traditional stability conditions that would otherwise ensure \"frequentist-validity\", and yet \"bayesian-validity\" is still achieved. Finally, numerical experiments examine the nature of the convergence that is guaranteed by these results.", "questions": "# Question 1: \nThe authors write *\"$\\ldots$ Lai and Wei showed that condition (i) of Theorem 1 is necessary for MLE consistency (which our proof relies on), as there are counterexamples where the condition is only marginally violated but consistency fails [20, Example 1].\"* \n\nThis line is rather confusing. If I'm understanding correctly the literal meaning of this statement, the authors are indicating that MLE consistency implies condition (i) because of the existence of counterexamples? There is something very imprecise here, considering later on, the authors also dive into an outlined hypothetical: *\"Second, even if $\\hat{\\beta}_n$ is consistent but condition (i) fails, it may not be the case that $\\ldots$\", which their earlier statement, if taken literally, would preclude the possibility of. \n\nAs far as I understand, Lai and Wei actually show that condition (i) (and other assumptions) suffice for there to be (strong) consistency, and then their cited example is meant to show that condition (i), in their words,*\"is in some sense weakest possible.\"* So I'm guessing the authors actually meant to say something like, Lai and Wei showed that condition (i) of Theorem 1 is *nearly* necessary for MLE consistency.\n\n# Question 2:\nThe restriction to a Gaussian statistical model, as outlined in Section 3 $\\ldots$ can the authors speak on the extent that their analysis relies on this?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I'm not entirely sure what to make of the fact that this theory paper has a missing appendix and hence no proofs. This leaves me on the fence, and I am preliminarily submitting a borderline rating that leans towards accept (I'm trying to be optimistic), and will look forward to the rebuttal phase and the comments/guidance from the chair. \n\n# Strengths:\n- There are some helpful, high-level explanations in the main body of the role that various hypotheses and conditions ( e.g. condition (i) ) have in the theorems. There are also informative summaries of how the proofs (though missing) proceed. As well, I appreciated the examples, remarks and discussions surrounding the possibility of finding frequentist-invalidity but Bayesian-validity. \n- The topic is motivated, and incorporates some Bandit settings \n\n# Weaknesses:\n- Appendix appears to be missing $\\ldots$ so no proofs provided in this manuscript. However, this omission of the Appendix seems to be accidental. A bit awkward to assess the level of theoretical contribution without the proofs. There are certainly numerical experiments that illustrate the Bayesian validity, but this work is a theory paper.\n- The authors largely restrict to a Linear Gaussian observation scheme", "was_revised": true, "final_justification": "My initial assessment was already positive (see above), though I pointed out that the appendix section containing all the proofs did not appear attached. The authors kindly pointed me to the separately zipped supplementary folder containing the appendix. I have since reviewed the proofs in the appendix. \n\nIf I'm not mistaken there are some minor typos (e.g. line 424), but they didn't get in the way of understanding the arguments. And I now raise the overall score from 4 (Borderline accept) to 5 (Accept).", "rebuttal": "We apologize for any confusion regarding the appendix section. We included it in the supplementary material - after unzipping the file, there should be a folder containing a file named “appendix.pdf” and a folder named “code”. The appendix may be found in the “appendix.pdf” file. These files are showing in our system as having been uploaded and when we download them ourselves, they appear as we expect, but let us know if there is an issue with accessing our appendix and we can try to fix it. We believe that the supplementary material was accessible to at least some of the other reviewers, as we have received comments from other reviewers regarding our proofs (which appear in the appendix).\n\nWe briefly discuss in Appendix F reasons why our proof technique does not extend to the setting of general parametric models; please let us know if our discussion in Appendix F fails to clarify the reason that our proof technique does not generalize. We apologize if this was not accessible for you when you reviewed our paper.\n\nThank you for catching this issue in Question 1! We should have written it differently, and we will be sure to clarify in our revision what we meant to say: “First, Lai and Wei showed that condition (i) of Theorem 1 is _nearly_ necessary for MLE consistency. In particular, if condition (i) of Theorem 1 is removed, the statement becomes false.”\n\nThank you for pointing out the formatting mistake on line 92 - it will be corrected."}]

Author rebuttal:

Meta-review: The paper proves a Bernstein-von Mises theorem for the regression coefficients in a parametric linear-Gaussian model where covariates are chosen according to an arbitrary function. It then applies these results to Bayesian uncertainty quantification in Gaussian multi-armed bandits and adaptive Gaussian linear bandits.

Reviewers found the paper well-written, with mathematically sound proofs, useful bandit applications, and reasonably mild assumptions. They noted limitations stemming from the reliance on linear-Gaussian structure and a relatively limited discussion of BvM in non-i.i.d. settings. The authors’ rebuttal addressed these concerns convincingly. I recommend acceptance.
Decision: Accept (poster)