Bandit Quickest Changepoint Detection

Abstract: Many industrial and security applications employ a suite of sensors for detecting abrupt changes in temporal behavior patterns. These abrupt changes typically manifest locally, rendering only a small subset of sensors informative. Continuous monitoring of every sensor can be expensive due to resource constraints, and serves as a motivation for the bandit quickest changepoint detection problem, where sensing actions (or sensors) are sequentially chosen, and only measurements corresponding to chosen actions are observed. We derive an information-theoretic lower bound on the detection delay for a general class of finitely parameterized probability distributions. We then propose a computationally efficient online sensing scheme, which seamlessly balances the need for exploration of different sensing options with exploitation of querying informative actions. We derive expected delay bounds for the proposed scheme and show that these bounds match our information-theoretic lower bounds at low false alarm rates, establishing optimality of the proposed method. We then perform a number of experiments on synthetic and real datasets demonstrating the effectiveness of our proposed method.


Reviews and author responses:

 * We would like to point out that this paper is the first to exhibit detection delay bounds, both upper and lower, as a function of the false alarm rate parameter $\alpha$, for the problem of adaptive, sequential quickest changepoint detection. 

* We completely characterise optimal detection delay performance in the low false alarm ($\alpha \ll 1$) regime, as evidenced by our upper and lower bounds in Theorems 1 and 2. There has been nothing in this regard known rigorously for this problem before this. 

* **Relevance of this regime**: This is the low false alarm regime which is the de facto standard regime commonly encountered in practice, and studied widely for the following problems:
A) Classical changepoint detection: The entire literature on classical (i.e., non-adaptive or non-bandit) quickest change detection considers $\alpha \to 0$ and shows optimality results, see e.g., every optimality result in the survey by Banerjee-Veeravalli: https://arxiv.org/pdf/1210.5552.pdf

B) Bandit best arm identification: The modern theory of best arm identification in stochastic multiarmed bandits and stochastic linear bandits is almost completely about optimality results for algorithms in the $\delta \to 0$ regime, where $\delta$ here is the false alarm or error probability. Well-regarded examples of this regime being studied in a long line of work in this regard are: 
* The Track-and-Stop algorithm of Garivier-Kaufmann, COLT-2016: https://proceedings.mlr.press/v49/garivier16a.html
* LinGame by Degenne et al, ICML-2020: https://proceedings.mlr.press/v119/degenne20a/degenne20a.pdf
* Soare et al: https://papers.nips.cc/paper/2014/file/f387624df552cea2f369918c5e1e12bc-Paper.pdf
* The RAGE algorithm of Fiez et al, NeurIPS-2019: https://papers.nips.cc/paper/2019/file/8ba6c657b03fc7c8dd4dff8e45defcd2-Paper.pdf
* The ELIMTIL algorithm of Tao et al, ICML-2018: https://proceedings.mlr.press/v80/tao18a.html

We also remark that the bandit change detection problem is a nonstationary and hence more challenging problem than best arm identification in stationary, stochastic bandits, and we have given a satisfactory resolution in this regime. 

We hope that you can reconsider your evaluation in light of the above points about the value of the theoretical results obtained here. Sure, there is the high dimensional regime that we have not been able to study in terms of bridging lower and upper bounds, but we hope that this does not take away from the value that the sharp optimality argument advanced for the low false alarm rate regime brings to the table here. 

 


 We appreciate reviewer feedback, and are cognizant of the reviewer's prerogative in scoring. Nevertheless, the fact that the paper does not clarify/discuss the precise nature of the results, namely, optimality at low false alarms is simply **not true**. Please see lines 14, 60, 112, 150-153, 287-291. In this context, the types of modifications to the current paper that we foresee are somewhat cosmetic, such as adding a few of the equations that we discussed in the responses, and expanding our discussions. 

On the other hand, that we must solve the high-dimensional case and close the gap on BQCD before it can be published is an unrealistic and unfair expectation. The low-false alarm rate setting for the fixed dimension case is a real problem, and our paper does deal with this case comprehensively. 

 Hi,
Thanks for responding to my concerns, particularly for clarifying that [Gun+21] was using a data set from a previous paper (this was not clear from the previous response and it seemed strange to take a dataset from [Gun+21] and not compare to the algorithm there). However, at present I am inclined to keep my score the same as although there are some nice ideas in this paper, the gap between the upper and lower bounds does not seem to have been properly investigated. It sounds like the authors have some idea for how to close this gap, so ideally this should be done before the paper is published, or at the very least this issue should be discussed honestly in the paper and the cases where the approach is sub-optimal clearly explained. Also, as mentioned in my original review, the readability of the paper could be significantly improved and I encourage the authors to do this.
 

 While we appreciate reviewer sentiment that an ML paper should ideally benchmark proposed method against baselines drawn from state-of-art works, the problem here is new, and to the best of our knowledge the two prior works, [ZM20] (which is unpublished) and [Gun+21] are only tangentially related. As such, it is not possible to apply these methods to the full range of experimental settings reported in the paper (see comments titled ``why not compare Gun+21 ...'' and "ZM20's proposal ..." for more details (see https://openreview.net/forum?id=mxowVJFe8D5&noteId=NJBBfW0F5v and https://openreview.net/forum?id=mxowVJFe8D5&noteId=KoWeH4t5ws ).

That said, we agree that a proposed method must be benchmarked against strong alternatives. Indeed this is exactly what we do. Let us explain this point further. Together with the recently proposed tabulation by reviewer R5bk (see "Experimental results for the Actual Algorithm Theoretically Analyzed" at https://openreview.net/forum?id=mxowVJFe8D5&noteId=F-cBtgAgjPI ), we compared against 3 other baselines. These taken together result in strong benchmarks for the proposed method. Let us point out why this is the case.

**URS (experiment design).** R5bk suggests experimental design. Nevertheless, for our case with homogeneous sensors, URS is indeed the optimal experimental design for our experiments. As such this method is a good sanity check on calibrating the benefits of adaptation. 

**EGCD (decoupled)** is our proposed method. Here, the data used for making a stopping decision ("exploitation") and for estimating the changed distribution ("exploration") are utilised separately. 

**EGCD(full).** This method is the stronger (i.e., more data efficient) variant in that parameter estimation utilizes both data collected during exploration rounds and the data collected for the exploitation rounds. 

**Oracle.** This is an omniscient strategy in that it is aware of both the pre-change and post-change distributions ahead of time, and consequently does not require exploration to estimate the post-change parameter. On the other hand it has no knowledge of stopping time, and thus the problem reduces to the conventional quickest change detection. The Oracle characterizes the gold-standard in that no adaptation is required, and it gives the absolute smallest detection delay for any BQCD problem. 

**Key Findings**
Our proposed method is sandwiched between URS and the stronger variants (EGCD-full) and Oracle. 

1. The fact that we perform substantially better than URS uniformly across all of the experiments demonstrates our gains through adaptive sampling. 

2. The fact that the proposed method closely tracks EGCD (full) suggest that our decoupled scheme is capable of fully leveraging the observations, and the cost of decoupled exploration is small. 

3. The fact that the proposed method tracks Oracle qualitatively across all experiment setups indicates that our method is close to optimal. This is because through adaptivity we achieve performance similar to a competitor who requires no adaptation (recall that oracle knows the change parameters and thus does not need to explore). In point of fact we note in passing that any other method would at best lie between our proposed method and the Oracle. As such this gap is already quite small as observed in experiments.

4. As such, we firmly believe that the results we report and the diverse experiments we conduct provide a rich exploration of problem space, particularly for a paper such as this, whose focus is theoretical.

 We use the well-known MIMII sounds audio dataset which is also used by Gun+21 but for a different problem setting (in fact, this dataset is very commonly used as the basis for testing algorithms for many different problems, not necessarily bandit change detection). More specifically, the MIMII audio dataset gives raw audio data measured from a variety of normal and abnormal machines. We use it as the basis to set up a sequential changepoint detection problem where only localised (i.e., at the level of a single machine) change occurs, whereas Gun+21 use it for setting up a different online changepoint problem.

It is important to note that Gun+21 formulate an essentially different problem of multiple changepoint detection when the sensing actions specifically correspond to fidelities of sampling a data stream, and moreover have costs associated to them. Additionally their setup does not allow for clearly expressing the notion of localised sensing. Moreover, their setting contains additional Bayesian assumptions built into the environment (problem instance) as well as their algorithm. 

Even if we were to contemplate modifying and applying their approach in our experiments, there are many tuning parameters left open such as the crucial choice of Bayesian prior distributions, the cost structure of the actions, and how to translate global sensing with varying fidelities to express local sensing -- this is a nontrivial and unclear exercise. We are not convinced that we can faithfully retrofit their strategy to apply to our setting which is different.

 There appears to be some confusion with regards to what our objective is, and let us try to clarify this here. The main goal of our work is the theoretical understanding and characterisation of BQCD. With this in mind, let us restate the essence of our theoretical findings.

**Gap between upper and lower bounds**
Our work is the first to show a fundamental characterization for the delay performance of BQCD, and certainly the first that exposes the fundamental tradeoff between delay and false-alarm in the adaptive changepoint detection setting. As in any work, our work leaves open questions, which we hope will get resolved in future works.

In particular, the main objective of our work is to characterise the expected delay as a function of false alarm level $\alpha$ (together with the typical no-stop false alarm time $m$). In this context, we view all of the other (problem-dependent) parameters as fixed, and first derive a novel, oracle-type information-theoretic lower bound, which is algorithm-independent. We then seek BQCD algorithms whose expected delay exhibits the same trade-off as the lower bound in terms of the false alarm level. 

Our specific upper bound is that the expected delay of $\epsilon$-BQCD satisfies

$$
\mathbb{E}^{(\nu,\theta^*)}[(\tau-\nu)^+] \leq O(\frac{\log \frac{1}{\alpha}}{ (1-\epsilon)\max_{a} D({\theta^*(a)}||{\theta^{(0)}(a)})}),
$$ 
when $\alpha$ is suitably small, where this regime of $\alpha$ is governed by the specific problem instance (i.e., pre- and post-change parameters and sensing observation distributions and the size or dimension of the parameters). 

This expression matches our lower bound, in the sense that the expected delay of *any* algorithm must satisfy, 
$$
\mathbb{E}^{(\nu,\theta^*)}[(\tau-\nu)^+] \geq \Omega(\frac{\log \frac{1}{\alpha}}{\max_{a} D({\theta^*(a)}||{\theta^{(0)}(a)})}).
$$

We expose the precise dependence of the delay in terms of $1/\alpha$, along with a worked out example showing the dominant term, in the response to reviewer r5bk available at https://openreview.net/forum?id=mxowVJFe8D5&noteId=PzekUXf4uVF \\

**Optimal characterization for fixed-dimension (or fixed instance) setting**
Unpacking our point above, what this means is that, with dimensions and other problem parameters held fixed, there is an $\alpha_0$, that can depend on the specific problem instance (i.e., pre- and post-change parameters and sensing observation distributions and the size or dimension of the parameters) such that for all $\alpha \leq \alpha_0$,
$$
\mathbb{E}^{(\nu,\theta^*)}[(\tau-\nu)^+] \leq \frac{8\log \frac{1}{\alpha}}{ (1-\epsilon)\max_{a} D({\theta^*(a)}||{\theta^{(0)}(a)})}.
$$
Order-wise, this expression matches our lower bound. 

Additionally, after a few algebraic manipulations it is not hard to show that, our result also implies, the following asymptotic statement, again for a fixed instance:
$$ \limsup_{\alpha \to 0} \frac{\mathbb{E}^{(\nu,\theta^*)}[(\tau-\nu)^+]}{ \frac{8}{(1-\epsilon)} \log\frac{1}{\alpha}} \leq  \frac{1}{\max_{a} D({\theta^*(a)}||{\theta^{(0)}(a)})  },$$
while, our lower bound implies that every algorithm for that instance must satisfy:
$$\liminf_{\alpha \to 0} \frac{\mathbb{E}^{(\nu,\theta^*)}[(\tau-\nu)^+]}{ \log\frac{1}{\alpha}} \geq \frac{1}{\max_{a} D({\theta^*(a)}||{\theta^{(0)}(a)})  }.$$



**Characterization for high-dimensions**
We believe that the reviewer is asking for a more complete characterisation of the detection delay as a function of $\alpha$ and $d$ in all possible regimes, including the 'high dimensional' regime where $\alpha$ is fixed but $d$ can be large (or more generally, the problem instance can vary). This is outside the scope of our current work. 

That said, as we have stated earlier in point 8 of the response to Reviewer p3ra, we expect that a tighter lower bound containing a dependence on $d$ should hold (although this requires a formal proof). Such characterisations in the so-called 'moderate confidence' regime have indeed been derived for, say, the simpler and stationary problem of best arm identification in stochastic multi-armed bandits by Simchowitz et al: http://proceedings.mlr.press/v65/simchowitz17a.html . This formalises the intuition that in a high dimensional setting with moderately high error probabilities allowed (the equivalent of $\alpha$ being bounded away from 0 in our setting), the amount of sampling effort must depend on $d$ (the size of the problem instance, number of arms) due to the core of the learning task essentially being a *search* problem over all  parametric instances. We believe that such a task can be accomplished even for the changepoint problem here, but it is in a different spirit and a subject for future work.

 No. It is not the theoretical assumptions but rather the applicability of the method to the full-range of our experiments that is the main issue. As we have commented before (point 6), ZM20 proposes a strategy for a much more limited setting (it assumes independent sensors, precise knowledge of pre and post-change parameters, and sparse and independent changes in parameters). Thus, its setting does not apply to our full range of experiments which involve diffuse, structured / correlated changes. 

In passing, we also remark that, ZM20's algorithm requires the specification of two nontrivial Bayesian prior distributions for (a) the change point time and (b) for the Thompson sampling based subroutine to work. It is unclear how we can faithfully adapt their prescription to our experimental setting. 

Finally, ZM20 does not provide a code base that can be examined, and, as such, we believe we are not expected to get into the details of whether and how it can be modified to apply to our experimental setup, considering that it is not published.

 Hi,
Thank you for addressing some of my comments. There are still a few areas where I would appreciate more clarity.
Firstly, I was not convinced by the reasoning given for not comparing experimentally to prior work. As far as I understand your main argument was that the assumptions made by your algorithm and [ZM20] were different and [ZM20] has no theoretical guarantees, have I understood this correctly? For experimental comparison I do not think this is such an issue, as often algorithms can work in settings where theoretically it is not understood if they should. In the response you also say that you use a data set taken from [Gun+21], is this correct? If so, I find it surprising that no experimental comparison is given to this paper. In general, to properly understand the empirical performance, it would be good to have a stronger baseline than is currently used (another reviewer also mentioned potentially using experimental design as a baseline).
Secondly, I am still curious about the gap between the upper and lower bounds. Your points 7 and 8 above seem to suggest that the lower bound is in fact loose and should contain a log|Theta| term, have I understood correctly? In many cases (aside from the sparse examples you have given), this means that the upper bound will have a linear dependence on d while the lower bound is largely dimension independent, is this correct? I understand your point that in the upper bound, in some cases log(1/alpha) will dominate, but I don't think this will hold for every problem and every alpha. 

 We hope the argument and example clarifies reviewer questions, and we welcome further feedback if there are any further questions.

**Claim.**
*For sufficiently small $\alpha$ the first term dominates the second term in the delay bound expression, Eq. 1 of Theorem 2.*

To establish this claim consider the expected delay bound in lines 733-734 of the supplementary section, for the case of change at time $\nu = 1$. We showed that, upto problem independent constants $C$:
$$\mathbb{E}^{(1, \theta^*)}[\tau]   \leq \frac{8 \log \beta}{\mu^* (1-\epsilon)} +  \sum_{\theta : a^*_{\theta} \neq a^*_{\theta^*}}   \frac{2e^{-\frac{\epsilon^2\Delta_\theta^2 \log \beta}{4\gamma\mu^*(1-\epsilon)}}}{1-e^{-\frac{\epsilon^2 \Delta_\theta^2}{8\gamma}}} +C. $$
The first and second terms corresponds to the first and second terms in the statement of Theorem 2.

Observe that, $\log \beta = \log \frac{m |\Theta^{(1)}|}{\alpha}$. This means that as $\alpha \rightarrow 0$, we have $\beta \rightarrow \infty$. Focusing on the individual terms: as $\log(\beta) \rightarrow \infty$ we notice that the first term approaches infinity linearly in $\log(\beta)$. On the other hand the second term approaches zero at an exponential rate in $\log(\beta)$. Since these terms are continuous expressions in $\beta$, the second term dominates the third term for sufficiently large $\beta$ (equivalently, sufficiently small $\alpha$). Relating this observation to Theorem 2, we indeed see that for sufficiently small $\alpha$, the first term dominates the second term.

*We emphasize that our main goal in the paper was precisely to show that in the limit of small false alarms, the dominant term matches our lower bound expression, in terms of $\log (\frac{1}{\alpha})$.* This is easily seen from the above argument. To simplify the exposition we did not focus on individual values of $\alpha$ that are bounded away from zero case. Nevertheless, the bounds can be improved to obtain tighter constants in terms of $\log(\frac{1}{\alpha})$. 

**Example.** Consider the situation of $d$ sensing actions, represented by the canonical basis vectors in $\mathbb{R}^d$. The pre-change parameter is $0 \in \mathbb{R}^d$, and there are $d$ candidate post-change parameters, each of which is $\delta$ times a canonical basis vector, i.e., $\Theta^{(1)} = {\delta e_i, i = 1, \ldots, d}$. The observation from applying the $k$-th action, $k \in [d]$, for system parameter $\theta$, 
is Gaussian-distributed with mean $\theta_k$ and variance $\sigma^2$. Thus, the aim is to detect a (sparse) change of magnitude $\delta$ in one of the $d$ coordinates in $\theta_0 = 0$, when at any time only a single coordinate can be (noisily) sensed. 

For this example we let the number of dimensions, $d=2$; the false alarm rate of $\alpha = 0.001$ as reviewer suggests, along with the specification that in the event of no-change we should not stop before $m=1000$. We also let $\delta=3$ and $\sigma=0.5$, and an exploration rate $\epsilon=0.5$; and $\gamma=1$. With these choices in place, let us plug-in values for $\mu^*=\frac{\delta^2}{2\sigma^2}$, which is the most-informative KL divergence and $\Delta_{\theta}=\frac{\mu^*}{2d}=4.50$; and finally $\beta = {m|\Theta^{(1)}|\over \alpha}=2\times 10^6$.

With these substitutions, let us compute the first and third terms. We note:

**First term** $\frac{8 \log \beta}{\mu^* (1-\epsilon)}=$ **12.90**

**Second term** $$\frac{2e^{-\frac{\epsilon^2\Delta_\theta^2 \log \beta}{4\gamma\mu^*(1-\epsilon)}}}{1-e^{-\frac{\epsilon^2 \Delta_\theta^2}{8\gamma}}}  = \text{{\bf 0.55}}$$
which is an order smaller than the second term.**

 Thank you for your responses. I think most issues were clarified; the only thing which is still there is comparison to something more substantial and clarity of explanations.

I understand that as alpha->0, the first term dominates and this is indeed interesting. Would you argue that the first term is dominating even for a small values such as 0.001, i.e. 1/1000 false alarm? Can you given an argument or an example?

 **1. Concrete Example grounded in notation would be useful.**
>Concrete examples with clearly enunciated observation and action spaces appear in Sec.~6 (see lines 372-384 and 319-413). We will describe this briefly here for reviewer reference, and bring this discussion forward in the revised version. 

>As a case in point, consider a sensor network composed of $N$ magnetometers in a spatial field, where the goal is to detect sudden appearance of targets. A reasonable model for the scenario is what is written on line 375. Observation at sensor $j$ follows: $S_j(t)=\theta_j \mathbf{1}_{t \geq \nu} + W_j(t)$; $S_j(t)$, $\theta_j$ and $W_n(t)$ are real-valued. $W_n(\cdot)$ is assumed to be IID zero mean Gaussian. A sensor observation exhibits elevated activity if there is change ($\theta_j > 0$), and this elevated activity manifests only among $K$ sensors within a small spatial range of the activity, i.e. the support of $[\theta_j]$ is a K-connected set. The support, the size of $\theta_j$'s and the time of change, $\nu \in \mathbb{R}$ are unknown. Connectivity is defined in terms of spatial proximity of the sensors. Sensors outside this region do not observe changes. The post-change parameter set $\Theta^{(1)}$ is the set of all $[\theta_j]$ whose non-zero support forms a $K$ connected set. Our action set ${\cal A}$ depending on application can admit only pointy actions corresponding to single node observations, or a subset of streams (multi-node observations) or a mixture (linear supervision of node observations), and a query $a_t \in {\cal A}$ results in observation $X_t=\langle a_t, S(t)\rangle$.

**2. Assumptions on parameter knowledge strong.**
>We disagree. The assumption of pre-change parameter makes sense because one usually has sufficient data to calibrate sensors under normal operating regime (see example above). This is analogous to conventional batch anomaly detection problems (see One-class SVMs). In this context, our theoretical assumption is more general than any of the prior works [ZM20,Gun+21] (see below). 
>The example in **1** suggests that it is reasonable to assume post-change process is characterized by a parametric family. Such sensor network scenarios and models have appeared for the static case (see Sec 2 Arias-Castro et.al., Near-Optimal Detection of Geometric Objects, 2005).

**3. non-stationary combinatorial bandits with piecewise constant functions or mixture of different streams?**
>As described in **1,2** above our framework allows for diverse action sets -- from a subset of streams (similar to a combinatorial semi-bandit) or (potentially randomised) linear and non-linear functions of all streams and return observations of any dimension (not necessarily equal to ``the number of streams touched''). However, we emphasize that the task here is fundamentally different from  (combinatorial) bandits in the traditional sense -- there is no notion of `reward' given in the input to maximise as in standard regret minimisation in bandit problems. To see this consider a multi-armed bandit problem with a single arm, then, regardless of whether or not reward distribution is non-stationary, the cumulative regret is identically zero. For our BQCD case, with false alarm approaching zero the detection delay over any horizon, $T$ scales as $O(T)$ detection delay, regardless of pre and post change separation (see Theorem 1). 
    
**4. detecting only one change:** 
>Understanding the limits of performance for adaptively detecting a single change is a prelude to tackling the more general problem of multiple changes. Indeed, the majority of classical changepoint detection theory is focused on understanding delay-vs-false alarm tradeoffs in the single change setting. A natural application of the strategy described in this work to handle multiple changes would be to restart the algorithm each time it guesses a change has occurred, and examine how many multiple changes one can detect successfully in the long run. However, bear in mind that this will require additional assumptions on how often change occurs, when it occurs, etc. Besides, it is known that changes occurring too close by in time are essentially undetectable [see e.g., Mai19 cited in the paper], so more assumptions will be needed. 
    
**5. error in knowledge of pre-change parameter:** This can be handled as a misspecified log likelihood ratio function used in the formulation. As long as the misspecification is small enough that the information gaps $\Delta_\theta$ are not significantly affected, we should be able to recover essentially similar performance bounds (for e.g., if the misspecification reduces gaps by at most a multiplicative factor $1-\eta$).
    
**6. paper similar to [ZM20]. no experimental comparison to this work. results presented here concerned with stopping early in the event of no change, as well as minimizing the delay in detecting a change, whereas prior work just concentrates on the detection delay.**
>Characterizing delay under false alarm constraint is fundamental to the problem (see **3** and R5bk **2**).

>Disagree about [ZM20]. It is unpublished, and its analysis results are too weak to merit a comparison to our work. As such [ZM20] *does not analyze detection delay* for the thresholds chosen to control false alarms. [ZM20] only shows that ``in principle'' their method will control false alarms for sufficiently large thresholds, and independently of false alarm control would asymptotically not ignore sensors where changes manifest. This serves only as a sanity check. In contrast, we are the first to derive fundamental detection delay bounds under a false alarm rate constraint, thus fully characterizing the BQCD problem.
>>*Experiments*. 
This being a theory paper, our goal for the experiments was to carry out more fine-grained performance evaluation of the algorithm beyond the theoretical results. In this context, we experimented with the audio dataset from [Gun+21], and investigated the impact of structured and unstructured spatio-temporal anomalies in the context of fine-grained and diffuse action sets. These experiments depict the role of diverse observation models on performance. [Gun+21] due to its Bayesian setting does not align with our setup.

>>*Applicability of [ZM20]*.
We cannot compare [ZM20] because: (a) [ZM20] assumes independent sensors, precise knowledge of pre and post-change parameters, and sparse and independent changes in parameters. This is an issue because this does not handle correlated or structured anomalies, which is a significant aspect of our part of our experiments; (b) [ZM20] has a number of tuning parameters, the code is not published, and we were not confident in our implementation; (c) [ZM20] in its generic embodiment (distribution $G$ is a point mass--see sec 3.1.2) without these tuning parameters is in essence a pure exploitation approach, and it is unclear that this would work. This makes direct comparison difficult.

**7. beta (Theorem 2) is a linear function of the size of the parameter space, and can imply slow stopping. Basis parameter is trivial. does improvement hold for a more realistic parameter space (e.g. $|\theta| = a^d$ for some** $a$)?
>First, it is worth clarifying role of $\beta$. In our response to R5bk (**2.,10**) we point out that $\beta$ is an internal algorithmic parameter that takes problem specs $(m,\alpha)$ and the action set and chooses an algorithm that delivers performance in the form of Theorem 2. Namely, at $\alpha$ false alarm control (line 258), bounds detection delay (line 260). So what matters is really how $\beta$ manifests in line 258 and 260, and as seen is logarithmic.

>Furthermore, the threshold for the log-likelihood ratio, $Q_t^{(2)}$ in Theorem 2 to trigger stopping is $\log \beta$, not $\beta$ (we inadvertently used $\beta$ in line 4 of algorithm, it should be $\log(\beta)$). This means the detection delay will be the time taken by a positively drifting random walk to reach level $\log \beta$. In this sense, the dependence of the threshold on parameter space size, as suggested by the setting of Theorem 2, part 1 is logarithmic in the number of overall post change distributions. So when $|\Theta^{(1)}| = a^d$ then $\log \beta$ will be $O(d)$.

>Basis parameter space trivial: We disagree. The audio experimental setting -- where it is known that machine failures are rare and hence only a machine is likely to fail -- is a reasonably common example of such a situation (also the example in **1** above).

**8. When we are in the so-called ‘low false alarm’ setting, it seems that the delay detection time is dominated by the first term in Theorem 2, and in this setting it scales logarithmically with the size of the parameter space. This does not appear in the lower bound so I disagree with the claim that the only difference is the $1/(1-\epsilon)$ penalty for epsilon-greedy.**

>Yes, the dependence of the first term of the upper bound on the parameter space is $\log |\Theta^{(1)}|$ as pointed out above. The low false-alarm setting also means that $\alpha = o(1)$ or equivalently $|\Theta^{(1)}| = o(1/\alpha)$, so $\log |\Theta^{(1)}| $ is dominated by $\log(1/\alpha)$, which is what appears in the the upper bound of Theorem 1 (note that Theorem 1 is a special case of the more general Theorem 3 proven in the appendix for any changepoint time $\nu \geq 1$). Therefore, the difference is indeed as stated.

>Is dependence on $\|\Theta^{(1)}\|$ fundamental?
We believe so. We focused on the fundamental interplay between false alarm on detection delay. However a careful analysis should yield a dependence on $\|\Theta^{(1)}\|$. We can draw an analogy to sparse recovery, and specialize to 1-sparse vectors (similar to post-change along a basis vector). We know that information theoretically adaptive sequential design exhibits  $\log\|\Theta^{(1)}\|$ dependence. These problems are quite similar, and so we believe change detection must have a similar dependence.

 **8. (Clarity) 190-198 - why this is the right likelihood - perhaps at least reference to some review text that gives context to people not familiar with this matter.**
>Line 193 spells out explicitly the likelihood of seeing data $X_1, …, X_t$, subject to having played the corresponding actions $A_1, …, A_t$, when a change from $\theta^{(0)}$ to $\theta^\* \in \Theta^{(1)}$ takes place at time $v$; this is just the chained product $\prod_{\ell = 1}^{v-1} f_{\theta_0}(X_\ell|A_\ell)^{E_\ell}  \prod_{\ell=v}^{t-1} f_{\theta}(X_\ell|A_\ell)^{E_\ell}$.
>Finding a pair $(v, \theta)$ for which this likelihood is maximized is the MLE for the true change point $\nu^*$ and post change parameter $\theta^*$. Such an MLE is equivalent to the “tail” arg max likelihood ratio on line 194, and forms the basis of classical (non-adaptive) changepoint detection theory, see e.g., <https://arxiv.org/pdf/1210.5552.pdf> [pages 22-23, equation (49)].
>From our perspective, the focus of classical theory is on detecting change for a fixed action. Our paper extends this theory to the case where we can adaptively choose an action from a set of actions. The major insight of QCD is that for optimal detection delay, one should use tail likelihood ratio processes across candidate post-change distributions and compare them (their maximum or a prior-based average) to a threshold to stop and output change. The *adaptive* changepoint detection problem studied here, however, presents a different added dimension of decision making where a choice of sensing action must also be made along with the usual decision of when to stop.

**9. (Clarity) 267-275 - its not clear to me what exactly is the statement about the second term and what is the interpretation of this distance.**
>We presume that this question is about the notion of gap $\Delta_\theta$ defined in line 272 (recall that $\theta^\*$ is assumed to be the true post--change distribution). Put rather simply, $\Delta_\theta = \frac{1}{2} \min ( \bar{D}(\theta^\*, \theta_0) , \bar{D}(\theta^\*, \theta) )$. We will be glad to include this more compact and transparent form of the gap in the revision.
>The notion of gap $\Delta_\theta$ arises when bounding the probability of the rare event that the ‘inferior’ random walk $Q_t^{(1)}(\theta)$ corresponding to a non-true parameter $\theta$ overtakes $Q_t^{(1)}(\theta^*)$, the random walk/queue of the true parameter $\theta^*$ -- this is explicitly spelt out in the Appendix in lines 665-689. More precisely, the crux of the argument is as follows. The true parameter’s random walk (queue) drifts upward at rate $a:= \bar{D}(\theta^*, \theta_0)$ whereas the non-true one’s walk drifts upward OR downward at rate $b:= \bar{D}(\theta^*, \theta_0) - \bar{D}(\theta^*, \theta)$, say. If $a - b > 0$ (Case 1, line 686), we have a competition between two growing random walks, and we use their midpoint drift $a-\frac{b}{2}$ as a ‘barrier’ to separate them with high probability. On the other hand, if $a-b \leq 0$ (Case 2, line 688), then we have a competition between a growing and a decreasing random walk, so we use the growing barrier with drift rate $\frac{a}{2}$ to separate them with high probability.

**10. (Clarity) 287-291 - It's not clear to me what implications this selection of $m$ has on the practicality of the problem. It seems to me the first term would dominate only if the distance between the two parameters was very small, much smaller than distances within the $\Theta^{(1)}$ set. I would like this to be clarified.**
>We emphasize that in our formulation, $m$, together with $\alpha$, is to be viewed as a problem description parameter that specifies the false-alarm constraint of the problem, which is that when no change occurs at all, the algorithm must refrain from stopping before time $m$ with high probability (at least $1-\alpha$); roughly the `rate’ of false alarm (inverse of duration to false alarm) must be no more than $1/m$ w.h.p. $\beta$, on the other hand, is a parameter that arises in the design of our algorithm, and Theorem 2 describes, in part, how it must be chosen in running the algorithm to correctly meet the false alarm constraint governed by the tuple $(m, \alpha)$. So if $m$ is given to be large enough such that $m = \omega(\log(1/\alpha))$ and $\log(m) = o(\log(1/\alpha))$ (e.g., $m = \log^2(1/\alpha)$), and $\alpha$ is very close to $0$ (this is common in many applications with a very ‘tight’ or rare false alarm requirement), then $\beta = \frac{m |\Theta^{(1)}|}{\alpha}$ becomes quite large -- of order $\log(1/\alpha)/\alpha$ -- and thus makes the first term of (1) dominate the second term which does {\bf not} depend on $m$ or $\alpha$.

**11. (Clarity) 313 - ??**
>Apologies for the missing equation reference (??). It was a reference to an equation which was subsequently phased out, so we will remove the reference and replace with text suitably.

**12. (Clarity) In many places I was not sure if $\nu = 1$**    
>Our results address the general case of $\nu \geq 1$.

**13. Theorem 2 assumes $\nu = 1$? Or is this just for the sketch proof?**
>$\nu = 1$ is assumed {\em only for the proof sketch}, for ease of exposition. Theorem 2 is stated for any $\nu \geq 1$. In the special case of $\nu = 1$, the term $\mathbb{E}^{(\infty)} [Q_\nu^{(1)}]$ trivially becomes $0$.


**14. Similarly is lower bound for $\nu = 1$ or more general?**
>Thanks for this question. We notice that Theorem 1, as stated in the main body, is only for $\nu=1$. However, it is extended in Theorem 3 (Appendix A) to all $\nu \geq 1$. We did not add the more general case in the main paper (this was primarily to keep the already-technical exposition simple), but will be happy to replace Theorem 1 with Theorem 3 if it helps convey the generality of the result.

 **1. novelty of this approach. something must exist from prior literature, at least in the non-adaptive case. No non-trivial baseline little suspicious (see below).** 
>The statistics community (see survey [VB14]) has indeed addressed non-adaptive quickest change detection. Adaptive case is new as also pointed out by [ZM20,Gun+21]. These works are contemporaneous to ours. [Gun+21] proposes a Bayesian approach, and does not provide any theoretical guarantess. [ZM20] is unpublished, and their analysis falls short of providing any bounds on detection delay with false alarm guarantees (see also response to p3ra (**6.**)). In point of fact, ours is the first paper that derives theoretical bounds for BQCD (see **2** below). The major insight in non-adaptive case is that for optimal detection delay, one should use tail likelihood ratio processes across candidate post-change distributions and compare them (their maximum or a prior-based average) to a threshold to stop and output change (see **8.**). The *adaptive* changepoint detection problem studied here, however, presents a different added dimension of decision making where a choice of sensing action must also be made along with the usual decision of when to stop.

**2. Triggering the algorithm too early not desired, yet not captured in the delay metric. Would this be meaningful to induce? From Theorem 2 it seems to me that the early detection is controlled via the parameter $m$ which defines $\beta = O(m)$. there is no clear cut choice of m since the true $\nu$ is not known? How do you address the choice of m in practice?**
>We believe this comment stems from a fundamental misunderstanding of quickest change detection (QCD). First, we emphasize QCD cannot be framed as a regret minimization problem (see response 3 to reviewer p3ra (non-stationary combinatorial bandits)). This kind of false alarm duration ($\geq m$ with probability $\geq 1-\alpha$) specification is widely employed in classical literature on non adaptive change detection (see [VB14]). The tuple $(m,\alpha)$ is part of the problem specifications akin to issuing a statement of ``correctness,'' and analogous to specifying a PAC algorithm’s correctness requirement that its returned answer be $\epsilon$-correct with probability at least $1-\delta$. To build intuition note that we must expect that with $m\rightarrow \infty$ (never trigger) then $\alpha \rightarrow 1$ (poor guarantee). Thus QCD focuses on optimizing detection delay under the $(m,\alpha)$ specification, exactly like a PAC algorithm, which optimizes sample complexity under the $(\epsilon,\delta)$ specification.

>Following the above point, one should view $\beta$ as an internal parameter of our algorithm that takes $(m,\alpha)$ as input, finds a suitable threshold for the algorithm, and asserts that it is correct (line 258) and bounds the detection delay (line 260).

**3. In general I am not sure what is the influence of larger $\nu$ in the context of the theory. It seems to me that the larger the $\nu$ the more difficult is it for the likelihood test to finish since the longer time it will take the likelihood estimator to saturate or am I wrong?**
>This is a very pertinent question, and technically not a fully understood one for the adaptive case. In the non-adaptive setting (only one action and one post-change distribution), there are crisp results for optimal change detection algorithms such as CUSUM that say that the conditional expected detection delay after the change time, i.e., $\mathbb{E}^{(\nu, \theta_1)}[\tau - \nu | \tau \geq \nu]$ is bounded by $\log(d)/D(\theta_1 || \theta_0)$ where $d$ is like the average time to false alarm (in no change). Thus, if we uncondition by multiplying with $\mathbb{P}^{(\nu, \theta_1)}[\tau \geq \nu] = \mathbb{P}^{(\infty)}[\tau \geq \nu]$ (which will fall off exponentially with $\nu$), then $\mathbb{E}^{(\nu, \theta_1)}[(\tau - \nu)^+]$ will fall off to $0$ rapidly with $\nu$. 

>However, a key difference in the adaptive setting investigated here is that the same type of conditional expected detection delay after the change time cannot be proven easily, and the best we can show is an unconditional detection delay bound as in Theorem 2, which will be loose for large change point times $\nu$ (it does not decay with increasing $\nu$). The source of difficulty is intuitively because if we explicitly condition on the event $\{ \tau \geq \nu \}$, it can result in strange distributions of the likelihood ratio queues for each parameter via the behavior of the algorithm. In the extreme case, the conditional distribution of the queue lengths could be such that suboptimal (non-true) post change parameters’ queues dominate the true parameter’s queue, preventing it from being selected for a long time as the winner. We suspect that some kind of ‘most likely trajectory’ to have reached $\tau \geq \nu$ should prevail, as in large deviations theory for sample paths, which may help in analyzing the conditional expected delay. 

**4. What about D-experimental design for the linear case as baseline; or a specific form of G-experimental design where one would take a supremum over all possible responses in $\Theta^{(1)}-\theta_0$. This is a better baseline than just uniformly at random, and yet seems to be an obvious choice.**
>Optimal experimental design can certainly be used to tune the $\epsilon$-exploration distribution in the paper's algorithm -- this could be a further level of optimization that could be carried within the scheme set out here. However, the primary focus of the paper is show how a baseline exploration distribution such as this can be used to give an improved adaptive sampling policy that learns to focus on the location of change. Moreover, this is achieved with explicit guarantees on the false alarm and detection times. 

**5. The approach seems to me to be rather suboptimal at the cost of being theoretically analyzable. ... differentiation of $Q^{(1)}$ and $Q^{(2)}$ ... theoretical such that one can split the proof technique for $t_1$ and $t_2$.**
>The approach of partitioning the past data, and using $Q^{(1)}$ for learning the post change distribution and $Q^{(2)}$ for triggering stopping, is indeed chosen for theoretical convenience.

>More precisely, the stopping rule could, in principle, rely on *all* past data to make the stopping decision instead of only $Q^{(2)}$ -- this would still satisfy the false alarm property as this depends only on a generic martingale argument (Lemma 3 in line 631, appendix). However, using all the past data to identify the MLE for the post change parameter (like line 14 of Algorithm 1) would make it difficult to argue a consistent upward drift of the true parameter's CUSUM statistic to reach the threshold in the detection delay proof. This is specifically because there would be random exploration rounds interspersed between exploitation rounds when the statistic is attempting to drift upward, and these random exploration rounds can introduce negative drift in this phase over which we have little control. The argument in lines 712-719 in the appendix would no longer work.

>This separation of data for decision and exploration, for proving theoretical bounds, is not new -- it occurs, e.g., in typical bandit regret algorithms such as $\epsilon$-greedy or Explore-Then-Commit (see e.g., `Bandit Algorithms', Lattimore \& Szepesvari), even though practical implementations usually eschew it. Even in our paper's numerical experiments, we ended up using the more data efficient approach of using all data to define the stopping rule, and this also seems to give improved detection delay performance.

>We believe, however, that such an explicit separation can be circumvented with more delicate approaches which is a subject of future work.

**6. This strategy seems to be very simple. given what we know for linear bandits.**
>Though we agree that the exploration involved in the approach presented is perhaps `simple', we believe that this is more a virtue than a drawback (Indeed, even the simplistic but efficient $\epsilon$-greedy is one of the first, and still widely adopted, approaches to solving bandit regret minimization problems.). Also, we believe that the idea of learning the 'most informative action' gradually using exploration and the GLRT CUSUM statistic is non-trivial, although perhaps deceptively *simple with the benefit of hindsight*.

>Please note that this is a statistical inference (change detection) problem framed in a non-stationary environment (one where the distribution changes at an arbitrary time). It is also fundamentally different than a traditional bandit problem in which the goal is of optimizing a notion of total  *reward*, as opposed to this setting where the task is to minimize total *time to stop* and also, on the other hand, respecting a certain minimum duration to false alarm stopping. Additionally, as the lower and upper bounds in the paper show, an approach that is more sophisticated in terms of exploration can only help improve detection delay performance in an additive sense compared to EGCD, in the low false alarm (low $\alpha$ and high $m$) regime. Our approach is quite reminiscent of asymptotically optimal approaches for, say, best arm identification in bandits like Track-and-Stop ( <https://arxiv.org/abs/1602.04589> ) which use uniform exploration together with greedily solving an instance-dependent optimization problem for exploitation.

 **Review: It would also be helpful to include the experimental results for the actual algorithm that was theoretically analysed, not just the modification that the authors found worked better in practice.**

*Response:* By way of comparison, we did benchmark even the version stated in the main paper (and covered by Theorem 2). The conclusion is that it is a tad slower in terms of detection delay performance. For e.g., in the context of Table 1 in the main paper (Synthetic experiment with isolated change and pointy actions), while EGCD(full data) is shown to achieve mean detection delays of 100, 120, 170 and 190 for graph sizes 10, 15, 20 and 25, respectively, the corresponding detection delay values for the 'theoretical' version of EGCD are 127, 171, 213 and 263. Note that these are still much shorter than those reported for URS which are 300, 450, 630 and 790. We can add these results to those already in the paper if desired. 

 **Concern: It seems that you also use a variant of stopping rule that is not as in the algorithm but you do not describe in detail in the main paper. Can you comment on these aspects?**

*Response:* Along with the explanation given in another comment for the theoretical difficulties of analysing the EGCD algorithm that uses *all* past data to compute the MLE (and which is used for the experiments), by way of comparison, we benchmarked even the version stated in the main paper (and covered by Theorem 2). The conclusion is that it is a tad slower in terms of detection delay performance. For e.g., in the context of Table 1 in the main paper (Synthetic experiment with isolated change and pointy actions), while EGCD(full data) is shown to achieve mean detection delays of 100, 120, 170 and 190 for graph sizes 10, 15, 20 and 25, respectively, the corresponding detection delay values for the 'theoretical' version of EGCD are 127, 171, 213 and 263. Note that these are still much shorter than those reported for URS which are 300, 450, 630 and 790. We can add these results to those already in the paper if desired. 

 **Comment 1. The distribution pi is not discussed, yet seems to be of moderate relevance to the additive error terms. It is also not specified how pi is selected for the numerical experiments. Additional clarification on the impacts of pi would be helpful.**
    
*Response 1.*
The choice of exploration distribution $\pi$ indeed influences the additive overhead term in the detection delay bound (Equation (1) in the paper). More precisely, its influence is via the average KL divergences

>$$D_{\theta^*, \theta} \triangleq \sum_{a \in {\cal A}} \pi(a) D_{\theta^*, \theta}(a)$$

that governs how fast the CUSUM statistic $Q^{(1)}_t(\theta)$ for each candidate parameter $\theta$ grows after the changepoint [lines 667-677 in the proof in the Appendix]. We can add a clarification for this in the main text. 

In the experiments, $\pi$ is taken as the uniform distribution over sensing actions, which is the same as what the URS strategy employs.

**Comment 2. Some oddities with numerical results; Table 2, URS for size = 20 mean is 236 while standard deviation is 1. Any intuition or explanation for the generally lower std dev of epsilon-GCD (and URS for diffuse actions) versus Oracle?**

*Response 2.*
Apologies -- upon further checking, it appears that the computation of standard deviation for detection delay was incorrect in our code. This was because in the process of preparing the Supplementary experiments-related material, a code update mistakenly led to a wrong parenthesis in a script -- in essence, $\text{Variance}[X-c]$ was being computed as $\text{Variance}[X]-c$ for the stopping time data $X$ and a constant (true changepoint) $c$. 
    
We have fixed the error in the code and the revised versions of Tables 2, 3 and 4 of the appendix are given at this link: https://bit.ly/3lN20gv . The results are in line with the intuition that the `amount of randomness used' by the 3 algorithms (and hence the standard deviation of stopping times) increases in the order Oracle $<$ $\epsilon$-GCD $<$ URS.

**Comment 3. The condition in 269 seems to be brittle and potentially very large. For example, if one possible parameter was** $\theta^\prime = \theta^\* + w$ **for small perturbation** $w$, and $a^*(\theta^\*) + w$ **was another possible action (i.e., this new** $\theta^\prime$ **had a different most informative action than** $\theta^\*$), **then it seems as though the delay would scale with how long it takes for** $\hat \theta_t$ **to settle to** $\theta^\*$ **(the quantity** $t_1$**), which is unnecessary. Is there a clear way to avoid this issue of an alternate** $\theta^\prime$ **with arbitrarily small** $\Delta_{\theta^\prime}$, **and a different optimal action? I understand the overlap with the future work of extending to continuous parameter spaces.**
    
*Response 3.*
This is a valid and pertinent question, thanks. It should be possible to extend the current proof technique to extract a time to settle to a set of 'neighboring' actions which are, say, $(1-\eta)$-most informative where $\eta \in (0,1)$. This will yield bounds that are near optimal, cover smooth perturbations and be more robust in this way. More generally, we believe that the extension to continuous parameter spaces can be carried out via a standard covering argument where each neighborhood of the cover contains parameters with likelihood ratios that are at most, say, $(1\pm \eta)$ multiplicatively distorted $\Leftrightarrow$ the log-likelihood ratios are at most $(1\pm \eta)$ additively distorted. By choosing $\eta$ to be small enough to not significantly affect the drift rates of the CUSUM statistics and also large enough to not make the finite cover size too large (the cover size will now play the role of $|\Theta^{1}|$), we should be able to derive similar near-optimal detection delay bounds as with the finite parameter case. This technical task is proposed to be accomplished in the future. 
    
**Comment 4. Discussion and future work should be in the main text, not in the appendix; with the extra page allotment, it wold be good if this could be brought to the main text.**

*Response 4.* We will definitely consider doing this with the extra page allotted for the final version, thank you.
    
**Comment 5. Minor errata: It is stated that** $a^\*$ **may not be unique in 136, however in 326 it is assumed to be. The exploration indicator** $E_l$ **is not defined prior to its use in 193. In assumption 4, the log is missing in front of the log likelihood ratio in 245. Appendix: Line 764, URS corresponds to** $\epsilon=1$. **Lines 313 and 350 have missing cross references.**

*Response 5.*
Thank you for catching these errors. We are sorry for the inconvenience caused due to the confusion, and will certainly fix these inconsistencies at the earliest. 

 **Extension to infinite parameter spaces**
>We believe the adaptation to infinite parameter spaces can be accomplished by carefully 'binning' the parameters so that KL divergences do not significantly change, and then using a union bound over these finitely many bins instead of the one on line 629. This is in the spirit of chaining methods to control the fluctuations of suprema of empirical processes (see, e.g., `Concentration Inequalities' by Boucheron, Lugosi, Massart). 
    
>More precisely, it should be possible to extend the current proof technique to extract a time to settle to a set of 'neighboring' actions which are, say, $(1-\eta)$-most informative where $\eta \in (0,1)$. This will yield bounds that are near optimal, cover smooth perturbations and be more robust in this way. More generally, we believe that the extension to continuous parameter spaces can be carried out via a standard covering argument where each neighborhood of the cover contains parameters with likelihood ratios that are at most, say, $(1\pm \eta)$ multiplicatively distorted $\Leftrightarrow$ the log-likelihood ratios are at most $(1\pm \eta)$ additively distorted. By choosing $\eta$ to be small enough to not significantly affect the drift rates of the CUSUM statistics and also large enough to not make the finite cover size too large (the cover size will now play the role of $|\Theta^{1}|$), we should be able to derive similar near-optimal detection delay bounds as with the finite parameter case. 
    
>But we defer this technical task to future work and focus on the finite parameter setting to bring out the new and essential concepts in adaptive changepoint theory here. 

 **However it seems to me that Theorem 2 as stated does not reveal much about the problem (at least as stated in the main text). I think it makes sense to provide a specialized result (for linear bandits) and context. I found little value in the actual statement of Theorem 2 since the second term was left very vague.**

*Response:* 
The section on Benefits of Adaptivity (292-309) helps explain Theorem 2 by specializing to a particular (linear) bandit setting, which is the case of sensing actions being able to measure orthogonal directions in the parameter space (the directions being the unit vectors without loss of generality here). 

The main point that we would like to bring out with Theorem 2's detection delay bound is its first term which scales inversely with $\max_a D(\theta^*(a) || \theta(a))$, which is the most amount of information yielded by any of the sensing actions to resolve $\theta^*$ from $\theta_0$. Such a term can be easily computed for any given sensing/information structure including those of the linear bandit type. 

For e.g., with each action $a \in \mathbb{R}^d$ yielding a Gaussian-distributed observation with mean $a^T \theta$ and variance $\sigma^2$ with $\theta$ being the (distribution) parameter in place at the current time instant, the maximum information gain for any true post-change parameter $\theta^* \in \mathbb{R}^d$ simply becomes $\max_a \frac{(a^T(\theta_0 - \theta^*))^2}{2\sigma^2}$. So in this case the most informative action has the geometric interpretation of being the vector $a$ which is 'most aligned' with the 'direction of change' $\theta^* - \theta_0$. 

The extra additive overhead in Theorem 2's detection delay bound does {\em not} depend on the threshold $\beta$ (and hence false alarm considerations), and becomes small compared to the first term when $\beta$ is large, i.e., the system is operated with a very 'stiff' false alarm constraint in mind. It depends on how much delay it takes for the true post-change parameter $\theta^*$ (and thus its most information action) to get picked up for sampling starting from when an actual change has taken place, and is a consequence of analysing how long the random walk $Q_t^{(1)}(\theta^*)$ takes to overtake any other random walk $Q_t^{(1)}(\theta)$, for $\theta \neq \theta^*$ and $\theta, \theta^* $ not having the same most informative action. This is the most important and  part of the performance analysis of this paper, and brings to the fore new challenges in analysing nonstationary processes.

 **The authors propose to use the `detection delay' as the performance metric instead of the classical regret. This is fine, but it could be motivated better.

>Detection delay is one of the most natural cost measures in monitoring applications -- machine degradation, quality control, reliability theory, surveillance, intrusion detection, anomaly detection, etc. You typically want to alert a downstream system or take a retaliatory or constructive action (repair a faulty machine, close a valve, shutdown the premises, invoke security protocols) as soon as possible in time after an undesirable event has taken place. In other words, every unit of time delay spent in not detecting a change that has occurred costs machine failure or theft or some undesirable cost. At the same time, triggering a false alarm in case of no change is not desirable as it causes a cost of unnecessary shutdown or evasive procedures employed, disrupting normal operations. A wealth of literature in anomaly detection and quickest changepoint detection, over several decades, is testimony to this, see e.g., the first paper by Lorden Lor71 and the authoritative textbook BN93. 

**What is ‘the most informative action’ as described? Is there some connection to a partial monitoring problem?**
>The most informative action, alluded to in the Introduction, is clearly spelt out in lines 164-165.

**Connection to partial monitoring:**
>Partial monitoring is a different task than bandit quickest change detection since it is about maximising reward -- a special case of which is the usual multi-armed bandit. The only commonality is  that (sensing) actions do not reveal full information about the underlying system state/parameter. Also, stochastic partial monitoring problems do not model change in the environment distribution, whereas the quickest change detection problem fundamentally aims to detect a change in a nonstationary system. 

**Is** $Q^(2)$ **ever defined in Algorithm 1?**
>$Q_t^{(2)}$ is indeed defined completely in Algorithm 1: At $t=1$ it is initialised to $0$ and for $t> 1$ it is recursively updated as per line 16 of the algorithm. 

**Is** $N_{\tau}(a)$ **defined on line 178?**

>$N_t(a)$ represents the total number of plays of action $a$ up to time $t$. We will supply an explicit definition of this in the revision, thanks. 

**I don’t see the relevance of the queuing in section 4**
>The CUSUM-type statistics $Q_t^{(1)}, Q_t^{(1)}$ can be interpreted as queues, which is a synonym for random walks/accumulating processes that are not allowed to dip below $0$. Each successive log likelihood ratio $E_\ell \log \frac{f_{\theta}(X_\ell|A_\ell)}{f_{\theta_0}(X_\ell|A_\ell)}$, incorporated into $Q_t^{(1)}$, admits the interpretation of an arriving customer or packet into the queue. We felt that this viewpoint is useful in giving a physical perspective on the CUSUM-type statistics, in the sense that before the changepoint, all queues have negative expected drift (and so stay close to $0$), but  after a changepoint, the queue for the true post-change parameter has a large positive drift. The learning problem is to try to find this largest growing queue as soon as possible. 
Exploration helps to focus attention on sampling and growing this queue as much as possible (by playing its most informative action) to overflow the level $\log \beta$.

The paper analyzes online change point detection. A user can play actions that reveal information about the post-change distribution. Some actions reveal more information that others. The post change distribution belongs to a set of possible distributions and is unknown apriori. The authors provide an algorithm that estimates the post-change distribution out a priori known set, and then plays the most informative action such that sequential testing can terminate quickly. They analyze "delay" - time after the changepoint when change is certified by their algorithm. They provide lower and upper bounds for their algorithm and compare it to some other baselines.   Problem 
-------------
I think the studied problem has a clear structure and is very practically relevant as well as to NeurIPS community. Many real-world applications follow this pattern. I enjoyed the problem statement and was not aware of instantiation of the linear bandit model in this context.

Clarity 
-------------
While I was able to follow the paper on a general level and I think authors made across what they wanted.  I had hard times to follow 
- 190-198 - why this is the right likelihood - perhaps at least reference to some review text that gives context to people not familiar with this matter
- 267-275 - its not clear to me what exactly is the statement about the second term and what is the interpretation of this distance
- 287-291 - It's not clear to me what implications this selection of “m” has on the practicality of the problem. It seems to me the first term would dominate only if the distance between the two parameters was very small, much smaller than distances within the \Theta^(1) set. I would like this to be clarified.
- 313 - ??
- In many places I was not sure if \nu = 1

The introduction seemed like a lot of concerts which were not properly defined and I was unable to benefit from it. I could only understand it after reading later sections. 

Novelty
-------------
I am unfortunately not familiar with the related literature that authors mention. I cannot comment on the novelty of this approach, but it seems to me that on some level this problem should be addressed by something from prior literature, at least in the non-adaptive case. The fact no non-trivial baseline algorithms are provided seems a little suspicious (see below) to me but I leave other authors familiar with related works to comment on this. 

Execution and Questions
----------------------------------
- Triggering the algorithm too early is also not desired, yet it's not captured in the delay metric. Would it not be more meaningful to induce this in the metric somehow? 
From Theorem 2 it seems to me that the early detection is controlled via the parameter “m” which defines beta = O(m). Also, it seems to me that there is no clear cut choice of m since the true \nu is not known? How do you address the choice of m in practice?
-  Theorem 2 assumes \nu = 1? Or is this just for the sketch proof?

-  Similarly is lower bound for \nu = 1 or more general?

- In general I am not sure what is the influence of larger \nu in the context of the theory. It seems to me that the larger the \nu the more difficult is it for the likelihood test to finish since the longer time it will take the likelihood estimator to saturate or am I wrong? Could you comment on this, I am not familiar with general changepoint literature which may address this?

 - I actually liked the proof sketch intuition and the reduction for the largest drift identification. I do not know all the details and I am not familiar with queuing theory, but I followed in on a high level this makes sense to me.

- However it seems to me that Theorem 2 as stated does not reveal much about the problem (at least as stated in the main text). I think it makes sense to provide a specialized result (for linear bandits) and context. I found little value in the actual statement of Theorem 2 since the second term was left very vague. 

-Addition of a baseline: What about D-experimental design for the linear case as baseline; or a specific form of G-experimental design where one would take a supremum over all possible responses in \Theta^(1)-\theta_0. This is a better baseline than just uniformly at random, and yet seems to be an obvious choice. If the main point is adaptivity I am sure this baseline will be worse sometimes, but might be more difficult to beat in practice due to the simple nature of the algorithm.

-The approach seems to me to be rather suboptimal at the cost of being theoretically analyzable. Namely the differentiation of Q^(1) and Q^(2) seems to me to be purely theoretical such that one can split the proof technique for t_1 and t_2. First under Q^(1) one finds once the true optimal parameter is identified and then with Q^(2) this concentrates. It seems to me that one could have only one sequence and adapt the GLR test to account for it. It seems that you also use a variant of stopping rule that is not as in the algorithm but you do not describe in detail in the main paper. Can you comment on these aspects?

-Not a humble opinion that should be taken with a grain of salt: While I might be underestimating the true difficulty of the problem since I am not familiar with all the mathematical tools. This strategy seems to be very simple. Even too simple given what we know at least for linear bandits.


Current Verdict
--------------------
While I like the applications and the problem statement is indeed very interesting, there are some definiciences and clarifications that need to be addressed before acceptance. 
 This is mainly theoretical work and the sociatal impact were not analyzed. 

The authors propose an adaptive algorithm for quickest change detection under bandit feedback. Their proposed epsilon greedy algorithm learns the most informative action, and is able to achieve minimal delay. The authors provide theoretical guarantees for their scheme, as well as an information theoretic lower bound to show optimality in the small false alarm regime. Experimental validation is provided on synthetic and real datasets.  1. The distribution pi is not discussed, yet seems to be of moderate relevance to the additive error terms. It is also not specified how pi is selected for the numerical experiments. Additional clarification on the impacts of pi would be helpful.
2. Some oddities with numerical results; Table 2, URS for size = 20 mean is 236 while standard deviation is 1. Any intuition or explanation for the generally lower std dev of epsilon-GCD  (and URS for diffuse actions) versus Oracle?
3. The condition in 269 seems to be brittle and potentially very large. For example, if one possible parameter was theta’ = theta*+w for small perturbation w, and a*(theta*) + w was another possible action (i.e., this new theta’ had a different most informative action than theta*), then it seems as though the delay would scale with how long it takes for \hat{\theta}\_t to settle to theta* (the quantity t_1), which is unnecessary. Is there a clear way to avoid this issue of an alternate theta’ with arbitrarily small \Delta_{\theta’}, and a different optimal action? I understand the overlap with the future work of extending to continuous parameter spaces. 
4. Discussion and future work should be in the main text, not in the appendix; with the extra page allotment, it wold be good if this could be brought to the main text.
5. Minor errata: It is stated that a* may not be unique in 136, however in 326 it is assumed to be. The exploration indicator E_l is not defined prior to its use in 193. In assumption 4, the log is missing in front of the log likelihood ratio in 245. Appendix: Line 764, URS corresponds to epsilon=1. Lines 313 and 350 have missing cross references. 

The authors clearly mention the limitations of this approach in their discussion section, most critically to more adaptive procedures, when \theta_0 is unknown, and extending their results to continuous parameter spaces. The motivation for the paper is natural, the approach is well thought out, and the lower bound is fundamental. There are no negative societal impacts in this theoretical paper that focuses on changepoint detection under bandit feedback.

This paper studies the problem of trying to detect a change in a multivariate time series when only a subset of the time series can be queried at a time. The authors consider the case where the initial parameter is known and we wish to minimize the time until we detect a potential change in it, whilst also making sure that there is a low probability of detecting a change when there is not one. The authors present an epsilon-greedy algorithm, which when exploring samples an action randomly and when exploiting selects the action that maximizes the KL divergence between the initial distribution and the distribution with the current maximum likelihood. This is a natural algorithm and the authors are able to prove theoretical guarantees on its performance when used with a likelihood ratio stopping criteria. By means of a lower bound, the authors also show that the proposed algorithm can be near optimal in some specific cases. Finally, they show that their algorithm outperforms a very naive baseline experimentally.  It was quite difficult to understand the problem which is being studied in this paper. In the introduction/abstract several terms were used without being defined which made it quite hard to follow. Some terms in the problem definition were also not rigorously defined (e.g. X_t was just defined as an observation, no details were given on which values this can take, what dimension it has etc.). It would also have been very helpful to see a concrete example of how one of the motivating problems can be written using this notation making specific the actions and observations. Is the general idea that the actions are just choosing a subset of the streams to observe (in which case the problem could be viewed as non-stationary combinatorial bandits with piecewise constant functions) or can the actions be a mixture of different streams? The fact that the learner knows the pre-change parameter seems like a very strong assumption but is not discussed/justified. Additionally, the aim is just to detect one change which seems quite restrictive. What happens when there is some error in our knowledge of the pre-change parameter?

I am a bit concerned about the limited comparison to the prior work. The problem studied in this paper seems very similar to that studied in [ZM20]. This is discussed a bit in the related work section, but the bounds obtained in this paper are not compared to those in [ZM20]. Additionally, there is no experimental comparison to this work. It seems that the results presented here are concerned with being able to stop early in the event of no change, as well as minimizing the delay in detecting a change, whereas prior work just concentrates on the detection delay. It should therefore be possible to compare the two approaches both theoretically and experimentally. It would also be helpful to include the experimental results for the actual algorithm that was theoretically analysed, not just the modification that the authors found worked better in practice.

The lower bound in Theorem 1 is nice, and it is pleasing that it shows a sort of trade-off between a false alarm rate and detection delay.

The algorithm that is presented seems to be a natural application of epsilon greedy. From reading the pseudo-code in Algorithm 1 it is possible to get a good understanding of what the algorithm is doing. I did not find the text in section 4 particularly helpful for understanding the algorithm. I also did not follow intuitively why the proposed algorithm would adapt to the no change setting. I am curious about the stopping threshold beta. In Theorem 2, it is stated that this should be tuned to be a linear function of the size of the parameter space. When the parameter space is large, this means that the stopping criteria will only be satisfied when there is a huge difference in the likelihoods, and so it seems like the algorithm will be very slow to stop (I also think that this means that the adaptation to infinite parameter space may be more involved than the authors claim). When we are in the so-called ‘low false alarm’ setting, it seems that the delay detection time is dominated by the first term in Theorem 2, but I don’t have any intuition for what the result will be in the other case. In particular the discussion in lines 276 to 286 was very difficult to follow and a lot of stuff seems to be hidden in the appendix. In the ‘low-false alarm’ setting, the upper bound in theorem 2 scales logarithmically with the size of the parameter space. This does not appear in the lower bound so I disagree with the claim that the only difference is the 1/(1-epsilon) penalty for epsilon-greedy. In many natural settings with large parameter spaces, this log(beta) term can represent a significant difference between upper and lower bounds. The example in line 292 considers a somewhat trivial example where the parameter space is just a basis. Does the improvement still hold for a more realistic parameter space (e.g. |theta| = a^d for some a)? I also found the proof outline of Theorem 2 very difficult to understand.


Minor comments:
- The authors propose to use the ‘detection delay’ as the performance metric instead of the classical regret. This is fine, but it could be motivated better. From the introduction it is also unclear what this detection delay is actually measuring, more intuition should be given.
- What is ‘the most informative action’ as described? Is there some connection to a partial monitoring problem? 
- the formatting of Algorithm 1 is strange, there should be a line space below.
-  Is Q^(2) ever defined in Algorithm 1?
- Is N_tau(a) defined on line 178?
- I don’t see the relevance of the queuing in section 4 The limitations of this work could be made clearer. The societal impact is not discussed, but this is probably OK for this paper.

Meta-review: The paper studies the problem of change point detection in a multi-sensor setting, where the learner can read observations from a set of available sensors to monitor whether any change in observation distribution happened or not. While this problem is clearly connected with change point detection in the offline setting and to bandit in non-stationary environments, it has an interest on its own and it poses very specific challenges that need to be carefully analyzed, starting from an accurate definition of the performance measure. The proposed algorithm builds on solid intuitions and some known techniques (such as CUMSUM) and it is shown to achieve near-optimal performance in the limit where false rate tends to zero. I think these contributions are valuable and they represent a very interesting step into understanding this problem. For this reason, I'm proposing acceptance.

Nonetheless, the paper generated a considerable amount of discussion and the authors provided very much needed clarification on some points including the interpretation of the theoretical results and the gap between upper and lower bound. In particular, the latter point shows an existing gap depending on the dimensionality d between the upper and lower bound. While it is perfectly fine to have such a gap, the authors should clearly state in the revised this aspect so that other people can understand where the limitations of the current work lie. Overall, the paper would greatly benefit from integrating the additional insights provided during the rebuttal into the final version of the paper.



Decision: accept