Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper designed a hypothesis testing procedure for detecting changes in episode sequential data.  For online operation, it also proposed a novel Bootstrap mechanism for False alarm rate control. The method is demonstrated based on a non-iid and non-Gaussian setting for reward signals.

In all the method is theoretically sound and seems useful for detecting changes, however there is no experiments provided to show that the method can help improving the performance in RL tasks. The strength of this paper would be significantly boosted if the proposed method can be used to solve an non-stationary RL problem.

Moreover, it is a little unclear the specific setting of this work. Is the underlying sequential decision making problem based on Markov decision processes(MDPs)?
There is no surprise that the RL feedback from an environment is highly correlated over consecutive time-step if the underlying problem is an MDP. It seems that the paper mainly deal with per-step reward without considering state and action information when detecting changes. 
Some related work that also applies hypothesis testing procedure for non-stationary MDPs might worth mentioning. E.g,

Banerjee et al. Quickest change detection approach to optimal control in Markov decision processes with model changes

“In setup 2, … we consider a fixed trained agent … , “ It is unclear if the trained agent exhibit an optimal behavior or arbitrary behavior(sub-optimal)? 

In figure 2, does each curve correspond to a fixed policy?

=========after rebuttal===========

I appreciate the authors’ effort to address my questions. I still think this paper is below my expectation especially if it is put into the context of RL.  I would expect to see how this method can solve or help solving a fundamental problem (e.g., reducing sample complexity) in RL or be applied to a novel application (e.g., nonstationary RL tasks). Otherwise I didn’t see why it is necessary to compare this method with other statistical testing methods in the context of RL, if they can be easily made in other non-iid settings without mentioning RL. 

review_title: Interesting and solid idea, but lacks demonstration of its use 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a sequential testing procedure for a fixed RL agent reward drift. 

Major issues:

1. The authors state in some places of the text that their method can be applied in statistic domain other than RL. However, for me, the entire proposed methodology does not relate to RL and is more general: it can be applied to any sequential statistics (series of random variables) that have a proper [auto]correlation structure (just see how the main statements). Due to this, I have concerns in the way how these proposed approaches are presented: I believe that a paper requires a notable refactoring to improve the presentation.


2. Seem that the contribution is not enough for the current venue.  All the proposed statements (Theorem 4.1, Theorem 4.2) looks very straightforward. Possibly, there untrivial tricks in their proofs, but the current proofs (proof sketch) does not reflect them at all (see). I expect a clearer presentation in the main text why these results are non-trivial and non-incremental. As for now, the provided theoretical grounds looks as non-enough for publication @ ICLR.



Other issues:
Paper cites the paper
 [Vineet Abhishek and Shie Mannor. A nonparametric sequential test for online randomized experiments. Proceedings of the 26th International Conference on World Wide Web Companion, pp.
610–6, 2017] 
where seq. testing is applied to A/B tests, but there is an earlier paper on this approach:
 [Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. Sequential testing for early stopping of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’15, pages 473–482, New York, NY, USA, 2015. ACM.]

================
After the author response, I raise my score by 1 (see my comment to them)
review_title: Seemingly incremental paper with results that may have an extended application domain

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a novel change detection test for non i.i.d. data motivated by applications in RL. At first, they provide an offline version of the test, then they extend it to the online setting.

The paper is clearly written and presents both theoretical results and convincing experimental results. My two concerns are about the novelty of what has been proposed w.r.t. standard CDT procedures and on the fact that a consistent part of the material of what has been proposed in the paper is deferred to the appendix.

I would like to have more discussion on the difference between what has been proposed here and the standard multivariate CDTs, for instance:
Kuncheva, Ludmila I. "Change detection in streaming multivariate data using likelihood detectors." IEEE transactions on knowledge and data engineering 25.5 (2011): 1175-1180.
Boracchi, Giacomo, et al. "Quanttree: histograms for change detection in multivariate data streams." International Conference on Machine Learning. 2018.
A strong motivation of the novelty w.r.t. to the literature might make me increaese the paper score.

In my opinion, the paper is not self-contained. Not even the main theorem proof are included (the sketches are not useful in understanding the proof line) and the experimental setting is described in details only in the additional material. I think you should rearrange some of the material from the appendix to the main paper and viceversa.

I would have appreciated a more detailed description of Algorithm 1. In this version of the paper it is difficult to understand the procedure you proposed, if you do not refer to the appendix for details.

In your setting the change in the episodic reward is only about the expected values. What happens if the new reward distribution changes in terms of covariance \Sigma?

Minor:
"in RL ... life-time of the task." I would have preferred a citation about this statement. Showing evidence using your experiment is a bit premature at this stage of the presentation.
assume strong assumptions -> require strong assumptions

--------------------------------------------------------------------------------------------------
After rebuttals the authors significantly improved the presented work, including and discussing some relevant work which was previously missing. 

review_title: Interesting work, it is not clear to me how much it is novel w.r.t. standard multivariate CDT.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the drift detection for episodic data, where data episodes are assumed to be i.i.d. but data within each episodic can be correlated. It is assumed that the pre-change (nominal) mean and covariance of each episodic is perfectly known or can be accurately estimated from reference data. The Uniform Degradation Test (UDT) and Partial Degradation Test (PDT) are proposed to detect the mean shift. Moreover, this paper uses bootstrap to control the false alarm rate by setting the threshold as empirical quantiles of the detection statistic computed from reference data. 

Overall, this paper is well written and well-structured. The most impressive part is that they include comprehensive details, covering almost every related aspect. And this paper has tried to make the problem as general as possible, such as considering non-Gaussian distributions, etc.

However, the technical contribution is very incremental. And the problem itself, although is proposed in the reinforcement learning setting, is of no fundamental difference with the classical mean shift in change-point detection literature, for example, change-point detection for a mean shift in multivariate Gaussian distributions. And I didn't see the classical Hotelling T^2 test mentioned in this paper, which is a classical and also widely-used method to detect the change in mean/covariance, and it also utilizes the pre-change covariance matrix in the detection statistic. Moreover, the proposed Uniform Degradation Test methods are a direct consequence of the likelihood ratio test; the false alarm control by bootstrap is also widely used in change-point detection literature for problems where the theoretical characterization of false alarm rate (or average run length) is difficult to obtain. 

Looking forward, I think this paper may be improved by expanding the horizon of the problem set-up and possibly leading to new theoretical findings. 

================ After the author response, I raise my score by 1 ================
Thanks to the authors for the detailed response and extended discussion on theoretical results and experiments, and I have raised my score by 1. 
review_title: Comments to "Drift Detection in Episodic Data"

Meta-review: The paper's initial evaluation was below par, but the author feedback helped clarify several crucial points after which two of the reviewers increased their scores by a point, bringing the current evaluation to borderline. 

The paper addresses a relevant and challenging problem in the RL domain. However, in my opinion, from the reviewers' and authors' remarks and from my own reading of the paper, there are concerns that need to be addressed before the paper can be publication worthy. Primary among these is the quantum of novelty -- as many reviews point out, the key idea of viewing an episodic trajectory as a multivariate (vector) sample for running hypothesis tests is not novel in itself, as is the claim that new tests have been devised. Another crucial issue is the (parametric) assumption of normality for the episodic reward sequence which is not adequately justified in the paper -- even a two time-step trajectory with normal rewards per state transition can exhibit a mixture-of-Gaussians type reward distribution for the second state, breaking the assumption. As it transpired from the reviews of Reviewer4, reducing environment shift/degradation to just a mean change problem, without even considering a change in the variances (2nd order statistics), seems to be too stylized to be effective. There are other, nonparametric approaches in statistics based on testing for changes in the distribution function (kernel density estimation approaches, for instance), which could perhaps be applied without normality assumptions and yield favourable results. The experimental results for detection delay often show significant overlaps of the delay distributions for different procedures (e.g., Hotelling vs. Mean vs. UDT etc.), which does not indicate an advantage of the proposed method. 

I would urge the author(s) to assimilate the feedback and delve deeper as to why and how parametric procedures based on normality assumptions may or may not succeed, so as to significantly strengthen the theoretical and practical message of this work.  
Decision: Reject