Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes novel rules for false discovery rate control (FDRC) for online anomaly detection in time series 
main_review: This paper is slightly out of my area, so this will be a low confidence review. In particular, I cannot strongly comment on the novelty.

Having said that, this is clearly the best paper in my bunch (of seven). Well motivated, very well written and possessing deep insights in statistics.  The paper clearly identifies its limitations 

The paper ends with “We hope to be able to follow up on this work once higher-quality public datasets become available.” I would love to see this (long journal) paper, I think it would be a very useful resource for the community. However, within the page limits available this current work makes a strong contribution. 

limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses techniques by which some limitations of current online false discovery rate control algorithms can be overcome. Importantly, the sequential decision thresholds from current algorithms decay to zero in the absence of recent rejections of the null hypothesis -- a situation called alpha-death. A technique to overcome this is proposed.

================
Update: The scope of contribution has been clarified in the rebuttal. The authors address the lack of real-world experiments somewhat, but more is desired to improve the strength of the paper.
main_review: 1. It was a bit hard to figure out the main novel contribution in this paper. It seems that the equation after line 216 (smoothed FDR) is the claimed contribution, but since the equation is not numbered, it does not come out clearly. Moreover, most equations on page 8 are also unnumbered and perhaps part of the contribution of this paper. All these are minor variations over existing techniques.

2. The experiments section (line 275) states that the paper does not aim to show that the technique results in yielding higher anomaly detection accuracy than alternatives. This is a serious drawback and coupled with the fact that no real-world datasets have been used perhaps the paper in its current state is only fit for a workshop rather than a full conference.
limitations_and_societal_impact: Limitations have been discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This research seeks to control the false-discovery rate, while simultaneously avoiding alpha-death, where the alpha values are shrunk to near zero for multiple-hypothesis testing with extremely rare events. The authors propose a method based on memory-decay that is extensible to other algorithms of the alpha-investing class.
main_review: The paper was well structured in setting up the problem, detailing the previous work and limitations, providing a solution and an extension, and providing experimental results. The exeprimental results could be fleshed out more, although I have trouble deciding from where the extra room would have come, besides perhaps the "False discovery rate control" section or removing some of the formulations in the "Oracle rules" section. 

Overall, I believe their extensible decay contributions to be useful to the field. I believe this research provides ample starting point for others, or the authors themselves, to explore.
limitations_and_societal_impact: The authors did an adequate job of discussing the limitations of their research.There were no obvious potential negative societal impacts that I believe the authors missed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper address online false discovery rate (FDR $\simeq 1-\mathrm{Precision}$) control in anomaly detection, where we obtain observations sequentially and immediately decide whether we classify them into anomaly or not.
The authors adopted smoothed FDR (sFDR) for criteria and then derived rejection thresholds that offer high detection power (Recall) for anomalies without shrinking its detection threshold to zero while maintaining target sFDR. It is made possible by the fact that FDR for no rejection is defined by the explicit parameter $\eta$ in the sFDR.
They also proposed a way to handle local dependency in time-series with a similar procedure.
Experimental results on synthetic datasets demonstrate that the proposed method performs better than existing methods, especially when an anomaly is very rare.
main_review: ### Originality:
* This work proposed novel rejection thresholds, $\alpha_t$s, in Eqs. (8) and (9) for anomaly detection with a capability of online FDR control. It is derived from well-studied sFDR [18].

### Quality:
* Related work is well-written and would be informative for many readers.

* Experimental evaluation is weak.
    - It is only on synthetic datasets.
    - They even did not conduct experiments on time-series or sequential data.
    - It seems that in cases where anomalies are rare, the system worked well. However, in cases where anomalies are more common, the performance deteriorates. I would still like to see how the proposed method works in real data, especially in more realistic scenario where we consider decay and local dependency.
    - The authors stated that "The appendix contains further experiments taking local dependency into account." in l.288, but there would be no such experiment in the appendix. I guess that they had tried to add experiments after submission but they could not. It could be unfair.

### Clarity:
* They did not clearly state the contributions of this paper. It would be better to state them.

* It would be better to cite conference/journal not arXiv, such as [23] is published in Neurips2019 and [32] is published in AISTATS2020.

* In l.142, what is "reward" in this context?

* In l.267, sFDR would be mFDR?

* Figure 2 is not mentioned in the main text. Also, in general, it is better to locate figures on top.

### Significance:
* The proposed method could be practically imoportant since it offers high recall for anomalies while maintaining target precision. However, the significance on real data is not clear because they did not provide experiments on real datasets.
limitations_and_societal_impact: They addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: After the rebuttal phase, the paper now has only positive reviews, and as such could 
be accepted. The main issues found during the review phase are the somewhat unclear novelty
and the insufficient experiments. After carefully reading the paper again, I need to add 
another concern: the independence assumption of the p-values. While the authors try to 
address this issue in Section 5, their approach is somewhat limited:

- It is assumed that sufficiently far away p-values are independent, this is measured by the 
  distance L in time, see (10). However, it remains unclear whether and when this assumption is satisfied,
  and the authors do not discuss this issue.
  
- The only proposed method working for (10) needs to know L, which again sounds rather unrealistic.

In summary, despite having a good average rating and a positive rebuttal phase, I personally see 
this as a true borderline case.
Decision: Accept (Poster)