Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a plug-and-play method for robust estimation of perturbed data in time series identification and prediction. The proposed method assumes perturbations in a part of time series data and performs estimation by aggregating multiple masked and imputed time series data outputs. Experimental results with multiple data sets and multiple machine learning models are reported.
strength_and_weaknesses: Strengths
1. Extensive experimental results are reported using multiple methods such as MLP, RNN, CNN, Transformer, and MLP-Mixer on multiple datasets of time series data.

Weaknesses
1. The authors should proofread the manuscript repeatedly. There are examples found in Section 1.
  - ell_0-robustness -> \ell_0-robustness
  - guaranteeing the robustness for each prediction mathematically. -> The first letter of the word at the beginning of a sentence should be capitalized.
  - BDPA should be first introduced by its correct name, not the abbreviation.
  - The authors should break paragraphs before describing the first contribution point.
1. Mathematical explanations are also hard to follow.
  - In equation (1), what does it mean that the delta index decreases as the x index increases?
  - Most critical point is that the definition of the mask in equation (2) is unclear; the reader is ultimately forced to imagine it. Surprisingly, there are three different capital M's in equation (2) and Algorithm 1: Roman, Italic, and Calligraphic Ms. As for Roman's M and Calligraphy's M, their definitions cannot be understood. Since the purpose is to mask the time-series data, the reviewer can only imagine that Ms are series data, with some being 1 and others being 0; the indexes for the value 0 are not stated clearly. Some figures showing mask patterns would be helpful.
1. (Yoon et al., 2022) is cited but not compared to the proposed method in the experimental results.
1. The second paragraph of the introduction makes the point about heuristic defense methods, arguing that a certified defense is needed to end the cat-and-mouse game. However, the algorithm in this paper also assumes that only a local part of the time series data is perturbated. Would the cat-and-mouse game continue even with the proposed method when other types of attacks are considered?
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality
- As mentioned in the first and second weaknesses, several drawbacks exist.
- This is just a question, but even if a mask hides the perturbed area, as this paper claims, it seems to make little sense since it is aggregated with the results from other masked time series where the perturbed area remains. How do the authors think the assumption makes sense?

Novelty
- Masking some parts of data is well-known to the community for self-supervised learning and data augmentation.

Reproducibility
- Sufficient reproducibility is expected. The protocol description of the experiment is detailed, and the source code is provided.

summary_of_the_review: Overall, the reviewer leans toward rejecting this paper. A response from the authors regarding the above weaknesses could improve the score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for detecting $l_{0}$ norm perturbation in time series. The key idea is to do multiple masking over the time series by sliding windows, impute the masked areas individually, and measure the disagreement of the predictions using the imputed time series. A disagreement is an indicator of perturbation. The method is a flexible framework and can be applied to different base predictive models.
strength_and_weaknesses: Strength
1. The paper studies a new problem and introduces a three-step approach for detecting $l_{0}$ norm perturbation in time series, which is easy to apply to different time series predictive models.
2. The paper provides some theoretical analysis about the robustness of the proposed method, and comparison with randomized smoothing defenses.
3. The experimental results demonstrate the effectiveness of the proposed method by comparing with a couple of baseline methods.

Weakness
1. The practical significance to detect $l_{0}$ norm perturbation in time series is obscure. The paper could provide some examples to illustrate why detect $l_{0}$ norm perturbation in time series is practically useful, compared to $l_{2}$ norm perturbation.
2. The presentation of the technical development is hard to follow. Many equations are given without sufficient description on their intuition. For example, Eq. 1, it is unclear why the norm $L_{adv}$ is used as the time point index and what is the relationship between $\delta$ and $\delta_{t}$. Eq. 4, it is unclear on why should all M predictions be consistent instead of allowing some tolerance. Some theoretical analysis is desired. Similarly, Eq 5, 7, etc., are presented without introducing intuition.
3. Below Eq 1, the authors claim in most cases short-term events should not have large impacts on the long-term outcomes. Then the question is why resistance to the short-term perturbation is essential.
4. The proposed method requires the masks to cover arbitrary temporally-localized perturbation of $L_{adv}$. It is better to provide some guidelines on how to ensure this requirement is satisfied.
5. In proposition 1, to conclude the forecast/label cannot be changed, it is better to quantify the change and define what is considered unchanged quantitatively.
6. In Remark 3, it is better to also describe the impact of certificate only for $f_{MIA}$ instead of $f$.
7. It is good to see the comparison to randomized smoothing defenses. It is better to further justify on why this comparison is important.
8. Is there any discussion on whether the proposed method useful for both univariate and multivariate time series?

clarity,_quality,_novelty_and_reproducibility: See strength and weakness.
summary_of_the_review: The paper presents a method for dealing with a new problem on detecting $l_{0}$ norm perturbation in time series. The method is a flexible framework to different forecasting/classification methods. The practical significance of the studied problem requires some justification. The presentation of the method design should be improved for clarity. The current paper is not easy to evaluate the validity of the proposed method. See details in strength and weakness.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Masking Imputing Aggregation (MIA), a framework based on masking and aggregation, to achieve certified robustness for time series forecasting models against temporally localized $\ell_0$-norm-bounded perturbations. For the masked time steps, an imputation model is trained to fill in and then new time series is fed into the old model for prediction. When consensus is achieved, certified robustness of the pipeline is achieved. Experiments on Exchange Rate, Traffic, UCI Electricity, DistalPhalanxTW, MiddlePhalanxTW, ProximalPhalanxTW against two baselines extended from previous work (Derandomized Smoothing and Randomized Ablation) demonstrate the effectiveness.
strength_and_weaknesses: Strengths:
- A practical and rigorous certification framework with significantly better performance than baselines.
- The framework is novel since no prior work achieves certified time series prediction against $\ell_0$ predictions and no prior work can protect base models without retraining in this domain.

Weaknesses:
- The idea is relatively straightforward: it is like an application of [1,2] in the time series domain. The unique point of imputation model is spiritually similar to [3].

- The writing quality needs further improvements. For example, many typos exist even on the first page:
  1. First para: $ell_0$-robustness -> $\ell_0$-robustness
  2. Second para: "Athalye et al. (Athalye et al., 2018)" repeated - you can use `\citet{}`
  3. Second para: "the concept of certified defense is proposed. guaranteeing the robustness for each prediction mathematically." -> additional "."

In later sections:

4. Eqn. (1): the first line $x_{1:t_0} + \delta, \text{subject to} \|\delta\|_0 \le L\_{adv}$ does not contain the temporal localization constraint, so it is not equal to the second line.

5. Eqn. (3): $G(\mathbf{x}^{(m)}\_{1:t_0})$ -> $G(\mathbf{\hat x}^{(m)}\_{1:t_0})$

6. Remark 1, last sentence: the inference cost could not be 0. It should at least be 1.

7. What is the $\sigma$ value used in practice in Eqn. (7)? It is not specified in later sections. Also, why Gaussian augmentation is needed here? Unlike other randomized smoothing work, here no Gaussian augmentation is used in the inference time, so the use of Gaussian augmentation in the training time may induce domain mismatch and needs further justifications.

8. Eqn. (9): I think the guarantee of RHS should be $2 * (k+L_{adv}+1)$ instead of $k + L_{adv}+1$, since both terms of LHS can change by 1 for each step perturbation leading to gap reduced by 2.

9. Section 5, first paragraph: MiddlePhalanxTW ProximalPhalanx -> MiddlePhalanxTW, ProximalPhalanxTW

[1] PatchGuard: A Provably Robust Defense against Adversarial Patches via Small Receptive Fields and Masking. USENIX Security 21

[2] PatchCleanser: Certifiably robust defense against adversarial patches for any image classifier. In 31st USENIX Security Symposium (USENIX Security 22)

[3] Denoised Smoothing: A Provable Defense for Pretrained Classifiers. NeurIPS 2020
clarity,_quality,_novelty_and_reproducibility: The writing is generally clear but not of high quality IMHO. The novelty may be a bit limited from the technical perspective but should be sufficient from the application perspective. The reproducibility is great given the open-source code.
summary_of_the_review: Due to the limited technical novelty and rough writing, the submission may be not ready for publication at the current stage. However, I appreciate the novel application of masking and imputation framework in the certified robustness of the time series domain.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors develop a framework for detecting and certifying robustness to perturbations to time series inputs that are localized in time with an l0 norm constraint. The motivation for the thread model comes from subsequence anomaly detection, a notion that has been previously studied in the time series literature.

Strengths:
1. New framework for robustness certification for l0 localized perturbations to time series models.
2. Empirical results demonstrating effectiveness of the framework.

Weaknesses:
1. The contributions relative to prior work on the theory of randomized smoothing is minimal.
2. The application to subsequence anomaly detection seems highly specific and may not be appropriate for the ICLR conference. 

justification_for_why_not_higher_score: Application is narrow and theoretical results are not significant.
Decision: Reject