RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies
Authors: Hao Cheng
Qingsong Wen
Yang Liu
Liang Sun
Venue: ICLR 2024 poster (2024)
Keywords: Robust time series forecasting; learning with noisy labels
Primary area: general machine learning (i.e., none of the above)
TL;DR: An simple and efficient algorithm to deal with anomalies in time series data

Abstract: Time series forecasting is an important and forefront task whose techniques have been applied to electricity forecasting, trajectory prediction, labor planning, etc. However, most of time series forecasting techniques assume that the training data is clean without anomalies. This assumption is unrealistic since the collected time series data can be contaminated in practice. The forecasting model will be inferior if it is directly trained by time series with anomalies. Thus it is essential to develop methods to automatically learn a robust forecasting model from the contaminated data. In this paper, we first statistically define three types of anomalies, then theoretically and experimentally analyze the loss robustness and sample robustness when these anomalies exist. Based on our analyses, we propose a simple and efficient algorithm to learn a robust forecasting model. Extensive experiments show that our method is highly robust and outperforms all existing approaches. The code is available at https://github.com/haochenglouis/RobustTSF.

Published as a conference paper at ICLR 2024
ROBUSTTSF: TOWARDS THEORY AND DESIGN OF Ro- BUST TIME SERIES FORECASTING WITH ANOMALIES
Hao Cheng', Qingsong Wen Yang Liu', and Liang Sun?
University of California, Santa Cruz, AAlibaaa Group (haocheng, yangliuj@ucs edu, gingsongedu@@mai1. com, liang. Suunaalll wwwllllbaaaaiiinnc.mm com
ABSTRACT
Time series forecasting is important and forefront task in many real-world ap- plications. However, most of time series forecasting techniques assume that the training data is clean without anomalies. This assumption unrealistic since the col- lected time series data can be contaminated in practice. The forecasting model will be inferior if directly trained by time series with anomalies. Thus is essential to develop methods to automatically learn robust forecasting model from the con- taminated data. In this paper, we first statistically define three types of anomalies, then theoretically and experimentally analyze the loxs robusiness and sample robust- ness when these anomalies exist. Based on our analyses, we propose a simple and efficient algorithm to learn robust forecasting model. Extensive experiments show that our method is highly robust and outperforms all existing approaches. The code is available httpp::: / httttpp:///ituub..oo/ com httpa//w.w....cccccccooooooo//// hatpaanggoooooo//ouosossst
INTRODUCTION
As the development of Internet of Things, the monitoring of cloud computing, and the increase of connected sensors, the applications of time series data increase significantly (Lai et al. [021 Wen al., [022] Lim & Zohren| 2021). However, many time series data collected are contaminated by noises and anomalies. In time series forecasting with anomalies (TSFA), we study how to make robust and accurate forecasting on contaminated time series data. The earliest work of TSFA can date back to (Connor et al. 1994), where the classic
pipeline is proposed. In the pipeline, a forecasting model first trained using the data with outliers and noises. Then the trained model is used to predict values for each time step (detection). If the predicted value is far from the observed value, we will regard this time step as anomaly time step and then use the predicted value to replace the observed value this time step (imputation). After imputation, the imputed (filtered) data are utilized to retrain a new forecasting model. The innrttiinimmuaattionneeee work with any forecasting algorithm and improves performance in many TSFA tasks. However, it is very sensitive to the training threshold and anomaly types, Note that TSFA is also related to time series anomaly detection (TSAD) (Blázquez- García et al.) 2021), especially forecasting model learned to compute the reconstruction error, which is then used to determine anomaly.
pipeline
"TSFA shares similarities with learning with noisy labels (LNL) (e.g., see (Natarajan et al. 2013 Song et al. 2022),, but their settings differ significantly. LNL typically involves datasets with clean input samples X and noisy labels Y, such mislabeled images in image classification tasks. In time series forecasting, we typically use past history, sometimes aided by external factors, to predict future targets. In this context, TSFA faces the challenge of handling anomalies in both covariates and targets simultaneously. This characteristic makes applying LNL methods directly to TSFA challenging. Moreover, the diverse nature of time series data in real-world scenarios leads to various types of anomalies. Existing algorithms struggle handle these diverse anomalies effectively. For instance, (Li et al., (022) addresses a similar problem to TSFA, emphasizing anomaly detection and drawing from LNL ideas. However, the small loss criterion in LNL does not universally apply common anomaly types in TSFA, resulting in inconsistent improvements across TSFA anomaly types."
denote all covariates X and the target we predicting Y. Notably,
"Correponniig author

---

Published as a conference paper at ICLR 2024
In this paper, we introduce unified framework for TSFA, bridging the gap between TSFA and LNL. We address various anomaly types commonly found in real-world applications. Within this framework, we present RobustTSF, an efficient algorithm that outperforms state-of-the-ee methods in both single-step and multi-step forecasting. In contrast to the intiactioniimpuattio--eeree RobustTSSF identifies informative samples by assessing the variance between the original input time series and its trend. Subsequently, it employs robust loss function to improve the forecasting process. This distinctive approach substantiated by our analyses comparing TSFA and LNL. Contributions can be summarized as follows:
pipeline,
We analyze the impact of three formally defined anomalies on model performance, exploring loss robustness with target anomalies and sample robustness with covariate anomalies.
We propose RobustTSF,
efficient algorithm for TSFA, which is both theoretically grounded and empirically superior to existing TSFA approaches.
We conduct extensive experiments, including single-step and multi-step time series forecasting with different model structures, to validate our method's performance.
To the best of our knowledge, this is the first study to extend the theory of LNL to time series forecasting, which builds bridge between LNL. and TSFA tasks.
2 RELATED WORKS
Robust Time Series Forecasting: There is line of works learn robust forecasting model in different data bias settings: (Yoon et al.) 2022) proposes to make forecasting model robust to input perturbations, motivated by adversarial examples classification (Goodfellow et al. 2015). (Arik) et al. 2022: Woo et al. 2022; Wang et al. 202 propose to make the forecasting model robust to non-stationary time series under concept drift or distribution shift assumption (Ditzler al. 2015 Lu et al. 2018). Different from these settings, our paper considers learning a robust time series forecasting model under the existence of anomalies (TSFA). Some existing approaches (Connor et al.)
1994] SScheke-Schneide et al.) 2020 Li et al. 2022) deploy detection detection-imputationn canpputtttio-ettrrnnnnn pipeline, or use small loss criterion to TSFA tasks which lack a deep understanding of how these anomalies affect model performance which makes their methods less explainable.
Robust Regression Our setting is also similar to robust regression (Pensia et al. 2020 Kong et al., 2022), which has some rigorous guarantees when the input and the label have anomalies. However, the theoretical guarantees are based on (generalized) linear regression. The alternating minimization (Pensia et al. 2020) also needs to calculate the inverse of X" X'X For time series forecasting with deep neural networks (DNNs), it is not practical to calculate the inverse of the source data since the model of DNNs is not the simple linear layer. Thus it is hard to generalize the algorithm and analyses in robust linear regression into robust time series forecasting with DNNs.
The goal of LNL is to mitigate the noisy label effect in classification and learn a robust model from the noisy dataset (Natarajan et al. 2013 Liu 2021). The problem of LNL has been researched for long time in the machine learning community, and many methods have been proposed in recent years, such as sample selection (Jiag et al. 2018 Han et al. 2018). robust loss design (Zhang & Sabuncu 2018 Liu & Guol 2020,[Cheng et al. a021ap b Zhu et al. 2021a), transition matrix estimation (Patrini et al. 2017 /huu et 20) etc. Among all these methods, arguably the most efficient treatment noise transition matrix estimation always involve training multiple networks or need
Learning with Noisy Labels:
adopt robust losses, since sample selection and
training. We say a loss is noise-robust in LNL if it satisfies the following equation (Ghosh et al.
2017
Xu et al. 2019 Ma et al. 2020 Liu & Guo 2020):
arg min FEJ (x. yx ((((()) = arg min feF T. TX. (e((x))
(1)
where f the classifier, X is the input, Y and Y are clean labels and noisy labels, respectively. Equation (I) implies that minimizing E over clean dataset with respect to is equivalent to minimizing E over a noisy dataset, demonstrating f's robustness to label noise. However, applying robust losses from LNL directly to TSFA faces two challenges: 1) TSFA deals with regression problems rather than classification problems where label noise is typically modeled. The effectiveness of robust losses from LNL in TSFA is uncertain without proper anomaly modeling. 2) In LNL, noise is limited to Y, while TSFA involves anomalies in both X and Y. We address these issues by analyzing loss and sample robustness in Sections 4 and of this paper.
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARY AND FORMULATION
27) be time series (without anomalies) of length T. Assume that the time series is partitioned into set of time windows with size (1 < < T), where each window is a sequence of predictors 2n = (Zn, a). This partition gives the training set D
Problem Formulation: Let (2, 22,
-n-a-1) and its corresponding
label is Ya.. an+a. where E {1. 2... {(x1: Vee ). (IN, N/wN) }} where N = T a, which can be assumed to be drawn according to an unknown distribution, D, over X' y. In real-world applications, time series may have anomalies (anomaly types are defined later), and we can only observe time series with anomaly signals. The observed training set is denoted by D ((ã1, Mn, ).... (ØN, Bw }}, where" denotes the observed value. The robust forecasting task aims to identify a model f that maps X to Y accurately using the observed training set with anomalies.
Anomaly types: In the paper, we mainly consider point-wise anomalies (we also provide preliminary studies for subsequence anomaly in Section 7.3). y. Define the noise (anomaly) rate as where < 17 The observed value ž in the time series can be represented as:
with probability 1 1-7)
with probability
where is the ground-true value and zt is the value of anomaly. We consider three types of of Constant type anomaly: zf = 2t + e, e is constant.
Missing type anomaly: 2 e, constant.
Gaussian type anomaly: =A = 2e + e, (0,
The N denotes Gaussian distribution, and the is noise scale. Note that c in Constant type anomaly and o² in Gaussian type anomaly should be large that they can represent anomalies instead of small random noise (e.g., white noise). Generally, the noise scale is within the range of (-oc, +oo).
Constant anomalies exhibit fixed difference between the anomaly value and the ground-truth value, Missing anomalies are independent of the ground-truth value, and Gaussian anomalies follow a Gaussian distribution. While (Connor et al., 1994) used Constant and Gaussian anomaly types for experiments without explicit definitions, they effectively simulate real-world anomalies, For instance, Missing anomalies represent missing sensor data (e 0), Constant and Gaussian anomalies model sensor functionality affected by external factors like temperature or humidity. We provide theoretical and experimental analyses of these anomaly types in this paper, with illustrations in Appendix [
Loss ROBUSTNESS: ANOMALY EFFECT IN Y
Understanding how anomalies affect model performance is hard when anomalies exist in both X and Y. Thus we first assume the input is clean and anomalies only exist in Y. In this case, D = ( ) Næ Day) } (we can assume these data are non-overlapping segments of the time series). Va., can be represented as:
Ven
with probability - 17
Ya.
with probability 17
Thus this setting is similar to the vanilla LNL setting except that for time series forecasting, the problem is regression instead of classification. Next, we examine the robustness of loss functions for different anomaly types defined earlier.
Theorem 1. Let l be the loss function and be the forecasting model. Under Constant and Missing rype anomalies with anomaly rate < 0.5, if for each xc, ((f(x) Bn) ) + ((f())) v = Ca, where
Oz is constant respect to the choice of f. Then we have:
Ex((f(() Y)] - Enl(((((( Y))
(2)
where > O and 72 are constants respect to f.
Our proposed Theorem suggests that a loss function satisfies {(f(x), yz) + ((f()), vz) Ca. then it is robust to constant and missing type anomalies. Equation (2) implies that minimizing & under D with respect is identical minimizing & under D (Equation
3

---

Published as a conference paper at ICLR 2024
C33 Chean iraut lime pertes
U6 Dausssan anemaatiss (haer)
Icl Dlouordann srwmatee (erabeie)
Mosrosan antemaates charn)
20-
-3
-1.8 LAN
85
300
100
20
Figure I: Visualization of time series with Gaussian anomalies different positions. (a): Clean input time series which is the sine function of time steps. We normalize the time series to 0 mean and I std. (b) (c) (d): Time series (sine) with Gaussian anomalies located in front, middle and back of the input time series, respectively.
Proposition 1. From Theorem is not robust.
7 MAE is robust to Constant and Missing type anomalies while MSE
Proposition 2. Assume Co can be estimated correctly. Let & be any possible loss function operated on fx) and ya. Define knorm = c. Then foor is robust to Constant and Missing rype anomalies.
Proposition analyzes the robustness of common MAE and MSE loss functions. Interestingly, MAE has been found to be robust to symmetric label noise in classification as well (Ghosh et al.) 2017). However, the robustness conditions in (Ghosh et al. 2017) and our work differ. In their work, robustness requires C C, whereas our approach does not necessitate such a strong condition. | Jadresssss the scenario in which C2 remains constant with respect to the forecasting constant. In this situation, any loss function
While Theorem
model, Proposition explores the case where C,, can be made robust to Constant and Missing type anomalies, provided that accurate estimation of Cz is achievable. However, practical estimation of C2 may be challenging, particularly when we lack prior knowledge of the anomaly generation process. In summary. Theorem [heorrem[rrpeee represents a generalization from noisy labels in classification to anomalies in regression.
Theorem 2. Let be MAE or MSE loss function and be the forecasting model. Under Gaussian rype anomaly, let f arg fEF (E)]((((X))))) Y)). Then have f =Y
Theorem iimplies that MAE and MSE are both robust to Gaussian anomalies in Y. Intuitively, this is understandable since the mean of yd. is the ground truth yz., It is also worth noting that these conclusions are made statistically. If the size of the training set is relatively small or the anomaly rate is high, the performance may still drop. However, in our experiments, we find that MAE can be very robust in some real-world datasets.
Detailed proofs for the above Theorems and Propositions can be found in Appendix A It may appear straightforward that MAE is more robust than MSE, but the extent and conditions of MAE's robustness are not well understood. For instance, prior research in label noise (LNL.) (Ghosh et al. (017) has demonstrated MAE's inherent robustness to symmetric label noise and, under certain conditions, to asymmetric label noise in classification tasks. However, these analyses are Blasiffication--ocused. To our knowledge, no study has explored how MAE performs regression when dealing with various types of anomalies in time series forecasting. Notably, previous works (Connor et al. 1994] Bohlke-Schneider al. 2020 Li et al., 2022) predominantly use MSE TSFA tasks. Our analysis indicates that MAE is inherently robust to Constant and Missing type anomalies, while both MAF and MSE exhibit robustness to Gaussian type anomalies. Experimental evidence supporting these thcoretical findings is presented in Appendix C
SAMPLE ROBUSTNESS: ANOMALY EFFECT IN X
In this section, we analyze how anomalies in X impact performance while keeping the labels clean. It's worth noting that a related study (Li et al. 2021) also investigates the influence of input time series outliers on prediction performance, using a point-wise anomaly model (Equation (1) in (Li) et al. (02T)). T) In particular, (Li et al. 2021) introduces metric called SIF to quantify outlier effects on predictions. However, calculating SIF necessitates knowledge of the outlier generation process and does not propose an efficient algorithm for mitigating outlier effects. In our context, we lack information about outlier (anomaly) generation, mirroring gal-world scenarios where only observed time series available. Our goal is to leverage understanding of anomaly effects in X develop an efficient performance improvement algorithm. We begin by assuming that not all input samples equally contribute to performance, even if they statistically exhibit the same number of anomalies.
Our assumption highlights the significance of anomaly position. To validate this, we conducted simulations involving time series with anomalies placed at various positions. For instance, Figure])
4

---

Published as a conference paper at ICLR 2024
Table I: Evaluating how different positions of anomalies in the input time series affect prediction performance. For each dataset, we simulate different types of anomalies with anomaly rates 0.1 and 0.2. The loss is MAE and the network structure LSTM. Detailed setting left in Appendix E
Electricity with Anomaly Position
Traffic with Anomaly Position
Anomaly Type
front
middle
back
front
middle
back
Clean
0. 0.12200.770
0.
0.182/0.070
0. 1.1
0.194/0. 105
0.194/0.100 105
Const. (n 0.1)
0, 0.18
0.183/0.071
0.195/0.078
0. 1119000100 105
0.200/0.1 115
0.215/0. 131
Const. (n) 0.2)
0.182/0.069
0.183/0.071
0.219/0.094
0.194/0.109
0.195/0. 106
0.236/0.153
Missing. (q) = 0.1)
0. 0812//0/070
0.16//0.073
0.204/0.083
0.203/0. 118
0.203/0. 120
0.221/0.127
Missing. (1) 0.2)
0.182/0.070
0.13/00007
0.223/0.
0.204/0.116
0.205/0.116
0.251/0.
Gaussian. (n) 0.1)
0.184/0.071
0.187/0.071
0.196/0.075
0.204/0.120
0.206/0.114
0.211/0.133
Gaussian. (n 0.2) 171
0. 086/000072
0/213/0.087
0.206/0.121
0.212/0.127
0,226/0. 147
(b), (c), and (d) depict a simple example involving Gaussian type anomalies, where the number of anomalies remains consistent. Our experiments, conducted on the Electricity and Traffic datasets. demonstrate the impact of anomaly position on model performance, as summarized in Table [ Notably, when anomalies are located at the front or middle of input time series, performance remains close to that of clean data training, with minimal performance degradation. Conversely, when anomalies are positioned at the end time series, performance significantly declines as the anomaly rate increases. This is understandable since the label is situated towards the end of the input time series. Further evidence on various time series datasets can be found in Appendix D Interestingly, similar findings have been reported in temporal graph learning (Wang al. 2021b). which emphasizes the informativeness of more recent edges for target predictions."
INSIGHTS AND OUR ALGORITHM
In this part, based on the analyses in Section $ and Section 5
we show how to achieve loss robustness in TSFA Equation ① in LNL when anomalies exist in both inputs and labels.
CONNECTION BETWEEN LNL AND TSFA
Our design proceeds as follows:
Denote the original observed inputs and labels X and Y. We first select X' from X where the anomalies of X" are sparse in the time series points located near the label. Denote the ground-truth label and observed label of X' as Y' and Y", respectively. From our analyses of sample robustness in Section 5 training on (X', .Y') has very similar prediction performance compared to training (X.Y). Le., X' acts like clean input time series. Mathematically, have
arg min Exy (((()),)) a min Ex y [1((X')) /
Since the selection process does not involve the labels, the distribution of Y' with respect to X' is unchanged. Let be the robust loss (MAE) in the setting of anomalies. From our analyses of loss robustness in Section we have:
arg min E & (U((')) Y"Y' zu arg min Ex',, x'',"" ((((())))) Y
Combining the above two equations, we have:
arg arg min E ((S(') s"rggarr Ex y ((((') y.y' arg Ex,y Ex.,
(3)
Equation (3) implies that by using selective samples to train the model, the performance can be close to the clean samples training without anomalies in both inputs and labels. Equation (3) can be viewed as generalization Equation (T) which considers the noise in the inputs TSFA tasks.
We use instead of because our input time series contain anomalies. In the absence of anomalies in the input, as guaranteed by Section 4 strict equality in Equation (3) can be achieved. Unlike LNL, which assumes clean inputs, time series data can be contaminated. Fortunately, our analysis in Section JJshows that by selecting input time series with minimal anomalies near the label, we can make them comparable to clean series, shown in Table T The of x reflects this observation. In summary, while LNL. relies on strict robustness, our approach incorporates a selection procedure to ensure robustness in the presence of potential anomalies.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 RobustTSF Algorithm
Input: Initialized DNN, Time series Iteration:
{21, Zr). Robust loss Number of epochs E, Batch size M.
1. Input the time series into Equation to get the trend of time series 8.
2. Segment Σ and to build the dataset { 81, Hor),... (EN, NN. Ua x)
for e
1 to 15 do
.V for 1 to 32 do
3. Randomly select M triplets from D
4. Calculate anomaly score for each triplet from Equation (
5. Train DNN on triplets using robust loss from Equation
end for end for
Output: DNN model
6.2 ROBUSTTSF ALGORITHM
From Section 6.1 to achieve robustness in TSFA (Equation 3 it suffices to show how to select X" from X. We derive an efficient algorithm as follows:
Given time series 2 = ".,,... J.
we first calculate its trend {81, ... ,8) by solving:
T
T-1
E min Σ
81
8t-1 - 2. 8t + s141
where À > 0 is the hyper-parametee parameter, is the t-th value in s. is worth noting that Equation is variation of the original trend filtering algorithm (Kim et al. 2009). We change (2, 82) to
[. - St to improve robustness. After trend filtering, we can segment and to get the training set D { 81, ", ... NØN, N Mæx which consists of N triplets. Note that 8, is the trend of nn Let A(T.) denote the anomaly score of (In, 8n, Væ.), which is defined follows:
K
A(T,)
Σ (kk
1 sh
(5)
where K is the length of the input En. T and s are the k-th value in Hn. and 8n. respectively. ae *t denotes the extent of anomaly at time step k. w(k) is weighting function that is designed
non-decreasing, and we consider the following two functions for w(k):
Exponential: w(k) = e (k-K)); Dirac: w(k) = 0 if k < K', w(k) = 1i < K.
The reason for designing w(k) as non-decreasing function is because the position of anomalies matters for the prediction performance, as we analyzed in Section 5 we enlarge the weights when anomalies close the labels. In practice, find Dirac function performs slightly better than exponential functions. We design the final loss RobustTSF as:
(A(G,.) < <) I(@n, Fm.):
where 1() represents the indicator function, equaling when the condition is met and 0 otherwise. > O serves as the threshold and / is selected as the MAE loss function. It is worth noting that our primary focus on discerning the extent of anomalies within time series points situated in proximity to the label, followed by appropriate filtering, rather than merely determining whether given time series point qualifies as an anomaly. This focus steers our use of the hyper-parameter 7 in Equation (6). streamlining and simplifying the anomaly detection within our method. Experiments in the Appendix [.6 demonstratee that setting т to 0.3 yields favorable results across diverse datasets and settings. We term our approach RobustTSF, summarized in Algorithm [ There wwwriooniinuuuttttnneeeeeee
major pipeline (Connor
differences between our approach and previous
et al. 1994 Schhnier al. 2020):
Our method does not need pre-trained forecasting DNN to detect anomalies. Instead, our method only needs calculate anomaly score (see Equation 4 and (5). which is more time efficient.

---

Published as a conference paper at ICLR 2024
pipeline, removing anomalies post-detection is viable approach but can introduce time series discontinuities, posing a crucial challenge in TSFA. (Connor et al. 1994) employs offline value imputation, while (Boblke-Schneide
In the wiwpoonniintttttinnnneeee
et al. 2020) opts for online imputation. However, these imputations can he noisy and lack robust theoretical foundations, which RobusTSF mitigates by eliminating the need for imputation. Implicit detection in our approach serves efficient sample selection. With sample selection and robust loss, we achieve the desired robustness, as expressed in Equation (3)
While there are potential alternatives to replace Equation (4) for trend calculation and selective approaches replace Equation (5). our primary focus this paper is establish a connection between LNL and TSFA. Our aim is to demonstrate that we can leverage insights from LNL to significantly enhance TSFA performance. Therefore, we 'e opted to keep our method simple and effective, leaving room for future improvements and exploration of alternative approaches.
7 EXPERIMENTS
Here we demonstrate the effects of RobustTSF on TSFA tasks. We simply introduce experiment setting here, leaving the detailed description of experiment setup, hyper-parameter configurations, model structure, etc, to Appendix
Model and Datasets Most recent papers on robust time series forecasting use LSTM-based structures in their experiments Yoon ct al. (2022): Wang et al. (2021a): Wu et al. (202I). Similarly, related works on TSFA tasks also employ LSTM models. Consequently, we primarily use LSTM in our main paper's ss experiments. However, since our framework is model-agnostic, it can be adapted to other structures like Transformers and TCN, as demonstrated in Appendix F.12 To ensure fairness in comparisons, all methods utilize the same model structure.
We evaluate our methods on the Electricity and Traffic dattassssssss splitting each dataset into training and test sets with a 7:3 ratio and introducing anomalies into the training set. Notably, we do not create a validation set from the training set for TSFA tasks due to two reasons: 1) Training sets in TSFA are noisy, making validation set selection unreliable for model tuning (as demonstrated in the Appendix F.4): 2) The common practice in LNL suggests that DNN models tend to fit clean samples first and then noisy ones, leading to an initial increase and then decrease in performance, making it insufficient to evaluate TSFA methods solely based on the validation set. Hence, we follow the evaluation protocol from a prominent benchmark method in LNL Li et al. (2020). This protocol records DNN performance on the clean test set at the end of each training epoch, and we report both the best and last epoch test set performances for each method after training concludes. The datasets are normalized to 0 mean and std before adding anomalies and training.
In the Appendix, we extend our evaluation more datasets, including real-world time series with intrinsic anomalies (Appendix F.I. and also address the scenario where the test set contains anomalies (Appendix [.II."
Comparing methods and evaluation criterion: We compare our method to the following approaches: Vanilla training, which directly trains the model on time series using MSE or MAE loss functions, Offline imputation (Connor et al., 1994). Online imputation (Bohlke-Schneider et al., 2020). and Loss-based sample selection (Li et al., 2022). We use MSE and MAE (see Appendix [ for details) the test set to evaluate each method. It's worth noting that some methods, such as (de Bézenac et al., (2020) and ARIMA (Ho & Xie 1998), while not designed for TSFA tasks, have
shown good performance in scenarios with skewed time series. We provide additional comparisons in the Appendix F.9
Training setup: Each method is trained for 30 epochs using the ADAM optimizer. The learning rate is set to 0.01 for the first 10 epochs and 0.001 for the subsequent 20 epochs. For RobustTSF, we maintain fixed byperparameters: À = 0.3 (Equation (4). r = 0.3 (Equation (6)). and K' = K - 1 (Dirac weighting) throughout all experiments presented in the paper. It's important to note that our paper primarily focuses on experiments with varying anomaly rates while keeping the anomaly scale constant, in line with conventions in the literature of LNL and related TSFA works. However, we also conduct experiments exploring different anomaly scales, as detailed in Appendix F.7
LoaDiiiraa 201 12014
hhtp://peems.do..aggov
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of different methods Electricity and Traffic dataset for single-step forecasting. We report the best and last epoch test performance for all methods, represented as MAE/MSE. We
also report A = |best lastl, which is the average of |best - last] for all anomaly settings of each
method, to reflect the stableness of each method. The best results are highlighted in bold font. We
also establish the statistical consistency of our results, as demonstrated in Appendix F.17
Dataset
Mothod
Clean
0
Constnn Anomaly
0.3
0.3
- 0.1
Missing Anomaly
0.3
Geuszaan Anomaly
0.3
0.3
A
(MSE)
Vanilla
best
last
0,205/0.082
0.18100071
0.21900.811
19900077
012
023900095
0.22800097
0.35410.97 0.229000988
0.5500.157
0.213000087
0.19220076
0,255/0.124
0.221000066
0.019
(MAK)
Vanilla
best
last
0.182000700
09890009
0. 1917077777
0-22340.090 096
00266000066
0.2500.190 109
0.215000000
D. 0:205/02866
0 0:225000922
0. 1999X02080
0.20330 082
0.216800056
0.219010996
0.015
Elecricity
Offine
best
last
0.187/0:072
0.1800071
18770011
0. 0911000073
021
0.23//00998
0.203/0.022
1.000.0
0.12/0.884
0.237/0.100
0.00000070
0.186/0.011
0. 0.207/0.083
0.19310075
0.009
Online
best
last
0.183/0.070
0.201/0:077
0.21200.044
0.9900.077
0 0.22//0009
0,25000.00 104
0.21500.084
0.19510.075
0.234H0.997
0,27/000933
0.20440.078
0.1940,074
0.21500087
O 0099/0.084
0.01
Loss sel
best
last
0. 0.18800068
0. 0.982
0.20880081
0. 21208A0.00 085
0.217A0089
0.20900081
D. 0.205A00083
0. 19840078
0.217A0090
0.214000899
0.19660777
O. 20200000033
O. 0.210NL0886
0.204R0083
0.004
RS
best
lass
0.177/0.066
0.177A0,065
0.177/0066
0.18220.068
0.190/0.011
0.18380070
0.8814068
0.187/0.071
0,204/0.079
0.19400074
0.18220.069
0.17640067
0.177/0.068
0.177A00077
0.004
(MSE)
Vanilla
best
last
119
0.199/0.118 116
0.220/0.126
023300122
2.3300.130
0.275/0.173
0.5110.357
2.29 121 35
0.233/0.144
O 0.5044..226
0.007
(MAE)
Vanilla
best
last
0.
0.20//0.114
0.210/01133
0.209/0.122
1.24/0.1511
0231100136
0.272/01866
0 0.266/0170
0.707002220
0.2160,1110
0.224/0.18
0.33000158
0.234.0.59
0.088
Offiiee
hesa
0222211133
11221A1133
0.218ALI25
0.005
Traffic
lasi
0.193/01110
0.220A0.133
0
2.22/001355
O 0.81001187
209900.123
O 0.33001331
Online
best
last
0,208/0.120
021660113
0.229/0.136
0.248/0.13
023900116
11
0.285/0.177 123000.126
023
023
02440.141 02240112
0,256/0.160
0.016
Loss sel
best
0.196/0.115
199990111
120
0.226/0 144
151
25577001600
26001170
0.241701533
1-24100-53
0.52500235
029241225
0211300133
0-220N0-33
O 0626N00119 179
26260.199 179
0.004
RobuxTSF
best
last
0.185/01000
0.185/0.100
0.19800113
0.200/0.112
0.209001116
0.203/01111
0.203/0.114
0.20100113
0.18511101 1.0
0.250N0.152
0.185/0.101
0.205/0.119
0.202A0.116
0.003
Table 3: Comparison of different methods on Electricity dataset for multi-step forecasting. We report the best epoch performance for all the methods by MAE/MSE. The anomaly ratio for each anomaly type is 0.2. The prediction length is 4 and 8. The best results are highlighted in bold font.
Clean
Constont (5) 0.2)
Missing in 0.2)
Gaussion (1 0.2)
Method
Vanilla (MSS)
175
0.32120.191
0.38420.622
0.36170.229
0:4230.299
0. 0.32480.191 191
0.309N 241
Vanilla (MAE)
0,292/0.165
0.302/0.174
0.382/0.264
0,327/0.192
0.394/0 283
0.310N0.181
0.371/0.247
Offline
0,302/0.179
0.314/0.184
0,399/0286
0.334/0.216
0.432/0.358
0.312/0.188
0.407/0.300 306
Online
0.30800.
0,30690.176
0.41300.308
0.337/0213
0147704122
0.34683.216
0.471A0.31
Lass sel
0. 184
0.31990.97
0.38300.34
0.34200217
0111200118
1.319/0.195
RS
0.280001158
0,292/0.163
0.371/0.52
0.3140.185
0.371/0.249
0.284:0.16
0.383/0.234
7.1 SINGLE-STEP TIME SERIES FORECASTING
We first perform experiments on single-step forecasting following exact problem formulation in Section 3 The length of input sequence is 16 for Electricity and Traffic. The overall results are reported in Trom which we can observe some interesting phenomenons:
1) Among all anomaly types, missing anomalies have the most significant negative impact on model performance 2) Offline imputation, online imputation, and loss selection approaches do not consistently improve performance across all anomaly types within each dataset. For instance, the online imputation method improves performance for the missing anomaly type in the Traffic dataset but decreases it for constant and Gaussian anomalies. This phenomenon also occurs with boss-based sample selection. The difference arises because, unlike LNL, TSFA involves noise (anomalies) in both X and Y, making the small loss criterion in LNL less suitable for TSFA tasks. 3) Loss selection and RobustTSF are the most stable methods (with small A values). Both approaches select reliable training samples, but RobustTSF outperforms the loss selection approach. 4) RobustTSF consistently improves performance compared to vanilla training across all anomaly types and datasets. This suggests the effectiveness of our selection module. Notably, even without manually adding anomalies (clean), RobustTSF still exhibits clear improvements. This is because sensor-recorded data inevitably contain anomalies. Thus, our method serves as a general approach for time series forecasting tasks.
7.2 GENERALIZATION TO MULTI-STEP FORECASTING
Our approach can be easily generalized to multi-forecasting tasks, i.e.. predicting multi-step values given input time series. For this part of experiment, we evaluate RobustTSF on Electricity dataset. The input length is fixed to be 96 and the prediction length O {4.8). The best epoch performance for each method is reported in Table 3 We can observe that RobustTSF still has consistent improvement compared to the other baseline methods, including the setting of clean time series.
8

---

Published as a conference paper at ICLR 2024
7.3 GENERALIZATION TO OTHER ANOMALIES
Even though our theories in Section 3 are based on three common point-level anomalies, i.e., Constant, Missing and Gaussian Anomalies, RobustTSF works well when anomalies follow other heavy-tailed distributions like student-t distribution and Generalized Pareto distribution. The corresponding evaluations are left in Appendix F.10. Furthermore, RobustTSF can also be adopted case of subsequence anomaly. Considering simple subsequence anomaly modeled by Markov process:
2t:
with probability -1 with probability 1]
If ž is clean,
21, E2- + c ((0000,.) with probability Φ
with probability 1 -
If 2 is anomaly,
(8)
Equation behaves like a point-level anomaly if the previous time step is clean. However, when the previous time step is an anomaly, the property of subsequence anomaly suggests that the current time step is likely to also be an anomaly, with a value close to the previous anomaly (Equation (8). We find that RobustTSF performs well in this scenario. This is because our filtering scheme (Equation (6) primarily selects samples with clean time steps near the label, allowing anomalies to be modeled by Equation (7) after sample selection. Empirical experiments in Table confirm this observation, as RobustTSF outperforms other methods in the subsequence anomaly setting. It's worth noting that for more complex anomaly types that may not satisfy P((Y)) A P( X), MAE may not be theoretically robust. Analyzing and developing loss functions for these anomaly types presents an interesting avenue for future research.
Table 4: Comparison of different methods on Electricity dataset for single-step forecasting with subsequence anomaly. We report the best epoch performance for all methods by MAE/MSE. We fix The best results are highlighted in bold font.
? 0.3 in Equation 7 and vary @ in Equation 8
Gaussian 0.3)
Method
á 0.1
0 = 0.3
à 0.5
6 = 0.7
6 - 0.9
Vanilla (MSE)
0,206/0.083
0.214/0.099
0.253/0. 111
0.337/0.209
0.47970.8 388
Vanilla (MAE)
0,201/0.080
0.19//0.076
0.194/0.076
0,202/0.081
0.206/0.084
Offline
0.194/0.075
0.194/0.075
0.191/0.074
0.198/0.078
0.203/0.083
Online
0.196/0.075
0.197/0.077
0.197/0.075
0.197/0.077
0,20000.079
sel
0. 0.198/0.75
0.199/0.079
0.199/0.079
0,202/0.082
0.203/0081
RobustTS:
0.17970.067
0.180/0.068
0.185/0.072
0.187/0.073
0.198/0.079
7.4
TUNING AND DISCUSSIONS
To further study the effect of each component of RobustTSF, we conduct Slyper-parameter Tuning towards the loss, weighting strategy and anomaly detection method shown in Table showing the advantage of each component of our design. More discussions (including test set with anomalies, innrtiioniimpaattionn-to
forecasting visualization, sensitivity of efficiency analyses, RobustTSF on more datasets, RobustTSF on Transformer and TCN models, comparing RobustTSF with more methods, more discussion on RobustTSF such significance and future work, etc) are summarized in Appendix F
pipeline, parameter sensitivity,
Table 5: performance (MAE) for all methods. The best results are highlighted in bold font.
Tuning of RobustTSF on Electricity dataset. We report the best epoch
Method
Const
Missing
Gaussian
==0 0.1
=
0.3
== 0.1
= 0.3
== 0.1
0.3
RS (MAE, Dirac weighting. tread filier detection)
0.177
0.183
0.181
0.194
0.176
0.177
RSF (MSE, Dirac weighting, trend filter detection)
0.180
0.193
0.182
0.249
0.177
0.181
RobustTSF (MAE, exponential weighting, trend lilter detection)
0.178
0.183
0.182
0.199
0.179
0,180
RS (MAE, Dirac weighting. prediction based detection)
0.191
0,217
0.195
0.206
0.180
0.184
8 CONCLUSIONS
This paper aims to deal with TSFA (time series forecasting with anomaly) tasks. We first define common types of anomalies and analyze the loss robustness and sample robustness on these anomaly types. Then based on analyses, develop efficient algorithm to improve model performance on TSFA tasks and achieve SOTA performance. Furthermore, this is the first study to form a bridge between LNL task and TSFA task and may potentially inspire future research this topic.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sercan O Arik, Nathanael C Yoder, and Tomas Pfister. Self-adaptive forecasting for improved deep learning on non-stationary time-series. arXiv preprint arXiv:2202.02403, 2022.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrett networks for sequence modeling. arXiv preprint arXiv:1803.0127) 2018.
Ane Glrequ, Angel Conde, Usue Mori, and Jose A Lozano. A review outlier/anomaly detection in time series data. ACM Compuling Surveys (CSUR), 54(3):1-33, 2021.
Michael Bohlke-Schneider Shubham Kapoor, and Tim Januschowski. Resilient neural forecasting systems. Proceedings of ACM SIGMOD Workshop on Duta Management for End-to- tn- Machine Learning (SIGMOD-DEEM 2020), 2020.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021a.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference on Learning Representations, 2021b.
Jerome T Connor, R Douglas Martin, and Les E Atlas. Recurrent neural networks and robust time series prediction. IEEE transactions neural networks, 5(2):240 254, 1994.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke- Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances Neural Information Processing Systems, 33:2995-3007, 2020.
Gregory Ditzler, Manuel Roveri, Cesare Alippi, and Robi Polikar. Learning in nonstationary environ- ments: A survey. IEEE Computational Intelligence Magazine, 10(4):12-25, 2015.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems. pp. 3527-8537, 2018.
Siu Lau Ho and Min Xie. The use of arima models for reliability forecasting and analysis. Computers & industrial engineering, 35(1-2):213-216, 1998.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. MentorNet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning (ICML), pp. 2304-2313, 2018.
Seung-Jean Kim, Kwangmoo Koh, Stephen Boyd, and Dimitry Gorinevsky. E trend filtering. SIAM review, 51(2):339-360 2009.
Weihao Kong, Rajat Sen, Pranjal Awasthi, and Abhimanyu Das. Trimmed maximum likelihood estimation for robust learning in generalized linear models. arXiv preprint arXiv:2206.07777, 2022.
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 104, 2018.
10

---

Published as a conference paper at ICLR 2024
Kwei- Lai, Daochen Zha, Junjie Xu, Yue Zhao, Guanchu Wang. and Xia Hu. Revisiting time series outlier detection: Definitions and benchmarks. In Thirty-fifth conference on neural information processing systems (NeurIPS), 2021.
Jianbo Li, Lecheng Zheng, Yada Zhu, and Jingrui He. Outlier impact characterization for time series data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11595-11603, 2021.
Junnan Li, Richard Socher, and Steven C.H. Hoi. DivideMix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020.
Wenkai Li, Cheng Feng, Ting Chen, and Jun Zhu. Robust learning of deep time series anomaly detection models with contaminated training data. arXiv preprint arXiv:2208. 01841, 2022.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A, 379(2194):2220020, 2021.
Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In Interna- tional Conference on Machine Learning, pp. 7735, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise
rates. In International Conference on Machine Learning (ICML). pp. 6226 6236, 2020.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions Knowledge and Data Engineering, 31(22)2334--266, 2018.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normal- ized loss functions for deep learning with noisy labels. In International Conference Machine Learning (ICML), pp. 6543-6553, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1954-1952, 2017.
Ankit Pensia, Varun Jog, and Po-Ling Loh. Robust regression with covariate filtering: Heavy tails and adversarial contamination. arXiv preprini arXiv:2009. / 2296. 2020.
David Salinas, Michael Bohlke-Sehneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank Gaussian copula processes. Advances in neural information processing systems, 32, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3); 1181-1191, 2020.
Hwanjun Song, Minscok Kim, Dongmin Park, Yooju Shin, and Jac-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions Neural Networks and Learning Systems, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
Yingjie Wang, Xianrui Zhong, Fengxiang He, Hong Chen, and Dacheng Tao. Huber additive models for non-stationary time series analysis. In International Conference on Learning Representations, 2021a.
Yiwei Wang, Yujun Cai. Yuxuan Liang, Henghui Ding, Changhu Wang, Siddharth Bhatia, and Bryan Hooi. Adaptive data augmentation on temporal graphs. Advances Neural Information Processing Systems, 04::1440-1522 2021b.
11

---

Published as a conference paper at ICLR 2024
Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An industrial perspective. Proceedings of the 28th ACM SIGKDD Conference on Knowledge 2022.
Discovery and Data Mining (KDD), pp.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. DeepTIMe: Deep time- index meta-learning for non-stationary time-series forecasting. arXiv preprint urXiv:2207.06044 2022.
Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang. Haifeng Chen, and Susan B Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 651-659, 2021.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L.dmi: A novel information eeoormttin-theoreetic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems, volume 32, 2019.
Taeho Yoon, Youngsuk Park, Ernest Ryu, and Yuyang Wang. Robust probabilistic time series forecasting. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp. 1336-1358, 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference artificial intelligence, volume 37, pp. 11121-11128, 2023.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, pp. 8778-8788. 2018.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedingy the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 101 13-10223, 2021a
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning (ICML), pp. 12912- 12923, 2021b.
12

---

Published as a conference paper at ICLR 2024
APPENDIX ARRANGEMENT
The appendix is arranged as follows:
Section
proving the theorems in the main paper;
Section
illustrating the common anomaly types by visualization;
Section
experiments Loss robustness;
Section
more experiments on Sample robustness;
Section
detailing the experimental settings in the paper,
Section
More experiments and discussions:
Section F.1
applying RobustTSF when test set also has anomalies;
Section
visualizing forecasting results;
Section
more experiments on subsequent anomalies and multi-step forecasting;
Section
showing the unreliability of model selection via noisy validation set; showing the sensitivity of pnpectoonimpuuaatio-rrrraaaiiiii Pipeline;
Section
Section F.6
expriments for parameter sensitivity analyses;
Section
experiments with respect to anomaly scales;
Section F.8
efficiency analyses for each method;
Section
comparing RobustTSF with more methods;
Section 10
applying RobustTSF on other anomaly distributions;
Section
applying Robus(TSF on more datasets;
Section
evaluating RobustTSF on Transformer and TCN Models;
Section
discussing the originality, significance and future work of RobustTSF.
Section
evaluating RobustTSF for low noise ratios.
Section F. F.15
additional elucidation for Equation (4)
Section
additional ablation studies for RobustTSF
Section
F.17
proof statistically consistent results for RobustTSF
A PROOF FOR THEOREMS
A.1 PROOF FOR THEOREM
Let RC(F) Exl(f(X), Y)
ER ((f(x), vw) and RT(() Ez((f(x),))
Eg. (f((()) Nz). respectively. Then
R²())
Br.ie & (() Vo))
Ryp Ey lac BR Vu 1 (()(x). Fz)
- le [11 11) -((((x), Mee) + m ((f(x)) vall
=Rap Eyy loe [11 11) ((((xx) ymm + n (C. {(f(x)) yz))
=(1 2 17) Re( 17- Ez C2.
where 1 > 0 and 72 = 1) Ez.g. Ca. which are constants respect to f.
A.2 PROOF FOR PROPOSITION
Let x then we are validating the value of Elg, yæ) + E(g, 2 For MAE loss, the value is represented as Vzl lq v²l- From the property of absolute function, Un lq /
achieves minimum when q lies in the range between Uæ and y Thus when min {ye V < q <
max (va, v we have lg ynl + lq Vol C.
Va V
This conclusion can not be fit for MSE loss, since (q - yn) + (q y2) is not constant when min (Ua, v < q < max y²)
13

---

Published as a conference paper at ICLR 2024
A.3 PROOF FOR PROPOSITION 2
Following the notation in Section A.2 constant respect to
it is easy to verify that foorm (g, Mee) + Enorm (4, ya) -1, which
A.4 PROOF FOR THEOREM 2
Following the notation in the proof for Theorem I, we aim to show that E,, Eye (n((11))-(((((a) Un) )m)
7 ((f(x), v. v22 (NVa, o²) achieves minimum when f(x) V/.
MSE loss, let q = f(x), then we are to show that (1 - 17) (q yo ya) + 1 (4 - y
achieves minimum when q = y/- Since (1 17) (q )=)² achieves minimum when y yn. we only need prove --x g(v2) (yá dy achieves minimum when q = y/ where g(v
(3C - is the pdf of the distribution of v
V250 exp(
20
Let s()) = Byg (vz =)=== -to +o s(V2) (u d then
ds(g)
dq
= q 4-2 2 Eng g (1 - = g =
Thus 1-x ((v²) (y - 1003-g222 dy achieves minimum when
MAE loss can also be proved with similar procedure as follows.
Let 8(g) = Eyg we gl = - ( 1uc2 gldy then
0s(g)
0
0q
(v) 1)
E
1y q
E 11 9} 1 4))
Ply < 7) = P(y > 9)
V
Thus 1t s(vn) lv gldy achieves minimum when Væ.
B VISUALIZATION FOR COMMON ANOMALY TYPES
As discussed in Section three types of anomalies are considered in this paper. Figure 2 shows visualization examples for these anomaly types.
C EXPERIMENTS ON LOSS ROBUSTNESS
In Section 4 we provided our Theorems on loss robustness when there are anomalies Y. Here, we examine our Theorems in two real-world time series datasets: Electricity and Traffic, which are used by many time series forecasting papers (Yon et al. 2022 Wang al. 2021a] Wu et al. 2021). Since many works (Connor et al. 1994 Bohlke-Schneider et al. 2020 Li et al. (022) dealing with TSFA problems deploy RNN-based structure to conduct experiments, we choose to use LSTM to validate our Theorems. Experiments are shown in Table 6 It can be observed that for Constant and Missing type anomalies, MAE is very robust whose performance does not vary too much when anomalies exist while MSE degrades the performance with the increase of the anomaly rate. For Gaussian type anomalies, MAE and MSE can both robust. These results support our theoretical analyses for loss robustness when anomalies only exist in the label.
14

---

Published as a conference paper at ICLR 2024
Clean series
[b] Constant type anomalles
Missing type anomalles
(0) Gaussian type anomolies
2.5
1.-
1.0-
0.5
0.5
1.0
0.-
0.5
0.0
0.0
-05
-0.5
-1.2
-1.0
sinc
sories
Lirne
arrornaly
anomaly
ansomnaly
-1.-
-1.
60 1.0
80
20
60
Figure 2: Visualization of time series with different types of anomalies. (a): Clean time series which the sine function of time steps. We normalize the time series to O mean and std. (b) (c) (d): Time series (sine) with Constant, Missing, and Gaussian type anomalies. The noise rate for these types of anomalies 0.2. The noise scale is 1.0 for Constant and Gaussian anomaly and for Missing anomaly.
Table 6: Comparison of loss functions different anomaly types with anomaly rate 0.1 and 0.2. For each setting, we report the best performance on test set, presented as MAE/MSE, The model structure is LSTM. Detailed experimental setting is left in Appendix E
Constant Anomaly
Missing Anomaly
Gaussian Anomaly
Dataset
Loss
1j 0.1
11 0.2
71 = 0.1
1i 0.2
TJ 0.1
? = 0.2
MAE
0.185/0.070
0.187/0.071
0.199/0.071
0.195/0.078
0.18//0.073 0.184/0.070
Electricity
MSE
0.190/0.071
0.205/0.081
0.209/0.082
0.217/0.089
0.187/0.071
0.188/0.074
MAE
0.194/0. 108
O. 199
0.206/0.119
0.210/0. 123
O. 198/0.112
0.206/0.117
Traffic
MSE
0.208/0.119
0.222/0.124
0.249/0.148
0.302/0.191
0.204/0.121
0.214/0.131
Table 7: Evaluating how different positions of anomalies in the input time series affect prediction performance. For each dataset, simulate different types of anomalies with anomaly rates 0.1 and 0.2. The loss is MAE and the network structure is LSTM. we report best epoch MAE for each setting
ETT-hl with Anomaly Position
Exchange with Anomaly Position
Anomaly Type
front
middle
back
front
middle
back
Clean
0.051
0.051
0.051
0.047
0.047
0.047
Const. (1) 0.1)
0.051
0.051
0.052
0.047
0.049
0.052
Const. (1) 0.2)
0.052
0.051
0.055
0.049
0.050
0.060
Missing, (1) = 0.1)
0,052
0.052
0.054
0,049
0.054
0.173
Missing. 0.2)
0.052
0.052
0.058
0.050
0.050
0.214
Gaussian. (17 0.1)
0.051
0.051
0.053
0.048
0.049
0.057
Gaussian. (n) 0.2)
0.052
0.052
0,058
0.050
0.051
0.062
D MORE EXPERIMENTS ON SAMPLE ROBUSTNESS
We show more experiments regarding to Sample Robustness to support our hypothesis in Section The datasets used are ETT-h1 (Zhou et al. (02T) and Exchange (Lai al. (2018). The results in Table 7 demonstrates minimal performance drop for distant anomalies, consistent with Section
E DETAILED EXPERIMENTAL SETTING
We use the same LSTM model structure to implement all the methods for fair comparison. Note that all the methods in Table 2 are model agnostic, except for online imputation which needs specific auto-regressive RNN-based model. The LSTM Table I Table 2] Table 3 Table Table 12 consists of two recurrent layers with the hidden size 10
Model structure:
in the paper (Table 6
and fully connected layer.
The Transformer model we use in the paper (Table 24 consists of a Positional Encoding layer with feature size 250, vanilla transformer encoding layer with 10 heads, and fully connected layer.
Dataset preprocessing: We first normalize the clean training set to 0 mean and 1 std and add anomalies. When adding constant type anomaly, the noise scale is 0.5 std = 0.5; when adding
15

---

Published as a conference paper at ICLR 2024
missing type anomaly, the noise scale is O; when adding Gaussian type anomaly, the noise scale c < 2- std = 2; The clean test set is also scaled using the scaling parameter of training set when evaluating each method, i.e., Stest Zeo where μf and ó are the mean and std of the training respectively.
Evaluation: We use MAE and MSE on the test set to evaluate each method:
MAE 1
1 V 0)²
Σ y úl
MSE
T2
1=1
Shared syper-parameter for all the methods We train each method for 30 epochs and report the best epoch test performance and last epoch test performance. The learning rate is 0.1 for the first 10 epochs and 0.01 for the last 20 epochs. The optimizer Adam and the batch size is 128.
Specific hyper-parameter for offline and online imputation Since deteecion-immuuttion-eeenee pipeline requires a threshold determine whether each time step an anomaly step and this pipeline can be sensitive to anomaly types (Section F.5)). When training ieteectionimmputatii retraining pipeline on different anomaly types, we choose the threshold from {0.5.0.6.0.7.0.8)) get the best result.
Specific
for loss-based sample selection: For each anomaly type, we pretrain the model for 3 epochs and record the loss of all the samples after each training epoch ends. Then we calculate the mean and std the loss for the samples and select the samples with small loss and small loss std. Note (Li et al. 2022) assumes the anomaly ratio is known. Thus we select (1 anomaly ratio)- Num. of Samples.
Specific pprem-prremeter for RobustTSF: We fix À = 0.3 (Equation 4), 0.3 (Equation (
K' K - 1 (Dirac weighting scheme, where K is the length of the input time series) for all the experiments in the paper.
F MORE EXPERIMENTS AND DISCUSSIONS ON ROBUSTTSF
F.1 APPLYING ROBUSTTSE WHEN TEST SET ALSO HAS ANOMALIES
In the main paper, we assume the test set clean without anomalies by following the setting of learning with noisy labels and related works TSFA tasks (Connor et al. 1994). However, the test set may also have anomalies in the real-world scenario. Here we show how to apply RobustTSF when the test set is noisy.
Assume anomalies exist in in the test set. Since is used to justify the performance of each method, noisy Y cannot determine the method is learning towards the right target. When performing testing, we first use a modified trend filter (Equation (5) in the main paper) to calculate the trend of each input æ. the amount of anomaly the last time step is smaller than т, then we do modify Otherwise, we use the last value of the trend to replace the value of the last time step in æ. Then we use the trained model to evaluate the test set. We report the performance (best epoch MAE) on the Electricity dataset shown in Table It can be observed that RobustTSF outperforms other methods on test sets with anomalies, providing valuable insights for practical applications.
Table 8: Comparing RobustTSF with other methods noisy test set anomalies
Constant
Missing
Gaussian
Method
17 0.01
0.05
0.01
17 = 0.05
0.01
1)
0.05
MAE
0.192
0.253
0.201
0.313
0.200
0.258
MSB
0.211
0.281
0.237
0.387
0.212
0.275
Offline
0.190
0.232
0.198
0.324
0.191
0,255
Online
0.197
0.270
0.202
0.362
0.212
0.250
Loss sel
0.207
0.235
0.208
0.311
0.199
0.236
RobustTSF
0.174
0.219
0.183
0.302
0.178
0.190
16

---

Published as a conference paper at ICLR 2024
F.2 VISUALIZING FORECASTING RESULTS OF EACH METHOD
In this part, we visualize the forecasting results of each method to see if RobustTSF can learn intrinsic patterns within the data. Each method trained on the Electricity dataset with Gaussian noise
(1) 0.3) and tested on the clean test set. The results are shown in Figure 3 It can be observed that
only RobustTSF can learn underlying patterns within the time series.
(a) Imputation method
(b) selection method
(c) RobustiSF
0.4
0.4
preckoitor
prediciton
prediciton gonund trith input time veries
0.2
genund truth
genund truth
0.2
imp of time series
0.00
input time series
0.0
-0.
-025
-0.2
-0.4
-0.4
-0.6
-0755
-0.6
-1.00
-0.8
-.0
1225
-1.0
-12
15
20
15
20
5
20
time Stop
time stop
tim stop
Figure 3: (a) The forecasting result of offline imputation. (b) The forecasting result of loss- sample selction. (c) The forecasting result of RobustTSF. The length of input time series is 16 and the forecasting horizon is 4.
F.3 MORE SUBSEQUENCE ANOMALY EXPERIMENT AND MULTI-STEP FORECASTING EXPERIMENT
In the main paper, show the result of Gaussian-based subsequence anomaly. We provide experi- ments of constant and missing type-based subsequence anomalies in this section. Results are shown in Table 9 It can be observed that RobustSTF is still superior the other methods.
Table 9: Comparison of different methods on Electricity dataset for single-step forecasting with subsequence anomaly. We report the best epoch performance for all methods, represented as MAE/MSE. We fix = 0.3 in Equation (7) and vary & in Equation The best results are highlighted in bold font.
Constant (1) 0.3)
Missing (n) 0.3)
Method
= 0.5
6 0.7
6= 0.9
Ø = 0.5
$ = 0.7
o= 0.9
Vanilla (MAE)
0.207/0.082
0.210/0.083
0.210/0.085
0.213/0.090
0.212/0.085
0.202/0.081
Offline
0.202/0.079
0.208/0.083
0.210/0.085
0.206/0.084
0.209/0.084
0.202/0.022
Online
0.216/0.087
0.215/0.084
0.208/0.082
0.285/0.144 149
0.295/0.156
0334/0.220
Loss sel
0.204/0.081
0.208/0.084
0.209/0.085
0.213/0.088
0.210/0.086
0.237/0. 104
RobustTSF
0.188/0.072
0.194/0.077
0.200/0.081
0.192/0.074
0.192/0.074
0.193/0.075
For the multi-step forecasting, we use another setting to perform experiments with input length of 12 and prediction length of 12. The experiments on Electricity dataset are presented in Table I0 The results affirm that Ro maintains good performance with input and output length of 12.
F.4 MODEL SELECTION VIA NOISY VALIDATION SET
In this section, we aim to show that the noisy validation is not reliable for selecting models. We split the Electricity data into separate training, validation, and testing sets with a ratio of 6:2:2 and add the anomalies in the training set and validation set. The best epoch test accuracy and the test accuracy using the model selected by the validation set are reported. The results are shown in Table IT It can be observed that A is relatively large whose value is even similar to |best last in Table Thus using the noisy validation set to select the model is not reliable.
17

---

Published as a conference paper at ICLR 2024
Table 10: Comparing RobustTSF with other methods for multi-step forecasting with input and output length of 12. Best epoch MAE is reported.
Constant
Missing
Gaussian
Method
1 = 0.1
1 = 0.3
7) = 0.1
1 = 0.3
7= 0.1
7) = 0.3
MAE
0.550
0.575
0.544
0.572
0.541
0.563
MSE
0.551
0.578
0.545
0.595
0.545
0.568
Offline
0.539
0.552
0.527
0.616
0.531
0.546
Online
0.623
0.627
0.617
0.680
0.605
0.654
Loss sel
0.560
0.564
0.535
0.623
0.549
0.563
RobustTSF 0.512
0.543
0.525
0.547
0.516
0.523
Table II: Validating the reliability of noisy validation on Electricity dataset. best represents the best epoch test accuracy, val_best represents the test accuracy using the model selected by the validation set. The performance is represented MAE/MSE. We also report A |best val best to reflect the reliability of using the noisy validation set to select the models.
Constant Anomaly
Missing Anomaly
Gaussian Anomaly
Method
71 = 0.2
17 =
== 0.2
1 U.3
1 = 0.2
1 = 0.3
Vanilla
0.217/0.086
0.222/0.097 0.252/0.112
0,234/0.094
0,240/0.
0,205/0.083
0.203/0.078
0.014
(MAE)
beet
0,238/0.101
0,247/0.107
0.251/0.111
0,218/0.090
0.221/0.092
sensitivity of 5 Gaussian anomaly
sensitivity for Missing anomaly
220
0.7
0.7
5 0.9
6 0.9
0.26
025
best MAE
0.22
2.2
0.28
233
0.30
0.15 anomaly ratio
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
anomaly ratio
Figure 4: Sensitivity of 8 with Gaussian and Missing anomaly for offline detection-impuration: retraining pipeline.
F.5 SENSITIVITY OF DETECTIINNIMUUTTIIONN RETTRRIANTTTOONEERIIINNNNG PIPELINE
pipeline (Connor al. 1994, Soolke-Schheidee et al. 2020 et al. (022) is the threshold for determining whether each time step is an anomaly, i.e,, 2, 2, > ò, where ž, the model prediction at time step 2, the original observed value at time step i and 5 is the threshold. We conduct an experiment on Electricity dataset shown Figure It can be observed that for Gaussian anomaly, different values of result in similar performances and 8 0.7 performs slightly better than others. However, for Missing type anomaly, values of 8 can make performance vary very much and 5 = 0.9 performs better than others. wwwnnn-mmmuattttttruutte
The most important hyper-parameee for detection innrriiiommmuutttiinniie
This part of experiment shows that the syper-parameter of sensitive to anomaly type. Recall that for RobustTSF, we fix all the hyper-parameters parameters for all the settings. Thus RobustTSF is more robust to parameers..
pipeline
18

---

Published as a conference paper at ICLR 2024
sensitivity A (Constant Anomaly)
sensitivity (Constant Anomaly)
sensitivity of (Gaussian Anomaly)
sensitivity of (GaussianAAnmmally
0.23
0.22
0.22
0.22
0.22
0.21
0.21
0.21
0.21
baxt MAR
0.20
0.20
0.20
0.20
0.19
0.19
0.19
0.19
0.17
0.17
0.17
A-0.2
0.17
6.16-
1:3
0.16
T=
0.16
A=03
0.16
1=0.3
1=0.4
0.4
A=4
0=
0.15
0.15
0.15
0.15
0.10
0.15
0.20
0.25
0,30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
sensitivity of [Constant Anomaly)
sensitivity (Constant Anomaly)
sensitivity of (Gaussian Anomaly)
sensitivity of (Gaussian Anomaly)
0.21
0.21
0.21
0.21
bext MAE
0.20
0.20
0.20
0.20
0.19
0.39
0.19
0.19
0.18
0.18
6.18
0.18
0.16
0.16
0.16
A-0,3
0.16
T=0.3
1-04
4-044
T-0.4
0.15
0.15
0.15
0.15
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
0.10
0.15
0.20
0.25
0.30
anomaly ratio
anomaly ratle
anomaly ratlo
anomaly ralio
Figure 5: The figure illustrates experiments concerning the hyperparameters and for RobustTSF on both the Electricity and Traffic datasets, each containing different types of anomalies. The first row of subfigures corresponds to the Electricity dataset, while the second row corresponds to the Traffic dataset. Notably, the results indicate that RobustTSF remains robust across various settings of hyperparameters. We reiterate that we maintain consistent hyperparameters across all settings when comparing with other methods.
F.6 PARAMETER SENSITIVITY ANALYSES
To further study the influence of the hyper-parameter of RobustTSF, we conduct the parameter sensitivity analyses on the Electricity and Traffic dataset. The results can be observed in Figure can be seen that RobustTSF is quite robust to these hyper-parameters. Each setting can outperform other baseline methods.
We also perform experiment to show the influence of different sample selection metric shown Table 12 It can be observed that random selection makes performance worse while the weighting schemes, which are all non-decreasing function with respect to the time step, have similar performance on TSFA tasks.
F.7 EXPERIMENTS WITH RESPECT TO ANOMALY SCALES
There exists three hyper-parameter parameters, namely y, and characterizing three anomaly types we defined in the paper. our main paper, we have concentrated our experiments on the r) hyyer-paaaaette which aligns with the conventions found in the literature of LNL and related TSFA works. This choice is rooted in the fact that even the original TSFA paper Connor et al. 1994) employed fixed y² value throughout their experiments.
For both e and 7, we have extended our experimentation efforts, as reflected in Figure Specifically, Figure (a) and (c) delineate the experimentation for the Constant anomaly, while Figure 6 (b) and (d) illustrate the e experiment for the Gaussian anomaly. It's worth noting that in the context of Gaussian anomaly, the value of 72 characterizes the distribution of In Figure 6 (b) and (d),
19

---

Published as a conference paper at ICLR 2024
Table 12: Comparison of different sample selection metric for RobustTSF. 'Random' means randomly select N * (1 - anomaly ratio) samples for training. We report the best epoch performance for all the methods, represented as MAE/MSE.
Constant
Missing
Gaussian
Method
1) = 0.3
1 0.3
1) 0.3
Random
0.222/0.089
0.211/0.086
0.201/0.079
Exponential weighting
0813//0.070
0.196/0.075
0.179/0.067
Dirac weighting K'= K
0. 0.183/0.070
0.194/0.074
0.177/0.067
Dirac weighting K" = K 2
0.187/0.072
0.188/0.073
8.180/0.066
Dirac weighting K" =K=K KK 4
0.189/0.073
0.200/0.079
0. 0.186/0.073
[a] Constant anomaly (Electricily)
(b) Gaussian anerraly (Electricity)
(c) Constant anomaly (Traffic)
(d) Gaussian anomaly (Traffic)
0.22
0.25
0.21
0.22
0.24
0.23
0.21
0.20
0.20
0.19
0.22
0.21
0.19
0.18
0.18
0.20
0.20
0.17
Disseline RocustTSF
0.17
Baseliee MAL
Baseline
Baseline MAE
RobusITSF
RobustTSF
0.36
0.16
0.18
0.18
0.6
0.4
anomaly scale
anomaly scale
anomaty scale
anomaly scale
Figure 6: The figure depicts experiments on various anomaly scales, with a fixed anomaly ratio of 0.3 across all settings. The x-axis represents the anomaly scale and the y-axis represents the Best epoch MAE for each method. Notably, RobustTSF demonstrates robustness to different anomaly scales across diverse anomaly types. While the baseline also maintains robustness against Gaussian anomaly scales, its performance is impacted by Constant anomaly scales. It's worth mentioning that anomaly scale experiment isn' 't feasible for the Missing type anomaly, as the anomaly scale remains consistently due to the presence of missing values.
the x-axis corresponds to 2 Notably, the findings unveil that while the baseline (MAE) retains its robustness in the face of varying 72 (i.e., the variance of anomaly does not affect the methods very much), its performance does exhibit sensitivity to Constant anomaly scales.
F.8 EFFICIENCY ANALYSES
We report training time for each method with LSTTM structure on Electricity dataset with constant anomaly (17 = 0.3) to compare the efficiency. The running hardware is MacBook with an Apple M1 chip.
The results are shown in Table 13 The small increased time of RobustTSF compared to Vanilla training is due trend computation (Equation (4)). However, compared detection-imen iimiiittiitttttiinn retraining pipeline, Robus(TSF is much more efficient.
Table 13: Training time comparison of each method.
Method
Vanilla Training
Offline imputation 68.09
Online imputation
RobustTSF 36.84
Time (s)
27.41
89.33
F.9 COMPARING ROBUSTTSF WITH MORE METHODS
There exist some methods, such the recently proposed normalized Kalman filter (NKF) (de Bézenac ctt al. 2020), even not specifically designed for TSFA tasks, showing good performance when time series has missing data. Since NKF has better forecasting performance than DeepAR (Salinas et al.
20

---

Published as a conference paper at ICLR 2024
2020), K (Krishnan al. 2017), and GP-Copula (Salinas al. 2019) on missing data settings, in this section, we mainly compare RobustTSF with the NKF method The results are shown in Figure 7 which shows that RobustTSF still exhibits better performance than NKF under different ratios of missing data.
We also compare RobustTSF with ARIMA, popular non-DNN approach for time series forecasting. Specifically, we employed a rolling ARIMA model to suit one-step forward TSFA. Essentially, at each time step, we extended the noisy training time series by one step and used ARIMA to fit the new noisy training sequence, subsequently predicting the next step. However, ARIMA differs from DNN-based methods and cannot split time series. Comparing ARIMA and DNNs might introduce bias due to ARIMA's increased anomaly exposure. To mitigate this, we assume anomalies exist in both training and test. Appendix E1 provides details on applying RobustTSF to noisy test set. For ARIMA, we utilized the statsmodels package and finetuned (p.d.q) parameters (Due to ARIMA's characteristics, the validation time long). Table T6 illustrates its performance falls short of RobustTSF due to its non-specialization for TSFA.
Another approach we compare Dlinear (Zeng et al. 2023) which exhibits better performance than transformer-based models. The results are summarized in Table 14 (reporting best epoch MAE). The results demonstrate that, while DLinear may have slight advantage in specific anomaly scenarios, RobustTSF consistently outperforms DLinear, particularly in Gaussian-type anomalies.
Table 14: Comparing RobustTSF with Dlinear. Reporting best epoch MAE for each method.
Constant Anomaly
Missing Anomaly
Gaussian Anomay
Dataset
Method
11 0.1
7= 0.3
1 0.1
= 0l.3
1 0.1
0.3
Dlinear
0.176
0.189
0.179
0.198
0.195
0.347
Electricity
RohustTSF
0.177
0.183
0.181
0.194
0.176
0.177
Dlincar
0.219
0,238
0.232
0.318
0.277
0.472
Traffic
RobustTSF
0.198
0.203
0.200
0.243
0.185
0.202
There also exists some potential approaches to deal with anomalies. For examples, after detecting anomalies, we can impute the values using interpolations or using the calculated trend as new time series to train the model. Table [able[[]showw [5]shows the comparison with these methods on the Electricity dataset. The results show the superiority of RobustTSF over other methods.
Table 15: Comparing RobustTSF with interpolation- based and trend-only methods. The anomaly ratio for each anomaly type is 0.1 and 0.3, respectively. We report best epoch MAE for each method
Constans
Missing
Gaussian
Method
1 0.1
0.3
== 0.1
7= 0.3
1) 7= 0.1
1) 7= 0.3
zero- cerooiiputation
0.191
0.212
0.200
0.224
0.190
0.232
interpolate- impllattio
0.185
0.200
0.191
0.209
0.186
0.187
Trend-only
0.214
0.216
0.213
0.215
0.215
0.217
RSbustTSF 0.182
0.190
0.187
0.204
0.182
0.177
Table 16: Comparing RobustTSF with non-DNN approach ARIMA. We report best epoch MAE for each method on Electricity dataset.
Method
Gaussian Anomaly (1 = 0.1)
Gaussian Anomaly (n = 0.2)
Gaussian Anomaly (n 0.3)
ARIMA (1,0,0)
0.213
0.294
0.419
ARIMA (I,1.)
0.232
0.329
D.363
ARIMA (5,1,0)
0.230
0.327
0.405
RobustTSF
0.178
0.182
0.190
F. APPLYING ROBUSTTSF ON OTHER ANOMALY DISTRIBUTIONS
In the main paper, we consider three types of anomalies, i.e. Constant, Missing and Gaussian Anomaly. In this section, we consider more possible distributions for modeling anomalies, i.c.,
2, = 2 + c, where c follows heavy-tailed distributions as
hhtns///gthhaaccc//saaammmmmmm irreg TS
21

---

Published as a conference paper at ICLR 2024
(a) MAE comparison of each method
(b) MSE comparison of each method
0.775
0.255
0.12
MAE
MSE
0.30
0.77
1.99
0.55
NKT
NKI
RobustTSF
0.36
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.50
0.05
0.30 0.10
0:15
0.25
missing ratio
missing ratio
Figure 7: MAE and MSE comparison between RobustTSF and NKF (de Bézenac et al. 2020) on Electricity dataset with different ratios of missing data.
Student-t distribution: f(e,v) = r(
(x) (1+ E -
(1t))
Generalized Pareto distribution: f(e,c) (1 c
In the above, f denotes pdf of the distributions and v, are hyper-parameterd determining the shape of the distributions. We perform RobustTSF on these distributions and the results are shown in Table
17 and Table [8
Table 17: Comparison of different methods on Electricity dataset for single-step forecasting with anomaly followed by student-t distribution. We report the best epoch performance for all methods, represented MAE/MSF, The best results are highlighted in bold font.
Student-T T (v-
Student- T (v
Method
= 0.1
1y = 0.2
= 0.3
1) = 0.1
27 = 0.2
== 0.3
Vanilla (MAE)
0. 187/0.0771
0.205/0.084
0.224/0.098
0.184/0.068
0.210/0.086
0.216/0.092
Offline
0.18770.070
0.189/0.073
0.190/0.074
0.179/0.067
0.198/0.078
0.195/0.077
Online
0. 196/0.075
0. 1999/00074
0.217/0.089
0.19//05075
0.200/0.081
0,21970.091
Loss sel
190/0.077
0.198/0.078
0.208/0.087
0.194/0.076
0.203/0.083
0.203/0.084
RobustTSF
0.177/0.066
0.178/0.067
0.178/0.067
0.178/0.067
0.182/0.069
0.182/0.070
Table 18: Comparison of different methods on Electricity dataset for single-step forecasting with anomaly followed by generalized Pareto distribution. We report the best epoch performance for all methods, represented as MAE/MSE. The best results are highlighted in bold font.
Generalized Pareto (c 1)
Generalized Pareto (c 0.5)
Method
17 = 0.1
1= 07=0.2 0.2
=
11 = 0.1
1) = 0.2
= 0.3
Vanilla (MAE)
195//008
0.209/0.087
0.223/00997
0.203/0.080
0.207/0.08
0.220/0.094
Otfline
0.181/0.069
0.193/0.075
0.195/0.078
0.187/0.071
0.197/0.079
0.193/0.076
Online
0. 192//00.773
0.201/0.078
0.209/0.086
0.193/0.074
0.207/0.083
0.212/0.086
Loss sel
0. 195/0.007
0.204/0.084
0.205/0.085
0.191/0.074
0.19//0.076
0,208/0.086
RobustTSF
0.176/0.066
0.181/0.069
0.181/0.069
0.179/0.067
0.184/0.068
0.184/0.069
F.11 APPLYING ROBUSTTSF ON MORE DATASETS
In this section, evaluate RobustTSF on more datasets, including ETT (Zhou et al.) 2021 and Exchange (Lai et al. 2018). The results are shown Table 19, Table 20 and Table 21 Overall, RobustTSF still achieves better performance than other existing algorithms.
22

---

Published as a conference paper at ICLR 2024
Table 19: Comparison of different methods ETT and Exchange dataset. The model structure is LSTM. We report the best performance on test set, presented as MAE/MSE.
Constant Anomaly
Missing Anomaly
Gaussian Anomaly
Dataset
Method
1/= 0.1
1) 0.3
71 = 0.1
1) 0.3
== 0.1
= 0.3
Vanilla (MAE)
0.054/0.006
10.01/
0.053/0.005
0.070/0.008
0.053/0.006
0.055/0.0006
Offline
0.055/0.006
0,06/0.006
0.053/0.000
0.057/0.006
0.052/0.005
0.055/0.006
ETThh
Online
0.057/0.006
0.074/0.009
0.054/0.00 006
0,057/0.006
0.055/0.006
0.069/0.009
Loss sel
0.055/0.006
0.062/0.007
0.055/0.006
0.065/0.009
0.052/0.00
0.060/0.007
RS
0.052/0.005
0.054/0.005
0.052/0.005
0.055/0.006
0.052/0.00 0.05
0.055/0.006
Vanilla (MAE)
0.054/0.007
0.075/0.011
0.046/0.00 004
0,062/0.009
0.04 004
0.050/0.006
Offline
0.041/0.003
0.074/0.012
0.045/0.00 005
0.060/0.009
0,042/0. 004
0.048/0.004 004
ETT-h2
Online
0.057/0.005
0.115/0.019
0.053/0. 006
0,079/0.012
00000000005
0.090/0.014
Loss sel
0.042/0.003
0,080/0.014
0.047/0.00 005
0.061/0.009
0.044/0. 004
0.057/0.007
Robus(TSF
0.040/0.003
0.058/0.007
0.041/0.003
0.061/0.008
0.041/0.004
0.049/0.005
Vanilla (MAE)
0.026/0.001
0.029/0.002
0.026/0.001
0.028/0.002
0.026/0.001
0.027/0.002
Offline
0.026/0.001
0.031/0.001
0.026/0.001
0.027/0.002
0.026/0.002
0.026/0.002
ETT-ml
Online
0.038/0.002
0.048/0.004
0.030/0.002
0.034/0.00 0.10
0.026/0.00
0.027/0.001
Loss sel
0.026/0.001
0.038/0.003
0.026/0.001
0.031/0.002
0.026/0.002
0.027/0.002
RobusTTSF
0.025/0.001
0.026/0.00 0011
0.026/0.001
0.026/0.001
0.025/0. 101
0.026/0.001
Vanilla (MAE)
0.123/0.029
0.131/0.029
0.160/0.050
O. T.1400.039
0.124K0:032
0.208/0.085
Offline
0.094/0.016
0.098/0.016
0.052/0.004 004
0.110/0.023
0.064/0.000 000
0.128/0.031
Exchange
Online
0.070/0.008
0.063/0.006
0.053/0.005 005
0,074/0.010
0.054/0.005
0.333/0.200
Loss sel
0.121/0.027
0.228/0.095
0.130/0.033 030
0.229/0.101
0.141/0.06 143//0.03
0.253/0.124
RobustTSF
0.072/0.010
0.076/0.011
0.052/0.004
0.057/0.006
0.053/0.005
0.065/0.008
Table 20: Comparison of different methods on ETT and Exchange dataset. The model structure is LSTM. We report the best performance test set, presented as MAE/MSE.
Student-T (v/
Student-T T (v
6)
Dataset
Method
1) 0.1
? 0.2
= 0.3
1)= 0.1
11 0.2
= 0.3
Vanilla (MAE)
0.053/0.006
0.054/0.006
0.061/0.000
0.052/0.006
0.054X0.006
0.056/0.009
Offline
0.053/0.005
0.055/0.006
0.062/0.007
0.053/0.005
0.054/0.006
0.058/0.007
ETTh1
Online
0.056/0.006
0.062/0.007
0.065/0.008
0.058/0.007
0.060/00077
0.065/0.008
Loss sel
0.054/0.006
0.062/0.009
0.083/0.017
0,058/0.007
0.063/0.009
0.077/0.015
RSbustTSF
0.052/0.005
0.052/0.005
0.053/0.005
0.051/0.005
0.052/0.005
0.053/0.006
Vanilla (MAE)
0.04270.004
0.048/0.005
0.052/0.006
0.04570.004
0.046/0.000 0.00
7.057/0.007
Offline
0.045/0.004
0.046/0.005
0.053/0.006
0.042/0.00 004
0.055/0.00 007
0.00//0.011
ETT-22
Online
0.056/0.005
0.079/0.012
0.096/0.018
0.054/0.007
0.071/0.00 009
0.098/0.018
Loss sel
0.042/0.004
0.055/0.006
0.061/0.009
0.042/0.004
0.052/0.00 006
0.058/0.008
RobusiTSF
0.042/0.004
0.048/0.004
0.052/0.005
0.043/0.004
0.045/0.005
0.050/0.005
Vanilla (MAE)
0.026/0.002
0.026/0.002
0.027/0.002
0.026/0.002
0.026/0.000 0.0
0.027/0.002
Offline
0.026/0.002
0.026/0.002
0.026/0.00 0000
0.026/0.002
0.026/0.00 0.00
0.026/0.002
ETT-ml
Online
0.026/0.002
0.026/0.002
0.026/0.00 0002
0.026/0.002
0.026/0.00 0.00
0.026/0.002
Loss sel
0.026/0.002
0.027/0.002
0.034/0.003
0.026/0.002
0.027/0.002
0.030/0.002
RobusiTSF
0.025/0.001
0.025/0.001
0.026/0.002
0.025/0.001
0.025/0.00)
0.025/0.001
Vanilla (MAE)
0.085/0.015
0.182/0.066
0.160/0.04P
0.147/0.042
0.172/0.059
0.218/0.087
Offline
0.082/0.013
0.102/0.019
0.137/0.036
0.087/0.014
0. 0.106/0.021
0.133/0.033
Exchange
Online
0.053/0.004
0.104/0.021
0.203/0.075
0.058/0.008
0.095/0.016
0.092/0.013
Loss sel
0.174/0.063
0.198/0.082
0.254/0.1 125
0.146/0.044
0. 0.153/0.06
0.321/0.091
RobustTSF
0.046/0.003
0.051/0.004
0.081/0.012
0.060/0.006
0.061/0.007
0.083/0.014
We also conduct experiments on time series datasets with real-world anomalies. The NAB bench-
mark dataset is from http::/odds. performace of RobustTSF is superior than other methods.
oonbbrook..o...o..o edu ttt//tabblel Table 23 shows the
We provide full description of the datasets in the paper in Table 22
23

---

Published as a conference paper at ICLR 2024
Table 21: Comparison of different methods ETT and Exchange dataset. The model structure is LSTM. We report the best performance on test set, presented as MAE/MSE.
Generalized Pareto (c
1)
Generalized Pareto (c 0.5)
Dataset
Method
1) = 0.1
y= 0.2
" = 0.3
f) === 0.1
11 0.2
= 0.3
Vanilla (MAE)
0.052/0.0066
0.054/0.006
0.057/0.006
0.053/0.005
0.053/0.006
0.064/0.008
Offline
0.052/0.006
0.053/0.006
0.053/0.006
0.052/0.005
0.053/00005
0.056/0.006
ETT-h
Online
0.054/0.006
0.055/0.006
0.061/0.006
0.055/0.008 006/
0.058/0.006
0.064/0.00
Loss sel
0.052/0.006
0.058/0.006
0.065/0.008
0.054/0.006
0.058/0.008
0.059/0.007
RSb
0.052/0.005
0.053/0.005
0.055/0.006
0.052/0.005
0.052/0.000 0052/0.005
0.056/0.006
Vanilla (MAE)
0.040/0.00 004
0,046/0.005
0.054/0.006 006
0,043/0.008 004
0.049/0. 006
0.060/0.007
Offline
0.041/0.008 004
0,044/0.005
0.049/0.00 005
0,046/0.008 004
0,046/0.00 004
0.051/0.005
ETT-h2
Online
0.049/0.005
0.055/0.007
0,075/0.010
0,049/0.004
0.062/0.00 006
0.075/0.009
Loss sel
0.042/0.004
0,046/0.005
0.060/0.00 008
0,042/0.004
0.047/0.00 005
0.058/0.008
Robus(TSF
0.040/0.003
0.043/0.00
0.051/0.005
0.041/0.003
0.045/0.004
0.050/0.005
Vanilla (MAE)
0.025/0.002
0.026/0.002
0028/0.002
0.026/0.002
0.026/0.002
0.027/0.002
Offline
0.026/0.002
0.026/0.002
0.026/0.003
0.026/0.002
0.026/0.002
0.026/0.002
ETT-ml
Online
0.028/0.002
0.028/0.002
0.032/0.002
0.026/0.002
0.028/0.
0.045/0.003
Loss sel
0.026/0.002
0.026/0.002
0.026/0.002
0.026/0.002
0.026/0.002
0.029/0.002
RobusTTSF
0.026/0.001
0.026/0.00 .002
0.026/0.002
0.026/0.001
0.026/0. 101
A026/0.002
Vanilla (MAE)
0.090/0.019
0.104/0.023
0.077/0.012
0.070/0.007
0.112/0.024
0.091/0.015
Offline
0.060/0.006
0.084/0.013
0.074/0.009
0.070/0.010
0.087/0.014
0.093/0.016
Exchange
Online
0.384/0.247
0.482/0.371
0.533/0. 434
0.568/0.517
0.396/0.25 258
0.438/0.320
Loss sel
1100/0.024
0.144/0.040
0.141/0.038
O. 0.105/0.022
0.137/0.037
0.175/0.054
RobustTSF
0.060/0.007
0.064/0.008
0.071/0.009
0.044/0.003
0.064/0.008
0.078/0.012
Table 22: Description of the datasets used the paper
Dataset
LEN
FREQ
Manullay adding anomalies
Train Test ratio
Electricity
26304
Ih
Yes
7:3
Traffic
17544
Ih
Yes
7:3
ETT-h1
17420
Yes
7:3
ETT-h2
17420
Ih
Yes
7:3
ETT-ml
69680
15 min
Yes
7:3
Exchange
7588
day
Yes
7:3
NAB (real-cloud)
4032
15
Yes
7:3
M4-monthly
1871
monthly
Yes
7:3
Table 23:
Performance each method on NAB benchmark (cloud dataset) and M4 forecasting dataset. Best epoch MAE is reported for each method. It can be observed that RobustTSF also performs well on world anomaly dataset and large forecasting dataset.
Vanilla MAE
Offline
Online
Loss sel
RobustTSF
NAB (real-cloud)
1.28
1.20
1.14
2.11
0.87
M4-monthly
2,66
2,63
2.61
2.61
2.58
M4 M4-monthly (Const ano ratio 0.2)
2.95
2.85
2.76
3.21
2.61
F.12 EVALCATING ROBUSTTSF ON TRANSFORMER AND TCN MODELS
We use the transformer Vaswani al. 2017 f and temporal convolutional networks (TCN) (Bai et al.
2018 to evaluate RobustTSF in single-ster s-sppp forecasting with different anomaly types on Electricity dataset. Results are shown in Table 24 and Table 25 respectively. It can be observed that RobustTSF also improves model performance on Transformer and TCN architectures.
24

---

Published as a conference paper at ICLR 2024
Table 24: Evaluating RobustTSF on Transformer model for time series forecasting. We report the best epoch performance for the methods, represented as MAE/MSE. The anomaly ratio for each anomaly type is 0.1 and 0.3, respectively. The best results are highlighted in bold font.
Clean
Constant
Mixsing
Gaussian
Method
0
11 0.1
0.3
0.1
0.3
0.1
0.3
Vanilla (MAE)
0.266/0.139 139
0.271/0.139
0.304/0. 184
0.287/0.152
O. 3.5/
0.263/0. 141
0.302/0. 188
RSh
1,264/0.137
0.269/0.137
0.277/0.152
0.271/0.411
0.287/0.622 0.266/0.33 0,286/0.155
Table 25: Evaluating RobustTSF on TCN model for time series forecasting, We report the best epoch performance for the methods, represented MAE/MSE. The anomaly ratio for each anomaly type is 0.1 and 0.3, respectively. The best results are highlighted in bold font.
Clean n= 0
Constant
Missing
Gaussian
Method
== 0.3
77 0.1
1
0.1
7
Vanilla (MAE)
0.186/0.083
0. 0.999/0.083
0.218/0.093
0.205/0.088
0.232/0.108 0.206/0.087
0.090000733
0.21
RobustISF
0,185/0.078
0. 0.189/0.080
0.192/0.081
0.186/0.078
0.185/0.075
0.186/0.075
F.13 DISCUSSING THE ORIGINALITY, SIGNIFICANCE AND FUTURE WORK OF ROBUSTTSF
In this section, we discuss the originality, significance, and future work of RobustTSF, which collectively serve as a comprehensive conclusion to our paper.
originality: Our paper delves into the realm of Time Series Forecasting with Anomalies (TSFA), relatively underexplored area within the time series domain. Traditional approaches to TSFA often employ the Detection-Impulee -nnpuuttttoo-Retraiinnn (DIR) pipeline, which, while intuitive, lacks robust theoretical analyses and guarantees. In contrast, we draw inspiration from the advancements in Learning with Noisy Labels (LNL) and approach the TSFA task from more principled perspective. We begin by providing a rigorous statistical definition of the TSFA problem. Leveraging our analyses on loss and sample robustness, we introduce efficient filtering method called RobustTSF. This method stands in stark contrast to the DIR pipeline, delivering consistent and compelling empirical results across extensive experiments.
significance: Our analytical journey leads us to establish a notable connection between TSFA and Learning with Noisy Labels (LNL). a vibrant research direction in machine learning. Although applying LNL methods directly to TSFA is infeasible due to the regression nature of TSFA tasks and the presence of anomalies in the inputs of time series, our work demonstrates how to achieve robustness in TSFA tasks, aligning with the goals of LNL. This positions our proposed method, RobustTSF, with theoretical and empirical foundations. More significantly, underscores the potential for drawing lessons from LNL to tackle TSFA tasks in the future.
Much like the evolution of LNL over recent years, progressing from modeling asymmetric/symmmett label noise instance-depender label noise, modeling of time series anomalies and theoretical analyses can serve as a catalyst for future research. This can inspire the development of methods tailored to handle more intricate anomaly patterns in TSFA tasks.
Our primary focus centers on point-wiss wise anomalies, with limited exploration of sequence- wise anomalies through specific experiments. The theoretical analysis of sequence-wist anomalies remains somewhat elusive. Real-world sequence anomalies can be intricate, and establish- ing statistical models and theoretical robustness against these anomalies poses challenging aspect in our current work. While this topic falls outside the scope of this paper, it holds promise for future research endeavors.
Future work:
Limitation: We present the limitation of the work below:
Our method focuses on point-wise anomalies, with limited exploration of sequence-wing wise anomalies. The theoretical analysis of sequence- sisence-wiseeeee anomalies remains somewhat un- clear, as real-world sequence anomalies can be complicated. The statistical modeling and theoretical robustness against these anomalies present challenging and unexplored aspect.
25

---

Published as a conference paper at ICLR 2024
Our theoretical analysis is grounded in statistical perspective, aligned with robust learning theory in the Learning with Noisy Labels (LNL) domain. However, in cases of limited training size, potential disparity might arise between theoretical analyses and empirical findings.
From the limitation, the training size may impact the performance RobustTSS, given that the theoretical analyses of losses are from a statistical perspective. Therefore, an implicit assumption of RobustTSF is that the training size should not be very small; otherwise, applying RobustTSF may not surpass baselines. While the datasets used in the paper suggest that training size may not be significant issue, we conducted additional experiments to explore the gap between the baseline and RobustTSF on the Electricity dataset (reporting best epoch MAE for each setting) concerning the training size under a Gaussian anomaly setting (anomaly rate = 0.3). The results are presented in Table 26 which demonstrates that RobustTSF outperforms baselines when the training size of the dataset is not very small.
Table 26: Description of the datasets used the paper
fraction of original dataset
0.5%
1%
2%
10%
50%
100%
MAE
1.08
0.901
0.713
0.578
0.240
0.210
RobustTSF 1.08
0.913
0.606
0.418
0.181
0.177
F.14
EVALUATING ROBUSTTSF FOR LOW NOISE RATIOS
We conduct experiments of RobustTSF Electricity dataset with low noise ratios and report the best epoch MAE for each setting. The results are reported in Table 27
F.15 ADDITIONAL ELUCIDATION FOR EQUATION
To show that substitution of squared terms with absolute values in Equation 4 lead improvement in robustness. we use simple time series function (sine function) to visualize the result of different options. Figure g shows that absolute term makes the trend more smooth and stable which would facilitate our anomaly score calculation in Equation
F.16
ADDITIONAL ABLATION STUDIES FOR ROBUSTTSF
We consider the effect of LNL and sample selection separately and conduct the ablation studies of RobusiTSF Electricity and Traffic datasets in Table 28
F.17 PROOF OF STATISTICALLY CONSISTENT RESULTS FOR ROBUSTTSF
We reran the pivotal experiments outlined in Table 2 of our main paper, focusing on the Electricity dataset. We present the best epoch MAE as the mean (standard deviation) for the three runs of each configuration in Table 29 These results demonstrate the statistical consistency of our findings.
26

---

(a) time series with constant anomaly
(b) trend filter with square term
trend filter with absolute term
0.5
vilue
0.5
-0.5
0
100
200
300
0
100
200
300
0
100
200
300
(d) time series with missing anomaly
(e) trend fllter with square term
(t) trend filter with absolute term
vols
0.0
0.0
-0.5
-1.0
-1.0
-1.0
100
200
300
100
200
300
100
200
300
[g)
time sertes with gaussian anomaly
(h) trand fllter with square term
trand filter with absolutte term
2.0
1.0
Value
0.0
-0.5
-0.
-1.0
-1.0
-1.5
200
380
200
300
700
300
stop
stop
stop
Figure 8: Illustration of the impact of square term and absolute term in trend filter.
Table 27: Comparing RobustTSF with other methods for low noise ratios. We report best epoch MAE for each method
Constant
Missing
Gaussian
Method
7 0.01
== 0.05
1 0.01
1)
0.05
0.01
7) 0.05
MAE
0.187
0.190
0.187
0.193
0.185
0.192
MSE
0.187
0.197
0.192
0.215
0.191
0.192
Offline
0.180
0.185
0.183
0.187
0.182
0.184
Online
0.187
0.193
0.188
0.190
0. 0.188
0.190
Loss sel
0.192
0.195
0.192
0.
0.187
0.190
RobustTSF
0.177
0.177
0.178
0.179
0.177
0.177
Table 28: Ablation studies of RobustTSF. Reporting best epoch MAE for each method.
Constant Anomaly
Missing Anomaly
Gaussian Anomaly
Dataset
Method
! == 0.1
1 = 0.3
7 = 0.1
" == 0.3
1 =
" =
Only LNI. loss
0.191
0.206
0,205
0.225
0.199
0.210)
Electricity
Only sample selection
0.183
0.186
0.184
0.237
0.180
0.182
RobustTSF
0.177
0.183
0.181
0.194
0.176
0.177
Only LNI. loss
0.209
0.231
0.266
0.295
0.216
0.233
Traffic
Only sample selection
0.199
0,220
0.224
0.310
0.196
0.210
RobustTSF
0.198
0.203
0,200
0.243
0.185
0.202
Table 29: Comparing RobusiTSF with other methods. We present the best epoch MAE the mean (standard deviation) for three runs of each configuration.
Clean
Constant
Missing
Gaussian
Method
" 0
7 0.1
0.3
0.1
0.3
0.1
0.3
MSE
0.187(0.002)
0.200 (0.003)
0.234 (0.003)
0.227 (0.005)
0.309 (0.015)
0.185
0.192 (0.003)
MAE
0.183(0.002)
0.191(0.004)
0.209 (0.004)
0.207(0.003)
0.223 (0.006)
0.198 (0.003)
0.210 (0.004)
Ottine
0.184(0.003)
0.188 (0.003)
0.217 (0.004)
0.192 (0.003)
0.213 (0.004)
0.184 (0.002)
0.191 (0.003)
Online
0. 0.18470.002)
0.199 (0.003)
0.225 (0.003)
0.194 (0.003)
0.230 (0.003)
0.595 (0.003)
0.209 (0.002)
Loss sel
118000.0055
0.205(0.003)
0.209 (0.004)
0.201 (0.002)
0.215(0.003)
0.196 (0.002)
0.201 (0.003)
RoF
0.177(0.002)
0.177 (0.002)
0.184 (0.002)
0.180K0.002)
0.195(0.001)
0.175(0.001)
0.177 (0.001)
27


Reviews:
[{"review_id": "atnolv6twr", "reviewer": "Reviewer_5W9f", "summary": "This paper presents an algorithm for time series forecasting in the presence of anomalies, together with some theory and experimental results.", "questions": "1. In section 3, the fourth line, I suspect that $T-\\alpha$ should be $T/\\alpha$. Is that correct?\n2. [Yoon 2022a] and [Yoon 2022b] are the same citation.\n3. In appendix B, the lines after the first one seem to have been cut off and are hidden behind figure 2.\n4. For table 12, what computer system was used for training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper explores a bit of variability in various hyper parameters and dataset normalization---more than the typical submission. Although, as I point out in weaknesses, some more information is needed.\n2. The proofs are clearer than the typical ML paper.", "weaknesses": "1. The ranges of hyper parameters, noise scales, and stds, and other variables related to normalizing the data and running the algorithms need to be clearly identified.\n2. In table 7, and some others where noise rates are varied, show that the lowest noise rate is 0.1. That is a rather high rate. The noise rate needs to be decreased to at least 0.01 and maybe lower depending on the dataset.\n3. The characteristics of the chosen datasets need to be described. Hopefully they cover a large range of variability, relationships between input variables, fraction of dataset that have anomalies, etc.\n\nMinor weakness:\n1. \"Proposation\" -> \"Proposition.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p9Sn7vw1gW", "reviewer": "Reviewer_FUy4", "summary": "This article presents a model for forecasting time series containing anomalies. The article begins by specifying the three main types of anomalies of interest and then introduces the theorems that will be used for the proposed algorithm. These theorems are primarily proofs of the robustness of Mean Squared Error (MSE) and Mean Absolute Error (MAE) in the presence of anomalies. The proposed algorithm mainly involves filtering to detect potential anomalies and penalizing the cost for windows containing these anomalies in order to not consider them. Experiments are finally conducted to demonstrate the method's efficiency in various settings", "questions": "Can you provide experiments with the usual baseline of the state of the art ? (PatchTST, Dlinear, Autoformer, Pyformer, ...)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The state of the art is well presented, the work presented is well contextualized, and the theoretical aspects are well introduced and explained\n* The experiments are numerous, they are entirely reproducible, and the appendices contain very useful information.\n* The issue of anomalies is well introduced, formally presented correctly, and intriguingly", "weaknesses": "* The first weakness of the article is the presentation of the main algorithm's idea. Its explanation is spread across several pages, and one has to wait until the 6th page to get an idea of how the algorithm will work, even though the idea itself is rather simple: identifying anomalies in the signal and applying very weak penalties to the cost in the corresponding region. It would be much easier to read the article if even a brief description were introduced very early on to understand the relevance of the different theorems formulated.\n\n* Regarding novelty, the main theorems seem to be adapted quite directly from other works, so there isn't much novelty there. Furthermore, the idea of filtering and training on robust windows is not very new either, and the filtering method is also pre-existing. Apart from adapting the framework for anomalies to time series, there isn't a significant novelty.\n\n* Regarding Section 5, which analyzes the effect of the position of anomalies on forecasting, it seems quite normal that anomalies near the end of the signal disrupt the prediction more than anomalies at the beginning. The system's state is disrupted, making forecasting more challenging. In contrast, if the anomalies are at the beginning, the model has time to stabilize on the 'correct' values.\n\n* However, the biggest weakness of the article lies in the experiments, which do not include any modern time series forecasting baselines (such as PatchTST, DLinear, Autoformer, ...). From a few quick experiments I conducted using the authors' code, DLinear seems to easily outperform the proposed model on the noisy data. This raises the question of the method's relevance if state-of-the-art models do not struggle with the anomalies considered by the authors. Of course, I may be mistaken, but in any case, it would be helpful to have the performance of state-of-the-art algorithms on this kind of problem to understand the challenges posed by the envisaged anomalies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuRX3Gy7xW", "reviewer": "Reviewer_442h", "summary": "The paper introduces a method that combines the learning with noisy labels (LNL) technique and time series forecasting with anomalies (TSFA). The key distinction is that LNL deals with label noise in the target variable Y, while TSFA handles noise in both the input variable X and the target variable Y, making TSFA more complex than LNL. The authors categorize anomalies into three types: constant, missing, and Gaussian type anomalies. They analyze the loss robustness when anomalies affect Y and the sample robustness when anomalies affect X.\nThe authors propose a method called RobustTSF that integrates LNL and TSFA. Specifically, they provide a method for selecting time series with minimal anomalies. The process involves computing the trend of a given time series, denoted as S, and constructing a triplet D = (X, S, Y). An anomaly score A is defined as the difference between X and S, with an associated weight w. After calculating the anomaly scores for all samples, the authors design the final loss function for RobustTSF, employing the mean absolute error (MAE) as a more robust loss function. They utilize the anomaly score to measure the degree of anomalies in the time series data and perform appropriate filtering.\nImportantly, this method does not rely on any specific deep learning model and can be compatible with any DNN model for time series forecasting.", "questions": "1.In Theorem 1, why is it necessary for the anomaly rate to be less than 0.5?\n2.In your method, you mentioned three types of anomalies, and your analysis of loss robustness indicates that MAE is robust to all three types of anomalies, while MSE is only robust to one of them. Can we understand from this that MAE is more suitable as a loss function than MSE in all situations? Are there any criteria to evaluate the robustness of other loss functions?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.Originality: This article presents a novel method that integrates LNL and TSFA, representing the first application of LNL to the domain of time series prediction. This pioneering approach demonstrates significant originality and provides valuable guidance for future research directions.\n2.Quality: This article exhibits high quality as it presents a well-supported argument, effectively connecting LNL and TSFA from a new perspective, allowing readers to comprehend the rationale behind the author's design choices. Furthermore, the author validates the effectiveness of this approach through various experiments, enhancing the overall quality of the work.\n3.Clarity: The author demonstrates exceptional clarity through a meticulously organization, guiding readers step by step in understanding the algorithm. Moreover, the author employs precise terminology, minimizing ambiguity and ensuring a clear and unambiguous understanding of the concepts presented.\n4.Significance: The author's work holds great significance as they propose a novel framework for addressing anomalies in time series data, applicable to any forecasting model. This contribution provides a fresh direction for researchers in the field of time series prediction, offering an alternative approach to further enhance prediction accuracy. The implications of this work are of paramount importance to the community, as it opens up new avenues for exploration and advancement in the field.", "weaknesses": "1. I noticed that in your experimental section, you used a large input length to predict a relatively small output length. For example, in section 7.2, you used an input length of 96 to predict an output data of length [4, 8]. However, for time series forecasting tasks, the typical setting is an input length of 12 and an output length of 12. Therefore, I am unsure if your model is applicable to this setting. I recommend conducting an additional set of experiments where the input series has a length of 12 and the output series has a length of 12, in order to assess the performance of your model in this specific scenario.\n2. In Section 6.2, I propose that additional elucidation should be provided: Why does the substitution of squared terms with absolute values in Equation (4) lead to an improvement in robustness?\n3. Please note the spelling error: \"k-the\" below Equation (5) should be corrected to \"k-th.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HFu5iHbhJ", "reviewer": "Reviewer_Swrw", "summary": "The paper studies time series forecasting problem in the presence of anomalies, a setting where the difficulty comes from the fact that anomalies produce noise in the labels and in the input data at the same time. To deal with anomalies in the labels, authors propose to use Learn with Noisy Labels (LNL) framework and present analysis of LNL in the regression setting. They also introduce a heuristic approach to deal with anomalies in the input data by discarding samples where anomalies appear towards the end of an input sample. They combine both techniques in RobustTSF algorithm and compare it on two datasets with existing state-of-the-art approaches.", "questions": "* Proposition 2: the statement is not clear: does it hold under the conditions of Theorem 1? If yes, then the statement is obsolete, if not, then where does C_x comes from?  \n* Section 6.2: why notations change from z to x? It’s not clear what x is and how they are different from z\n* End of page 6: while RobustTSD does not need a pre-trained DNN, it relies on limited trend estimation precedure in eq. (4) that implicitly assumes the smooth behavior of the time series. I’d see this as a disadvantage that one will have to adapt eq. (4) in unclear way to different datasets, while DNN would adopt automatically via pre-training. \n* It’s great that authors look at scenario where the test set contains anomalies as well as this is the most realistic case. However, this sections misses a comparison of RobustTSF to other approaches on the test set with anomalies. It would be very relevant for practical applications if this evaluation is expanded.\n* Ablation study section: what is presented is not an ablation study, but rather a hyperparameter tuning on the test dataset. An ablation study would, for example, consider the effect of LNL and sample selection separately, i.e. what happens when one applies only LNL or only sample selection. \n* Parameter \\tau seems to be scale dependent and while appendix mentions that the time series are normalized, the main text omits this detail.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## Originality \n* The analysis of mean absolute and mean square errors in LNL regression setting for 3 presented types of anomalies is novel\n* The sample selection technique to avoid samples with late anomalies is novel\n## Quality \n* Authors compare their method to diverse set of baselines\n* Empirical evaluation shows the advantage of presented techniques\n## Clarity \nPresentation is clear and main ideas are explained well\n## Significance\nI assess the paper to be of limited significance: while the presented method uses novel ideas in the context of time series forecasting with anomalies, evaluation is limited to two datasets with very similar properties and the limitations of the presented approach are not discussed.", "weaknesses": "* Hyper parameter tuning of RobustTSF is missing: it’s never discussed how \\tau of 0.3 was chosen and Figure 5 suggests that choice of \\tau would affect the comparison to the baselines. \n* Overall limitations of the approach are not discussed. When does it break? What implicit assumptions are made? What time series aspect one would need to consider to choose this approach over another in practice?\n* No confidence bounds in the experimental results makes it hard to judge the significance of the evaluation results\n* Only two datasets of very similar properties are used for evaluation, which again limits the assessment of method applicability\n\nOverall, my main concern is with the evaluation - choice of hyperparameters for the method and the lack of confidence bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)