Task-oriented Time Series Imputation Evaluation via Generalized Representers
Authors: Zhixian Wang
Linxiao Yang
Liang Sun
Qingsong Wen
Yi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: time series imputation
downstream task oriented imputation evaluation
Primary area: other

Abstract: Time series analysis is widely used in many fields such as power energy, economics, and transportation, including different tasks such as forecasting, anomaly detection, classification, etc. Missing values are widely observed in these tasks, and often leading to unpredictable negative effects on existing methods, hindering their further application. In response to this situation, existing time series imputation methods mainly focus on restoring sequences based on their data characteristics, while ignoring the performance of the restored sequences in downstream tasks. Considering different requirements of downstream tasks (e.g., forecasting), this paper proposes an efficient downstream task-oriented time series imputation evaluation approach. By combining time series imputation with neural network models used for downstream tasks, the gain of different imputation strategies on downstream tasks is estimated without retraining, and the most favorable imputation value for downstream tasks is given by combining different imputation strategies according to the estimated gain.

Task-oriented Time Series Imputation Evaluation via Generalized Representers
Zhixian Wang 1.2, Linxiao Yang", Liang Sun?, Qingsong Wen', Yi Wang
The University of Hong Kong, DAMO Academy, Alibaba Group
zxwangdeee hku.hk, hk, linxiao, y www.ylllwlaaaaaaiai. qingsongedu@gmail. com, yi yiwangSeee hku.
com, liang. sun@alibaba-ing com,
Abstract
Time series analysis is widely used in many fields such power energy, economics, and transportation, including different tasks such as forecasting, anomaly detec- tion, classification, etc. Missing values are widely observed in these tasks, and often leading unpredictable negative effects on existing methods, hindering their further application. In response to this situation, existing time series imputation methods mainly focus on restoring sequences based on their data characteristics, while ignoring the performance of the restored sequences in downstream tasks. Considering different requirements of downstream tasks (e-g.. forecasting), this paper proposes efficient downstream task-oriented time series imputation evalu- ation approach. By combining time series imputation with neural network models used for downstream tasks, the gain of different imputation strategies down- stream tasks is estimated without retraining, and the most favorable imputation value for downstream tasks is given by combining different imputation strategies according to the estimated gain. The corresponding code can be found the repository htpp::////ttht www.thu..cmmhkueeeee.eoeo coom/hkued/Taast
diieeted-Imppttae
Introduction
Time series analysis plays crucial role in many real-world applications, such as energy, finance,
healthcare, and other fields [2 3). For example, forecasting load series forms the basis for further decision-making power dispatch the power grid system, thereby generating significant amount of economic benefits 4 5 6. However, collecting time series data, especially high-quality ones, is challenging. Due the instability the external environment, sensor failures, and even ethical and legal privacy issues, missing values are prevalent in time series data [7]. For instance, in the BDG2 load series dataset (8), widely used in building energy analysis, the ratio of complete time series data is less than 10%.
To handle missing values in time series data, numerous methods have been proposed for time series imputation in the literature. Based on the features of the imputation methods, these approaches can be divided into statistical and machine learning methods, such as ARIMA and KNN [0] as well 13 [4]. Both types of methods generally use reconstruction errors of missing values to guide learning and perform evaluation. Recently, some researchers have turned their attention to evaluation strategies based on downstream task performance [15. However, in most cases, downstream tasks are classification tasks [33, while forecasting tasks, as another important branch of time series-related tasks, have not been fully considered. The main challenge for
as deep barning-based methods [1] 12
time series forecasting is that the time series serves as both input and label (output) for the model
during training, whereas in classification tasks, only serves as input for the model.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In supervised learning, training labels influence the calculation of the loss function, which in turn affects the optimization of model parameters and, ultimately, the performance of the model on the test set. [16] indicates that noise in input data (missing data can be considered a type of noise) often has a limited impact on forecasting results. contrast, label noise can significantly affect the model and, consequently, the final test results from the beginning to the end of the time series. Therefore, when evaluating the impact of different time imputation methods on downstream forecasting tasks, is essential to focus assessing the quality of training labels constructed through various imputation methods.
To evaluate how the quality of the imputation labels affects downstream forecasting tasks, it is important to clarify that an excellent imputation strategy does not necessarily mean that the imputed value at each time step is superior to any other method. [15] provides a benchmark for various methods time series imputation tasks. Although SAITS [3], as one of the latest SOTA methods, has achieved remarkable results, there are still methods that surpass SAITS in some cases. This demonstrates that time series imputation is a complex task, making difficult to find a universal method capable of handling all situations, let alone considering the performance of downstream forecasting tasks. A more realistic scenario that while one method may perform better overall, it may not outperform other methods locally. In time series, this means that one method may excel in some time steps, while others do better different time steps. However, to examine the impact of each time step on forecasting, retraining the forecasting model multiple times is necessary, which is impractical due to time and computational costs. Therefore, an efficient estimation method is needed to examine the impact of each time step with different imputation methods. Additionally, since finding a universal method is difficult, it is natural to shift focus toward combining the advantages of current methods obtain better imputation strategy. Consequently, determining how to combine different strategies becomes a challenge.
Based on the above situation, we have developed a task-oriented time series imputation evaluation strategy. Specifically, we summarize our contributions into the following points.
1. We propose a strategy that evaluates each time series imputation method by estimating the impact of missing (imputed) labels at each time step downstream tasks without requiring multiple retraining, which significantly reduces time and computational consumption. To the best of our knowledge, we the first to consider the impact of missing values in time series as labels on downstream forecasting tasks.
2. We introduce a simple and effective similarity calculation method based on the characteristics of long time series estimate the impact of imputed values more quickly, striking balance between performance and computational cost.
3. We develop a time series imputation framework guided by maximizing the gains of down- stream tasks, enabling the combination of advantages from different time series imputation strategies to achieve better one. This results in improved performance in the downstream forecasting task.
1.1 Related Work
Time Series Imputation Time series imputation can be primarily classified into two categories: traditional techniques and neural network- based techniques. Traditional methods replace missing values with statistics, such as the mean value or the last observed value [17. They include simple statistical models like ARIMA 188, ARFIMA, SARIMA 19. and machine learning techniques such as KNNI 20), TIDER (21) MICE 22), BAyOTIDE 23 In recent years, deep learning imputation methods have demonstrated remarkable capabilities in capturing intricate temporal relationships and complex variation patterns inherent in time series data. These methods employ deep learning models
13 generative neural networks such as VAES [2] 25), GANs [26] 27], and diffusion models (28 to capture complex dynamic relationships within time series data. Although different methods exhibit various advantages, no universal method currently outperforms others in consider combining existing advanced methods in this work to achieve better time series imputation strategies.
like Transformers 24
all scenarios and datasets. This observation inspires
Sample-based based Explaination Sample-based explainable methods can be divided into two cate- gories 229.. One is based on retraining, which evaluates the importance of corresponding data by comparing the impact of removing data points on the model and even the final test results (30 31 32

---

Among them, the introduction of shapley value by [3] naturally ensures the faimess of data attribu- tion. The other type is based on gradient methods, which directly estimate the influence of data points without the need for retraining. This type of method can be subdivided into three main categories, which are based on representative theories, Influence Function, and training loss trajectory. [3 is representative of the method of the first type, whose core idea is to fix the other layers and only focus on the last layer of the neural network, so that the influence of each sample can be explicitly calculated. On the other hand, the Influence Function [34] is based on the assumption of convergence and uses the Hessian matrix to estimate the influence samples. The last type of method takes into account the entire training process of the neural network, continuously tracking the impact of samples on each parameter update [35]. In addition, [36 summarizes the gradient- based method and unifies them as generalized representers.
2 Methodology
2.1 Problem Statement
Consider multivariate time series dataset represented by (:))=1 incorporating n samples. In this dataset, X; E DxL corresponds to a feature matrix containing D) distinctive features over L1 temporal intervals, whereas y; E Ax signifies the target time series, which spans L2 temporal intervals. It is crucial to recognize that y; may include several missing entries, common complication within real-world datasets. For example, in the context of electrical load forecasting, X; encompasses daily weather-related time series data, comprising variables such as temperature, humidity, and wind speed, while y; represents the electrical load time series of a given day, possibly containing missing entries due to issues in data collection transmission.
Addressing missing values in {y:} through imputation is a fundamental preprocessing step for machine learning tasks involving this data, underscoring the necessity to assess the effectiveness
of various imputation methods. Consider ( and {y; (y,²)) as two time series resulting from the imputation of } via two different methods. The goal is to ascertain whether the imputation performed on { (2) } is superior to that on ( Moreover, we seek evaluate the quality of imputation at each temporal interval, determining if the imputation of the I-th interval in y; is more accurate than that in y: (1)
Conventionally, the quality of imputation is quantified by measuring the discrepancy between the imputed values and the actual data, favoring methods that minimize this deviation. In this study, however, we propose to assess imputation quality based on the performance of subsequent tasks.
One step further, we evaluate the quality of imputation timestep basis, examining if the imputation (1) for the 1-th interval in y exhibits improved efficacy over y; thereby offering a more nuanced and comprehensive evaluation of imputation methodologies.
Let us define the loss function for the downstream task as ((f(X,)) y), where f(, O) denotes the model used in the downstream task parametered by 0. And let {(X; y. y/) constitute test
dataset that will be used gauge model performance. We denote (1) and Va.i as the 1-th entries of
(2)
y (1) and y respectively. According to our intuition, if v!! is superior to U swapping vi for (2)
ya. should result in a decrease in the test set's loss. Guided by this rationale, we define the indicator
function T(i, 1), which discerns whether Vg
(2) is preferable over v as follows:
1(1,1) = Σ 11,1, 1=1
(C(((X), C(((X 02).
L(f(X 0), 9), ())
s.t. 0 = arg min
0
0 arg min ((f 0). Ds
(201) Σ O).
(1)
0
3

---

Here, D, (2 denotes a vector identical to y except at the I-th entry, which matches that of (2) (1) Vil leads to a decreased test set loss,
Clearly, [(i. 2 implies that the substitution of Vi,1 with
(2)
(1) suggesting that Vi, is superior. Conversely, if [(i, )) < O, it suggests that Vi.i is preferable to Vi,1
Despite the effectiveness of the definition provided by Equation (1), computing ( for every missing value in the dataset is impractical due the extensive model retraining required, which can be prohibitive in terms of time. To overcome this challenge, in the next section, we put forth an efficient methodology for estimating [(i, () without retraining the model.
2.2 Approximation Model Construction
To compute I(i, () efficiently, we propose a retrain-free method in this subsection. As both y and (1) (2) are imputation of then we assume that Vi,7 is close to Vi with which we approximate [ii, using the first order Tolyer expansion
(2)
OC(f 0). vk) m
T(i,1) R
(ULl
(1)
(2)
Oy:.
(1)
(C(S(X) k: 0), 0f(X k. 0)
0f(x) 0)
(3
Vi.
(2)
Oyt
Equation provides an approximation for computing I(i, l), where 0 measures how the
training target Vs,t affect the prediction of the test data, and PC(1(X),0) computes how the 0f(x700
changing of the prediction of X affect the final loss. Note that the symbolic expression 0f(X;.0) By.a
can be conceptually broken down into 0f 8e elucidating the role of the label ye. in shaping 0e Ty..
the model parameters & throughout the training process. This, in turn, has repercussions on the ky 0). By focusing on the derivative @ our goal is to assess the extent which changes in label values y/,1 influence dyg.t
model's prediction when evaluated on unseen data from the test set, i.e. f (x
the model's predictions on the test set, thereby affecting overall model efficacy.
When it comes back to the estimation, dispite (o(((EE,))..t and Vi - Vill are easy to compute, ;,,))
estimating 0f(X is difficult. The difficulty comes from two aspects. Firstly, for the complex Dy..
f(; 0). the final parameter is not only affected by the training data, some other factors, such as the structure of the network and learning rate during the learning process. Secondly, all of the training samples affect the parameters of the model, leading to the mixture of the effect of data points on the final model. Thus isolating the effect of a single data point difficult.
To overcome these two difficulties, we propose to approximate @s(X;.0)
89.. using white-box model, where how each training datapoint affects the final prediction clear from the design of the model. (f(X;.0) using kernel machine, i.c. a,,KK K (X; X k)) where K (X,,-) is a kenerl between the training sample X; and test samples measuring the similarity between the X; and X and is learnerable hyperparameter. It can be proven that the indicator function based on this definition satisfies many desirable properties (please see Appendix for details) to construct axiomatic attribution. Formally, the coefficient ai, I can be computed by solving the following optimization problem:
To this end, we propose to approximate By.,
La ΣΣΣS
0f(X,
â argmin
L
a (X,,
aER"XIL' x342
dy:.
To solve this problem,
f(x,0,))5(x,.. can act as a substitute of 85(X, 0y.1
since there is no ground
-
truth. However, the problem is still not practical to solve because we can not obtain s(X,, 02) without retraining the model. Even though it isn't necessary to traverse all and to obtain the

---

complete data, it still goes against our original intention to calculate [ () efficiently. Furthermore, also difficult for us determine how much data is sufficient ensure the accuracy of the solution. Fortunately, with the help of Remark [ we can bypass the process of finding enough and directly approximate 0flXx,01
Vid -
Oyi,
Remark 1. Given two infinitely differentiable functions fæ) and g(x: in a bounded domain D E R", []f(x) g(>)ll is always less than c. For any given 5 and cg. there exists such that, in the not greater than e2. >
domain D, the measure of the region that satisfying @f(m)) Bg(m) iee m(1) S E2.
du
Remark gives us the intuition that approximation. However, some issues still need to clarified. First, we consider neural networks used in the downstream task as infinitely differentiable functions since in practical applications, it is unlikely for computed flotting-poin numbers to precisely equal non-differentiable points. Second, Remark limits the definition domain to bounded region D. Time series data is usually bounded (for example, the renewable generation sequence cannot be greater than the installed capacity), making this assumption reasonable. Finally, Remark] can be rephrased the better the approximation of
avoid retraining downstream models by first-order
the original function, the better the approximation of its derivative, that is, we can have a (DD) <a m(D)
probability of fitting the derivative well. Therefore, the optimization problem (3) can be transformed into easier one. (Note that indicate the existence of that meets the conditions in this remark, but make no restrictions on e, while it often difficult for us to make the approximation error of the original function sufficiently small. If the e the remark is infinitely close will encounter difficulties. However, in gradient descent-basse neural network training, such situations often do not hinder our practical applications. Due the space limit, the full theoretical discussion is provided in the Appendix.)
practical applications
á' argmin a e'ER" xB
Σ
Σ a
K (x., ((x,xx)))))((.00
(4)
1=1
0á's
0g,1 Oy..
(5)
Now the problem is converted to solving the problem the RKHS subspace spanned by the kernels,
4 Intuitively, we solve it by projecting it onto
La α Kid
à' argmin a x31
C
K X x(),
+
(6)
2
fx((;)
|| RK
where K; is the kernel gram matrix defined as Ki,j K (X X,),. Considering the first-order
optimality condition, d'
36
37 Recalling our goal of estimating the
(X,)
relationship between â' and y. in (5) their relationship is still unclear. This because the objective of the optimization problem to construct approximation model without considering the role of label value yr,1- To clarify the effect of ys.t. we introduce an approximation by trigonometric inequality that L (fx (X,) (X, (), < Cyy (X,, 0)) LfK (X:) ,,). The second term on the right side is the loss function corresponding to the downstream model, and in the case of training convergence, this should be close constant. Therefore, the optimization problem (6) can be rewritten as
n ( (X., C(k (X.),
La
à' argmin af a'ER"X2
Σ
a Kid's
(7)
[l 1Rx
5

---

Formally, the solution are a; ac ((x( and ب = o (fx((xx)..)) Furthermore, 0fx ())09..
xx,
since fx(-) is used to approximate f(., 0), we use f(- 0) to replace fx( to simplify the calculation. With the above preparation, the [(i, 1) can be represented as follow with NTK kernel (3X.
m
LCC((XX) 0),9%) (C 0) vi)
(x 0) ( 0)²
(8)
n a 0) 0, Bys,
00
d0
NT Kernnel
ai,1
2.3 Similarity Calculation Acceleration
In the previous section, to gauge the effects of substituting Vir with Vill on the downstream task, we utilized the Neural Tangent Kernel assess the similarity between the model outputs for inputs X; and X Given that the model's output length is L2, the computational complexity of calculating [(i, 1) for all time steps in y scales as O(mL2P), where P denotes the total number of parameters in the model f(- 0). i.c., I0). In numerous time series applications, such as forecasting, L2 can be substantially large (e.g., 128), rendering the evaluation process for all imputations time- consuming. To mitigate this challenge, we propose a method to compress the size of (f(x.,)) Our approach is inspired by the observation that in time series forecasting, the output of J(: typically exhibits smooth variability across different I values. Therefore, we posit that the model output ((x,, ;; 0) resides in low-dimensional space spanned by limited number of smooth basis functions. In mathematical terms. f(x;: ;, 0) = A ( 0) where pr consists of rows each representing a predefined smooth vector, AT Ra is the pseudo-inverse of A, and r, which is significantly smaller than L2, denotes the number of basis functions employed to approximate
(1)
(2)
29
f(x,, s, 0). Consequently, can approximate OS(x,0)
08
0f (X 0)
DAf OX 0)
(9)
00
00
This approximation allows for the compression of the model output, thereby reducing the number of gradients that require computation. Through this simplification, the computational complexity for calculating (ii, () decreases to (mrP), substantially less than the original complexity.
In experiments, we further simplify by assuming A to be block diagonal matrix, defined as blkdiag(11, .2...., 1c). where 1, ... 1c-1 are vectors of length /2/c] with all elements equal to 1, and 1. E (e-1)(a-) is a vector with all elements equal to 1.
2.4 Task-oriented Imputation Evaluation
(1) We have introduced a method for computing the indicator T(i, I), which assesses if replacing Vi.! with vi,
(2) results in reduced loss for the downstream task. Given two sets of imputation results, (v, y and (y; (2) y derived from distinct imputation techniques, we can evaluate T(i, 1) across all samples and time steps, and identify that y( outperforms y; (1) at time step if T(i, 1) greater than zero and vice versa if the value is lesser. In contrast conventional evaluation strategies, our proposed method does not necessitate the availability of the ground truth y, thereby enhancing its practical utility in myriad real-world scenarios where actual values remain unobtainable. This feature renders our approach significantly more adaptable to situations where empirical truths are elusive.
2.5 Task-oriented Imputation Emsemble
Given that our proposed method can evaluate the quality of two imputations at the time step level, a natural extension is to combine these two sets of imputations to derive an improved result. Specifically. we can generate a new set imputations yy for the i-th sample, where the 1-th entry is v.r if [ii,t) > O, and yi. otherwise. Based on the definition of [ii, we anticipate that model trained using y will incur lower loss compared to one trained with y (1) thereby yielding better imputation results. It is important to note, however, that the calculation of 1(i, L) is predicated

---

on the scenario where only the /th timestep in the ith sample from { is substituted with Vi. This consideration omits the potential interactions among samples. Consequently, in practical implementations, we opt to substitute only the timesteps that rank within the top e% of I(i, () values. Our experiments. detailed in 3.3.2 confirm the efficacy of our proposed task-oriented imputation ensemble method. Our proposed ensemble process is summarized Algorithm
Algorithm 1: Task-oriented Imputation Emsemble.
Data: Training data ((X,, y, hn and {(X,, e: (2) ) Jm: Estimated gain on Validation data
{I(i. := Downstream task model f(,, 0); Replacing percentage c%
Calculate threshold = (i (i) > 0})
2 for i = to n, 1= 1 to I2 do
if 2( threshold then
4
Let Vl,1 = V.
(2)
5
else
Let v/.1- Vi,l
(1)
end
end
9
Train ((,0) on ((X 1,2))nn Result: f):
3 Experiment
3.1 Datasets and Experiment Setup
To validate our method, conduct experiments six datasets: the GEF load forecasting competition dataset with the corresponding temperature (39), the UCI dataset (electricity load and air quality) (40). the Traffic dataset containing road occupancy rates?] P and two transformer datasets, ETTHI and ETTH2 [1 Note that we use the hourly resolution version the UCI electricity dataset from 42 in the main experiment. In our main experiment, we set the downstream task as univariate time series forecasting, with both input sequence and prediction lengths set to 24. In addition to the GEF dataset, we implement our method on the 'OT' time series in ETHI and ETH2, the mean value of the road occupancy rate in Traffic, the temperature in the UCI air quality dataset, and the total electricity consumption of 321 users in the UCI electricity dataset. original missing values in these datasets. To simulate the missing values situation, we randomly set masks with lengths in [2. 4. 6. 12, 24, 48, 96, 120], and replace the original values with the average value at the corresponding positions as the baseline. For the missing rate setting, if the missing rate is too low, the difference between different imputation methods may be small, while the missing rate is too high, the even best imputation method will also be difficult to obtain reasonable results. Based on the above considerations, we mainly consider 40% missing rates as our main experimental setup. Meanwhile, we will provide experimental results under other missing rate settings in [30%, 50%, 60%] in the Appendix.
important to note that there are no
3.2 Time Series Imputation Methods
To verify the performance of our strategy in evaluating different time imputation methods, we introduce multiple advanced time imputation methods. Firstly, as mentioned in the last subsection, we use the basic mean value imputation the baseline. Secondly, we consider several time series imputation methods based on deep neural networks. They are GPVAE (25) and USGAN (27) based on generative neural network, mRNN 43) and BRITS [IT] based on RNN structure, SAITS [3]. and ImputeFormer [44] based on attention mechanism. All the implementations of the above models are with the help of the toolkit called PyPOTS 45). In addition. we also implement the network structure of spatiotemporal assisted method called SPIN [46] for time series imputation. All the details of the neural network setting will be described in the Appendix.
dot co.

---

3.3 Experimental Results
3.3.1
Application Estimate the Gain
MLP
DLinear
0.8
0.8
0.6
Corr
Seq sim
0.4
INF
0.4
Seg
0.2
Seg
Seg-1
0.2
95%
95%
90%
Accuracy
90%
85%
80%
75%
85%
70%
65%
80%
60%
55%
75%
50%
10%
15%
20%
1%
5%
10%
15%
20%
Percentage
Percentage
Figure 1: The correlation and accuracy comparison between the estimation of imputation value gain and actual gain (MSEL), where INF (section (D55 represents our modified Influence Function, Seq-sim represents our original method, and Seg-N represents the acceleration method divided by N segments. The horizontal axis here represents selecting the sample with the highest x% influence based on the absolute value of the estimation.
In this section, examine the estimated gains of imputation for each time step. We divide the GEF dataset into a training set and a test set, where the training set includes load data from 2011 and the test includes load data from 2012. We replace 40% of the load data in the training set with the average value at the corresponding position as the baseline, apply the linear interpolation method to replace the corresponding baseline, and them as training labels separately. In the training set, have obtained a total of 8760 training samples. Therefore, we replace the labels of each sample one 8760 sets of samples, retrain 8760 forecasting models, test their performance
by to construct on the test set, and compare the performance of the new model with the model trained on the mean value-based samples. Note that although our estimation is done at each time step of each sample, the time consumption of replacing each time step and retraining is too high. Therefore, we replace all the time steps in each sample and sum up the benefit estimates for each time step as the benefit estimate for the entire sample after replacement. We adopt the 3-yayrr MLP structure used in [35] and extend be used for time series. In addition, have also added simple and widely used forecasting model, DLinear
47 as our forecasting backbone model.
Table Time comparison between different methods.
Time
Soq-sim
Sog-4
Sog-2
Seg-1
Retraining
MLP Diiinar
46s
18s
17s
TIs
248h
8s
8s
>48h
As shown in Figure I we use two metrics compare the performance of different methods, where Corr represents the correlation between estimated gain and actual gain, and accuracy represents the percentage of same sign between actual gain and estimated gain. the MLP model, the Influence Function achieves good correlation. However, in terms of accuracy, the accuracy of the Influence Function rapidly decreases as the percentage estimated data to total data increases. On the contrary, method exhibits good characteristics at different percentages, In addition, the

---

segmented acceleration method generally shows that the more segments there are, the better the performance. Although it performs poorly in small percentages, it exhibits good performance in a wide range of data. In the DLiner model, both our method and the segmented acceleration method present similar situations to those in the ML.P model. However, the Influence Function exhibits poor performance, with estimation accuracy and correlation significantly lagging behind our method. In addition, Table [ repoorts the total time required for different methods to run on the 4TX4090, indicating that retraining requires a significant amount of time while our method achieves a good balance between performance and time consumption. It is worth mentioning that when we focus not only the small portion of time steps with the greatest impact but also examine the impact of relatively large time steps on the forecasting results, our segmented acceleration method achieves very good performance while reducing the required time.
3.3.2 Application 2: Combine Different Time Series Imputation
Similar to the previous section, we use mean value as baseline to estimate the benefits on the validation set (not test set) obtained by imputation at each time step and replace the 10% time step with the highest benefits to train the forecasting model. In addition, the missing values in the time series damage the original characteristic of the time series. Although imputation values can repair it to some extent, they may still have adverse effects on downstream tasks. Therefore, we also introduce the Influence Function as a comparison, using it to estimate the 10% points with the worst impact on the forecasting results after imputation and remove them, which is often the case in the application of Influence Function
34
Table 2 reports the comparison of MSE for downstream forecasting tasks (here we use the DLinear as the backbone model), and we will analyze from two aspects.
I. Comparison between Original and Gain Estimation. Part I and II in Table 2 demonstrate that combining different imputations can enhance the performance of downstream tasks on all datasets. The improvements in the GEF dataset, Electricity dataset, and Traffic dataset are significant, while the enhancement in others is relatively less noticeable. The primary reason is that, in those datasets, the performance of other imputation methods considerably lags behind the original one, making the replacement of the original label with a newly imputed label seem less impactful. However, after incorporating our estimation, there is still slight improvement in forecasting performance. On the other hand, when considering combining two imputation methods with similar original performance, incorporating our method will bring significant gains (as seen in the GEF datasets). It's worth noting that the mean value used as baseline outperforms other imputation methods in most cases, primarily because the training data input is also based on mean value imputation, facilitating unified and convenient comparisons. In practical applications, we can also apply other advanced imputation methods to the input data and modify the labels based on the estimated benefits.
II. Comparison between Gain Estimation and Influence Function. Part I, II. and III Table 2 in- dicate that discarding a certain number of samples according to the Influence Function can indeed improve the performance of forecasting; however, such improvement is not universal. In the AIR dataset, discarding some data can negatively impact the performance of most methods. This may be due to the small amount of data contained in the AIR dataset, resulting in greater adverse effect when discarding data. The operation of discarding data can only consider one imputation method, while our method can combine any two imputation methods to achieve better results. Furthermore, the results of repeated experiments show that the strategy of modifying values at specific time steps can make performance more stable, as its variance is significantly smaller than that of discarding data.
In addition to the univariate input results displayed in Table 2 we also include the results of multivariate inputs, which are common in practical applications. For instance, when predicting power loads, temperature is a crucial external variable. A large amount of research has focused on
studying the relationship between load and temperature [48 49 In this experimental setting, unlike multivariate forecasting, temperature plays an auxiliary role in load forecasting while there is no need to forecast temperature itself. Consequently, we conduct experiments on the GEF dataset, inputting temperature an external variable into the model to forecast loads. Table presentss the performance of our model under multivariate input, which is consistent with the univariate input scenario, and incorporating our method proves to be beneficial.

---

Table 2: MSE, in the downstream forecasting task with univariate input, every experiment is done times.
Method
Datasets
GEF
ETTHI
ETTH2
ELECTRICITY
TRAFFIC
AIR
LOriginal
Mean
0,1750
0,0523
0.1797
0,1123
0,4359
0,0
SAITS
0,198000 0009)
0 0.2098(0.0125) 0.1027)
0.1176(0.01 10)
0.5006(0.022 0,431100.0151)
I)
BRITS
0.2021 (O.00)7
0.1642(0.0105)
0.2384(0.0018)
0.150330.00033
0.6979(0.0086) 0.4535(0.0001)
MRNN
0.2052(0.0000) (0.001))
0.2184(0.0016)
0.2317(0.0001)
0.44000.00000)
0.7965(0.0018)
GPVAE
0.208700.0019)
0. 11991 (0.0072)
0.2365(0.0022)
0. (0.000))
0.4465(0.0001)
0.6968(0 0044)
USGAN
0.2048(0.0023)
0.1549/0.0179
0.2238(0.0085)
0.1447(0.0011)
0,4742(0.00488
0,684000.00 (306))
SPIN
0.21200.00029)
0.200000.05099)
0.2414(0.0327)
0.1588K0.01 13)
0.460910.01488)
0.6604(0.0802)
ImputeFormer
0.1820000016)
0. 0533(0.0033)
0.2125(0.0022)
0.1076(0.0012)
0.4249(0.0060)
0.6900/0.01199
II With Gain estimation
Mean+SAIIS
0.0522/0.00000 0.165300000008
0.1797(0.0000
0.095700.0000
0.4147(0.0233)
0.1491(0.0001)
Mcan+BRITS
0.1694(0.0000)
0.0522(0.0000)
(..7950.0000)
0. 1068000000))
0,4318(0.0000)
0.1507(0.0000
Mcan+MRNN
0.1696(U.D00X)
0.052330.0000)
0.1794(0.0000)
0.4319(0.00000)
0,150S(0. (0000)
Meun-GPVAE
0,1696(0..00)) (000))
0.0522(0.00 (.000)
(..090))00000)
(058(0.0001)
0,4290(0.0000)
0.1507
Mcan+USGAN
0. (698(000001)
0.0522(0. (0000)
(..795(0.0000)
O. 1069(00.000) 0003)
0.4215(0.0004)
0,150 (000)
Mcan+SPIN
0.167900.0016)
0.0523000001)
0.1784(0.0000)
0.103800.000 0007)
0.4276000013)
0.1502(0.0005
Mcan+Imputed Impulcboreee
01667(0.0003)
0.0522(0.0000)
(0.795(0.0000)
0.0077(0.0002)
0.4178(0.0015)
0.1498(0.0000)
With Influence Function
SATIS+INP
0.0053
O. (.16600000E1)
0.20 0.207((000115)
0.4294(000139) 0.0
0.52070.0213)
BRITS+INE
0.2326(0.0005) 1.1637(0.0091) 0.1952(0.00099
0.1302(0.0022)
0.4419(0.0008)
0.7110(0.0069)
MRNN-INF
0,1972(0.0002 0002)
0. 1905(0.0017) 7)
(..250(0.0003)
(.4001(0.0002)
0.7758(0.002))
GPVAE-INE
0.2013(0.0018)
154330.0073)
0.2314(0.0031)
(.227))00022)
0,4347(0.0005)
0.7096(0.00211
USGAN-INF
(.1984(0.00124)
0. (48)().0120)
0.1263(0.0013 (Q.000)))
(.55970000045)
(..961().019))
SPIN
0,210620.0433 0.2195(0.0046)
0.2551
O. 1133 (0.0224)
0.4728(0.0194)
0.7629 1007)
ImputeFormer-INF
0.1776(0.0009)
0.1461(0.0013)
0.2085(0.0020)
0.103340.0070)
0.4197(0.0058)
0.6498(0.0046)
Table 3: MSEJ in the downstream forecasting task with multivariate input, every experiment is done
3 times.
Method
Mean
SAITS
BRITS
MRNN
GPVAE
USGAN
Formerrrrerr
Original +Ours
0.1845
0.1848(0.0022)
0.2153(0.0136)
0.239210.0321)
0.203810.0006
0.1895(0.0004)
0.182200.011)
0.1748000002) 0.178800.0000 0.179700.0005) 0.180000.0000 0.17800000044 0.1747(0.001))
4 Conclusion and Future Work
In this work, we propose to evaluate the imputation values at each time step for the impact on downstream forecasting tasks. On the one hand, our method can accurately estimate the gain of each imputation value without retraining. On the other hand, our method can also combine different time series imputation strategies based on the estimation of gain to obtain better imputation for downstream tasks. To ensure the applicability of this method practical scenarios, we also provide an accelerated calculation method. In the future, we will focus on further downstream tasks, such as optimization tasks based on prediction values, and build an end-to-end evaluation strategy.
5
Acknowledgement
The work was supported in part by the National Key R&D Program of China (2022YF00141000) in part by the Research Grants Council of the Hong Kong SAR (HKU 27203723), and in part by the Alibaba Group through Alibaba Research Intern Program.
References
[1] M. Jin, H. Y. Koh, Q. Wen, D. Zambon, C. Alippi. G. I. Webb, I. King, and S. Pan, "A survey on graph neural networks for time series: Forecasting, classification, imputation. and anomaly detection," IEEE Transactions on Patlern Analysis and Machine Intelligence, 2024.
[2] Y. Liang, H. Wen, Y. Nie, Y. Jiang, M. Jin, D. Song, S. Pan, and Q Wen, "Foundation models for time series analysis: A tutorial and survey." in Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2024, pp. 6555-6565.
10

---

[3] J. D. Hamilton, Time series analysis. Princeton university press, 2020.
[4] D. Qin, C. Wang, Q. Wen, W. Chen, L. Sun, and Y. Wang, "Personalized federated darts for electricity load forecasting of individual buildings," IEEE Transactions on Smart Grid, vol. 14, 6, pp. 4888 4901, 2023.
[5] Wang, L. Han, X. Zhang, Y. Wang, and S. Zhang, "Electrical load forecasting based on variable t-distribution and dual attention mechanism," Energy, vol. 283, 128569, 2023.
[6] M. Grabner, Y. Wang. Q. Wen, B. Blažič, and V. Struc, "A global modeling framework for load forecasting in distribution networks," IEEE Transactions on Smart Grid, vol. 14, no. 6, pp. 4927-4941, 2023.
[7] W. Du, J. Wang. L. Qian, Y. Yang, F. Liu, Z. Wang, Z. Ibrahim, H. Liu, Z. Zhao, Y. Zhou et al., "Tsi-bench: Benchmarking time series imputation," arXiv preprint arXiv:2406. 12444 2024.
[8] C. Miller, A. Kathirgamanathan, B. Picchetti, Arjuman, J. Y. Park. Z. Nagy Raftery, B. W.
Hobson, Z. Shi, and F. Meggers, "The building data genome project 2, energy meter data from the ASHRAE great energy predictor III competition," Scientific Data, vol. 7, p. 368, Oct. 2020.
[9] M. Steinbach and P.-N. Tan, "knn: k-nearest neighbors," in The top ten algorithms in data mining. Chapman and Hall/CRC, 2009, pp. 165-176.
[10] R. H. Shumway, D. Stoffer, R. H. Shumway, and S. Stoffer, "Arima models," Time series analysis and its applications: with R examples, pp. 75-163, 2017.
[1]] W. Cao, D. Wang. J. Li, H. Zhou, L. Li, and Y. Li, "Brits: Bidirectional recurrent imputation for time series," Advances neural information processing systems, vol. 31, 2018.
[12] L. Pinheiro Cinelli, M. Araújo Marins, E. A. Barros da Silva, and S. Lima Netto, "Variational autoencoder," Variational Methods for Machine Learning with Applications to Deep Networks. Springer, 2021, pp. 111-149.
[13] W. Du, D. Côté, and Y. Liu, "Saits: Self-atiention--assed imputation for time series," Expert Systems with Applications, vol. 219, p. 119619, 2023.
[14] J. Cheng, C. Yang. W. Cai, Y. Liang, Q. Wen, and Y. Wu, "NuwaTS: Mending every incomplete time series," arXiv preprint arXiv:2405.15317. 2024.
[15] Wang, W. Du, W. Cao, Zhang, W. Wang, Y. Liang, and Q. Wen, "Deep learning for multivariate time series imputation: A survey," arXiv preprint arXiv:2402.04059 2024.
[16] H. Cheng. Q. Wen, Y. Liu, and L. Sun, "RobustTSF: Towards theory and design of robust time series forecasting with anomalies," in The Twelfth International Conference on Learning Representations (ICLR), 2024.
[17] M. Amiri and R. Jensen, "Missing data imputation using fuzzy-rough methods," Neurocomput- ing, vol. 205, pp. 152-164, 2016.
[18] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time series analysis: forecasting and control. John Wiley & Sons, 2015.
[19] C. Hamzaçebi, "Improving artificial neural networks' performance in seasonal time series forecasting," Information Sciences, vol. 178, no. 23, pp. 1550-4559) 2008
[20] N. Altman, "An introduction to kernel and nearest-neighhor nonparametric regression," " The American Statistician, vol. 46, no. 3. pp. 175-185, 1992.
[21] S. Liu, X. Li, G. Cong, Y. Chen, and Y. Jiang, "Multivariate time-series imputation with disentangled temporal representations"" in The Eleventh International Conference on Learning Representations, 2022.
[22] S. Van Buuren and K. Proothuis-Oudshoom "mice: Multivariate imputation by chained equations in r," Journal of statistical software, vol. 45, pp. 1-67, 2011.
11

---

[23] S. Fang, Q. Wen, Y. Luo, S. Zhe. and L. Sun, "Bayotide: Bayesian online multivariate time series imputation with functional decomposition." " in Forty-first International Conference on Machine Learning (ICML), 2024.
[24] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[25] V. Fortuin, D. Baranchuk, G. Rätsch, and S. Mandt, "Gp-vae: Deep probabilistic time series imputation." in International conference on artificial intelligence and statisties. PMLR, 2020,
pp. 1651-1661.
[26] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," Communications of the ACM, vol. 63, no. 11, pp. 139-144, 2020.
[27] X. Miao, Wu, Wang. Y. Gao, X. Mao, and Yin, "Generative semi-supervised learning for multivariate time series imputation." in Proceedings of the AAAI conference artificial intelligence, vol. 35, no. 10, 2021, pp. 8983-8991.
[28] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances in Neural Information Processing Systems, vol. 33, pp. 6840-6851, 2020.
[29] Z. Hammoudeh and D. Lowd, "Training data influence analysis and estimation: A survey," Machine Learning, pp. 1-53, 2024.
[30] A. Ghorbani and J. Zou, "Data shapley: Equitable valuation of data for machine learning," in International conference on machine learning. PMLR, 2019, pp. 2242-2251.
[31] T. Wang and R. Jia, "Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity," arXiv preprint 2rXhv:2205.15466, 2022.
[32] Y. Kwon and J. Zou, "Beta shapley: a unified and noise-reduced data valuation framework for machine learning," arXiv preprint arXiv:21 14049, 2021.
[33] C.-K. Yeh, Kim, I. E.-H. Yen, and P. K. Ravikumar, "Representer point selection for explaining deep neural networks," Advances in neural information processing systems, vol. 31, 2018.
[34] W. Koh and P. Liang, "Understanding black-box predictions via influence functions." in International conference on machine learning. PMLR, 2017, pp. 1885-1894.
[35] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan, "Estimating training data influence by tracing gradient descent," Advances in Neural Information Processing Systems, vol. 33, pp. 19 920- 930, 2020.
[36] C.-P. Tsai, C.-K. Yeh, and Ravikumar, "Sample based explanations via generalized represen- ters," Advances in Neural Information Processing Systems, vol. 36, 2024.
[37] B. Schölkopf, R. Herbrich, and J. Smola, "A generalized representer theorem," in Interna- tional conference on computational learning theory. Springer, 2001, pp. 416 426..
[38] A. Jacot, Gabriel, and C. Hongler, "Neural tangent kernel: Convergence and generalization in neural networks," Advances in neural information processing systems, vol. 31, 2018.
[39] T. Hong, P. Pinson, and S. Fan, "Global energy forecasting competition 2012," pp. 357-363, 2014.
[40] D. Dua, C. Graff et al., "Uci machine learning repository," 2017.
[41] H. Zhou, Zhang, J. Peng, S. Zhang, J. Li, Xiong, and W. Zhang, "Informer: Beyond efficient transformer for long sequence time-series forecasting." in The Thirty-Fisth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, vol. 35, no. 12. AAAI Press, 2021, pp. 11 106-11 115.
12

---

[42] G. Lai, W.-C. Chang, Y. Yang, and H. Liu, "Modeling long-and short-term temporal patterns with deep neural networks." " in The s1st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 95-104.
[43] J. Yoon, W. R. Zame, and M. van der Schaar, "Estimating missing data in temporal data streams using multi-directional recurrent neural networks," IEEE Transactions on Biomedical Engineering, vol. 66, no. 5, pp. 1477-1490, 2018.
[44] T. Nie, G. Qin, W. Ma, Y. Mei, and J. Sun, "Imputeformer: Low mankness-induced transform- ers for generalizable spatiotemporal imputation." in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 2260-2271.
[45] W. Du, "PyPOTS: Python toolbox for data mining on Partially-Observed Time Series," 2023. [Online]. Available: https://arxiv.orr/as//2535....1
[46] I. Marisca, A. Cini, and C. Alippi, "Leearnng to reconstruct missing data from spatiotemporal graphs with sparse observations," Advances in Neural Information Processing Systems, vol. 35, pp. 32 069-369922002 082, 2022.
[47] A. Zeng, M. Chen, L. Zhang, and Q. Xu, "Are transformers effective for time series forecasting?" in Proceedings of the AAAI conference on artificial intelligence, vol. 37, no. 9, 2023, pp. 11 121-11 128.
[48] L.. Yang, R. Ren, X. Gu, and L. Sun, "Interactive generalized additive model and its applica- tions in electric load forecasting," in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 5393-5403.
[49] Xie and T. Hong. "Temperature scenario generation for probabilistic load forecasting," " IEEE Transuctions Smart Grid, vol. 9, no. 3, pp. 1680-1687,200
[50] E. Winter, "The shapley value," Handbook of game theory with economic applications, vol. 3, pp. 2025-2054, 2002.
[51] L. Bottou, "Large-scale machine learning with stochastic gradient descent," in Proceedings of COMPSTAT"2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers. Springer, 2010, pp. 177-186.
[52] D. Kingma, "Adam: a method for stochastic optimization," in Int Conf Learn Represent, 2014.
[53] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga al., "Pytorch: An imperative style, segh-performanc deep learning library," Advances in neural information processing systems, vol. 32, 2019.
13

---

A Why We Need Task-oriented Imputation
Here toy example illustrate that, in some cases, not directly the accuracy of imputation instead of downstream tasks to evaluate the imputation method. We want to point out that better imputation accuracy does not always mean better forecasting performance, and simulate dataset based on the GEF dataset to illustrate this viewpoint, experimenting with a predicted length of 24. Suppose that we only observed the value the time step nk (k20) and nk+1 (k> 1), just for the convenience of linear interpolation. In the first case(represented by D), we Set = 4, fill the missing value with linear interpolation, and uniformly add Gaussian noise N (0.00,,0.3). In the second case(represented by II), we set n = and only do the linear interpolation (shown in Figure 2 We put two data into MLP and calculated the forecasting error shown in the following Table
Simulated data visualization
true
2.5
-X- Case
Value
0.5
0.0
0.5
10
15
20
Time
Figure 2: Visualization of simulated data
Table 4: Imputation and Forecasting accuracy on simulated dataset
MSEJ
Imputation
Forecasting
O. 1039
0.1140
0.0576
0.1395
B
Axioms
36), there are several axioms desirable for a fair attribution. this section, we modify them to version suitable for our task and demonstrate that our method satisfies such properties. Here we use Ii,, Xx) to represent the impact the perturbation of the 1-th step the i-th sample on sample X
Definition (Efficiency Axiom). For any model f(,00) and test point X an indication function T( satisfies the efficiency axiom iff:
L2
1(,,1)
CC - - 02) v())
L (f (X () 0) 1) k)
s.t. 01 arg min k=
0, arg min
L ( (X 0) ,1
14

---

This is a counterpart of the efficiency axioms in Shapley values [50], and our method is naturally satisfying.
Definition (Self-Explanation Axiom). An indication function T(.. :..) satisfies the self-explanation axiom iff there exists any training point X; having effect on itself, i.e. T(i, X;) 0, the training point should not impact any other points, i.. [íi, 1, X) for all X.
Similar to the dummy axiom in the Shapley values, method naturally satisfies.
Definition 3 (Symmetric Zero Axiom). An indication function I(....)) satisfies the symmetric zero axiom iff any two training points X, X, such that if [(i, X;) # 0 and I(j. X,) * 0, then
I x)) + Xi),,
(10)
This situation holds if and only if K (X, Xj) = O, therefore, our method satisfies.
Definition 4 (Symmetric Cycle Axiom). An indication function I(,,,,) satisfies the symmetric cycle axiom iff for any set of training points Xiy.... X with possible duplicates, and Xt.11 X holds that:
k II (4 (46, X..)) i=1
II (
(11)
The original definition of (() does not satisfy the above properties, goal is to estimate
impact of label perturbations on the loss. However, when removing 8C[F(X, (.)... from the definition, our method satisfies this property.
0f(X..0)
Definition 5 (Continuity Axiom). An indication function [(s satisfies the continuity axiom iff it continuous wrt the test data point X, for any fixed training point X;
lim I(j x') I(j,,, x
(12)
Definition (Irreducibility Axiom). An indication function ((,',-) satisfies the irreducibility axiom iff for any number of training points X , Xs.
711,,X1 I(,,,, X²) X12,;x
I(1,,, Xx)
I(2,,Xy)
del
2 0.
(13)
I(k,,,X))
I,,,,,
I(k, Xx)
sufficient condition for an attribution A(-) satisfy the irreducibility axiom is for
[T x,)) Σ ,(,,,x,))
(14)
When selecting the NTK kernel, this property naturally satisfies.
C Discussion on Remark 1
Remark 1. Given two infinitely differentiable functions fxx and g(x) in a bounded domain D E R (I - g(x)]l is always less than e. For any given 6 and there exists an such that, in the domain D, the measure of the region! 1 that satisfying @f(()) 8g(x) ie, m(I)= & 62-
8 is not greater than ez.
Correctness. Firstly, can relax the restrictions on the function by requiring that each dimension of the function on R. be continuous. Then, we can simplify the problem into a one-di mie-immenioonal case on R. Secondly, transformed into proving that for any given and 62, there exists an such that, in the interval [a, b), the measure of the region satisfying / (x:) > 8 is less than e2.
let h(x) fz) g(x) (note that [h(z) < c), and then our problem be
Let the domain represent the set all that satisfy [b)()) > We first need to prove that can be rewritten as the union of several disjoint intervals 1, that satisfy
15

---

11, Vz E 1,,(2) > or h(x) --
Since (x)) itself is continuous function, the division is obvious. The problem is that we need to prove the number of I is countable.
Assuming 1, is uncountable, we perform bisection on [a, bj to obtain numbers of smaller intervals like [a (1)), a + ( ) If 4 is not countable, then matter how large n is, we can always find two points and in the interval that satisfy h' h'(x) 8, h' h'(y) 8 (here mainly consider greater cases, the smaller case is the same), then
5/6 & [h(z) - h(y)) & [k(z)] + [b(y)] & 2e,
and this equation does not hold as e + O, which makes contradiction. Therefore, we can rewrite I as U 4. To simplify, assume that can be written in the form of finite number of unions I U 4. If the remark does not hold, we have max m(1)) k. then Nk 2 62. Obviously,
[ok] < 2e, then k < 1/ Finally, we have
NK2 E2,
which makes contradiction as + O. If the number of 4; is infinite, we can always find that is big enough to satisfy Nk 2 C2 and the rest is the same.
Discussion on Application. As shown in the analysis above, for some given and cz, we only require cad that |c] |zc y| and |e| < 2NV Even though the real boundary is highly related the specific scenarios and is difficult to tell in real applications, we can have a look at some widely used examples. Here, we mainly discuss two kinds of optimizers that are widely used in neural network training and
they are SGD 51 and Adam [22
For the loss function L to be set MSE. Note that
X and Xtest to represent the input data in the training set and test set, separately. is the training label and yi is the value of a time step in it. We use represent the training epoch and there will be T epochs total. Our goal is to use (Xtest Vi,1) to approximate f f(ttes, Or, 11,1) first and then use (r 09..1
Ogl XisscsMa.) to approximate 8yaa
During the training process of forecasting model, we let he(,))) represent (C(F(X.0....)).)) and 09c
8 ( will be zero. Recalling that our goal is to approximate the gradient 0f(X 0r.sta) by 8yh,1
approximating (Xtest, 0r, Vi.1) first and then take the gradient. However, the function that we really approximate is 801 since Of(Xsert v(.)) Of(Xsest. O 0r..1.1) 80r and Or is dys,
0yt
80r
Dyc.
00r
constant for given Xtest-
For SGD (as well as its variants SGD with momentum), 0, = 0o Σt 1yy, (yr.). Therefore, the 02 0, which means that the gradient of the function want to approximate is constant. In this 0y?;
case, our approximation will be pretty good.
For Adam, on the one hand, (35] has claimed that the first-order approximation in the SGD situation remains valid, as long as a small step-size is used in the update. On the other hand, let 0r =
8o -E (he (Vr,)), where represent the terms in Adam. In this situation, aer will be an 8.H,1
algebraic function with only finite number of monotonic intervals. Therefore, for any given d and e, the 62 will not be really high since the Vi, that makes 1e( 0g(X only appear near the local extreme point, whose number is finite.
> can 0y..
By.t
D Implementation Details
Table summariize the dataset partitioning we used. Except for the GEF data, the rest are multivariate datasets. We forecast the 'OT' sequences in ETHl and ETH2, well the combined electricity
16

---

consumption of all users in ELECTRITY, the average occupancy rate of all roads in TRAFFIC, and the temperature series in the AIR dataset.
D.1
Datasets for forecasting
Table 5: Datasets used in the forecasting task.
Datsst
Validaios
GRF
3-1--01-01 01-00-20440140 00000
2014-01-01 00:00-201 4014-19-11 01:00
2014-09-01
01:00- 5-01-01
00:00
ETTHI
2016-07-01 0000-2007-47-11 00000
201 7-07-01
00:00-2008-----56600 01.00
201 5-18--3--66
01:00-200
8-06-26
19:00
ETTH2
2016-07-01 00000-201--0701 0000
201 7-07-01 10:00-2016-01-2000 01:00
201 5-00-26
01:00-
6-06-26 19:00
EIBTTRICITY
2012-01-01 02000-201--08-11 00000
201 3-88-01 10:00-2014-06--00 01:00
2014-06-26 01:00-2000400000000000 201-0001------ 12-11 23:00
TRAFFIC
2016-07-01 120002207-12-31 25.00
2017-12-31 23:00--2018-04-01 81.00
2015-34-01
01:00-201 8-07-02
01.00
AIR
2004-03- 1800-2009-11201 00000
2004-12-01 00:00-2005-0--01 00:00
2005-02-00
00:00-200--0440 13:00
D.2 Implementation of time series forecasting model
We include two models in our experiment. The first one is a 3-layer MLP in which the input size and output size are both 24 while the hidden size is 128. In addition, we mainly apply the simple and bigh-performance DLiner with default setting in 47 as our forecasting model backbone. In addition, to adapt to situations where the input and output dimensions are different, we constructed an output layer at the end of the DLiner model, mapping the output of multiple variables to the output of a specified dimension. We use the torch.SGD optimizer (53 to optimize the parameters of the model, where the learning rate is set to 0.1. The maximum epochs for each training are 300, and the patience is set 10.
D.3 Implementation of time series imputation model
We have introduced a total of five imputation methods for comparison, and all experiments were based on PyPOTS 45) except for SPIN. The hyperparameters for each method are shown in Table 6 In addition, referring to (13), we set the total training epoch to 100 and the patience to 10, while other hyperparameters are the default setting.
Table Hyperparameters of the time series imputation.
Method
Hyperpammeton.
SAITS
n. layleers-22, d. model-64, d fin-32, n, heads-4, 1-166 d y- 16, dropont-0.1
BRITS
MRNN
mm sioco=6
GPVAR
Intent, size-64
USGAN
mu bhikknnssissaaaae
size=61
ImputeFormer lalayers - 2dd inpurtembeed embed -64, Jeamableeennaee emhed d. proj 32.d f2n - 64 aa temporra heads -
D.4
Implementation of our estimation
In section 2.2
we gives the estimation of I(i, as follows,
OF (X 0),):) ac(f(X) 0), vi) 0f 0) 0f 0)T
(15)
n Of (X,,, 0.) dy..
0f 0) 00
00
NT Kkernel
However, depending on the solution to the optimization problem 77). we may have different forms of estimation for T(i,1). l). Referring to 336 35) when we no longer only consider the downstream model parameters 0 at the moment of training convergence but also consider the entire training process, we can obtain another form of solution the optimization problem that
17

---

t (x( where t, B(0, and 17()) here represent the & th
di,1 = - -
BEE
| 0
epoch, the batch size, and the learning rate, respectively. And the [(i,)) in this situation will be
r PC( (X, 0) ys)
ac(f(Xi k. 0). 0f (X,,0) .). 0f(X) k. 0)
Σ
B
0f4-1) (X;, 0.) 1yy.
Of(X k:
00
00
BC)
0=0
(
L (f(-1) (X 0) ys)
0f((-1) (X 0)¹ Oc(f(X),
by 0). y
Σ
B(O 0 (X;, 0y.,
00
00
BETT)EBB
0=0(t)
(16)
Compared the original estimation, repeated calculations will bring significant computational burden. However, based on our acceleration method, the time required for such multiple calculations still controlled within reasonable range. In application, we calculate for each parameter updates in each epoch.
D.5
Implementation Influence Function
In section 3.3. we compared the performance of our method with the Influence Function that we modified. Below, we will describe how to modify the Influence Function to our task.
For a training point (X, v). define V1.5 [y).... 11 + 6, ..... :Vr.g). Consider the perturba- tion y Vt. and let Or,s del eL (f(X,)) 91,,,)) Then we have
arg minege E L (J( (), eL (J(X, 0),
d0
= -11 (VoL(f(XX)) VoL(F(X,0),))
(17)
de
c==0
2 -11 V VoL(f(X, 0).
(18)
Therefore, [l,, X teat) - VeL (f(X test: 0), H V VeL(f(X, 6). 8. Note that
(e
we applied the Conjugate gradients mentioned in 34 to accelerate its computation and compare with our methods.
D.6 Hareware usage
We use NVIDIA GTX 4090 GPU with 24GB of memory for all our experiments.
E
Potential Social Impact
Our estimation may not be 100% accurate compared to the actual situation, so is possible to introduce bias in the evaluation among different imputation strategies, which may further have adverse effects on downstream tasks.
F
Supplementary Experimental Results
F1
Acceleration method
F.1.1
Performance of the acceleration method
In our practice, we mainly examine the benefits of modifying each time step on downstream tasks. Therefore, we mainly focus on whether the gain estimation is positive or negative without providing precise values. Based on this idea, we provide methods for accelerating calculations Section 2.3 Here, we present comparison between the accelerated estimate and the original estimate. Note that we conduct this experiment on three datasets and they are GEF, ELECTRICITY. and generated time series, denoted by Brown, based the following Python code.
18

---

from fbm import FBM FBM (2-288,, hurst=0.74 75, 1ength=1, method='daviesharre
The result is shown in the Figure 3 the main text.
Note that here we also use correlation and accuracy mentioned in
GEF
GEF
1.00
100%
98%
96%
Accuracy
94%
Carr
92%
0.90
Seg
90%
Seq-8
86%
Seg
84%
Seg 4
82%
0.80
80%
ELE
ELE
1.00
100%
98%
0.95
96%
Accuracy
94%
92%
90%
88%
0.85
84%
82%
0.80
80%
Brown
rowwn
1.00
100%
98%
0.95
96%
Accuracy
94%
Cer
92%
0.90
88%
0.85
86%
82%
0.80
80%
1% 5% 10%115%20%
50%
1%65% 10%115%20%
50%
Percentage
Percentage
Figure 3: The correlation and accuracy comparison between the estimation of our original method and the acceleration method. F.1.2 MSE comparison on downstream forecasting task
Table MSEL in the downstream forecasting task.
Method
EH
Datasets
ELECTRICITY
TRAFFIC
LOnginal
0.0523
0.1797
0.1123
0.4399
SAITS
0.1980(0.0092)
.022(000022))
0.2098/0.0125)
0.1176(0.010))
0.43110000551
0.5006/0.0251)
BRITS
0.2021
0. (692((00100))
0.2784(0.0018)
0. (000000000033)
0.45350000001)
0.6979(0.0088
MRNN
0.2052(0.9001)
0.2184/0.0016)
0.2317/0.0001)
0,454000.0000
0.7965(0.0018))
GPVAR
0.208700.0119)
0. (0.007))
0.2365(0.0022)
0.1471(0.0001)
0.446500.000)
0.6968(0.0044
USGAN
0.2048/0.00023
0. 15499).0119)
0.2238(0.0085)
0.1447(0.0011)
0.4742(0 (0.088)))))))
0.6840X0.0305)
SPIN
0.212000.0029)
0.200000.0509)
0.2414(0.0327)
0.158)
0,4609/0 01008)
0.6604(0.8002)
ImputeFomser
0.1820000016)
0.155800.0053
0.2125(0.0022)
0.1076(0.0012)
0.42690000060)
0.63000000 19)
1.Wih Sog-4 Gain estimation
Mean-SAITS
0.1666(0.00077
0.0522/0.0000)
0.1 11796/0.00000
0.0922000006)
0.4182(0.00222
0.149IN.0000
Mean-BRITS
0. 0.1704/0.0000
(.02220000000
0. 1.79550.00000
0.1078(0.0000)
0.4352(0.0000))
0.1 (07((00000))
Mean-MRNN
0.17070.00000
(0000)
(795((0.000))
0,4333/0 (000)
0.1508(0.0000
Mean-GPVAN
(17080000000)
0.0522(0.0000)
(7955(00000)
O. (0090000000)
0 0.004)
0.1 (.15070.0000)
Mcan-USGAN
0.1 (70040000001
0.0522(0.0000)
0.1 0.1795(0.0000
0.1076O0.0
0,4251
0,150600 0000))
MeNn
0. 0.163/00.011 3)
0.0523(0.0001)
0.18000000033
0.104700.0001)
0.4302(0 (00002000000000
0.1 0.1502/0.0000
Mean-Inguttoomer
0.1666(0.0002)
0.0522(0.0000)
0.1794(0.0000)
(.099100.0002)
(-4203(0.0014)
O. 0.0000)0000
III. With Scp-2 Gain estimation
Mean
0. 16866000005)
0.0522(0.0000)
0.1799(0.0001
0,421200.00177 (.0000000000))
0.1 (1.0000))
0.1724/0.0000)
0.0522(0.0000)
0.1795(0.0000)
0.1105(0.0000)
0.4355ID (000)
0.15070000000
Mean-MRNN
0.17300000000
0.0523(0.0000)
0.1795(0.0000)
0.435660 (435660.0000)
0.1508(0.00 0000)
Mcan-GPVAE
0.1730(0.0000)
0.0522/0.0000)
0.1795(0.0000)
0.1097(0.0001
0.433510 0003)
0.1507/0.0000
Mcan-USGAN
(11724000001)
0.0522(0.0000)
0.1795(0.000)
0.1098/0.000 0003)
0,429000 0001)
(560000000)
Mcan+SPIN
0.0033)0.0008)
0.0523/0.0000 D0U0)
0.1803/0.0004
(.1093(0.0003)
0.434340.0001
0.150340.0005
Mean--nguttFomer
0.168((00004)
0.0523/0.0011
0.1795(0.000):
0.102100.0002)
0,423110.0000
0.049)(00000)
Table 7 summarizes the performance of our acceleration methods Seg-4 and Seg-2 in downstream forecasting tasks. Overall, performance gradually improves as the number of segmented segments
19

---

gradually increases. In addition, even if it is only divided into two segments, our method can still bring some gain with minimal additional computational burden.
F.1.3 A larger dataset
For large-scale data, we applied our method to a 15 15-minute resolution UCI electricity dataset (with approximately 100000 training points) and we adjusted our experimental setup to input 96 points and output 96 points, and here is the result.
Table 8: MSE comparison larger dataset.
ELETRICITY
Mean 0,249
SATTS 0.307
+IF
+ours
+ours seg 16
+OUrS Seg
8
MSFL
0.248
0.238
0.236 264.27
0.237
Time(s)
126.33
1053.28
118.53
F.2
Additional missing rate
In addition the 40% missing rate in the main experiment, we also conduct several experiments in the ELECTRICITY dataset with missing rates in [30%, 50%, 60%].
Table 9: MSE + comparison on different missing rate.
ELETRICITY STRATEGY
Mean
SATTS
BRITS
GPVAR
USGAN
ImputeFormer
(.005(0.0010))
0.1083(0.0016)
0.007))0.0002
0.0045(0:0006
0.0063)0
30%
+OUrS
0.079%0.0088)
0.0855(0.0000)
0.0853/0.0000) (.0051)).0002)
0.0814(0.000)
50%
0.1410
(.168)(0.0320)
0.2082(0.0010)
0.2044(0.0021)
(.207100.0099)
0.1347(0.0010)
0.1126(0.0026)
(1313/0.0000)
0,1289(0.0005)
0,1319(0.0009) 0009)
0,0)52(0.0002
0.1938
(.2541(0.0350)
0,309400. 0000)
0.303210.0166 0.3057(0.0066)
0.1889(0.001)
60%
0.1351(0.0020)
0.1724(0.0001)
0.1691(0.0015)
0.1741(0.0016)
0.1431(0.0011)
F.3
Combination with robust time series forecasting
In addition to our solution, some kinds of methods, such as robust time series forecasting, to deal with missing (anomaly) values have been proposed these days. Here we combine our method with [16], which is one of the SOTA of such kind of method to illustrate that this kind of method is not contradictory to our approach but can be combined. Note that the hyperparameters are the same as the original paper in [16] and we replace the dataset with ours.
Table 10: MSE + comparison on the ELECTRICITY dataset combining our method and RobustTSF.
MSEL
Mean
SAITS
BRITS
USGAN
GPVAE
SPIN
Impute Former
TSF
0.056
0.050
0.092
0.084
0.099
0.053
0.076
+Robus!TSF+oom
0.046
0.050
0.052
0.046
D.048
0.051
F.4 Illustration on multivariate forecasting
We conduct our method multivariate forecasting tasks and give small example of the ELEC- TRICITY dataset. Note that we apply our method the first three users (columns) in the dataset.
Table 11: MSE comparison on multivariate forecasting.
MSEL
BRITS
USGAN
ImputeFormer
Original
0.1776
0.2252
0,3404
0.3404
U.3355
0.223]
+Ours
0.1497
0.1704
0.1717
0.1704
0.1470
F55
Forecasting results
and 9 show the visualization of the forecasting performance of the model. We mainly compared the combination of the SAITS method and the baseline model with the original SATIS method, and it can be seen that combining the two imputations will bring benefits to the forecasting.
Figures 4 5
20

---

-0.5
-1.0
Value
-1.5
-2.0
-2.5
true
saits
salts mean
10
20
30
40
50
60
70
Time
Figure 4: Visualization of forecasting result AIR
0.5
0.0
Value
-0.5
true
salts
-1.5
Salts mean
20
60
70
Time
Figure 5: Visualization forecasting result on ELECTRICITY
saits
saits +mean
2
1
-1
-2
10
50
60
Time
Figure 6: Visualization of forecasting result on Traffic
21

---

0.5
0.0
-0.5
Value
-1.0
-1.5
true
saits
-2.0
salts mean
20
40
50
60
70
Time
Figure 7: Visualization of forecasting result on GEF
true
-0.6
salts
salts mean
-0.8
Value
-1.2
-1.4
-1.6
10
20
40
50
60
70
Figure 8: Visualization of forecasting result on ETTHI
1.25
1.00
0.75
0.50
0.00
saits
Time
Figure Visualization of forecasting result on ETTH2
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have summarized our contributions both in the abstract section and at the end of the introduction. 'The claims made are supported by the results in the paper's experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We cover some limitations of this work in our conclusion section (the part left for future work).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification:
See Section 2 and Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide details about our experiment settings in the experiment section to ensure reproducibility as well as the link the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provide the public accessible link to both the code and dataset we used for experiment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policyylllyyyooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiieeeeep hutpi///guidessoeeeuuiiiii SroodeSubiissi.e ssiiiioicccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We cover all these information in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes),
Justification:
Every experiment is done 3 times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
We cover these information in our appendix sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We confirm that our research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of time series forecasting are provided in the Introduction section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited all the models used for the experiment and ensured the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets..
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "34ya4ijQTq", "reviewer": "Reviewer_aC85", "summary": "This paper focuses on the imputation of missing values in time series. By noticing that different imputation methods might affect the downstream forecasting tasks, this paper proposes the imputation evaluation approach regarding the downstream tasks' performance. \nThen, the authors also developed some methods to improve the efficiency of the evaluation approach and did experiments against some baselines.", "questions": "See above.", "limitations": "No potential negative societal impact is found in this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposed an evaluation approach to time series imputation methods from the perspective of the impact on the downstream forecasting task. This is a seemingly interesting perspective. \n\n2. The proposed method has mathematically formulated the intuition of evaluating imputation methods based on the prediction performance, and the authors developed the approximate method to improve the efficiency.", "weaknesses": "1. The evaluation criteria defined in (1) are based on binary comparison and do not look realistic. In real-world applications, the difference in prediction errors between the two methods can be marginal and insignificant, and calculating this binary indicator makes little sense. Meanwhile, in real-world data, the percent of missing values would be very low, and it is doubtful that imputed missing values would lead to significant differences in downstream predictions, especially considering that there are various techniques to regularize models for more robust predictions.  \n\n2. Some mathematical presentation is confusing. For instance, how is Eq.(2) derived?  $f(X_k^v, \\theta)$ is irrelevant to $y$, and thus why is there the term of the derivative of  $f(X_k^v, \\theta)$ w.r.t. $y$ in Eq.(2)\n\n   According to line 148,  Eq.(3) is for approximating $\\frac{\\partial f(X_k^v, \\theta)}{y_{i, l}}$, while on line 156, it presents the substitute of $\\frac{\\partial f(X_k^v, \\theta)}{y_{i, l}}$ again. This is very confusing. \n\n3. The experiments have unjustified and confusing setups and results. For instance, the description between lines 259 and 267 is very unclear, e.g., what is the label in a time series prediction task? why specifically replace 40% of data? \nIn Fig. 1, what is the percentage w.r.t. on the x-axis?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and we will address each of the concerns raised by the reviewer as outlined below.\n>Weakness 1\n\nFirstly, we would like to clarify the importance and necessity of handling missing values even though there is just a small difference. The phenomenon of missing time series is widely present in fields such as power systems[1] and healthcare[2] and the missing rate might not be low in the real world. For example, the missing rate in PhysioNet2012[2] might be as high as 80%. The missing values in the power system can affect downstream tasks such as load forecasting and power dispatch, where a small deviation is enough to cause huge economic losses. In healthcare, such deviations are closely related to the lives of patients. With a responsible attitude towards people's lives and property, we believe that even minor deviations are worth paying attention to, not limited to these areas. \n\nSecondly, we highly appreciate the value of the robust forecasting methods mentioned. However, these methods are not contradictory to our approach but can be combined. We applied our method to the latest related work RobustTSF[3], and the results on the ELE dataset are shown in the table below.\n\n|       MSE       |  Mean | SAITS | BRITS | USGAN | GPVAE | ImputerFormer |  SPIN |\n|:---------------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-----:|\n|    +RobustTSF   | 0.056 | 0.050 | 0.092 | 0.084 | 0.099 |     0.053     | 0.076 |\n| +RobustTSF+ours |   -   | **0.046** | 0.050 | 0.052 |**0.046**  |     0.048     | 0.051 |\n\nFrom the results, it can be seen that our method can help further improve the accuracy of robust forecasting.\n\n>Weakness 2\n\nWe are very sorry for the misunderstanding caused. \n\n1. The symbolic expression $\\frac{\\partial f\\left(X_k^v, \\theta\\right)}{\\partial y_{i,l}}$ can be conceptually broken down into $\\frac{\\partial f\\left(X_k^v, \\theta\\right)}{\\partial \\theta} \\cdot \\frac{\\partial \\theta}{\\partial y_{i,l}}$, elucidating the role of the label $y_{i,l}$ in shaping the model parameters $\\theta$ throughout the training process. This, in turn, has repercussions on the model's prediction when evaluated on unseen data from the test set, i.e. $f\\left(X_k^v, \\theta\\right)$. By focusing on the derivative $\\frac{\\partial f\\left(X_k^v, \\theta\\right)}{\\partial y_{i,l}}$, our goal is to assess the extent to which changes in label values $y_{i,l}$ influence the model's predictions on the test set, thereby affecting overall model efficacy.\n\n2. Your understanding of Eq.(3) as an approximation for the partial derivative $\\frac{\\partial f\\left(X_k^v, \\theta\\right)}{\\partial y_{i, l}}$, where $X_k^v$ represents samples from the test set, is correct. When we aim to approximate the derivatives for the test set using Eq.(3), we essentially leverage information obtained from the training set. This involves utilizing the derivatives $\\frac{\\partial f\\left(X_j, \\boldsymbol{\\theta}\\right)}{\\partial y_{i, l}}$ calculated at various points $X_j$ within the training set to construct or \"fit\" a function that can then estimate these derivatives for any given point in the test set.\n\nAs for other content that may cause misunderstandings, we will carefully review and revise it in the revised version of the paper.\n\n>Weakness 3\n\nWe are very sorry for the misunderstanding. \n\n1. We detailed the experimental setup in section 3.1 of our paper. The label represents the values of the next 24 data points following the current time point. Given that our data is recorded at an hourly resolution, this allows us to predict the time series for the full 24 hours ahead.\n\n2. The missing rate of 40% is a widely used experimental setting[4]. Due to time and computational resource limitations (mainly from the imputation method itself rather than our method), in the following, we only provide the experimental results with a missing rate of 50%. The results again show the superiority of our method. We will include more results with various missing rates in our revised paper.\n\n|       MSE in ELE(50% missing)       |  Mean | SAITS | BRITS | USGAN | GPVAE | ImputerFormer |  \n|:---------------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|\n|      -    | 0.141 | 0.168 | 0.208 | 0.207| 0.204 |     0.134     | \n| +ours |   -   | **0.112** | 0.131 | 0.131 |0.128  |     0.115     | \n\n3. For the x-axis in Fig. 1, we described it in the caption of Fig. 1 in the original paper, the meaning here is that we selected the time step with the highest absolute value in gain estimation, which is the top x% time step that impacts most after the imputation, and compared the accuracy and correlation between our method's estimation and the actual gain. This kind of evaluation is widely used in related work [5].\n\n[1] Park J Y, Nagy Z, Raftery P, et al. The Building Data Genome Project 2, energy meter data from the ASHRAE Great Energy Predictor III competition[J].\n\n[2] Silva I, Moody G, Scott D J, et al. Predicting in-hospital mortality of ICU patients: The physionet/computing in cardiology challenge 2012[C]//2012 computing in cardiology. IEEE, 2012: 245-248.\n\n[3] Cheng H, Wen Q, Liu Y, et al. RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies[C]//The Twelfth International Conference on Learning Representations.\n\n[4] Du W, Côté D, Liu Y. Saits: Self-attention-based imputation for time series[J]. Expert Systems with Applications, 2023, 219: 119619.\n\n[5] Koh P W, Liang P. Understanding black-box predictions via influence functions[C]//International conference on machine learning. PMLR, 2017: 1885-1894.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YQ5y8RCisv", "reviewer": "Reviewer_wGbj", "summary": "This paper presents a novel strategy to evaluate time series imputation methods based on their impact on downstream tasks, without requiring multiple model retrainings. The proposed method leverages a similarity calculation to estimate the effect of imputed values efficiently, balancing performance and computational cost. Furthermore, the authors introduce a framework that combines various imputation strategies to enhance downstream task performance, demonstrating significant improvements in forecasting accuracy.", "questions": "1. Is the proposed method a post-processing evaluation or an in-processing method during model training?\n2. Is the method only comparing one pair of models at a time?\n3. What is the approximation error of the indicator function and how does this impact the imputation performance?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Unlike traditional methods that focus solely on data restoration, this paper shifts the evaluation criterion to the performance of imputed data in subsequent forecasting tasks. This innovative perspective addresses a significant gap in the field and offers a more practical assessment of imputation quality.\n\n2. The research is supported by thorough theoretical analysis and experiments on multiple benchmark datasets. The authors provide clear justifications for their approach and present detailed comparisons with existing methods.", "weaknesses": "1. Although the paper introduces a retrain-free method to estimate the impact of missing labels, it does not thoroughly discuss the computational efficiency of this approach compared to traditional methods, particularly for large-scale datasets.\n\n2. The motivation of this paper is not very clear: why considering the impact of missing values as labels in downstream forecasting tasks is a valuable formulation, and why is this the optimal way to combine existing advanced methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and we have addressed each of the concerns raised by the reviewer as outlined below.\n\n>Weakness 1\n\nTo our knowledge, we are the first to examine the impact of missing values as training labels on downstream forecasting tasks, so there is no absolutely fitting traditional method. In section 3.3.1 and Appendix C.4, we modified the Influence function(IF) [1] and made it applicable to our tasks. Theoretically, the complexity of our method is O (pL), where p is the number of network parameters and L is the predicted length. If further optimization is not considered, the complexity of IF will be O ($p ^ 3$+pL). If the L-BFGS algorithm is used to approximate the inverse of the Hessian matrix [1], the complexity of IF will also be O (pL). However, with the accelerated calculation method in Section 2.3 using the properties of time series, our approach can achieve less computation time than IF and outperforms it.\n\nFor large-scale data, we applied our method to a 15-minute resolution UCI electricity dataset (with approximately 100000 training points). To make the experiment even more complex, we adjusted our experimental setup to input 96 points and output 96 points, and here is the result.\n\n|   ELE   |  Mean | SAITS | Mean+SAITS+IF | Mean+SAITS+ours | Mean+SAITS+ours_seg16 | Mean+SAITS+ours_seg8 |\n|:-------:|:-----:|:-----:|:-------------:|:---------------:|:---------------------:|:--------------------:|\n|   MSE   | 0.249 | 0.307 |     0.248     |      0.238      |       **0.236**       |         0.237        |\n| Time(s) |   -   |   -   |     126.33    |     1053.28     |         264.27        |      **118.53**      |\n\nIt can be seen that the computational complexity is acceptable in practice.\n\n>Weakness 2\n\nWe first need to point out that imputation tasks are usually not the goal, but often run as a preprocessing part, and the processed time series needs to be applied to various types of downstream tasks. Evaluating imputation strategies based on the performance of downstream tasks has received increasing attention[1] and the time series forecasting task is an important component of time series-related tasks. Although this may not necessarily be the best solution, establishing an end-to-end method for filling data applied to time series forecasting is valuable. In addition, we want to point out that better imputation accuracy does not always mean better forecasting performance. We simulate a dataset based on the GEF dataset to illustrate this viewpoint and conduct an experiment with a predicted length of 24. We suppose that we only observed the value at the time step nk(k$\\geq$0) and nk+1(k$\\geq$1). Note that this setting is for the convenience of linear interpolation. In the first case(represented by I), we set n = 4, fill the missing value with linear interpolation, and uniformly add Gaussian noise $\\mathcal{N}$(0.05,0.3). In the second case(represented by II), we set n=6 and only do the linear interpolation (visualization can be seen in the pdf uploaded). We put two data sets into MLP and calculated the forecasting error as shown in the following table.\n\n| MSE | Imputation | Forecasting |\n|:---:|:----------:|:-----------:|\n|  I  | 0.1039 |   **0.1140**  |\n|  II |  **0.0576** |  0.1395 |\n\nThis result indicates that there is no causal relationship between imputation accuracy and forecasting accuracy. When we focus on the impact on downstream forecasting tasks, directly focusing on the accuracy of downstream tasks would be a better choice.\n\n>Question 1\n\nOur method can calculate both after training and during training (note that it is the model in the downstream forecasting model rather than the imputation model). The difference between them is whether we examine the parameters at every moment during the entire training process or only focus on the parameters after the training is completed. In our experiment, we performed calculations synchronously during the training of downstream forecasting models.\n\n>Question 2\n\nYes, our method mainly involves pairwise comparisons. In fact, from Tables 2 and 3 in the original paper, it can be seen that a simple pairwise comparison can achieve better results than the original (imputation) model.\n\n>Question 3\n\nWe can analyze the approximation error from two parts.\n\nFirstly, in Section 2, we first approximated the effect\nof the label $y_{i,l}$ with a first-order term. This is reasonable for the MSE loss function since the second-order term doesn't affect the model parameters and thus the test set performance($\\frac {\\partial^2\\frac{\\partial \\mathcal{L}(f_\\theta(x),y)}{\\partial \\theta }}{\\partial y_{i,l}^2} = 0$). Furthermore, Lemma1 shows that with a sufficiently small fitting error, the derivatives' fitting error can also be small with any probability $p$. Notably, in practice, we only need to determine if the imputation is beneficial at each step, focusing on positive or negative gain. This implies that our estimation doesn't need high accuracy yet still brings significant gains to downstream tasks (see experiments in section 3.2.2).\n\nSecondly, the above analysis indicates that our fitting error depends on the error of using the NTK kernel function to fit the neural network. It is difficult to directly analyze this error. However, many related works have demonstrated the rationality of doing so through extensive experiments[2,3].\n\n\n[1] Wang J, Du W, Cao W, et al. Deep learning for multivariate time series imputation: A survey[J]. arXiv preprint arXiv:2402.04059, 2024.\n\n[2] Jacot A, Gabriel F, Hongler C. Neural tangent kernel: Convergence and generalization in neural networks[J]. Advances in neural information processing systems, 2018, 31.\n\n[3] Tsai C P, Yeh C K, Ravikumar P. Sample-based explanations via generalized representers[J]. Advances in Neural Information Processing Systems, 2024, 36.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zgsg00xJAf", "reviewer": "Reviewer_z2pu", "summary": "The paper proposes a task-oriented time series imputation evaluation approach that assesses the impact of different imputation strategies on downstream forecasting tasks, rather than just the accuracy of the imputed values. The authors introduce a similarity-based method to efficiently estimate the impact of imputed values on downstream task performance, and develop a time series imputation framework that combines the advantages of different imputation strategies. The paper also discusses several axioms that the proposed method satisfies, such as the efficiency, self-explanation, and symmetric zero axioms.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a task-oriented time series imputation evaluation method, focusing on evaluating the impact of different imputation strategies on downstream prediction tasks, rather than just evaluating the accuracy of imputed values. This approach is aligned with practical application needs.\n\n2. The authors propose a similarity-based method that achieves a balance between performance and computational cost, and can effectively estimate the impact of imputed values on the performance of forecasting tasks.\n\n3. The authors have developed a time series imputation framework that combines the advantages of different imputation strategies, and achieves better performance on downstream forecasting tasks.", "weaknesses": "1. The authors state in the abstract that missing values can be observed in many time series analysis tasks, but the task-oriented imputation evaluation method proposed by the authors is only designed for forecasting tasks. In the problem definition, the label of a time series dataset is similarly defined as a time series, however in the classification task, the label is a discrete value. In addition, for some unsupervised learning time series tasks, such as clustering [1], missing values can affect the performance as well. In conclusion, the authors' method is too limited in its scope of application compared to other imputation methods.\n\n[1] Time series cluster kernel for learning similarities between multivariate time series with missing data. PR 2018.\n\n2. The authors' motivation is to provide imputation values that are more beneficial to downstream forecasting tasks, while ignoring the accuracy of the imputation values themselves. However, for the time series imputation task, performing accurate imputations is also one of the non-negligible purposes [2]. The authors ignored the requirement for imputation accuracy when designing their methodology, and did not show experimentally whether their method would reduce imputation accuracy.\n\n[2] Deep learning for multivariate time series imputation: A survey. Arxiv 2402.04059.\n\n3. The authors' ultimate aim is to make more accurate predictions in the presence of missing values, which is highly relevant to the study of robust time series forecasting [3,4]. However the authors don't mention them in the paper and don't compare them as baselines in their experiments. In addition the authors should discuss the advantages and disadvantages of imputation-based and missing value robust-based forecasting methods.\n\n[3] Weakly Guided Adaptation for Robust Time Series Forecasting. VLDB24.\n\n[4] RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies. ICLR 24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and we have addressed each of the concerns raised by the reviewer as outlined below.\n\n>Weakness 1\n\nWe apologize for the misunderstanding caused by our statements. Our main focus here is on time series forecasting tasks. Compared with other time series tasks, the characteristic of forecasting tasks is that their labels are all based on the time series itself, which can be severely affected by missing values and anomalies [1]. Therefore, the relationship between forecasting tasks and time series imputation tasks is highly close, and this requires attention. In response to such situations, we propose a new perspective of evaluating imputation methods by utilizing the impact of label imputation on downstream forecasting tasks and providing practical tools for application. Due to the importance and widespread application of time series forecasting tasks [2], we believe that we have sufficient motivation to incorporate the effectiveness of time series forecasting into the evaluation of time series imputation. As for other kinds of tasks, we can conduct further research in the future.\n\n>Weakness 2\n\nWe fully agree with the value of evaluating the accuracy of imputation as you mentioned. However, we need to emphasize that compared to direct evaluation, evaluating through downstream forecasting tasks has universality and irreplaceability. Universality refers to the fact that unlike time series forecasting tasks, which can obtain true labels through future observations, imputation tasks cannot obtain true labels in reality. Therefore, comparing the accuracy of imputation can only be done on simulated datasets, which limits the objects of evaluation. Irreplaceability refers to the absence of a causal relationship between the accuracy of imputation and the accuracy of downstream forecasting tasks, therefore they cannot replace each other but complement. One of the contributions of our paper is to propose this viewpoint and provide a practical calculation method for evaluation. More innovative time series imputation evaluation methods are also one of the current focuses in the field [3].\n\nIn addition, we have two points that need to be clarified, the first is whether our method may harm the accuracy of imputation. Because forecasting models rely on capturing trends, periods, and other characteristics of time series, which in turn has a supervisory effect on time series imputation. Therefore, our method is unlikely to cause significant damage to time series imputation. (shown in the table below).\n\n| MSE(imputation) |    ELE    |\n|:---------------:|:---------:|\n|      SAITS      |   0.398   |\n|      BRITS      |   0.512   |\n|      GPVAE      | 0.493     |\n|      USGAN      | 0.488     |\n|        IT       | 0.358     |\n|       SPIN      | 0.522     |\n|    SAITS+ours   | **0.343** |\n|    BRITS+ours   | 0.368     |\n|    GPVAE+ours   | 0.364     |\n|    USGAN+ours   | 0.368     |\n|     IT+ours     | 0.347     |\n|       SPIN+ours      | 0.352     |\n\nAnd then the second is that a better imputation accuracy does not always mean a better forecasting performance. We simulate a dataset based on the GEF dataset to illustrate this viewpoint. We conduct experiments on GEF with a predicted length of 24, that is, the label length was 24. We suppose that we only observed the value at the time step nk(k$\\geq$0) and nk+1(k$\\geq$1). Note that this setting is for the convenience of linear interpolation. In the first case(represented by I), we set n = 4, fill the missing value with linear interpolation, and uniformly add Gaussian noise $\\mathcal{N}$(0.05,0.3). In the second case(represented by II), we set n=6 and only do the linear interpolation (visualization can be seen in the pdf uploaded).  We put two data sets into MLP and calculated the forecasting error as shown in the following table.\n\n| MSE | Imputation | Forecasting |\n|:---:|:----------:|:-----------:|\n|  I  | 0.1039 |   **0.1140**  |\n|  II |  **0.0576** |  0.1395 |\n\nThis result indicates that there is no causal relationship between imputation accuracy and forecasting accuracy. When we focus on the impact on downstream forecasting tasks, directly focusing on the accuracy of downstream tasks would be a better choice.\n\n>Weakness 3\n\nWe highly appreciate the value of the robust forecasting method you mentioned and we have a brief discussion\nabout conclusions in robust forecasting methods like [1] in the introduction. Here we want to clarify that our method and robust forecasting methods such as RobustTSF[1] are not in a competitive relationship but in a cooperative relationship, as our method can provide them with data with less noise and higher information data. Specifically, the table below shows the experimental results on the ELE dataset, which indicate that our method can further improve forecasting accuracy based on robust forecasting methods.\n\n|       MSE       |  Mean | SAITS | BRITS | USGAN | GPVAE | ImputerFormer |  SPIN |\n|:---------------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-----:|\n|    +RobustTSF   | 0.056 | 0.050 | 0.092 | 0.084 | 0.099 |     0.053     | 0.076 |\n| +RobustTSF+ours |   -   | **0.046** | 0.050 | 0.052 |**0.046**  |     0.048     | 0.051 |\n\n[1] Cheng H, Wen Q, Liu Y, et al. RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies[C]//The Twelfth International Conference on Learning Representations.\n\n[2] Lim B, Zohren S. Time-series forecasting with deep learning: a survey[J]. Philosophical Transactions of the Royal Society A, 2021, 379(2194): 20200209.\n\n[3] Wang J, Du W, Cao W, et al. Deep learning for multivariate time series imputation: A survey[J]. arXiv preprint arXiv:2402.04059, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIJuK6T76K", "reviewer": "Reviewer_56K9", "summary": "The authors propose a strategy that evaluates the effectiveness of various imputation methods used to fill missing values ​​at different timestamps (time series). The effectiveness of each imputation method is evaluated based on the downstream task gain. Subsequently, rather than filling missing values with a single imputation method, each missing value at a specific timestamp is filled with the most accurate imputation method at that timestamp.", "questions": "- Can you please formally detail how you go from the indicator function $I(i,l)$ (equation 1) to its approximation with the first order Taylor expansion (equation 2)? It is quite confusing because in the equation we have the losses with respect to the learnable parameters $\\theta_1$ and $\\theta_2$ and in Equation 2 only the loss with respect to the $\\theta$ parameter is used. Also, where is the initial term of the first order Taylor expansion? Did you just assume that it is equal to zero?\n\n- Shouldn't NTK be equal to $\\sum_{p=1}^{P}\\frac{\\partial f (\\mathbf{X}_i, \\theta)}{\\partial \\theta_p}\\frac{\\partial f (\\mathbf{X}_k^v, \\theta)}{\\partial \\theta_p}$ in Equation 8?\n\n- Can you please clarify for what purpose linear interpolation is used (line 259-261)? Is it to fill in missing values ​​in the labels and thus obtain two labels, one filled in by the empirical mean and the other filled in by this interpolation?\n\n- Have you really retrain 8760 forecasting models (line 263), or is this a typing error?\n\n- Does the actual gain represent the gain obtained using a single imputation method, and does the estimated gain represent the gain that could be obtained using a combination of imputation methods? If not, can you explain these gains?\n\n- Can you please elaborate more about the influence function?\n\n- When evaluating the proposed solution with multivariate time series (MTS), are the labels also MTS, or are they still univariate time series?", "limitations": "- Unless the reviewer has missed something, it appears that the proposed method is only applicable when the labels are univariate time series. This makes it less generalizable, since in many real-world forecasting applications the objective is to forecast multivariate time series at consecutive discrete timestamps;\n\n- The reviewer would suggest adding the computation time to obtain the estimation gain for each model in Table 2;\n\n- Please correct the typo on line 139, Taylor expansion;\n\n- The reviewer suggests that as a future work, the authors empirically evaluate the gain obtained by combining imputation methods/models that adopt the same or different approaches to filling missing values. For example, it would be interesting to see how the emprirical mean imputation works with GRU-D[1], which also uses the empirical mean to fill missing values. Another example could be to study the gain obtained when imputation models based on graph neural networks and message passing are used exclusively or combined with other imputation models, such as those based on recurrent neural networks.\n\n\n[1] Z. Che, S. Purushotham, K. Cho, D. Sontag, Y. Liu, Recurrent neural networks for multivariate time series with missing values, Scientific reports 8 (1) (2018) 6085.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Originality: The originality of this work lies in the fact that, unlike previous works that study the impact of missing value in inputs, this study evaluates the impact of missing value in labels (time series). Moreover, it proposes an alternative solution that does not use a single imputation method but several that maximize the gain of the downstream task by providing the most accurate imputed value at each timestamp;\n\n- Quality: The paper quality is good. The document is well-structured;\n\n- Significance: The reviewer believes that the proposed solution could have a significant impact in the community. Indeed, one of the inherent problems with time series data is missing values. The proposed solution allows for an optimal choice of various imputation models or strategies to fill these missing values. Moreover, their solution is time series forecasting task-agnostic.", "weaknesses": "- The experimental section 3.3.1 is unclear and very difficult to follow. Authors must refer to the questions to know the reviewer's concerns;\n\n- The proposed strategy may fail when the labels are multivariate time series (MTS). Indeed, in the case of MTS, many imputation methods/models (e.g. deep learning models) consist of filling missing values based on all observed values, i.e. within and across streams. Consequently, combining these imputation methods/models with others that fill in missing values by considering only the observed values of the corresponding univariate time series (e.g. the empirical mean) can lead to a high variance in the imputed values. This high variance will introduce noise into the model calculation scheme and may affect the gain of the downstream task;\n\n- Using the mean as the sole baseline is not sufficient. Authors should adopt at least two baseline imputation methods to perform robust experiments;\n\n\n- The limitations of the model are not discussed. The authors simply indicate the directions for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and we have addressed each of the concerns raised by the reviewer as outlined below.\n\n>Weakness 1\n\nWe sincerely apologize for the misunderstanding we caused, and we will make detailed modifications in the revised version.\n\n>Weakness 2 & Question 7 & Limitation 1\n\nThe experiment in Table 3 uses a multivariate time series to forecast a univariate one. However, our method can theoretically be applied in multivariate forecasting, as in multi-step forecasting, the label $y$ itself is a multidimensional variable. We only need to make simple modifications to the code to apply our method. Below, we present a simple experiment to demonstrate the effectiveness of our method in multivariate forecasting. For simplicity (the main limitation here comes from the time consumption of the imputation method itself rather than our method), we predicted the first three sequences in the original ELE dataset.\n\n| Multi forecasting on ELE |   MSE  |\n|:-----------------:|:------:|\n|        Mean       | 0.1776 |\n|       SAITS       | 0.2252 |\n|     SAITS+ours    | **0.1497** |\n\n>Weakness 3\n\nUsing a simple average as the baseline is convenient for evaluation, as each imputation method has different results under different random numbers, and using them as the baseline for evaluation will become very complex. However, to demonstrate the effectiveness of our method under different baselines, we validated it on ELE data using SAITS and ImputeFormer(IF). If the reviewer consistently believes in the necessity of multiple baselines, we will include them in the revised version.\n\n|    MSE       |     ELE    |\n|:------------------:|:----------:|\n|        Mean        |   0.1123   |\n|        SAITS       |   0.1176   |\n|         IF         |   0.1076   |\n| SAITS(baseline)+IT | **0.1013** |\n| IT(baseline)+SAITS |   0.1051   |\n\n>Weakness 4\n\nWe discussed the limitations in the Appendix and we will add them to the conclusion in the revised version.\n\n>Question 1\n\nWe apologize for the misunderstanding. Our goal is to obtain the value of $\\theta_2$ without retraining given $\\theta_1$ so that we can now the performance of the model on the parameter $\\theta_2$ and get the gain for changing from $\\theta_1$ to $\\theta_2$. The $\\theta$ in Eq (2) should be theta $\\theta_1$ and we take $y_ {i, l} ^ {(1)} $ as the baseline and examine the benefits of replacing $y_ {i, l} ^ {(1)} $with $y_ {i, l} ^ {(2)} $. At this point, the init term is $y_ {i, l} ^ {(1)}$.\n\n>Question 2\n\nDue to the need for multi-step output in our network, the size of the two terms in the NTK kernel is ($L_2$, p) and (p, $L_2$), where $L_2$ is the length of the output. Therefore, the NTK kernel here is not a single value, but rather a matrix of size ($L_2$,$L_2$).\n\n>Question 3\n\nThere is no particular reason for using linear interpolation here, mainly for the sake of simplicity. Our main goal is to demonstrate that our method can estimate the impact of replacing average interpolation with linear interpolation at each time step on downstream forecasting tasks. Replacing linear interpolation with other imputation methods will not bring about essential differences.\n\n>Question 4\n\nThere is no error and we train 8760 different models so that we can get the real difference between the original model and the model that replaces the baseline value with the other ones during the training process, to test the effectiveness of our estimation. If necessary, we can put them on anonymous Google Drive after the rebuttal.\n\n>Question 5\n\nSuppose we have two imputation methods, one serves as the baseline and we want to combine the other one. The gain obtained by replacing the baseline value with the value from the other method at each time step is the actual gain, and the estimated gain is our method's estimation of the actual gain. Therefore, higher accuracy and correlation represent better performance.\n\n>Question 6\n\nThe Influence function examines how the model changes on the test set when a training sample is missing during the training process, and uses this to explain the importance of the sample. Our method mainly examines how the model changes on the test set after a sample is transformed from one imputation method to another during the training process. The application methods of the two are also different. The former usually discards samples with negative impacts to seek better performance on the test set, while our method can evaluate which imputation method is used at each time point in the sample to perform better on the test set.\n\n>Question 7 & Limitation 1\n\nSee Weakness 2\n\n>Limitation 2\n\nHere is the time consumption in Table 2 with RTX 3080Ti and the time consumed by our methods is very small after our optimized programming.\n\n| Time(s) |   GEF  |  ETTH1 | ETTH2  | ELE    | Traffic | AIR    |\n|:----:|:------:|:------:|--------|--------|---------|--------|\n|      | 19.908 | 20.233 | 20.908 | 30.170 |  29.132 | 14.701 |\n\n>Limitation 3\n\nThank you for your pointing out and we will correct it and carefully correct the similar mistake in the revised version.\n\n>Limitation 4\n\nThank you very much for your suggestion. We strongly agree that a single imputation method is difficult to apply in various scenarios. How to combine the advantages of different methods will be an interesting direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7UzEbdtzXc", "reviewer": "Reviewer_xbnY", "summary": "This paper studies the problem of evaluating time series imputation methods in terms of the performance on downstream tasks. It proposes a fine-grained metric and uses RKHS to efficiently estimate the metric. Experiments demonstrate that the proposed method achieves better estimation than the influence function approach.", "questions": "See weaknesses and limitations.", "limitations": "L1: The proposed method uses RKHS to approximate the function. However, it seems difficult to check in practice whether the approximation is accurate or not. \n\nL2: The problem setting does not seem to be well motivated. As what people usually care about is just an aggregated metric (like Table 2), it is unclear why it is worthy to evaluate in such a fine granularity (i.e., I(i,l)) using more computation. \n\nL3: This paper focuses on missing labels but does not consider missing input data. As the authors have dicussed in Section 1, missing input data can have a big impact on forecasting results if the time of missing data is close to the test data.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: This paper studies an interesting perspective for the evaluation of imputation methods: performance on downstream tasks. \n\nS2: This paper borrows an RKHS-based method from the field of model explanation. This may be of independent interest even for a broader research community.\n\nS3: The authors have provided documented code to facilitate reproducibility.", "weaknesses": "W1: The proposed Section 2 & Appendix A are basically Sections 4.1 & 3 in [28], respectively. Thus, the paper should clearly attribute these contributions to [28]. \n- [28] Tsai et al. Sample based explanations via generalized representers. NeurIPS, 2023.\n\nW2: Since this paper focuses on evaluation, it should provide a comprehensive evaluation. However, this paper mainly uses older imputation methods, and a number of latest imputation methods are missing (e.g., [SPIN,PoGeVon]).\n- [SPIN] Marisca et al. Learning to reconstruct missing data from spatiotemporal graphs with sparse observations. NeurIPS, 2022.\n- [PoGeVon] Wang et al. Networked time series imputation via position-aware graph enhanced variational autoencoders. KDD, 2023. \n\nMinor issue: Some sentences are unclear and hard to understand. For example, Lines 103-104 wrote \"$X_i^{D\\times L_1}$ corresponds to ... $L_1$ temporal intervals,\" but what they actually mean seems to be \"one time range of length $L_1$.\" It might be better if the authors further polish their writing to improve clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and we have addressed each of the concerns raised by the reviewer as outlined below.\n\n>Weakness 1\n\nThank you for your comments and we will revise the relevant wording in the revised version to attribute more properly. In addition, we would like to clarify the differences between us and [1], which can be summarized into three points. \n\n**New research objects and techniques**: [1] mainly focus on how to examine the impact of each sample on test results, while we focus on the impact of sample labels on test results and provide **Lemma 1** to ensure the feasibility of our approach.\n\n**New application scenarios**: [1] mainly focus on the influence of samples. When the samples show negative effects, the only way is to discard the negative samples, which may also discard some useful information. We focus on time series imputation tasks, obtaining imputation that is more advantageous at each time step, and minimizing information loss to the extent possible.\n\n**New perspective**: [1] mainly focuses on how to interpret the output of neural networks, and we propose the idea of using downstream forecasting tasks to evaluate time series imputation methods, which is also a current focus in this field [2].\n\n>Weakness 2\n\nWe have added two new time series imputation methods as supplements to Table 2(also shown in the pdf uploaded), including the Spin and the ImputeFormer(IF) [3] published in KDD 2024. As for PoGeVon, we are unable to implement it in a short time due to the lack of relevant open-source code. If the reviewer thinks it's necessary, we can try adding it to the revised version of the paper.\n\n|            |     GEF    |    ETTH1   |    ETTH2   |     ELE    |   TRAFFIC  |     AIR    |\n|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|    Mean    |   0.1750   |   0.0523   |   0.1797   |   0.1123   |   0.4359   |   0.1508   |\n|    SAITS   |   0.1980   |   0.1027   |   0.2098   |   0.1176   |   0.4311   |   0.5006   |\n|    SPIN    |   0.2120   |   0.2000   |   0.2414   |   0.1588   |   0.4609   |   0.6604   |\n|     IF     |   0.1820   |   0.1558   |   0.2125   |   0.1076   |   0.4249   |   0.6300   |\n| SAITS+ours | **0.1653** | **0.0522** |   0.1797   | **0.0957** | **0.4147** | **0.1491** |\n|  SPIN+ours |   0.1679   |   0.0523   | **0.1784** |   0.1038   |   0.4276   |   0.1504   |\n|   IF+ours  |   0.1657   | **0.0522** |   0.1795   |   0.0977   |   0.4178   |   0.1498   |\n\nFrom the experimental results, it can be seen that the conclusion is consistent with the original paper, and our method can help the imputation method achieve better results. \n\n>Limitations 1\n\nWe can analyze the approximation error from two parts.\n\n1. In Section 2, we approximate label $y_{i,l}$ using a first-order approximation. This is reasonable for the MSE loss function, as the second-order term doesn't affect model parameters($\\frac {\\partial^2\\frac{\\partial \\mathcal{L}(f_\\theta(x),y)}{\\partial \\theta }}{\\partial y_{i,l}^2} = 0$). Lemma 1 shows that with small fitting errors, we can make the error between derivatives of fitted and original functions small with any probability. In practice, we only need to determine if imputation at each step is beneficial, focusing on positive or negative gains. This implies that our estimation can bring significant gains to downstream tasks.\n\n2. The above analysis indicates that our fitting error depends on the error of using the NTK kernel function to fit the neural network. It is difficult to analyze this error. However, many related works have demonstrated the rationality of doing so through extensive experiments[1].\n\n>Limitations 2\n\nWe admit that aggregated metric has many advantages like simplicity. However, detailed information is also important in the real world. For example, in the energy system, the peak load(which is a small part of all the load series) caused by extreme weather these days has received increasing attention, as it is the scenario most likely to lead to the collapse of the energy system and result in significant economic losses[4]. Similarly, in the field of healthcare, fine-grained information also helps doctors make correct judgments, which is closely related to patients' lives. All of these indicate the necessity of conducting a detailed analysis of time series.\n\n>Limitations 3\n\nAs the reviewer said, the impact of $x_{j,k}$ in the input on the prediction is influenced by its location, and in practical applications, we have no way to control the position where missing values appear in the input, which is not conducive to evaluating different imputation methods. Therefore, our main focus here is on the impact of label y. However, our method can also be easily applied to examine the impact of input value $x_{j,k}$, simply replacing $y_{i,l}$ in $\\frac{\\partial^2 \\mathcal{L}\\left(f\\left(X_i, \\theta\\right), y_{i}\\right)^T}{\\partial f\\left(X_i, \\theta \\right) \\partial y_{i, l}}$ (Eq.(8) in the original paper) with $x_{j,k}$ will be applicable. Here we give an example of how our approach can help improve more than 20% of the accuracy in the table below.\n\n| Replacing input x |   ELE  |\n|:-----------------:|:------:|\n|       SAITS       | 0.1492 |\n|     SAITS+ours    | **0.1120** |\n\n>Minor issue\n\nThank you very much and we will polish our paper in the revised version.\n\n[1] Tsai C P, Yeh C K, Ravikumar P. Sample-based explanations via generalized representers[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2] Wang J, Du W, Cao W, et al. Deep learning for multivariate time series imputation: A survey[J]. arXiv preprint arXiv:2402.04059, 2024.\n\n[3] Nie, T., Qin, G., Mei, Y., & Sun, J. (2024). ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation. KDD 2024. \n\n[4] Thangjam A, Jaipuria S, Dadabada P K. Time-varying approaches for long-term electric load forecasting under economic shocks[J]. Applied Energy, 2023, 333: 120602.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable time and detailed comments, and we appreciate that the reviewer recognized the strength of our paper like the **theoretical analysis**(Reviewer **aC85**, **wGbj**), **innovative perspective**(Reviewer **wGbj**, **56k9**, **xbnY**), and **practical value**(Reviewer **z2pu**). We have carefully read the comments and provide a detailed response to each comment below. Here we highlight our major responses to each reviewer below. We hope our responses can properly address your concerns.

* In response to the comments of Reviewer **xbnY**'s feedback that we should add more up-to-date imputation methods for comparison, we add two of the SOTA methods called Imputeformer and SPIN and use them to strengthen Table 2 in the original paper(reviewers can find it in the attached pdf). The experiment shows that our methods can help improve even the most up-to-date methods.

* Regarding the concerns of the reviewers **aC85** and **z2pu** about whether there is an overlap between our method and the robust forecasting method. We clarified that our relationship with the robust forecasting method is not a competition but a win-win cooperation, and added experiments showing that our method can further improve the accuracy of robust forecasting.

* We demonstrated to reviewers **56K9** and **xbnY** the scalability of our method, which can be extended to multivariate time series forecasting and also to examine the impact of input data $X$.

* To address the concerns of reviewers **wGbj** and **56K9** about the computational efficiency of our method, we provided the computation time of our method and validated it on a larger scale of data, demonstrating the computational efficiency of our method and the effectiveness of the acceleration method mentioned in section 2.3 of the original paper.

* To better explain to reviewer **z2pu** and **wGbj** the necessity of incorporating the performance of downstream forecasting tasks into the evaluation of time series imputation, we constructed a simulated dataset (relevant visualizations can be found in the PDF file we uploaded) to demonstrate that there is no causal relationship between higher time imputation accuracy and higher forecasting accuracy. When we focus on the impact on downstream forecasting tasks, directly focusing on the accuracy of downstream tasks would be a better choice.

Meta-review: # Summary
This paper proposes a novel approach to evaluating time series imputation methods, focusing on their impact on downstream forecasting tasks rather than just the accuracy of the imputed data. The proposed approach efficiently estimates the effect of different imputation strategies using a similarity-based method, avoiding the need for multiple model retrainings.   Additionally, the authors present a framework that combines the strengths of various imputation techniques to enhance overall task performance, demonstrating significant improvements in downstream forecasting tasks.  

The strengths of the paper include its innovative perspective, strong theoretical foundation, and practical relevance. It addresses a gap in the field by providing a more meaningful assessment of imputation quality in real-world applications.

# Concerns and Author Feedback
However, there are concerns about unclear mathematical explanations, limited scope (focusing only on forecasting), and the computational efficiency of the proposed method. Some reviewers also noted that the paper lacks comparisons with the latest imputation techniques and questioned its generalizability to other tasks and multivariate data.

In their feedback, the authors clarified several points, added comparisons with state-of-the-art methods, took steps toward demonstrating the scalability and efficiency of their approach, and provided limited results for the multivariate setting showing consistent improvements as with univariate. They also provided additional experiments to address concerns about the necessity of focusing on downstream tasks.

# Pros and Cons
- Pros:
  - Innovative approach that shifts the evaluation of imputation methods to their impact on downstream tasks.
  - Strong theoretical foundation and practical relevance.
  - Demonstrated forecast performance improvements and scalability.
- Cons:
  - Mathematical explanations and some experimental setups are unclear.
  - Limited scope, focusing only on forecasting tasks.
  - Concerns about computational efficiency and lack of comparisons with state-of-the-art imputation methods.
  - Questions about the generalizability to other time series tasks and multivariate data.

# Final ratings
After rebuttal and discussions 4/5 reviewers were on the accept side, with one full accept, all others including reject on the borderline, with various different shortcomings listed above being the reason for maintaining borderline ratings. This is a majority on the accept side with at least one feeling more strongly than borderline for acceptance. 

# Final Decision Justification:
The decision to accept the paper is based on its innovative and practically valuable approach, which outweighs the concerns raised. The authors' responses and additional experiments further strengthen the paper's contributions, making it a worthwhile addition to the field.
Decision: Accept (poster)