Title: Task-oriented Time Series Imputation Evaluation via Generalized Representers
Venue: NeurIPS 2024
Accepted: yes
Keywords: time series imputation

downstream task oriented imputation evaluation
Abstract: Time series analysis is widely used in many fields such as power energy, economics, and transportation, including different tasks such as forecasting, anomaly detection, classification, etc. Missing values are widely observed in these tasks, and often leading to unpredictable negative effects on existing methods, hindering their further application. In response to this situation, existing time series imputation methods mainly focus on restoring sequences based on their data characteristics, while ignoring the performance of the restored sequences in downstream tasks. Considering different requirements of downstream tasks (e.g., forecasting), this paper proposes an efficient downstream task-oriented time series imputation evaluation approach. By combining time series imputation with neural network models used for downstream tasks, the gain of different imputation strategies on downstream tasks is estimated without retraining, and the most favorable imputation value for downstream tasks is given by combining different imputation strategies according to the estimated gain.

Task-oriented Time Series Imputation Evaluation via Generalized Representers

Abstract

Time series analysis is widely used in many fields such as power energy, economics, and transportation, including different tasks such as forecasting, anomaly detection, classification, etc. Missing values are widely observed in these tasks, and often leading to unpredictable negative effects on existing methods, hindering their further application. In response to this situation, existing time series imputation methods mainly focus on restoring sequences based on their data characteristics, while ignoring the performance of the restored sequences in downstream tasks. Considering different requirements of downstream tasks (e.g., forecasting), this paper proposes an efficient downstream task-oriented time series imputation evaluation approach. By combining time series imputation with neural network models used for downstream tasks, the gain of different imputation strategies on downstream tasks is estimated without retraining, and the most favorable imputation value for downstream tasks is given by combining different imputation strategies according to the estimated gain. The corresponding code can be found in the repository https://github.com/hkuedl/Task-Oriented-Imputation .

1 Introduction

Time series analysis plays a crucial role in many real-world applications, such as energy, finance, healthcare, and other fields [1, 2, 3]. For example, forecasting load series forms the basis for further decision-making in power dispatch in the power grid system, thereby generating a significant amount of economic benefits [4, 5, 6]. However, collecting time series data, especially high-quality ones, is challenging. Due to the instability of the external environment, sensor failures, and even ethical and legal privacy issues, missing values are prevalent in time series data [7]. For instance, in the BDG2 load series dataset [8], widely used in building energy analysis, the ratio of complete time series data is less than 10 % .

To handle missing values in time series data, numerous methods have been proposed for time series imputation in the literature. Based on the features of the imputation methods, these approaches can be divided into statistical and machine learning methods, such as ARIMA and KNN [9, 10], as well as deep learning-based methods [11, 12, 13, 14]. Both types of methods generally use reconstruction errors of missing values to guide learning and perform evaluation. Recently, some researchers have turned their attention to evaluation strategies based on downstream task performance [15]. However, in most cases, downstream tasks are classification tasks [13], while forecasting tasks, as another important branch of time series-related tasks, have not been fully considered. The main challenge for time series forecasting is that the time series serves as both input and label (output) for the model during training, whereas in classification tasks, it only serves as input for the model.

In supervised learning, training labels influence the calculation of the loss function, which in turn affects the optimization of model parameters and, ultimately, the performance of the model on the test set. [16] indicates that noise in input data (missing data can be considered a type of noise) often has a limited impact on forecasting results. In contrast, label noise can significantly affect the model and, consequently, the final test results from the beginning to the end of the time series. Therefore, when evaluating the impact of different time imputation methods on downstream forecasting tasks, it is essential to focus on assessing the quality of training labels constructed through various imputation methods.

To evaluate how the quality of the imputation labels affects downstream forecasting tasks, it is important to clarify that an excellent imputation strategy does not necessarily mean that the imputed value at each time step is superior to any other method. [15] provides a benchmark for various methods in time series imputation tasks. Although SAITS [13], as one of the latest SOTA methods, has achieved remarkable results, there are still methods that surpass SAITS in some cases. This demonstrates that time series imputation is a complex task, making it difficult to find a universal method capable of handling all situations, let alone considering the performance of downstream forecasting tasks. A more realistic scenario is that while one method may perform better overall, it may not outperform other methods locally. In time series, this means that one method may excel in some time steps, while others do better in different time steps. However, to examine the impact of each time step on forecasting, retraining the forecasting model multiple times is necessary, which is impractical due to time and computational costs. Therefore, an efficient estimation method is needed to examine the impact of each time step with different imputation methods. Additionally, since finding a universal method is difficult, it is natural to shift focus toward combining the advantages of current methods to obtain a better imputation strategy. Consequently, determining how to combine different strategies becomes a challenge.

Based on the above situation, we have developed a task-oriented time series imputation evaluation strategy. Specifically, we summarize our contributions into the following points.

We propose a strategy that evaluates each time series imputation method by estimating the impact of missing (imputed) labels at each time step on downstream tasks without requiring multiple retraining, which significantly reduces time and computational consumption. To the best of our knowledge, we are the first to consider the impact of missing values in time series as labels on downstream forecasting tasks.

Weintroduce a simple and effective similarity calculation method based on the characteristics of long time series to estimate the impact of imputed values more quickly, striking a balance between performance and computational cost.

We develop a time series imputation framework guided by maximizing the gains of downstream tasks, enabling the combination of advantages from different time series imputation strategies to achieve a better one. This results in improved performance in the downstream forecasting task.

1.1 Related Work

Time Series Imputation Time series imputation can be primarily classified into two categories: traditional techniques and neural network-based techniques. Traditional methods replace missing values with statistics, such as the mean value or the last observed value [17]. They include simple statistical models like ARIMA [18], ARFIMA, SARIMA [19], and machine learning techniques such as KNNI [20], TIDER [21], MICE [22], BayOTIDE [23]. In recent years, deep learning imputation methods have demonstrated remarkable capabilities in capturing intricate temporal relationships and complex variation patterns inherent in time series data. These methods employ deep learning models like Transformers [24, 13], generative neural networks such as VAEs [12, 25], GANs [26, 27], and diffusion models [28] to capture complex dynamic relationships within time series data. Although different methods exhibit various advantages, no universal method currently outperforms others in all scenarios and datasets. This observation inspires us to consider combining existing advanced methods in this work to achieve better time series imputation strategies.

Sample-based Explaination Sample-based explainable methods can be divided into two categories [29]. One is based on retraining, which evaluates the importance of corresponding data by comparing the impact of removing data points on the model and even the final test results [30, 31, 32].

Among them, the introduction of shapley value by [30] naturally ensures the fairness of data attribution. The other type is based on gradient methods, which directly estimate the influence of data points without the need for retraining. This type of method can be subdivided into three main categories, which are based on representative theories, Influence Function, and training loss trajectory. [33] is a representative of the method of the first type, whose core idea is to fix the other layers and only focus on the last layer of the neural network, so that the influence of each sample can be explicitly calculated. On the other hand, the Influence Function [34] is based on the assumption of convergence and uses the Hessian matrix to estimate the influence of samples. The last type of method takes into account the entire training process of the neural network, continuously tracking the impact of samples on each parameter update [35]. In addition, [36] summarizes the gradient-based method and unifies them as generalized representers.

2 Methodology

2.1 Problem Statement

Consider a multivariate time series dataset represented by { ( X i , y i ) } n i =1 , incorporating n samples. In this dataset, X i ∈ R D × L 1 corresponds to a feature matrix containing D distinctive features over L 1 temporal intervals, whereas y i ∈ R L 2 signifies the target time series, which spans L 2 temporal intervals. It is crucial to recognize that y i may include several missing entries, a common complication within real-world datasets. For example, in the context of electrical load forecasting, X i encompasses daily weather-related time series data, comprising variables such as temperature, humidity, and wind speed, while y i represents the electrical load time series of a given day, possibly containing missing entries due to issues in data collection or transmission.

Addressing missing values in { y i } through imputation is a fundamental preprocessing step for machine learning tasks involving this data, underscoring the necessity to assess the effectiveness of various imputation methods. Consider { y (1) i } and { y (2) i } as two time series resulting from the imputation of { y } via two different methods. The goal is to ascertain whether the imputation performed on { y (2) i } is superior to that on { y (1) i } . Moreover, we seek to evaluate the quality of imputation at each temporal interval, determining if the imputation of the l -th interval in y (2) i is more accurate than that in y (1) i .

Conventionally, the quality of imputation is quantified by measuring the discrepancy between the imputed values and the actual data, favoring methods that minimize this deviation. In this study, however, we propose to assess imputation quality based on the performance of subsequent tasks.

One step further, we evaluate the quality of imputation on a timestep basis, examining if the imputation for the l -th interval in y (2) i exhibits improved efficacy over y (1) i , thereby offering a more nuanced and comprehensive evaluation of imputation methodologies.

Let us define the loss function for the downstream task as L ( f ( X , θ ) , y ) , where f ( · , θ ) denotes the model used in the downstream task parametered by θ . And let { ( X v i , y v i ) } i m =1 constitute a test dataset that will be used to gauge model performance. We denote y (1) i,l and y (2) i,l as the l -th entries of y (1) i and y (2) i , respectively. According to our intuition, if y (2) i,l is superior to y (1) i,l , swapping y (1) i,l for y (2) i,l should result in a decrease in the test set's loss. Guided by this rationale, we define the indicator function I ( i, l ) , which discerns whether y (2) i,l is preferable over y (1) i,l as follows:

̸

Here, y i (2 ,l ) denotes a vector identical to y (1) i , except at the l -th entry, which matches that of y (2) i . Clearly, I ( i, l ) ≥ 0 implies that the substitution of y (1) i,l with y (2) i,l leads to a decreased test set loss, suggesting that y (2) i,l is superior. Conversely, if I ( i, l ) < 0 , it suggests that y (1) i,l is preferable to y (2) i,l .

Despite the effectiveness of the definition provided by Equation (1), computing I ( i, l ) for every missing value in the dataset is impractical due to the extensive model retraining required, which can be prohibitive in terms of time. To overcome this challenge, in the next section, we put forth an efficient methodology for estimating I ( i, l ) without retraining the model.

2.2 Approximation Model Construction

To compute I ( i, l ) efficiently, we propose a retrain-free method in this subsection. As both y (1) and y (2) are imputation of y , then we assume that y (1) i,l is close to y (2) i,l , with which we approximate I ( i, l ) using the first order Tolyer expansion as

Equation (2) provides an approximation for computing I ( i, l ) , where ∂f ( X v k , θ ) ∂y i,l measures how the training target y i,l affect the prediction of the test data, and ∂ L ( f ( X v k , θ ) , y v k ) ∂f ( X v k , θ ) computes how the changing of the prediction of X v k affect the final loss. Note that the symbolic expression ∂f ( X v k , θ ) ∂y i,l can be conceptually broken down into ∂f ( X v k , θ ) ∂ θ · ∂ θ ∂y i,l , elucidating the role of the label y i,l in shaping the model parameters θ throughout the training process. This, in turn, has repercussions on the model's prediction when evaluated on unseen data from the test set, i.e. f ( X v k , θ ) . By focusing on the derivative ∂f ( X v k , θ ) ∂y i,l , our goal is to assess the extent to which changes in label values y i,l influence the model's predictions on the test set, thereby affecting overall model efficacy.

When it comes back to the estimation, dispite ∂ L ( f ( X v k , θ ) , y v k ) ∂f ( X v k , θ ) and y (1) i,l -y (2) i,l are easy to compute, estimating ∂f ( X v k , θ ) ∂y i,l is difficult. The difficulty comes from two aspects. Firstly, for the complex f ( · , θ ) , the final parameter is not only affected by the training data, some other factors, such as the structure of the network and learning rate during the learning process. Secondly, all of the n training samples affect the parameters of the model, leading to the mixture of the effect of data points on the final model. Thus isolating the effect of a single data point is difficult.

To overcome these two difficulties, we propose to approximate ∂f ( X v k , θ ) ∂y i,l using a white-box model, where how each training datapoint affects the final prediction is clear from the design of the model. To this end, we propose to approximate ∂f ( X v k , θ ) ∂y i,l using a kernel machine, i.e. α T i,l K ( X i , X v k ) , where K ( X i , · ) is a kenerl between the training sample X i and test samples measuring the similarity between the X i and X v k , and α is a learnerable hyperparameter. It can be proven that the indicator function based on this definition satisfies many desirable properties (please see Appendix for details) to construct an axiomatic attribution. Formally, the coefficient α i,l ∈ R L 2 can be computed by solving the following optimization problem:

To solve this problem, f ( X j , θ 1 ) -f ( X j , θ 2 ) y (1) i,l -y (2) i,l can act as a substitute of ∂f ( X j , θ ) ∂y i,l since there is no ground truth. However, the problem is still not practical to solve because we can not obtain f ( X j , θ 2 ) without retraining the model. Even though it isn't necessary to traverse all i , j , and l to obtain the

complete data, it still goes against our original intention to calculate I ( i, l ) efficiently. Furthermore, it is also difficult for us to determine how much data is sufficient to ensure the accuracy of the solution. Fortunately, with the help of Remark 1, we can bypass the process of finding enough f ( X j , θ 1 ) -f ( X j , θ 2 ) y (1) -y (2) and directly approximate ∂f ( X j , θ ) ∂y i,l .

i,l i,l

Remark 1. Given two infinitely differentiable functions f ( x ) and g ( x ) in a bounded domain D ∈ R n , || f ( x ) -g ( x ) || is always less than ϵ . For any given δ and ϵ 2 , there exists an ϵ such that, in the domain D , the measure of the region I that satisfying || ∂f ( x ) ∂ x -∂g ( x ) ∂ x || > δ is not greater than ϵ 2 , i.e, m ( I ) ≤ ϵ 2 .

Remark 1 gives us the intuition that we can avoid retraining our downstream models by first-order approximation. However, some issues still need to be clarified. First, we consider neural networks used in the downstream task as infinitely differentiable functions since in practical applications, it is unlikely for computed floating-point numbers to precisely equal non-differentiable points. Second, Remark 1 limits the definition domain to a bounded region D . Time series data is usually bounded (for example, the renewable generation sequence cannot be greater than the installed capacity), making this assumption reasonable. Finally, Remark 1 can be rephrased as the better the approximation of the original function, the better the approximation of its derivative, that is, we can have a m ( D ) -ϵ 2 m ( D ) probability of fitting the derivative well. Therefore, the optimization problem (3) can be transformed into an easier one. (Note that we indicate the existence of ϵ that meets the conditions in this remark, but make no restrictions on ϵ , while it is often difficult for us to make the approximation error of the original function sufficiently small. If the ϵ in the remark is infinitely close to 0, practical applications will encounter difficulties. However, in gradient descent-based neural network training, such situations often do not hinder our practical applications. Due to the space limit, the full theoretical discussion is provided in the Appendix.)

Now the problem is converted to solving the problem (4). Intuitively, we solve it by projecting it onto the RKHS subspace spanned by the kernels,

where K l is the kernel gram matrix defined as K l,ij = K ( X i , X j ) l . Considering the first-order optimality condition, ˆ α ′ i,l = -1 n ∂ L ( ˆ f K ( X i ) ,f ( X i , θ ) ) ∂ ˆ f K ( X i ) l [36, 37]. Recalling our goal of estimating the relationship between ˆ α ′ and y i,l in (5), their relationship is still unclear. This is because the objective of the optimization problem is to construct an approximation model without considering the role of label value y i,l . To clarify the effect of y i,l , we introduce an approximation by trigonometric inequality that L ( f K ( X i ) , f ( X i , θ )) ≤ L ( y i , f ( X i , θ )) + L ( f K ( X i ) , y i ) . The second term on the right side is the loss function corresponding to the downstream model, and in the case of training convergence, this should be close to a constant. Therefore, the optimization problem (6) can be rewritten as

Formally, the solution are ˆ α ′ i = -1 n ∂ L ( ˆ f K ( X i ) , y i ) ∂ ˆ f K ( X i ) and ˆ α i,l = -1 n ∂ 2 L ( ˆ f K ( X i ) , y i ) ∂ ˆ f K ( X i ) ∂y i,l . Furthermore, since f K ( · ) is used to approximate f ( · , θ ) , we use f ( · , θ ) to replace f K ( · ) to simplify the calculation. With the above preparation, the I ( i, l ) can be represented as follow with NTK kernel [38],

2.3 Similarity Calculation Acceleration

In the previous section, to gauge the effects of substituting y (1) i,l with y (2) i,l on the downstream task, we utilized the Neural Tangent Kernel to assess the similarity between the model outputs for inputs X i and X v k . Given that the model's output length is L 2 , the computational complexity of calculating I ( i, l ) for all time steps in y scales as O ( mL 2 P ) , where P denotes the total number of parameters in the model f ( · , θ ) , i.e., | θ | . In numerous time series applications, such as forecasting, L 2 can be substantially large (e.g., 128), rendering the evaluation process for all imputations timeconsuming. To mitigate this challenge, we propose a method to compress the size of ∂f ( X i , θ ) ∂ θ . Our approach is inspired by the observation that in time series forecasting, the output of f ( · , θ ) typically exhibits smooth variability across different l values. Therefore, we posit that the model output f ( X i , θ ) resides in a low-dimensional space spanned by a limited number of smooth basis functions. In mathematical terms, f ( X i , θ ) ≈ A † A f ( X i , θ ) where A ∈ R r × L 2 consists of rows each representing a predefined smooth vector, A † ∈ R L 2 × r is the pseudo-inverse of A , and r , which is significantly smaller than L 2 , denotes the number of basis functions employed to approximate f ( X i , θ ) . Consequently, we can approximate ∂f ( X i , θ ) ∂ θ as:

This approximation allows for the compression of the model output, thereby reducing the number of gradients that require computation. Through this simplification, the computational complexity for calculating I ( i, l ) decreases to O ( mrP ) , substantially less than the original complexity.

In our experiments, we further simplify by assuming A to be a block diagonal matrix, defined as blkdiag ( 1 1 , 1 2 , . . . , 1 c ) , where 1 1 = · · · = 1 c -1 are vectors of length ⌊ L 2 /c ⌋ with all elements equal to 1, and 1 c ∈ R L 2 -( c -1) ⌊ L 2 /c ⌋ is a vector with all elements equal to 1.

2.4 Task-oriented Imputation Evaluation

We have introduced a method for computing the indicator I ( i, l ) , which assesses if replacing y (1) i,l with y (2) i,l results in a reduced loss for the downstream task. Given two sets of imputation results, { y (1) i } n i =1 and { y (2) i } n i =1 , derived from distinct imputation techniques, we can evaluate I ( i, l ) across all samples and time steps, and identify that y (2) i outperforms y (1) i at time step l if I ( i, l ) greater than zero and vice versa if the value is lesser. In contrast to conventional evaluation strategies, our proposed method does not necessitate the availability of the ground truth y , thereby enhancing its practical utility in myriad real-world scenarios where actual values remain unobtainable. This feature renders our approach significantly more adaptable to situations where empirical truths are elusive.

2.5 Task-oriented Imputation Emsemble

Given that our proposed method can evaluate the quality of two imputations at the time step level, a natural extension is to combine these two sets of imputations to derive an improved result. Specifically, we can generate a new set of imputations y ′ i for the i -th sample, where the l -th entry is y (2) i,l if I ( i, l ) > 0 , and y (1) i,l otherwise. Based on the definition of I ( i, l ) , we anticipate that a model trained using y ′ i will incur a lower loss compared to one trained with y (1) i , thereby yielding better imputation results. It is important to note, however, that the calculation of I ( i, l ) is predicated

on the scenario where only the l th timestep in the i th sample from { y (1) } n i =1 is substituted with y (2) i,l . This consideration omits the potential interactions among samples. Consequently, in practical implementations, we opt to substitute only the timesteps that rank within the top c % of I ( i, l ) values. Our experiments, detailed in 3.3.2, confirm the efficacy of our proposed task-oriented imputation ensemble method. Our proposed ensemble process is summarized in Algorithm 1.

Algorithm 1: Task-oriented Imputation Emsemble.

3 Experiment

3.1 Datasets and Experiment Setup

To validate our method, we conduct experiments on six datasets: the GEF load forecasting competition dataset with the corresponding temperature [39], the UCI dataset (electricity load and air quality) [40], the Traffic dataset containing road occupancy rates 2 , and two transformer datasets, ETTH1 and ETTH2 [41]. Note that we use the hourly resolution version of the UCI electricity dataset from [42] in the main experiment. In our main experiment, we set the downstream task as univariate time series forecasting, with both input sequence and prediction lengths set to 24. In addition to the GEF dataset, we implement our method on the 'OT' time series in ETH1 and ETH2, the mean value of the road occupancy rate in Traffic, the temperature in the UCI air quality dataset, and the total electricity consumption of 321 users in the UCI electricity dataset. It is important to note that there are no original missing values in these datasets. To simulate the missing values situation, we randomly set masks with lengths in [2, 4, 6, 12, 24, 48, 96, 120], and replace the original values with the average value at the corresponding positions as the baseline. For the missing rate setting, if the missing rate is too low, the difference between different imputation methods may be small, while if the missing rate is too high, the even best imputation method will also be difficult to obtain reasonable results. Based on the above considerations, we mainly consider 40% missing rates as our main experimental setup. Meanwhile, we will provide experimental results under other missing rate settings in [30%, 50%, 60%] in the Appendix.

3.2 Time Series Imputation Methods

To verify the performance of our strategy in evaluating different time imputation methods, we introduce multiple advanced time imputation methods. Firstly, as mentioned in the last subsection, we use the basic mean value imputation as the baseline. Secondly, we consider several time series imputation methods based on deep neural networks. They are GPVAE [25] and USGAN [27] based on generative neural network, mRNN [43] and BRITS [11] based on RNN structure, SAITS [13], and ImputeFormer [44] based on attention mechanism. All the implementations of the above models are with the help of the toolkit called PyPOTS [45]. In addition, we also implement the network structure of a spatiotemporal graphs-assisted method called SPIN [46] for time series imputation. All the details of the neural network setting will be described in the Appendix.

3.3 Experimental Results

3.3.1 Application 1: Estimate the Gain

0.2

0.4

0.6

0.8

Corr

MLP

Seq-sim

INF

Seg-4

Seg-2

Seg-1

0.2

0.4

0.6

0.8

DLinear

1%

5%

10%

15%

20%

Percentage

75%

80%

85%

90%

95%

1%

5%

10%

15%

20%

Percentage

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

In this section, we examine the estimated gains of imputation for each time step. We divide the GEF dataset into a training set and a test set, where the training set includes load data from 2011 and the test set includes load data from 2012. We replace 40% of the load data in the training set with the average value at the corresponding position as the baseline, apply the linear interpolation method to replace the corresponding baseline, and use them as training labels separately. In the training set, we have obtained a total of 8760 training samples. Therefore, we replace the labels of each sample one by one to construct new 8760 sets of samples, retrain 8760 forecasting models, test their performance on the test set, and compare the performance of the new model with the model trained on the mean value-based samples. Note that although our estimation is done at each time step of each sample, the time consumption of replacing each time step and retraining is too high. Therefore, we replace all the time steps in each sample and sum up the benefit estimates for each time step as the benefit estimate for the entire sample after replacement. We adopt the 3-layer MLP structure used in [35] and extend it to be used for time series. In addition, we have also added a simple and widely used forecasting model, DLinear [47], as our forecasting backbone model.

As shown in Figure 1, we use two metrics to compare the performance of different methods, where Corr represents the correlation between estimated gain and actual gain, and accuracy represents the percentage of the same sign between actual gain and estimated gain. In the MLP model, the Influence Function achieves a good correlation. However, in terms of accuracy, the accuracy of the Influence Function rapidly decreases as the percentage of estimated data to total data increases. On the contrary, our method exhibits good characteristics at different percentages. In addition, the

segmented acceleration method generally shows that the more segments there are, the better the performance. Although it performs poorly in small percentages, it exhibits good performance in a wide range of data. In the DLiner model, both our method and the segmented acceleration method present similar situations to those in the MLP model. However, the Influence Function exhibits poor performance, with estimation accuracy and correlation significantly lagging behind our method. In addition, Table 1 reports the total time required for different methods to run on the GTX4090, indicating that retraining requires a significant amount of time while our method achieves a good balance between performance and time consumption. It is worth mentioning that when we focus not only on the small portion of time steps with the greatest impact but also examine the impact of relatively large time steps on the forecasting results, our segmented acceleration method achieves very good performance while reducing the required time.

3.3.2 Application 2: Combine Different Time Series Imputation

Similar to the previous section, we use mean value as a baseline to estimate the benefits on the validation set (not test set) obtained by imputation at each time step and replace the 10% time step with the highest benefits to train the forecasting model. In addition, the missing values in the time series damage the original characteristic of the time series. Although imputation values can repair it to some extent, they may still have adverse effects on downstream tasks. Therefore, we also introduce the Influence Function as a comparison, using it to estimate the 10% points with the worst impact on the forecasting results after imputation and remove them, which is often the case in the application of Influence Function [34].

Table 2 reports the comparison of MSE for downstream forecasting tasks (here we use the DLinear as the backbone model), and we will analyze from two aspects.

I. Comparison between Original and Gain Estimation . Part I and II in Table 2 demonstrate that combining different imputations can enhance the performance of downstream tasks on all datasets. The improvements in the GEF dataset, Electricity dataset, and Traffic dataset are significant, while the enhancement in others is relatively less noticeable. The primary reason is that, in those datasets, the performance of other imputation methods considerably lags behind the original one, making the replacement of the original label with a newly imputed label seem less impactful. However, after incorporating our estimation, there is still a slight improvement in forecasting performance. On the other hand, when considering combining two imputation methods with similar original performance, incorporating our method will bring significant gains (as seen in the GEF datasets). It's worth noting that the mean value used as a baseline outperforms other imputation methods in most cases, primarily because the training data input is also based on mean value imputation, facilitating unified and convenient comparisons. In practical applications, we can also apply other advanced imputation methods to the input data and modify the labels based on the estimated benefits.

II. Comparison between Gain Estimation and Influence Function . Part I, II, and III Table 2 indicate that discarding a certain number of samples according to the Influence Function can indeed improve the performance of forecasting; however, such improvement is not universal. In the AIR dataset, discarding some data can negatively impact the performance of most methods. This may be due to the small amount of data contained in the AIR dataset, resulting in a greater adverse effect when discarding data. The operation of discarding data can only consider one imputation method, while our method can combine any two imputation methods to achieve better results. Furthermore, the results of repeated experiments show that the strategy of modifying values at specific time steps can make performance more stable, as its variance is significantly smaller than that of discarding data.

In addition to the univariate input results displayed in Table 2, we also include the results of multivariate inputs, which are common in practical applications. For instance, when predicting power loads, temperature is a crucial external variable. A large amount of research has focused on studying the relationship between load and temperature [48, 49]. In this experimental setting, unlike multivariate forecasting, temperature plays an auxiliary role in load forecasting while there is no need to forecast temperature itself. Consequently, we conduct experiments on the GEF dataset, inputting temperature as an external variable into the model to forecast loads. Table 3 presents the performance of our model under multivariate input, which is consistent with the univariate input scenario, and incorporating our method proves to be beneficial.

4 Conclusion and Future Work

In this work, we propose to evaluate the imputation values at each time step for the impact on downstream forecasting tasks. On the one hand, our method can accurately estimate the gain of each imputation value without retraining. On the other hand, our method can also combine different time series imputation strategies based on the estimation of gain to obtain better imputation for downstream tasks. To ensure the applicability of this method in practical scenarios, we also provide an accelerated calculation method. In the future, we will focus on further downstream tasks, such as optimization tasks based on prediction values, and build an end-to-end evaluation strategy.

References

M. Jin, H. Y. Koh, Q. Wen, D. Zambon, C. Alippi, G. I. Webb, I. King, and S. Pan, 'A survey on graph neural networks for time series: Forecasting, classification, imputation, and anomaly detection,' IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Y. Liang, H. Wen, Y. Nie, Y. Jiang, M. Jin, D. Song, S. Pan, and Q. Wen, 'Foundation models for time series analysis: A tutorial and survey,' in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6555-6565.

J. D. Hamilton, Time series analysis . Princeton university press, 2020.

D. Qin, C. Wang, Q. Wen, W. Chen, L. Sun, and Y. Wang, 'Personalized federated darts for electricity load forecasting of individual buildings,' IEEE Transactions on Smart Grid , vol. 14, no. 6, pp. 4888-4901, 2023.

J. Wang, L. Han, X. Zhang, Y. Wang, and S. Zhang, 'Electrical load forecasting based on variable t-distribution and dual attention mechanism,' Energy , vol. 283, p. 128569, 2023.

M. Grabner, Y. Wang, Q. Wen, B. Blažiˇ c, and V. Štruc, 'A global modeling framework for load forecasting in distribution networks,' IEEE Transactions on Smart Grid , vol. 14, no. 6, pp. 4927-4941, 2023.

W. Du, J. Wang, L. Qian, Y. Yang, F. Liu, Z. Wang, Z. Ibrahim, H. Liu, Z. Zhao, Y. Zhou et al. , 'Tsi-bench: Benchmarking time series imputation,' arXiv preprint arXiv:2406.12747 , 2024.

C. Miller, A. Kathirgamanathan, B. Picchetti, P. Arjunan, J. Y. Park, Z. Nagy, P. Raftery, B. W. Hobson, Z. Shi, and F. Meggers, 'The building data genome project 2, energy meter data from the ASHRAE great energy predictor III competition,' Scientific Data , vol. 7, p. 368, Oct. 2020.

M. Steinbach and P.-N. Tan, 'knn: k-nearest neighbors,' in The top ten algorithms in data mining . Chapman and Hall/CRC, 2009, pp. 165-176.

R. H. Shumway, D. S. Stoffer, R. H. Shumway, and D. S. Stoffer, 'Arima models,' Time series analysis and its applications: with R examples , pp. 75-163, 2017.

W. Cao, D. Wang, J. Li, H. Zhou, L. Li, and Y. Li, 'Brits: Bidirectional recurrent imputation for time series,' Advances in neural information processing systems , vol. 31, 2018.

L. Pinheiro Cinelli, M. Araújo Marins, E. A. Barros da Silva, and S. Lima Netto, 'Variational autoencoder,' in Variational Methods for Machine Learning with Applications to Deep Networks . Springer, 2021, pp. 111-149.

W. Du, D. Côté, and Y. Liu, 'Saits: Self-attention-based imputation for time series,' Expert Systems with Applications , vol. 219, p. 119619, 2023.

J. Cheng, C. Yang, W. Cai, Y. Liang, Q. Wen, and Y. Wu, 'NuwaTS: Mending every incomplete time series,' arXiv preprint arXiv:2405.15317 , 2024.

J. Wang, W. Du, W. Cao, K. Zhang, W. Wang, Y. Liang, and Q. Wen, 'Deep learning for multivariate time series imputation: A survey,' arXiv preprint arXiv:2402.04059 , 2024.

H. Cheng, Q. Wen, Y. Liu, and L. Sun, 'RobustTSF: Towards theory and design of robust time series forecasting with anomalies,' in The Twelfth International Conference on Learning Representations (ICLR) , 2024.

M. Amiri and R. Jensen, 'Missing data imputation using fuzzy-rough methods,' Neurocomputing , vol. 205, pp. 152-164, 2016.

G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time series analysis: forecasting and control . John Wiley & Sons, 2015.

C. Hamzaçebi, 'Improving artificial neural networks' performance in seasonal time series forecasting,' Information Sciences , vol. 178, no. 23, pp. 4550-4559, 2008.

N. S. Altman, 'An introduction to kernel and nearest-neighbor nonparametric regression,' The American Statistician , vol. 46, no. 3, pp. 175-185, 1992.

S. Liu, X. Li, G. Cong, Y. Chen, and Y. Jiang, 'Multivariate time-series imputation with disentangled temporal representations,' in The Eleventh International Conference on Learning Representations , 2022.

S. Van Buuren and K. Groothuis-Oudshoorn, 'mice: Multivariate imputation by chained equations in r,' Journal of statistical software , vol. 45, pp. 1-67, 2011.

S. Fang, Q. Wen, Y. Luo, S. Zhe, and L. Sun, 'Bayotide: Bayesian online multivariate time series imputation with functional decomposition,' in Forty-first International Conference on Machine Learning (ICML) , 2024.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, 'Attention is all you need,' Advances in neural information processing systems , vol. 30, 2017.

V. Fortuin, D. Baranchuk, G. Rätsch, and S. Mandt, 'Gp-vae: Deep probabilistic time series imputation,' in International conference on artificial intelligence and statistics . PMLR, 2020, pp. 1651-1661.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, 'Generative adversarial networks,' Communications of the ACM , vol. 63, no. 11, pp. 139-144, 2020.

X. Miao, Y. Wu, J. Wang, Y. Gao, X. Mao, and J. Yin, 'Generative semi-supervised learning for multivariate time series imputation,' in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 10, 2021, pp. 8983-8991.

J. Ho, A. Jain, and P. Abbeel, 'Denoising diffusion probabilistic models,' Advances in Neural Information Processing Systems , vol. 33, pp. 6840-6851, 2020.

Z. Hammoudeh and D. Lowd, 'Training data influence analysis and estimation: A survey,' Machine Learning , pp. 1-53, 2024.

A. Ghorbani and J. Zou, 'Data shapley: Equitable valuation of data for machine learning,' in International conference on machine learning . PMLR, 2019, pp. 2242-2251.

T. Wang and R. Jia, 'Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity,' arXiv preprint arXiv:2205.15466 , 2022.

Y. Kwon and J. Zou, 'Beta shapley: a unified and noise-reduced data valuation framework for machine learning,' arXiv preprint arXiv:2110.14049 , 2021.

C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, 'Representer point selection for explaining deep neural networks,' Advances in neural information processing systems , vol. 31, 2018.

P. W. Koh and P. Liang, 'Understanding black-box predictions via influence functions,' in International conference on machine learning . PMLR, 2017, pp. 1885-1894.

G. Pruthi, F. Liu, S. Kale, and M. Sundararajan, 'Estimating training data influence by tracing gradient descent,' Advances in Neural Information Processing Systems , vol. 33, pp. 19 92019 930, 2020.

C.-P. Tsai, C.-K. Yeh, and P. Ravikumar, 'Sample based explanations via generalized representers,' Advances in Neural Information Processing Systems , vol. 36, 2024.

B. Schölkopf, R. Herbrich, and A. J. Smola, 'A generalized representer theorem,' in International conference on computational learning theory . Springer, 2001, pp. 416-426.

A. Jacot, F. Gabriel, and C. Hongler, 'Neural tangent kernel: Convergence and generalization in neural networks,' Advances in neural information processing systems , vol. 31, 2018.

T. Hong, P. Pinson, and S. Fan, 'Global energy forecasting competition 2012,' pp. 357-363, 2014.

D. Dua, C. Graff et al. , 'Uci machine learning repository,' 2017.

H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, 'Informer: Beyond efficient transformer for long sequence time-series forecasting,' in The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference , vol. 35, no. 12. AAAI Press, 2021, pp. 11 106-11 115.

G. Lai, W.-C. Chang, Y. Yang, and H. Liu, 'Modeling long-and short-term temporal patterns with deep neural networks,' in The 41st international ACM SIGIR conference on research & development in information retrieval , 2018, pp. 95-104.

J. Yoon, W. R. Zame, and M. van der Schaar, 'Estimating missing data in temporal data streams using multi-directional recurrent neural networks,' IEEE Transactions on Biomedical Engineering , vol. 66, no. 5, pp. 1477-1490, 2018.

T. Nie, G. Qin, W. Ma, Y. Mei, and J. Sun, 'Imputeformer: Low rankness-induced transformers for generalizable spatiotemporal imputation,' in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 2260-2271.

W. Du, 'PyPOTS: a Python toolbox for data mining on Partially-Observed Time Series,' 2023. [Online]. Available: https://arxiv.org/abs/2305.18811

I. Marisca, A. Cini, and C. Alippi, 'Learning to reconstruct missing data from spatiotemporal graphs with sparse observations,' Advances in Neural Information Processing Systems , vol. 35, pp. 32 069-32 082, 2022.

A. Zeng, M. Chen, L. Zhang, and Q. Xu, 'Are transformers effective for time series forecasting?' in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 9, 2023, pp. 11 121-11 128.

L. Yang, R. Ren, X. Gu, and L. Sun, 'Interactive generalized additive model and its applications in electric load forecasting,' in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 5393-5403.

J. Xie and T. Hong, 'Temperature scenario generation for probabilistic load forecasting,' IEEE Transactions on Smart Grid , vol. 9, no. 3, pp. 1680-1687, 2016.

E. Winter, 'The shapley value,' Handbook of game theory with economic applications , vol. 3, pp. 2025-2054, 2002.

L. Bottou, 'Large-scale machine learning with stochastic gradient descent,' in Proceedings of COMPSTAT'2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers . Springer, 2010, pp. 177-186.

D. Kingma, 'Adam: a method for stochastic optimization,' in Int Conf Learn Represent , 2014.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , 'Pytorch: An imperative style, high-performance deep learning library,' Advances in neural information processing systems , vol. 32, 2019.

A Why We Need Task-oriented Imputation

Here we use a toy example to illustrate that, in some cases, we can not directly use the accuracy of imputation instead of downstream tasks to evaluate the imputation method. We want to point out that better imputation accuracy does not always mean better forecasting performance, and we simulate a dataset based on the GEF dataset to illustrate this viewpoint, experimenting with a predicted length of 24. Suppose that we only observed the value at the time step nk (k ≥ 0) and nk+1 (k ≥ 1), just for the convenience of linear interpolation. In the first case(represented by I), we set n = 4, fill the missing value with linear interpolation, and uniformly add Gaussian noise N (0.05,0.3). In the second case(represented by II), we set n = 6 and only do the linear interpolation (shown in Figure 2). We put two data sets into MLP and calculated the forecasting error as shown in the following Table 4.

0

5

10

15

20

Time step

0.5

0.0

0.5

1.0

1.5

2.0

2.5

Value

Simulated data visualization

true

Case I

Case II

B Axioms

In [36], there are several axioms desirable for a fair attribution. In this section, we modify them to a version suitable for our task and demonstrate that our method satisfies such properties. Here we use I ( i, l, X k ) to represent the impact of the perturbation of the l-th step of the i-th sample on sample X k .

Definition 1 (Efficiency Axiom). For any model f ( · , θ ) , and test point X v k , an indication function I ( · , · , · ) satisfies the efficiency axiom iff:

This is a counterpart of the efficiency axioms in Shapley values [50], and our method is naturally satisfying.

Definition 2 (Self-Explanation Axiom). An indication function I ( · , · , · ) satisfies the self-explanation axiom iff there exists any training point X i having no effect on itself, i.e. I ( i, l, X i ) = 0 , the training point should not impact any other points, i.e. I ( i, l, X ) = 0 for all X .

Similar to the dummy axiom in the Shapley values, our method naturally satisfies.

̸

Definition 3 (Symmetric Zero Axiom). An indication function I ( · , · , · ) satisfies the symmetric zero axiom iff any two training points X i , X j such that if I ( i, · , X i ) = 0 and I ( j, · , X j ) = 0 , then

̸

This situation holds if and only if K ( X i , X j ) = 0 , therefore, our method satisfies.

Definition 4 (Symmetric Cycle Axiom). An indication function I ( · , · , · ) satisfies the symmetric cycle axiom iff for any set of training points X t 1 , . . . X t k , with possible duplicates, and X t k +1 = X t 1 , it holds that:

The original definition of I ( · , · , · ) does not satisfy the above properties, as our goal is to estimate the impact of label perturbations on the loss. However, when removing ∂ L ( f ( X k , θ ) , y k ) ∂f ( X k , θ ) from the definition, our method satisfies this property.

Definition 5 (Continuity Axiom). An indication function I ( · , · , · ) satisfies the continuity axiom iff it is continuous wrt the test data point X , for any fixed training point X i :

Definition 6 (Irreducibility Axiom). An indication function I ( · , · , · ) satisfies the irreducibility axiom iff for any number of training points X 1 , . . . , X k ,

A sufficient condition for an attribution A ( · ) to satisfy the irreducibility axiom is for

̸

When selecting the NTK kernel, this property naturally satisfies.

C Discussion on Remark 1

Remark 1. Given two infinitely differentiable functions f ( x ) and g ( x ) in a bounded domain D ∈ R n , || f ( x ) -g ( x ) || is always less than ϵ . For any given δ and ϵ 2 , there exists an ϵ such that, in the domain D , the measure of the region I that satisfying || ∂f ( x ) ∂ x -∂g ( x ) ∂ x || > δ is not greater than ϵ 2 , i.e, m ( I ) ≤ ϵ 2 .

Correctness. Firstly, we can relax the restrictions on the function by requiring that each dimension of the function on R n be continuous. Then, we can simplify the problem into a one-dimensional case on R . Secondly, we can let h ( x ) = f ( x ) -g ( x ) (note that | h ( x ) | < ϵ ), and then our problem can be transformed into proving that for any given δ and ϵ 2 , there exists an ϵ such that, in the interval [ a, b ] , the measure of the region satisfying | h ′ ( x ) | > δ is less than ϵ 2 .

Let the domain I represent the set of all x that satisfy | h ′ ( x ) | > δ . We first need to prove that I can be rewritten as the union of several disjoint intervals I i that satisfy

Since h ′ ( x ) itself is a continuous function, the division is obvious. The problem is that we need to prove the number of I i is countable.

Assuming I i is uncountable, we perform n bisection on [ a, b ] to obtain numbers of smaller intervals like [ a +( 1 2 ) n , a +( 1 2 ) n +1 ] . If I i is not countable, then no matter how large n is, we can always find two points x and y in the interval that satisfy h ′ ( x ) > δ, h ′ ( y ) > δ (here we mainly consider greater cases, the smaller case is the same), then

and this equation does not hold as ϵ → 0 , which makes contradiction. Therefore, we can rewrite I as ⋃ ∞ i =1 I i . To simplify, assume that I can be written in the form of a finite number of unions I = ⋃ N i =1 I i . If the remark does not hold, we have max i m ( I i ) = k , then Nk ≥ ϵ 2 . Obviously,

| δk | ≤ 2 ϵ , then k ≤ | 2 ϵ δ | . Finally, we have

which makes contradiction as ϵ → 0 . If the number of I i is infinite, we can always find N that is big enough to satisfy Nk ≥ ϵ 2 and the rest is the same.

Discussion on Application. As shown in the analysis above, for some given δ and ϵ 2 , we only require that | ϵ | < | x -y | δ and | ϵ | < ∣ ∣ ϵ 2 δ 2 N ∣ ∣ . Even though the real boundary is highly related to the specific scenarios and is difficult to tell in real applications, we can have a look at some widely used examples. Here, we mainly discuss two kinds of optimizers that are widely used in neural network training and they are SGD [51] and Adam [52].

For the loss function L to be set as MSE. Note that we use X and X test to represent the input data in the training set and test set, separately. y is the training label and y i,l is the value of a time step in it. We use t to represent the training epoch and there will be T epochs total. Our goal is to use g ( X test , y i,l ) to approximate f ( X test , θ T , y i,l ) first and then use ∂g ( X test ,y i,l ) ∂y i,l to approximate ∂f ( X test ,θ T ,y i,l ) ∂y i,l .

During the training process of a forecasting model, we let h t ( y i,l ) represent ∂ L ( f ( X,θ t ,y i,l ) ,y ) ∂θ t and ∂ 2 h t ( y i,l ) ∂y 2 i,l will be zero. Recalling that our goal is to approximate the gradient ∂f ( X test ,θ T ,y i,l ) ∂y i,l by approximating f ( X test , θ T , y i,l ) first and then take the gradient. However, the function that we really want to approximate is ∂θ T ∂y i,l since ∂f ( X test ,θ T ,y i,l ) ∂y i,l = ∂f ( X test ,θ T ,y i,l ) ∂θ T ∂θ T ∂y i,l and ∂f ( X test ,θ T ,y i,l ) ∂θ T is a constant for given X test .

For SGD (as well as its variants SGD with momentum), θ T = θ 0 -∑ T t =1 ηh t ( y i,l ) . Therefore, the ∂ 2 θ T ∂y 2 i,l = 0 , which means that the gradient of the function we want to approximate is constant. In this case, our approximation will be pretty good.

For Adam, on the one hand, [35] has claimed that the first-order approximation in the SGD situation remains valid, as long as a small step-size η is used in the update. On the other hand, let θ T = θ 0 -∑ T t =1 ηv t ( h t ( y i,l )) , where v t represent the terms in Adam. In this situation, ∂θ T ∂y i,l will be an algebraic function with only a finite number of monotonic intervals. Therefore, for any given δ and ϵ , the ϵ 2 will not be really high since the y i,l that makes || ∂f ( X test ,θ T ,y i,l ) ∂y i,l -∂g ( X test ,y i,l ) ∂y i,l || > δ can only appear near the local extreme point, whose number is finite.

D Implementation Details

Table 5 summarizes the dataset partitioning we used. Except for the GEF data, the rest are multivariate datasets. We forecast the 'OT' sequences in ETH1 and ETH2, as well as the combined electricity

consumption of all users in ELECTRITY, the average occupancy rate of all roads in TRAFFIC, and the temperature series in the AIR dataset.

D.1 Datasets for forecasting

D.2 Implementation of time series forecasting model

We include two models in our experiment. The first one is a 3-layer MLP in which the input size and output size are both 24 while the hidden size is 128. In addition, we mainly apply the simple and high-performance DLiner with default setting in [47] as our forecasting model backbone. In addition, to adapt to situations where the input and output dimensions are different, we constructed an output layer at the end of the DLiner model, mapping the output of multiple variables to the output of a specified dimension. We use the torch.SGD optimizer [53] to optimize the parameters of the model, where the learning rate is set to 0.1. The maximum epochs for each training are 300, and the patience is set to 10.

D.3 Implementation of time series imputation model

We have introduced a total of five imputation methods for comparison, and all experiments were based on pyPOTS [45] except for SPIN. The hyperparameters for each method are set as shown in Table 6. In addition, referring to [13], we set the total training epoch to 100 and the patience to 10, while other hyperparameters are the default setting.

D.4 Implementation of our estimation

In section 2.2, we gives the estimation of I ( i, l ) as follows,

However, depending on the solution to the optimization problem (7), we may have different forms of estimation for I ( i, l ) . Referring to [36, 35], when we no longer only consider the downstream model parameters θ at the moment of training convergence but also consider the entire training process, we can obtain another form of solution to the optimization problem that

ˆ α i,l = -∑ t ∈ [ T ]: i ∈ B ( t ) η ( t ) | B ( t ) | ∂ 2 L ( f ( t -1) ( X i , θ ) , y i ) ∂f ( t -1) ( X i , θ ) ∂y i,l , where t , B ( t ) , and η ( t ) here represent the t th epoch, the batch size, and the learning rate, respectively. And the I ( i, l ) in this situation will be

Compared to the original estimation, repeated calculations will bring a significant computational burden. However, based on our acceleration method, the time required for such multiple calculations is still controlled within a reasonable range. In application, we calculate for each parameter updates in each epoch.

D.5 Implementation of Influence Function

In section 3.3.1, we compared the performance of our method with the Influence Function that we modified. Below, we will describe how to modify the Influence Function to fit our task.

For a training point ( X , y ) , define y l,δ def = [ y 1 , · · · , y l + δ, · · · , y L 2 ] . Consider the perturbation y ↦→ y l,δ and let θ ϵ,δ def = arg min θ ∈ Θ 1 n ∑ n i =1 L ( f ( X i , θ ) , y i ) + ϵL ( f ( X , θ ) , y l,δ ) -ϵL ( f ( X , θ ) , y l,δ ) . Then we have

Therefore, I ( l, X test ) = -∇ θ L ( f ( X test , θ ) , y test ) ⊤ H -1 ˆ θ [ ∇ y l ∇ θ L ( f ( X , ˆ θ ) , y ) ] δ . Note that we applied the Conjugate gradients mentioned in [34] to accelerate its computation and compare it with our methods.

D.6 Hareware usage

We use 1 NVIDIA GTX 4090 GPU with 24GB of memory for all our experiments.

E Potential Social Impact

Our estimation may not be 100% accurate compared to the actual situation, so it is possible to introduce bias in the evaluation among different imputation strategies, which may further have adverse effects on downstream tasks.

F Supplementary Experimental Results

F.1 Acceleration method

F.1.1 Performance of the acceleration method

In our practice, we mainly examine the benefits of modifying each time step on downstream tasks. Therefore, we mainly focus on whether the gain estimation is positive or negative without providing precise values. Based on this idea, we provide methods for accelerating calculations in Section 2.3. Here, we present a comparison between the accelerated estimate and the original estimate. Note that we conduct this experiment on three datasets and they are GEF, ELECTRICITY, and a generated time series, denoted by Brown, based on the following Python code.

The result is shown in the Figure 3. Note that here we also use correlation and accuracy mentioned in the main text.

0.80

0.85

0.90

0.95

1.00

Corr

GEF

Seg-12

Seg-8

Seg-6

Seg-4

80%

82%

84%

86%

88%

90%

92%

94%

96%

98%

100%

GEF

0.80

0.85

0.90

0.95

1.00

Corr

ELE

80%

82%

84%

86%

88%

90%

92%

94%

96%

98%

100%

ELE

1%5% 10%15%20%

50%

Percentage

0.80

0.85

0.90

0.95

1.00

Corr

1%5% 10%15%20%

50%

Percentage

80%

82%

84%

86%

88%

90%

92%

94%

96%

98%

100%

F.1.2 MSE comparison on downstream forecasting task

Table 7 summarizes the performance of our acceleration methods Seg-4 and Seg-2 in downstream forecasting tasks. Overall, performance gradually improves as the number of segmented segments

gradually increases. In addition, even if it is only divided into two segments, our method can still bring some gain with minimal additional computational burden.

F.1.3 A larger dataset

For large-scale data, we applied our method to a 15-minute resolution UCI electricity dataset (with approximately 100000 training points) and we adjusted our experimental setup to input 96 points and output 96 points, and here is the result.

F.2 Additional missing rate

In addition to the 40% missing rate in the main experiment, we also conduct several experiments in the ELECTRICITY dataset with missing rates in [30% , 50% , 60%] .

F.3 Combination with robust time series forecasting

In addition to our solution, some kinds of methods, such as robust time series forecasting, to deal with missing (anomaly) values have been proposed these days. Here we combine our method with [16], which is one of the SOTA of such kind of method to illustrate that this kind of method is not contradictory to our approach but can be combined. Note that the hyperparameters are the same as the original paper in [16] and we replace the dataset with ours.

F.4 Illustration on multivariate forecasting

We conduct our method on multivariate forecasting tasks and give a small example of the ELECTRICITY dataset. Note that we apply our method to the first three users (columns) in the dataset.

F.5 Forecasting results

Figures 4, 5, 6, 7, 8, and 9, show the visualization of the forecasting performance of the model. We mainly compared the combination of the SAITS method and the baseline model with the original SATIS method, and it can be seen that combining the two imputations will bring benefits to the forecasting.

0

10

20

30

40

50

60

70

Time

2.5

2.0

1.5

1.0

0.5

Value

true

saits

saits+mean

0

10

20

30

40

50

60

70

1.5

1.0

0.5

0.0

0.5

Value

true

saits

saits+mean

Time

0

10

20

30

40

50

60

70

Time

2

1

0

1

2

3

Value

true

saits

saits+mean

0

10

20

30

40

50

60

70

Time

2.0

1.5

1.0

0.5

0.0

0.5

Value

true

saits

saits+mean

0

10

20

30

40

50

60

70

Time

1.6

1.4

1.2

1.0

0.8

0.6

Value

true

saits

saits+mean

0

10

20

30

40

50

60

70

Time

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Value

true

saits

saits+mean

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have summarized our contributions both in the abstract section and at the end of the introduction. The claims made are supported by the results in the paper's experiment section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We cover some limitations of this work in our conclusion section (the part left for future work).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 2 and Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide details about our experiment settings in the experiment section to ensure reproducibility as well as the link of the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provide the public accessible link to both the code and dataset we used for experiment.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We cover all these information in our experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] ,

Justification: Every experiment is done 3 times.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] ,

Justification: We cover these information in our appendix sections.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that our research adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The societal impacts of time series forecasting are provided in the Introduction section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited all the models used for the experiment and ensured the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets..

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of evaluating time series imputation methods in terms of the performance on downstream tasks. It proposes a fine-grained metric and uses RKHS to efficiently estimate the metric. Experiments demonstrate that the proposed method achieves better estimation than the influence function approach.\n\nSTRENGTHS:\nS1: This paper studies an interesting perspective for the evaluation of imputation methods: performance on downstream tasks. \n\nS2: This paper borrows an RKHS-based method from the field of model explanation. This may be of independent interest even for a broader research community.\n\nS3: The authors have provided documented code to facilitate reproducibility.\n\nWEAKNESSES:\nW1: The proposed Section 2 & Appendix A are basically Sections 4.1 & 3 in [28], respectively. Thus, the paper should clearly attribute these contributions to [28]. \n- [28] Tsai et al. Sample based explanations via generalized representers. NeurIPS, 2023.\n\nW2: Since this paper focuses on evaluation, it should provide a comprehensive evaluation. However, this paper mainly uses older imputation methods, and a number of latest imputation methods are missing (e.g., [SPIN,PoGeVon]).\n- [SPIN] Marisca et al. Learning to reconstruct missing data from spatiotemporal graphs with sparse observations. NeurIPS, 2022.\n- [PoGeVon] Wang et al. Networked time series imputation via position-aware graph enhanced variational autoencoders. KDD, 2023. \n\nMinor issue: Some sentences are unclear and hard to understand. For example, Lines 103-104 wrote \"$X_i^{D\\times L_1}$ corresponds to ... $L_1$ temporal intervals,\" but what they actually mean seems to be \"one time range of length $L_1$.\" It might be better if the authors further polish their writing to improve clarity.\n\nQUESTIONS:\nSee weaknesses and limitations.\n\nLIMITATIONS:\nL1: The proposed method uses RKHS to approximate the function. However, it seems difficult to check in practice whether the approximation is accurate or not. \n\nL2: The problem setting does not seem to be well motivated. As what people usually care about is just an aggregated metric (like Table 2), it is unclear why it is worthy to evaluate in such a fine granularity (i.e., I(i,l)) using more computation. \n\nL3: This paper focuses on missing labels but does not consider missing input data. As the authors have dicussed in Section 1, missing input data can have a big impact on forecasting results if the time of missing data is close to the test data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a strategy that evaluates the effectiveness of various imputation methods used to fill missing values ​​at different timestamps (time series). The effectiveness of each imputation method is evaluated based on the downstream task gain. Subsequently, rather than filling missing values with a single imputation method, each missing value at a specific timestamp is filled with the most accurate imputation method at that timestamp.\n\nSTRENGTHS:\n- Originality: The originality of this work lies in the fact that, unlike previous works that study the impact of missing value in inputs, this study evaluates the impact of missing value in labels (time series). Moreover, it proposes an alternative solution that does not use a single imputation method but several that maximize the gain of the downstream task by providing the most accurate imputed value at each timestamp;\n\n- Quality: The paper quality is good. The document is well-structured;\n\n- Significance: The reviewer believes that the proposed solution could have a significant impact in the community. Indeed, one of the inherent problems with time series data is missing values. The proposed solution allows for an optimal choice of various imputation models or strategies to fill these missing values. Moreover, their solution is time series forecasting task-agnostic.\n\nWEAKNESSES:\n- The experimental section 3.3.1 is unclear and very difficult to follow. Authors must refer to the questions to know the reviewer's concerns;\n\n- The proposed strategy may fail when the labels are multivariate time series (MTS). Indeed, in the case of MTS, many imputation methods/models (e.g. deep learning models) consist of filling missing values based on all observed values, i.e. within and across streams. Consequently, combining these imputation methods/models with others that fill in missing values by considering only the observed values of the corresponding univariate time series (e.g. the empirical mean) can lead to a high variance in the imputed values. This high variance will introduce noise into the model calculation scheme and may affect the gain of the downstream task;\n\n- Using the mean as the sole baseline is not sufficient. Authors should adopt at least two baseline imputation methods to perform robust experiments;\n\n\n- The limitations of the model are not discussed. The authors simply indicate the directions for future work.\n\nQUESTIONS:\n- Can you please formally detail how you go from the indicator function $I(i,l)$ (equation 1) to its approximation with the first order Taylor expansion (equation 2)? It is quite confusing because in the equation we have the losses with respect to the learnable parameters $\\theta_1$ and $\\theta_2$ and in Equation 2 only the loss with respect to the $\\theta$ parameter is used. Also, where is the initial term of the first order Taylor expansion? Did you just assume that it is equal to zero?\n\n- Shouldn't NTK be equal to $\\sum_{p=1}^{P}\\frac{\\partial f (\\mathbf{X}_i, \\theta)}{\\partial \\theta_p}\\frac{\\partial f (\\mathbf{X}_k^v, \\theta)}{\\partial \\theta_p}$ in Equation 8?\n\n- Can you please clarify for what purpose linear interpolation is used (line 259-261)? Is it to fill in missing values ​​in the labels and thus obtain two labels, one filled in by the empirical mean and the other filled in by this interpolation?\n\n- Have you really retrain 8760 forecasting models (line 263), or is this a typing error?\n\n- Does the actual gain represent the gain obtained using a single imputation method, and does the estimated gain represent the gain that could be obtained using a combination of imputation methods? If not, can you explain these gains?\n\n- Can you please elaborate more about the influence function?\n\n- When evaluating the proposed solution with multivariate time series (MTS), are the labels also MTS, or are they still univariate time series?\n\nLIMITATIONS:\n- Unless the reviewer has missed something, it appears that the proposed method is only applicable when the labels are univariate time series. This makes it less generalizable, since in many real-world forecasting applications the objective is to forecast multivariate time series at consecutive discrete timestamps;\n\n- The reviewer would suggest adding the computation time to obtain the estimation gain for each model in Table 2;\n\n- Please correct the typo on line 139, Taylor expansion;\n\n- The reviewer suggests that as a future work, the authors empirically evaluate the gain obtained by combining imputation methods/models that adopt the same or different approaches to filling missing values. For example, it would be interesting to see how the emprirical mean imputation works with GRU-D[1], which also uses the empirical mean to fill missing values. Another example could be to study the gain obtained when imputation models based on graph neural networks and message passing are used exclusively or combined with other imputation models, such as those based on recurrent neural networks.\n\n\n[1] Z. Che, S. Purushotham, K. Cho, D. Sontag, Y. Liu, Recurrent neural networks for multivariate time series with missing values, Scientific reports 8 (1) (2018) 6085.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a task-oriented time series imputation evaluation approach that assesses the impact of different imputation strategies on downstream forecasting tasks, rather than just the accuracy of the imputed values. The authors introduce a similarity-based method to efficiently estimate the impact of imputed values on downstream task performance, and develop a time series imputation framework that combines the advantages of different imputation strategies. The paper also discusses several axioms that the proposed method satisfies, such as the efficiency, self-explanation, and symmetric zero axioms.\n\nSTRENGTHS:\n1. The paper proposes a task-oriented time series imputation evaluation method, focusing on evaluating the impact of different imputation strategies on downstream prediction tasks, rather than just evaluating the accuracy of imputed values. This approach is aligned with practical application needs.\n\n2. The authors propose a similarity-based method that achieves a balance between performance and computational cost, and can effectively estimate the impact of imputed values on the performance of forecasting tasks.\n\n3. The authors have developed a time series imputation framework that combines the advantages of different imputation strategies, and achieves better performance on downstream forecasting tasks.\n\nWEAKNESSES:\n1. The authors state in the abstract that missing values can be observed in many time series analysis tasks, but the task-oriented imputation evaluation method proposed by the authors is only designed for forecasting tasks. In the problem definition, the label of a time series dataset is similarly defined as a time series, however in the classification task, the label is a discrete value. In addition, for some unsupervised learning time series tasks, such as clustering [1], missing values can affect the performance as well. In conclusion, the authors' method is too limited in its scope of application compared to other imputation methods.\n\n[1] Time series cluster kernel for learning similarities between multivariate time series with missing data. PR 2018.\n\n2. The authors' motivation is to provide imputation values that are more beneficial to downstream forecasting tasks, while ignoring the accuracy of the imputation values themselves. However, for the time series imputation task, performing accurate imputations is also one of the non-negligible purposes [2]. The authors ignored the requirement for imputation accuracy when designing their methodology, and did not show experimentally whether their method would reduce imputation accuracy.\n\n[2] Deep learning for multivariate time series imputation: A survey. Arxiv 2402.04059.\n\n3. The authors' ultimate aim is to make more accurate predictions in the presence of missing values, which is highly relevant to the study of robust time series forecasting [3,4]. However the authors don't mention them in the paper and don't compare them as baselines in their experiments. In addition the authors should discuss the advantages and disadvantages of imputation-based and missing value robust-based forecasting methods.\n\n[3] Weakly Guided Adaptation for Robust Time Series Forecasting. VLDB24.\n\n[4] RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies. ICLR 24.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel strategy to evaluate time series imputation methods based on their impact on downstream tasks, without requiring multiple model retrainings. The proposed method leverages a similarity calculation to estimate the effect of imputed values efficiently, balancing performance and computational cost. Furthermore, the authors introduce a framework that combines various imputation strategies to enhance downstream task performance, demonstrating significant improvements in forecasting accuracy.\n\nSTRENGTHS:\n1. Unlike traditional methods that focus solely on data restoration, this paper shifts the evaluation criterion to the performance of imputed data in subsequent forecasting tasks. This innovative perspective addresses a significant gap in the field and offers a more practical assessment of imputation quality.\n\n2. The research is supported by thorough theoretical analysis and experiments on multiple benchmark datasets. The authors provide clear justifications for their approach and present detailed comparisons with existing methods.\n\nWEAKNESSES:\n1. Although the paper introduces a retrain-free method to estimate the impact of missing labels, it does not thoroughly discuss the computational efficiency of this approach compared to traditional methods, particularly for large-scale datasets.\n\n2. The motivation of this paper is not very clear: why considering the impact of missing values as labels in downstream forecasting tasks is a valuable formulation, and why is this the optimal way to combine existing advanced methods\n\nQUESTIONS:\n1. Is the proposed method a post-processing evaluation or an in-processing method during model training?\n2. Is the method only comparing one pair of models at a time?\n3. What is the approximation error of the indicator function and how does this impact the imputation performance?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the imputation of missing values in time series. By noticing that different imputation methods might affect the downstream forecasting tasks, this paper proposes the imputation evaluation approach regarding the downstream tasks' performance. \nThen, the authors also developed some methods to improve the efficiency of the evaluation approach and did experiments against some baselines.\n\nSTRENGTHS:\n1. This paper proposed an evaluation approach to time series imputation methods from the perspective of the impact on the downstream forecasting task. This is a seemingly interesting perspective. \n\n2. The proposed method has mathematically formulated the intuition of evaluating imputation methods based on the prediction performance, and the authors developed the approximate method to improve the efficiency.\n\nWEAKNESSES:\n1. The evaluation criteria defined in (1) are based on binary comparison and do not look realistic. In real-world applications, the difference in prediction errors between the two methods can be marginal and insignificant, and calculating this binary indicator makes little sense. Meanwhile, in real-world data, the percent of missing values would be very low, and it is doubtful that imputed missing values would lead to significant differences in downstream predictions, especially considering that there are various techniques to regularize models for more robust predictions.  \n\n2. Some mathematical presentation is confusing. For instance, how is Eq.(2) derived?  $f(X_k^v, \\theta)$ is irrelevant to $y$, and thus why is there the term of the derivative of  $f(X_k^v, \\theta)$ w.r.t. $y$ in Eq.(2)\n\n   According to line 148,  Eq.(3) is for approximating $\\frac{\\partial f(X_k^v, \\theta)}{y_{i, l}}$, while on line 156, it presents the substitute of $\\frac{\\partial f(X_k^v, \\theta)}{y_{i, l}}$ again. This is very confusing. \n\n3. The experiments have unjustified and confusing setups and results. For instance, the description between lines 259 and 267 is very unclear, e.g., what is the label in a time series prediction task? why specifically replace 40% of data? \nIn Fig. 1, what is the percentage w.r.t. on the x-axis?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNo potential negative societal impact is found in this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}