Masking the Gaps: An Imputation-Free Approach to Time Series Modeling with Missing Data

Authors: Abhilash Neog, Arka Daw, Sepideh Fatemi Khorasgani, Aanish Pradhan, Mary E. Lofton, Bennett J. McAfee, Adrienne Breef-Pilz, Heather L. Wander, Dexter W Howard, Cayelan Carey, Paul Hanson, Anuj Karpatne
Primary area: learning on time series and dynamical systems
Keywords: Time-Series, Deep Learning, Masked Autoencoders, Missing Data, Data Imputation

Abstract:
Modeling time series is important in a variety of domains, yet it is  challenged by the presence of missing values in real-world time-series datasets. Traditional frameworks for modeling time-series with missing values typically involve a two-step process, where the missing values are first filled-in using some imputation technique, followed by a time-series modeling approach on the imputed time-series. However, existing two-stage approaches suffer from two major drawbacks: first, the propagation of imputation errors into subsequent time-series modeling performance, and second, the inherent trade-offs between imputation efficacy and imputation complexity. To this end, we propose a novel imputation-free approach for handling missing values in time series termed {Miss}ing Feature-aware {T}ime {S}eries {M}odeling ({MissTSM}) with two main innovations. {First}, we develop a novel embedding scheme that treats every combination of time-step and feature (or channel) as a distinct token, encoding them into a high-dimensional space. {Second}, we introduce a novel {Missing Feature-Aware Attention (MFAA) Layer} to learn latent representations at every time-step based on partially observed features. We evaluate the effectiveness of MissTSM  in handling missing values over multiple benchmark datasets using two synthetic masking techniques: masking completely at random (MCAR) and periodic masking, and a real-world missing-value dataset.

Decision: reject
Reviewer scores: 5, 3, 3, 8

Reviews and discussion:
<thread>
creation date:
1729283500634

thread_type:
official_review

<thread_object>
creator: reviewer_7jfe

<content>
summary:
This work proposes an imputation-free approach for handling missing values in time series termed Missing Feature-aware Time
Series Modeling (MissTSM). MissTSM is composed of two primary modules. The first, Time-Feature Independent (TFI) Embedding, encodes each combination of time-step and feature into a high-dimensional space. The second module, the Missing Feature-Aware Attention (MFAA) Layer, is a novel mechanism designed to capture latent representations at each time-step by leveraging partially observed features. Empirical evaluations conducted under Masking Completely at Random (MCAR) and periodic masking validate the effectiveness of MissTSM in modeling time series data with missing features.

strengths:
1. The intuition behind the idea is clear, effectively circumventing the imputation step typically used in traditional methods for handling missing data.
2. The presentation is strong, with clear figures that make the content easy to read and understand.
3. The experiments are comprehensive, covering both time series forecasting and classification, and include an insightful ablation study.

weaknesses:
1. While the motivation is clear, the experimental results do not show significant improvement over the other baselines.
2. Time series missingness is a well-studied and widely researched field, making it difficult to assess the significance of the contribution here.
3. This work primarily represents an empirical study rather than a theoretical advancement.

questions:
1. The author mentions two drawbacks of imputation: the propagation of imputation errors and the trade-off between imputation efficacy and complexity. Could the author clarify how MissTSM circumvents these drawbacks? I'm confused because: (1) it seems that errors in TFI would aggregate and influence MFAA, and (2) there's a clear trade-off between complexity and efficiency—for example, adding a dynamics module may improve performance but also increase complexity. Could the author clarify to what extent MissTSM addresses these two challenges?
2. Please clarify the intuition behind the 2D Positional Encodings (Line201-210) and explain why and how equations (2) and (3) are used in this context? Additionally, could the author elaborate on why these encodings provide an effective representation?
3. Can the authors explain the rationale for considering 60%-70% missing values in the experiments? Is it because MissTSM shows more significant improvements with a higher percentage of missing values? Typically, a missing range of 10-40% is more common, so additional clarification would be helpful.
4. My main concern lies in the significance of the experimental results. It appears that MissTSM's performance heavily depends on the dataset. For example, MissTSM only demonstrates comparable results on the ETTm2 and Weather datasets (Table 1, Figure 5), and the ablation study doesn’t clearly show the effectiveness of TFI and MFAA. Could the authors explain this? If additional results could be provided, specifically concerning MissTSM without TFI-Embedding but with MFAA, and address the previous questions, I would be happy to increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730457925576

thread_type:
official_review

<thread_object>
creator: reviewer_g7mn

<content>
summary:
To address the missing values in time series datasets, this work considers the imputation-free time series modeling. MCAR and periodic masking are considered in experiments by comparing with two stage baselines, consisting of imputation and modeling approaches.

strengths:
S1. Various imputation baselines are considered in experiments.

S2. Time series modeling is important.

S3. Time series forecasting and classification are considered in experiments.

weaknesses:
W1. The introduction is not well-written, which seems more like the related work. It is suggested to further polish the manuscript, to improve the readability.

W2. In the Introduction, the authors claim that all existing works for time-series modeling with missing values are two stage approaches, where missing values are first imputed, followed by feeding the imputed time-series to a modeling approach. Unfortunately, there already exists existing study [m1] considering the robust time series modeling over low quality data, where missing values and anomalies are associated with low importance in modeling, instead of treated as a two stage framework. Therefore, the motivation is unpersuasive, claiming that all the existing works are two stage approaches.

W3. In practice, in addition to the missing values, there also exist various data quality problems, e.g., anomaly, errors, staleness, etc. Only focusing modeling the time series data with missing values is insufficient to serve real applications. Please discuss how the approach might be extended or adapted to handle other types of data quality problems, or explain why you chose to focus specifically on missing values.

W4. MCAR and periodic masking are widely used operations in time series imputation, which are surprising to be proposed by this manuscript, at the end of the Introduction.

W5. Various imputation methods are considered in the experiments, but the most related works, i.e., robust modeling methods such as [m1], are totally ignored.

W6. The SOTA imputation baselines are ignored in the comparison, such as [m2]. Please include the comparison with it.

W7. In addition to the MCAR and periodic masking, MAR and MNAR are also widely considered in the experiments.

[m1] Cheng, Hao, et al. "RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies." The Twelfth International Conference on Learning Representations.

[m2] Tashiro, Yusuke, et al. "Csdi: Conditional score-based diffusion models for probabilistic time series imputation." Advances in Neural Information Processing Systems 34 (2021): 24804-24816.

questions:
W1-W5
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638180504

thread_type:
official_review

<thread_object>
creator: reviewer_uwvs

<content>
summary:
This paper introduces a DL-based time-series method called MissTSM, which is designed to handle missing values in time series data without relying on imputation techniques. The key components of MissTSM include a Time-Feature Independent (TFI) embedding that treats each time-step and feature combination as a distinct token and a Missing Feature-Aware Attention (MFAA) layer that learns latent representations at each time-step based on the observed features (using missingness indicator when assigning the attention values). The authors evaluate MissTSM on multiple benchmark datasets using synthetic (MCAR and periodic) and real-world missing value scenarios. The results show that MissTSM achieves comparable performance to a wide variety of time-series classification and forecasting methods (equipped with state-of-the-art imputation methods).

strengths:
-	The paper is in general well-written and easy to follow.

weaknesses:
-	The authors have presented a limited perspective on time-series methods that avoid imputation. A broader range of approaches exists to tackle missing values in time series. One such approach involves augmenting the input data with indicators for each feature-time variable, which can be particularly informative when missingness is not random. Another line of research focuses on handling multivariate time series with irregular time intervals. This includes methods like GRU-D [A], which directly incorporates missingness indicators and time intervals as inputs, and ODE/SDE-based approaches [B]-[D] that inherently model such irregular time series. 
-	The authors should consider a broader range of baseline models. This could include evaluating existing time-series methods that incorporate missingness indicators as input or exploring some of the mentioned techniques designed for handling irregular time intervals.
-	Treating each feature at each time point as a separate embedding and treating missingness indicator to bias the attention score are not technically novel. Moreover, the performance gain seems marginal and sometimes being lot worse than the benchmarks. 
-	While the utility of the imputation methods is often measured by the performance in downstream tasks, imputation is not designed to achieve higher discriminative/prediction power. More simpler imputation methods – such as mean/mode imputation or last observation carried forward – should be included as well in the experiments. 

References
- [A] Che et al., “Recurrent Neural Networks for Multivariate Time Series with Missing Values,” Scientific Reports, 2018.
- [B] Chen et al., “Neural ordinary differential equations,” NeurIPS 2018.
- [C] Rubanova et al., “Latent ODEs for Irregularly-Sampled Time Series,” NeurIPS 2019.
- [D] Park et al., “Neural Stochastic Differential Games for Time-series Analysis,” ICML 2023.

questions:
-	It seems that attention scores for the masked feature-time points can be naturally removed when computing those self-attention scores. What is the benefit of using the mask indicators as the bias terms rather than explicitly removing those from computation?
-	Why the quarries do not depend on the time point whereas the keys and values do?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731515331956

thread_type:
official_review

<thread_object>
creator: reviewer_szvs

<content>
summary:
The paper presents an imputation-free approach called MissTSM to handle missing data in time series. The MissTSM framework is innovative, and the experiments demonstrate its effectiveness.

strengths:
1. The paper is well organized and clearly articulates the methodology behind MissTSM;
2. The authors provided a link to their source code for reproducibility;

weaknesses:
1. Three datasets for each task are limited and not enough to verify a framework;
2. Too many experiment setup details are missing. For example, a). hardware information is missing in the computational cost comparison; b). the authors do not mention how they determine models' hyperparameters;

questions:
1. How did the hyperparameters of the baseline models get optimized? Please discuss and reveal more details regarding this point.
2. Baseline models are missing in the code repository. Did the authors reproduce the baseline methods by using their official code or other unified Python libraries (e.g.  Time-Series-Library [1], PyPOTS [2])? Data processing is quite different across imputation algorithms while unified interfaces can ensure fairness in the experiments.
3. Now that only three datasets are for the forecasting task, why select both ETTh2 and ETTm2 from ETT datasets for experiments? Generally, datasets from different application domains are more reasonable.

### References
[1] https://github.com/thuml/Time-Series-Library

[2] Wenjie Du. PyPOTS: a Python toolbox for data mining on Partially-Observed Time Series. In KDD MiLeTS Workshop, 2023. https://github.com/WenjieDu/PyPOTS
</content>
</thread_object>

</thread>
<thread>
creation date:
1732478766921

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Review Comments

comment:
We sincerely thank all the reviewers for providing constructive feedback. We are encouraged that the reviewers found 
- Our proposed idea to be intuitive and clearly motivated [*Reviewer 7Jfe*] 
- Our evaluation to be comprehensive with multiple imputation baselines and time-series tasks, including forecasting and classification [*Reviewers G7Mn, 7Jfe*] 
- Our paper to be well-written and easy to follow [*Reviewers sZVS, UWVS, 7Jfe*] 

Before addressing each of the reviewer’s comments individually, we address some of the shared concerns raised by the reviewers in our general response below: 

**Comment 1: Improvement in MissTSM’s performance appears marginal (UWVS, 7Jfe)** 
> We acknowledge that MissTSM’s performance improvement over other baselines may appear marginal in some scenarios and is not always the best-performing model across the extensive set of experiments considered in this paper, which is difficult to achieve given the natural variations expected in most empirical analyses. However, MissTSM shows *competitive performance* close to the best-performing model *consistently* across varying datasets, horizon windows, masking schemes, and masking fractions. 

> To quantify this claim, we have now added a new column in Table 1 that shows the average rank for every method across all three forecasting datasets and horizon windows for every synthetic masking scheme (using a fixed masking fraction). From Table 1, we can see that MissTSM is a *close second* in all three masking schemes with average ranks of 1.9, 2.7, and 4.1 for no masking, MCAR masking, and periodic masking, respectively. However, the average ranks of Table 1 still do not comprehensively summarize performance trends across all experiments of the paper. Instead, the performance of MissTSM varies across datasets (we are consistently better than other baselines on ETTh2 but not on Weather), horizon windows, and masking fractions, as presented in Figures 4 to 7 of the main paper. We make the following **key remarks** summarizing the comparison of MissTSM with baselines across all experiments of our paper:
> 1. MissTSM shows significant performance gains for larger masking fractions such as 90% (see Figure 4) and larger horizon windows such as 720 (see Figure 5). This suggests that MissTSM is particularly useful in extreme testing environments where other baseline methods break down. 
2. MissTSM shows worse performance than baselines only in two scenarios: EMG dataset (classification) with masking fraction 0.2, and Weather dataset (forecasting) for smaller horizon windows (less than 720). This suggests that there is scope for improving MissTSM for smaller masking fractions and horizon windows. 
3. While the 6 benchmark datasets used in our work are currently the de facto standards in the time-series modeling community, these datasets also have their weaknesses. For example, we can see from Table 1 that the spline interpolation generally performs better if not at par with the more complex SAITS imputation, suggesting that these datasets are simpler with regular periodic structures than what we may expect in real-world settings.  
4. On Physionet, a real-world dataset with 80% missing values, we can see from Figure 8 that MissTSM performs significantly better than all other baselines. This suggests the practical utility of our work on datasets with real patterns of missing values, as opposed to datasets generated from synthetic masking schemes that may have limitations.

> Apart from these four remarks on the performance of MissTSM, we would also like to emphasize that MissTSM is imputation-free and hence (a) does not suffer from the computational costs of imputation, and (b) does not involve error propagation from imputation to forecasting (see Appendix C), making it an ideal choice for handling missing values in multivariate time-series data. 

**Comment 2: More baselines need to be included in the experiments (UWVS, G7Mn)**
> We have performed new experiments with GRU-D and Latent ODE methods for classification and forecasting tasks, as suggested by [UWVS] and with RobustTSF as suggested by [G7Mn]. Our results indicate that MissTSM significantly outperforms these baselines in their respective tasks and datasets.  

We hope that our responses to the individual review comments provided below address the main concerns of the reviewers. If we missed any detail, we will be happy to provide more clarifications during the discussion period. If our responses have adequately addressed your concerns, we kindly request that you consider revising your scores. Thank you very much for your time and effort.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an imputation-free approach called MissTSM for handling missing values in time series data, using Time-Feature Independent embedding and Missing Feature-Aware Attention mechanisms. The paper has a clear presentation and comprehensive experimentation. However, as raised by the reviewer, the novelty and the experiments are somewhat limited.