GITAR: GENERALIZED IRREGULAR TIME SERIES REGRESSION VIA MASKING AND RECONSTRUCTION PRETRAINING

Authors: Yu Wu, Ting Dang, Cecilia Mascolo
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Time series, irregular time series, self-supervised learning

Abstract:
Multivariate time series regression, encompassing forecasting and interpolation, is crucial for numerous real-world applications, particularly in healthcare, climate science, ecology, and others. While recent work has focused on improving modeling for time series regression, two main limitations persist. First, the prevalence of irregularly sampled time series with missing values poses significant challenges.
For instance, healthcare applications often involve predicting future or missing observations from irregular data to enable continuous patient monitoring and timely intervention. As current approaches mainly rely on the assumptions of regular time series such as strong periodicity, when applied to irregular ones they exhibit performance degradation. Second, while some state-of-the-art methods (SOTA) do model irregularity and perform regression tasks on irregular data, they are often trained in a fully supervised manner. This limits their ability to generalize easily to different domains (e.g., training and testing datasets with different numbers of variables). To address these challenges, we propose GITaR, a Generalized Irregular Time Series Regression model via masking and Reconstruction pertaining mechanism, aiming to capture the inherent irregularity in time series and learn robust, generalizable representations without supervision for downstream regression tasks. Comprehensive experiments on common real-world regression tasks in healthcare, human activity recognition, and climate science underline the superior performance of GITaR compared to state-of-the-art methods. Our results highlight our model’s unique capability to generalize across different domains, demonstrating the potential for broad applicability in various fields requiring accurate temporal prediction and interpolation.

Decision: reject
Reviewer scores: 5, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730152794279

thread_type:
official_review

<thread_object>
creator: reviewer_hwl2

<content>
summary:
The paper introduces a generalized model for irregular time series regression, leveraging a masking and reconstruction pretraining mechanism. Experimental evaluation is given to show its performance.

strengths:
The proposed model utilizes a masking and reconstruction pretraining mechanism to capture irregular patterns without requiring labeled data, enabling robust and generalized representations.

weaknesses:
1. Although some modules are integrated, the motivation is missing in the paper, such as why this specific masking approach is useful over others for handling irregular time series. The authors are suggested to give more theoretical discussions and an intuitionistic example to strongly motivate the work. Note that in terms of the results reported in Table 3,  the PrimeNet method is able to obtain comparable performance than that of the proposed model in most cases.

2. What is the distinction between the two critical research questions the authors propose? They appear to address the same issue, as a robust self-supervised learning framework inherently implies the ability to adapt to different domains. Could the authors clarify how these questions are fundamentally different?

3. The description for Fig1 lacks clarity, making it difficult to understand the intended meaning. It is unclear whether “v1-v3” represents different variables, tasks, or something else. Additionally, if the figure is meant to illustrate a different domains/transfer learning issue, a more detailed explanation would help readers grasp the specific context and purpose of the comparison.

4. From the method and the experimental part, I cannot find how the proposed model learns and captures the claimed local semantics (i.e., irregular patterns) and global temporal dependencies. Could the authors provide more detail on how these aspects are addressed?

5. In addition to self-supervised learning, there are numerous deep generative models for irregular time series, including GANs and diffusion models [1-3]. The authors have not discussed or compared with these types of methods. Besides, the metrics used in Table 3 are not clearly specified. Also, while the authors claim “across PhysioNet to MIMIC,” in the caption of Table 3, it does not seem to present results specifically for the PhysioNet dataset.


[1] Tashiro, Yusuke, et al. "Csdi: Conditional score-based diffusion models for probabilistic time series imputation." Advances in Neural Information Processing Systems 34 (2021): 24804-24816.

[2] Zhang, Weijia, et al. "Irregular Traffic Time Series Forecasting Based on Asynchronous Spatio-Temporal Graph Convolutional Networks." Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.

[3] Li, Yan, et al. "Generative time series forecasting with diffusion, denoise, and disentanglement." Advances in Neural Information Processing Systems 35 (2022): 23009-23022.

questions:
As provided in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469261858

thread_type:
official_review

<thread_object>
creator: reviewer_saec

<content>
summary:
In this paper, the authors propose an architecture and training scheme for
irregular time series. They propose to patch and mask not specific amounts of
data points, but times-spans that can include more points in denser and lesser
points in sparser regions. Furthermore, the authors propose an architecture
which first use attention inside the patches (Called Irregular Patch Encoding)
and then an attention mechanism between different patches (Global Temporal
Encoding).

The authors test their approach both on regular regression task and on
out-of-domain forecasting.

strengths:
+ The Architecture and the design decisions seem useful and reasonable. 
+ The results are promising, as Gitar outperforms t-patchGNN which outperforms a lot of the ODE Models.
+ Ablation study shows that indeed all components are needed

weaknesses:
- My strongest critique point is the paper writing. The paper on the hand contains a lot of self-advertising sentences which distracts from the "real content". Example:
  - Page 4, Starting at 207: "The first module preserves the original irregular patterns
    while masking and segments the multivariate time series into synchronized, channel-independent
    patches. This approach ensures effective learning of local semantics (i.e., irregular patterns) within
    patches and enhances generalization capability via channel-independent design, reducing process-
    ing complexity, and mitigating long-range information loss"
    -> Why not simply say: "For each channel, we randomly mask a time-span of range q_d and we patch by the time-span lengths instead of    by the amount of points in the patch."
   - On the other hand, a lot of crucial information is missing in the paper:
     - The decoder: How does it look like? Is it an MLP? Just one layer?
     - What does the following mean:
        "This pre-trained reconstruction model R will then be fine-tuned for downstream tasks such as
        forecasting or interpolation on irregular multivariate time series data. To validate its generalization
        capability, we fine-tune R across various domains, encompassing different datasets with varying
        input channels and irregular patterns"
        How exactly does that look like? How does the fine-tuning look like? How do you do for example forecasting? With a forecasting MLP- head as in PatchTST? Or do you just assume that the full forecasting horizon is masked and you apply your normal reconstruction?
      - How do you mask? Randomly select a time-span? How long is the time-span? Does masking means zeroing out? I would formalize and explain such crucial points in detail as instead of just mentioning them shallowly.
    - I was also not able to have a look at the code to answer these questions by myself. I could unfortunately not find a link in the paper and also I could not see any supplementary material.

- In the ablation study, I could not find the answer to the following question: Do you need pre-training at all? How good is your model when being only trained on target forecasting tasks?
- Missing comparison method: Can you compare your method against (https://ojs.aaai.org/index.php/AAAI/article/view/29560)? This paper also outperforms all ODE-based models and may be competitive to your model.

## Final Conclusion
The results look promising and I thus think this paper has indeed potential. However, I think the paper writing has to be modified before publishing, as a lot of crucial details are missing.

questions:
- How does your model perform without any pre-training at all? When being just trained on the forecasting tasks?
- How did you collect your baseline results? Do you re-implement all baselines? Because your results for t-PatchGNN, are different then the results reported in the t-PatchGNN paper itself, have a look the table: (https://github.com/usail-hkust/t-PatchGNN). Regarding the results here, t-Patch GNN would be more competitive to your method.
- How exactly does your fine-tuning approach work?
- How exactly does your decoder look like?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557284486

thread_type:
official_review

<thread_object>
creator: reviewer_oigx

<content>
summary:
The paper proposes a new generalized regression framework for irregular time series data.  The model is trained in the self-supervised mode by relying on a patching and masking mechanism that permits to define and learn the model on a variety of regression tasks. The task sampling schema used for training the model is proposed. After the self-supervised training the model can be used to support a variety of downstream regression tasks.  The model is tested on prediction and interpolation tasks on four datasets showing promising prediction performance against baselines.

strengths:
Originality: A novel attempt to train the models supporting regression inferences on irregularly sampled time series data. Also the idea of using mTAN approach for encoding irregular time series patches to regular representation is interesting and can be applicable to future time series neural network architectures

Significance: Irregularly sampled time series data are important for many practical applications, most prominently healthcare. The objective of developing models that are able to support many downstream regression tasks  is well justified.  

Experiments: •	Authors show a promise of the GITaR method against baseline methods in interpolation, forecasting and transfer learning tasks on multiple irregular time series datasets.

weaknesses:
**** Methodology ****

The novelty of the patch schema is unclear and very much resembles recent work on T-PatchGNN.

Weijia Zhang, Chenlong Yin, Hao Liu, Xiaofang Zhou, and Hui Xiong. Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. ICML, 2024 I 

It appears that the Global Temporal Encoding $E^i$ is over individual time series and does not capture dependencies across different time series. It is unclear if/how these patterns and dependences are captured in the model. 

The description of the decoder architecture is missing in the paper. It is understandable that the model transforms the irregular observations to a regular latent representation. It is unclear how the model then makes the predictions of time series variables at irregular times from this regular latent representation.


**** Experiments ***** 

Evaluating the performance of models designed for regular time series (Re) against those specifically tailored for irregular time series (IR) on irregular time series regression tasks  doesn’t add much value to the analysis. Perhaps authors intend to just highlight the importance of incorporating time explicitly in the model is important, which makes sense. However, including only 3 IR model baselines in the main results section is limiting. More IR baselines from the families: RNN-based (e.g., GRU-D [1]), Graph-based (e.g., Raindrop [2]), and ODE-based (e.g., ContiFormer [3]) should be considered in the experimental analysis for robust evaluation.

[1] Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values.

[2] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series.

[3] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contiformer: Continuous-time transformer for irregular time series modeling, 2024.

Interpolation experiments are performed only on Physionet dataset, and do not have MAE metric in the final comparison.

The results section on generalization capabilities is promising but limited to draw meaningful conclusions from. It is unclear why training is performed only on the Physionet dataset and tested on the rest. It would be worthwhile to see if the benefits of this generalization hold up for when the model is pretrained on other datasets or on a combination of them and tested on the remaining set. For example, train on MIMIC and test on the rest; train on MIMIC+Physionet and test on the rest would help to evaluate GITaR’s generalization capabilities.

**** Code *****

The code necessary for reproducing the reported results is not included in the submission.

questions:
- Can you explain the difference of patching mechanism in T-PatchGNN and Time-sensitive Patching proposed in your work? 

- The text for section on generalization capabilities can be improved. Can you clarify how exactly the results are computed? Is there a transfer learning step (i.e., training) on the target domain? Is it the forecasting task? If so, what is the forecasting horizon? Is it comparable to the forecasting results (i.e., Table 2)?

 Are there any special or extreme cases where task sampling schema for supporting SSL proposed in the paper may fail? If the masking of observation has uniform timespans, a sparsely sampled time series will have a smaller number of samples masked than a densely sampled time series. Because of these unequal masked observations, the model is biased to learn densely sampled time series better than sparsely sampled one.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640169321

thread_type:
official_review

<thread_object>
creator: reviewer_x6xd

<content>
summary:
In order to handle irregular multivariate time series data, the paper presents GITAR, a self-supervised learning (SSL) framework that combines irregular-temporal encoder, time-sensitive patching, and irregular-sensitive masking. The suggested model aims to address drawbacks of current approaches, including limited adaptability to new domains and domain-specific inter-channel dependencies. According to tests conducted on a number of real-world datasets, GITAR outperforms other state-of-the-art (SOTA) techniques in irregular time series regression tasks.

strengths:
- The paper addresses a relevant and challenging issue in irregular time series analysis, which is applicable across a variety of fields. 

- The framework is technically well-executed with clear mathematical formulations. The irregular-time attention mechanism (ITA) and continuous time embeddings are rigorously defined, providing robustness for learning temporal dynamics.

- The experiments cover multiple datasets, which helps evaluate the framework across different irregular time series contexts.

weaknesses:
- The theoretical foundations of the proposed methods are weakly articulated. The notation and assumptions behind the continuous embeddings and ITA mechanism are not thoroughly explained. Key concepts, such as the initialization and constraints of embedding parameters, are underexplored.

- The model’s generalization claim would be better supported by comparing it to a broader array of baselines that specifically address irregular time series and variable sampling rates. 

- The model’s main innovations appear to be modest adaptations of existing techniques (e.g., masked autoencoders). Without stronger theoretical or empirical evidence, GITAR’s contribution to the field of irregular time series analysis seems limited.

questions:
- In case of continuous time embeddings, why are specific sinusoidal terms chosen over other possible basis functions? What benefits do $\omega$ and $\alpha$ provide in capturing irregular periodicities? Is there theoretical support indicating they are optimal for this purpose? How do these embeddings behave under highly irregular sampling intervals?

- What theoretical or empirical basis supports the choice of attention as the primary mechanism for learning temporal dependencies in irregular time series? Would other mechanisms (e.g., kernel-based methods,  spectral embeddings, or Neural ODE/SDE) potentially provide similar or improved performance?

- Is there any analysis or empirical validation demonstrating that these attention weights maintain meaningful interpretations across different irregularity patterns?

- The choice of patch size and masking ratio is critical for the performance of GITAR. How does the model’s accuracy change as these hyperparameters vary? Additionally, how are these hyperparameters selected for datasets with different irregularity patterns?

- Given the high complexity of long time series sequences, how does GITAR address computational challenges associated with global attention mechanisms?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733136015192

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Genearl Response

comment:
Dear AC and Reviewers:

We sincerely thank the reviewers for their insightful comments and discussions, which have significantly helped us to address unclear aspects and improve the quality of our paper. Below is a summary of the questions and responses following the reviewer's subsequent comments:

- **Performance Difference** For *Reviewer X6xD*, *Reviewer oiGX* who raised concerns about performance mismatches between our results and the baseline-reported results, we have clarified that the discrepancy is mainly due to the different metrics used. We use RMSE, whereas the references use MSE. When converting to the same evaluation metric of MSE, our proposed method consistently outperforms the referenced baseline systems, e.g., with relative improvements of 12.7% for interpolating 10% of missing data. We chose to use RMSE to better highlight the performance differences among models. Notably, we have ensured a fair comparison by reproducing the baselines according to their official guidelines and maintaining consistent settings, such as using the same number of variables.

- **Baselines** For *Reviewer hwL2*, who suggested a more comprehensive comparison with GAN- and diffusion-based models, we have carefully reviewed the current literature and identified only one relevant study, whose performance is also surpassed by one of our baseline methods. Additionally, there is no strong evidence suggesting that diffusion-based methods are more effective and suitable for irregular time series regression tasks [1]. More importantly, our study focuses on designing SSL frameworks that effectively learn from irregular time series data and generalise to various unseen domains. We followed the standard evaluation protocols of pre-training, fine-tuning, and evaluation, and included state-of-the-art baselines and datasets, consistent with existing work in this area [2,3]. Therefore, comparisons to general diffusion-based methods are outside the scope of this study.

- **Methods**
    - **Multivarite Dependency:** For *Reviewer oiGX* who raise the question of why our channel-independent approach outperforms methods that capture channel correlations, we have provided both the experimental results and conceptual reasoning. Our experiments addressed the reviewer's concern regarding the large dataset size, confirming that it is not the reason for the benefits of our designed channel independence. We have clarified the two reasons as follows: i)Multivariate irregular time series often exhibit distinct sampling rates and dynamic missing patterns across channels, which existing models like Crossformer, designed for regular time series, fail to handle effectively. Learning each variable independently allows our model to bypass these limitations, leading to superior performance. and ii) To achieve generalization across unseen datasets with varying irregularities and channel configurations, our approach avoids pre-training on fixed channel dependencies. For instance, capturing dependencies in 12-lead ECG data would hinder transferability to 2-channel PPG data due to differences in channel types and patterns. Channel-independent learning ensures robust generalization across diverse datasets. 

    - **Theoretical Analysis:** For *Reviewer X6xD* who suggested theoretical analysis of parameter configuration, we have provided the practical insights of the parameters constraints. Our paper primarily focuses on empirical analysis supported by comprehensive experiments, and we will clarify the rationale for parameter selection based on real-world dataset characteristics.

Finally, we want to emphasize that the novelty and objective of our work lie in designing a self-supervised learning framework that generalizes across unseen domains with varying irregular patterns, ratios, and numbers of signal channels. This is the first study aimed at transferring a pre-trained model for irregular time series to diverse domains with different data types and channel configurations. Current studies often focus on task-specific models or pre-training and testing on the same dataset due to challenges like data sparsity and unpredictable missing patterns. In contrast, our work presents a promising SSL framework that effectively captures irregular data patterns and transfers them to unseen domains.

Our specific design consists of multiple modules, including an irregular attention module and a temporal dependencies module, within the mask and reconstruction pretraining task. This design effectively captures the intrinsic characteristics of irregular time series. Additionally, we employ a channel-independent training approach to enhance generalization capability across unseen domains.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a new method for irregular time series regression using masking and reconstruction. The reviewers agreed that the paper is well-motivated, the method is technically sound and the experiments contain a variety of datasets. 

However, the paper also suffers from drawbacks including the novelty of the method (a reviewer pointed to T-PatchGNN, another to GAN-based methods as being similar), and several other baselines were recommended (such as diffusion models). While the authors conducted some experiments, they did not test against everything the reviewers recommended. 

Overall, while there might be merits to the paper, I cannot recommend acceptance in its current form.