Title: Rotary Masked Autoencoders are Versatile Learners
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning

Transformer

Masked Autoencoder

irregular time-series

Rotary Positional Embeddings
Abstract: Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables interpolation and representation learning with multidimensional continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and audio, demonstrating that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability to reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's relative position property.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes RoMAE, which extends Masked Autoencoders to handle irregular time-series by adapting RoPE for continuous positions. The key contribution is Continuous Axial RoPE that encodes real-valued timestamps, enabling a single unified architecture to achieve competitive performance across irregular time-series classification, standard image tasks, and interpolation without specialized modifications, demonstrating that general foundation models can match domain-specific approaches when equipped with appropriate continuous positional encoding.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe authors provide formal analysis of position reconstruction capabilities and empirically validate these claims, offering both theoretical understanding and practical guidance on when to use [CLS] tokens for absolute vs. relative positioning.\n2.\tThe paper demonstrates that RoMAE achieves competitive or superior performance on irregular time-series, standard vision tasks, and interpolation benchmarks, providing convincing evidence that a unified architecture can match domain-specific methods.\n3.\tThe proposed method addresses the important challenge of handling irregular sampling in time-series data using standard Transformer components, eliminating the need for specialized architectures.\n\nWeaknesses:\n\n1.\tWhile the application is novel, the main technical innovation—using RoPE with continuous values—is relatively straightforward since RoPE naturally accepts real-valued inputs, and the theoretical analysis mostly confirms expected behavior rather than revealing innovative insights.\n2.\tThe approach is restricted to D ≤ 3 dimensions due to exponential token growth, and while the authors mention computational overhead for varying positions, they provide insufficient analysis of the practical computational costs compared to specialized time-series methods.\n3.\tThe experimental evaluation lacks comparison with recent large-scale time-series models like Time-LLM or other foundation models specifically designed for time series data.\n\nQUESTIONS:\n1.\tHow exactly do the authors handle the continuous extension of RoPE when timestamps have vastly different or when dealing with multi-dimensional irregular data where different dimensions have different temporal resolutions?\n2.\tThe proposed masking appears to be uniform across patches, but irregular time-series naturally have varying information density. Are there any experiments with adaptive masking strategies that account for the irregularity pattern? \n3.\tThe experiments cover various datasets, but can it be characterized how RoMAE's performance varies with different types of irregularity patterns (e.g., completely random sampling vs. burst sampling vs. periodic gaps)?\n\nLIMITATIONS:\n1.\tThe paper lacks discussion of how continuous RoPE approximations in practice might deviate from their theoretical guarantees, especially for extreme timestamp ranges. \n2.\tLimited analysis of how sensitive the method is to key hyperparameters (masking ratio, patch size, p-value for truncated RoPE) across different irregularity patterns.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

comments: SUMMARY:
The paper proposes a variation of rotary positional embeddings (RoPE), which is applied to the MAE architecture. On various benchmark tests, the proposed method shows robust performance.

STRENGTHS AND WEAKNESSES:
Strengths

- The RoPE method is nicely harmonized into the MAE framework. 
- By using Axial RoPE on continuous values and by using patch-level input that comes with real-valued position, it implements the RoPE version that works on irregularly sampled data.
- It shows strong performance on Irregular classification tasks (e.g., ELAsTiCC, UEA datasets), interpolation tasks (e.g., PhysioNet, Synthetic), and irregular regression (Pendulum). 

Weaknesses
- It has been mentioned that models like Latent ODE and HeTVAE can compete with their ability to use continuous time representation. It would be more insightful if these models are analyzed with computational costs. 
- The proposed method shows strong performance on the interpolation task, but extrapolation beyond the scope of the observed data is not well studied. 
- While the experiments were done on some irregular datasets, it is not thoroughly discussed how varying degrees of irregularity affects the performance. Very sparse data is not rare in machine learning applications. 
- The proposed method appears to be somewhat incremental as it's based on the existing Axial RoPE method and patchfication.

QUESTIONS:
- For sparse cases, the patch size might matter as they can sometimes cover a too large area to have a meaningful number of samples, or a too small patch could result in too local representations. How does the proposed method handle such trade-off?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is focused on extending masked autoencoders (MAE) model for handling time-series data, but where the data are sampled at irregular time points. One way to do so would be to integrate position encodings/embeddings. This paper shows that incorporating rotary positional embeddings for continuous position encoding is a good idea. So, adapting and showing the potential value of RoPE to encode continuous temporal positions in MAE is the main contribution of this work. The experiments do show proof of feasibility in that in that on the datasets tested, the model achieves competitive performance without requiring specialized architectural modifications. There is some merit in showing that one model can handle both irregular temporal measurements and standard MAE use-cases.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n\n+As indicated in the summary, I find value in the experiments that a single model can achieve reasonable performance across multiple modalities (irregular time series, images, audio) without domain-specific modifications. \n\n+The use of ROPE, while not very novel, addresses the need of applying MAE models on irregular temporal data. The theoretical analysis does not provide any new insight, but is still OK. \n\n+As a proof of feasibility, I think that the experimental evaluation is reasonable. A few diverse tasks are covered and the model performs well across these applications.\n\n+The approach likely maintains efficiency advantages of MAE although this is not really discussed in deep detail in the paper. \n\nweakness:\n\n-An inability to handle extrapolation is more limiting than the paper acknowledges. Forecasting is one of the core use cases of time-series models and a model that is natively inapplicable to this setting will only be useful in interpolation and representation learning. While this does not mean that such a model is not useful at all, but it does question the claim about versatility highlighted in the paper's title. \n\n- The core technical piece is relatively incremental, substituting RoPE for standard positional encoding in MAE. The paper shows that this works - and so there is value in it -- but it does diminish the significance of the technical contribution.\n\n- Experimental evaluations leave out recent Mamba-based approaches and focus on UEA (why MIMIC, UCR etc were left out is not clear). In terms of positioning, I would have expected the paper to make a stronger case for why state space models are not inherently better suited -- especially when the sequence lengths become much longer. This issue is also relevant given that the paper uses a sequence flattening approach which can make scalability an even bigger issue.\n\nQUESTIONS:\n1. what were the core factors influencing the choice of baselines and datasets? Why were Mamba-based models and other more challenging datasets (where similar to what's described in the paper, irregular sampled data can be derived) not used?  \n\n2. The delta with S5 is small. Is there enough justification (due to significant benefits in other regime) to choose masked autoencoders for this task? There is at least some results questioning whether time-series data are a good fit for transformer based model. \n\n3. How pathological are the irregular sampled data that doing a standard imputation as a pre-processing, they cannot be fed directly to Chronos or TimesFM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Rotary Masked Autoencoder (RoMAE) adds a number of positional encoding improvements into Masked Autoencoders, studying in detail the variations of behavior that the resulting positional encodings allow. In a broad series of experiments across regular and irregular time-series, images, and audio, RoMAE is compared to existing baselines, quantified empirically, and studied technically to show the benefits of adding positional information into the MAE framework. This results in a simplification and unification in terms of architecture, compared to baselines in the respective domains studied.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper features extensive empirical findings, experimental results, and extensions into a broad appendix with many interesting findings. The discussions about the subtlety of including the [CLS] token are practically relevant, and Figure 1 is a great introduction to the method, and the overall early description and background are clear. The results make sense and are compelling for the domains, many sections help the reader understand the work, and the RoMAE architecture seems useful and potentially simple to reproduce for new applications. The work seems broadly applicable to many areas given the generality of the MAE architecture, and the authors do a good job of conveying the various configurations of RoMAE that may be applicable to a given application area.\n\nWeaknesses:\nOne drawback from the broad swath of experimental and technical findings here, is that the paper flow feels a bit convoluted. I would strongly encourage the authors to re-visit the narrative structure of the paper, and re-organize. Delegating a number of core results tables to the appendix, while also having a half-page dedicated to irregular time-series, and numerous tables surrounded by empty white space, shows that a deep-clean in terms of paper format can both increase the information density of the paper, and also improve the flow for interested readers. Details about dataset construction are more appropriate for appendix material (beyond brief discussion in the main paper), rather than core findings.\n\nThe description of how to apply continuous axial RoPE to irregular time-series is a bit unclear to me, and would be greatly boosted by a figure showing the irregular time-series processing. Section 4.3 seems to take up a lot of space in the paper to describe a simple (but useful) finding, and might do with some changes to compress the information.\n\nGiven the core of MAE is fundamentally about representation learning, some qualitative exploration of learned representations could be a useful bridge between the detailed discussion of CLS token and absolute position, and dataset specific benchmarks in the later sections. Similar experiments were done in both VQ-VAE and MAE using linear probing, and may be useful to guide ideas in this direction.\n\nA second drawback from the breadth of experiments, is that the comparisons (generally) lack depth. It may be out of scope for this revision round, but consider a more direct application of this approach to \"standard\" benchmarks, adding direct classification scoring to the Tiny ImageNet results and adding in numerous relevant benchmarks, adding these representations as grafted representations to a large language model for multimodal tasks (similar to approaches like miniGPT, LLaVa, Freeze-Omni, and so on), or other benchmarks (direct comparisons to ViT, even if a small one, on some standard tasks). As it stands, the existing benchmarks seem time-series focused, but time-series model benchmarking usually covers an even larger set of datasets, so the paper currently sits in the middle of two camps, and stronger multimodal benchmarking would help me improve the score if that is the preferred direction for the authors.\n\nI do not find a key \"hero result\" to latch onto here, in a similar spirit to the (well-done) \"hero\" Figure 1. Generally, a lot of time is devoted to the details of absolute position prediction, which is useful but some portion of that discussion may be better suited to the appendix.\n\nQUESTIONS:\nRelated to weaknesses above, can you clarify a bit more the exact application of the continuous axial RoPE to multivariate, irregular timeseries? This seems intricate but critical for a number of areas, and a potential strength of RoMAE.\n\nOn infilling tasks, do you notice any particular bias in the model toward low or high frequency components? Most of the examples shown seem to have low frequency patterns, but showing that the model is also good at dealing with high-frequency, repeating ones would be useful for many applications.\n\nIn the limitations section, it is briefly mentioned that \"RoPE in RoMAE has some additional computational overhead if the positions are\n336 different with each forward pass, e.g., with any continuous irregular time-series\". This seems like a very significant drawback to me - as the speed of RoPE application can sometimes drives technical modeling decisions. Are there any empirical studies around this speed difference, compared to for example a model with no RoPE at all? Or one that receives the continuous position value directly as an additional input?\n\nThere are numerous studies around the interpolation of RoPE parameters for context expansion (YaRN for example). The interpolation section didn't mention much, if any, about these methods but they seem directly applicable for things like training on lower resolution images, and applying to higher resolution ones, or downsampled timeseries training, etc. Have the authors looked into this, or have some clarification about potential applications here?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}