Rotary Masked Autoencoders are Versatile Learners
Authors: Uros Zivanovic
Serafina Di Gioia
Andre Scaffidi
Martín Emilio de los Rios
Gabriella Contardo
Roberto Trotta
Venue: NeurIPS 2025 poster ()
Keywords: deep learning
Transformer
Masked Autoencoder
irregular time-series
Rotary Positional Embeddings
Primary area: deep_learning
TL;DR: RoMAE is a single, drop-in masked autoencoder that utilizes continuous (axial) RoPE to excel in interpolation and representation learning across modalities.

Abstract: Applying Transformers to irregular time-series typically requires specializations to their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is an extension to the Masked Autoencoder (MAE) that enables interpolation and representation learning with multidimensional continuous positional information while avoiding any time-series-specific architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and audio, demonstrating that RoMAE surpasses specialized time-series architectures on difficult datasets such as the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability to reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's relative position property.

Rotary Masked Autoencoders are Versatile Learners
Uros Zivanovic'. Serafina Di Gioia 2, Andre Scaffidi', Martín de los Rios', Gabriella Contardo 7.3 3. and Roberto Trotta 5,5,,
University of Trieste, Italy
Abdus Salam International Centre for Theoretical Physics (ICTP), Italy SScuola Internazionale Superiore di Studi Avanzati (SISSA), Italy
IINFN National Institute for Nuclear Physics, Italy
SICSC - Centro Nazionale di Ricerca in High Performance Computing, Italy IImperial College London, United Kingdom University of Nova Gorica, Slovenia
Abstract
Applying Transformers to irregular time-serieg typically requires specializations their baseline architecture, which can result in additional computational overhead and increased method complexity. We present the Rotary Masked Autoencoder (RoMAE), which utilizes the popular Rotary Positional Embedding (RoPE) method for continuous positions. RoMAE is extension to the Masked Autoencoder (MAF) that enables interpolation and representation learning with multidimen- sional continuous positional information while avoiding any simessiiiessppeccffcc architectural specializations. We showcase RoMAE's performance on a variety of modalities including irregular and multivariate time-series, images, and au- dio, demonstrating that RoMAE surpasses specialized time-series architectures difficult datasets such the DESC ELAsTiCC Challenge while maintaining MAE's usual performance across other modalities. In addition, we investigate RoMAE's ability reconstruct the embedded continuous positions, demonstrating that including learned embeddings in the input sequence breaks RoPE's Ess relative position property.
Introduction
The framework offered by foundation models has shifted the machine learning landscape by estab- lishing new benchmarks variety of modalities and tasks. Specifically, Transformers (58) have achieved state-of-the-art performance across many domains, from vision [15] to natural language processing (6]). Given the ability of Transformers to handle sequential data such as natural language, they naturally became appealing for time-series, which arise in large variety of domains, includ- ing health, finance and astrophysics. Such data can often be irregularly sampled in the temporal dimension. Being originally designed for sequences of text, the base Transformer architecture is not able to deal with such irregularly sampled data, by default only supporting quantized positional information as is found in natural language. This lack of support for continuous positional information becomes limitation when extending Transformers other modalities. degrading performance on tasks requiring precise temporal modelling and hindering the model's ability to capture complex patterns in non-unifommly sampled time-series.
Various specializations to the Transformer have been proposed to overcome this limitation. These can be divided into two main types: modifications of the internal architecture of the Transformer (e.g. modifying the feedforward layer in the Transformer Encoder Block (19 46)) and novel positional embeddings (e.g. using neural ODE 12). Alternatively, modern state space models
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

like Mamba [22] or S5 [ST are natively able to model various modalities such as text and images in addition to irregular time-series. Extending the capability of Transformers to irregular time-series while staying within existing frameworks developed for fields such as Natural Language Processing (NLP) and computer vision would allow one to easily benefit from ongoing developments within the Transformer "ecosystem".
To this end, we propose new representation learning method utilizing Rotary Positional Embed- dings (RoPE) [54] for continuous position in combination with Masked Autoencoder (MAE) 23 pre-training: the Rotary Masked Autoencoder (RoMAE). We investigate the performance of this framework on variety of tasks with different modalities. RoMAE obtains highly competitive results when compared specialized,
satte-of-hheaas approaches for individual tasks while maintaining MAE's s excellent performance in computer vision and audio. RoMAE is therefore extremely versatile while being built up from only standard Transformer methods. Our contributions are threefold:
1. Continuous Positional Embedding with RoPE:
We investigate how RoPE can be used to embed continuous positions, expanding on its original concept introduced in the RoFormer architecture [54], which did not address non-uniform or real-valued timestamps. We show that learned input embeddings can invalidate RoPE's relative distance guarantees, and we provide new empirical insights into RoPE's ability to embed varying positional scales.
2. RoMAE:
An expansion of MAE that works natively with irregular multivariate time-series without sacrificing any performance on standard modalities such as images and audio. Utilizing standard off-the-shelf methods developed for Transformers in NLP and Computer Vision, RoMAF shows that a specialized architecture is not required to achieve strong performance on irregular time-series.
3. Experimental Analysis:
We compare RoMAE with sfute-of-th-ass deep learning (DL) models, conducting experiments on the following tasks and modalities: (i) irregularly sampled multivariate time-series classification, (ii) image classification, (iii) irregularly sampled time-series interpolation and (iv) audio classification.
This work is structured as follows: Section 2 covers related works on MAE, RoPE, and irregular time-series. Section 3 provides the necessary background material. Section[deetail the workings of ROMAE and establishes the theoretical framework we use to tackle a variety modalities. Section 5 presents the results of our experiments. Section 6 discusse our results. Section concludes the paper.
2 Related Work
S4 as a simple and Rotary Positional Embeddings: RoPE was initially proposed in RoFormer
effective Relative Position Embedding (RPE) method that is independent from the Multi Multi-Head Attention (MHA) implementation being used. Despite RoPE encoding relative position, models incorporating RoPE have been shown not to generalize sequences longer than the ones shown during training. To improve sequence length extrapolation, various works have proposed increasing RoPE's base wavelength from 10 000 up to 500 000 63
62
Alternatively. YaRN 38 proposes
to interpolate RoPE's s frequencies 0g during inference to avoid distri--iittiittinn angles. There has also been recent discussion on the usefulness of the long-term decay property of RoPE 4. this work, we provide additional experimental evidence supporting the argument against the long-term decay of RoPE
RoPE in Vision Transformers: 2D Hand-crafted and learned Absolute Positional Embedding (APE) methods have both been shown to give similar performance on Computer Vision benchmarks when used with Vision Transformers (ViT) [15). Later works [25] 34] have shown the impact of RoPE on multi-resolution inference, finding that RoPE improves ViT's extrapolation performance. To extend RoPE to 2D, Axial RoPE (17] applies RoPE twice, once for each dimension. Additional learnable parameters can be added to Axial RoPE to encode diagonal positional information as well (25] Taking advantage of RoPE's independence from the specific Multi- Head Attention (MHA) implementation, Vision X-former 29 (ViX) a variant of VIT that utilizes RoPE with linear MHA implementations, making it more computationally efficient. Overall, adding RoPE to VIT has been shown to be a beneficial change, improving model performance, and enabling extrapolation to higher resolutions during inference.

---

ND Patchification
RoMAE
RoMAF Encoder/Decoder
Lx
Parch reconstruction
MLP
MASK
Output logits
RMS
RoMAE Decoder
MASK
Concat & Lincar
Flanten
MASK
Scaled Dut-Product Atlention
Patch Sequence
Intermediate representations
Linear Projection
Linear
Lincor
Lincar
RoMAE
Inpul Embeddingss
RMSNonn
inpul emboddings
Visualization of data embedding via multi- dimensional (ND) patchification for illustrative data realisations in 1, and 3D. Centre: Full depiction RoMAE architecture. The optional [CLS] token is omitted from the input sequence for simplicity. Right: The RoMAE encoder/decoder with RoPE operations denoted by rotational arrows.
Figure Overview of the RoMAE pipeline. Left:
have shown that self- supervised pre-training tasks greatly boost the downstream fine-tuning performance of Transformers NLP. MAE [23] is an approach that adapts BERT's masked modelling pre-training task to images. Although MAE is not the first work to conduct masked image modelling, it is one of the most widely used methods. MAE has been shown be both data-cfficient [57] and scalable 23 59 It has also been adapted successfully to variety of modalities other than images such as video [57 18). audio (28). and point clouds (55] MAE has also been combined with RoPE in MAETok f which uses the trained model as tokenizer for diffusion models. Although MAE has been used in variety of contexts, highlight that many of these contexts have required task-specific specializations to the backbone Transformer architecture, unlike the approach we present here.
Masked Autoencoders:
Architectures such BERT 14 and GPT 40
Transformers for Irregular Time-series: Adapting the Transformer architecture for irregular time- series is a long-standing research topic with many methods having been proposed [60]. When tokenizing the input, a popular approach to insert the time for each point through positional embeddings an approach recently used by models such as ContiFormer (12), Timer [33], and the concurrent work TrajGPT [52), which, similar RoMAE, uses RoPE for the task. Alternatively, one can also convert the time-series data into 2D images and process them using off off-the-shelf ViT 332. As a pre-training task, methods focus either on autoregressive trajectory prediction [52 33) or BERT-style interpolation through masked modelling [37. MAE has not been used for irregular time-series pre-training yet.
Background
Attention: We follow the formulation for Attention proposed in the original Transformer (58). Let
be a sequence of k embeddings z; E Rddess for E [1, 2,.,,.. k) and dmoael be the dimensionality of the model. Three linear layers are applied to transform into sequences containing queries (g:), keys (ks)), and values (w²). E.g., 4 = W Wy k; = W 2 = Z. The matrices Q, K, containing g.k.v are then passed through Scaled Dot-Produce Attention (SDPA) defined in Equation (
QKT
Attention(4) K, V) softmax
(I)
Vimaneel

---

The key operation in SDPA is the dot product between qi and k; which determines how the values V will be mixed with one another. Because Attention is permutation-invartt must be encoded within d and k; to allow SDPA to reason about position.
positional information
Regular and Irregular Dimensions: RoMAH is designed to work with both regular and irregular dimensional data. Specifically, we consider inputs of the form: X E Id xd2xx..xd where D is the number of dimensions in x and d, is the size of dimension i for i E [1. D..
Definition 3.1 (Regular and irregular dimensions). A regular dimension is one where all points equally spaced, while an irregular dimension is one where the distance between points varies.
Some dimensions in the input x may be irregular while others could be regular. For example, a set of images sampled at irregular times has height and width as two regular dimensions and time as one irregular dimension.
3.1 Rotary Positional Embeddings
Given imput 2m E I2 with position m and even dimensionality dz, RoPE partitions 2., into disjoint 2D subspaces (t) with i E [1, ..., dm/ rotating each subspace as:
cos m0; sin m0;) sin m0; cos m0l:
(2)
The 0 values are generated in the same way as for sinusoidal positional embeddings 58 : 0; 10000-2(i-11/4, Therefore, each subspace is rotated by different amount depending on the individual 6,. RoPE is applied directly to the queries and keys before they enter SDPA. Because the dot product relies only on the angle between two vectors and their magnitude, RoPE is a RPE method.
p-RoPE; In this work we make use of p-RoPE 4 a truncated version of RoPE where only the percent of smallest 0; values are kept. This cuts out fraction (1 p) of the frequencies in RoPE and thus leaves a portion of the input embedding space unchanged. The unchanged region in the embedding space provides the model with a data channel can use to pass information into SDPA without any modifications by RoPE, making the model more robust to varying sequence length. We use value p = 0.75, which has been shown by Barbero et al.
work well.
To encode multi-dimeensonal position, we utilize Axial RoPE [17. In Axial RoPE, the input is split into D subspaces of size dmodel/ D. where D is the number of positional dimensions. Then we apply p-RoPE to each subspace, encoding the positional value for that dimension. We note that since RoPE requires that embeddings be even and Axial RoPE requires that embeddings be divisible by D, this puts constraints on the possible values that dmodel can take.
Axial RoPE:
4 Method
An overview of RoMAE is shown in Figure I
N-Dimensional (ND) Patchification: Given inputs x as described in Section 3 we define a patch size (p1,- .PD) and divide each dimension into N₂ d:/p; non-overlapping segments, where d, is the size of dimension i. These are flattened, creating a sequence of patches with length = I1 N; and number of elements per patch n. = 11 Pt. Finally, a linear layer W is applied to each patch to project into the embedding dimension dimaaee- This process illustrated in Figure and is the same as employed by ViT [15 for images and ViVIT (I] for video. It can also be understood as using non-overlapping N-dimensional convolutions with the number of channels equal to dmmocet- After ND- -D-patchificatton, we have a sequence of embeddings which can be passed into the RoMAE Encoder.
Proposition 4.1. For any irregular dimension d, in x, the corresponding patch size for that dimension P) must be equal to 1.
Discussion: This limitation emerges from the requirement that each patch has the same number of data points np inside it. Note that mixing irregular and regular dimensions is not an issue. E.g., for an

---

time-series of images, one could choose a patch size of (1, 16, 16) for the time, height, and width respectively.
Because the NDD-atchification process flattens all dimensions into a single sequence, RoMAE is able to jointly model and attend to all patches across all dimensions at once. The drawback of this is that the number of tokens grows exponentially with the number of dimensions. In this work we only utilize this process up to D 3. For highly irregular multivariate time-series, also utilize different approach that is described in Section 4.2
4.1 Overall Structure
RoMAE's 's structure follows MAE's, using an asymmetric encoder/decoder, with the encoder being much larger than the decoder. Although both the encoder and decoder in RoMAE are a Transformer Encoder similar to BERT [4. we bring over recent developments in NLP from models such as
Llama 43 Specifically, we utilize the popular Sigmoid Linear Unit [24 16] for the non-lineane non-linearity. We also use R.MSNorm [66] instead of Layer Normalization 2 as it has been shown be more computationally efficient while maintaining the same level of model convergence, For architectural details, including definitions of various model sizes, we refer to Appendix A.1 and Appendix [A.2
Given a sequence of input patches, we uniformly mask a percentage of them. After projecting the unmasked patches into embeddings, a learned [CLS] token is optionally appended to the start of the sequence. This token becomes useful during fine-tuning, when an MLP head can be placed on top of it conduct classification. The embeddings are then passed through the RoMAE Encoder to create an intermediate representation z'. A set of learnable [MASK] tokens is then appended to z', with each [MASK] token receiving the positional information corresponding a patch that was masked out. This sequence is then passed through the RoMAE Decoder, after which the model head predicts np values for each patch that was masked out. After pre- mre-training, the decoder is removed and the intermediate representations z' are used for downstream tasks.
Pre-Training Task:
4.2 Positional Information in RoMAE
Although RoPE is originally designed be used with discrete positions such as those found in text, we observe that Equation (2 works with any E R. We make use of this in RoMAE to encode continuous position. Specifically, alongside the input values x, ROMAE also accepts sequence
Continuous Axial RoPE:
[s1,... ".,,....,,,, sk], 8; E DR containing the positional information for each patch. This is then applied within RoMAE using Axial RoPE as described in Section 3.1
Dealing With Many Irregular Dimensions: Although we are able to encode multi multi-dimensional large number dimensions
positional information using Axial ROPE, this does not scale well
due to having to divide dmasee by D. To overcome this, we optionally reserve a dimension in Axial
RoPE that is used to store the dimensional index for [1, 2, ..... D). E.g., if an embedding belongs to dimension 4, it will receive a positional encoding of 4. When using this approach we
include the learned [CLS] token, which allows the model to recover the dimensional index despite RoPE being RPE method (as a consequence of Proposition 4.2) We utilize this method with the ELAsTiCC dataset Section 5.4 which allows us to reduce the number of positional dimensions from 6 to 2.
4.3
Effects of Relative Position in RoMAE
Here we present an analysis of the effects of switching from absolute to relative position in RoMAE.
Proposition 4.2 (Reconstructing absolute position). When learned [CLS] token included in the input sequence, the RoMAE Encoder is able to reconstruct the original set of positions as described in Section
The [CLS] token provides the model with an "anchor". This allows the model compare each input embedding to the [CLS] token, and reconstruct its absolute position. We provide a proof in Appendix [ well as experimental evidence in Section 5.1
Intuition:
Corollary 4.1 (Translational invariance in the RoMAE Encoder). When no learned [CLS] token is included in the input sequence, positional information the RoMAE Encoder is relative, This makes the RoMAE Encoder invariant to translations of the input embedding positions.
5

---

Discussion: Translational invariance in the RoMAE Encoder makes the pre- -re-traiiing task more difficult because the model cannot make predictions based on the overall position of token in the input. E.g., with absolute position, the model can learn that objects of interest may often appear near the centre of an image.
Corollary 4.2 (Effect of distance absolute position reconstruction). RoMAE is able recover the absolute position of any embedding 2; with regard to Proposition 4.2 irrespective of the position 8; of that embedding.
Discussion: A claimed property of RoPE is that it causes the dot product between queries and keys decay their positions grow further apart (54). In Appendix [..] we provide empirical evidence showing that RoMAE is able to reconstruct position almost perfectly over two different scales of distance. We also refer the proof by Barbero al. [4] showing that is possible construct key for any non-zero query and any distance such that the softmax value in Attention is maximized at that distance.
Overall, relative position is a key element that influences the training dynamics RoMAE. This allows us to investigate RoPE from new angle, drawing new conclusions the effect of learned embeddings and supporting prior claims that the long-term decay property is not significant to the functioning of RoPE.
5 Experiments
Throughout the experiments we make use of different sizes of RoMAE: RoMAF-tiny, RoMAE-small, and RoMAE-base, detailed in Appendix A.I
Compute Details:
The experiment on the Tiny ImageNet data set (Section
5.2
was run on one node of Slurm cluster,
utilizing two NVIDIA Tesla V100 GPUs for 5 hours.
The experiment on the DESC ELASTICC Challenge (Section 5.4
was run on Slurm cluster using 4 nodes for 4 hours with each having 4 Nvidia A100 (with 64GB memory) GPUs.
Together, the experiments on the UEA Time-Series Archive
3 (Section 5.4). Pendulum dataset [5]
(Section and absolute position experiments (Section 5.1
were run on a 1080ti GPU for a total
of ~ 1.5 hours.
All interpolation experiments (Sec. 5.3) were run on a single NVIDIA A100-PCIE--40GB GPU (internal cluster), utilising <5 GB memory, 10min for the spirals dataset, ~ 30 mins for the synthetic dataset, and ~ 3 hours for PhysioNet.
Model and experimental code for RoMAE is made public through a convenient Python package ?
5.1 Reconstructing Absolute Position
To verify the model's ability reconstruct absolute positional information according Proposi- tion we give the model sequence of 10 identical values as input. Each embedding is then given ID position sampled uniformly between 0 and 50. We then use the same linear head to predict the position for all tokens. Because the model dimension dmodel has an effect on the number of 0, values RoPE uses, we also test a variety of model sizes. We run each test times and report the mean and standard deviation. Our generated training set has 20 000 samples while our generated test set has
http:://bottal.nnnss.ee//tttssss http:///.....seeeeooooouo/uu////
TD
htterrommweiiioo.igtthoooo githuu. 1ooMMMeE-oooMMMMMEMM Website/
Table 1: Position reconstruction MSE (mean + std) for
Table 2: Results on Tiny ImageNet across
various sizes of RoMAE.
various versions of RoMAF-small
Model size
With [CLS]
No [CLS]
Model
F-score (+ std)
RoMAE-tiny
0.062 (0.007)
200.33 (0.001)
RoMAE (no [CLS])
0.500 (0.011)
RoMAE-smal
0.0057 (0.002)
200.33 (0.001)
RoMAE ([CLs])
0.475 (0.006)
RoMAE-base
0.0031 (0.002)
200.33 (0.002)
RoMAE (absolute)
0.479 (0.010)

---

4000. Reported Mean Squared Error (MSE) is the average MSE over the test set. Results are shown in Table
We observe a clear difference between the model that uses the [CLS] token and the one that does not. When supplied with the learnable token, RoMAR is able reconstruct the original absolute position almost perfectly. Larger models seem to achieve better MSE, although all sizes perform well. For all experimental details and an additional experiment on absolute position reconstruction we refer to Appendix D.5 and Appendix B. respectiiv....
5.2
Tiny ImageNet
To investigate the effect of positional embedding and the learned [CLS] token on RoMAE, we train three versions of RoMAE on Tiny ImageNet [31]: RoPE with the [CLS] token, RoPE without the
[CLS] token, and absolute sinusoidal positional embeddings (58) with the [CLS] token. When fine- tuning RoMAE without the [CLS] token, we place the classification head on top of the mean of the output embeddings, otherwise we place the head top of the [CLS] token. The final configuration with absolute positional embeddings is very similar to MAE, making for a good comparison. We use patch size of (16, 16) and mask 75% of the input, similar to MAE. After pre-training each model for 200 epochs, we fine-tune for another 15. We also follow the procedure outlined by MAE to compute the pre-training loss using normalized patch values instead of pixel values.
For full experimental details we refer to Appendix D.I. The results are shown in Table 2 Although all 3 models perform similarly, we highlight that RoMAE with RoPE and no [CLS] token performs slightly better than both RoPE with [CLS] and absolute positional embeddings. This could be because of the models translation invariance (Corollary 4.1). The difference could also come from placing the classification head on top of the mean of output embeddings instead the [CLS] token. Overall, our results indicate that RoMAE performs least well MAE on images.
5.3 Audio benchmark
Table 3: Results for the ESC-50 benchmark for audio datasets, for the RoMAB-small model.
Model
Accuracy
SSAST (Librispeech)
80.0
RoMAE-sman (Librispeech)
83.2
SSAST (AudioSet-20k)
82.2
RoMAE-small (AudioSet-20k)
84.7
We chose to test RoMAE's ability to classify audio files, after a self-supervised pre-training on unlabeled audio datasets, inspired by the SSAST pretraining strategy [21]. SSAST is the first Vision transformer-bart based model that introduces a self-supervised pretraining strategy, supporting arbitrary
patch size, for the audio representation learning.
As our pretraining datasets, we used modified version of the Audioset 20 and Librispeech [36 datasets. AudioSet is a 2017 multi-label audio event classification dataset. Using carefully structured hierarchical ontology of 635 audio classes guided by manual curation and expert knowledge, the authors collected data from human labelers to probe the presence of specific audio classes, including, for example, human sounds, animal sounds, music, natural sounds, in 2 million 10-second segments of YouTube videos. However, access to the original 2 million audio clips is fraught with difficulty, as a consistent subset of YouTube videos no longer available. In order to conduct reproducible experiment, we decided to
training set the balanced training AadioSet-20k. made available on Hugging Face I We thus pretrain RoMAE using two different data sets: AudioSet- 20k and the Librispeech dataset. For the Audioset dataset used the ciining/validation split provided by the downloaded dataset, while for Librispeech, downloaded and preprocessed using the scripts provided on the SSAST Github repo, used 70/30 split.
In order to apply our model to the audio datasets, we first transform the audio waveforms to Mel spectrograms. First, the input audio waveform of length seconds is converted into sequence
7

---

of 228-dimensional log-Mel filterbank (fbank) features computed with a 25ms Hamming window every 10ms. This results in a 128 100 spectrogram. For the pretraining step, we followed the patchification strategy adopted in [21] and we split the spectrogram into sequence of N (16 x 16) patches, where N = 12(100t - 16)/10 10 is the number of patches and the effective input sequence length for the model. We refer to Appendix D.2 for the list of hyperparameters adopted for the pretraining and finetuning of the model. We would like to highlight here that the pretraining was run for 150 epochs, without the [CLS] token, while we choose to adopt mask ratio equal to 0.75, that is very similar to the masking fraction associated with the SSAST-patch 400 model.
For the finetuning audio classification benchmark, we used the ESC-50 dataset [39], consisting of 2000 5-second environmental audio recordings classified into 50 classes. The current best results on ESC-50 are accuracies of 86.5 and 94.7 obtained with supervised training (on AudioSet-2MM respectively by SOTA-S and SOTA-P models. The SSAST model, which is the only ViT model adopting self-supervised training for this task, achieved accuracies of 82.2 and 84.6 when trained, respectively, on AudioSet-20K and AudioSet-2M (as reported in Table 2 of (1). showing that the size and richness of the pretraining dataset impacts, in a non-negligible way, the performance of that model on the finetuning tasks. Since we did not have sufficient computational resources to pretrain RoMAE on AudioSet-2M, we chose to compare our model performance with that of the SSAST model pretrained on AudioSet-20K, and Librispeech, respectively. We report the benchmark results in Table 3 showing that RoMAE performs better than the SSAST model in these two cases.
5.4 Irregular Time-series Classification
Table 4: Light curve classification results on ELAsTiCC. A / or X corresponds to whether the Alert Mask (AM) is used.
Table 5: Regression MSE x x10 (mean + std) on the Pendulum dataset. We use custom RoMAE model size.
Method
AM
F-score
Model
Regression MSE (x10-$)
Transformer [58
0.5256
ODE-RNN 44 RKN-A, 5
7.26 (0.41)
ATAT 7
0.6270
5,09 (0.40)
RoMAE-tiny-ssallln
0.6649
ContiFormer
4.63 (1.07)
RoMAE-tiny-shallow
X
0.7106
CRU [45] 51
3,94 (0.21)
RoMAE-tiny
0.7205
S5 [51]
3.41 (0.27)
RoMAE-tiny
X
0.8029
RoMAE
3.32 (0.13)
DESC ELASTiCC Challenge The DESC ELAsTiCC Challenge is a multivariate irregular time- series dataset consisting of ~1.8M simulated light curves and 36 classes of astronomical objects. Each light curve consists of 6 irregularly sampled channels (called "bands"). To embed this data in RoMAE, we follow the procedure described in Section 4.2 This results a dimensional positional embedding, where one dimension embeds the time, and the second embeds the channel index. Although ELASTiCC has additional metadata for each light curve, we compare performance only across raw light curves. Some points in the light curve marked through an alert mask being unlikely to contain useful information. We evaluate RoMAEs performance both with these points and without. For more details how these points are chosen we refer to Appendix [D6] We train all RoMAE models by conducting full pre-training for 200 epochs with a masking ratio of 50% then fine-tuning for 25 epochs. for full details and a visualization of the data we refer to Appendix [.6
Table shows our results using two sizes of ROMAE, and compares with ATAT 77. a Transformer architecture specialized for ELAsTiCC. Despite the latter's specialization, RoMAE-tiny-shlllow (with a comparable number of parameters as ATAT) improves over ATAT by about .04 F-score when using the alert mask. The larger RoMAE-tiny with the alert mask achieves an even greater improvement of about. II F-score. A key reason for RoMAE's better performance might be that ATAT does not conduct any pre- pre-training. In the case of RoMAE-tiny, the larger scale of the model also likely plays a role. We also find notable that when ignoring the alert mask and using all points, RoMAE performs even better, suggesting that perhaps these points should not be ignored.
'The Mel Filterbank transform computes weighted averages of bins to provide spectral power estimates on a logarithmic frequency scale, which is more affine human hearing resolution.
8

---

Table 6: Accuracy across various datasets from the UEA Time-series Archive.
Model
Dataset
TST 65
mTAAN 46
S5 51
ContiFormer
[1]
RoMAE-tiny
BM
0.9667
0.9917
0.9833
0.9750
0.9917
CT
0.9742
0.9529
0,9610
0,9833
0.9882
EP 0.9589
0.9203
0,9074
0.9324
0,9517
HB 0.7398
0.7789
0.7333
0.7561
0.7447
LSST 0.5520
0.5307
0.6389
0.6004
0.6225
Table Results for the interpolation experiments discussed in Section 5.5
Synthetic (MSE)
Spirals (RMSE)
HeTVAE 35
0.223 0.070
RoMAE-tiny
0.233 = 0.007
Transformer 58
1.37 + 0.14
Latent ODE [TO
2.09 + 0.22
PhysioNet (MSE)
ContiFormer
0.49 + 0.06
HeTVAE
35
0.562 + 0.022
RoMAE-tiny
0.0183 + 0.007
RoMAE-tiny
0.467 + 0.021
UEA Multivariate Time-series Archive We evaluate RoMAE on variety of datasets from the UEA Multivariate Time-series Archive 3 To make the datasets irregular we follow the procedure outlined by Kidger et al. [30], dropping 30% of the observations. Because all variates are present at each time-step, the only irregular dimension is time. Therefore, to embed this data in RoMAE, combine all variates per time-step into one embedding. This is much simpler setup than the one used for the ELASTiCC dataset in Section 5.4 For each dataset we conduct pre-training for 400 epochs. When fine-tuning, found it necessary change hyper-parameters between different datasets. For more details on our experimental setup we refer to Appendix [33
Table 6 presents the mean accuracy from 3 full training runs (pretraining finetuning), as well as published results from a variety of models. Overall, RoMAE-tiny performs similarly or better than the comparators, and is able to handle the various datasets without issues. All the datasets trained on are relatively small, with some being on the order of hundreds of samples. Therefore, this experiment also shows how data-efficient RoMAE can be.
The Pendulum dataset 51 is an irregular time-series dataset consisting of irregularly sampled images of a pendulum. To embed the images in RoMAE, we use patch size of (1, 24, 24) for (time, height, width). This corresponds to embedding per time-step/image. RoMAE is trained directly on regression without any pre-training, predicting the sine and cosine of the angle of the pendulum which follows non-linear dynamical system. We use custom size for RoMAF which contains only layers and an MI.P hidden size of 30. This is much smaller than RoMAF-tiny and provides for fairer comparison with other models trained on this dataset. For additional information on the dataset and full experimental details we refer to Appendix D.4 After training 20 different seeds, the mean MSE and standard deviation of RoMAE on the Pendulum dataset are reported in Table RoMAE outperforms specialized Transformer based models such ContiFormer [12] well state space models such as S5 [5]]
Irregular Time-series Regression: Pendulum Dataset
5.5 Interpolation
We evaluate RoMAE on three interpolation tasks with increasing dimensionality and sampling irregularity. (i) Spiral: A 2D synthetic benchmark of 300 noisy Archimedean spirals as in Ref. [12] (ii) Synthetic: The 50-step univariate task from Ref. [48] and (iii) PhysioNer: 48-hour ICU records containing clinical variables steps. To create irregular time-series data, time points are randomly selected from the first half of each spiral, which are used for interpolation. For (ii), the interpolation task between random subsample including between and 10 points per trajectory. For (iii, we follow the 50% masking protocol
49 For (i), each spiral is discretized into evenly-spaced time

---

of [35]: half of the time, rows with at least one observation are hidden and must be reconstructed from context. We provide additional experimental details in the Appendix, respectively D.7 D.8 and D.9 All experiments are conducted to ensure a direct comparison with the results of the Transformer [58), LatentODE [9] and ContiFormer [12] for (i), and HeTVAF for (ūi) and (iii). Across
scales, our RoMAE-tiny configuration consistently scores competitively with respect to benchmarked MSE/RMSE, as seen in Table, ? We lastly remark on RoMAE's ability to retain progressively higher frequency modes for interpolation with time-series data in Appendix B.4
6 Discussion
shoos shows that one tiny/small RoMAE model, pre-trained once with a generic objective and no task-specifie architectural tuning, matches or surpasses specialised baselines across three increasingly difficult interpolation datasets. On the 2D Spiral benchmark, we improve by an order of magnitude over the previous best result by ContiFormer, which we attribute MAE tubelet- masking enforcing long-range reasoning. For the 50-step Synthetic task we obtain 0.233 + 0.007 MSE, comparable to HcTVAE's 0.223 = 0.070 but with markedly tighter standard deviation (five seeds). On the irregular, 41-channel PhysioNet-2012 ICU data we achieve 0.570 + 0.014 MSE, within half a standard deviation of VVA's''s heteroscedastic decoder. We have verified by retraining HeTVAE on the PhysioNet interpolation task that it indeed excels on densely sampled variables, such as heart-rate, whereas ROMAE maintains more balanced interpolation across the sparsely observed channels; consequently, while the aggregate MSE over all the features is similar, RoMAE delivers a simpler, single-stage model without task specific adaptations.
These results indicate that RoMAE pre-training can be universally beneficial for continuous time interpolation over irregular time-series data. The RoMAE architecture is able to scale from low- dimensional position time embeddings (50,1), to large (2880,41) points with extremely sparse observations differing across features, without refinement of the inherent architecture, suggesting that MAE-style models can serve strong, task-agnostic baselines for continuous- interpolation. RoMAE's classification results on datasets sizes ranging from a few hundreds to millions show how pre-training enables RoMAE to be both scalable and data-efficient.
Our tests on position reconstruction demonstrate that care must be taken when working with learned embeddings. Given how prominent the [CLS] token is when working with Transformer Encoders, our results are relevant for a multitude of models, including RoFormer [54].
Limitations: RoPE in ROMAE has some additional computational overhead if the positions are different with each forward pass, e.g., with any continuous irregular time-series. If positions stay constant, however, in images, the overhead becomes negligible. For details the additional compute incurred by continuous RoPE, see Appendix B.2, RoMAE is also not well suited for very long sequences, as it uses standard Attention which has (n2)) memory complexity with regards to sequence length. Lastly, RoMAE's ability to perform on extrapolation tasks is limited, as discussed in Sec. B3
We believe that RoMAE'S flexibility can extend representation learning to scientific fields where it may not have been easily adopted thus far, e.g., similar to what we showed with the ELASTiCC Challenge. The potential for the misuse of RoMAE is similar to that of MAE and other foundational models.
Broader and Societal Impact:
Conclusion
This paper introduced RoMAE, extension to the Masked Autoencoder architecture that allows it to accept muutti-dimessional data with continuous positions as input. We investigated the theoretical implications of using relative positional embeddings for MAE model, showing how it changes the difficulty of the masked pre-training task. We also showed how the model can use a learned [CLS] token to recover absolute position. Results across large variety of modalities and tasks have shown that RoMAE achieves excellent performance in modalities requiring continuous position while maintaining the performance of MAE on images and audio. Future work will include building more robust theoretical understanding of the implications of using RoPE. Because RoPE is compatible with various Attention implementations, one could also adapt ROMAE to use a linear Attention variant, which would allow it to work with much larger input sequences. Finally, we envisage the exploration of the many potential modalities that RoMAE could work with that were not tested here.
10

---

Acknowledgments
RT and AS acknowledge funding from Next Generation EU, the context of the National Recovery and Resilience Plan, Investment PE1 Project FAIR "Future Artificial Intelligence Research". This resource was co-financed by the Next Generation EU [DM 1555 del 11.10.22]. RT is partially sup- ported by the Fondazione ICSC, Spoke 3 "Astrophysics and Cosmos Observations", Piano Nazionale Ripresa e Resilienza Project CN00000013 "Italian Research Center on High-Performance Com- puting, Big Data and Quantum Computing" funded by MUR Missione 4 Componente Investimento 1.4: Potenziamento strutture di ricerca e creazione "campioni nazionali di R&S (M4C2-19)) Next Generation EU (NGEU). We also acknowledge the use of computational resources provided by the Italian AREA Science Park supercomputing platform ORFEO Trieste. GC supported by the Eu- ropean Union's ' Horizon Europe research and innovation program under the Marie Sklodowska-Cure COFUND Postdoctoral Programme grant agreement No. 10108 1355- SMASH and by the Republic of Slovenia and the European Union from the European Regional Development Fund. Disclaimer: Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

---

References
[1] A. Arnab, M. Dehghani, G. Heigold, Sun, M. Lucic, and C. Schmid. Vivit: A video vision transformer. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 6826..666.. IEEE, 2021. doi: 10.1109/ ICCV48922.2021.0066
URL https: /tt:://oii..o http::://doooor/////// 1110///111111111499222222 202.199922222200.0000 00676
[2] L. J. Ba, R. Kiros, and G. Hinton. Layer normalization. CoRR, abs/1607.064500, 2016. URL http://arxiv org 0rg/
[3] A. J. Bagnall, H. A. Dau, J. Lines, M. Flynn, Large, A. Bostrom, P. Southam, and E. J. Keogh. The UEA multivariate time series classification archive, 2018. CoRR, abs/ 1tt//81111000000 111000075, 2018. URL htttp://axxiv... wwt./orr////b/aeeeeeeeee 0rg1000//111100075
[4] F. Barbero, A. Vitvitskyi, C. Perivolaropoulos, R. Pascanu, and P. Velickovic. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Confer- ence on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL httpps::/ www.nneooooonddttuuuuuu
[5] Beeker, H. Pandya, G. H. W. Gebhardt, C. Zhao. C. J. Taylor, and G. Neumann. Re- current kalman networks: Factorized inference in high-dimensional deep feature spaces. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Confer- ence on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Leurning Research, pages 544-552. PMLR, 2019, URL http::/ppoceeee
mlr. pwwps/ v97 becker199..0 html
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M. Bal- can. and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020. URL httpss://proceeednnnng neurips. httpp/cc/aaaerr22///a/aaaa http://c....bbbtac.........
[7] Cabrera-Viver G., Cartan-Ca D., Astorga, N., Reyes-Jainaaa, I., Förster, F., Huijse, P., Arredondo, J., Muñoz Arancibia, A. M., Bayo, A., Catelan, M., Estévez, P. A., Sánchez-Sáez, P., Álvarez, A., Castellanos, P., Gallardo, P., Moya, A., and Rodriguez-Mancini D. Atat: Astronomical transformer for time series and tabular data. A&A, 689: 2024. doi: 10.1051/ 0004-6361/20224494755 URL htttps://do1. org/ 10. 10011//0004463611222499000999090
[8] H. Chen, Y. Han, Chen, X. Li, Y. Wang, Wang. Z. Wang, Z. Liu, D. Zou, and B. Raj. Masked autoencoders are effective tokenizers for diffusion models. CoRR, abs/2502.03444, 2025. doi: 10448550/ARXIV.2502..044
httpss: org/10. 48550/ arviv 25.. 03444
[9] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural Ordinary Differential Equations. In Advances Neural Information Processing Systems (NeurIPS), pages 6571- 6583, 2018
[10] Т. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6572 6572-6583, 2018. URL https:t::///oooteteee
www.nnnuip.... wwwpppppe///a///e 2018/ html
[11] Y. Chen, K. Ren, Y. Wang, Y. Fang, W. Sun, and D. Li. Contiformer: Continuous- time transformer for irregular time series modeling. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10
12

---

16. 2023, 2023. URL http:://ppppps..ppp....pppeeeeep 33220F55666400111//ff17777-2-222
html
[12] Y. Chen, Q. Wang, and Y. Fu. Continuous-time Transformer for Irregular Time-series Predictions. In Advances Neural Information Processing Systems (NeurIPS). 2023.
[13] E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le. Randaugment: Practical automated data aug- mentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-
12, 2020, virtual, 2020. URL https: http::es//proceeinnge pen.reerinnns..uu.......... nwwrips http:/appppp////meae
-bsta000000020---aaaaaaa- html
[14] Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding, In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLE 2019, Minneapolis, MN, USA. June 2-7. 2019, Volume (Long and Short Papers), pages 4171 4171-4186. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/N19111 1423. URL hRthts::///oo org/ 10. 10..1663/v1/11- 1423
[15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, Uszkoreit, and N. Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. /ttps:tts////oeerrree
[16] S. Elfwing, E. Uchibe, and K. Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Nenworks, 107:3-11, 11, 2018. doi: 10.1016/I. NEUNET.2011.12.011
URL http:: //oii. org wtw.//..eeeettt......m..aaaaa... neunennet.....m 2017 12.011
[17] Y. Fang, Q. Sun, X. Wang, T. Huang, X. Wang, and Cao. EVA-02: visual representation for neon genesis. Image Vis. Comput., 149:105171, 2024, doi: 10. 10.1106/J...AVI.... 2AW.MMMAIII..22.10517777
URL https: org/ 10. 1016/j www.iiiiiii......ii 2024. 105171
[18] C. Feichtenhofer, H. Fan. Y. Li, and K. He. Masked autoencoders as spatiotemporal learn- ers. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Anmual Conference on Neural In- formation Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. http:: wwps.cc/papieeeeeea poc/paper___ files/ Cttp:://tt---ottat---a----oo-oe- Confereece .ttll
files/paper/ happ:/apppp//// /022/
[19] S. Gao, T. Koker, O. Queen, T. Hartvigsen, T. Tsiligkaridis, and M. Zitnik. Units: A unified multi-task time series model. Advances in Neural Information Processing Systems, 37:140589 140631, 2024.
[20] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 20/7 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE, 2017.
[21] Y. Gong, C. Lai, Y. Chung, and J. R. Glass. SSAST: self-supervised audio spectrogram trans- former. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Sym- posium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 10699-10709, AAAI 2022. doi: 10.1699/AAALV36110222533 URL https: //doi. org/ 1609/ 06000/00/0/...111/.a........ wtw.ii.v.i..... 221155
[22] A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces. CoRR, abs/2312.00752, 2023. doi: 10.48550/ARXX1.23122.0777 URL. hhtpps::/ : //doi .rgg/00
48550/ aarvvv 2312, 2312.00752
13

---

[23] K. He, X. Chen, S. Xie, Li, P. Dollár, and R. B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979-15988. IEFF, 2022. doi: 10. 10.9 SEVR526688.022.01155
URI.httpss penttps://doi.. //o.i//oii..o org/10.
[24] D. Hendrycks and K. Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, 2e////066.84111 2016. http://///aaxii org hetg/bb//1606.... .11110016111555661
[25] B. Heo, S. Park, D. Han, and S. Yun. Rotary position embedding for vision transformer. In A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol, editors, Computer Vision FCCV 2024 18th European Conference, Milan, Italy, September 29-October 2024, Proceedings, Part X, volume 15068 of Lecture Notes in Computer Science, pages 289-305. Springer, 2024, doi: 10.1007/978-3--33-77888 978-3 978-3-031-72684-2.11
17. StRpsttts:///ot 0r./10.10077
[26] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. CoRR, abs/1207 ha/p/107.008880 000000000... 2012.
httptttp:///rreee org httgg/bb/11007/ .5800
[27] G. Huang. Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision - ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science pages 646 661. Springer, 2016. doi: 10. _9
URL https::/doii owapeii.rr////// 10t1://0001100088----333333 319---33--------------
39
[28]
Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhoferr Masked autoencoders that listen. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November
28 December 9, 2022, 2022. htt:: http:hhttppppppee //pp:er//ppppeeeep/eee nips cc/ wwwpppappe.iieeeeeeeeeeeeeeeeeee www.ffiillsppppe www.feseepppppr/22/2.. hash/ http://99622299990b11 htw.a//0000999911110361666116111 htt.://aee..aaaoooaaaoaaoooorrrr .tmll
[29] Jeevan and A. Sethi. Resource-efficient hybrid x-formers for vision. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI. USA, January 3-8, 2022, pages 3555-3563, IEEE, 2022. doi: 10. 10111A/V54455:5222.00006 URL https://don org/10. 11109//AAV14555 WAC251458.2002..... 00361
[30] P. Kidger, J. Morrill, J. Foster, and T. J. Lyons. Neural controlled differential equations for irregular time series. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con-
ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-
12. 2020, virtual, 2020. URL http:://prrccceiinggg http:///prroeedngnneeeeippppppsp wttpe/nps....apeee////////.. ww/ppppee//2///aaeeeee
html
[31] Y. Le and X. S. Yang. Tiny imagenet visual recogmition challenge. 2015. URL http: //vision. sta.ford. edu wwweecchhinnn/2222//rrrrrr/////0 ptrpr/////pppeseeeepppreeeeetct pdf
[32] Z. Li, S. Li, and X. Yan. Time series images: Vision transformer for irregularly sampled time series. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL httt:://ppppeeeeeeee nips cc cttccc//ppaoo htcppappr__fee fileffieee//ee wwwes//apeeeeee//eeeeeae paaper/023/haahh Ba.Tc1ee88861111255eddd.. ca80d Aetrodd-bsssaat... Conf erence html
[33] Y. Liu, H. Zhang, C. Li, X. Huang, J. Wang. and M. Longg Timer: Generative pre-trained transformers are large time series models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https //openrevies
[34] Z. Lu, Z. Wang, D. Huang, C. Wu, X. Liu, W. Ouyang, and L. Bai. Fit: Flexible vision transformer for diffusion model. In Forty-first International Conference on Machine Learning. ICML. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://
openreview wwt.:e//ooormm//t-22ee222222
14

---

[35] M. Moor, B. Rieck, M. Horn, C. R. Jutzeler, and K. Borgwardt. Early Recognition of Sepsis with Heteroscedastic Temporal Variational Autoencoders. In International Conference on Machine Learning (ICML), pages 7781-7792, 2021.
[36] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: An ast corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015. doi: 10.1 10.1109/CCASSP.205.77773
[37] H. Patel, R. Qiu, A. Irwin, S. Sadiq, and S. Wang. EMIT event-based masked auto encoding for irregular time series. In E. Baralis, K. Zhang. E. Damiani, M. Debbah, P. Kalnis, and X. Wu, editors, IEEE International Conference on Data Mining, ICDM 2024, Abu Dhabi, United Arab Emirates, December 92 2024, pages 370-379. IEEE, 2024. doi: 10.1 109/CCDM59182.2000 00044, URL Stthstps::.. /doi.. org/ 10. 11111119///00 IIM//912DMMMM 2024. 00044
[38] B. Peng. J. Quesnelle, H. Fan, and E. Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttttt//pppenreeeee
net htt/fformmidd-wlBB.
[39] K. J. Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM International Conference on Multimedia, MM '15, page 1015-1018, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781 9781450334594, doi: 10.1145/2733373. 2806390. URL https://doi. org/ 1145 2806390
[40] A. Radford, Narasimhan, Salimans, I. Sutskever, et al. Improving language under- standing by generative pre-training. OpenAl blog, 2018. https: h/tp:://///doooooii....aaoa openai............... com/ research- sawaarcchcooeer/langgaggaea
understaanding_pppert pdf
[41] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Lan- guage models unsupervised multitask learners. OpenAl blog, 2019.
URL
https://cdn. https://ccd.ooeaaaii... www/bettereeeoeeeeeeeeeeee waw.--alnggggemeeeee maaeee///mmmmeeeaaaeeeeeeeee lowguagge_ooeeesssssooeoeeeeeeee _
pdf
[42] N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, and A. Courville. On the spectral bias of neural networks. In International conference machine learning, pages 5301-5310. PMLR, 2019.
[43] B. Rozière, J. Gehring. F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov. I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferree A. Grattafiori, W. Xiong, A. Défosse7, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve. Code llama: Open foundation models for code. CORR, abs/2308. 12950, 2023. doi: 10.4 144555//ARX...33....... 12950. URL
10. 20l.4/8500/40 arxiv 2308. 12950
[44] Y. Rubanova, T. Q. Chen, and D. Duvenaud. Latent ordinary differential equations for irregularly-samplee time series. In H. M. Wallach, H. Larochelle, A. Beygelz- imer, F. d' d'Acché-Bue, E. B. Fox, and R. Gamett, editors, Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5321-5331, 2019, URL httpt:h/////oot pttp:////oceeeenns./memm neur wet.rpp.cccpppper20011.111 cc/papprr/019/haelt htt.::o4ww....----barttt... Abstract,
[45] M. Schirmer, M. Eltayeb, Lessmann, and M. Rudolph. Modeling irregular time series with continuous recurrent units. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 19388-19405. PMLR, 2022. URL https: httpae//roocceededeeeeeee moriinnnn.........mmmm.mm.m .pess// v162 ht00//chhrmme22e html
[46] Shukla and B. M. Marlin. Multi-time attention networks for irregularly sampled time series. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URI, www.reerrvieee.ne///rrrrr net/forum? id=4c0J61w04
15

---

[47] S. Shukla and B. M. Marlin. Heteroscedastic temporal variational autoencoder for irregularly sampled time series. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httpttt///oppeeeeiieeee het/forum?an A27
[48] N. Shukla and B. M. Marlin. Heteroscedastic temporal variational autoencoder for irregularly sampled time series. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. UtLtttss::// net/forum?ice Ap7obD-EE--
[49] I. Silva, B. Moody, D. Scott, L. Celi, R. Mark, and G. Clifford. The PhysioNe//Computing in Cardiology Challenge 2012: Predicting In- In-Hospital Mortality from ICU Data. In Computing in Cardiology, pages 245-248, 2012.
[50] I. Silva, G. Moody, J. Scott, I. A. Celi, and R. G. Mark. Predicting in-hospital mortality of ieu patients: The physionet/computing in cardiology challenge 2012. In 2012 computing in cardiology, pages 245-248. IEEE, 2012.
[51] T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /ttps:///penrrevier net/
[52] Z. Song, Q. Lu, H. Zhu, D. Buckeridge, and Y. Li. Trajgpt: Irregular time-series representation learning for health trajectory analysis. CoRR, abs/2410.02133. 2024. doi: 10.48550/ARXIV. 2410.02133. URL htt.ttttt//.....co http////ooo.rr////// 48550/ arxiv. 2410.02133
[53] N. Stroh. Trackgpt-a generative pre-trained transformer for cross-domain entity trajectory forecasting, arXiv preprint arXiv:2-42.000666 2024.
[54] J. Su, M. H. M. Ahmed, Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. doi: 10. NB1CCOM 127063. WRL/https://ddot org/10. 10. 1016/j neucom. 2023. 127063
[55] K. Su, Q. Wu, P. Cai, X. Zhu, X. Lu, Z. Wang, and Hu. RI-MAE: autoencoders for self-supervised point cloud representation learning. In T. Walsh, J. Shah, and Z. Kolter, editors, AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 March 4, 2025, Philadelphia, PA, USA, pages 7015-7023. AAAI Press, 2025. doi: 10. 10.1609/AAA. V3917.2275 URL httph:tt////ttt org/
masked
/togg//////// 10. 1609/ aaai
v3917 32753
[56] C. Szegedy, V. Vanhoucke, S. Ioffe, Shlees, and Z. Wojna. Rethinking the inception architec- ture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818 8-26. IEEE Computer Society, 2016. doi: 10. N..110//CVR..2016.30
URL https::p/// : //doi org/10 1109/CVPR. 2016 008
[57] Z. Tong, Y. Song. J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL. http: /tttpe//papers.. wwwpeerrs..is...oo ctaps.cc/pappe files/ /twes//appr////22/.. pappp//022//aass/s 115550832766111c222bb33333322233
Cenesraact-Coffeeentse Conf html
[58] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fer- gus, V. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017. URL URI /tt.:/ttttt:::e/eeeetiiiii wtw.eeeeeddnng..u.iiiiieeee neurips cc/ cc/paper/ / 2017 hash/ http://////atttteeeeeeeeeeeeetna http://acc..aaaaaaaaaaaaaraaa... Abstract.html
16

---

[59] L. Wang, B. Huang, Z. Zhao, Z. Tong. Y. He, Y. Wang, Y. Wang, and Y. Qiao. Videomae V2: scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17- 24, 2023, pages 14549.14560. IEEE, 2023. doi: 10.1 10I100CV22279922220000000 URL https: //doi. org/10..//. 10. 1110//1V0////T 2ww.2/..............22222. 2.023.01390
[60] Q.
Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun. Transformers in time series: survey. In Proceedings of the Thirty-Secone International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao. SAR. China, pages 6778 6786. ijcai.org, 2023. doi: 10.24963/IICAA22223333 Uttpttss:://oo org/10. 24963/ ijcii 2023/759
[61] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: State-of-the- art natural language processing. In Proceed- ings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[62] W. Xiong. J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz, M. Khabsa, H. Fang, Y. Mehdad, S. Narang, K. Malik, A. Fan, S. Bhosale, S. Edunov, M. Lewis, S. Wang. and H. Ma. Effective long-context scaling of foundation models. In K. Duh, H. Gómez-Adorma and S. Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 4643 4663. Association for Computational Linguistics, 2024. doi:
URL httphstss: o htte//.0.18653//10 18653/v1/
2024. naacl- 260
[63] M. Xu, X. Men, B. Wang, Q. Zhang, H. Lin, X. Han, and W. Chen. Base of rope bounds context length. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024. NeurIPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024. URL http://papers.aa nips ott.r//rreeeeeeetooooooooooooooo
2024/ ht/4//hah//aae /9tp///22222222222222 htw.aa/aaaaa.aaaeeeeaeeeeeooooo. Aaceceebbbsbbatcc Conferenoeeeeeee html
[64] Z.-Q. J. Xu, Y. Zhang, and T. Luo. Overview frequency principle/speccraal bias in deep learning. Communications on Applied Mathematics and Computation, 7(3):827-864, 2025.
[65] G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff. transformer-based framework for multivariate time series representation learning. In F. Zhu, B. C. Ooi, and
C. Miao, editors, KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Virtual Event, Singapore, August 14-18. 2021, pages 2124-2124. ACM, 2021. doi: 10.1 04.114//3445548.4474444
org/ 1145311444454444555 3467401
[66] B. Zhang and R. Sennrich. Root mean square layer normalization. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019. December 8-14, 2019, Vancouver, BC, Canada. pages 12360- 2019. /ttpsp/t/////////ooooeeiiio
wwt.ggs..eriis...o pttpcc/////////////aaaa//// (a5a112222244444333377777111433 htwp://bbbbbbsttttt.t.. html
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction states that we showcase RoMAE's abilities on irregular and multivariate time-series (shown in Section 5.4), images (shown in Section
5.2 and 5.4), audio (details shown in Section (.3), where we provide performances of state-of- the-art baselines. The investigation of RoMAE's ability reconstruct embedded positions and analysis of relative position property are shown in Section 5.1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section and as well as in Appendix B.3
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary details for reproducibility are provided in Section and Ap- pendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Link to code is provided in Section https://chromeilionn .at.hitthb... io/ public.
Experiment/evaluation code and some checkpoints have also been made
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/ lttipss.ccppublicuuu.eeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments report performance mean and standard deviation obtained several runs as described in Section 3 and Appendix, except for the DESC ELCCTiCC Challenge, as this significantly larger dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp::/neeurpptt http:/rpp...pppbllcccccccccctaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss how our model can be applied positively in downstream deploy- ments and draw parallels with other foundation models regarding negative impacts; see Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No LLMs or image generator are presented. The datasets used here are all publicly available, and the tasks addressed should not be risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in this paper properly cited. We have also cited the code assets our model relies on both in the paper and in our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code for our model and our training utilities. We also specify all training details in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ottp:e///eerrrpo.cccoo/oeeeereee for what should or should not be described.
24

---

Appendix A
Model Details
A.1
Model Sizes
Table 8: All RoMAE model sizes.
Parameter
tiny-shallow
tiny
small
base
dmodel
180
180
432
720
Nhead
3
3
6
12
Depth
2
12
12
Dim. feed-forward
720
720
1728
2880
Num. parameters
0.782M
4.67M
26.9M
74.7M
We define set of model sizes for RoMAE which based on the original BERT 14 and ViT 15
model sizes, and are given in Table 8 The most important difference between RoMAE sizes and the sizes of other BERT-style models is in the d'model parameter: RoMAE adopts a different dimensionality because of the constraints regarding Axial RoPE described in Section 3.1 Specifically, we choose dmwdeel such that the same model dimensionality works up to positional (axial) dimensions.
Decoder Size: Although we vary the size of the RoMAE Encoder throughout various training runs, always use RoMAE tiny-shallow the decoder size when pre-training. Our choice is motivated by MAE [23]. which also uses a small and shallow decoder.
A.2
Architectural, Normalization, and Regularization Details
Here we describe the components of RoMAE in more detail, the normalization techniques we use during various training runs and technologies we use speed up training.
RMSNorm: RoMAE uses RMSNorm 666 which defined as follows:
danatel
04
a. RMS(a) Ji,
RMS(a)
danodel
where a is sequence of embeddings. 9i is learned parameter that rescales each E and is small value added for numerical stability. Notably, R.MNNrm does not centre the input like LayerNorm 2 does. Re-centring has been shown not to be necessary and removing it saves compute.
Patch Reconstruction: After passing all [MASK] tokens through the RoMAE decoder, pass the same reconstruction head over all [MASK] tokens to predict the original patch values. This head consists of RMSNorm followed by a linear layer Wd
The classification head we use has the same structure as the patch reconstruction head, using RMSNorm and linear layer X We place the head on top of the [CLS] token when it is available. Otherwise we take the mean of the output embeddings and place the head on top of this.
Classification Head:
In some runs we use stochastic depth (27, which is form of dropout where whole layers are zeroed out. Specifically, each layer 4n which has depth m, has a probability Nage be zeroed out, where À is the probability of the final layer being zeroed out.
Stochastic Depth:
Am to
Mixed Precision Training: Although all final results are in full FP32 precision, we also tried mixed precision training through PyTorch Automatic Mixed Precision (AMP) When training with AMP, some operations are conducted in a lower precision (either 16-bit brain floating-pon point (BF16) or 16-bit Coating-point (FP16)) instead of the usual 32-bit floating point. This speeds the model up greatly, resulting in significantly less compute resources being used. In our experiments we found that RoMAE still converged well when using mixed precision.
Dropout: When using dropout 26 we apply it to the attention scores and to the MLP hidden layer
with the same probability.
https://docs wwwprch. htw.:/rrww.rrggebo////taaale///e
25

---

Label smoothing: To help reduce overfitting, label smoothing [56] prevents the model from becoming overconfident. This is done by changing the model target labels, reducing each correct class label from 1 to confidence value and increasing all incorrect class labels from a value of O to value of - c) Relases.
Appendix B Additional Experiments and Discussion
B.1 Additional Absolute Position Reconstruction Results
20
(0, 1000)
(0, 100)
15
MSE
10
5-
0.00
0.25
0.50
0.75
1.00
Percentage domain maximum
Figure 2: RoMAE position reconstruction MSE across two positional ranges.
Here provide additional experiment showing how RoMAE is able to to reconstruct absolute position across a wide range of values. To conduct the experiment, we pass only one token into RoMAE, giving it random position drawn from uniform distribution Uja, b] training the model to predict this position. We also pass the [CLS] token. The experiment is conducted over two domains; (0, 100) and (0, 1000). Hyperparameters and training details discussed in Section D.5 After training, we evaluate how well the model performs across the range of values it was trained on. Results are plotted in Figure
We find that the model is generally able to reconstruct all position values within the two domains tested, except when the position close to the edges of the domain. This effect occurs already when the domain is relatively small and worsens as it becomes larger. We also find that the model performs better overall on the smaller domain. These results provide empirical support for Proposition 4.2 and show how the model is able to learn to reconstruct absolute position across a large domain. That the loss grows the position nears the edges of the domain shows that the model does not find solutions that generalize to out-of -isttrriibution positions. These results also indicate that it may be beneficial to rescale positions be within a smaller range.
Discussion:
B.2 Compute Performance
Table 9; Relative speed of RoMAE when used with regulat/irregular positions.
Positional Embedding
Relative speed
Absolute (sin/cos)
RoPE (quantized)
0.98 0.87
RoPE (continuous)
We evaluate the performance of RoMAE when using different positional encoding methods, specif- ically: absolute sin/cos (58), RoPE with integer (quantized) positions, and RoPE with continuous positions. The workload we test on uses positional dimensions for RoPE and image-like inputs to the model. Therefore, this experiment representative of what one would encounter when using
26

---

data such as what we have in the Tiny ImageNet experiment (Section 5.2 or in the ELCsTiCC experiment (Section 5.4). The results, calculated on an NVIDIA 1650Ti GPU, are shown Table 9
Although the performance of regular quantized RoPE is not far from standard absolute positional embeddings, when switching continuous RoPE the model is only 87% of the original speed. This is because we are unable to cache the RoPE frequencies between forward passes. With quantized position on the other hand, everything can be cached once before-hand and reused. While continuous RoPE incurs a notable performance penalty, it is not drastic. We note that other architectures specialized in irregular time-series also suffer from this issue, e.g., ContiFormer [[]] is reported as being 6 times slower than the vanilla transformer. The performance of RoMAE could likely be improved through a more optimized RoPE implementation. Quantizing the positions could also address this issue in datasets where it is reasonable to do.
B.3 Extrapolation
Being BERT-style model, ROMAE is not well suited for extrapolation. During training the bidirectional encoder sees all tokens that lie inside the observed temporal window, therefore it never learns an inductive bias for causal ordering or forward progress in time, and struggles with out-of- distribution positions during inference. Recent work on causal Transformers, for example GPT-family models equipped with RoPE [52 or exponential relative embeddings (54 [53] shows that a strictly unidirectional attention pattern together with position embeddings that extrapolate to unseen indices can capture temporal trends far more effectively. We argue that despite this limitation, RoMAE has a
place representation learning and interpolation framework, similarly to BERT in language.
B.4
Retaining Frequencies
PFT
0
Time domain
0.15
0.0015
MSC
0.10
0.0010
0.0055
21
Frequency (Hp)
Frequency (Hz)
Figure 3: Average MSE obtained from the interpolation task using RoMAE-tiny for time-series with a single varying frequency component. Left: MSE computed on the Fast Fourier Transform (FFT). Right: MSE in the time domain. We generate 200 time-series per individual frequency, with 50 observed noisy points and 50 masked (interpolated) points, thus limiting frequency of 25 according to Nyquist-Shannon sampling theorem. Error bars show the standard deviation of the MSE obtained for each individual frequency.
Here we investigate RoMAE's ability to retain high frequency modes interpolation tasks. It is known, for example as shown in References [2 64 that neural networks can exhibit spectral bias, in that the networks preferentially learn low-frequency components before high- high-frequency details. Examining how RoMAF reconstructs patterns at different frequencies provides insight into whether the rotational encoding allows to capture fine- mne-graaned structure during interpolation, with implications for understanding the inductive biases introduced by this positional encoding scheme.
To empirically assess RoMAE's ability reconstruct signals at different frequencies, we designed a controlled toy dataset of noisy sine waves. Each time series is defined over E [0, 1] and generated as the sum of one or two frequency components (with equal probability), where integer frequencies are sampled uniformly from f E [0, 24] Hz and Gaussian noise ~0, 0.01) is added. Each time-series has 100 data points, 50 of which are taken as input and 50 of which are masked for interpolation. We train RoMAF-tiny for 200 epochs on 10,000 examples. We then evaluate this model on (i) time-series with a single frequency mode as shown Figure 3 (ii) time-series with two frequencies modes as shown in Figure Using the 50 predicted (interpolated) points, compute the
27

---

MSE FFT (x10')
MSE Time Domain
0..00
+0000
035
0/21
-00000) 0.008
-0022
0122
022
0330
UUASE 01000
-0107
0143 UACA
Feequency 2 116
01000
1LEGS
0L.0N
11.05
Floquency 2 (Ho)
0131
0161
005 ILEC
Men MSE
01003
0.005
01062
0.065
05.35
0.005
0.67
0100
0166
00001
00.130
05151
0.124
0.111
BO88
0.127
Moat:
-0011
0.112
10S
0.00
6:079
CJISS
01221
01218
0.178
0.-45
0.100
0.127
0.120
01122
( 162
0.151
0.126
0.0C7
0.105
4.15
2X2
11201
I1217
0.170
0.142
0.:17
0223
2117
1:372
0.100
0.127
02122
9.10
0100
01:23
01.00)
01207
01218
0:229
0.133
0.173 1L129
01444
255
11220
1:215
0.202
0.177
157
0.100
0.10
03322
013022
01.081
(1.29)
0.200
0:312
02144
11:00
10
0157
(1254)
(.210
0.257
0.203
0223
0223
0.201
016
21
94
18
21
94
Prequemecy
(11x)
Proquemeyy
(llz
Figure 4: Same as Figure 3 but now for time-series with two frequency modes present in the signal. Left: MSE computed on the FFT. Right: MSE in the time domain. The time-series have 50 observed noisy points and 50 masked (interpolated) points.
fi
Hz,
Hu
Trith
Prediction
Prodiction
Oberrvation
0.6
Magnitade
Volue
0.4
0.2
1
0.00
0.25
0.50
0.75
1.00
0
10
20
30
40
Time
Frequency (HE)
Figure 5: Illustrative realisation from the evaluation of RoMAE on a the bi-frequency time series. Left: Interpolation in the time domain for composite sinusoidal signal with base frequencies and 5 Hz. Right: FFT of the ground truth and predicted waveform.
MSE in both the time-domain and Fourier domain. We plot sample prediction from the evaluation F This analysis was conducted with a mean signal to noise ratio (SNR) of 43. over entire training set. We acknowledge that the distribution of the MSE will be affected by an increasing SNR, resulting less sensitivity to higher frequency modes.
As expected, we observe general degradation of the reconstruction for higher frequencies, with an approximately linear trend error for frequency above 9Hz. For the case two modes, we observe the same overall trend with slight preference toward two higher frequency modes, as opposed to one low and one high mode. This due the sampling rate of the observations and the fact that the FFT for two higher frequencies has uni-modal power spectrum, yielding slightly better reconstruction. Lastly, we have checked that the above observations are maintained for non-sinusoidal signals. We repeated the analysis using non-sinusoidal periodic functions, specifically a square wave and a cycloid. We observed similar behaviour for the retention of high frequencies as with the sinusoidal experiment.
28

---

Appendix C Proofs
Notation: Here we define additional notation, on top of what is presented in Section 3.1 We define the block-diagonal rotation matrix R which contains the 2D rotation matrices corresponding to all 0 's:
1
cos 8, sin 0:) - sin 8
2
0
R
(4)
cos 8
When applying RoPE, R exponentiated by position m, then multiplied by 2m- E.g.: R Tm
Definition C.1 ([CLS] token). The Iearned [CLS] token is a vector Ieus E Rudd consisting of learnable parameters, that is appended to the start of sequence described in Section 4 The position of 7ets always zero.
C.1 Reconstructing Absolute Position Using the [CLS] Token
We now prove Proposition 4.2 by construction. The proof is based on the proof by Barbero et al. 4. showing that RoPE can be maximized for any relative distance Z. Here generalize this result to continuous position r E Q
Proof: Consider distance E + C R. query +b that is non-zero by assumption and key corresponding to the [CLS] token as described Definition C.I such that R' sb. Assume that the query is position j E Q+. We compute the dot product between rotated q and k:
(R'q) (R (R'k)
R - R -R-++
(5)
We now write this as the sum of dot products between the O,'s and each 2D subspace (0,
dmusaa/22
T t-
(b(10)
(6)
dantae/22
(6(18) / cos ((-j (--++r)0)
(7)
Because both j and r are Q+, and 0g is never multiple of क by definition, the unique maximum occurs when j A similar proof applies when r, E Q
Appendix D Full Experimental Details and Hyperparameters
D.1
Tiny Imagenet Experimental Setup
We present the unified Tiny ImageNet pre-training and fine-tuning hyperparameters in Table 10 All Tiny ImageNet pre- pre-training and fine-tuning runs use the same hyperparameters. Although our final results use FP32, we also tested mixed-preciste precision training and found that FP16 precision works with Tiny-ImageNet as opposed to our experiences on the ELAsTiCC dataset discussed Section D.6
When training, we normalize using the ImageNet mean and standard deviation. Each patch is individually normalized when calculating loss as done MAE 23 During fine-tuning, we also use RandAugment [ with 2 operations and magnitude of 9. While the model would likely benefit from more epochs during the pre-training and fine-tuning stages, we consider this sufficient for the purposes of an ablation study.
29

---

Table 10: Tiny ImageNet unified pre-training and fine-tuning hyperparameters.
Hyperparameter
Pre- Tre-irainng
Fine-Tuning
Optimizer
AdamW
AdamW
A betas
B1 = 0.9, B3 = 0.95
81 = 0.9, 3g = 0.999 0.05
Weight Decay
0.05
Base LR
2 x 10-4
x 10
Batch size
1024
1024
Epochs
200
15
Gradient clip
I 2000
Linear LR warmup steps
500
LR schedule
Cosine
Cosine
Dropout
0
0
Stochastic depth
0
Label smoothing c
0.9
Precision
FP32
FP32
D.2 Audio Experimental Setup
In Table II we present the unified pre-training and fine-tuning hyperparameters for the audio representation learning and classification experiments, discussed Section [.3 in the main text.
When training, both on Audioset [20) and Librispeech [36], we normalize the data using the mean and standard deviation estimated on the entire set of spectrograms. Each patch is individually normalized before calculating the loss as done MAE [23].
Table 11: Audio Experiment unified pre-training and fine-tuning hyperparameters.
Hyperparameter
Pre-Traie Pre-Training
Fine- Tine-Tuning
Optimizer
AdamW
AdamW
AdamW betas
0.9, 3o
0.95
B1 0.9, B2 0.02
0.99
Weight Decay
0.05
Base LR
10 10-4
10
Batch size
64
48
Epochs
150
50
Gradient clip
1
1
Linear LR warmup steps
1000
50
LR schedule
Cosine
Cosine
Dropout
Stochastic depth A
0
0
Label smoothing e Precision
0.8
FP32
FP32
D.3 UEA Multivariate Time-series Experimental Setup
We the same pre-training setting across all datasets, shown in Table 13 When fine-tuning, find necessary to have per-dataset hyperparameters. These are shown in Table 12 Because the datasets have varying sizes, resulting in vastly different numbers of training steps, we choose to scale the number of learning rate warmup steps as a percentage of total steps.
D.4 Pendulum Dataset
When training on the Pendulum dataset, we use a custom model size shown Table 14 The model size is chosen such that the MLP hidden dimension is equal to that of other models benchmarked on the dataset. To create the dataset, we follow the procedure from S5 [5I], using their published code
30

---

Table 12: UEA Multivariate Time-series Archive fine-tuning hyperparameters. Values that are constant across runs are reported only once.
Hyperparameter
BM
CT
EP
HB
LSST
Optimizer
SGD
Momentum
0.9
Weight Decay
0.
Base L.
10-2
10-3
2 10-3
2 10-2
3 10-2
Batch size
8
16
16
32
16
Epochs
50
100
150
30
15
Gradient clip
1
1
1
2
10
Linear LR warmup percentage
10%
LR schedule
Cosine
Dropout
0
0
0.2
0
0.2
Stochastic depth À
0
0
0.2
0
0.2
Label smoothing Precision
0.9
0.8
1
0.9
FP32
Table 13: UEA Multivariate Time-series Archive unified pre-training hyperparameters.
Hyperparameter
Value
Optimizer
AdamW
A betas
31 = 0.9, B3 0.95 0.05
Weight Decay
Base LR
3 10
Batch size
64
Epochs
400
Gradient clip
Linear LR warmup percentage
0.1
LR schedule
Cosine 0
Dropout
Stochastic depth X
0
Precision
FP32
Training RoMAE on the Pendulum dataset is generally very fast because of the small model size and the lack of pre-training.
D.5 Absolute Position Reconstruction Hyperparameters
Here provide full details for the experiments shown in Section 5.1 and Appendix B. These results can be seen Table 16 and Table 17 respectively. Across both experiments we keep the model size equal RoMAE-tiny as described in Table Table/Jexcepp for dioodel which set to 960 for the experiment Section B.I and set according to the corresponding model size for the experiment in Section 5.1 All experiments in Section 5.I use the same hyperparameters shown in Table 17 For both experiments we report the mean and standard deviation across different seeds.
D.6 ELASTiCC Experimental Setup
Full pre-trainie pre-training and fine-tuning hyperparameters for Section 5.4 can be found in Table [8 When creating the train/test split we use the code and pre-processing provided in the ATAT [7] code release" The alert mask removes points whose flux is either saturated or has high estimated error. A saturated flux occurs when the flux is higher than the maximum amount that the instrument measuring it can record.
Because the input values in ELAsTiCC nearly always larger than what would find with images (e.g., of the order of 10-50 when standardized as opposed to between and with images),
31

---

Table 15: Pendulum dataset end-to-end training hyperparameters.
Hyperparameter
Value
Table 14: Pendulum dataset custom model
size.
Optimizer
AdamW
AdamW betas
B1 0.9, B2 0.999
Model Parameter
Value
Weight Decay
0.01
Base LR
3 10-4
dmodel
60
Batch size
16
Mead
2
Depth
2
Epochs
50
Gradient clip
Dim. feed-forward
30
Linear LR warmup steps
1000
Num. parameters
37.4K
LR schedule
Cosine
Dropout
Stochastic depth À Precision
FP32
Table 16: End-to-end training hyperparameters for the absolute reconstruction range experiment (Section B.I. Values that are constant across all runs are reported only once.
Hyperparameter (0, 100)
(0, 1000)
Optimizer
SGD
Momentum
0.9
Weight Decay
0.
Base LR
5 x 10-6 6
5 x 10
Batch size
64
Epochs
10
Gradient clip
inf
Linear LR warmup steps
625
LR schedule
Cosine
Dropout
0
Stochastic depth
0
Precision
FP32
Table 17: End-to-end training hyperparameters for the absolute reconstruction MSE experiment
(Section 5.1).
Hyperparameter
Value
Optimizer
AdamW
Adam W betas
B1 = 0.9, 32 = 0.999
Weight Decay
0.01
Base LR
5 x 10Th
Batch size
64
Epochs
10
Gradient clip
inf
Linear LR warmup steps
625
LR schedule
Cosine
Dropout
0
Stochastic depth À
Precision
FP32
we found beneficial to increase the gradient clip threshold from the common value of In order to handle the variable number of points per sample utilize padding, applying pad mask to the attention scores. Although our final model was trained using full FP32 precision, we tested RoMAE with both FP16 and BF16 for mixed precision training, and found that FP16 resulted in NaN
32

---

Table 18: ELASTiCC full training hyperparameters.
Hyperparameter
Pre- Tre-Trainng
Fine-Tuning
Optimizer
AdamW
AdamW
A betas
B1 = 0.9, B3 = 0.95
81 = 0.9, 3g = 0.999 0.05
Weight Decay
0.05
Base LR
6.4 x 10-3
x 10
Batch size
16384
4096
Epochs
200
25
Gradient clip
10
10
Linear LR warmup steps
2000
2000
LR schedule
Cosine
Cosine
Dropout
0
0.2
Stochastic depth
0
0.2
Precision
FP32
FP32
values. This is likely due to to the increased input range ELAsTiCC interacting poorly with the reduced range of FP16. We found that BF16, with its larger range, worked well.
Each light curve in the ELAsTiCC dataset contains recordings of both the flux difference and variance. An example training sample is visualized in Figure showing how each band has different number of observations, each of which is at different time than the others. To convert each point in the light curve to an embedding we use a patch size of (1, 2) for time and flux/variance respectively. Therefore, during pre-training the model predicts not only the masked flux difference values but also variance for each point, while time and band index are embedded using position.
15
-8-
15
Normalizzel Flux
-15
25
-50
40
40
t
100
150
300
450
600
750
900
Time (days)
Figure 6: A training example from the ELASTiCC dataset. The flux difference each band has already been normalized.
33

---

Table 19; End-to-end training hyperparameters for the spirals dataset.
Hyperparameter
Value
Model size
Tiny Adam W
Optimizer
Adam betas
B1 0.9, 32 0.999 0.01
Weight Decay
Base L.R
3 10-1
Batch size
32
Epochs
500
Gradient clip
Linear LR warmup steps
2000
LR schedule
Cosine
Dropout
Stochastic depth À Precision
0
FP32
D.7 Spiral dataset
We construct dataset of 300 spirals as the prescription from Ref. [12. similarly allocating 200 for training and 100 for testing. Each spiral is randomly assigned to either clockwise or counter-clockwise, with parameters drawn from normal distributions
N/O, a:) and b~~N(033, a)),
where a 0.02. For the results presented in Table. and for comparison with ContiFormer, we add Gaussian noise sampled from A(0, (3) to the training samples, setting 3 = 0.1. The spirals were truncated at times corresponding to in both cases, and only the parts the spiral corresponding to the interpolation task carried out by Ref. [12 were used. Each spiral is discretized into 75 evenly spaced time steps. To create irregular time series data, time points are randomly selected from the first half of each spiral, which used for interpolation. We show the model hyperparameters used to generate the results Table
12
2
8
6-
Ground Trush
Sparse Oberrvaioos
4
Masked Observations
Predicted Values
0
2
0-
-2
-10.0
-7.5
-5.0
-2.5
0.0
2.5
3
5
X
X
Figure 7: Two sample realisations of differing chirality from the test set of spirals. The green line is the ground truth trajectory. The Red points are the 75 stochastic inputs of which 45 are masked. The blue points are the interpolated predictions.
The x and y coordinates of spirals are embedded using a patch size (1,2) for time and x/y coordinates respectively. We train for 400 epochs with learning rate of 10-3 Uncertainties presented Table 7 represent the evaluation uncertainties after trials with randomly seeded batches of 100 test spirals. The addition of the CLS token was observed to not significantly improve
34

---

performance. The code for generating the exact spirals used for this experiment, as well the details of the experimental setup for ContiFormer on their github
D.8 Synthetic dataset
Table 20: Synthetic dataset training hyperparameters
Hyperparameter
Value
Model size
Tiny Adam W
Optimizer
AdamW betas
B1 0.9, 32 0.999 0.01
Weight Decay
Base LR
1 10-3 3
Batch size
8 50
Epochs
Gradient clip
1 200
Linear LR warmup steps
LR schedule
Cosine
Dropout
Stochastic depth À Precision
0
FP32
using the same code generate the datase The dataset comprises 2,000 univariate time series, each with 50 uniformly spaced time points in [0, 1]. With patch size of (1, 1), each individual point is converted to token. Each trajectory generated by sampling 10 latent variables 2 NO.0 1) at reference times rk 0.1 k, and applying RBF kernel smoother with bandwidth a 120.0 to interpolate values across the timeline. Gaussian noise (0, 0.01) added simulate measurement error. mimic irregular sampling, we randomly select between 3 and 10 observed points per trajectory. The dataset split into 80% training, 10% validation, and 10% test sets. Performance assessed using mean squared error (MSE). We display results from some test samples for number of observed points n = 2, 10 and 20 in Figure g The addition of the CLS token was observed not significantly impact performance, We show the model hyperparameters used to generate the results in Table 20
We evaluate RoMAE on a synthetic interpolation task introduced by Ref. 47
3
10
20
1.0
Spose Slee koe
Manood Cheerotiee
Value
0.5
Value
Preeloted Valoes
Value
0.0
-1.0
0.00
0.25
0,50 Time
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,50
0.75
1.00
Time
Time
Figure 8: Samples from interpolation tests using 3, and 20 observations.
D.9 PhysioNet
We adopt the pre-processed release of the PHYsiONET/CinC 2012 Challenge (50, comprising multivariate clinical seme-seriss collected during the 48h window following intensive- carssiveecar---ee caaee-crreemmme unit (ICU) admission. Static covariates (Age, Gender, Height, ICU type) occupy feature indices 0-3 and are always observed, whereas the remaining 37 channels are sparsely and irregularly sampled.
35

---

In order to benchmark RoMAE we directly compare performance on the interpolation task using the same pre-processed version of the dataset produced by Ref. [35 which rounds the observation times to the nearest minute resulting in 2880 possible measurement times per time series. The data set includes 8000 instances that can be used for interpolation experiments. We additionally use the same experimental protocols which involve masking 50% of observed time points. Each multivariate record the PHYSIONET 48 clinical dataset is converted into a sequence of scalar tokens that RoMAE can process. Let z; E R denote the value of feature de (1,...,41) measured minute-resolune resolution time step (I...., T (T 2880); let m {0, 1} be the corresponding observation mask (1 = measured). We flatten the list (( pa)) with
grid into a one mne-dimmesional token
(d) In 24
Pr. [t/T, d)
N E
The two diodenenniooal positional vector P.. encodes (i) the normalised time t/T E [0, 1] and (ii) the feature index d, providing the plps = co-ordinates required by RoMAE. During training we stochastically subsample 50% of the observed tokens. The final input tensor hence has length N for the values, and shape (2, N) for the positions, and a Boolean mask of length N indicating which tokens RoMAE must reconstruct, exactly matching the interpolation protocol of the HcTVAE benchmark.
We show the results of interpolation study in Table 7 where we compare to HetVAE [3 as well as other models benchmarked in that study. We show the model hyperparameters used to generate the results in Table aloog with the addition of the CLS token that was seen to improve the results. Lastly, official Physionet challenge can be found on their website
Table 21: PhysioNet dataset training hyperparameters
Hyperparameter
Value Tiny Adam W
Model size
Optimizer
AdamW betas
81 0.9, 82 0.999 0.01
Weight Decay
Base LR
1 x 10-4
Batch size
16
Epochs
200
Gradient clip
1 100
Linear LR warmup steps
LR schedule
Cosine
Dropout
0
Stochastic depth À Precision
0
FP32
httg:r///hhssssaarreerrrroooeeed 2002/1...0/
36


Reviews:
[{"review_id": "gxIl8DGSth", "reviewer": "Reviewer_wtxL", "summary": "The Rotary Masked Autoencoder (RoMAE) adds a number of positional encoding improvements into Masked Autoencoders, studying in detail the variations of behavior that the resulting positional encodings allow. In a broad series of experiments across regular and irregular time-series, images, and audio, RoMAE is compared to existing baselines, quantified empirically, and studied technically to show the benefits of adding positional information into the MAE framework. This results in a simplification and unification in terms of architecture, compared to baselines in the respective domains studied.", "questions": "Related to weaknesses above, can you clarify a bit more the exact application of the continuous axial RoPE to multivariate, irregular timeseries? This seems intricate but critical for a number of areas, and a potential strength of RoMAE.\n\nOn infilling tasks, do you notice any particular bias in the model toward low or high frequency components? Most of the examples shown seem to have low frequency patterns, but showing that the model is also good at dealing with high-frequency, repeating ones would be useful for many applications.\n\nIn the limitations section, it is briefly mentioned that \"RoPE in RoMAE has some additional computational overhead if the positions are\n336 different with each forward pass, e.g., with any continuous irregular time-series\". This seems like a very significant drawback to me - as the speed of RoPE application can sometimes drives technical modeling decisions. Are there any empirical studies around this speed difference, compared to for example a model with no RoPE at all? Or one that receives the continuous position value directly as an additional input?\n\nThere are numerous studies around the interpolation of RoPE parameters for context expansion (YaRN for example). The interpolation section didn't mention much, if any, about these methods but they seem directly applicable for things like training on lower resolution images, and applying to higher resolution ones, or downsampled timeseries training, etc. Have the authors looked into this, or have some clarification about potential applications here?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThis paper features extensive empirical findings, experimental results, and extensions into a broad appendix with many interesting findings. The discussions about the subtlety of including the [CLS] token are practically relevant, and Figure 1 is a great introduction to the method, and the overall early description and background are clear. The results make sense and are compelling for the domains, many sections help the reader understand the work, and the RoMAE architecture seems useful and potentially simple to reproduce for new applications. The work seems broadly applicable to many areas given the generality of the MAE architecture, and the authors do a good job of conveying the various configurations of RoMAE that may be applicable to a given application area.\n\nWeaknesses:\nOne drawback from the broad swath of experimental and technical findings here, is that the paper flow feels a bit convoluted. I would strongly encourage the authors to re-visit the narrative structure of the paper, and re-organize. Delegating a number of core results tables to the appendix, while also having a half-page dedicated to irregular time-series, and numerous tables surrounded by empty white space, shows that a deep-clean in terms of paper format can both increase the information density of the paper, and also improve the flow for interested readers. Details about dataset construction are more appropriate for appendix material (beyond brief discussion in the main paper), rather than core findings.\n\nThe description of how to apply continuous axial RoPE to irregular time-series is a bit unclear to me, and would be greatly boosted by a figure showing the irregular time-series processing. Section 4.3 seems to take up a lot of space in the paper to describe a simple (but useful) finding, and might do with some changes to compress the information.\n\nGiven the core of MAE is fundamentally about representation learning, some qualitative exploration of learned representations could be a useful bridge between the detailed discussion of CLS token and absolute position, and dataset specific benchmarks in the later sections. Similar experiments were done in both VQ-VAE and MAE using linear probing, and may be useful to guide ideas in this direction.\n\nA second drawback from the breadth of experiments, is that the comparisons (generally) lack depth. It may be out of scope for this revision round, but consider a more direct application of this approach to \"standard\" benchmarks, adding direct classification scoring to the Tiny ImageNet results and adding in numerous relevant benchmarks, adding these representations as grafted representations to a large language model for multimodal tasks (similar to approaches like miniGPT, LLaVa, Freeze-Omni, and so on), or other benchmarks (direct comparisons to ViT, even if a small one, on some standard tasks). As it stands, the existing benchmarks seem time-series focused, but time-series model benchmarking usually covers an even larger set of datasets, so the paper currently sits in the middle of two camps, and stronger multimodal benchmarking would help me improve the score if that is the preferred direction for the authors.\n\nI do not find a key \"hero result\" to latch onto here, in a similar spirit to the (well-done) \"hero\" Figure 1. Generally, a lot of time is devoted to the details of absolute position prediction, which is useful but some portion of that discussion may be better suited to the appendix.", "was_revised": true, "final_justification": "The authors addressed a number of my questions, and the promised revisions should strengthen the paper. Given the updates, I think the paper is acceptable for the conference, but without application to categories of broader interest might be somewhat niche in its appeal - particularly the extrapolation aspect discussed by the other reviewers, as well as hinted at in my own review, is very important and is key within time-series applications. That said, the interpolation ability and simplicity of this approach make it appealing for a large number of time-series tasks as well. I do think it is a reasonably strong paper for the niche, once the focus on clarifying details has been added. Particularly, the author replies to reviewer questions has been thorough and answered some important questions around the method.", "rebuttal": "We thank the reviewer for their time and their useful feedback. We have taken on board several points highlighted in the review that we believe will strengthen our paper. \nLet us first address the questions:\n\n## Question 1:\nWhen dealing with multivariate irregular time series, we distinguish 2 cases depending on the type of irregularity of the data:\n\na) All the features present the same irregularity pattern (in time), meaning that all variates (features) are observed at each available time-step,\n\nb) Several axes are irregular (e.g. we observe one variate at a time step, then another at a different time-step),\n\nIn case a),  we utilize patchification to create one embedding combining all variates for each available time-step, embedding only time through the application of 1D p-RoPE.\n\nWhen dealing with case b), in practice, we flatten the irregular dimensions into a 1D sequence, saving their original “dimensional index” in their positional embedding. Any features present at *all* time-steps can either be encoded in additional positional dimensions, e.g., the time attribute could be encoded in its own dimension as every point has a time value, *or* through patchification. In the case of ELAsTiCC, we use two Continuous Axial RoPE dimensions; one storing the time of each token, and the other storing the index of the variate, with the value of each point being inserted through patchification. \n\nWe also point out that in case b), our novel theoretical analysis of absolute position reconstruction was required  to show that the model is able to deduce the “dimensional index” of each point. We are working on an additional Figure to make the irregular multi-positional encoding process clearer in the paper, along with improving the description in the text. \n\n## Question 2: \nThis is a very interesting question. Given that our model is trained using MSE, we hypothesize that it might suffer from the “spectral bias” towards prioritizing low frequencies (Rahaman et al, 2019 “On the Spectral Bias of Neural Networks”). However, and as highlighted in Rahaman et al., the picture might not be that straightforward depending on the data shape and the problem (see Sec 5 in their paper). Therefore, from a purely theoretical point of view, it is not trivial to give a clear-cut answer to this question. \n\nTo investigate more empirically if RoMAE is also good at handling high-frequency patterns, we are currently running experiments on a controlled toy dataset mixing sine-waves at different frequencies to investigate if higher frequencies are well retained during interpolation; this will be integrated in the revised manuscript. \nIt is also relevant to note that the ELAsTiCC dataset used in our Section 5.3 experiment contains a good diversity of frequency patterns; It is a dataset where each class corresponds to a type of astrophysical object with specific variability patterns and frequencies (some recurring, some not). RoMAE’s good performance on this dataset would thus tend to indicate that it is robust for various frequencies.\n\n## Question 3\nWe agree that evaluating the additional overhead incurred by continuous RoPE for irregular sequences is important. We would like to firstly point out that “Rotary Position Embedding for Vision Transformer” (Heo et al. 2403.13298)​​ showed that regular RoPE adds negligible additional cost to the overall computational cost. In particular, they showed that during inference, injecting 2-D RoPE into ViT-B adds 1.8 M FLOPs, just 0.01 % of the model’s total GFLOPs ( 17.6 G FLOPs).\nWe note that in general, positional embeddings for irregular time-series cannot be cached, as they change with every forward pass, which makes RoPE on irregular time-series slower than regular RoPE.\nWe have performed a quick test demonstrating relative speed up for a typical image (224x224) workload. The results are as following: \n| Model | Relative speed (Baseline = MAE)|\n|-------------|-------------:|\n| MAE  | 1 |\n| RoMAE (regular positions)  | 0.98  |\n| RoMAE (irregular positions)  | 0.87  |\n\n## Question 4: \nYes, using context expansion (for example as with YaRN) would absolutely be possible here, e.g. for images (as mentioned by the reviewer, training on lower-resolution and evaluating on higher resolution). We will add a sentence on that regard in the manuscript. \n\n## General comments\n**Hero result**: We would like to point out that our results on the ELAsTiCC dataset would constitute a remarkable result, given that RoMAE is the best performing model on this dataset (to our knowledge) which is of great interest within the astrophysics community. We therefore envisage follow-up studies, using models like RoMAE on real data, will be possible when real data from the LSST survey is released. The fact that analysis on such a dataset is even tractable, let alone feasible, should motivate readers to consider this result our ‘hero’ result. We did not emphasise this in the manuscript due to the scope of the conference, but , given your comment, we understood that a little more context is maybe needed for the camera ready version. \nWe also want to acknowledge the recommendations on the manuscript’s organization to improve its clarity and structure; we will restructure by moving the patchification subsection to the Appendix, collating Tables of results and moving the audio result in the main part, and will add an illustration for the positional embeddings in the multi-variate irregularly sampled case.\n\nWe also appreciate the comment regarding a qualitative analysis of the learned representations, using e.g. the linear probing. We agree that it is relevant for a variety of applications. We are currently running linear probing experiments on our TinyImageNet embeddings to compare the effect of CLS and absolute-position encoding, which will be included in the revised version.\n\nWe also agree that investigating the integration of RoMAE in multi-modal frameworks would be interesting and particularly relevant for e.g. applications in astronomy (related to the ELAsTiCC dataset), among others. However, we feel such an analysis would be out of the scope of the current manuscript (and realistically not possible for us to conduct properly within this round of rebuttal as acknowledged by the reviewer); we hope nonetheless that the breadth of experiments conducted (various settings, nature of the data and problems) are sufficient to showcase the advantages (and limitations) of RoMAE and motivate future works investigating multi-modal settings."}, {"review_id": "VpyrgYlCcZ", "reviewer": "Reviewer_mZNG", "summary": "This paper is focused on extending masked autoencoders (MAE) model for handling time-series data, but where the data are sampled at irregular time points. One way to do so would be to integrate position encodings/embeddings. This paper shows that incorporating rotary positional embeddings for continuous position encoding is a good idea. So, adapting and showing the potential value of RoPE to encode continuous temporal positions in MAE is the main contribution of this work. The experiments do show proof of feasibility in that in that on the datasets tested, the model achieves competitive performance without requiring specialized architectural modifications. There is some merit in showing that one model can handle both irregular temporal measurements and standard MAE use-cases.", "questions": "1. what were the core factors influencing the choice of baselines and datasets? Why were Mamba-based models and other more challenging datasets (where similar to what's described in the paper, irregular sampled data can be derived) not used?  \n\n2. The delta with S5 is small. Is there enough justification (due to significant benefits in other regime) to choose masked autoencoders for this task? There is at least some results questioning whether time-series data are a good fit for transformer based model. \n\n3. How pathological are the irregular sampled data that doing a standard imputation as a pre-processing, they cannot be fed directly to Chronos or TimesFM?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strengths:\n\n+As indicated in the summary, I find value in the experiments that a single model can achieve reasonable performance across multiple modalities (irregular time series, images, audio) without domain-specific modifications. \n\n+The use of ROPE, while not very novel, addresses the need of applying MAE models on irregular temporal data. The theoretical analysis does not provide any new insight, but is still OK. \n\n+As a proof of feasibility, I think that the experimental evaluation is reasonable. A few diverse tasks are covered and the model performs well across these applications.\n\n+The approach likely maintains efficiency advantages of MAE although this is not really discussed in deep detail in the paper. \n\nweakness:\n\n-An inability to handle extrapolation is more limiting than the paper acknowledges. Forecasting is one of the core use cases of time-series models and a model that is natively inapplicable to this setting will only be useful in interpolation and representation learning. While this does not mean that such a model is not useful at all, but it does question the claim about versatility highlighted in the paper's title. \n\n- The core technical piece is relatively incremental, substituting RoPE for standard positional encoding in MAE. The paper shows that this works - and so there is value in it -- but it does diminish the significance of the technical contribution.\n\n- Experimental evaluations leave out recent Mamba-based approaches and focus on UEA (why MIMIC, UCR etc were left out is not clear). In terms of positioning, I would have expected the paper to make a stronger case for why state space models are not inherently better suited -- especially when the sequence lengths become much longer. This issue is also relevant given that the paper uses a sequence flattening approach which can make scalability an even bigger issue.", "was_revised": true, "final_justification": "I appreciate the feedback from the authors. I find the experimental evidence valuable overall. The scope of the problem and the novelty of the technical contribution remain somewhat moderate.", "rebuttal": "We thank Reviewer 2 for their insightful response.\n\n## Question 1\nWe selected datasets that are both widely used for irregular multi-variate time-series benchmarks (UEA, PhysioNet) and directly comparable to prior MAE-style work (e.g., the synthetic interpolation dataset from HeTVAE). We could not run experiments on MIMIC due to its restricted credentialed access requirements.\n\nFurthermore, ELAsTiCC and UEA already provide a wide diversity in the datasets, with ELAsTiCC being significantly challenging for instance. We agree that adding evaluations on UCR would add value to the paper’s results, and are willing to conduct additional experiments on the UCR datasets.\n\nRegarding Mamba: its architecture is designed for causal, autoregressive generation, which is very different from our bidirectional approach. More simply, it aims to compete with autoregressive LLMs and not BERT-like models such as RoMAE. Instead, we found it more prudent to compare RoMAE to S5 (a state-space model known to excel at long sequences) to represent the strongest SSM baseline. This keeps our comparison “apples-to-apples”.\n\n## Question 2\nIt is true that on some tasks the raw accuracy gain over S5 is modest (and for one dataset S5 performs better); however, we want to point out that it is not always so (see e.g. results for EP, CT, in Table 4 of the manuscript). Ultimately, our goal is to demonstrate that RoMAE **can match** state-of-the-art SSMs on irregular time-series tasks without task-specific architectural tweaks, utilizing our smallest RoMAE architecture RoMAE-tiny. We also feel that our better results compared to other transformer-based models such as Contiformer (another masked-reconstruction model) exacerbate this fact. Lastly, bidirectional attention naturally captures global context for the interpolation task, a capability that causal models lack, making MAE‐style training particularly well suited for representation learning on sparse time series.\n\n\n## Question 3\nIn several cases, doing imputation (or interpolation) on the time series would be extremely difficult (and a problem in its own right). See for instance one example of an ELAsTiCC multi-variate time-series in Fig 4 in the Appendix which shows large windows of missing values; in addition to the windows of missing data to interpolate, in the specific case of ELAsTiCC (but likely relevant in other applications), the diversity of objects in the dataset makes it complicated to choose an interpolation/imputation strategy to model “well” all the time-series in the first place. Many researchers have expressed interest in approaches like RoMAE solely for its interpolation capabilities. The additional caveat of performing imputation as a pre-processing step is that if the imputation/interpolation is somewhat inaccurate or poor, this will inevitably impact the models used down the line (say e.g. as suggested Chronos). Avoiding this problem altogether seems like a benefit, at least in such cases where imputation would be non-trivial and potentially detrimental.\n\nWhen considering comparisons to autoregressive models such as Chronos or TimesFM, it’s important to note that they focus on forecasting on regularly sampled data, not on *masked-reconstruction* for representation learning and interpolation. RoMAE’s use of continuous RoPE within the MAE framework uniquely enables interpolation, a feature that is lacking from Chronos or TimesFM. Thus, their design goals and evaluation protocols differ fundamentally from ours: We aim for a unified encoder that can be fine-tuned or probed across downstream tasks (classification, interpolation), rather than a forecasting or point-forecasting model.\n\n\nBelow, we additionally address some points the reviewer brought up:\n### Extrapolation\nAs a BERT-style architecture, RoMAE is designed for interpolation and representation learning. RoMAE is, by design, not a GPT which is tailor-fit for next token prediction, a limitation we will now state explicitly in Sec 6 and the Abstract to avoid misleading claims or expectations. \nThe limited extrapolation capacity we observe is not unique to RoMAE. Zeng et al. (2023) (arxiv:2205.13504) report that vanilla Transformer/GPT-style forecasters are outperformed by a one-layer linear baseline on nine long-horizon datasets, attributing the gap to poor temporal-length extrapolation. Likewise, Chen et al. (2024) (arxiv:2402.10635) note that discrete-time Transformers ‘have limitations in generalizing to continuous-time data paradigms, motivating their ContiFormer variant. Similar constraints are reported with GPT models for NLP utilizing regular RoPE (see ALiBi arxiv:2108.12409). Combining RoMAE with a causal decoder, for example, is promising future work.\n\n###  “Incremental” technical novelty\nWhile our contribution with RoMAE might come across as incremental (combining existing techniques), we want to point out that it is nonetheless novel (as this combination was not investigated before) and relevant given the performances shown. Additionally, our paper clarifies that RoMAE is more than a simple RoPE swap: we (i) extend RoPE to multi-axis, real-valued “continuous Axial RoPE” (ii) provide the first theoretical results on when a [CLS] token suffices for absolute-position recovery, using these theoretical results to justify the integrity of our approach; (iii) demonstrate a single encoder that, without domain-specific heads or pre-training, reconstructs images, audio, and highly irregular time-series across many tasks."}, {"review_id": "uLAMcdfA8r", "reviewer": "Reviewer_joLj", "summary": "The paper proposes a variation of rotary positional embeddings (RoPE), which is applied to the MAE architecture. On various benchmark tests, the proposed method shows robust performance.", "questions": "- For sparse cases, the patch size might matter as they can sometimes cover a too large area to have a meaningful number of samples, or a too small patch could result in too local representations. How does the proposed method handle such trade-off?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- The RoPE method is nicely harmonized into the MAE framework. \n- By using Axial RoPE on continuous values and by using patch-level input that comes with real-valued position, it implements the RoPE version that works on irregularly sampled data.\n- It shows strong performance on Irregular classification tasks (e.g., ELAsTiCC, UEA datasets), interpolation tasks (e.g., PhysioNet, Synthetic), and irregular regression (Pendulum). \n\nWeaknesses\n- It has been mentioned that models like Latent ODE and HeTVAE can compete with their ability to use continuous time representation. It would be more insightful if these models are analyzed with computational costs. \n- The proposed method shows strong performance on the interpolation task, but extrapolation beyond the scope of the observed data is not well studied. \n- While the experiments were done on some irregular datasets, it is not thoroughly discussed how varying degrees of irregularity affects the performance. Very sparse data is not rare in machine learning applications. \n- The proposed method appears to be somewhat incremental as it's based on the existing Axial RoPE method and patchfication.", "was_revised": true, "final_justification": "Although the method is a combination of existing ideas, the paper presents meaningful innovations.", "rebuttal": "We thank the reviewer for their useful feedback. To answer their question on the patch size:\nRegarding the case of patches that are too small resulting in local representations: because of our use of bidirectional attention, every single embedding will have access to the full “information” globally, even if each original patch was very local. At the output of the model, there are no embeddings with only local information. However, the possible limitation here is that this could lead to many tokens. If this is the case, then one can follow what RoFormer (original RoPE implementation) did and use a linear attention variant. We would also like to add that the MAE framework in conjunction with a high masking ratio can also alleviate this issue.\n\n## Computational cost vs. Latent ODE / HeTVAE\nWe feel this is an excellent suggestion. We are currently working on such comparisons for the revised version of the manuscript. Due to the limited time for this rebuttal, we have performed a quick test demonstrating our relative speedup for a typical image (224x224) workload. The results are as follows: \n| Model | Relative speed (Baseline = MAE)|\n|-------------|-------------:|\n| MAE  | 1 |\n| RoMAE (regular positions)  | 0.98  |\n| RoMAE (irregular positions)  | 0.87  |\nWe will add further comparisons to, e.g., Latent ODE or HeTVAE for the final version of the manuscript.\n\n## Extrapolation\nAs a BERT-style architecture, RoMAE is designed for interpolation and representation learning. RoMAE is, by design, not a GPT-like model which is tailor-fit for next token prediction (extrapolation), a limitation we will state explicitly in Sec 6 and the Abstract to avoid misleading claims or expectations. \n\nThe limited extrapolation capacity we observe is not unique to RoMAE. Zeng et al. (2023) (arxiv:2205.13504) report that vanilla Transformer/GPT-style forecasters are outperformed by a one-layer linear baseline on nine long-horizon datasets, attributing the gap to poor temporal-length extrapolation. Likewise, Chen et al. (2024) (arxiv:2402.10635) note that discrete-time Transformers ‘have limitations in generalizing to continuous-time data paradigms, motivating their ContiFormer variant. Similar constraints are reported with GPT models for NLP utilizing regular RoPE (see ALiBi arxiv:2108.12409). Combining RoMAE with a causal decoder, for example, is a promising future work.\n\n## Sensitivity to the degree of irregularity\nThis is an interesting point which was also brought up by Reviewer Wjq7 (4). Indeed, this proposed study could be helpful for the community to understand the robustness of RoMAE to the degree/pattern of irregularity. However, a proper, thorough analysis would require designing a (synthetic) experiment where we control the irregularity and monitor the performance of RoMAE and other baselines. Moreover, we note that, to our knowledge, this type of investigation is not commonly done in the literature, and is perhaps better reserved for future work.\n\nMore qualitatively however, some of the current illustrations in the manuscript showcase that the datasets we used have varying degrees and patterns of irregularity that don’t seem to crucially impact RoMAE; e.g., in Fig 4 in the Appendix, a typical sample from ELAsTiCC, with large time-windows missing and –semi-periodic gaps–, and more subtle different sampling in each modality, while the pendulum dataset is somewhat more uniform. Section A.8.4  in the Appendix also provides an illustration of the interpolation results at test-time with varying degrees of sparsity (3, 10, 20 points), when trained on sequences with 3-10 datapoints.\n\n## The method is “incremental”\nWhile our contribution with RoMAE might come across as incremental (combining existing techniques), we want to point out that it is nonetheless novel (as this combination was not investigated before) and relevant given the performances shown. Additionally, our paper clarifies that RoMAE is more than a simple RoPE swap: we (i) extend RoPE to multi-axis, real-valued “continuous Axial RoPE” (ii) provide the first theoretical bound on when a [CLS] token suffices for absolute-position recovery; (iii) demonstrate a single encoder that, without domain-specific heads or pre-training, reconstructs images, audio, and highly irregular time-series across many tasks."}, {"review_id": "PHn1DTTFAu", "reviewer": "Reviewer_Wjq7", "summary": "This paper proposes RoMAE, which extends Masked Autoencoders to handle irregular time-series by adapting RoPE for continuous positions. The key contribution is Continuous Axial RoPE that encodes real-valued timestamps, enabling a single unified architecture to achieve competitive performance across irregular time-series classification, standard image tasks, and interpolation without specialized modifications, demonstrating that general foundation models can match domain-specific approaches when equipped with appropriate continuous positional encoding.", "questions": "1.\tHow exactly do the authors handle the continuous extension of RoPE when timestamps have vastly different or when dealing with multi-dimensional irregular data where different dimensions have different temporal resolutions?\n2.\tThe proposed masking appears to be uniform across patches, but irregular time-series naturally have varying information density. Are there any experiments with adaptive masking strategies that account for the irregularity pattern? \n3.\tThe experiments cover various datasets, but can it be characterized how RoMAE's performance varies with different types of irregularity patterns (e.g., completely random sampling vs. burst sampling vs. periodic gaps)?", "limitations": "1.\tThe paper lacks discussion of how continuous RoPE approximations in practice might deviate from their theoretical guarantees, especially for extreme timestamp ranges. \n2.\tLimited analysis of how sensitive the method is to key hyperparameters (masking ratio, patch size, p-value for truncated RoPE) across different irregularity patterns.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe authors provide formal analysis of position reconstruction capabilities and empirically validate these claims, offering both theoretical understanding and practical guidance on when to use [CLS] tokens for absolute vs. relative positioning.\n2.\tThe paper demonstrates that RoMAE achieves competitive or superior performance on irregular time-series, standard vision tasks, and interpolation benchmarks, providing convincing evidence that a unified architecture can match domain-specific methods.\n3.\tThe proposed method addresses the important challenge of handling irregular sampling in time-series data using standard Transformer components, eliminating the need for specialized architectures.\n\nWeaknesses:\n\n1.\tWhile the application is novel, the main technical innovation—using RoPE with continuous values—is relatively straightforward since RoPE naturally accepts real-valued inputs, and the theoretical analysis mostly confirms expected behavior rather than revealing innovative insights.\n2.\tThe approach is restricted to D ≤ 3 dimensions due to exponential token growth, and while the authors mention computational overhead for varying positions, they provide insufficient analysis of the practical computational costs compared to specialized time-series methods.\n3.\tThe experimental evaluation lacks comparison with recent large-scale time-series models like Time-LLM or other foundation models specifically designed for time series data.", "was_revised": true, "final_justification": "The authors' rebuttal addressed my primary concerns by providing new empirical data on computational overhead and offering clear technical clarifications on handling multi-dimensional irregular data. Given the thoroughness of the rebuttal and the new information provided, I raise my score from \"3: Borderline reject\" to \"4: Borderline accept\".", "rebuttal": "We thank the reviewer for their useful feedback. We would like to firstly address some misunderstandings that were conveyed by the reviewer when specifying perceived limitations of our paper. \n\n\n## Limitations \n\n**1.**\nSee Appendix A.5 (Figure 2): We have explicitly investigated how RoMAE’s position reconstruction error changes depending on the range of the positions. This identifies that the error does in fact increase as the range of the positions increase. We will add an additional (more extreme) case (going up to ~10^6) to the plot as well as some additional discussion of this for the camera ready version.\n\n**2.**\nDuring our testing of the model we evaluated the relationship between irregular absolute position reconstruction and p-RoPE p value and found that using a value of 0.75 did not have any adverse effects on the reconstruction error. We agree that investigating the effect of irregularity on hyperparameters is an important consideration and will add some additional discussion regarding this to the Appendix.\n\n\nTurning now to the questions:\n## Question 1 \nRegarding different temporal resolutions and/or ranges: this is a very interesting point, which might be very relevant in some applications (e.g. astrophysics with different observational facilities/types, healthcare monitoring,...), thank you for pointing this out. There are different ways to handle such situations with RoMAE. First, it is possible to use RoMAE as is, since the position embedding through RoPE contains a large range of frequencies: the model then has the ability to choose to focus on relevant frequencies depending on the time-scale it is interested in, e.g., lower frequencies tell the model more about long time-scales (see for instancem  Barbero et al. 2024, “Round and round we go! What makes Rotary Positional Encodings useful?”). \n\n\nHowever, this “out-of-the-box” approach might suffer from limitations for extremely large positional ranges (potential numerical issues) and/or in the case of temporal imbalance between different data modalities. Another option is thus to use separate positional dimensions to encode different temporal resolutions (especially relevant in the case of different modalities observed at different time resolutions, as suggested by the reviewer). In this case, one could store, for example, seconds in one positional dimension, and years in another.\n\n\n## Question 2 \n\nRegarding adaptive masking, our current experiments do not use dataset-specific masking strategies as we are interested in providing a general baseline on which any adaptive masking strategy would only improve upon. We highlight that RoMAE is compatible with any masking strategy the user may want. For example, RoMAE would likely benefit from the tube masking strategy proposed by VideoMAE (arxiv:2203.12602) when being trained on video. Additionally, when considering interpolation, utilizing domain knowledge to construct a mask that is representative of actual missing data in the particular dataset being studied would also likely provide an improvement over a regular uniform mask. We will update the discussion for the camera ready version to clarify these points. \n\n\n## Question 3\nRegarding analyzing the sensitivity/robustness of the performance with respect to the irregularity pattern (or degree): this is an interesting point which was also brought up by Reviewer 3 (joLj).   Indeed, this could be helpful for the community to understand the robustness of RoMAE to the degree/pattern of irregularity. A proper, thorough analysis would require designing a (synthetic) experiment where we control the irregularity and monitor the performance of RoMAE and other baselines. We note that to our knowledge this is not commonly done in the literature, and perhaps this experiment is better reserved for future work.\n\nMore qualitatively, some of the current illustrations in the manuscript showcase that the datasets we used have varying degrees and patterns of irregularity that don’t seem to crucially impact RoMAE; e.g., in Fig 4 in the Appendix, a typical sample from ELAsTiCC, with large time-windows missing and –semi-periodic gaps–, and more subtle different sampling in each modalities, while the pendulum is somewhat more uniform while still irregular (see Fig 3 in Appendix); Section C7 in the Appendix also provides an illustration of the interpolation results at test-time with varying degree of sparsity (3, 10, 20 points), when trained on sequences with 3-10 datapoints.\n\n\n## Considerations regarding dimensionality \nWe would like to address some of the weaknesses brought up by the reviewer, notably regarding the claim that the approach is limited to D < 4 dimensions due to exponential token-growth. We want to clarify here that this only becomes an issue when dealing with many *irregular* positional dimensions. We have additionally demonstrated via use of the ELaSTICC dataset, how RoMAE can be used with 6 irregular dimensions. The usage of MAE in conjunction with a high masking ratio (in places where this is applicable) also helps alleviate numerical issues with token-growth. We also reiterate that, as mentioned in the text, RoMAE is compatible with linear attention variants, emphasizing its versatility. \n\nWith regard to the comment about computational overhead (which has also been made by the other reviewers), we agree that it is important to measure this. Due to the limited time for this rebuttal, we have performed a quick test to estimate the  relative computational cost for a typical image (224x224) workload. The results are as follows: \n| Model | Relative speed (Baseline = MAE)|\n|-------------|-------------:|\n| MAE  | 1 |\n| RoMAE (regular positions)  | 0.98 |\n| RoMAE (irregular positions)  | 0.87 |\n\nWe also invite the reviewer to look at our response to  Rev. 1 Q1 for more details and considerations on irregular dimensions with RoMAE . \n\n## Comparisons to LLM’s for time-series\nSimilar points were also raised by Reviewer’s 2 (mZNG) and 3 (joLj).\n\nWhen considering comparisons to autoregressive models such Time-LLM (Jin et al, 2024) and TimesFM (Google Research, 2024), that focus on the forecasting task, not on *masked-reconstruction* for representation learning and interpolation. RoMAE’s use of continuous RoPE within the MAE framework uniquely enables interpolation, a feature that is lacking from Time-LLM or TimesFM. Thus, their design goals and evaluation protocols differ fundamentally from ours: We aim for a unified encoder that can be fine-tuned or probed across downstream tasks (classification, interpolation), rather than a forecasting or point-forecasting model."}]

Author rebuttal:

Meta-review: This paper improves the positional encoding of Masked Autoencoders to handle time-series data and evaluates them in a range of domains and against existing baselines. The feedback for this paper had been mixed, with reviewers arguing that the scope of the problem and the novelty are moderate, as well as with questions around handling extrapolation. During the discussion period, the authors addressed some of these questions and provided a thorough rebuttal to some of these questions. Overall, I recommend acceptance of this paper.
Decision: Accept (poster)