GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings
Authors: Jingyun Xiao
Ran Liu
Eva L Dyer
Venue: ICLR 2024 poster (2024)
Keywords: Time-series
Transformer
Spatiotemporal
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a group embedding framework which is aware of latent spatiotemporal structure of time-series data

Abstract: Analyzing multivariate time series is important in many domains. However, it has been difficult to learn robust and generalizable representations within multivariate datasets due to complex inter-channel relationships and dynamic shifts. In this paper, we introduce a novel approach for learning spatiotemporal structure and using it to improve the application of transformers to timeseries datasets. Our framework learns a set of group tokens, and builds an instance-specific group embedding (GE) layer that assigns input tokens to a small number of group tokens to incorporate  structure into learning. We then introduce a novel architecture, Group-Aware transFormer (GAFormer), which incorporates both spatial and temporal group embeddings to achieve state-of-the-art performance on a number of time-series classification and regression tasks. In evaluations on a number of diverse timeseries datasets, we show that GE on its own can provide a nice enhancement to a number of backbones, and that by coupling spatial and temporal group embeddings, the GAFormer can outperform the existing baselines. Finally, we show how our approach discerns latent structures in data even without information about the spatial ordering of channels, and yields a more interpretable decomposition of spatial and temporal structure underlying complex multivariate datasets.

GAFORMER: ENHANCING TIMESERIES TRANSFORM- ERS THROUGH GROUP-AWARE EMBEDDINGS
Jingyun Xiao*! Ran Liu', Eva L. Dyer!
Machine Learning Center
2 Department of Biomedical Engineering Georgia Institute of Technology, Atlanta, GA, 30332
ABSTRACT
Analyzing multivariate time series is crucial in numerous domains, yet learning robust and generalizable representations within such datasets remains challeng- ing due to complex inter-channel relationships and non-stationary dynamics. this paper, introduce novel approach for learning data-adaptive position em- beddings to incorporate learned spatial and temporal structure into transformer architectures. Our framework introduces group tokens and constructs instance- specific group embedding (GE) layer that assigns input tokens to a select num- ber of learned group tokens, thereby incorporating structural information into the learning process. Building on this, we propose novel architecture, the Group-Aware Transformer (GAFormer), which integrates both spatial and tem- poral group embeddings to achieve state-of state-of-the-art performance on various time series classification and regression tasks. Through evaluations on diverse time se- ries datasets, we demonstrate that GE alone can significantly enhance the perfor- mance of several backbone models, and that the combination of spatial and tempo- ral group embeddings allows GAFormer to surpass existing baselines. Moreover, our approach effectively discerns latent structures in data without prior knowledge of the spatial ordering of channels, leading to more explainable decomposition of the spatial and temporal structure underlying complex timeseries datasets. Code is available https///gthubbccommetttaa///AAA
INTRODUCTION
Multivariate time series (MTS) arise in variety of domains, from finance and traffic prediction to healthcare (Ghosh et al., 2009; Tsay, 2013; Che et al., 2018). MTS consist of many channels of univariate time series, where each channel has its own temporal dynamics, and many channels interact through latent interactions or dependencies across channels. The temporal dynamics, or temporal structure of each channel, together with the relationships across different channels, or the channel-wise structure, are both important when analyzing time series data (Zerveas et al., 2021; Zhang Yan, 2022). Being able learn shared spatial and temporal structures multivariate time series data is essential for obtaining robust representations and building inferences in downstream tasks.
Transformers have demonstrated impressive performance when being used to extract representations from MTS (Zerveas et al., 2021; Nie et al., 2022). To learn nonlocal interactions across different tokens in sequence, position embeddings are critical to encode the relative ordering between channels and over different points in time (Vaswani et al., 2017). However, standard approaches for position embedding (PE) that are used in language and vision are used can be problematic for the following reasons:
For general timeseries datasets, there is predetermined ordering or "spatial position" for different channels. Thus, unlike in language or vision, it is challenging to use positional embed- dings to build inductive bias to understand the relationships across channels (Su et al., 2021).
The relationships across channels and time segments might be instance-specific. For example, when localizing an event in the brain (e.g., seizure) using multiple electrodes spanning different
"Contact information: jxiao766 gatech.edu; evadyer gatech.edu.

---

regions in the brain, the dynamics would be altered depending the specific brain region where the seizure occurs (Shah et al., 2018). In this case, having a fixed set of positional embeddingg may lead to poor generalization.
These characteristies of timeseries make conventional positional embedding methods inadequate and ineffective. Moving forward, will be necessary to design novel types of embeddings techniques tailored to the spatiotemporal structure of MTS.
In this work, we present a novel framework for learning both channel and temporal structure in time series data and then integrating this information into our tokens through "group embeddings" (Figure 1). Our approach learns con- cise set of group-level tokens across the dataset and deter- mines how adaptively assign them to individual samples based on the similarity between the group embedding and specific sample embeddings. This versatile methodology can be employed across sequences to integrate group-level structures into transformer layers in either space time.
Typical
Transformer
Daturdees
Anstsssterrbedingr
Group Embedding Layer
Transformer
Building on this idea, we introduce the Awaue-Aware Trans- former (GAFormer). By integrating spatial and tempo- ral group embeddings with a spatiotemporal transformer architecture, GAFormer processes multivariate time series (MTS) data uniquely. It examines interactions across both temporal and spatial dimensions, allowing it to form more unified understanding of the structure within the data. Ad- ditionally, by decomposing our grouping into either the spa- tial or temporal dimension, to enhanced interpretability, making GAFormer a more ex- plainable architecture for modeling time series.
Dalo
GE
Sample
Sample
Group embedding layer On top, we show a standard position cmbed- ding layer used in transformers. Below, we show our proposed group embedding (GE) layer which learns a set of group to- kens and assigns them to different input tokens in data-dependent fashion.
Figure 1:
show that this also leads
We tested our proposed technique on both classification and regression tasks spanning number of different multivariate time series datasets. Our results suggest that group embed- dings can be used boost performance with various back- bones, and that by combining both temporal and spatial group embeddings, GAFormer can provide state-of-the-are performance
comparison to previous methods. When we further analyzed the learned group structure, found that GAFormer can reveal meaningful structure in data without any prior knowledge of the channel or temporal grouping in the data.
The major contributions of our work are follows:
We introduce novel data-adaptive group embedding (GE) technique that can be used to learn both spatial and temporal structure in multivariate time series datasets. GE can be applied flexibly to any transformer encoder, and we show its application in multiple backbones and architectures for incorporating both spatial and temporal group awareness.
We develop a group-aware transformer, named GAFormer, which provides a robust solution for learning spatial and temporal patterns that leads to improved classification.
In addition to providing enhanced performance, GAFormer offers meaningful explanations in variety of different types of time series datasets by revealing both spatial and temporal grouping structures within the data.
2
METHOD
In this section, we introduce our approach for building "group embeddings" and then introduce GAFormer architecture which uses a combination of spatial and temporal group embeddings.
2.1 GROUP EMBEDDINGS
Transformers provide powerful architecture for processing a wide range of data modalities, ranging from text and images, to temporal data. In all of these cases, the raw data must first be "tokenized"
2

---

Channel independent
Spatial group embeddings
Projector
Temporal group embeddingg
Patchified multi-channel
Te TH
Channel reduction
Combining spatial and temporal group embeddings with a channel-invarian transformer archi- Jecture. To learn both spatial and temporal groupings in multivariate timeseries data, GAFormer starts with a channel-Jee level tokenization of the imput before passing the data through channel- encoder. In the next stage of processing, we then learn spatial group tokens and embed this information in spatial group embedding (SGE) encoder;
Figure 2:
temporal transformer
stage, we learn temporal group tokens and embed this information in temporal group emebdding (TGE) encoder.
convert into sequence of tokens that can be effectively processed by a transformer. After tokenizing our data, we consider sequence of tokens X = X1...., x.]] E NNXD where N is the total number of tokens the sequence, and is the token dimensionality.
Due to the permutation invariant nature of self-attention position embeddings (PEs) are critical for the success of transformers (Huang al., 2020; Su et al., 2021). With both learnable and fixed position embeddings (e.g.. sin--cos), each token is augmented with set of non-adaptive embeddings P [p PNd R-xD to create a new sequence XPE = |x| + Pi..... XN + pNv. Thus, in traditional position encoding scheme, each time sequence is passed into the model, a token in a specific position in the sequence will be augmented with fixed embedding associated with that sequence position. We can also write this operation XPE + X P.
In contrast to this fixed scheme for PEs, we propose to build group embeddings (GEs) for input sequence in data-adaptive manner. To assign the group embeddings to input to- kens, we pass the input sequence to an transformer encoder layer Enc(.) to obtain a sequence (Enc(X)I,..... Ene (ne(X) N... Each token in this sequence is projected to K D dimensions through a learned weight matrix W E DxK, After projecting the sequence into a lower-dimensional space, then apply softmax function obtain the weights; this operation will effectively sparsify the coefficients that assign group tokens to input tokens. This group embedding operation GE(X) can then be written as follows:
GE(X) SoftMax(Enc(()) W)
where SoftMax(-) represents the softmax function applied to each sequence (column). The group embedding GE(X) is then added to the input tokens X, resulting in:
XGE + X GE(X).
(2)
Because we apply the softmax, we can sparsify the assignment weights and thus select small number of group embeddings each token. We find that in practice, this assignment is often 1- sparse and tokens in sequence are each mapped to single group embedding.
2.2 GAFORMER: A GROUP-AWARE SPATIOTEMPORAL TRANSFORMER
Based on the proposed group embedding technique, we propose GAFormer, spatiotemporal trans- former that concurrently extracts both temporal and spatial grouping structures through learning group embeddings in both dimensions (Figure 2). GAFormer is designed to learn representations from multivariate time series X Cxf, where C. represents total amount of channels, and T represents total amount of timepoints patches along the temporal dimension.
Tokenization Layer: Following the channel-invarian design of Nie et al. (2022), we first divide the complete temporal sequence from each channel into smaller "patches" (chunks in time), creating a tensor of shape X E x Pxt. where P denotes the total number of patches, and L represents the number of timepoints in each patch. We then forming = Token(X) E PxD, In practice, to both retain the channel-wise separation and also inform the model about the temporal semantics of each channel, we implement Token(-)
encoder Token(.) to tokenize each patch,
3

---

(A) Spatial correlation
(B) Results different settings
Model
2
object-
Stable Setting
Shuffle Setting
Biased Setting
Identify
100
channels
1
90.42
96.76
70-
learnable PE
Classify energy
sin-cos
60
Group tokens
50
11
Figure 3: Synthetic experiments on trajectories from a noisy many-body system. (A) To build the spatial correlation, we use mady- system with noise, and use the energy of the system to perform classification. Thus, the model first needs identify relevant objects, and then classify their total energy. (B) We compare the performance three settings: (Ieft) where the channel order is fixed in train and test, (middle) where the channel ordering is permuted during training and testing, and (right) where channel ordering training is distributed differently than in test. Our results show that group embeddings can provide robust performance in the presence of channel mismatch and distribution shifts.
as a transformer encoder with learned positional embeddings that processes each channel in X independently.
Spatial (Channel-Wise) Group Embeddings: To next extract the channel-wise interactions, we then slice the spatiotemporal sequences spatially, building denote the new set of P spatial sequences. In this case, we can use our group embedding strategy learn spatial or across-channel structure through a spatial group embedding (SGE) layer; here, we learn spatial set of group tokens Gs E R where Ks is the number of groups and assignments as described in the previous subsection. The spatial operations jointly update the latent embeddings Zs as below:
P sequences of length C. Let Zs
2' = Trans-S(Zss SGE(Zs)),
where Trans-S(-) denote spatial transformer encoder that operates sequences of tokens that are at the same point in time but vary across their channel dimension. Critically, since each tem- poral patch of data is an independent sample the spatial group embedding module SGE(-), can extract different spatial groupings for each time period. This gives GAFormer impressive ex- pressiveness, such that it gives customized spatial grouping to each temporal period based on the structure of the complete training dataset.
Temporal Group Embeddings: Next we apply dimension reduction layer H(-) to extract Zr ([ Øp]) E P2''' where channels D-dimension tokens are bottlenecked into one token of D'-dimension. We pass this temporal sequence through temporal transformer encoder Trans-T(-) and apply a temporal group embedding module TGE(-) to apply temporal grouping structure to the sequence. This results in final output which can be written as
z fnnl Trans- Tass-TTZ----- TGE(Zr)),
(4)
where GAFormer maintains a temporal set of group tokens Gr RK with Kr groups.
The architecture is trained end-to-end, where the parameters of the tokenization layer, the spatial encoder. the dimension reduction layer, and the temporal encoder are jointly optimized. When performing discriminative tasks, the tokens in Zfinal are averaged and fed into linear classifier or regressor depending on the task.
3 RESULTS
3.1 AN INTUITIVE EXAMPLE: NOISY MANY-BODY SYSTEMS
We first provide an intuitive example to motivate the benefits of our proposed group embedding. Specifically, examine system where the classification can only be performed after correctly identifying the channel groupings. In this scenario, we inspect the shortcomings existing posi- tional embedding strategies and assess the necessity of employing our proposed GF. technique.
4

---

Noisy Many-Body Systems: To dissect the performance of model, we consider mutivariate time series generated from the trajectories of many-body systems consisting of mutually interacting particles (Greydanus et al., 2019; Liu et al., 2022). In our experiments, the task is to classify the total energy of a many-body system to decide if it belongs to a high-energy or a low-energy system. However, to make the problem more challenging. we pollute the system with irrelevant objects that are just passing by and not interacting with the system. To solve this task, we assume that the system needs to identify the objects that are within the interactive system, and then (ii) classify the total energy of the system. We denote the resulting system as the noisy many-body system (Figure 3(A)).
For each noisy many-body system, we first initialize a pair of interacting objects and a pair of non-interacting objects (Liu et al., 2022), and then solve the trajectories with the Explicit Runge-Kuun Kutta method (Dormand & Prince, 1980). We sample 20 consecutive observations with a gap of 0.2 within time span [0, 10] for each system, and randomly generate 30,000 trials to provide rich variability of the trajectories. We compute the total energy of each system and keep the top and bottom quartiles for the classification labels. We then perform 60/40% train/test random split, and used the same generated dataset throughout.
Experiment Setup:
To benchmark the effectiveness of GE, we train the exact same transformer architecture with (i) learnable positional embedding (Gehring et al., 2017; Radford et al., 2018; 2019), (ii) parameter- free sin-cos positional embedding (Vaswani et al., 2017), and (iii) GE. Each architecture contains 4-layer transformer blocks with 4 attention heads and 32 dimensions, where for GE we share the first transformer layer the Enc(-) that generates group embedding coefficients depth of the transformer consistent. We train all models with learning rate of 0.0001 using the Adam optimizer (Kingma& & Ba, 2014), with batch size of 64 for 200k steps until the model converges. All runs are repeated for five random seeds.
keep the total
We examine the robustness of GE in three settings: (i) Stable setting, where the relative position of objects (channels) never shifts; (ii) Shuffle setting, where the observed objects could be any position and are randomized similarly in training and testing; (iii) Biased setting, where the observed objects have different positions that are randomly sampled from non-overlapping sets for the training and testing data.
Results:
As shown in Figure 3(B), the synthetic results demonstrate that GE is necessary for the successful modeling of MTS with varying spatial structures. When the channel structure known and fixed (Stable), GE performs similar as the other two position embedding techniques. However, when the channel structure is randomized and thus exhibits rich variability (Shuffle), GE shows impressive im- provement over learnable position embedding 10%) albeit having similar amount of parameters, and outperforms sin-cos position embedding by a large margin of 44 Finally, when the spa- tial structure distribution differs from the train and test (Biased), GE becomes the only embedding method that gives robust performance (=77%), while the other two baseline methods fail to learn (=:5%) The results on synthetic experiments demonstrate that GE is a necessary component for effective learning on datasets where the token-wise structure is unknown or biased.
3.2 TIME-SERIES CLASSIFICATION TASKS
In this section, we aim to validate: (i) The effectiveness of GE when is added into other architec- tures; (ii) The effectiveness of GAFormer when is used in multivariate seressrres.
Univariate and Multivariate Datasets: We validate the effec- tiveness of GE on both the univariate datasets and the muiltivariate datasets; and further examine the performance of GAFormer on muiltivariate datasets. Both datasets are selected from the UEA Time Series Classification bench- mark (Bagnall et al., 2018), where univariate time-series datasets contain InlineSkate (7 classes) (Mörchen, 2006), Earthquakes (2 classes), Adiac (37 classes) (Jalba al., 2004); While the multivariate time-series datasets contain MotorImagery
Table Classification performance on univariate time-series
datasets, We bold the best model and underline the second best.
InlineSkate
Farthquakes
Adiac
GRUDey Salem,
28.00
74.82
37.08
TCN 2017)
22.55
74.28
58.06
MVTS (Zeveus et al,, 1021)
22.18
74.82
57.54
MVTS TGE
34.73
76.26
61.64
A
12..5
11.44
14.10
Auto Transoces at, 2022)
33.09
75.54
67.02
AutoTrans + TGE
34.73
76.98
75.45
A
11.64
11.44
18.43
5

---

Table 2: Classification performance on multivariate time-seriet from the UEA benchmark. Results obtained through our (GE) approach are highlighted below each architecture in terms of change in accuracy A. The top performing model is boldface and second model is underlined.
SeliRegSCP2
FaceDetect
Ethanol
MotorImagery (c=64)
(c=7)
(c=144)
(c=3)
Avg.
NN DTW, DTW
48.30
51.90
29.30
45.13
53.30
51.30
30.40
39.0
43.5
53,90
52.90
32.30
50.0
47.28
GRU 2017)
51.11
56.56
34.60
51.0
48.32
TCN 2017)
53.89
66.60
30.04
50.0
50.13
MVTSazeveee al. 2021)
51.11
55.82
25.10
50.0
45.51
MVTS + TGE
51.67
61.75
30.42
55.0
49.71
A
10.56
15,93
15.32
15.0
14,20
Auto Trans et al,, 2023)
44.78
65.12
27.76
53.0
47.67
Auto'Trans + TGE
52.78
68.05
27.00
56.0
50.96
18.00
12.93
J0.76
13.0
13.29
PatchT (Nic 2023)
50.56
$4.99
25.86
54.0
46.35
GAFormer
56.11
67.99
41.44
61.0
56.64
(64-channel ECoG, 2 classes) (Lal et al., 2004), SelfRegSCP2 (7-channel EEG, classes) (Birbaumer et al., 2001), FaceDetect (144-channel MEG, 2 classes), and Ethanol (3-channel Spectrometer, 4 classes) (Large et al., 2018). The selected datasets present a multifaceted spectrum of challenges inherent to time series analysis.
Baselines:
For univariate experiments, we incorporate temporal GE (TGE) on top of two transform- ers for time-series classification: MVTS (Zerveas et al., 2021) and Auto Tutorranffommer (Ren et al., 2022). We also selected two non-transformer architectures, GRU (Dey Salem, 2017) and TCN (Lea ct al., 2017), for comparison.
For our experiments on multivariate time-series, we implemented a supervised version PatchTST (Nie et al., 2022), apply multivariate versions of the same baselines used for the univariate experi- ments, and compare against results for a nearest neighbor classifier (NN) (Peterson, 2009) and two dynamic time warping approaches that either use the same (DTW,) or different (DTWp) p) warping factors across dimensions, as reported in Bagnall et al. (2018). More details of benchmarks and model implementation stated in Appendix B.
For each dataset, we perform an 80/20% train/val split on the original training dataset, and select the best model on the validation set obtain results the testing set. We perform consistent evaluation on all experiments, where we train all models with the Adam optimizer with an initial learning rate of 0.0003, cosine annealing scheduler with warm restarts (Loshchilov & Hutter, 2016), a restarting period of 5 epochs, and the multiplying factor of 2. We set the maximum number of epochs as 300. All models are trained until convergence. We provide more details of the models and the optimization process in Appendix B.
Experiment Setup:
Univariate Results: Table 1, we studied the performance of baseline architectures with and with- out temporal group embeddings on number of univariate datasets. In all datasets, we show that when adding the temporal group embedding module (TGE), the performance of baseline transform- ers can be greatly improved. Especially, in InlineSkate, adding TGE to MVTS gives an impressive 112.55% increase accuracy; while Adiac, adding TGE to nutorransfrrmer gives 18.43% in- crease in accuracy. The experimental results demonstrate that adding temporal group embeddings to univariate time-series provides an impressive boost performance.
In Table 2, report the classification performance of GAFormer and the performance gain provided by GE across various multivariate datasets. Similar to the univariate results, show that integrating the temporal group embeddings TGE into the transformer archi- tectures MVTS and Tranffossorrmrr gives notable improvements in accuracy, and in the case of the SelfRegSCP2 dataset, the classification accuracy gain 18.00% Furthermore, when combin- ing spatial and temporal grouping structures. we show that GAFormer can obtain further boosts in performance, giving an average classification performance increase of 16.. Our analysis demon- strates the benefits of group embeddings, highlighting its versatility both on its own and when im- plemented as an independent architecture.
Multivariate Results:

---

Table 3: Performance on neural population decoding including both line-series classification tasks (Mihi-
Chewie) and regression tasks (NLB). We bold the best model and underline the second best one.
Classification (Acc)
Regression (R*)
Chewie-1
Chewie-2
Mihi-1
Mihi-2
NLB-Maze
NLB-RTT
GRU Salem,
75.00
94,44
73.81
86,05
0,8887
0.5951
TCN at., 2017)
78.13
91.67
90.48
81.40
0.8946
0,5407
(D & Pandarimat,, 1021)
81.06
88.89
88.10
90.70
0,8708
0,4621
EIT et 2022)
75.00
77.78
78.57
65.91
0.8791
0,4691
Fororrer
81.25
94.44
92.86
88.37
0.9136
0.5433
3.3 CLASSIFICATION AND REGRESSION TASKS ON NEURAL RECORDINGS
Brain-computer interfaces (BCIs) enable the direct translation neural activity into outputs that can control external devices, bridging the gap between the brain and machines. Crucially, the success of BCIs heavily relies on the accuracy of the neural decoding methods, which benefit from the identification of neuronal function groups. Thus, we examine GAFormer for neural decoding tasks.
Neural Decoding Datasets: We systematically evaluate GAFormer across six neural decoding datasets that capture neural population activities from the motor cortex of nonhuman primates gaged in different movement tasks (Pei et al., 2021; Liu et al., 2022). In neural recording datasets, the activities of individual neurons are sorted into distinct channels, and the number of spikes across temporal period (20ms) are counted to produce multivariate time-series.
We first test GAFormer the Mihi-Chewie reaching dataset for classification (Dyer et al., 2017). The dataset consists of stable behavior-based neural responses from different neuron populations and animals performing the same task (Dyer et al., 2017; Liu et al., 2021), where two rhesus macaques, Chewie and Mihi, were trained to reach one of eight locations. While executing different reaches, neural activity in their primary motor cortex was continuously recorded across the two subjects on two different days, forming a total of four sub-datasets
We also examine the performance of GAFormer on the Maze and the Random Target Task (RTT) tasks of the Neural Latents Benchmark (NIB) for regression (Pei et al., 2021). The Maze dataset records the neural activity from the dorsal premotor (PMd) and primary motor (M1) cortices, where the objective is to predict the hand movement trajectories of the observed subject. The RTT dataset is a unique self-paced sequential reaching task set amidst random elements of a grid. The dataset contains neural spikes from the primary cortex, which are used to predict the subject's 's hand position.
We benchmark GAFormer against existing state-of of-the-an art transformer-basee models for neural data, including NDT (Ye & Pandarinath, 2021) and EIT (Liu et al., 2022), as well as traditional methods GRU and TCN. For all experiments, we use the same GAFormer architec- ture stated in Section 3.2, and simply add one linear projection layer the final embeddings of GAFormer to predict the hand velocities for the regression task with an MSE loss. For all experi- ments, we train the model for 300 epochs, and optimize the network with an Adam optimizer and cosine annealing learning rate scheduler, and report the converged accuracy and R2 score on the test set. We report additional details about the hyperparameters in Appendix
Experimental Setup:
Results and Insights: We show the experimental results as in Table 3. Across the board, we find that GAFormer provides strong performance on all neural decoding tasks, with major improvements over the baseline methods in multiple instances. The good performance happens in both classi- fication tasks and regression tasks, where significant improvements in the Chewie-2 and Mihi-1 datasets of a classification accuracy of 94.44% and 92.86% are observed, surpassing the previous state-of-the-art by 13%. For the more complicated regression tasks, GAFormer also gives the new state-of-thed afrt, demonstrating robust decoding performance decoding neural activities. The neural decoding results demonstrate that GAForner can consistently outperform the previous state-of-the- arts, suggesting its adaptability and promise for a diverse range of tasks.
Overall, the GAFormer model showcases versatility across datasets, highlighting its potential as a robust tool for neural decoding in BCIs, even when compared against the previous state-of-the-an models. Besides the impressive classification performance, another major advantage of GAFormer its interpretability, which is demonstrated through the visualizations in Appendix C, where the structural information about neurons organizations as well as temporal stages are generated through group embedding assignments automatically.
7

---

3.4 ABLATION STUDIES
To understand the contributions of various components in our GAFormer architecture, we conducted a number of ablations the model. These experiments aimed quantify the impact of temporal group embedding and spatial group embedding within the system.
The first ablation revolved around understanding the performance of GAFormer without any of the group-aware embeddings (Table 4, Base). By comparing this stripped-down version with the full model, we can gauge the performance gain offered by our group-aware embedding mechanism. We found that training the model without group-aware embeddings resulted in suboptimal performance across all the UEA datasets that we tested. This demonstrates that our proposed group embedding strategy is integral to harnessing the intricate patterns present in multivariate time series.
The second ablation aimed to dissect the individual impacts of spatial and temporal group embeddings. To do this, we trained two distinct model variants: one exclu- sively leveraging spatial group embed- dings (+SGE) and the other reliant solely on temporal group embeddings (+TGE). Both variants improved over the base model with no group embeddings. How- ever, neither reached the performance of the combined spatial and temporal model, indicating that the joint utility of spatial and temporal tokens plays crucial role in the learning process.
Table 4: Ablotions. Our base channel invariant architecture (left), with temporal GE only, with spatial GE only, and using GAFormer to extract both spatial and temporal GE
Base
+TGE
+SGE
Ours
SelfRegSCP2
52.78
53.33
55.00
56.11
FaceDetecct
53.80
56.83
61.55
67.99
Ethanol
25.10
30.04
37.64
41.44
MotorImagery
50.0
54.0
58.0
61.0
3.5
VISUALIZZTIONS OF THE GROUP ASSIGNMENTS
key advantage of our group-awe aware architecture is that, once trained, we can examine the spatial and temporal group embedding assignments that learned by the model. Thus, studied the spatial and temporal embeddings learned by GAFormer in the Motorlmagery ECoG dataset (Figure 4) and the NLB RTT dataset (Appendix C.I). In both cases, found clear temporal grouping structure that appeared to segment the data in data-adaptive manner, with the temporal grouping structure of NLB RTT aligning well with the intrinsic structure the movement task (Pei et al., 2021). In the case of the Motorlmagery dataset, we also found distinguishable grouping structures along the channel (spatial) dimension. In this case, the imagined movements of fingers have huge variation (Figure 4 left), while the imagined movements of the tongue exhibit similar grouping structures across trials (Figure right). The distinction of grouping structures across the different classes seem to align with the temporal complexity of the task, the movement of the finger is originally richer than the movement of tongue.
-Fnger
SGEIXI Finger2
SGED0 TEEDO-Tonggu
SGETO Tonguc
Charnels
FEnge-Fnger1
TGEXI-Fingre
(GEXI-Tongan
TGEX)- Tongue
Figure 4: Spatial (top) and temporal (bottom) groupings for the Motorlmagery dataset. The two examples on the left and right are from Class (Finger) and Class 2 ((Toguee, respectively.
4 RELATED WORK
Position Embedding: Position embedding plays a critical role in the transformer architecture,
whether dealing with textual, images, to temporal data (Huang al., 2020; Su et al., 2021). Given
8

---

the order-agnostic nature of transformers, positional embeddings introduce the essential context needed to build complex relationships across different parts of a sequence. For this reason, numer- ous studies Li et al. (2021); Zheng et al. (2021); Ke et al. (2020); Wu et al. (2021), have delved into the specifics and refinements of simple positional embeddings. Among existing techniques, our approach shares similarities with the conditional positional embedding that is used in visual repre- sentation learning (Chu et al., 2021). However, their method focuses on instance-specific learning of positional embedding without specifically targeting the discovery of grouping structures, which is one unique advantage of our proposed approach. Moreover, it is worth noting that recent works on multivariate time series also explored the use of spatiotemporal embeddings that have been de- veloped for traffic forecasting (Liu et al., 2023) and future location prediction (Lin et al., 2021). Different from them, the robustness of GE relies on the design of the grouping structure, which effectively identifies distinct spatial and temporal structure in MTS datasets.
Set Discovery and Prediction: In the domain of computer vision, Locatello el. (2020) introduced slot attention, which is used for weakly supervised set discovery and prediction problems. Drawing parallels with our proposed methodology, both techniques require the construction of a fixed-size group tokens', which are referred to 'slots' in slot attention. However, embedding differs from slot attention and its successive works (Patrick et al., 2021; Siméoni et al., 2021; Ibrahim al., 2023) due to the unique multivariate nature of timeseries, and the data-adaptive nature of our approach. We hypothesize that the performance gain obtained by our approach is due to the uniqueness of the structure of MTS, which is often neglected in previous works.
proposed group
Transformers for Multivariate Timeseries: Initial approaches to integrating transformers for mul- tivariate time series involved tokenizing short context windows from sequences. These tokenized segments were subsequently embedded using Multi-Layer Perceptrons (MLP) or Temporal Con- volutional Networks (TCN) (Zerveas et al., 2021). Despite its foundational nature, this method exhibited limitations in extracting channel-specifie features and incorporating them into the training process. Recent methodologies, notably EIT (Liuu et al., 2022) and PatchTST (Nie et al., 2022), in- troduced a channel-independenn design, for classification and forecasting tasks, respectively. Build- ing on this work, Wang et al. (2023) introduces a spatiotemporal architecture for traffic prediction.
coaneelindeeendenn transformer
In the realm of forecasting, PatchTST (Nie et al., 2022) has introduced an innovative patch design tailored for time series predictions, maintaining channel-invarian design. Similarly. FrossFormer (Zhang & Yan, 2022) prioritizes patch designs but with a focus on exploiting inter-channel depen- dencies. Such methodologies underscore the importance of channel relationships for enhancing forecasting outcomes. Other significant contributions in this domain include works like (Zhou et al., 2021; 2022; Wang et al., 2024), which have expanded the use cases of mansformer-based forecasting.
5 DISCUSSION
In this work, we introduced novel framework for building data-adaptive position embeddings to enhance time series transformers. By adaptively assigning group-level tokens to individual samples based on similarity, our method effectively integrates channel -wise and temporal groupings into the transformer architecture. Empirical validations on several time series benchmarks demonstrate how group embeddings can be integrated into a number of existing transformer models. Furthermore, the Group-Aware Awaup- Transformer (GAFormer) architecture developed in this work offers a robust solution for capturing and leveraging the complex interactions within multivariate time series data, leading to enhanced performance and interpretability without the need for known ordering or spatial structure across channels.
Limitations and Future Work: While GAFormer has shown promising results, it partially relies on the lhannel-indeennen design as proposed in Nie et al. (2022); Liu et al. (2022). While this approach has many advantages, as discussed in Han et al. (2023), a channel-separable design might not adequately capture the complexities of data with intricate inter-channel dynamics, leading to suboptimal representations in certain datasets. In addition, as the spatial and temporal dimensions grow, more data is needed for the model to effectively extract reliable grouping structures. Moving forward, improving the group embedding module for effective training on high-dimensional MTS data would be an exciting line of research. Additionally, more in-depth examination of model interpretability, possibly using advanced quantification metrics or visualization tools, can shed light on how the group tokens capture and represent data dynamics.
9

---

ACKNOWLEDGEMENT
This project is supported by NIH award IR01EB029852-01, NSF award IIS-203744,, NSF award IIS-2212182, NSF CAREER award IIS-2146072, as well as generous gifts from the Alfred Sloan Foundation, the McKnight Foundation, the CIFAR Azrieli Global Scholars Program.
REFERENCES
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint 111: 00075, 2018.
N Birbaumer, H Flor, N Ghanayim. T Hinterberger, Iverson, E Taub, B Kotchoubey, A Kbler, and Perelmouter. brain-controlled spelling device for the completely paralyzed. Naturea398, pp- 297-298, 2001.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1 8(1):6085, 2018.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pp. 1597-1600. IEEE, 2017.
John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
Eva L Dyer, Mohammad Gheshlaghi Azar, Matthew G Perich, Hugo L Fernandes, Stephanie Naufel, Lee E Miller, and Konrad P Körding A cryptography-basn based approach for movement decoding. Nature biomedical engineering, 1(12):977-76,, 2017.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pp. 1243-1252. PMLR, 2017.
Bidisha Ghosh, Biswajit Basu, and Margaret O'Mahony. Multivariate short-term traffic flow fore- casting using time-series analysis. IEEE transactions on intelligent transportation systems, 10(2): 246 254, 2009.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Lu Han, Han-Jia Ye, and De-Chuan Zhan. The capacity and robustness trade-off: Revisit- ing the channel independent strategy for multivariate time series forecasting. arXiv preprint arXiv:2304.cm 05206, 2023.
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. arXiv preprint arXiv:2009.13658, 2020.
Muhammad Ibrahim, Naveed Akhtar, Saeed Anwar, and Ajmal Mian. Sat3d: Slot attention trans- former for 3d point cloud semantic segmentation. IEEE Transactions on Intelligent Transporta- fion Systems, 2023.
Andrei C Jalba, Michael HF Wilkinson, and Jos BTM Roerdink. Automatic segmentation diatom images for classification. Microscopy research and technique, 65(1-2):72-85, 2004.
Guolin Ke, Di He, and Tic- Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006. /5595, 2020.
10

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Thomas Lal, Thilo Hinterberger, Guido Widman, Michael Schröder, N Hill, Wolfgang Rosenstiel, Christian Elger, Niels Birbaumer, and Bernhard Schölkopf. Methods towards invasive human brain computer interfaces. Advances in neural information processing systems, 17, 2004.
James Large, E Kate Kemsley, Nikolaus Wellner, lan Goodall, and Anthony Bagnall. Detecting forged alcohol non-invasively through vibrational spectroscopy and machine learning. In Ad- vances Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part I 22, pp. 298-3092 Springer, 2018.
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 156-165, 2017.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34:15816-15529, 2021.
Yan Lin, Huaiyu Wan, Shengnan Guo, and Youfang Lin. Pre-training context and time aware loca- tion embeddings from spatial-temporal trajectories for user next location prediction. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 4241 144444448,22 2021.
Hangchen Liu, Zheng Dong, Renhe Jiang, Jiewen Deng, Jinliang Deng, Quanjun Chen, and Xuan Song. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting. arXiv preprint arxiv:2308. /0425, 2023.
Ran Liu, Mehdi Azabou, Max Dabagia, Chi- Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith Hengen, Michal Valko, and Eva Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in neural information processing systems, 14:1779-1,,,99 2021.
Ran Liu, Mehdi Azabou, Max Dabagia, Jingyun Xiao, and Eva Dyer. Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. Advances in neural information processing systems, 35:2377-2391, 2022.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:11525-11538 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399: 2016.
Fabian Mörchen. Time series knowlegde mining. Görich und Weiershäuser, 2006.
Yugi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint urXiv:2211.14730, 2022.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and Joao F Henriques. Keeping your eye on ball: Trajectory attention in video transformers. Advances in neural information processing systems, 14:12493-12506 2021.
Felix Pei, Joel Ye, David Zoltowski, Angi Wu, Raced H Chowdhury, Hansem Sohn, Joseph E O'Doherty, Krishna V Shenoy, Matthew Kaufman, Mark Churchland, et al. Neural latents benchmark'21: evaluating latent variable models of neural population activity. arXiv preprint arXiv:2109.04463. 2021.
Leif E Peterson. K-nearest neighbor. Scholarpedia, 4(2):1883, 2009.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
11

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yankun Ren, Longfei Li, Xinxing Yang, and Jun Zhou. Autotransformer: Automatic transformer architecture design for time series classification. In Pacific-Asia Conference on Knowledge Dis- covery and Data Mining, pp. 143-155. Springer, 2022.
Vinit Shah, Eva Von Weltin, Silvia Lopez, James Riley McHugh, Lillian Veloso, Meysam Gol- mohammadi, lyad Obeid, and Joseph Picone. The temple university hospital seizure detection corpus. Frontiers in neuroinformatics, 12:83, 2018.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supprvised transformers and no labels. arXiv preprint 2rXii:2119.. 14279, 2021.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0900 2021.
Ruey S Tsay. Multivariate time series analysis: with R and financial applications. John Wiley & Sons, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Yule Wang, Zijing Wu, Chengrui Li, and Anqi Wu, Extraction and recovery of spatio-temporal structure in latent dynamics alignment with diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
Zepu Wang, Yuqi Nie, Peng Sun, Nam H Nguyen, John Mulvey, and H Vincent Poor. St-mlp: A cascaded spatio-temporal linear framework with channel-independence strategy for traffic fore- casting. arXiv preprint S33V:23008.074966 2023.
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improv- ing relative position encoding for vision transformer. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 10033-10041, 2021.
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint arXiv:2108.011 0/2/00 2021.
George Zerveas, Srideepika Jayaraman, Dhaval Patel. Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer-basee framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference knowledge discovery & data mining, pp- 2114-2124 2021.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
Jianqiao Zheng, Sameera Ramasinghe, and Simon Lucey. Rethinking positional encoding. arXiv preprint arXiv:2107.02561. 2021.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 106-11115, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pp. 27268-27286 PMLR, 2022.
12

---

APPENDIX
A DATASETS
A.1 UEA DATASETS
Table Selected datasets from UEA Multivariate Time Series Classification archive.
Dataset
Train
Test
Channels
Length
Classes
MotorImagery
100
64
3000
2
SelfRegSCP2
200
180
7
1152
2
FaceDetection
5890
3524
144
62
2
Ethanol
261
263
3
1751
4
A.2 NEURAL DATASETS
Table Neural datasets.
Dataset
Train
Test
Units
Length
Classes
NLB-Maze
1721
574
182
140
NLB-RTT
810
270
130
120
Mihi-Day
167
42
162
76/84
8
Mihi-Day 2
172
43
152
73/87
8
Chewie- Day
32
163
Chewie-Daa 2
144
36
148
75/77
8
B EXPERIMENT DETAILS
MODEL IMPLEMENTATION AND HYPERPARRMETTRS
For the patching tokenization layer of PatchTST and GAFormer, set the patch window 10 for SelfRegSCP2, Motorlmagery, and Ethanol, set the patch window as 2 for FaceDetec since the data length short. For all neural datasets (Mihi-Chewie, NLB-Maze, NLB-RTT), window as to better evaluate the potential of our model for studying fine-grained dynamics of neural activities. For all experiments and all models including GRU, TCN, MVTS, AutoTrans, the token dimension and embedding size are 256. For Transformer layers, the number of head is set as 16. In GAFormer, we set the depth of spatial/temporal group embedding module and spatial transformer encoder as 3, and the depth for final temporal transformer encoder 6. For the added TGE module to other baseline models, we kept the depth and dimension consistent with the TGE in GAFormer. For fair comparison, the depth of MVTS, AutoTrans NDT and EIT are kept the same as the total depths of spatial and temporal Transformer layers GAFormer. We set the number of groups to be 10 in all group embedding modules. Following Zerveas et al. (2021) and Nic et al. (2022), we adopt batch normalization rather than layer normalization in all Transfomer architectures.
set the patch
B.2 TRAINING
We did use pretraining data augmentations for all experiments except for the MotorImagery dataset. In the Motorlmagery dataset, the number of samples is limited and the length of each trial
TGE
T-Fmeoder
Dim
Head
Group K
Patch size
3
6
256
8
10
10
Table 3: Hyperparameters used for univariate datasets.
13

---

Dataset
Chamnel-independent Encoder
SGE
S-Encoder
TGE
T-Encoder
Dim
Group K
Patch Size
FaceDetect
3
6
256
8
10
MotorImagery
3
256
8
SelfRegSCP2
3
8
256
8
10
Ethanol
3
3
3
8
256
8
10
Mihi-Chewie
3
3
6
256
16
1
NLB
3
8
256
8
Table 4: Hyperparameters used for maltiariaate datasets. The first columns represent the depth of each transformer component.
(3000 timepoints) is long even with patching tokenization. Thus we split each test sample from 3000 timepoints to be 3 samples of 1000 timepoints and formed larger dataset. In the training stage, we randomly select segment of 1000 timepoints from each training trial as strong data augmentation. For models with patching tokenization, we set the batch size as 32. For models with no patching tokenization (MVTS, AutoTrans), we set the batch size as 16 since the number of tokens large.
C ADDITIONAL VISUALIZATIONS
C.1 VISUALIZATIONS ON REAL-WORLD DATASETS
Notural lpaking data
S6100
TGE
1.0
channss
Chonnaas
1.0
channel
Figure 1: Visualization of Group Embeddings for Neural Spiking Datasets. (Left) Spiking data, (Middle) Spa- tial group embeddings, and (Right) Temporal Group Embeddings for the Neural Latents Benchmark Random Target Task.
14

---

Nouraa spikking dataa
SCE00
TGEDO
crannass
Time
Time
Figure 2:
Visualization of Group Embeddings for Neural Spiking Datasets, (Left) Spiking data, (Middle) Spatial group emheddiings, and (Right) Temporal Group Embeddings for the Neural Latents Maze Task.
As shown on the right in Figure 2, we observed consistent cutoff in the temporal group assign- ments, where the initial 50 timepoints were mostly categorized as group 1. We notice that this corresponds to the onset time (at 250ms in each trial, which is binned as the 50th timepoints with bin size of 50ms) of the movements of monkey subjects the studied MC.Maze subdataset (Ye & Pandarinath, 2021). This grouping structure indicates that GE facilitates the encoder's ability to discern nuanced, common structural patterns throughout the dataset.
C.2 VISUALIZATIONS ON SYNTHETIC DATASETS
We provide additional visualizations of token representations on the synthetic many-body datasets, as shown in Figure 3, Figure 4, and Figure 5.
15

---

Figure Token embeddings at the input (left) and out- put (right) layers for a trans- former encoder trained with learned position embeddings (PE). Different colors repre- sent the x and axis of object- (green and orange), object- 2 (blue and purple), and two random noise objects.
4-
Figure 4: Token embeddings at the input (left) and output (right) layers for transformer encoder trained with group embeddings (GE). Different colors represent the and y axis of object- 1 (green and orange). object-2 (blue and purple), and two ran- dom noise objects.
2
-2
-6
-50
50
-5
Figure 5: Token embeddings at the output for the same frozen transformer hackbone without (left) and with (right) GE. Sur- prisingly, with the same back- bone, applying group embed- dings at the input can produce representations that separable. Different colors repre- sent the x and y axis of object- (green and orange), object-2 (blue and purple), and two ran- dom noise objects.
16


Reviews:
[{"review_id": "K0C3Kgqksc", "reviewer": "Reviewer_Jppq", "summary": "This paper introduces a novel transformer architecture called GAFormer, which models the interations of spatial and temporal groups separately. The proposed model demonstrates effectiveness on both synthetic experiments and real-data.", "questions": "1. Does this model consisted of 2 layers of transformer? SGE and TGT?\n\n2. What's the specification of these transformers? It should be stated in the main paper.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and motivated. \n2. The group embedding is novel and very effective as shown by synthetic experiments.", "weaknesses": "1. The details of transformer used seems to be missing. see questions\n2. Why does this model not compare to (Nie et al.), which seems to be an important related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rpQoKtw8P8", "reviewer": "Reviewer_q3iU", "summary": "The paper tries to improve the transformers to time-series datasets by learning spatiotemporal structure. Specifically, the authors introduce an instance-specific group embedding layer that assigns input tokens to a small number of group tokens to incorporate structure into learning. Based on the group embedding layer, they introduce the GAFormer to incorporate both spatial and temporal group embeddings.", "questions": "1. The authors claim that the group embedding layer can assign input tokens to a small number of group tokens to incorporate structure into learning. I wonder how the structure information can be incorporated using group embedding.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies the multivariate time series which is an interesting and important problem.\n2. The paper provides a detailed literature review and\n3. The paper discusses its limitations.\n4. The paper provides extensive experimental results to demonstrate the effectiveness of the proposed method.", "weaknesses": "1, The proposed methodology is not well motivated and built on the existing Transformer. The paper tries to introduce the group embedding layer but does not provide clear motivation for the group embedding, i.e. how to define the \"group\" and why we need the group embedding.\\\n2. The explanation of the group embedding is vague. It is not easy to understand what the group embedding stands for.\\\n3. The paper aims to learn generalizable representation within multivariate datasets. I did not find the related experiments to demonstrate the generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9S4NdqYGdt", "reviewer": "Reviewer_cXyc", "summary": "The paper proposes a method to learn PEs for transformers in the multivariate temporal data setting. Specifically, the authors propose a Group Embedding (GE) to capture the group aware dynamics along the channels and also the temporal dimension. This Group Embedding is then induced into the transformer architecture called GAFormer to learn the spatial and temporal structure of the multivariate time series datasets. Experimental results on the synthetic dataset show the robustness of the proposed GE over baseline PEs. Moreover, when GE is induced in other transformer architectures for the univariate datasets the performance is enhanced. GAFormer also achieves SoTA performance on the multivariate datasets tested. The authors also claim the interpretability of the proposed method.", "questions": "- The authors don’t seem to have mentioned the computational complexity of the method in the paper. From the description, it seems to inherit the quadratic computational complexity of the transformer which is prohibitive for large timesteps and multi chaneled data\n- The group structure exhibited in Figure 4 is claimed to provide interpretability of the model. However, these are some groups that have been learnt and the interpretability aspect is not clear as to what it means if some channels or time signals are grouped. Further analysis and explanation by the authors may help.\n- As the authors have mentioned other mechanisms (spot attention etc.) to induce group structure, I would expect some empirical study to compare the proposed method with these PEs. Specifically, is the proposed method a unique way to induce PEs for multivariate time series forecasting or could other techniques provide similar or better performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Promising PE for multivariate time-series data. The PE show improved performance over the tasks.\n- The proposed GAFormer also achieves SoTA performance on the multivariate datasets tested.", "weaknesses": "- Since the model is designed to learn the group structure in a data-dependent manner, it may not be able to capture these properties effectively in the low data regime as the baseline PEs would.\n- The group structure exhibited in Figure 4 is claimed to provide interpretability of the model. However, these are some groups that have been learnt and the interpretability aspect is not clear as to what it means if some channels or time signals are grouped. This is similar to the groups formed in any attention mechanism and the method doesn’t seem to provide any added explanations or interpretability to the decisions. Thus it may not be right to claim interpretability based on group embedding structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbjNqlVAZl", "reviewer": "Reviewer_Hkvr", "summary": "This paper introduces a novel technique for improving the application of transformers to timeseries datasets by learning spatiotemporal structures. The proposed framework employs group tokens and an instance-specific group embedding layer to integrate structure into the learning process. The newly devised Group-Aware transFormer (GAFormer) demonstrates superior performance across various timeseries tasks, offering enhanced interpretability and the ability to discern latent structures in intricate multivariate datasets.", "questions": "- How to tokenise the MTS data?\n\n- What's the different between a text tokeniser and a MTS tokeniser?\n\n- How to determine K? \n\n- What's the effect of different K?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using *groupe mbedding*(GE) technique to learn both spatial and temporal structure in multi-variate timeseries datasets is novel.\n- The proposed model achieves the state-of-the-art performance on a number of time- series classification and regression tasks", "weaknesses": "Though the approach claims to offer \"a more interpretable decomposition,\" this is a subjective claim. The degree of interpretability might vary based on the user or the specific use case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FbOoyCKlH", "reviewer": "Reviewer_MEmY", "summary": "In this paper, authors present a novel approach for learning spatiotemporal structure and using it to improve the application of transformers to timeseries datasets. A key aspect of contributions lies in the creation of a specialized group embedding scheme designed specifically for transformer architectures. This scheme enables the adaptive learning of concise grouping tokens, which encompass both channel and temporal dimensions. By incorporating group-aware structural information into the representation space, GAFormer enhances the overall understanding and encoding of data patterns.", "questions": "See Weaknesses for details.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The method proposed by the authors is intuitive and validated on multiple datasets, and the experimental results prove the effectiveness of GAFormer.\n\n2.GAFormer achieves the adaptive discernment of channel-wise and temporal groupings without relying on any predefined structure or additional supervision beyond classification or regression tasks. Through the assignment of group embeddings to tokens, GAFormer enhances the interpretability of the model's representations.", "weaknesses": "1.The GAFormer goes through the SGE and then the TGE, what happens if the order of the two modules is reversed? Why is it modeled from the spatial point of view first?\n\n2.Can the author explain Figure 4 again, I don't really understand the interpretability of the authors' proposal.\n\n3.Table 4 shows that SGE is more effective than TGE, can the authors explain why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)