PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecasting

## Abstract

Abstract Time series forecasting remains a critical challenge across various domains, often complicated by high-dimensional data and long-term dependencies. This paper presents a novel transformer architecture for time series forecasting, incorporating two key innovations: parameter sharing (PS) and Spatial-Temporal Segment Attention (SegAtt). We also define the time series segment as the concatenation of sequence patches from the same positions across different variables. The proposed model, PSformer, reduces the number of training parameters through the parameter sharing mechanism, thereby improving model efficiency and scalability. The introduction of SegAtt could enhance the capability of capturing local spatio-temporal dependencies by computing attention over the segments, and improve global representation by integrating information across segments. The combination of parameter sharing and SegAtt significantly improves the forecasting performance. Extensive experiments on benchmark datasets demonstrate that PSformer outperforms popular baselines and other transformer-based approaches in terms of accuracy and scalability, establishing itself as an accurate and scalable tool for time series forecasting.

## 1 Introduction

Time series forecasting is an important learning task with significant application values in a wide range of domains, including the weather prediction (Ren et al., ; Chen et al., ), traffic flow (Tedjopurnomo et al., ; Khan et al., ), energy consumption (Liu et al., ; Nti et al., ), anomaly detection (Zamanzadeh Darban et al., ) and the financial analysis (Nazareth & Ramana Reddy, ), etc. With the advancement of artificial intelligence techniques, significant efforts have been devoted to developing innovative models that continue to improve the prediction performance (Liang et al., ; Wang et al., ). In particular, the transformer-based model family has recently attracted more attention for its proved success in nature language processing (OpenAI et al., ) and computer vision (Liu et al., ; Dosovitskiy et al., ). Moreover, pre-trained large models based on the transformer architecture have shown advantages in time series forecasting(Liu et al., ; Jin et al., ; Chang et al., ; Woo et al., ), demonstrating that increasing the amount of parameters in transformer models and the volume of training data can effectively enhance the model capability.

On the other side, many simple linear models (Zeng et al., ; Li et al., ) also shown competitive performance compared to the more complex designs of transformer-based models. One possible key reason for their success in time series forecasting is that they have low model complexities and are less likely to overfit on noisy or irrelevant signals. As a result, even with limited data, these models can effectively capture robust information representations. To overcome the limitations of modeling long-term dependencies and capturing complex temporal relationships, PatchTST (Nie et al., ) process temporal information by combining patching techniques to extract local semantic information, leading to superior performance. However, it applies channel-independent designs and leaves the significant potential for improvement in effectively modeling across variables.
Moreover, the unique challenges of modeling multivariate time series data, where the temporal and spatial dimensions differ significantly from other data types, present many unexplored opportunities. While past research (donghao & wang xue, ; Zhang & Yan, ; Ilbert et al., ) has largely treated these dimensions separately, the question of whether systematically mixing temporal and spatial information can further enhance model performance remains an open area for future investigation.

To reduce the model complexity in deep learning, parameter sharing (PS) is a crucial technique that can significantly reduce the amount model parameters and enhance computational efficiency. In CNNs, convolutional filters share weights across spatial locations, capturing local features with fewer parameters. Similarly, LSTM networks share weight matrices across time steps to manage memory and control information flow. By studying the sharing of attention weights, (Xiao et al., ; Kitaev et al., ) improved performance in the fields of natural language processing and computer vision. ALBERT (Lan et al., ) further extends parameter sharing in natural language processing by sharing weights across transformer layers, reducing parameter redundancy while maintaining performance. In multi-task learning, the Task Adaptive Parameter Sharing (TAPS) approach (Wallingford et al., ) selectively fine-tunes task-specific layers while maximizing parameter sharing across tasks, achieving efficient learning with minimal task-specific modifications.
Those studies demonstrate that parameter sharing has the potential for model size reduction, generalization ability enhancement and mitigating the over-fitting risks across various tasks.
Besides, employing advanced optimization technique is also essential in reducing the overfitting issue. (Ilbert et al., ) proposed a simple yet effective transformer-based network structure and equipped it with the SAM (Sharpness-Aware Minimization) optimization mechanism to achieve competitive performance compared with large models (Foret et al., ).

In this work, we explore innovative designs of transformer-based model for time series forecasting by incorporating the characteristics of time series tasks and the concept of parameter sharing. Unlike MOIRAI (Woo et al., ) that “flattens” multivariate time series by treating all variables as a single sequence, or SAMformer (Ilbert et al., ) that applies attention to the channel dimension to capture spatial dependencies, we propose a different and novel approach. We construct a transformer encoder with two-stage segment attention structure, where each network layer consists of a parameter-sharing block. This parameter-sharing block is composed of three fully connected layers with residual connections, which keeps the overall number of parameters in the model very small while enabling effective information sharing across different parts of the model. For segment attention, we use patching to divide the variables into different patches, then identify the patches at the same position across different variables and merge them into a segment. As a result, each segment represents the spatial extension of a single-variable patch. In this way, we decompose the multivariate time series into multiple segments. Attention is applied within each segment to enhance the extraction of local spatial-temporal relationships, while information fusion across segments is performed to improve the overall predictive performance. Additionally, by incorporating the SAM optimization method, we further reduce over-fitting while maintaining efficient training. We conduct extensive experiments on long-term time series forecasting datasets to verify the effectiveness of this segment attention structure with parameter sharing. Our model remains competitive when compared to previous state-of-the-art models, achieving the best performance on 6 out of 8 mainstream long-term time series forecasting tasks. The contributions are summarized as follows:

- •
We developed a novel transformer-based model structure for time series forecasting, where the parameter sharing technique is applied in the transformer block to reduce the model complexity and improve the generalization ability.
- •
We proposed a segment attention mechanism tailored for multi-variate data, which merges the temporal sequence of different channels to form a local segment and applies attention within each segment to capture both temporal dependencies and cross-channel interactions.
- •
Through extensive experiments in long-term forecasting tasks and ablation studies, we verified the effectiveness and superior performance of our proposed framework.

## 2 Related Work

### 2.1 Temporal modeling in time series forecasting

In recent years, time series analysis has received widespread attention, with more deep learning methods being applied to time series forecasting. These deep learning methods focus on establishing temporal dependencies within time series data to predict future trends. The models can be broadly categorized into RNN-based, CNN-based, MLP-based, and Transformer-based approaches. RNNs and their LSTM variants were widely used for time series tasks in the past, with related works such as DeepAR(Salinas et al., ). CNN-based methods like TCN (Bai et al., ) and TimesNet (Wu et al., ) have been designed to adapt convolutional structures specifically for temporal modeling. MLP-based approaches, such as N-BEATS (Oreshkin et al., ), RLinear (Li et al., ), and TSMixer (Chen et al., ), have demonstrated that even simple network structures can achieve solid predictive performance.
Moreover, Transformer-based models have become increasingly popular in time series forecasting due to the unique attention mechanism of Transformers, which provides strong global modeling capabilities. Many recent works leverage this to enhance time series modeling performance, such as Informer (Zhou et al., ), Autoformer (Wu et al., ), Pyraformer (Liu et al., ), and Fedformer (Zhou et al., ). Additionally, PatchTST (Nie et al., ) further divides time series data into different patches to enhance the ability to capture local information. However, the aforementioned models primarily focus on temporal modeling, with less emphasis on modeling the relationships between variables. Although PatchTST attempted to incorporate cross-channel designs, it observed degraded performance in their model.

On the other hand, some pre-trained large models have been applied to time series forecasting tasks (Das et al., ; Liu et al., ; Gao et al., ; Liu et al., ; Zhou et al., ; Jin et al., ). For example, Moment uses the patching method and mask pre-training to build a pre-trained model for time series, while GPT4TS (Zhou et al., ) also adopts the patching method and uses BERT as the backbone. The increase in model parameters has provided them with greater expressive power but also increased the difficulty of training.

### 2.2 Variate modeling in time series forecasting

In addition to modeling temporal dependencies, recent works have focused on modeling inter-variable dependencies (donghao & wang xue, ; Zhang & Yan, ; Ilbert et al., ; Woo et al., ; Liu et al., ). ModernTCN (donghao & wang xue, ) employs different 1-D convolutions to capture the temporal and variable dimensions separately; Crossformer (Zhang & Yan, ) constructs attention mechanisms across both time and spatial dimensions; SAMformer (Ilbert et al., ) applies attention to the variable dimension to model cross-channel dependencies; MORAI (Woo et al., ) ”flattens” multivariate time series into univariate sequences to merge information across variables; iTransformer (Liu et al., ) represents multivariate time series and captures global dependencies. All of these works emphasize the simultaneous modeling of both variable and temporal dependencies as critical directions for improving multivariate time series modeling, which helps establish global spatial-temporal dependencies. However, this may weaken the ability to capture local spatial-temporal dependencies. Additionally, expanding the global receptive field of spatio-temporal dependencies could increase model complexity, which in turn may lead to overfitting due to the larger number of parameters. Our work addresses these issues and proposes solutions to mitigate them.

Figure: Figure 1: Data transformation in PSformer. Multivariate time series data first undergo patching and cross-channel merging before being fed into the PSformer Encoder. Then after inverse transformation and linear mapping for the prediction length, the final time series results are obtained.
Refer to caption: extracted/5971143/pictures/data_flow.jpg

## 3 The PSformer Framework

### 3.1 Problem Formulation

We denote the input multivariate time series $X\in\mathbb{R}^{M\times L}$ as with $M$ variables and look-back window $L:(x_{1},x_{2},...,x_{L})$, where $x_{t}$ represents the $M$-dimensional vector at time step $t$. $L$ will be equally divided into $N$ non-overlapping patches of size $P$. $P^{(i)}$ denote the $i$-th patch with size $P$, where $i=1,2,3,...,N$. The $P^{(i)}$ of the $M$ variables forms the $i$-th segment, which denote cross-channel patch of length $C$, where $C=M\times P$. As shown in Figure 1, we transform X from $X\in\mathbb{R}^{M\times L}$ to $X\in\mathbb{R}^{C\times N}$ before feeding it into the model. We aim to predict the future values of $F$ time steps, e.g., $(x_{L+1},...,x_{L+F})$. Beside, we denote $\bm{X}^{in}$ and $\bm{X}^{out}$ as the input and output signals for the specified layers.

### 3.2 Model Structure

The PSformer model we constructed is depicted in Figure 2. The key components of the model include the PSformer Encoder, segment attention, and PS Block. The PSformer Encoder serves as the backbone of the model and contains both the segment attention and the PS Block. The PS Block provides the parameters for all layers within the Encoder, utilizing the parameter sharing technique.

Figure: Figure 2: PSformer network structure. The PSformer includes the PS Block for parameter sharing within the PS Encoder structure, as well as the two-stage segment attention.
Refer to caption: x1.png

Forward Process
The univariate time series of length $L$ for the $i$-th variable, starting at time index $1$, is denoted as $x_{1}^{(i)}=(x_{1}^{(i)},...,x_{L}^{(i)})$, where $i=1,...,M$. Then the input $(x_{1},...,x_{L})$ with $M$ dimensions is presented as $x_{1}\in\mathbb{R}^{M\times L}$, and $x_{1}$ is used as the input to the transformer network structure. Similar to other time series forecasting methods, we use a RevIN layer Kim et al. (), which is added at both the input and output of the model.

Spatial-Temporal Segment Attention

We introduce spatial-temporal segment attention (SegAtt), which merges patches from different channels at the same position into a segment and establishes spatial-temporal relationships across different segments. Specifically, the input time series $X\in\mathbb{R}^{M\times L}$ is first divided into patches, where $L=P\times N$, and then transformed into $X\in\mathbb{R}^{(M\times P)\times N}$. By merging the $M$ and $P$ dimensions, the input becomes $X\in\mathbb{R}^{C\times N}$, where $C=M\times P$, facilitating the subsequent cross-channel information fusion.

In this transformed space, identical $\bm{Q}\in\mathbb{R}^{C\times N}$, $\bm{K}\in\mathbb{R}^{C\times N}$, and $\bm{V}\in\mathbb{R}^{C\times N}$ matrices are generated by applying a shared block’s non-linear projection (applied by Parameter Shared Block), with weights $\bm{W}^{Q}\in\mathbb{R}^{N\times N}$ and $\bm{W}^{K}\in\mathbb{R}^{N\times N}$ used to project the input $X^{in}\in\mathbb{R}^{C\times N}$. The $\bm{Q}$ and $\bm{K}$ matrices are then multiplied using a dot-product operation to form the attention matrix $\bm{Q}K^{T}\in\mathbb{R}^{C\times C}$, which captures relationships between different spatial-temporal segments (in the $C$ dimension) and is used to act on the $\bm{V}$ matrix. While the computation of $\bm{Q}$, $\bm{K}$ and $\bm{V}$ involves nolinear transformations of the input $X^{in}$ across segments in the $N$ dimension (corresponding to the $d_{m}odel$ dimension of attention in the NLP domain), the scaled dot-product attention primarily applies attention across the $C$ dimension, allowing the model to focus on dependencies between spatial-temporal segments across channels and time.

This mechanism ensures that information from different segments is integrated through the computation of $Q$, $K$, and $V$. It also captures local spatial-temporal dependencies within individual segments by applying attention to the internal structure of each segment. Additionally, it captures long-term dependencies across segments over larger time steps. The final output is $X^{out}\in\mathbb{R}^{C\times N}$, completing the attention process.

Parameter Shared Block
In our work, we propose a novel Parameter Shared Block (PS Block), which consists of three fully connected layers with residual connections, as illustrated in Figure 2. Specifically, we construct three trainable linear mappings $\bm{W}^{(j)}\in\mathbb{R}^{N\times N}$ with $j\in\{1,2,3\}$.
The output of the first two layers is computed as:

$$ $\bm{X}^{out}=\text{GeLU}(\bm{X}^{in}\bm{W}^{(1)})\bm{W}^{(2)}+\bm{X}^{in}, \vspace{0.5ex}$ (1) $$

which follows a similar structure as the feed-forward network (FFN) with residual connections. This intermediate output $\bm{X}^{out}$ is then used as the input for the third transformation, yielding:

$$ $\bm{X}^{out}=\bm{X}^{in}\bm{W}^{(3)}.$ $$

Therefore, the PS Block as a whole can be expressed as:

$$ $\bm{X}^{out}=(\text{GeLU}(\bm{X}^{in}\bm{W}^{(1)})\bm{W}^{(2)}+\bm{X}^{in})\bm {W}^{(3)},\vspace{1ex}$ (2) $$

and we denote PS Block output as $\bm{X}^{out}=\bm{X}^{in}\bm{W}^{S}$, where $W^{S}\in\mathbb{R}^{N\times N}$ and $\bm{X}^{out}\in\mathbb{R}^{C\times N}$. The structure of the PS Block allows it to perform nonlinear transformations while preserving a linear transformation path. Although the three layers within the PS Block have different parameters, the entire PS Block is reused across different positions in the PSformer Encoder, ensuring that the same block parameters $\bm{W}^{S}$ are shared across those positions, as illustrated in Figure 2. Specifically, PS Block share parameters across three parts of each PSformer Encoder, which including two segment attention layer and the final PS Block. In the segment attention layer, the PS block outputs are used as the $Q$, $K$, and $V$ matrices to construct the attention mechanism. This parameter-sharing strategy reduces the overall number of parameters while maintaining the network’s expressive capacity.

PSformer Encoder
In the PSformer Encoder, as Figure 2, each layer shares the same parameters $\bm{W}^{S}$ of PS Block. For the input $\bm{X}^{in}$, the transformation in the PSformer Encoder can be expressed as follows:

SegAtt stage one is represented as: $\bm{Q}^{(1)}=\bm{X}^{in}\bm{W}^{S}$, $\bm{K}^{(1)}=\bm{X}^{in}\bm{W}^{S}$, $\bm{V}^{(1)}=\bm{X}^{in}\bm{W}^{S}$, Therefore, $\bm{Q}^{(1)}$, $\bm{K}^{(1)}$, $\bm{V}^{(1)}$ $\in\mathbb{R}^{C\times N}$.
Dot-product attention operation

$$ $\bm{O}^{(1)}=\text{Attention}^{(1)}(\bm{Q}^{(1)},\bm{K}^{(1)},\bm{V}^{(1)})= \text{Softmax}(\frac{\bm{Q}^{(1)}\bm{K}^{(1)}}{\sqrt{d_{k}}}){V}^{(1)}.$ $$

then with ReLU activation:$\bm{O}^{(1)}_{act}=ReLU(\bm{O}^{(1)})$ .

SegAtt stage two is represented as: $\bm{Q}^{(2)}=\bm{O}^{(1)}_{act}\bm{W}^{S}$, $\bm{K}^{(2)}=\bm{O}^{(1)}_{act}\bm{W}^{S}$, $\bm{V}^{(2)}=\bm{O}^{(1)}_{act}\bm{W}^{S},$
Therefore, $\bm{Q}^{(2)}$, $\bm{K}^{(2)}$, $\bm{V}^{(2)}$ $\in\mathbb{R}^{C\times N}.$
Dot-product attention operation

$$ $\bm{O}^{(2)}=\text{Attention}^{(2)}(\bm{Q}^{(2)},\bm{K}^{(2)},\bm{V}^{(2)})= \text{Softmax}(\frac{\bm{Q}^{(2)}\bm{K}^{(2)}}{\sqrt{d_{k}}})\bm{V}^{(2)}.$ $$

The two-stage SegAtt mechanism can be viewed as analogous to an FFN layer, where the MLP is replaced with attention operations. Additionally, residual connections are introduced between the input and output, and the result is then fed into the final PS Block.

The transformation in the final PS Block is represented as
$\bm{O}^{out}=\bm{O}^{in}\bm{W}^{S}.$
Therefore, the entire encoder can be expressed as

$$ $\bm{X}^{out}=(\text{Attention}^{(2)}(\text{ReLU}(\text{Attention}^{(1)}(\bm{X} ^{in})))+\bm{X}^{in})\bm{W}^{S},$ $$

with $\bm{X}^{out}\in\mathbb{R}^{C\times N}$. Then we apply a dimensionality transformation to obtain $\bm{X}^{out}\in\mathbb{R}^{M\times L}$, since $C=M\times P$ and $L=P\times N$.

After passing through $n$ layers of the PSformer Encoder, the final output is $\bm{X}^{pred}=\bm{X}^{out}\bm{W}^{F}$, where $\bm{X}^{pred}\in\mathbb{R}^{M\times F}$, and $\bm{W}^{F}\in\mathbb{R}^{L\times F}$ is a linear mapping, where $F$ is the prediction length. The $\bm{X}^{pred}$ is the final output of the PSformer model.

## 4 Experiment

Datasets
In this paper, we focus on the long-term time series forecasting. We follow the time series forecasting work in Ilbert et al. () and use 8 mainstream datasets to evaluate the performance of our proposed PSformer model. As shown in Table 1, these datasets include 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2), as well as Weather, Traffic, Electricity, and Exchange. These datasets have been used as benchmark evaluations in many previous time series forecasting studies.

Table: Table 1: Datasets for long-term forecasting. Dataset size is structured as (Train, Validation, Test).

Baselines
We select state-of-the-art (SOTA) models in the field of long-term time series forecasting, including not only Transformer-based models but also large models and other SOTA models. Specifically, baselines include (1) Transformer-based model: SAMformer (Ilbert et al., ), iTransformer (Liu et al., ), PatchTST (Nie et al., ), FEDformer (Zhou et al., ), Autoformer (Wu et al., ), (2) Pretrained Large model: MOMENT (Goswami et al., ), GPT4TS (Zhou et al., ), (3) TCN-based model: ModernTCN (donghao & wang xue, ) and (4) MLP-based methods: TSMixer (Chen et al., ), RLinear (Li et al., ).

Experimental Settings
The input time series length $T$ is set to 512, and four different prediction lengths $H\in\{96,192,336,720\}$ are used. Evaluation metrics include Mean Squared Error (MSE) and Mean Absolute Error (MAE). We train our constructed models using the SAM optimization technique as in
Ilbert et al. (). We collect baseline results of SAMformer, TSMixer, FEDformer and Autoformer from Ilbert et al. (), in which SAMformer and TSMixer are also trained with SAM, and the results of iTransformer, RLinear and PatchTST from Liu et al. (). We test ModernTCN with the fixed input length of 512 following donghao & wang xue (). For the pre-trained large models MOMENT and GPT4TS, the results are collected from (Goswami et al., ). More details about the baseline settings can be found in Section A.1.

Table: Table 2: Long-term forecasting task. All the results are averaged from 4 different prediction lengths {96, 192, 336, 720}. A lower MSE or MAE indicates a better performance. See Table 27 in Appendix for the full results with more baselines.

### 4.1 Results and Analysis

Summary of Experimental Results
The main experimental results are reported in Table 2. PSformer achieved the best performance on 6 out of 8 major datasets in long-term time series forecasting tasks, demonstrating its strong predictive capabilities across various time series prediction problems. This success is attributed to its segment attention mechanism, which enhances the extraction of spatial-temporal information, and the parameter-sharing structure, which improves the model’s robustness. The complete experimental results are in Appendix B.2.

Comparison with Other SOTA models
Compared to other Transformer-based models, PSformer demonstrates higher predictive accuracy, reflecting the effectiveness of dividing multivariate time series data into spatial-temporal segments for attention calculation. The neural network’s ability to extract information from all spatial-temporal segments enhances the prediction performance. In contrast to current large pre-trained models, PSformer not only achieves better accuracy but also reduces the amount of parameters through parameter sharing.
Although linear models are simpler and have fewer parameters with satisfactory performance in some cases, the ability to extract rich information from complex data is limited. In contrast, PSformer integrates the residual connections, thus enabling a linear path while retaining the capability to process complex nonlinear information.
Moreover, the ConvFFN component in ModernTCN tailored for temporal data also confirms that the convolutional mechanism, which actually embodies the idea of parameter sharing, is also effective in the time series domain. With the same spirit, we have successfully applied the parameter sharing to the transformer-based models in the time series field and achieved superior performance.

Comparison with PatchTST and SAMformer
PatchTST employs a channel-independent design and divides time series data into multiple patches, which demonstrates the effectiveness of the patching method in time series processing. However, its channel-independent approach does not fully consider the relationships between different channels, focusing only on processing each channel individually. On the other hand, SAMformer applies attention directly to the channel dimension via dimension transformations and utilizes a simplified model structure, achieving good predictive performance. However, it may fail to capture valuable local information without the patching method. PSformer combines the advantages of both models while addressing their limitations. By using spatial-temporal segment attention mechanism, PSformer effectively captures local temporal information and handles relationships among different channels. This design enables PSformer to outperform them in various time series forecasting tasks as validated by extensive experiments.

### 4.2 Ablation studies

The Effect of segments Numbers
Since PSformer employs a non-overlapping patching to construct segments, the model’s performance is affected by the number of segments. Therefore, we tested the model’s performance with different segment numbers on two datasets, ETTh1 and ETTm1. Given that the input sequence length is fixed at 512, the number of segments must be a divisor of the sequence length. Consequently, we set the number of segments to 8, 16, 32, and 64, and test the model on four different forecasting horizons. The test results are shown in Table 3, which indicate that the number of segments impacts the model’s prediction accuracy to some extent. Across both datasets, a moderate number of segments (such as 32) tends to perform the best, particularly for both short and long forecasting horizons. For shorter horizons (96, 192), using 16 or 32 segments generally yields the lowest MSE, while fewer segments (8) often result in poorer performance. As the horizon increases (336, 720), 32 segments consistently lead to optimal results, indicating that this number balances the extraction of both local and global temporal features effectively.

Table: Table 3: The MSE results of different number of segments ($N$) on the ETTh1 and ETTm1 dataset.

The Effect of PSformer Encoder Numbers
Since PSformer adopts the segment attention mechanism, with non-shared PS Block parameters across different Encoders (shared within Encoder), we tested the impact of varying the number of encoders on model performance. We conducted tests on the ETTh1 and ETTm1 datasets, varying the number of encoders from 1 to 4 with four different forecasting horizons. The experimental results are shown in Table 4. The results indicate that ETTm1 performs best with 3 encoders, while ETTh1 achieves better performance with just 1 encoder. This may suggest that for smaller datasets, fewer encoders result in better performance, as reducing the number of encoders decreases the amount of model parameters, thereby mitigating the risk of over-fitting.

Table: Table 4: The MSE results of different number of encoders on the ETTh1 and ETTm1 dataset.

Ablation of Parameter Sharing methods
We investigate the impact of parameter-sharing mechanism on the model performance. In addition to the default parameter-sharing approach, which shares parameters only within encoder (In-Encoder), we also test the following approaches: a. no parameter sharing, i.e., None; b. sharing parameters only across encoders (with different parameters used for the PS blocks within each encoder), i.e., Cross-Encoders; and c. sharing parameters both within and across encoders, i.e., ALL. We conducted experiments on the ETTm1 and Weather datasets, and the results are shown in Table 5. As can be seen, the In-Encoder method performs the best, followed by ALL, while None shows the worst performance. This indicates that the parameter-sharing mechanism contributes to improving the model performance.

**Table 5: Parameter Sharing with different methods**
| Sharing Methods | Cross-Encoders | In-Encoder | ALL | None |  |
| --- | --- | --- | --- | --- | --- |
| ETTm1 | 96 | 0.297 | 0.282 | 0.288 | 0.295 |
| 196 | 0.329 | 0.321 | 0.326 | 0.338 |  |
| 336 | 0.360 | 0.352 | 0.358 | 0.372 |  |
| 720 | 0.420 | 0.413 | 0.414 | 0.425 |  |
| Weather | 96 | 0.158 | 0.149 | 0.151 | 0.154 |
| 196 | 0.201 | 0.193 | 0.196 | 0.198 |  |
| 336 | 0.252 | 0.245 | 0.245 | 0.245 |  |
| 720 | 0.319 | 0.314 | 0.316 | 0.319 |  |

Ablation of Segment Attention methods
As an important component of PSformer, we designed segment attention to effectively capture temporal information while fully utilizing the correlations between channels. In the ablation study, we compared different attention mechanisms without using SegAtt under parameter sharing, as well as the performance of using only MLP. We conducted experiments on the ETTh1 and ETTh2 datasets.
In this study, w/o SegAtt means applying the attention mechanism to the input $x\in\mathbb{R}^{B\times M\times L}$ on the $M\times L$ dimensions. For the setting with channel independence’ applies the attention mechanism on the $N\times P$ dimensions. The results in Table 6 show the performance of PSformer variants with different attention mechanisms on the ETTh1 and ETTh2 datasets across various forecasting horizons:

- •
The default PSformer configuration (with SegAtt) consistently achieves the lowest MSE across all horizons, demonstrating the effectiveness of segment attention.
- •
When SegAtt is removed, the cross-channel attention is less effective at capturing both local and global temporal dependencies, resulting in slightly increased MSE .
- •
The variant with channel-independent attention shows further degradation, suggesting that neglecting inter-channel correlations impacts the model’s ability to capture temporal features.
- •
Using only MLP layers also results in higher MSE. Although it performs second best for the three forecasting horizons on ETTh2, it still falls short compared to the performance with SegAtt, highlighting the necessity of applying SegAtt.

Table: Table 6: Ablation analysis of SegAtt on ETTh1 and ETTh2 (MSE reported). Red: the best.

## 5 Conclusion and future work

In this work, we proposed the PSformer model for multivariate time series forecasting, which leverages segment attention technique to facilitate information transfer among time series variables and capture spatial-temporal dependencies. By employing parameter sharing, the model effectively improves parameter efficiency and reduces the risk of overfitting when the data size is relatively limited. This network structure, which combines parameter sharing with segment attention mechanism, achieves state-of-the-art performance on long-term multivariate forecasting tasks by enhancing model parameter efficiency and improving the utilization of channel-wise information. Future work can focus on applying this parameter sharing and segment attention technique to the development of pre-trained large models for time series forecasting, to overcome the issue of excessively large parameter counts in existing pre-trained models, and improve the capability of extracting information from multivariate time series. Additionally, exploring model architecture designs with parameter sharing to improve prediction performance still holds potential value.

## Acknowledgments

We would like to thank Haixu Wu for his valuable feedback and suggestions for improving our manuscript prior to submission. His contributions were greatly appreciated and helped shape the final version of our work.

## Appendix A Experimental Configuration

### A.1 Details of Baseline Settings

We conducted all of our experiments with look-back window $L=512$ and prediction horizons $H\in\{96,192,336,720\}$. Results of PSformer and ModernTCN reported in Table 11 come from our own experiments. The difference between our experiment with ModernTCN and its official code is that we standardized the look-back window to 512 and set drop_last=False for the test set in the dataloader to ensure consistency with our experimental settings for a fair comparison. For MOMENT and GPT4TS, the results are collected from (Goswami et al., ), except Exchange rate (which is tested on official code repository). The results of SAMformer, TSMixer, FEDformer and Autoformer are obtained from (Ilbert et al., ), while the results of iTransformer, PatchTST, and RLinear are taken from (Liu et al., ).

### A.2 Settings for PSformer

We provides an overview of the experimental configurations for the PSFormer model across various tasks and datasets in Table 7. The experiments cover multiple time-series datasets, including ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, Traffic, and Exchange. In all experiments, the input sequence length (Input Length $T$) is set to 512, and each input is equally divided into 32 non-overlapping segments (Segments Num. $N$). The model architecture uses 3 Encoders for tasks, while for the ETTh1, ETTh2, ETTm2 and Exchange, 1 Encoder are used.

The learning rate adjustment strategy (lradj) is set to ”constant” for all experiments, with a fixed learning rate (LR) of $10^{-4}$. The loss function used in all experiments is Mean Squared Error (MSE). The batch size is set to 16 for most tasks, except for the Traffic dataset, where the batch size is 8. Each experiment runs for 300 epochs, with a patience value of 30 for early stopping. A fixed random seed of 1 is applied across all experiments to ensure reproducibility.

Table: Table 7: An overview of the experimental configurations for PSformer.

### A.3 Model Size Comparison

Table 8 presents a comparison of the parameter size across different models, including the PSFormer and other baseline models such as SAMformer, TSMixer, ModernTCN, and RLinear. The comparison is conducted on ETTh1, Weather, and Traffic datasets, with prediction horizons $H\in\{96,192,336,720\}$. PSFormer is evaluated in two configurations: the full model and the encoder part. The parameters of the encoder part refer to the number of parameters after excluding the linear mapping in the output layer.
The table denote that both PSFormer (full) and SAMformer have parameter sizes that are close to RLinear, where RLinear’s parameters mainly stem from the linear mapping between input and output. Notably, the parameter sizes of these three models are relatively unaffected by the number of input channels. In contrast, TSMixer and ModernTCN exhibit significantly larger parameter sizes, with the number of input channels playing a major role in the overall parameter burden. The relative size comparison shows that TSMixer and ModernTCN have several times, or even thousands of times, more parameters than PSFormer(full). Finally, the parameter size of PSFormer(Encoder) is much smaller, indicating that optimizing the linear mapping layer in the output could further reduce the overall parameter count.

**Table 8: Comparison of the trainable model parameters**
| Dataset | Horizon | PSformer | SAMformer | TSMixer | ModernTCN | RLinear |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Full | Encoder |  |  |  |  |  |  |
| ETTh1 | H=96 | 52,416 | 3,168 | 50,272 | 124,142 | 876,064 | 49248 |
| H=192 | 101,664 | 3,168 | 99,520 | 173,390 | 1,662,592 | 98496 |  |
| H=336 | 175,536 | 3,168 | 173,392 | 247,262 | 2,842,384 | 172368 |  |
| H=720 | 372,528 | 3,168 | 369,904 | 444,254 | 5,988,496 | 369360 |  |
| Relative Size (Avg) | 1.0 | 0.014 | 0.987 | 1.408 | 16.192 | 0.948 |  |
| Weather | H=96 | 58,752 | 9,504 | 50,272 | 121,908 | 2,709,280 | 49248 |
| H=192 | 108,000 | 9,504 | 99,520 | 171,156 | 3,495,808 | 98496 |  |
| H=336 | 181,872 | 9,504 | 173,392 | 245,028 | 4,675,600 | 172368 |  |
| H=720 | 378,864 | 9,504 | 369,904 | 442,020 | 7,821,712 | 369360 |  |
| Relative Size (Avg) | 1.0 | 0.039 | 0.953 | 1.347 | 25.708 | 0.948 |  |
| Traffic | H=96 | 58,752 | 9,504 | 50,272 | 793,424 | 822,018,208 | 49248 |
| H=192 | 108,000 | 9,504 | 99,520 | 842,672 | 822,804,736 | 98496 |  |
| H=336 | 181,872 | 9,504 | 173,392 | 916,544 | 823,984,528 | 172368 |  |
| H=720 | 378,864 | 9,504 | 369,904 | 1,113,536 | 827,130,640 | 369360 |  |
| Relative Size (Avg) | 1.0 | 0.039 | 0.953 | 5.040 | 4530.574 | 0.948 |  |

### A.4 Running time Comparison

The average running time per iteration (s/iter) of different models on the ETTh1 and Weather datasets with varying prediction horizons is shown in Table 9. PSformer demonstrates relatively stable running times across different horizons. For the ETTh1 dataset, the running time remains between 0.011 and 0.012 seconds, while for the Weather dataset, it varies slightly between 0.026 and 0.027 seconds. PSformer also shows comparatively efficient running times across the datasets, with performance that remains competitive even as the prediction horizon increases. This indicates that PSformer manages computational costs effectively, especially when compared to more complex models.

**Table 9: Comparison of the running time (s/iter). We test the average running time per iteration of different models across the first five epochs on the ETTh1 and Weather datasets.**
| Dataset | Horizon | PSformer | PatchTST | ModernTCN | TSMixer | RLinear | iTransformer |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ETTh1 | H=96 | 0.012 | 0.049 | 0.241 | 0.013 | 0.032 | 0.020 |
| H=192 | 0.011 | 0.049 | 0.244 | 0.016 | 0.033 | 0.023 |  |
| H=336 | 0.012 | 0.054 | 0.245 | 0.015 | 0.034 | 0.022 |  |
| H=720 | 0.012 | 0.063 | 0.279 | 0.019 | 0.037 | 0.025 |  |
| Weather | H=96 | 0.026 | 0.165 | 0.388 | 0.013 | 0.030 | 0.022 |
| H=192 | 0.026 | 0.166 | 0.361 | 0.016 | 0.032 | 0.022 |  |
| H=336 | 0.027 | 0.176 | 0.730 | 0.016 | 0.033 | 0.027 |  |
| H=720 | 0.027 | 0.185 | 0.788 | 0.024 | 0.034 | 0.035 |  |

### A.5 Discuss about Positional Encoding

The reasons of eliminating positional encoding Similarly, SAMformer does not use positional encoding because it applies attention to the channel dimension, where there is no strict sequential relationship between channels. Although SegAtt is also a cross-channel structure, it involves local time series constructed by patches. We consider such local sequences as local representations (or tokens) rather than short sequences. The experimental results also demonstrate that this structure can similarly reduce the dependency on positional encoding while achieving good performance.

### A.6 Sharpness-Aware Minimization (SAM)

Optimization steps
SAM optimizer (Foret et al., ) modifies the typical gradient descent update to seek a flatten optimum. Below is the mathematical formulation:

Let $\theta$ be the model parameters, $\mathcal{L}(\theta)$ be the loss function, and $\epsilon$ be a small perturbation applied to the parameters. The SAM optimization process can be described in two steps:

- •
Find the adversarial perturbation that maximizes the loss in a neighborhood of the current weights $\theta$:
$\hat{\epsilon}=\arg\max_{\|\epsilon\|\leq\rho}\mathcal{L}(\theta+\epsilon)$
where $\rho$ is a small constant that controls the size of the neighborhood.
- •
Update the parameters in the direction that minimizes the loss with respect to the perturbed parameters:
$\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}(\theta+\hat{\epsilon})$
where $\eta$ is the learning rate.

As used in SAMformer, we also employ this optimization technique to train our models, which can generate promising results.

## Appendix B More Results and Analysis

### B.1 Investigation of hyper-parameter ρ 𝜌 \rho italic_ρ

The Effect of $\rho$ Since we employed SAM to ensure training stability, we also conducted sensitivity tests on the hyper-parameter $\rho$ in SAM. We divided the range of $\rho$ from 0 to 1 into 10 equal parts and tested its effect on model prediction performance across the ETTh1, ETTm1, ETTm2, and Weather datasets. The results are shown in Figure 3. It can be observed that as the parameter $\rho$ gradually increases in SAM can improve the model’s prediction performance to some extent. However, if $\rho$ is set too large, it may degrade the model’s performance. When selecting $\rho$, it’s important to consider the dataset’s complexity and noise levels, as well as the model’s architecture. For more complex datasets or larger models, a slightly larger $\rho$ can help smooth the loss landscape and improve generalization. Further, $\rho$ should also be balanced with the learning rate to avoid instability or performance degradation. As a comparison, we also report the $\rho^{*}$ used by PSformer, SAMformer and TSMixer in the Table 10.

Figure: Figure 3: Ablation analysis on hyper-parameter $\rho$. When taking $\rho$ values from 0 to 1 in steps of 0.1, the prediction loss will slightly decrease first and then increase significantly if the $\rho$ exceeds a threshold, which means the selection of $\rho$ should be careful.
Refer to caption: extracted/5971143/pictures/ablation_rho.jpg

Table: Table 10: Neighborhood size $\rho^{*}$ used by PSformer, SAMformer and TSMixer for SAM optimization to achieve their best performance on the benchmarks.

### B.2 Full Results

Table 11 presents the detailed experimental results of different models and forecasting horizons, providing a comprehensive evaluation of their performance in long-term time series forecasting tasks. The performance is measured using Mean Squared Error (MSE) and Mean Absolute Error (MAE). In the table, red values represent the best performance in the respective task and metric, while blue-lined values indicate the second-best performance.

The last row of the table summarizes the number of first-place results for each model across all tasks. As can be seen, PSformer achieved the best MSE performance in 20 out of 32 prediction tasks, and ranked second in 8 tasks. In terms of MAE, PSformer achieved the best results in 22 tasks and came second in 5 tasks. This clearly demonstrates the superior performance of PSformer compared to other baseline models in long-term time series forecasting tasks.

The next best-performing model is ModernTCN, which achieved the best MSE results in 6 tasks and the best MAE results in 3 tasks. While other models such as SAMformer and PatchTST also showed competitive performance in some tasks, their overall results are not as strong as those of PSformer and ModernTCN. In summary, PSformer’s strong performance across multiple benchmark tasks suggests its potential effectiveness in long-term forecasting.

Table: Table 11: Full long-term forecasting results. We set the forecasting horizons $H\in\{96,192,336,720\}$. A lower value indicates better performance. Avg means the average results from all prediction lengths. Red: the best, Blue lined: the second best.

### B.3 Training loss

Figure 4 illustrates the training and validation loss curves of the ETTh1 and ETTm1 datasets with prediction horizons $H=\{96,192\}$. In this experiment, we set the number of epochs to 200 and disabled early stopping to observe the complete training process. As shown in the plots, despite the model reaching convergence early in the training (as indicated by the flattening of the training loss curve), the validation loss remains consistently low throughout the training duration. This indicates the model’s stability and its ability to generalize well to unseen data over extended epochs.

Figure: (a) ETTh1-96
Refer to caption: extracted/5971143/pictures/loss_etth1_96.jpg

### B.4 Visualization

To better understand our method, we present the forecast curves and the corresponding attention maps for several selected samples. The attention maps visualize the attention weights in different stages of the model, providing insight into how the model processes the input data at each stage.

For the ETTh1 in Figure 5, the input sequence length is 512, and we display the last 100 time steps of the input. The prediction length is set to 96. We select five samples from the ETTh1 dataset, and for each sample, we visualize the attention maps for stage 1 and stage 2 of the SegAtt. From the attention maps, it is evident that there are significant variations in the attention distributions across different samples. Additionally, the attention maps from stage 1 and stage 2 also show noticeable differences, despite both stages sharing the same PS block parameters. This indicates that while the two-stage share parameters, they are able to handle and process the information differently, capturing different aspects of the input data at each stage of the model.

For the Weather in Figure 6, the input length is also 512, and the last 100 time steps are displayed, while the prediction length is 192. Since the model for this dataset employs a three-layer Encoder structure, we display the attention maps for both stages across each layer. Specifically, the notation“1-2” represents the attention map for layer 1, stage 2, and similarly for the other layers. The first two rows of attention maps correspond to the attention distributions from the three Encoder layers. Following that, the prediction curves for nine selected variates are plotted, providing a detailed view of the model’s forecast performance across different variates.

Figure: (a) SegAtt map: stage 1
Refer to caption: x2.png

Figure: (a) SegAtt map 1-1
Refer to caption: extracted/5971143/pictures/pic_hotmap_weather/pic_2_hotmap_att_1_1.jpg

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: See the Weaknesses.
soundness: 2
strengths: 1. The design ideas and motivations behind the model are very interesting.

2. The way the article is written is very good, making it very easy to follow.
summary: The article introduces a novel Transformer architecture, PSformer, for time series forecasting, incorporating parameter sharing (PS) and Spatial-Temporal Segment Attention (SegAtt). The authors conducted experiments on various benchmark datasets, comparing PSformer with baseline methods.
weaknesses: 1. The article has some minor errors, for example, I couldn't seem to find Table 27.

2. The article lacks research on important references, such as the author's focus on iTransformer at ICLR 2024, but does not consider contemporaneous models like TimeMixer[1], and FITS[2]. Furthermore, this still lacks a comparison with GNN-based methods like CrossGNN[3] and FourierGNN[4] from NeurIPS 2023 and other methods such as MICN[5] at ICLR 2023.

[1] Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., ... & ZHOU, J. (2024). TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[2] Xu, Z., Zeng, A., & Xu, Q. (2024). FITS: Modeling Time Series with $10 k $ Parameters. In The Twelfth International Conference on Learning Representations.

[3] Huang, Q., Shen, L., Zhang, R., Ding, S., Wang, B., Zhou, Z., & Wang, Y. CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. In Thirty-seventh Conference on Neural Information Processing Systems.

[4] Yi, K., Zhang, Q., Fan, W., He, H., Hu, L., Wang, P., ... & Niu, Z. FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective. In Thirty-seventh Conference on Neural Information Processing Systems.

[5] Wang, H., Peng, J., Huang, F., Wang, J., Chen, J., & Xiao, Y. (2023). Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations.

3. The experimental comparisons are insufficient. The methods mentioned in W2 and TimesNet[6] were also not compared by the authors, therefore, it cannot be concluded that PSformer achieves SOTA performance.

[6] Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., & Long, M. (2023). TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In The Eleventh International Conference on Learning Representations.

4. The author seems to have not provided details about the experimental platform. Additionally, it appears that the author did not specify the parameter search space for the compared methods. Has the author completed the work of determining the best model parameters on the validation set? If this work has not been done, it would not be fair to compare the performance of all models, as the specific choices of the experimental platform and model parameters can have a significant impact on the experimental conclusions. It is hoped that the author can clarify this point.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: See weakness
soundness: 3
strengths: 1. The method proposed in the paper has clear motivation and solid intuition.
2. The idea is presented with clarity and easy to follow.
3. Sufficient analysis is done to highlight the efficacy of the proposed updates on existing Transformer architectures.
summary: This paper presents a Transformer architecture for time series forecasting that highlights parameter sharing and cross-channel patching. Specifically, it applies attention across both channels and patches for spatio-temporal information fusion, and aligns the parameters of linear projections within an encoder block. Experiments show that these two designs contributes to the overall performance improvement on popular benchmarks.
weaknesses: 1. The accuracy improvement is relatively marginal, especially in ablation studies, and variances in metrics should be reported to support the significancy of the contribution from the proposed ideas.
2. An analysis about why PSformer does not work well on exchange would be helpful.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In the weakness part.
soundness: 3
strengths: a) This paper presents an innovative SegAtt mechanism and a parameter-sharing
approach, effectively advancing methods in time series forecasting and aligning well with
the field’s objectives.

b) The experimental evaluation is notably thorough, offering readers a well-rounded
perspective on the framework’s performance and the contributions of its various
components.

c) The writing is clear and of high quality, making the paper accessible and easy to
understand.
summary: This article presents a framework for multivariate time series forecasting, integrating
parameter sharing and a Spatial-Temporal Segment Attention mechanism. Extensive
experiments demonstrate that this approach consistently achieves higher accuracy and
efficiency compared to other state-of-the-art models
weaknesses: a) While SegAtt demonstrates notable strengths, it would be advantageous to evaluate the
selection of segmentation numbers across a more diverse range of datasets beyond
ETTh1 and ETTm1. Providing further guidance on practical approaches for selecting
segmentation numbers would also be valuable.

b) The experimental section could be enhanced by further validating the framework’s
parameter-saving capacity and examining its implications for pre-trained models.

c) To strengthen the robustness of the findings, incorporating results with multiple
random seeds would provide additional confirmation of the framework’s superior
performance.

d) A deeper exploration of channel-mixing techniques could enrich the analysis. A
comparative discussion, similar to PatchTST A.7, on the benefits of channel
independence versus channel-mixing strategies would be particularly insightful.

e) Regarding computational efficiency, could you clarify which specific components
within the framework contribute to its computational advantages over other state-of-the-
art models?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) Referring to **Weaknesses (2)**, could you provide a more in-depth analysis or empirical studies to illustrate the effectiveness of parameter sharing in time series forecasting?

(2) According to the authors, attention is applied within each segment to enhance the extraction of local spatio-temporal relationships. However, in PSformer, a token represents a down-sampled sequence along a channel. This might be confusing because it suggests that the attention is applied to capture the global dependencies across channels. Could you provide a more detailed explanation of how the segmentation process preserves local temporal information while allowing for cross-channel interactions.
soundness: 2
strengths: (1) The introduction of parameter sharing techniques in transformer-based models demonstrates its effectiveness and potential in the field as validated by experimental results.

(2) Experimental results demonstrate that PSformer achieves state-of-the-art performance in the most of the datasets.
summary: This paper introduces PSformer, a novel transformer-based architecture for time series forecasting that incorporates parameter sharing techniques and a Spatial-Temporal Segment Attention mechanism to capture local spatio-temporal dependencies.
weaknesses: (1) There are several writing errors present in the article.

(2) Although the parameter sharing techniques demonstrated effectiveness according to the ablation study, the authors did not provide detailed analysis or empirical studies to further elucidate this technique, such as comparisons of convergence rates with and without parameter sharing or analysis of how parameter sharing affects model capacity.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Are there any specific settings where PSFormer performs particularly well? For example, do they work better on datasets with strong correlations between variables? When looking at the benchmark datasets, it seems that the Electricity and Exchange datasets exhibited weaker performance compared to others. I am curious if there is any reason for this. 
2. Could the authors provide an ablation study comparing PSFormer’s performance with and without SAM, as well as with other optimization methods such as Adam or SGD? This would clarify the specific impact of SAM on performance and provide a basis for its inclusion in the PSFormer architecture.
3. The authors briefly explained the influence of positional encoding; however, have you conducted more specific experiments on the effects of this positional encoding on the model? I am curious about additional analysis results for various time series data. Could the authors perform experiments comparing the model’s performance with and without positional encoding on different time series types (e.g., seasonal vs. non-seasonal, stationary vs. non-stationary)? Such an analysis could clarify the benefits of positional encoding in diverse time series forecasting contexts.
4. In the ablation study, I observed the analysis results regarding the influence of the number of encoder layers and the number of segments. It seems that tuning these hyperparameters has a significant impact on the model’s performance. Could the authors detail their hyperparameter tuning process, or alternatively, discuss recommended tuning approaches (e.g., grid search, random search, Bayesian optimization) tailored to PSFormer? 
5.  Could the authors provide a detailed analysis of how parameter sharing in the PS Block affects the model’s temporal pattern capture? For instance, they could compare learned representations across layers with and without parameter sharing, illustrating its influence on time series modeling.
soundness: 3
strengths: 1. PSFormer reduces the number of parameters by parameter sharing, which allows to maintain both size and representational ability of the model. This design enhances the model’s scalability and helps mitigate overfitting in data-scarce scenarios.
2. PSFormer demonstrates shorter running time and smaller model size, which empirically shows efficacy of parameter sharing. 
3. The SegAtt mechanism effectively models spatio-temporal dependencies in multivariate time series by incorporating inter-variable information, boosting prediction accuracy.
summary: PSFormer suggests a new transformer-based model for multivariate time-series forecasting. This model utilizes parameter sharing(PS Block) and spatial-temporal segment attention(SegAtt) to decrease computational complexity while effectively modeling time and feature-wise dependency in data. SegAtt is designed to group patches located at the same positions across different variables, enabling efficient capture of local spatio-temporal dependencies. PSBlock allows the model to maintain linear path(residual connection) but also perform nonlinear transformation. As this block is shared, the model can reduce its parameter when it comes to the whole model architecture. With these advantages, PSFormer achieves strong parameter efficiency and enhances predictive accuracy across various benchmark datasets, showing greater scalability and forecasting performance than conventional Transformer models.
weaknesses: 1. The model’s performance varies with different hyperparameters, making hyperparameter tuning appear crucial. 
2. While SegAtt improves performance, it may underperform in cases of univariate time series or where there is little dependency between variables.
3. The paper briefly mentions the necessity of positional encoding, but additional experiments are needed to assess its impact on the generalization of sequences that require a strong temporal order.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See W1-W6.
soundness: 2
strengths: S1: PSformer experiments involve a wide range of datasets.

S2: PSformer and baselines are compared in terms of training parameters and running time.
summary: This paper proposes a Transformer-based model for time series forecasting (PSformer), which primarily consists of two core components: the Parameter Sharing Technique and the Spatial-Temporal Segment Attention mechanism. The former is designed to reduce the complexity of the model, while the latter is used to simultaneously model temporal dynamics and channel correlations.
weaknesses: W1: The paper overall lacks innovation. Parameter Sharing is merely a simple module parameter-sharing technique to reduce complexity. Additionally, Spatial-Temporal Segment Attention simply merges the channel dimension and patch size dimension together before modeling it with a Transformer.

W2: What is the purpose of including the PS Block in the Segment Attention? Why not just add the PS Block after the output of the Attention, which would eliminate the need for parameter sharing? Additionally, why can two SegAttn be viewed as one FFN layer?

W3: The integration of temporal and channel information in Spatial-Temporal Segment Attention may lead to incomplete capture of both time and spatial information, potentially resulting in negative effects. Could you further explain how this approach differs from the separated modeling methods like Crossformer and iTransformer, and visualize the advantages of PSformer in modeling time and channel?

W4: Why is there such a significant difference between the results of PatchTST in Table 1 and the original PatchTST? Please explain.

W5: Please add new baselines for comparison, such as PDF (ICLR2024) and Time-LLM (ICLR2024).

W6: There are many writing errors in the paper, such as "Moment()" on line 125 and "GPT4TS uses BERT as the backbone" on line 127.

Meta-review: This work proposes a new Transformer architecture consisting of i) pre-processing of time series, ii) spatiotemporal segment attention, iii) parameter sharing block, and iv) post-processing. The proposed architecture involves many recent enhancements for time series, i.e., RevIN, patching, and various attention mechanisms. Overall, the proposed method is a mixture of existing discoveries and the reviewers are also around the border level.

Decision: Reject