Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift

## Abstract

Abstract Time series forecasting, which aims to predict future values based on historical data, has garnered significant attention due to its broad range of applications.
However, real-world time series often exhibit complex non-uniform distribution with varying patterns across segments, such as season, operating condition, or semantic meaning, making accurate forecasting challenging.
Existing approaches, which typically train a single model to capture all these diverse patterns, often struggle with the pattern drifts between patches and may lead to poor generalization.
To address these challenges, we propose TFPS , a novel architecture that leverages pattern-specific experts for more accurate and adaptable time series forecasting. TFPS employs a dual-domain encoder to capture both time-domain and frequency-domain features, enabling a more comprehensive understanding of temporal dynamics. It then uses subspace clustering to dynamically identify distinct patterns across data patches. Finally, pattern-specific experts model these unique patterns, delivering tailored predictions for each patch. By explicitly learning and adapting to evolving patterns, TFPS achieves significantly improved forecasting accuracy. Extensive experiments on real-world datasets demonstrate that TFPS outperforms state-of-the-art methods, particularly in long-term forecasting, through its dynamic and pattern-aware learning approach. The data and codes are available: https://github.com/syrGitHub/TFPS .

## 1 Introduction

Time series forecasting plays a critical role in various domains, such as finance (Huang et al., ), weather (Bi et al., ; Wu et al., ; Lam et al., ), traffic (Long et al., ; Kong et al., ), and others (Wang et al., ; Liu et al., ), by modeling the relationship between historical data and future outcomes. However, the inherent complexity of time series data, including temporal dependencies and non-stationarity, poses significant challenges in achieving reliable forecasting results.

Recent works have shown the effectiveness of Transformer-based models for time series forecasting, due to their ability to capture long-range dependencies (Zhou et al., ; Wu et al., ; Liu et al., ). In particular, models like PatchTST (Nie et al., ) operate by splitting the continuous time series into discrete patches and processing them with Transformer blocks.
However, closely examining these patches shows that they often exhibit distribution shifts, which can be attributed to various factors such as concept drift (Lu et al., ). These shifts can manifest as sudden or gradual changes in the underlying patterns and distributions of the time series data. For example, patches corresponding to different regimes, seasons, or operating modes may have distinct statistical properties.
Despite the remarkable success of time series forecasting models (Nie et al., ; Zeng et al., ; Eldele et al., ), they adopt the Uniform Distribution Modeling (UDM) strategy, which fails to account for the drifts and discrepancies between patches. This problem may result in poor generalization, hindering the performance of time series forecasting.

To quantify these distributional shifts, we split the ETTh1 dataset into patches and examined sudden drift and gradual drift in the time and frequency domains as shown in Figure 1. Specifically, we compute the maximum mean discrepancy (MMD) between patches and display the results as a heatmap. Notably, sudden drift (Figure 1 (a)) leads to patches with distinctly different distributions, while gradual drift (Figure 1 (b)) results in more pronounced drift in the time domain, making forecasting more challenging. Furthermore, the information in the frequency domain offers a complementary perspective on concept shifts. These observations highlight that time series data often exhibits a complex structure that evolves over time, with different segments having varying densities and underlying patterns (Sanakoyeu et al., ).

To address the challenges posed by distribution shifts in time series data, we propose a novel Time-Frequency Pattern-Specific (TFPS) architecture to effectively model the complex temporal patterns.
In particular, TFPS consist of the following three key components.
First, TFPS employs a Dual-Domain Encoder (DDE) to capture temporal dependencies in the data. DDE extracts features from both time and frequency domains to provide a comprehensive representation of the time series data, enabling the model to capture both short-term and long-term dependencies.
Second, TFPS addresses the issue of concept drift by incorporating a Pattern Identifier (PI), that utilizes a subspace clustering approach to dynamically identify the distinct patterns present in different patches. Therefore, it can effectively handle nonlinear cluster boundaries and accurately assign patches to their corresponding clusters.
Finally, TFPS constructs a Mixture of Pattern Experts (MoPE), a set of specialized expert models, each tailored to a specific pattern identified by the PI. By dynamically assigning patches to the appropriate expert based on their identified patterns, MoPE enables the model to effectively handle patch sequences across varying patterns and densities. This pattern-specific modeling approach allows our TFPS model to capture the unique characteristics and dynamics of each pattern, leading to improved forecasting accuracy.

In summary, the key contributions of this work are:

- •
We introduce a novel pattern-specific modeling strategy that decomposes the complex, evolving time series into multiple segments. Each segment is modeled by specialized experts, in contrast to the conventional UDM approach.
- •
We propose the TFPS framework, which explicitly addresses concept drift in time series forecasting by leveraging both time-domain and frequency-domain features. Our approach employs a clustering mechanism to dynamically assign patches to specific experts, allowing the model to better handle distributional shifts.
- •
We evaluate our approach on eight real-world multivariate time series datasets, demonstrating its effectiveness. Our model achieves top-1 performance in 50 out of 64 settings, showcasing its competitive edge in improving forecasting accuracy.

Figure: (a) Sudden drift: A new concept occurs within a short time.
Refer to caption: extracted/5922744/figure/fig-1-a.png

## 2 Related Work

Time Series Forecasting Models.
In recent years, deep models with elaborately designed architectures have achieved great progress in time series forecasting (Liu & Wang, ; Qiu et al., ). Approaches like TimesNet (Wu et al., ) and ModernTCN (Luo & Wang, ) utilize convolutional neural networks with time-series-specific modifications, making them better suited for forecasting tasks.
Additionally, simpler architectures such as Multi-Layer Perceptron (MLP)-based models (Zeng et al., ; Ekambaram et al., ) have demonstrated competitive performance.
However, Transformer-based models have gained particular prominence due to their ability to model long-term dependencies in time series (Zhou et al., ; Wu et al., ; Zhou et al., ; Liu et al., ).
Notably, PatchTST (Nie et al., ) has become a widely adopted Transformer variant, introducing a channel-independent patching mechanism to enhance temporal representations. This approach has been further extended by subsequent models (Liu et al., ; Eldele et al., ).

While previous work has primarily focused on capturing nonlinear dependencies in time series through enhanced model structures, our approach addresses the distribution shifts caused by evolving patterns within the data, which is a key limitation of existing methods. We emphasize tackling these shifts to improve forecasting performance in complex real-world scenarios.

The Combination of Time and Frequency Domains.
Time-domain models excel at capturing sequential trends, while frequency-domain models are essential for identifying periodic and oscillatory patterns.
Recent research has increasingly emphasized integrating information from both domains to better interpret underlying patterns.
For instance, ATFN (Yang et al., ) demonstrates the advantage of frequency domain methods for forecasting strongly periodic time series through a time–frequency adaptive network.
TFDNet (Luo et al., ) adopts a branching structure to capture long-term latent patterns and temporal periodicity from both domains.
Similarly, JTFT (Chen et al., ) utilizes the frequency domain representation to extract multi-scale dependencies while enhancing local relationships modeling through time domain representation.
Yan et al. () propose TFMRN, which expands data in both domains to capture finer details that may not be evident in the original data.
Recently, TSLANet (Eldele et al., ) leverages Fourier analysis to enhance feature representation and capture both long-term and short-term interactions.

Building on these approaches, our proposed method, TFPS, introduces a novel Dual-Domain Encoder that effectively combines time and frequency domain information to capture both trend and periodic patterns. By integrating time-frequency features, TFPS significantly advances the field in addressing the complexities inherent in time series forecasting.

Non-stationary Time Series Forecasting.
Concept drift describes unforeseeable changes in the underlying distribution of data over time, posing a significant challenge for time series forecasting (Lu et al., ). These non-stationarities complicate predictive modeling, necessitating effective solutions to handle shifting distributions.
To address varying distributions, normalization techniques have emerged as a focal point in recent research, aiming to mitigate non-stationary elements and align data to a consistent distribution.

For instance, adaptive norm (Ogasawara et al., ) applies z-score normalization using global statistics and DAIN (Passalis et al., ) introduces a neural layer for adaptively normalizing each input instance.
Kim et al. () propose a reversible instance normalization (RevIN) to alleviate series shift.
Furthermore, Non-stationary transformer (Liu et al., ) points that directly stationarizing time series will damage the model’s capability to capture specific temporal dependencies. This work addresses the problem by introducing an innovative de-stationary attention mechanism within self-attention frameworks.
Recent advancement include (Fan et al., ), which identifies both intra- and inter-space distribution shifts in time series data, and SAN (Liu et al., ), which applies normalization at the slice level, thus opening new avenues for handling non-stationary time series data.
Lastly, SIN (Han et al., ) introduces a novel method to selecting the statistics and learning normalization transformations to capture local invariance in time series data.

However, over-reliance on normalization can lead to over-stationarization, where important patterns or variations in the data are smoothed out (Liu et al., ).
Additionally, different patterns follow distinct internal dynamics, making a unified modeling approach inefficient.
Our approach brings the intrinsic non-stationarity of the original series back to latent representation, enabling better handling of distribution shifts by tailoring the experts to the evolving patterns and densities within the data.

## 3 Method

### 3.1 Preliminaries

Time series forecasting aims to uncover relationships between historical time series data and future data.
Let $\mathcal{X}$ denote the time series, and $x_{t}$ represent the value at timestep $t$.
Given the historical time series data $X=[x_{t-L+1},\cdots,x_{t}]\in\mathbb{R}^{L\times C}$, where $L$ is the length of the look-back window and $C>1$ is the number of features in each timestep, the objective is to predict the future series $Y=[x_{t+1},\cdots,x_{t+H}]\in\mathbb{R}^{H\times C}$, where $H$ is the forecast horizon.

Figure: Figure 2: The structure of our proposed TFPS. The input time series is divided into patches, and positional embeddings are added. These embeddings are processed through two branches: time-domain branch and frequency-domain branch. Each branch consists of three key components: (1) an encoder to capture patch-wise features, (2) a clustering mechanism to identify patches with similar patterns, and (3) a mixture of pattern experts block to model the patterns of each cluster. Finally, the outputs from both branches are combined for the final prediction.
Refer to caption: extracted/5922744/figure/fig-2.png

### 3.2 Overall Architecture

Our model introduces three novel components: the Dual-Domain Encoder (DDE), the Pattern Identifier (PI), and the Mixture of Pattern Experts (MoPE), as illustrated in Figure 2.
The DDE goes beyond traditional time-domain encoding by incorporating a frequency encoder that applies Fourier analysis, transforming time series data into the frequency domain. This enables the model to capture periodic patterns and frequency-specific features, providing a more comprehensive understanding of the data. The PI is a clustering-based module that distinguishes patches with distinct patterns, effectively addressing the variability in the data. MoPE then utilizes multiple MLP-based experts, each dedicated to modeling a specific pattern, thereby enhancing the model’s ability to adapt to the temporal dynamics of time series. Collectively, these components form a cohesive framework that effectively handles concept drift between patches, leading to more accurate time series forecasting.

### 3.3 Embedding Layer

Firstly, the input sequence $X\in\mathbb{R}^{L\times C}$ is divided into patches of length $P$, resulting in $N=\lfloor\frac{(L-P)}{S}+2\rfloor$ tokens, where $S$ denotes the stride, defining the non-overlapping region between consecutive patches. Each patch is denoted as $\mathcal{P}_{i}\in\mathbb{R}^{C\times P}$. These patches are then projected into a new dimension $D$, via a linear transformation, such that, $\mathcal{P}_{i}\rightarrow\mathcal{P}^{\prime}_{i}\in\mathbb{R}^{C\times D}$.

Next, positional embeddings are added to each patch to preserve the temporal ordering disrupted during the segmentation process. The position embedding for the $i$-th patch, denoted as $E_{i}$, is a vector of the same dimension as the projected patch. The enhanced patch is computed by summing the original patch and its positional embedding: $X_{PE_{i}}=\mathcal{P}^{\prime}_{i}+E_{i}$, and $X_{PE}=\{X_{PE_{1}},X_{PE_{2}},\cdots,X_{PE_{N}}\}$. Notably, the positional embeddings are learnable parameters, which enables the model to capture the temporal dependencies in the time series more effectively.
As a result, the final enriched patch representations are $X_{PE}\in\mathbb{R}^{C\times N\times D}$.

### 3.4 Dual-Domain Encoder

As shown in Figure 1, both time and frequency domains reveal distinct concept drifts that can significantly affect the performance of forecasting models.
To effectively address these drifts, we propose a Dual-Domain Encoder (DDE) architecture that captures both temporal and frequency dependencies inherent in time series data.

We utilize the patch-based Transformer (Nie et al., ) as an encoder to extract embeddings for each patch, capturing the global trend feature.
The multi-head attention is employed to obtain the attention output $\mathbf{O}_{t}\in\mathbf{R}^{N\times D}$ as follows:

$$ $\begin{split}\mathbf{O}_{t}=\text{Attention}(Q,K,V)=\text{Softmax}\left(\frac{ QK^{T}}{\sqrt{d_{k}}}\right)V,\\ Q=X_{PE}\mathbf{W}_{Q},\quad K=X_{PE}\mathbf{W}_{K},\quad V=X_{PE}\mathbf{W}_{ V}.\end{split}$ (1) $$

The encoder block also incorporates BatchNorm layers and a feed-forward network with residual connections, as shown in Figure 2 (b). This process generates the temporal features $z_{t}\in\mathbb{R}^{C\times N\times D}$.

In parallel with the time encoder, we incorporate a Frequency Encoder by replacing the self-attention sublayer of the Transformer with a Fourier sublayer (Lee-Thorp et al., ). This sublayer applies a 2D Fast Fourier Transform (the number of patches, hidden dimension) to the patch representation, expressed as:

$$ $\displaystyle\mathbf{O}_{f}=\mathcal{F}_{patch}(\mathcal{F}_{h}(X_{PE})).$ (2) $$

We only keep the real part of the result, and hence, we do not modify the feed-forward layers in the Transformer. The structure of the Frequency Encoder is depicted in Figure 2 (c), yielding frequency features $z_{f}\in\mathbb{R}^{C\times N\times D}$.

By modeling data in both the time and frequency domains, the DDE provides a more comprehensive understanding of the underlying patterns in time series data. This dual-domain perspective enables the model to better address complexities like concept drift and evolving temporal dynamics, ultimately improving its robustness and predictive accuracy. Through this approach, we aim to create a versatile framework capable of adapting to the intricate nature of real-world time series data.

Figure: Figure 3: Illustration of the proposed Pattern Identifier and Mixture of Pattern Experts. The embedded representation $\mathbf{z}$ from DDE combines with subspace $\mathbf{D}$ to construct the subspace affinity vector, which yields the normalized subspace affinity $S$. Subsequently, the refined subspace affinity $\widetilde{S}$ is computed from $S$ to provide self-supervised information. Then, we assign the corresponding patch-wise experts to the embedded representation $\mathbf{z}$ according to $S$ for modeling.
Refer to caption: extracted/5922744/figure/fig-3.png

### 3.5 Pattern Identifier

To address the complex and evolving patterns in time series data, we introduce a novel Pattern Identifier (PI) module, an essential innovation within our framework. Unlike traditional approaches that treat the entire time series uniformly, our PI module dynamically classifies patches based on their distributional characteristics, enabling a more precise and adaptive modeling strategy.

The core of our approach lies in leveraging subspace clustering to detect concept shifts across multiple subspaces, as illustrated in Figure 3.
This enables the PI module to distinguish patches by identifying their underlying patterns in both the time and frequency domains.

The PI module’s uniqueness lies in its ability to directly analyze the properties of each patch, clustering them into distinct groups. In the time domain, PI allows our framework, TFPS, to detect shifts related to temporal characteristics, such as seasonality and trends, which can influence forecasting accuracy. In the frequency domain, the PI captures shifts associated with frequency-specific features, like periodic behaviors and spectral changes, offering a comprehensive view of pattern evolution across the entire time series.

This dual-domain capability represents a significant advancement in our framework, making it highly sensitive to pattern shifts occurring across both temporal and frequency dimensions. By focusing on both domains, our model overcomes the limitations of existing methods that rely solely on one representation, ensuring robustness in diverse scenarios.

To provide clarity, Figure 3 showcases an application of the PI module exclusively within the time domain. However, the insights and methodology seamlessly extend to the frequency domain, presenting a unified solution to the challenge of concept shifts.

The PI module refines subspace bases iteratively, where the improved subspaces, in turn, enhance the representation learning in deep neural networks. This iterative refinement of subspaces is a key contribution of our work, leading to more accurate identification and modeling of evolving patterns.
The PI module operates in three key steps explained next.

Construction of subspace bases.
We define a new variable $\mathbf{D}=[\mathbf{D}^{(1)},\mathbf{D}^{(2)},\cdots,\mathbf{D}^{(k)}]$ to represent the bases of $k$ subspaces, where $\mathbf{D}$ consists of $k$ blocks, each $\mathbf{D}^{(j)}\in\mathbf{R}^{q\times d}$, $\left\|\mathbf{D}_{u}^{(j)}\right\|=1,u=1,\cdots d,j=1,\cdots,k$.
To control the size of the columns of $\mathbf{D}$, we impose the following constraint:

$$ $\displaystyle R_{1}=\frac{1}{2}\left\|\mathbf{D}^{T}\mathbf{D}\odot\mathbf{I}- \mathbf{I}\right\|_{F}^{2},$ (3) $$

where $\odot$ denotes the Hadamard product, and $\mathbf{I}$ is an identity matrix of size $kd\times kd$.

Subspaces differentiation.
To ensure the dissimilarity between different subspaces, we introduce the second constraint:

$$ $\displaystyle\begin{split}R_{2}&=\frac{1}{2}\left\|\mathbf{D}^{(j)T}\mathbf{D} ^{(l)}\right\|_{F}^{2},\quad j\neq l,\\ &=\frac{1}{2}\left\|\mathbf{D}^{T}\mathbf{D}\odot\mathbf{O}\right\|_{F}^{2}, \end{split}$ (4) $$

where $\mathbf{O}$ is a matrix with all off-diagonal $d$-size blocks set to 1 and diagonal blocks set to 0. Combining $D_{1}$ and $D_{2}$ yields the regularization term for $\mathbf{D}$:

$$ $\displaystyle R=\alpha(R_{1}+R_{2}),$ (5) $$

where $\alpha$ is a tuning parameters, fixed at $10^{-3}$ in this work.

Subspace affinity and refinement.
We propose a novel subspace affinity measure $\mathbf{S}$ to assess the relationship between the embedded representation $\mathbf{z}$ from DDE and the subspace bases $\mathbf{D}$. The affinity $s_{ij}$, representing the probability that the embedded $\mathbf{z}_{i}$ belongs to the $j$-th subspace, is defined as:

$$ $\displaystyle s_{ij}=\frac{\left\|\mathbf{z}_{i}^{T}\mathbf{D}^{(j)}\right\|_{ F}^{2}+\eta d}{\sum_{j}(\left\|\mathbf{z}_{i}^{T}\mathbf{D}^{(j)}\right\|_{F}^ {2}+\eta d)},$ (6) $$

where $\eta$ is a parameter controlling the smoothness.
To emphasize more confident assignments, we introduce a refined subspace affinity $\widetilde{S}$:

$$ $\displaystyle\widetilde{s}_{ij}=\frac{s_{ij}^{2}/\sum_{i}s_{ij}}{\sum_{j}(s_{ ij}^{2}/\sum_{i}s_{ij})}.$ (7) $$

This refinement sharpens the clustering by weighting high-confidence assignments more heavily. The subspace clustering objective objective based on the Kullback-Leibler divergence is then:

$$ $\displaystyle\mathcal{L}_{sub}=KL(\widetilde{S}\parallel S)=\sum_{i}\sum_{j} \widetilde{s}_{ij}log\frac{\widetilde{s}_{ij}}{s_{ij}}.$ (8) $$

The clustering loss is defined as:

$$ $\displaystyle\mathcal{L}_{PI}=R+\beta\mathcal{L}_{sub},$ (9) $$

where $\beta$ is a hyperparameter balancing the regularization and subspace clustering terms.

### 3.6 Mixture of Pattern Experts

Traditional time series forecasting methods often rely on a uniform distribution modeling (UDM) approach, which struggles to adapt to the complexities of diverse and evolving patterns in real-world data. To address this limitation, we introduce the Mixture of Pattern Experts module (MoPE), which assigns specialized experts to patches based on their unique underlying patterns, enabling more precise and adaptive forecasting.

Given the cluster assignments $s$ obtained from the PI module, we apply the Patch-wise MoPE to the feature tensor $z\in\mathbb{R}^{C\times N\times D}$. The MoPE module consists of the following key components:

Gating Network:
The gating network $G$ calculates the gating weights for each expert based on the cluster assignment $s$ and selects the top $k$ experts. The gating weights are computed as:

$$ $\displaystyle G(s)=\text{Softmax}(\text{TopK}(s)).$ (10) $$

Here, the top $k$ logits are selected and normalized using the Softmax function to produce the gating weights.

Expert Networks:
The MoPE contains $K$ expert networks, denoted as ${E_{1},\dots,E_{K}}$. Each expert network is modeled as an MLP consisting of two linear layers and a ReLU activation. Given a patch-wise feature $z$, each expert network $E_{k}$ processes the input to generate its respective output.

Output Aggregation:
The final output $h$ of the MoPE module is a weighted sum of the outputs from all the selected experts, with the weights provided by the gating network:

$$ $\displaystyle h=\sum_{k=1}^{K}G(s)E_{k}(z).$ (11) $$

After the frequency branch is processed by the inverse Fast Fourier transform, the time-frequency outputs $h_{t}$ and $h_{f}$, are concatenated to form $h=\text{concat}(h_{t},h_{f})\in\mathbb{R}^{C\times N\times 2D}$.

Finally, a linear transformation is applied to the concatenated output $h$ to generate the prediction: $\hat{Y}=\text{Linear}(h)\in\mathbb{R}^{H\times C}$.

This approach ensures that the MoPE dynamically assigns and aggregates contributions from various experts based on the evolving patterns, improving the model’s adaptability and accuracy.

### 3.7 Loss Function

Following the approach outlined in Nie et al. (), we use the Mean Squared Error (MSE) loss to quantify the discrepancy between predicted values $\hat{Y}$ and ground truth values $Y$:
$\mathcal{L}_{MSE}=(\hat{Y}-Y)^{2}$.
In addition to the MSE loss, we incorporate the clustering regularization loss from the PI module, yielding the final loss function:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{MSE}+\mathcal{L}_{PI_{t}}+\mathcal{L}_{ PI_{f}}.$ (12) $$

This combined loss ensures that the model not only minimizes forecasting errors but also accurately identifies and maintains the integrity of pattern clusters across time. The algorithm is provided in the Appendix G.

## 4 Experiments

In this section, we present experimental results to demonstrate the effectiveness of our proposed TFPS framework, including its forecasting performance and model analysis.

### 4.1 Experimental Setup

Dataset and Baselines.
We conducted our experiments on eight publicly available real-world multivariate time series datasets, i.e., ETT (ETTh1, ETTh2, ETTm1, ETTm2), Exchange, Weather, Electricity, and ILI. These datasets are provided in (Wu et al., ) for time series forecasting.
We followed the standard protocol for data preprocessing. Specifically, we split the datasets into training, validation, and testing by a ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other dataset (Zeng et al., ). Table 1 shows the statistics of these datasets.

In our experiments, we employed a diverse set of state-of-the-art forecasting models as baselines, categorized based on the type of information they utilize as follows.
(1) Time-domain methods: PatchTST (Nie et al., ), DLinear (Zeng et al., ), TimesNet (Wu et al., ) and iTransformer (Liu et al., );
(2) Frequency-domain methods: FEDformer (Zhou et al., ) and FITS (Xu et al., );
(3) Time-frequency methods: TFDNet-IK (Luo et al., ) and TSLANet (Eldele et al., ).
We rerun all the experiments with codes provided by their official implementation.

Note that some recent foundation models such as Time-LLM (Jin et al., ) and MOIRAI (Woo et al., ) have demonstrated remarkable performance in time series forecasting by leveraging knowledge from diverse datasets through pretraining. However, our TFPS model and the baselines above focus on specific dataset for training and testing. Therefore, we did not include these pretrained foundation models in comparison.

**Table 1: The statistics of the datasets.**
| Datasets | ETTh1 & ETTh2 | ETTm1 & ETTm2 | Exchange-Rate | Weather | Electricity | ILI |
| --- | --- | --- | --- | --- | --- | --- |
| Variates | 7 | 7 | 8 | 21 | 321 | 7 |
| Timesteps | 17,420 | 69,680 | 7,588 | 52,696 | 26,304 | 966 |
| Granularity | 1 hour | 15 min | 1 day | 10 min | 1 hour | 1 week |

Experiments details.
Following previous works (Nie et al., ), we used ADAM (Kingma & Ba, ) as the default optimizer across all the experiments. We employed the MSE and mean absolute error (MAE) as the evaluation metrics, and a lower MSE/MAE value indicates a better performance. TFPS was implemented by PyTorch (Paszke et al., ) and trained on a single NVIDIA RTX 3090 24GB GPU.
We conducted grid search to optimize the following three parameters, i.e., $\text{learning rate}=\{0.0001,0.0005,0.001,0.005,0.01,0.05\}$, $K_{t}=\{1,2,4,8\}$, and $K_{f}=\{1,2,4,8\}$.
The lookback length $L$ was set to 104 for ILI and 96 for other datasets.

### 4.2 Overall Performance Comparison

Table 2 highlights the consistent superiority of TFPS across multiple datasets and prediction horizons, securing the top performance in 50 out of 64 experimental configurations.
In particular, TFPS demonstrates significant improvements over time-domain methods, with an overall improvement of 8.7% in MSE and 5.9% in MAE. Compared to frequency-domain methods, TFPS shows even more pronounced enhancements, with MSE improved by 15.8% and MAE by 11.6%.

While the time-frequency methods like TSLANet and TFDNet perform competitively on several datasets, TFPS still outperforms them, showing improvement of 4.5% in MSE and 1.9% in MAE. These substantial improvements can be attributed to the integration of both time- and frequency-domain information, combined with our innovative approach to modeling distinct patterns with specialized experts. By addressing the underlying concept shifts and capturing complex, evolving patterns in time series data, TFPS achieves more accurate predictions than other baseline models.

**Table 2: Multivariate long-term forecasting results with prediction lengths $H\in\{24,36,48,60\}$ for ILI and $H\in\{96,192,336,720\}$ for others. The input lengths are $L=104$ for ILI and $L=96$ for others. The best results are highlighted in bold and the second best are \ulunderlined.**
| Model | IMP. | TFPS | TSLANet | FITS | iTransformer | TFDNet-IK | PatchTST | TimesNet | DLinear | FEDformer |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Our) | () | () | () | () | () | () | () | () |  |  |  |  |  |  |  |  |  |  |  |  |
| Metric | MSE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
| ETTh1 | 96 | -1.1% | 0.398 | 0.413 | 0.387 | 0.405 | 0.395 | 0.403 | \ul0.387 | \ul0.405 | 0.396 | 0.409 | 0.413 | 0.419 | 0.389 | 0.412 | 0.398 | 0.410 | 0.385 | 0.425 |
| 192 | 4.8% | 0.423 | 0.423 | 0.448 | 0.436 | 0.445 | 0.432 | 0.441 | 0.436 | 0.451 | 0.441 | 0.460 | 0.445 | 0.441 | 0.442 | \ul0.434 | \ul0.427 | 0.441 | 0.461 |  |
| 336 | 1.8% | 0.484 | 0.461 | 0.491 | 0.487 | \ul0.489 | 0.463 | 0.491 | 0.463 | 0.495 | \ul0.462 | 0.497 | 0.463 | 0.491 | 0.467 | 0.499 | 0.477 | 0.491 | 0.473 |  |
| 720 | 3.0% | 0.488 | 0.476 | 0.505 | 0.486 | 0.496 | 0.485 | 0.509 | 0.494 | \ul0.492 | \ul0.482 | 0.501 | 0.486 | 0.512 | 0.491 | 0.508 | 0.503 | 0.501 | 0.499 |  |
| ETTh2 | 96 | -2.0% | 0.313 | 0.355 | \ul0.290 | 0.345 | 0.295 | \ul0.344 | 0.301 | 0.350 | 0.289 | 0.337 | 0.299 | 0.348 | 0.324 | 0.368 | 0.315 | 0.374 | 0.342 | 0.383 |
| 192 | -2.9% | 0.405 | 0.410 | 0.362 | 0.391 | 0.382 | 0.396 | 0.380 | 0.399 | \ul0.379 | \ul0.395 | 0.383 | 0.398 | 0.393 | 0.410 | 0.432 | 0.447 | 0.434 | 0.440 |  |
| 336 | 10.5% | 0.392 | 0.415 | \ul0.401 | \ul0.419 | 0.416 | 0.425 | 0.424 | 0.432 | 0.416 | 0.422 | 0.424 | 0.431 | 0.429 | 0.437 | 0.486 | 0.481 | 0.512 | 0.497 |  |
| 720 | 12.6% | 0.410 | 0.433 | 0.419 | 0.439 | \ul0.418 | \ul0.437 | 0.430 | 0.447 | 0.424 | 0.441 | 0.429 | 0.445 | 0.433 | 0.448 | 0.732 | 0.614 | 0.467 | 0.476 |  |
| ETTm1 | 96 | 4.1% | 0.327 | 0.367 | \ul0.329 | \ul0.368 | 0.354 | 0.375 | 0.342 | 0.377 | 0.331 | 0.369 | 0.331 | 0.370 | 0.337 | 0.377 | 0.346 | 0.374 | 0.360 | 0.406 |
| 192 | 2.6% | 0.374 | 0.395 | 0.376 | \ul0.383 | 0.392 | 0.393 | 0.383 | 0.396 | 0.376 | 0.381 | \ul0.374 | 0.395 | 0.395 | 0.406 | 0.382 | 0.392 | 0.395 | 0.427 |  |
| 336 | 4.2% | 0.401 | 0.408 | 0.403 | 0.414 | 0.425 | 0.415 | 0.418 | 0.418 | 0.405 | \ul0.410 | \ul0.402 | 0.412 | 0.433 | 0.432 | 0.414 | 0.414 | 0.448 | 0.458 |  |
| 720 | -0.7% | 0.479 | 0.456 | 0.445 | \ul0.438 | 0.486 | 0.449 | 0.487 | 0.457 | 0.471 | 0.437 | \ul0.466 | 0.446 | 0.484 | 0.458 | 0.478 | 0.455 | 0.491 | 0.479 |  |
| ETTm2 | 96 | 6.9% | 0.170 | 0.255 | 0.179 | 0.261 | 0.183 | 0.266 | 0.186 | 0.272 | \ul0.176 | 0.267 | 0.177 | \ul0.260 | 0.182 | 0.262 | 0.184 | 0.276 | 0.193 | 0.285 |
| 192 | 7.1% | 0.235 | 0.296 | \ul0.243 | 0.303 | 0.247 | 0.305 | 0.254 | 0.314 | 0.245 | \ul0.302 | 0.248 | 0.306 | 0.252 | 0.307 | 0.282 | 0.357 | 0.256 | 0.324 |  |
| 336 | 4.6% | 0.297 | 0.335 | 0.308 | 0.345 | 0.307 | 0.342 | 0.316 | 0.351 | \ul0.303 | \ul0.340 | 0.303 | 0.341 | 0.312 | 0.346 | 0.324 | 0.364 | 0.321 | 0.364 |  |
| 720 | 3.6% | 0.401 | 0.397 | \ul0.403 | 0.400 | 0.407 | 0.401 | 0.414 | 0.407 | 0.405 | \ul0.399 | 0.405 | 0.403 | 0.417 | 0.404 | 0.441 | 0.454 | 0.434 | 0.426 |  |
| Exchange | 96 | 12.7% | 0.083 | 0.205 | 0.085 | 0.206 | 0.088 | 0.210 | 0.086 | 0.206 | \ul0.084 | \ul0.205 | 0.089 | 0.206 | 0.105 | 0.233 | 0.089 | 0.219 | 0.136 | 0.265 |
| 192 | 11.2% | 0.174 | 0.297 | 0.178 | 0.300 | 0.181 | 0.304 | 0.181 | 0.304 | \ul0.176 | \ul0.299 | 0.178 | 0.302 | 0.219 | 0.342 | 0.180 | 0.319 | 0.279 | 0.384 |  |
| 336 | 10.4% | 0.310 | 0.398 | 0.329 | 0.415 | 0.324 | 0.413 | 0.338 | 0.422 | 0.321 | \ul0.409 | 0.326 | 0.411 | 0.353 | 0.433 | \ul0.313 | 0.423 | 0.465 | 0.504 |  |
| 720 | -13.3% | 1.011 | 0.756 | 0.850 | 0.693 | 0.846 | 0.696 | 0.853 | 0.696 | 0.835 | 0.689 | 0.840 | 0.690 | 0.912 | 0.724 | \ul0.837 | \ul0.690 | 1.169 | 0.826 |  |
| Weather | 96 | 15.6% | 0.154 | 0.202 | 0.176 | 0.216 | 0.167 | 0.214 | 0.176 | 0.216 | \ul0.165 | \ul0.209 | 0.177 | 0.219 | 0.168 | 0.218 | 0.197 | 0.257 | 0.236 | 0.325 |
| 192 | 10.6% | 0.205 | 0.249 | 0.226 | 0.258 | 0.215 | 0.257 | 0.225 | 0.257 | \ul0.214 | \ul0.252 | 0.225 | 0.259 | 0.226 | 0.267 | 0.237 | 0.294 | 0.268 | 0.337 |  |
| 336 | 9.1% | 0.262 | 0.289 | 0.279 | 0.299 | 0.270 | 0.299 | 0.281 | 0.299 | \ul0.267 | \ul0.298 | 0.278 | 0.298 | 0.283 | 0.305 | 0.283 | 0.332 | 0.366 | 0.402 |  |
| 720 | 4.1% | 0.344 | 0.342 | 0.355 | 0.355 | 0.347 | \ul0.345 | 0.358 | 0.350 | \ul0.347 | 0.346 | 0.351 | 0.346 | 0.355 | 0.353 | 0.347 | 0.382 | 0.407 | 0.422 |  |
| Electricity | 96 | 14.6% | 0.149 | 0.236 | 0.155 | 0.249 | 0.200 | 0.278 | \ul0.151 | \ul0.241 | 0.171 | 0.254 | 0.166 | 0.252 | 0.168 | 0.272 | 0.195 | 0.277 | 0.189 | 0.304 |
| 192 | 12.0% | 0.162 | 0.253 | 0.170 | 0.264 | 0.200 | 0.281 | \ul0.167 | \ul0.258 | 0.189 | 0.269 | 0.174 | 0.261 | 0.186 | 0.289 | 0.194 | 0.281 | 0.198 | 0.312 |  |
| 336 | 0.2% | 0.200 | 0.310 | 0.197 | 0.282 | 0.214 | 0.295 | 0.179 | 0.271 | 0.205 | 0.284 | \ul0.190 | \ul0.277 | 0.197 | 0.298 | 0.207 | 0.296 | 0.212 | 0.326 |  |
| 720 | 7.2% | 0.220 | 0.320 | \ul0.224 | \ul0.318 | 0.256 | 0.328 | 0.229 | 0.319 | 0.247 | 0.318 | 0.230 | 0.312 | 0.225 | 0.322 | 0.243 | 0.330 | 0.242 | 0.351 |  |
| ILI | 24 | 40.9% | 1.349 | 0.760 | 1.749 | 0.898 | 3.489 | 1.373 | 2.443 | 1.078 | 1.824 | \ul0.824 | \ul1.614 | 0.835 | 1.699 | 0.871 | 2.239 | 1.041 | 3.217 | 1.246 |
| 36 | 43.6% | 1.239 | 0.752 | 1.754 | 0.912 | 3.530 | 1.370 | 2.455 | 1.086 | 1.699 | \ul0.813 | \ul1.475 | 0.859 | 1.733 | 0.913 | 2.238 | 1.049 | 2.688 | 1.074 |  |
| 48 | 40.4% | 1.461 | 0.801 | 2.050 | 0.984 | 3.671 | 1.391 | 3.437 | 1.331 | 1.762 | \ul0.831 | \ul1.642 | 0.880 | 2.272 | 0.999 | 2.252 | 1.064 | 2.540 | 1.057 |  |
| 60 | 39.8% | 1.458 | 0.836 | 2.240 | 1.039 | 4.030 | 1.462 | 2.734 | 1.155 | 1.758 | \ul0.863 | \ul1.608 | 0.885 | 1.998 | 0.974 | 2.236 | 1.057 | 2.782 | 1.136 |  |
| $1^{\text{st}}$ Count | 50 | 3 | 1 | 2 | 6 | 1 | 0 | 0 | 1 |  |  |  |  |  |  |  |  |  |  |  |

### 4.3 Ablation Study

Table 3 presents the MSE results of TFPS and its variants with different combinations of encoders, PI, and MoPE.
1) Best Result.
The full TFPS model, i.e., both the time and frequency branches, along with their respective encoders, PI, and MoPE are included, performs the best across all the forecast horizons for both datasets.
2) Linear vs. PI.
We replace PI with a linear layer and find that it generally results in higher MSE in most cases, indicating that accurately capturing specific patterns is crucial.
3) Impact of Pattern-aware Modeling.
Additionally, when comparing the results with the encoder-only configuration, two variants with MoPE in each branch achieved improved MSE, further supporting the necessity of patter-aware modeling.
4) Importance of DDE.
Furthermore, we find that both the time encoder and frequency encoder alone yield worse performance, with the time encoder playing a more significant role.
In summary, incorporating both branches with PI and MoPE provides the best performance, while simpler configurations result in higher MSE.

**Table 3: Ablation study of TFPS components. The model variants in our ablation study include the following configurations across both time and frequency branches: (a) inclusion of the encoder, PI and MoPE; (b) PI replaced with Linear; (c) only the encoder. The best results are in bold.**
| Time Branch | Frequency Branch | ETTh1 | ETTh2 |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Encoder | PI | MoPE | Encoder | PI | MoPE | 96 | 192 | 336 | 720 | 96 | 192 | 336 | 720 |
| ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.398 | 0.423 | 0.484 | 0.488 | 0.313 | 0.405 | 0.392 | 0.410 |
| ✓ | ✓ | ✓ |  |  |  | \ul0.401 | 0.459 | \ul0.486 | \ul0.492 | \ul0.318 | 0.409 | \ul0.400 | \ul0.428 |
| ✓ | Linear | ✓ |  |  |  | 0.401 | \ul0.451 | 0.494 | 0.509 | 0.325 | 0.411 | 0.400 | 0.434 |
| ✓ |  |  |  |  |  | 0.414 | 0.460 | 0.501 | 0.500 | 0.339 | 0.411 | 0.426 | 0.431 |
|  |  |  | ✓ | ✓ | ✓ | 0.455 | 0.507 | 0.539 | 0.576 | 0.324 | \ul0.407 | 0.417 | 0.436 |
|  |  |  | ✓ | Linear | ✓ | 0.503 | 0.535 | 0.558 | 0.583 | 0.398 | 0.446 | 0.457 | 0.444 |
|  |  |  | ✓ |  |  | 0.552 | 0.583 | 0.591 | 0.594 | 0.371 | 0.426 | 0.418 | 0.463 |

### 4.4 Comparsion With Normalization Methods

**Table 4: Comparison between TFPS and normalization approaches.**
| Model | TFPS | FEDformer |  |  |
| --- | --- | --- | --- | --- |
| + SIN | + Dish-TS | + NST |  |  |
| ETTh1 | 0.448 | 0.458 | 0.461 | 0.456 |
| ETTh2 | 0.427 | 0.457 | 0.565 | 0.453 |
| ETTm1 | 0.422 | 0.465 | 0.689 | 0.462 |
| ETTm2 | 0.400 | 0.480 | 0.825 | 0.481 |
| Weather | 0.380 | 0.501 | 1.005 | 0.481 |

Normalization methods such as SIN (Han et al., ), Dish-TS (Fan et al., ), and Non-Stationary Transformers (Liu et al., ) can reduce fluctuations to enhance performance and are widely used for non-stationary time series forecasting.
We compare our TFPS with these state-of-the-art normalization methods and Table 4 presents the average MSE evaluation across all forecasting lengths for each dataset.
While normalization methods contribute to data stabilization, TFPS provides a more nuanced approach by leveraging distribution-specific modeling, leading to significant improvements with an average MSE decrease of 37.5%. Detailed results for all cases can be found in Appendix E.2.

### 4.5 Visualization

We visualize the prediction curves for ETTh1 with $H=192$. Given that DLinear exhibits competitive performance, we compare its results with those of TFPS in Figure 4 under two scenarios: (a) sudden drift caused by external factors or random events, and (b) gradual drift where the trend is dominant.
It is evident that DLinear struggles to achieve accurate predictions in both scenarios. In contrast, our TFPS consistently produces accurate forecasts despite these challenges, demonstrating its robustness in dealing with various concept dynamics.

Figure: (a) Sudden Drift
Refer to caption: extracted/5922744/figure/fig-4-a.png

Figure: (a) Expert 0
Refer to caption: extracted/5922744/figure/fig-5-a.png

### 4.6 Analysis of experts

Qualitative Visualizations of Pattern Identifier.
Through training, pattern experts in MoPE spontaneously specialize, and we present two examples in Figure 6. We visualize the expert with the highest score as the routed expert for each instance pair. In the provided examples, we observe that expert-0 specialize in downward-related concepts, while expert-4 focuses on parabolic trend. These examples also demonstrate the interpretability of MoPE.

Number of Experts.
In Figure 6, we set the learning rate to 0.0001 and conducted four sets of experiments on the ETTh1 and ETTh2 datasets, T1F1, T1F2, T1F4, T1F8, to explore the effect of the number of frequency experts on the results.
For example, T1F4 means that the TFPS contains 1 time experts and 4 frequency experts.
We observed that T1F2 outperformed T1F4 in both cases, suggesting that increasing the number of experts does not always lead to better performance.

In addition, we conducted three experiments based on the optimal number of frequency experts to verify the impact of varying the number of time experts on the results.
As shown in Figure 6, the best results for ETTh1 were obtained with T4F2, while for ETTh2, the optimal results were achieved with T2F2.
Combined with the average MMD in Table 5 (Appendix A), we attribute this to the fact that, in cases where concept drift is more severe, such as ETTh1 in the time domain, more experts are needed, whereas fewer experts are sufficient when the drift is less severe.

## 5 Conclusion

In this paper, we propose a novel pattern-aware time series forecasting framework, TFPS, which incorporates a dual-domain mixture of pattern experts approach.
Our TFPS framework aims to address the distribution shift across time series patches and effectively assigns pattern-specific experts to model them.
Experimental results across eight diverse datasets demonstrate that TFPS surpasses state-of-the-art methods in both quantitative metrics and visualizations.
Future work will focus on investigating evolving distribution shifts, particularly those introduced by the emergence of new patterns, such as unforeseen epidemics or outbreaks.

## Appendix A Dataset

We evaluate the performance of TFPS on eight widely used datasets, including four ETT datasets (ETTh1, ETTh2, ETTm1 and ETTm2), Exchange, Weather, Electricity, and ILI. This subsection provides a summary of the datasets:

- •
ETT (^1^11[https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset)) (Zhou et al., ) (Electricity Transformer Temperature) dataset contains two electric transformers, ETT1 and ETT2, collected from two separate counties. Each of them has two versions of sampling resolutions (15min & 1h). Thus, there are four ETT datasets: ETTm1, ETTm2, ETTh1, and ETTh2.
- •
Exchange-Rate (^2^22[https://github.com/laiguokun/multivariate-time-series-data](https://github.com/laiguokun/multivariate-time-series-data)) (Lai et al., ) the exchange-rate dataset contains the daily exchange rates of eight foreign countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore ranging from 1990 to 2016.
- •
Weather (^3^33[https://www.bgc-jena.mpg.de/wetter/](https://www.bgc-jena.mpg.de/wetter/)) (Wu et al., ) dataset contains 21 meteorological indicators in Germany, such as humidity and air temperature.
- •
Electricity (^4^44[https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014)) (Wu et al., ) is a dataset that describes 321 customers’ hourly electricity consumption.
- •
ILI (^5^55[https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html](https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html)) (Wu et al., ) dataset collects the number of patients and influenza-like illness ratio in a weekly frequency.

For the data split, we follow Zeng et al. () and split the data into training, validation, and testing by a ratio of 6:2:2 for the ETT datasets and 7:1:2 for the others. Details are shown in Table 5. The best parameters are selected based on the lowest validation loss and then applied to the test set for performance evaluation.

**Table 5: The statistics of the datasets.**
| Datasets | Variates | Prediction Length | Timesteps | Granularity | Average MMD^*<br>(Time Domain) | Average MMD^*<br>(Frequency Domain) |
| --- | --- | --- | --- | --- | --- | --- |
| ETTh1 | 7 | {96, 192, 336, 720} | 17,420 | 1 hour | 0.938 | 0.340 |
| ETTh2 | 7 | {96, 192, 336, 720} | 17,420 | 1 hour | 0.582 | 0.635 |
| ETTm1 | 7 | {96, 192, 336, 720} | 69,680 | 15 min | 1.371 | 0.328 |
| ETTm2 | 7 | {96, 192, 336, 720} | 69,680 | 15 min | 1.213 | 0.815 |
| Exchange-Rate | 8 | {96, 192, 336, 720} | 7,588 | 1 day | 0.805 | 0.485 |
| Weather | 21 | {96, 192, 336, 720} | 52,696 | 10 min | 0.129 | 0.236 |
| Electricity | 321 | {96, 192, 336, 720} | 26,304 | 1 hour | 0.026 | 0.005 |
| ILI | 7 | {24, 36, 48, 60} | 966 | 1 week | 0.125 | 0.234 |
| ^* A large MMD indicates a more severe drift. |  |  |  |  |  |  |

## Appendix B Maximum Mean Discrepancy

Maximum mean discrepancy (MMD) is a kernel-based statistical test used to determine whether given two distribution are the same.
Given an $X$, the feature map $\phi$ transforms $X$ to an another space $\mathcal{H}$ such that $\phi(X)\in\mathcal{H}$. $\mathcal{H}$ is Reproducing Kernel Hilbert Space (RKHS) and we can leverage the kernel trick to compute inner products in $\mathcal{H}$:

$$ $\displaystyle X,Y\text{\quad such that\quad}k(X,Y)=\langle\phi(X),\phi(Y) \rangle_{\mathcal{H}}.$ (13) $$

Feature means.
The mean embeddings of a probability distribution $P$ is a feature map that transforms $\phi(X)$ into the mean of each coordinate of $\phi(X)$:

$$ $\displaystyle\mu_{P}(\phi(X))=[\mathbb{E}[\phi(X_{1})],\cdots,\mathbb{E}[\phi( X_{m})]]^{T}.$ (14) $$

The inner product of the mean embeddings of $X\sim P$ and $Y\sim Q$ can be written in terms of kernel function:

$$ $\displaystyle\langle\mu_{P}(\phi(X)),\mu_{Q}(\phi(Y))\rangle_{\mathcal{H}}= \mathbb{E}_{P,Q}[\langle\phi(X),\phi(Y)\rangle_{\mathcal{H}}]=\mathbb{E}_{P,Q} [k(X,Y)].$ (15) $$

Maximum mean discrepancy.
The MMD measures the distance between the mean embeddings of two samples, $X$ and $Y$, in the RKHS:

$$ $\displaystyle\operatorname{MMD}^{2}(P,Q)=\left\|\mu_{P}-\mu_{Q}\right\|_{ \mathcal{H}}^{2},$ (16) $$

For convenience we omit the $\phi(\cdot)$ terms. If we use the norm induced by the inner product such that $\|x\|=\sqrt{\langle x,x\rangle}$, the Eq. 16 becomes:

$$ $\displaystyle\operatorname{MMD}^{2}(P,Q)=\langle\mu_{p}-\mu_{Q},\mu_{p}-\mu_{Q }\rangle=\langle\mu_{p},\mu_{p}\rangle-2\langle\mu_{p},\mu_{Q}\rangle+\langle \mu_{Q},\mu_{Q}\rangle.$ (17) $$

Using the Eq. 15, finally above expression becomes:

$$ $\displaystyle\operatorname{MMD}^{2}(P,Q)=\mathbb{E}_{P}[k(X,X)]-2\mathbb{E}_{P ,Q}[k(X,Y)]+\mathbb{E}_{Q}[k(Y,Y)].$ (18) $$

Empirical estimation of MMD.
In real-world applications, the underlying distribution are usually unknown. Thus, an empirical estimate of Eq. 18 can be used:

$$ $\displaystyle\operatorname{MMD}^{2}(X,Y)=\frac{1}{m(m-1)}\sum_{i\neq j}k\left( x_{i},x_{j}\right)-\frac{2}{mn}\sum_{i,j}k\left(x_{i},x_{j}\right)+\frac{1}{n( n-1)}\sum_{i\neq j}k\left(y_{i},y_{j}\right),$ (19) $$

where $x_{i}$ and $x_{j}$ are samples from $P$, $y_{i}$ and $y_{j}$ are samples from $Q$, and $k(x,y)$ is the kernel function, often the Gaussian (RBF) kernel.

## Appendix C Distribution shifts in both time and frequency domains

The time series $\mathcal{X}$ is segmented into $N$ patches, where each patch $\mathcal{P}_{n}=\{x_{n1},x_{n2},\dots,x_{nP}\}$ consists of $P$ consecutive timesteps for $n=1,2,\cdots,N$.
For the frequency domain, we apply a Fourier transform $\mathcal{F}$ to each patch $\mathcal{P}_{n}$, obtaining its frequency-domain representation as $\hat{\mathcal{P}}_{n}=\mathcal{F}(\mathcal{P}_{n})$.

Each patch’s probability distribution in the time domain is denoted as $p_{t}(\mathcal{P}_{n})$, representing the statistical properties of $\mathcal{P}_{n}$, while its frequency domain distribution, denoted as $p_{f}(\hat{\mathcal{P}}_{n})$, captures its spectral characteristics.

The distribution shifts between two patches $\mathcal{P}_{i}$ and $\mathcal{P}_{j}$ are characterized by the comparing their probability distributions in both time and frequency domains. These shifts are defined as:

$$ $\displaystyle\mathcal{D}_{t}(\mathcal{P}_{i},\mathcal{P}_{j})$ $\displaystyle=|d(p_{t}(\mathcal{P}_{i}),p_{t}(\mathcal{P}_{j}))|>\theta,$ (20) $\displaystyle\mathcal{D}_{f}(\hat{\mathcal{P}}_{i},\hat{\mathcal{P}}_{j})$ $\displaystyle=|d(p_{f}(\hat{\mathcal{P}}_{i}),p_{f}(\hat{\mathcal{P}}_{j}))|>\theta,$ (21) $$

where $d$ is a distance metric, such as MMD values or Kullback-Leibler divergence, and $\theta$ is a threshold indicating a significant distribution shift. If $\mathcal{D}_{t}(\mathcal{P}_{i},\mathcal{P}_{j})$ or $\mathcal{D}_{f}(\hat{\mathcal{P}}_{i},\hat{\mathcal{P}}_{j})$ exceeds $\theta$, this implies a significant distribution shift between the two patches in either domain.

## Appendix D Related Work

Mixture-of-Experts.
Mixture-of-Experts (MoE) models have gained attention for their ability to scale efficiently by activating only a subset of experts for each input, as first introduced by Shazeer et al. (). Despite their success, challenges such as training instability, expert redundancy, and limited expert specialization have been identified (Puigcerver et al., ; Dai et al., ). These issues hinder the full potential of MoE models in real-world tasks.

Recent advances have integrated MoE with Transformers to improve scalability and efficiency. For example, GLaM (Du et al., ) and Switch Transformer (Fedus et al., ) interleave MoE layers with Transformer blocks, reducing computational costs.
Other models like state space models (SSMs) (Pióro et al., ; Anthony et al., ) combines MoE with alternative architectures for enhanced scalability and inference speed.

In contrast, our approach introduces MoE into time series forecasting by assigning experts to specific time-frequency patterns, enabling more effective, patch-level adaptation. This approach represents a significant innovation in time series forecasting, offering a more targeted and effective way to handle varying patterns across both time and frequency domains.

## Appendix E More Model Analysis

### E.1 Analysis of experts

Detailed Results on the Number of Experts.

We provide the full results on the number of experts for the ETTh1 and ETTh2 dataset in Figure 7.

In Figure 7, we set the learning rate to 0.0001 and conducted four sets of experiments on the ETTh1 and ETTh2 datasets, T1F1, T1F2, T1F4, T1F8, to explore the effect of the number of frequency experts on the results.
For example, T1F4 means that the TFPS contains 1 time experts and 4 frequency experts.
We observed that T1F2 outperformed T1F4 in four cases, suggesting that increasing the number of experts does not always lead to better performance.

In addition, we conducted three experiments based on the optimal number of frequency experts to verify the impact of varying the number of time experts on the results.
As shown in Figure 7, the best results for ETTh1 were obtained with T4F2, T8F4, T4F4, T4F4, while for ETTh2, the optimal results were achieved with T2F2, T2F4, T4F2 and T4F2.
Combined with the average MMD in Table 5, we attribute this to the fact that, in cases where concept drift is more severe, such as ETTh1 in the time domain, more experts are needed, whereas fewer experts are sufficient when the drift is less severe.

Figure: Figure 7: Results of expert number experiments for ETTh1 and ETTh2.
Refer to caption: extracted/5922744/figure/fig-7.png

Comparing Inter- and Intra-Cluster Differences via MMD.

We present the heatmaps of inter-cluster and intra-cluster MMD values obtained using linear layers and PI in Figure 8.
The diagonal elements represent the average MMD values of patches within the same clusters. If these values are small, it indicates that the difference of patches within the same cluster is relatively similar.
The off-diagonal elements represent the average MMD values between patches from different clusters, where larger values mean significant differences between the clusters.
We observe that when using PI, the intra-cluster drift is smaller, while the inter-cluster shift is more pronounced compared to the linear layer. This indicates that our identifier effectively classifies and distinguishes between different patterns.

Figure: (a) Linear layer
Refer to caption: extracted/5922744/figure/fig-8-a.png

### E.2 Results of the Comparison between TFPS and Normalization Methods

In this section, we provide the detailed experimental results of the comparison between TFPS and five state-of-the-art normalization methods for non-stationary time series forecasting: SIN (Han et al., ), SAN (Liu et al., ), Dish-TS (Fan et al., ), Non-Stationary Transformers (NST) (Liu et al., ), and RevIN (Kim et al., ).
The results of SIN are from Han et al. (), other results are from Liu et al. ().
We report the evaluation of FEDformer over all the forecasting lengths for each dataset and the relative improvements in Table 6.
It can be concluded that TFPS achieves the best performance among existing methods in most cases. The improvement is significant with an average MSE decrease of 18.9%.
We attribute this improvement to the accurate identification of pattern groups and the provision of specialized experts for each group, thereby avoiding the over-stationarization problem often associated with normalization methods.

**Table 6: Detailed results of the comparison between TFPS and normalization methods. The best results are highlighted in bold and the second best are \ulunderlined.**
|  |  |  | FEDformer |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | TFPS | + SIN | + SAN | + Dish-TS | + NST | + RevIN |  |  |  |  |  |  |  |
| Model | IMP. | (Our) | () | () | () | () | () |  |  |  |  |  |  |  |
| Metric | MSE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |  |
|  | 96 | -1.0% | 0.398 | 0.413 | 0.413 | 0.372 | 0.383 | \ul0.409 | \ul0.390 | 0.424 | 0.394 | 0.414 | 0.392 | 0.413 |
|  | 192 | 3.8% | 0.423 | \ul0.423 | 0.443 | 0.417 | \ul0.431 | 0.438 | 0.441 | 0.458 | 0.441 | 0.442 | 0.443 | 0.444 |
|  | 336 | -0.3% | 0.484 | 0.461 | 0.465 | 0.448 | \ul0.471 | \ul0.456 | 0.495 | 0.486 | 0.485 | 0.466 | 0.495 | 0.467 |
| ETTh1 | 720 | 4.5% | 0.488 | 0.476 | 0.509 | 0.490 | \ul0.504 | \ul0.488 | 0.519 | 0.509 | 0.505 | 0.496 | 0.520 | 0.498 |
|  | 96 | 31.3% | \ul0.313 | 0.355 | 0.412 | 0.357 | 0.300 | \ul0.355 | 0.806 | 0.589 | 0.381 | 0.403 | 0.380 | 0.402 |
|  | 192 | 26.0% | \ul0.405 | 0.410 | 0.472 | 0.453 | 0.392 | \ul0.413 | 0.936 | 0.659 | 0.478 | 0.453 | 0.457 | 0.443 |
|  | 336 | 36.7% | 0.392 | 0.415 | 0.527 | 0.527 | \ul0.459 | \ul0.462 | 1.039 | 0.702 | 0.561 | 0.499 | 0.515 | 0.479 |
| ETTh2 | 720 | 37.9% | 0.410 | 0.433 | 0.593 | 0.639 | \ul0.462 | \ul0.472 | 1.237 | 0.759 | 0.502 | 0.481 | 0.507 | 0.487 |
|  | 96 | 4.1% | \ul0.327 | 0.367 | 0.373 | 0.320 | 0.311 | \ul0.355 | 0.348 | 0.397 | 0.336 | 0.382 | 0.340 | 0.385 |
|  | 192 | 2.9% | \ul0.374 | 0.395 | 0.394 | 0.366 | 0.351 | \ul0.383 | 0.406 | 0.428 | 0.386 | 0.409 | 0.390 | 0.411 |
|  | 336 | 5.3% | \ul0.401 | 0.408 | 0.418 | 0.405 | 0.390 | \ul0.407 | 0.438 | 0.450 | 0.438 | 0.441 | 0.432 | 0.436 |
| ETTm1 | 720 | -0.5% | 0.479 | \ul0.456 | 0.451 | 0.475 | \ul0.456 | 0.444 | 0.497 | 0.481 | 0.483 | 0.460 | 0.497 | 0.466 |
|  | 96 | 33.5% | 0.170 | \ul0.255 | 0.326 | 0.211 | \ul0.175 | 0.266 | 0.394 | 0.395 | 0.191 | 0.272 | 0.192 | 0.272 |
|  | 192 | 32.3% | 0.235 | 0.296 | 0.402 | 0.316 | \ul0.246 | \ul0.315 | 0.552 | 0.472 | 0.270 | 0.321 | 0.270 | 0.320 |
|  | 336 | 35.0% | 0.297 | 0.335 | 0.465 | 0.399 | \ul0.315 | \ul0.362 | 0.808 | 0.601 | 0.353 | 0.371 | 0.348 | 0.367 |
| ETTm2 | 720 | 35.9% | 0.401 | 0.397 | 0.555 | 0.547 | \ul0.412 | 0.422 | 1.282 | 0.771 | 0.445 | 0.422 | 0.430 | \ul0.415 |
|  | 96 | 28.4% | 0.154 | 0.202 | 0.280 | \ul0.215 | \ul0.179 | 0.239 | 0.244 | 0.317 | 0.187 | 0.234 | 0.187 | 0.234 |
|  | 192 | 23.3% | 0.205 | 0.249 | 0.314 | \ul0.264 | \ul0.234 | 0.296 | 0.320 | 0.380 | 0.235 | 0.272 | 0.235 | 0.272 |
|  | 336 | 19.8% | 0.262 | 0.289 | 0.329 | \ul0.293 | 0.304 | 0.348 | 0.424 | 0.452 | 0.289 | 0.308 | \ul0.287 | 0.307 |
| Weather | 720 | 18.4% | 0.344 | 0.342 | 0.382 | \ul0.370 | 0.400 | 0.404 | 0.604 | 0.553 | \ul0.359 | 0.352 | 0.361 | 0.353 |
| $1^{\text{st}}$ ($2^{\text{nd}}$) Count | 24 (8) | 9 (4) | 7 (24) | 0 (1) | 0 (1) | 0 (2) |  |  |  |  |  |  |  |  |

## Appendix F Metric Illustration

We use mean square error (MSE) and mean absolute error (MAE) as our metrics for evaluation of all forecasting models. Then calculation of MSE and MAE can be described as:

$$ $\displaystyle\text{MSE}=\frac{1}{H}\sum_{i=L+1}^{L+H}(\hat{Y}_{i}-Y_{i})^{2},$ (22) $$

$$ $\displaystyle\text{MAE}=\frac{1}{H}\sum_{i=L+1}^{L+H}\left|\hat{Y}_{i}-Y_{i} \right|,$ (23) $$

where $\hat{Y}$ is predicted vector with $H$ future values, while $Y$ is the ground truth.

## Appendix G Algorithm of TFPS

We provide the pseudo-code of TFPS in Algorithm 1.

Figure: Algorithm 1 Time-Frequency Pattern-Specific architecture - Overall Architecture.

## Appendix H Broader Impact

Real-world applications. TFPS addresses the crucial challenge of time series forecasting, which is a valuable and urgent demand in extensive applications. Our method achieves consistent state-of-the-art performance in four real-world applications: electricity, weather, exchange rate, illness. Researchers in these fields stand to benefit significantly from the enhanced forecasting capabilities of TFPS. We believe that improved time series forecasting holds the potential to empower decision-making and proactively manage risks in a wide array of societal domains.

Academic research. TFPS draws inspiration from classical time series analysis and stochastic process theory, contributing to the field by introducing a novel framework with the assistance pattern recognition. This innovative architecture and its associated methodologies represent significant advancements in the field of time series forecasting, enhancing the model’s ability to address distribution shifts and complex patterns effectively.

Model Robustness. Extensive experimentation with TFPS reveals robust performance without exceptional failure cases. Notably, TFPS exhibits impressive results and maintains robustness in datasets with distribution shifts. The pattern identifier structure within TFPS groups the time series into distinct patterns and adopts a mixture of pattern experts for further prediction, thereby alleviating prediction difficulties. However, it is essential to note that, like any model, TFPS may face challenges when dealing with unpredictable patterns, where predictability is inherently limited. Understanding these nuances is crucial for appropriately applying and interpreting TFPS’s outcomes.

Our work only focuses on the scientific problem, so there is no potential ethical risk.

## Appendix I Limitations

Though TFPS demonstrates promising performance on the benchmark dataset, there are still some limitations of this method.
First, the patch length is primarily chosen heuristically, and the current design struggles with handling indivisible lengths or multi-period characteristics in time series. While this approach works well in experiments, it lacks generalizability for real-world applications.
Second, the real-world time series data undergo expansion, implying that the new patterns continually emerge over time, such as an epidemic or outbreak that had not occurred before.
Therefore, future work will focus on developing a more flexible and automatic patch length selection mechanism, as well as an extensible solution to address these evolving distribution shifts.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the PI module, the number of experts K is currently adjusted manually through parameter tuning. Is it possible to directly relate the value of K to the number of patterns inherent in the time series data itself?
2. On line 317, there appear to be two identical objectives listed, which seems to be a typographical error.
soundness: 3
strengths: The authors conducted a detailed experimental analysis and identified that different time series exhibit unique patterns. Traditional models typically employ a unified modeling approach for these varying patterns, which can limit their effectiveness. To address this issue, the authors propose a Mixture of Experts (MoE) approach that enables tailored modeling for different patterns. Additionally, they decompose the time series data into two dimensions: time and frequency, using a dual encoder to model both types of information separately. To ensure that the MoE effectively learns distinct patterns, the authors introduce a novel gating mechanism. This mechanism leverages subspace affinity to allocate sequences with different patterns to the appropriate experts.
summary: The paper innovatively addresses the diversity of time series patterns by introducing a Mixture of Experts (MoE) approach for decoupled modeling. It leverages a unique PI module as the gating function to route different patterns to the appropriate experts. The experimental results convincingly demonstrate the superiority of this method, complemented by comprehensive ablation studies validating the effectiveness of each component. The writing is clear, with the authors articulating the problem and solution effectively. However, the design of the expert models appears somewhat simplistic, and the decoupling of time and frequency domains is already a well-established approach in time series analysis, making it less of an innovative contribution. Overall, the method shows sufficient technical depth and originality, despite some areas that could benefit from further enhancement.
weaknesses: The decoupling of time-domain and frequency-domain information, as discussed in the paper, is already a common approach in the field of time series analysis and cannot be considered a novel contribution. Additionally, regarding the MoE (Mixture of Experts) section, the use of multiple MLPs seems more akin to employing multiple output predictors with weighted combinations rather than a true MoE structure. Lastly, with respect to the normalization technique, would the authors consider adding experimental results comparing their approach to REVIN [1]? REVIN has shown strong performance across various time series forecasting tasks and could provide a valuable benchmark.


[1] Kim T, Kim J, Tae Y, et al. Reversible instance normalization for accurate time-series forecasting against distribution shift[C]//International Conference on Learning Representations. 2021.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: see weakness
soundness: 3
strengths: 1. The paper is well written and easy to follow.
2. The paper is well motivated.
3. The method shows promising results compared to the baselines.
summary: The paper addresses the challenge of distributional shift in time series data by proposing a novel framework called the Time Frequency Pattern-Specific (TFPS) architecture. This method is designed to effectively model complex patterns in time series, enhancing the model's ability to manage distributional shifts more robustly.
weaknesses: 1. The number of subspace variables $k$, does not appear to be clearly specified in the paper. Given that these variables are essential for modeling the patterns of patches, it is crucial to understand how to set an appropriate value for $k$. The authors should provide further discussion on this aspect. Additionally, an ablation study examining the impact of different values of $k$ would be beneficial.

2. The paper doesn't specify how the subspace variables are initialized. I think the initialization of subspace variables is crucial to achieve optimal performance.

3. Since the subspace variables is utilized for clustering. It is better to provide the relevant visualization (e.g., tsne) to show the effectiveness.

4. Some important information, such as the number of MoPE, the value of hyper-parameters $\alpha$, $\beta$ used for the results in the Table 2 are not clearly specified.

5. What do the notations $K_t$ and $K_f$ mean? The paper doesn't provide the clear definition.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Do you have ablation study on the hyper parameters in the loss function? Which ranges did you try and how sensitive are the results to the particular choice? Also, did you have to balance the contribution of the three terms in the loss?
soundness: 3
strengths: The paper is well written and relatively easy to follow. The proposed approach is interesting and explores time and frequency aspect of time series prediction. Authors conduct extensive experiments on real world datasets showing that the purposed approach outperforms a number of baselines.
summary: The paper proposes a pattern-based mixture of experts model for time series prediction. In this approach time series patches are first extracted and embedded into a sequence of "tokens". This sequence is then encoded with time and frequency encoders and mined for representative patterns that can aid prediction via a mixture of experts approach.
weaknesses: I have several concerns about this method. First, it is quite complex, the loss in Eq 12 has three terms and both L_{PI_t} and L_{PI_f} have three tunable hyper parameters \alpha, \beta and \nu (Eq 5, 6 and 9). So there are at least six hyper parameters to tune in addition to balancing the three loss terms that likely requires more hyper parameters. There is nothing in the experimental section on these hyper parameters and no ablation study is conducted. I've worked with these datasets myself, they are quite small and easy to overfit to which raises a major concern when one has to tune 6+ hyper parameters just in the loss function. Second, the proposed mixture of experts in Eq 10 and 11 looks nearly identical to self attention with the exception of the top-k selection. So I suspect that this method will, despite significant added complexity, have a similar representation power to simply stacking attention layers over patches, as all operations including subspace bases affinity are nearly identical to appropriately parametrized self attention.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to weeknesses.
soundness: 2
strengths: 1. Distribution shifts often lead to changes in frequency-domain information, making it a reasonable and effective approach to model time series from both time-domain and frequency-domain perspectives.
2. The main contribution of this paper is to utilize subspace clustering to detect concept drift between multiple subspaces and model them separately, thereby achieving more accurate and adaptive modeling. 
3. Extensive experiments demonstrate that the proposed method outperforms other state-of-the-art approaches across various time series datasets.
summary: Real-world time series often exhibit varying degrees of distribution shift, posing challenges for time series forecasting. This paper employs subspace clustering from both time-domain and frequency-domain perspectives to identify different patterns within data segments. It utilizes multiple experts to specifically model temporal patterns. Extensive experiments conducted on eight real-world time series datasets demonstrate that the proposed method outperforms other state-of-the-art approaches in terms of predictive performance.
weaknesses: 1. The frequency-domain encoder part of the dual-domain encoder implements a simple feedforward layer by discarding the imaginary part. This leads to a loss of information in the frequency domain. Does this contradict the original intention of using the dual-domain encoder to provide a comprehensive representation of the time series? Considering that the time-domain encoder directly utilizes the PatchTST encoder, the frequency-domain part could also use an existing frequency-domain encoder.
2. The Pattern Identifter module employs multiple hyperparameters  $\alpha、\beta$ to mix different components of the loss function. Given the importance of the loss function for experimental results, it is necessary to include ablation experiments for these hyperparameters.
3. In the final loss function, is it reasonable to simply add $\mathcal{L}_{MSE}、\mathcal{L}_{PI_t}、\mathcal{L}_{PI_f}$? Given that the severity of distribution shifts varies across different datasets, and that the importance of frequency-domain information is relatively low in such datasets, the model should be allowed to learn the appropriate weights for both components on its own.
4. It is noted that in the experiments, the sequence length (seq_len) was uniformly set to 96, which undermines the performance of other baselines, as FITS[1], PatchTST[2], and DLinear[3] have indicated that their methods perform better with longer input sequences. It is suggested to provide experimental results comparing seq_len as a hyperparameter rather than fixing it at 96.
5. It is noted that multiple experts are used in both the time domain and frequency domain, which will increase the runtime of the model. It is suggested to compare the runtime with other methods.

[1]Xu Z, Zeng A, Xu Q. FITS: Modeling Time Series with 10 k  Parameters[C]. In ICLR.

[2]Nie Y, Nguyen N H, Sinthong P, et al. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers[C]. In ICLR.

[3]Zeng A, Chen M, Zhang L, et al. Are transformers effective for time series forecasting? In AAAI.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) In Figure 1, the authors posit that a combined time and frequency domain perspective provides a more comprehensive view of data shifts. However, the provided heatmaps do not clearly support this claim. For instance, in Figure 1(a), the time-domain heatmap highlights significant MMD values for patch #10, while the actual drift begins at patch #9. Similarly, the frequency-domain heatmap includes patches #7 and #8 in the drift region, which is inaccurate. A similar discrepancy is observed in Figure 1(b), where the differences between patches 5, 6, 7, 8, and subsequent patches are not readily apparent. Consequently, this figure does not provide compelling evidence for the superiority of the combined domain approach in detecting shifts.

(2) Based on Figure 1, it appears that the sudden drift is detected due to the data's relatively consistent cyclical patterns across patches. How would the method perform if the data lacks such clear cyclical properties? Would the entire sequence of patches be incorrectly identified as a drift, even if the underlying data distribution changes gradually?

(3) Although this research draws comparisons to traditional time-domain and frequency-domain techniques, a more rigorous evaluation would involve comparisons with MoE-based methods. A list of relevant works is provided below.

* Dish-ts: a general paradigm for alleviating distribution shift in time series forecasting. (AAAI-23) .
* Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift. (ICLR 2022).
* OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling. (NeurIPS 2023).
* Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift. (KDD '24).
* Addressing Distribution Shift in Time Series Forecasting with Instance Normalization Flows. arXiv preprint arXiv:2401.16777.
* MixMamba: Time series modeling with adaptive expertise. (Inf. Fusion 2024).
* Mixture-of-Linear-Experts for Long-term Time Series Forecasting. (AISTATS 2024).

(4) Please elucidate the specific contributions of the PI the gate network to TFPS. Why both components are essential for the system’s performance would be beneficial. Furthermore, conducting an ablation study, which involves comparing the model’s performance with and without the separate PI module, would provide empirical validation of its significance.

(5) A rigorous computational complexity analysis would provide valuable insights into the trade-offs between model complexity and performance gains.

(6) I suggest that authors: Provide clear definitions for q and d in D. Explain the rationale behind controlling column size using Equation (3). Clarify the purpose of the second constraint in Equation (4). Include a theoretical justification for how these constraints improve representation learning. Conduct an ablation study comparing model performance with and without these constraints.

(7) As noted in Shazer et al.’s work on Mixture-of-Experts, gating networks often exhibit a tendency to select the same k experts for routing patches. To mitigate this issue and encourage diversity among the expert modules, regularization and load balancing techniques are typically employed. However, it remains unclear how the proposed model addresses this specific challenge and promotes diversity among its expert modules.

(8) The datasets employed in this study are relatively low-dimensional. It is recommended to conduct experiments on high-dimensional time series datasets, such as the Traffic dataset. Additionally, a discussion on potential scalability challenges and necessary modifications to the TFPS architecture for efficient handling of high-dimensional data would be valuable.

(9) Given that the Patch-TST module used as encoder in this work already incorporates an ReVIN layer to mitigate distribution shifts in time series data. I am wondering if authors consider this point and how it affect the process of PI?
soundness: 2
strengths: (1) Utilizing both time and frequency domain information provides a comprehensive perspective on data, capturing both trend and periodic patterns, which is essential for handling complex time series.
(2) The use of PI and MoPE allows TFPS to handle distribution shifts by learning and adapting to specific patterns within different data segments.
(3) Extensive experiments on real-world datasets demonstrate the effectiveness and competitiveness of TFPS compared to existing methods.
summary: This paper proposes a new architecture, Time-Frequency Pattern-Specific (TFPS), designed for time series forecasting, particularly under challenging conditions involving concept drift within the data. The key idea revolves around addressing the limitations of conventional uniform distribution modeling (UDM) by recognizing that different segments or "patches" within a time series exhibit distinct patterns and distributional shifts. These shifts may stem from factors such as sudden events, gradual trend changes, or different operating modes, making accurate forecasting difficult for models trained on a single global pattern. TFPS tackles this problem through a multi-faceted approach. First, it employs a Dual-Domain Encoder (DDE) that analyzes both time and frequency domain features, enabling a richer understanding of temporal dynamics and their potential shifts. Second, a Pattern Identifier (PI) utilizes subspace clustering to dynamically group patches exhibiting similar patterns. Finally, a Mixture of Pattern Experts (MoPE) leverages multiple specialized expert models, each trained on a specific pattern identified by the PI. This allows TFPS to adapt its predictions based on the identified pattern of a given patch, leading to improved accuracy.
weaknesses: (1) The authors acknowledge that the patch length is currently determined heuristically, which might limit generalizability and performance in cases with indivisible lengths or multi-period characteristics.
(2) The model's ability to adapt to entirely new patterns, such as those arising from unforeseen events, is not explicitly addressed and remains an open question.
(3) While not explicitly mentioned, the use of multiple expert models and the dual-domain encoding approach could increase computational complexity compared to simpler models. This aspect would require further analysis and optimization for practical implementation in real-time applications.

Meta-review: This paper presents a pattern-based mixture of experts model for time series prediction. Reviewers agreed that this paper is well written and easy to follow, and the paper aims to address a critical problem, i.e., patch-level distributional shift in time series forecasting. Meanwhile, reviewers raised many concerns about novelty, technical details, comparisons with baselines, missing related work, etc. During the post-rebuttal discussions, reviewers still believed that some major issues still remain. For instance, the paper fails to exhibit a significant efficiency advantage over SOTA baselines. Overall, this work in its current version is not ready for publication at ICLR.

Decision: Reject