PPT: Patch Order Do Matters In Time Series Pretext Task
Authors: Jaeho Kim
Kwangryeol Park
Sukmin Yun
Seulki Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Time Series Classification
Self-Supervised Learning
Pretext Task
Primary area: learning on time series and dynamical systems
TL;DR: Patch order aware self-supervised learning methodology for time series classification

Abstract: Recently, patch-based models have been widely discussed in time series analysis. However, existing pretext tasks for patch-based learning, such as masking, may not capture essential time and channel-wise patch interdependencies in time series data, presumed to result in subpar model performance. In this work, we introduce *Patch order-aware Pretext Task (PPT)*, a new self-supervised patch order learning pretext task for time series classification. PPT exploits the intrinsic sequential order information among patches across time and channel dimensions of time series data, where model training is aided by channel-wise patch permutations. The permutation disrupts patch order consistency across time and channel dimensions with controlled intensity to provide supervisory signals for learning time series order characteristics. To this end, we propose two patch order-aware learning methods: patch order consistency learning, which quantifies patch order correctness, and contrastive learning, which distinguishes weakly permuted patch sequences from strongly permuted ones. With patch order learning, we observe enhanced model performance, e.g., improving up to 7% accuracy for the supervised cardiogram task and outperforming mask-based learning by 5% in the self-supervised human activity recognition task. We also propose ACF-CoS, an evaluation metric that measures the *importance of orderness* for time series datasets, which enables pre-examination of the efficacy of PPT in model training.

Published as a conference paper at ICLR 2025
PPT: PATCH ORDER Do MATTERS IN TIME SERIES PRETEXT TASK
Jaeho Kim Kwangryeol Park Sukmin Yun?
Seulki Lee!
Ulsan National Institute of Science and Technology (UNIST)
Hanyang University ERICA
(k jh36900, pkr7098, seulki.. lee) @eniet..t....... .a..stttc...... sukmiayun@hanyange .r.
ABSTRACT
Recently, patch-based models have been widely discussed in time series analysis. However, existing pretext tasks for patch-based learning, such as masking, may not capture essential time and channel-wise patch interdependencies in time series data, presumed to result in subpar model performance. In this work, we introduce Patch order-aware Pretext Task (PPT), a new self-supervised patch order learning pretext task for time series classification. PPT exploits the intrinsic sequential order information between patches in time and channel dimensions of time series data, where model training aided by channel-wise patch permutations. The per- mutation disrupts patch order consistency across time and channel dimensions with controlled intensity provide supervisory signals for learning time series order characteristics. To this end, we propose two patch order-aware learning methods: patch order consistency learning, which quantifies patch order correctness, and contrastive learning, which distinguishes weakly permuted patch sequences from strongly permuted ones. With patch order learning. we observe enhanced model performance, e.g., improving up to 7% accuracy for the supervised cardiogram task and outperforming mask-based learning by 5% in the self-supervised human activity recognition task. We also propose ACF-CoS, an evaluation metric that measures the importance of orderness for time series datasets, which enables pre-examination of the efficacy of PPT in model training.
(A) Time Order Importance
(B) Channel Order Importance
Fold Wings
Spread Wings
Original Order
Permuted Order
(2)
(1)
Accelero meter
Original
Aligned Peaks x
Non-Aligned Peaks !?
Force Sensor
Reversed Time Order
Highly Similar Signal Model is confused
Reversed
Force
Sensor
Figure 1: Motivation. Why is patch order important in time series? (A) Time Order Importance. The aircraft handling signal task (Song et al., 2011) shows how actions can be misinterpreted if their time order ignored. For instance. "Fold Wings" and "Spread Wings" are distinct actions, but their time signals may appear similar when reversed, potentially causing confusion model training and highlighting the need for time order awareness classification. (B) Channel Order (Alignment) Importance. The smart shoe task (Kim et al., 2023) shows how permuting patch orders in one channel (accelerometer) can disrupt overall channel alignment, Peaks that were initially aligned across all channels during stepping action (Green shaded) shift out of alignment (Red shaded) when the top patches from the accelerometer are permuted, misaligning signals. Our proposed method, PPT. supervises this information during model training to address these challenges.
"Equal Contribution.
1 "ooresponniig author

---

Published as a conference paper at ICLR 2025
(1) t-SNE Visualization of "Selff-uperr Superviddd Learning
(2) Attribution Visualization
Time Signal
Patch Index
Normalized Attribution
(A) Recon.
Complementary 88.16%
(C) (Ours)
(A) Cross-Entropy (CE)
100
700
100
630
130
(B) CE PPT (Ours)
390
88.43%
92.33%
Figure 2: (1) I-SNE (Van der Maaten & Hinton, 2008) visualization of patches from the smart insole task (Kim al., 2023). All figures are derived from self-supervised linear probing using PatchTST (Nie et al., 2022). color-coded by patch index. (A) uses a mask-and
pretext task (Nie et al., 2022). (B) implements the Complementary patch contrastive learning (CL.) proposed by (Lee et al., 2023). (C) uses PPT (Patch order aware Pretext Task), where we observe a clear alignment of patch orders in latent space. (2) Feature
ablation (Kokhlikyan et al., 2020) to find patch importance in supervised training, Results on the ShapesAll
task (Dau et al., 2019) reveals high importance scores key time segments when using PPT compared to using cross-entropy (CE) alone. The details of the results are shown in Appendix A.
INTRODUCTION
Recent advances in time series have seen a notable trend toward the adoption of patch-based method- ologies (Cao et al., 2023; Chen et al., 2023; Lee et al., 2023; Wang et al., 2023). These methods segment the time series into "patches," each consisting of consecutive data points, moving away from the traditional "temporal " or "point-wise" approaches that model individual time points separately. Although the shift to using patches has generally improved model performance, current approaches used in patch-based models often struggle to effectively incorporate order information between time series patches. Positional encoding (Vaswani et al., 2017) and strategies such as musk-and-reconstruet (Nie et al., 2022) or patch complementary contrastive learning (CL) (Lee et al., 2023) still fail to fully capture the chronological and channel-specific order alignment essential in time series (Fig. 2-(1)). Modeling this order information is important as it leverages the unique characteristics of time series, specifically, their inherent sequential nature. By improving order awareness across patches, model can improve its performance, ensuring that the temporal dynamics and channel dependencies characteristic of time series are effectively captured and utilized.
To address this gap, we propose Patch order-aware Pretext Task (PPT), the first order learning pretext task for patch-based time series model, which creates supervising signals from the inherent order relationship between the time series patches. This order awareness across time and channel is achieved through channel-wise permutation of patches, where the "channel" refers to a specific feature (e.g., sensor) in time series. The motivation of PPT comes from critical observation: The sequence and arrangement of time series patches, spanning both the time and channel dimensions, represent natural inductive bias that can be exploited in learning. This awareness is crucial, as time series data exhibit complex interdependencies that are necessary for accurate classifications. For instance, as shown in (Fig. 2-(2)), visualization of the attribution scores for each patch demonstrates that incorporating PPT effectively identifies inflection points, while using cross-entropy (CE) as a standalone loss function fails to highlight meaningful time segments. Failure to account for these dependencies can lead to suboptimal model performance, diminishing the model's ability to understand the underlying data structure, and reducing accuracy and reliability tasks where order is crucial. We further illustrate our motivations with real-world examples in Fig. 1.
pretext task a "pre-designed tasks for a network to solve, where the supervising signal is generated from the data itself" (Jing & Tian, 2020). As pretext task, PPT is adaptable to self-supervised learning (SSL) and supervised learning. In SSL, PPT operates independently as a standalone loss function, while in supervised training. it supplements the standard cross-entropy (CE) minimization as an additional learning task without relying on labeled information. PPT is a plug-in in method for patch- based time series models that encode individual patches into latent representations (e.g.. PStchTST, PITS). By integrating PPT, the model gains insight into the relationships of structural order between patches, encompassing both the time and channel dimensions, resulting in consistent improvements in model performance. Furthermore, we introduce ACF-CoS (a (Autocorrelation Function with Cosine Similarity), a metric based on autocorrelation function, to quantify the "orderness" of time series. Briefly, orderness is the ability of a discriminative function to distinguish sequence from its permuted
httpps: //gi thub. com 1
2

---

Published as a conference paper at ICLR 2025
set, emphasizing sequence order and structural dependency (Definition 3.1). An extensive number of experiments demonstrate a positive correlation between the ACF-COS score and the performance enhancement brought by PPT, underlining its significance in exploiting structural data characteristics for patch-based time series. The key contributions of our work are summarized as follows.
We introduce Patch order-aware Pretext Task (PPT), the first order-aware pretext task for patch- based time series models. As a self-supervised learning method, we apply and assess PPT on two state-of-the-an patch-based time series models, i.e., Transformer-based PatchTST (Nie et al., 2022) and linear-based PITS (Lee et al., 2023), demonstrating PPT's effectiveness.
We propose two patch order-aware learning methods of PPT, i.c., patch order consistency and contrastive learning, which exploits inherent structural orders between patch sequences across time and channel dimensions, applicable to both self-supervised and supervised training.
We propose ACF-COS (Autocorrelation Function with Cosine Similarity), a metric for gauging the "orderness" of time series. It enables us to pre-examine whether a time series task could benefit from patch order learning, enabling targeted applications of PPT where the order is critical.
RELATED WORKS
Self-Supervised Learning in Time Series. TS-TCC (Eldele et al., 2021) a self-supervised learning (SSL) method designed for won-patch-basee time series, generating two views of the time series data using weak (jittering and scaling) and strong (permuting and jittering) augmentations. Then, an autoregressive model is used to construct context vector for temporal contrast between the two samples. Based on that, some supi-super (i.e. CA-TCC (Eldele et al., 2023)) and graph-structured learning (i.e. TS-GAC (Wang al., 2023)) methods have followed. Inspired by TS-TCC, we propose PPT that introduces a channel-wise patch permutation approach with varying strengths determined by the permutation frequency, termed 'weak' and 'strong' permutation. While the main objective of TS-TCC is to bring the weak and strong representations closer, the objective of PPT is the opposite, as it widens the distance between the weak and strong while placing the weak closer to the non-permuted 'original' set (Sec. 3.2). Moreover, unlike TS-TCC's random segment permutations (Um et al., 2017) made on the fly, PPT provides control over the strength of permutation, which enables the proposed time and channel-wis consistency and contrastive learning (Sec. 3.3). In addition, PPT is not only efficient due to its use of the permutation bank (i.e. only the permuted indices are stored) but also effective because it continuously exposes the model to a diverse array of permuted samples. Thus, PPT significantly diverges from TS-TCC in its focus for contrasting, making it unique and effective patch-based pretext task for time series.
Patch-Base Based Representation of Time Series. Patch- based methods have gained significant popular- ity across various time series applications, including time-series foundation models (Jin et al., 2023; Zhou et al., 2024; Gao et al., 2024), forecasting (Chen et al., 2023), and classification (Cheng et al., 2023). TimeMAE (Cheng et al., 2023) have integrated the Masked Autoencoder (He al., 2022) archi- tecture for time series, optimizing the reconstruction loss of randomly masked patches. PatchTST (Nie model, utilizes mask reconstruction
al., 2022), Transformer-basse method, establishing itself as strong baseline for various time series applications. TS-GAC (Wang al., 2023) is a graph- model where patches are represented as nodes and their correlations as edges, performing contrastive learning in both node and graph levels. PITS (Lee et al., 2023), an MI.P-based model, enhances supervised training by combining contrastive learning and masked modeling of patches. As a simple yet effective self-supervised learning method, PPT is applicable to many of those patch-based time series models.
self-supervised pretraining
Quantifying Time Series. Time series exhibit substantial variations in length, number of channels, and dynamics. As such, quantifying these series into a single metric provides an interpretable tool for understanding both the data and the behavior of models applied to it. Shannon entropy (SE) (Lin, 1991) and its temporal extension, Permutation entropy (PE) (Bandt & Pompe, 2002), are foundational metrics used measure the uncertainty and pattern frequency in time series. However, SE and PE may not adequately capture the intrinsic orderness of data sequences. Addressing this gap, we introduce the ACF-CoS metric (Sec. 4), which quantifies the orderness of time series by computing the cosine similarity between autocorrelation functions of the original and its permuted (shuffled) set.
Extended Related Works. Appendix B discusses related works for order-aware learning tasks in domains other than time series and SSL methodologies for time series in general.
3

---

Published as a conference paper at ICLR 2025
1. Patch
2. Channel-Wise
3-(A), Consistency Learning
Representations
Patch Permutation
OideeOrder Consistency
Channel Order Consistency
Time (T)
Channels (0)
Original
Strong
Original
Original
Strong
3-(B). Contrastive Learning
model f
Permute Bank
Time-Order Contrastive
Channel-Oree Contrastive
Original
Strong
Weal
Original
Permute
Weak
Bank
Patch Representations
Strong
Strong
Context Vector
Positive Pair Negative Pair
Pseudo label
Autoregressive Function
Figure 3: PPT employs three primary components: Patch Representations. time series patches encoded into latent patch representations; (2) Channel- Wise Patch Permutation. permuting patch orders channel- with permutation intensity governed by weak or strong criteria; (3-A) Consistency Learning.. an autoregressive approach for recognizing patch order at both time and channel levels, paired with (3-B) Contrastive Learning. to distinguish between original/wcakly permuted (positive) and strongly permuted (negative) patches of sequenees. For the negatives in contrastive learning, we use strongly permuted sequences from all channels. Best in color.
PPT: PATCH ORDER AWARE PRETEXT TASK FOR TIME SERIES CLASSIFICATION
Preliminaries. Let X E R be a raw time series, where T and D denote the time sequence length and the number of channels (features), respectively. For each channel, we derive N
non-overlapping patches, each of size s. The latent representation for the i-th patch from the d-th channel in X is denoted by ((.d) E Rdh, where da the dimensionality of the embedding, obtained via model f()). The sequence of patches in time order can be obtained for each channel d [1 DJ and is represented as P (c.d) = (P(1.)). ....: P P.N)))
) E Rd Additionally, sequence of patches in
channel order can be obtained for all i E [1: : N] and represented as P (t..) = (P(..))- P(. E Rd DD While time order P ((.df) captures the temporal evolution within individual channels, the channel order sequence P(a.)) represents snapshot of patterns that appear together across all channels at single time point, enabling the model to learn both temporal dynamics and concurrent cross- channel relationships. The objective of PPT to exploit these order relationships in the time and channel dimensions of the patches derived solely from X. Here, define patch order as:
Definition 3.1 (Order) Let Psime and Pehamnel be domains of time and channel order patch sequences, respectively. We define discriminative function g(-) such that g PP + R. and permute operation : P, where P = Pamee U Pehhameet An order exists in P, where is an element of P, if the discriminative function g' can distinguish between P and its permuted II(P).
Remark 3.1 The discriminatory power of function g suggests a non-random order or sequence dependency in P, which is disrupted through IT(P).
3.1 STRUCTURE OVERVIEW
We provide an overview of PPT Fig. 3, outlining a three-step process. (1) Time series are reordered into sequences of patches, with each patch transformed into patch embeddings through encoder (2) utilizing a buulbbilt permutation bank, patches undergo efficient channel-wist permutations within each channel, resulting in two permuted orders of patch sequences-termed "Weak" and "Strong" based on permutation intensity (Sec. 3.2). (3) These sequences are then applied in consistency and contrastive learning (Sec. 3.3), where consistency learning verifies the correctness of the order of the patch sequence, and contrastive learning pairs the Original and Weak sequences as positives against the Strong sequence negative. The learning occurs in both the time and channel dimensions.
paper,
training, ACF-COS
caleulation.
4

---

Published as a conference paper at ICLR 2025
3.2 CHANNEL- WISE PATCH PERMUTATION STRATEGY
Permutation Strategy. We propose a straightforward yet effective patch permutation strategy wherein patches are permuted within each channel. Permuting within each and every channel is sufficient to break the patch consistency in both time and channel dimensions. The permutation operator II, ((..d) P(jzid) selects and permutes the order of two distinct patches, P(i.d) and P( within the same channel d. The hyperparameter regulates the frequency of permutations performed for each channel. For instance, 7 = means that a single pair of patches within each channel will be permuted. We propose and investigate three selection methods that determine which patch pairs to be permuted, i.e., the vicinity, distant, and random pairs (See Appendix C). Our experiment shows that the random pair selection is most effective, as it encompasses both the vicinity and distance pairs. Consequently, we employ random pair selection for our patch permutations.
Furthermore, define the permuted sequence of patches as "Weak" and "Strong" based on the magnitude y. Selecting a small y leads to the permutation of only a few pairs of patches within channel, maintaining most of the original patch sequence order. In contrast, high value of substantially disrupts the order of the original sequence in both the time and channel. We utilize "Weak, "Strong " and the un-permuted "Original" sequence of patches in the training of PPT.
Permutation Time Complexity. The permutation operation efficiently managed using a pre-built bank of permuted indices, with each index randomly sampled to maintain diversity during minibatch training. In theory, permuting patch positions with respect to the number of patches N only involves reordering them, which has a time complexity of ((V) for each permutation. In practice, permuting a batch of 512 data samples takes merely 0.018 seconds on modern GPU hardware (Appendix F). The time complexity and the pseudo-algorithm for the permutation bank are in Appendix
3.3 CONSISTENCY AND CONTRASTIVE LEARNING
Time and Channel- Patch Consistency. PPT learns to differentiate between the "Original" and "Strong" patch sequences in both time and channel dimensions. It is referred to as learning "consistency" the order of patches. The motivation for this learning objective is based on the observations (see Fig. 1), where intrinsic order exists among patches, and any deviation from this order may result in patch sequences that appear to be inconsistent and/or errant. To quantify the consistency, autoregressive models Grime and 9channe1 summarize the patch sequences P ((L.)iE(I::]]
and P (( into context vectors CiE[1:N] Rd and Cae|1:D| E Ed, respectively. Here, we employ single-layer uni-directional LSTMs that process the patch sequences. The final cell state of each LSTM serves as the context vector, effectively summarizing the patch sequences. These LSTM modules are trained end-to-end with the backbone model (details in Appendix F). The context vectors are then utilized in pseudo-label training, where the learning task is to discriminate between the context vectors of the "Original (label 1)" and "Strongly (label 0)" permuted sequences. Based on the binary cross entropy, we define the consistency loss for time and channel-order patches as:
1 m7
CCS Time or Caannel
CCS
[ya log((s) + (1 - yi) log(1 - ]+))
1=1
where CCS with C]E[I:N] and CCS either in time and channel, y and ÿ are the pseudo-label and predicted label, respectively. The total
Channel with CdE(L:D). m is the number of patch sequence samples
consistency loss Les is then the sum of the losses from both dimensions, gmum Time Chanzel
CCS
Time and Channel-Order Patch Contrasting. The use of the consistency loss, L empowers the model to discern between the original and permuted sequences of patches in both time and channel dimensions. To further refine the model's ability to detect varying degrees of permutation, we introduce an InfoNCE loss (Oord et al., 2018), denoted as ger to detect the degree of permutation variations. Here, the "Original" serves as the anchor, the "Weak" as positives, and the "Strong" as negatives, Autoregressive models are employed for contrastive learning, taking context vectors to distinguish between degrees of permutation. We define the contrastive loss for time order 2 in Eq. (2), where is the number of channels, c is the context vectors, T represents the temperature hyperparameter, and sim denotes cosine similarity. Similarly, the contrastive loss for channel order CCT Chhannel is defined as Eq. (2) by replacing D with the number of patches N. Then, the total contrastive loss becomes the sum of the two losses as LCT LCT + LCT Channeel measured in both
5

---

Published as a conference paper at ICLR 2025
A) Patch Embeddings in Time-Level
B) Patch Embeddings in Channel-Level
PatchTST: 78.6%
PatchTST PPT: 2.99)
PatchTST:
PatchTST 5.8%)
Index: 1-4
Indexc 13 16
Index: 27
Accelerometer Sensers
Force Sensors
Figure 4: t-SNE (Van der Maaten Hinton, 2008) visualization patch embeddings from supervised PatchTST (Nie et al., 2022). (A) Time-Level Patch from the ECG data (Bousseljot et al., 1995) visualize temporal patch embeddings with patch index from i = 1 4 (Green), 13 16 (Yellow), (, and 27 - 30 (Purple). (B) Channel-T Level Patch from the smart insole task (Kim et al., 2023) shows the distinct embeddings for the accelerometer channels (Blue) and force sensor channels (Red). We observe clear distinction between time and channel patches with the use PPT, with performance increases detailed in Tab.
time and channel dimensions.
1 D
Σ d=1
D log
cxp(sim( (c exp(sim(c,) original c. //T) + > exp (sim((sss C,
original Cy )/T)
(2)
Time
original strony
Ck
Although consistency and contrastive learning losses are presented here for a multivariate time series, they can be applied to univariate time series by removing channel losses CCS
Channel and CCT Channel
3.4 OVERALL Loss SETUP
Supervupe Loss. In self-supervised (SSL.) training, we focus on optimizing the consistency and contrastive terms in Eq. (3). Here, aι and X2 are hyperparameters. However, avoid extensive manual tuning, they are replaced by learnable parameters that dynamically adjust the weight of each loss term, reflecting strategy based on homoscedastic uncertainty for multi-task learning (Kendall et al., 2018; Dong et al., 2024).
Supervised Loss. For supervised training, PPT works specific supervised loss Lr, the cross-entropy loss in the case of classification. For instance, Fig. compares the t-SNE emheddings of patches with and without PPT in supervised setup. The final objective function is Eq. (3), with the hyperparameter À replaced by learnable parameters as in SSL.
extra loss term along with the task-
SSelf-Supervised \ CCS C
and Luperviied = LT + X CCS Sum
Ag CCT
8.um
ACF-COS METRIC: MEASURING THE IMPORTANCE OF ORDERNESS
We propose ACF-CoS (Autocorrelation Function with Cosine Similarity) to measure the importance of orderness in time series data. This score allows
pre-assess the suitability of PPT
xr] given the mean I, the j-th pretext task for a given dataset. For a time series x = [21 .a...... 0T 1] is defined as in Eq. (4), with j as
element a; of the autocorrelation vector lag. We calculate two distinct autocorrelation vectors, a and a', from the original time series x and its
counterpart x', respectively. While the calculation of from the original time series x is straightforward. a' is obtained from the patch-permuted series x' by (1) reshaping into sequence of patches, (2) randomly shuffling those patches, and (3) reassembling them into the final series x'. The ACF-COS score is then computed by taking the cosine similarity between a and a' as:
ACF-CoS(a, a') = 1 - a a' aa/l|aa ||a|||||||||||a aj EE x) - x) D) (x, z)². (4)
The interpretation of ACF-COS is intuitive and aligned with the definition of patch order (Defini- tion 3)... A low ACF-COS score indicates that the two autocorrelation vectors are similar, meaning that permuting the sequences does not significantly alter their autocorrelation, suggesting a weak structural order dependency. In contrast, high ACF-CoS shows that permutation effectively disrupts autocorrelation, indicating strong dependence on the sequence order. Consequently, white noise has an ACF-CoS score close to zero, which means that there is no structural order dependency, whereas the step function shows ACF-COS close to one, which implies a significant dependency. quantify the order characteristics of the data, we treat each channel as an independent sequence and average the ACF-CoS scores of all data samples to obtain the final ACF-CoS score (Tab. 4). To mitigate the impact of patch size variations, we calculate the ACF-CoS scores over different patch sizes.

---

Published as a conference paper at ICLR 2025
Table 1: Self-Supervised Linear Probing Results. We compare PPT (ours) state-of- -ht sulf-supe learning methods. The asterisk (*) denotes patch-based methods. The baselines PatchTST and PITS use supf-supe training methods from their original work. The best scores are highlighted with red bold. The full experimental result, including the MS-HAR task, is given in Appendix J.
Dataset
Models
Accuracy
score
AUROC
AUPRC
Precision
Recall
Mixing-up
74.48 2.95
49.30+-253
7180+
52.22+-2.11
11.66+181
49.36=4.01
SimCLR
74.42=4.38
44.6446.34
48.3446.76
47.83-1707
47.19=8.02
TS2Vee
78.08=2.9
49.07433.00
78.00+3.58
50.95-+3.88
48.9743.52
49.87=2.88
TE-C
77.63=6.18
53,30:::600
82.22-45.31
54.8447.18
56,1443879
56.00=9.07
TS-TCC
75.61=3.49
48.47443.59
73.82++7.40
53,98-t3.77
54.7540.67
49.02=410
SimMTM
31.75=3.33
53.08 +3.68
81.35
58.70::4.05
57.75:44884
51.62=4.17
EMO
TImeMAE"
73.97=2228
42.44:12.07
70.11:1-443
43.25:::.14
42.81++2.37
42.43==2.09
TS-GAC
13.75=1.66
46.42++1.29
75.92-42.30
49.29:4:062
46.04++0.77
1..86=1.69
UnITS
78.37=0.13
29.51:10.53
65.95:12.95
44.84:1:.39
29.47:17.52
33.44=00.4
PatchTST
78.70=0:73
45.81
82.60
55,23::2.11
59,40::::32.
16.35=1311
PatchTST (+PPT)"
81.92=0.58
54.19:12.38
84.74:1
62.51
62.96:12.49
53.41
PITS"
69.63-2.04
43.73-4-1.06
68.84442.39
43.90-H1.00
44.07=40.82
45.68 L.31
PITS (+PPT)"
75.55=2.84
68.63-3330
25.5942.28
45.054:2.66
2..53-2.06
Mixing-up
82.35_0.93
88.82:10057
89 2.222
95.32.1 1.05
01.76.20772
97.23
SimCLR
80.41=1.14
87,89:10663
86.03:13.24
93.8411.77
79,30
98.57
TS2Vee
84.06=2811
89.75
91 36.66.228
96. 2.22
23.82:22.56
96. 96.62=0:95
TF-C
79.58=1.22
87.25
86 26.55.22
93.95:11.50
79.35:10:47
9.791
TS-TCC
19.58=1.18
86.98
79. 1.88
88.05 1,40
80.51:0000
94.59=1.47
SimMTM
80.75-0.29
88.08
88 66:2
94, 144.481177 1.23
79.58:10.31
98.61
PTB
TS-GAC"
TimeMAE"
83.26=1997
78.65-1.43
89.51
86.7840.86
91. 2.:6
92.97:4-085
95.37:10.96
81.7621199
98.90=0023
97.15=1.10
UniTS"
1.02
89.57:00.00
88.98
94.81 +:084
85.51:
94.04=0.23
PatchTST
79.89=1887
87.46-t-1.07
81.36-34.91
88.94-63.50
19.50=+1.54
97.21
(EPPTST(-PPTT)
78.50=1.76
88.67:40095
76.044+6..77
86.79453.5
96.83=0063
PITS
8..61=1.21
89.96:1-0.61
86.53-4-2.46
92.75
85.03:+2.00
95.55 1.32
PITS (+PPT)
86.48
91.24:10.26
91.83
96.26:10:85
85.67.10.61
97. 07.58:0086
Mixing-up
81.76=2.94
39.25
96.46:20.00
82.47-12.W0 2.80
79.99
1893
SimCLR
87.10=0.44
88.72-40.79
98. +-.15
92.69+
09.42+0.84
88.20 0.88
TS2Vec
0.67
89.92:10066
98. 98.35::0.12
93.93--0.10
90.12 00.12:90997
0.44
TF-C
88.23=0.73
92,90+1100 1.00
TS-TCC
87.98=1.22 1.28
88.53
98 08.660220.22
91.86+ 1.03
88, 1:45
$8.36=1.22
SimMTM
82.05=0.33
83.96:1-0.50
97 0.006
88.06:10.69
84.63:40.44
83.63
TimeMAE'
13.44=1.09
1.72
97.81
19.
86, 36.10:::2066
1.577=1.58
TS-GAC
92.32=0.85
92.79:10.84
99.04:1:..88
96.23-1:0.64
92.67:10.65
93.23=0092
UniTS
84.45=0:98
98.27-+0.18
90.03-4-0.95
83.41
PatchTST
88.43=0044
89.87:1:038
98. 08.79:10.11
94,34:1:.40
89.73
90.15=0.43
PatchTST (+PPT)
92.33=0.48
93.67:10.45
99. 99.28:10.10
96.8110044
93.54.10.37
93.90=0056
PITS"
87.23-1.04
87.1840.86
98.41
91.65:10.77
86 86.62=0.88
PITS (+PPT)"
92.07=0.78
92.77:0.63
99.28:10.19
96.87:11.46
92.99 70
92.83=00.5
5 EXPERIMENTS
This section evaluates PPT as pretext task in both self-supervised and supervised learning, demon- strating that enhancing patch order awareness improves model performance. In self-supervised scenarios, employ linear probing, semi-supervised training, and full fine-tuning (Appendix K). In linear probing, the encoder model f is trained, followed by applying logistic regression to the frozen representation. In semi-supervised training, self-supervised training is performed with model f. followed by fine-tuning with a fraction of the dataset and a reduced learning rate (Dong et al., 2024). In the supervised context, we perform an ablation study on the consistency and contrastive learning methods, revealing their individual and combined effects on model performance.
Implementation of PPT. We apply PPT to two representative patch- models: Transformer- based PStchTST (Nie et al., 2022) and linear- based PITS (Lee et al., 2023). We set permutation frequency ? 1, for weak shuffle and ? in length, we adjust the patch size 8 so that each sequence is divided into 10 to 40 non-overlapping patches. The detailed implementations and configurations are Appendix and Appendix
or 40 for strong shuffle. As each dataset differs
Self-Supervised Baselines. We make an extensive comparison to leading self-supervised base- lines such as Te2Vec (Yue et al., 2022), SimCLR (Tang et al., 2020), TF-C (Zhang et al., 2022), TS-TCC (Eldele al., 2021), and SimMTM (Dong et al., 2024), well as patch-based works like TimeMAE (Cheng et al., 2023), TS-GAC (Wang et al., 2023), UniTS (Gao et al., 2024), PatchTTT (Nie al., 2022), and PITS (Lee al., 2023). For PatchTST,
the mask-and-
7

---

Published as a conference paper at ICLR 2025
Table 2: Semi-SSuprvised Results. We perform semi-supervised training on the GL task. Models pre- vre-trained on unlabeled data using supervieee learning. then fine-tuned on 10% and 1% labeled data, Best scores are in red bold. Full fine-tuning and randomly initialized backbone results are in Appendix K.
Fraction
Models
Accuracy
Fl
AUROC
AUPRC
Precision
Recall
Mixing-up
02.85+00.99
90,44+00.91
98.74+-0.26
94.42+1111 1.14
90.59-+0..5
90.69-4-0.00
SimCLR
84.55.10.78
83.60.1
98.47.1..20
9..74.10.83
86.78.10..2
62.56.1..JJ
TS2Vee
88.12+ 1.58
85.51=1.13
96.46-t-0.78
89.37=:1.55 1.87
85 85.86-t00.99
85.97=
TF-C
03.35
82.73-10.448
97.96.1:0.09
03.95:1:0.33
82.10:1:0.59
TS-TCC
93.69 1.05
92.11110.84
99.41
97.76.10.76
93.69.10.38
91.83-10.88
10%
90.06:22.95
91.10:22.54
98.774:0.43
94.65-22.00
91.50:::2.46
90.83-42.66
SimMTM
91.94.10.58
91.35100.53
98.95.1.0.35
95.65.10.62
91.41.10.44
91,40.1..65
PutchTST
91.61:10.82
92.33:10.89
99.35:40.111
97.10:100.47
92.88:1:0.80
92.47:1:075
13.2
1..97
99.50.1 0.09
97.79.10.47
94.74.1
94.27
85.11:1..78
85.67:2..21
98.18:10.443
29.511:2.63
84.61
84.60:1..65
PITS (+PPT)
92.47 1,06
93.32:10.60
99.48:1:.112
07.28
93.17:10.69
93.07
Mixup
84.22.1..17
22.08
0.55
87.48
83.76.1..52
81.53-13.33
SimCLR
62661+11889
47.28
90,88++2.00
66.05
63.15:+9..38
51.63-+2.992
TS2Vec
77.41
75.17
16.17.1.
82.84 1.67
79.04.1 1.10
3..64
TF-C
65.34:2..50
52.8811.998
91.191 1.59
31.15
71.92:14.11
3..65
TS-TCC
85.77 1.08
83.02 1.16
97.82.1.0.25
89.85 1: 1.19
86.31
83.04 1.46
1%
TimeMAR
34.6
96.24410.58
3..35
37.58:13.69
73.57-1..11
SIMMTM
78.44=22.20
79.48 1.95
94.93=400.87
82.75 1.34
80,66:2:.440
19.31
PutchTST
20.55
96.77.L
26.44.1..83
81.50.13.58
81.52.13.58
PatchTST (+PPT)
84.80 1,68
86.92-21.48 1.48
98.08:1: 0.38
90.64:2
86.88:1 1,65
86.75-21.57
72.41:1:2.05
72.81
95.40:1.0.66
75.92.1:3.23
3..8331...77
70.45.14.71
PITS (+PPT)
81.04 1.86
83.71
97.68:10.30
87,26 1.62
81.25 1.45
82.05
reconstruct (Mask) pretext task, and for PITS. we use both Mask and complementary contrastive learning (CL) pretext tasks from their original works to differentiate from the use of PPT as pretext task. We use metrics like Accuracy, F1, AUROC, AUPRC, Precision, and Recall, consistent with previous works (Wang et al., 2024b).
Datasets. (1) EMOPain (Egede al., 2020), from the UEA repository (Bagnall al., 2018), is pain classification task using a surface electromyographic sensor (sEMG), with 30 channels (2) PTB (Bousseljot al., 1995) cardiogram signals (ECG), incorporating 15 channels from 198 users. (3) Gilon (GL) is large-scale smart insole-based human activity recognition (HAR) task (Kim et al., 2023) with 47,647 instances with 14 channels collected from 72 users. (4) Sleep EEG (Kemp et al., 2000) is a univariate brainwave (EEG) with 371,055 instances. (5) MS-HAR (Morris et al., 2014) has six sensors in an armband with 14,201 instances collected from 93 users. The full details of the datasets are in Appendix E. Here, five random seeds used to report the results.
5.1 SELF-SUPERVISED LINEAR PROBING
Setup. In the linear probing setup. PPT is trained self-supervised. optimizing Eq. (3). Strong performance in linear probing indicates robust learned representation. Following self-supervised training, logistic regression is applied to evaluate the representation (Yue et al., 2022). We used official baseline model implementations, making necessary adjustments (e.g., adapting for multivariate series when only univariate support was available). The hyperparameters for the baselines are in Appendix H.
Results. Partial results of three tasks (EMO, PTB, and GL.) are in Tab. with full results in Ap- pendix J. Here, we observe that PPT is an effective self-supervised pretext task, providing improved learning representation compared to leading methods. Specifically, PatchTST with PPT is the best- performing model in EMO and GL, while PITS with PPT performs best in PTB. Furthermore, PPT consistently outperforms or matches the original pretext methods both PatchTST and PITS across all tasks. In particular, PITS with PPT surpasses the original method by five percentage points in AUROC on the PTB task. This performance gain PTB supported by a high ACF-COS score of 0.318, indicating that order is essential in the cardiogram data. These results underscore that learning order information provides strong guiding signals, allowing the model to learn interdependencies between patches from both time and channel dimensions, leading to enhanced model performance.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation Study for Supervised Training. We perform an ablation on the L and L loss components on GI, Sleep EEG, and PTB ECG, using the supervised cross-entropy loss as the baseline for all experiments. the
The performance metries are the accuracy the original un-permuted test set (Original), accuracy permuted lest set (Permuted), and their Difference (Diff), with higher Original and Diff, and lower Permuted values indicating better performance. 'V' indicates inclusion, and 'X" indicates exclusion of the module. The best results are highlighted in red bold.
Dataset Name
GL HAR
SleepEEG
PTB ECG
Models
ee
Original
Permuted
Diff
Original
Permuted
Dif''
Original
Permuted
Diff 1
x
X
91,6g
88.8amm
2.76
61.6g11
58.521
3.03
78.6
76.3
2.28
PatchTST
96.6
89.2
7.33
61.8aus
58.0
3.79
78.8
23.7am
5.17
(2022)
x
97.2
89.0
8.17
17.9
3.56
81.8
73.5
8.33
97.4
88.7
8.65
63.5
58.7
4.69
8144ss
72.7:051
8,71
x
91.6
85.3
6,30
55.4
55.1
0.32
82. 02.0zam
71.62003
10.4
PITS (2023)
x
92.8
81.0axse
11.8
56.3ans
55.62155
0.65
85.
68.9
16.2
94.02m0
87.6gss
6,40
57.4
16.8
0.66
21.3
12.7
96.3
73.
23.3
1.95
65.00
24.6
5.2 SUPEVVVEE TRAINING
Setup. We evaluated the downstream performance of a self-supervised model through semi- supervised training. This involves pre-training the encoder with unlabeled samples, then adding a fully connected layer, and fine-tuning all parameters using cross-entropy loss (Wang al., 2024b) with limited labeled training data: 10%, and I %. We also provide the full fine-tuning results (using 100%) in Appendix K, where we also compare
work with randomly initialized backbone.
Results. Table 2 is the results of semi-supervised training on the GL task. PPT demonstrates superior performance. ranking as the top or second-best model in 10 out of 12 metries. Given GL's imbalanced nature, metrics beyond accuracy provide more meaningful insights. In both 10% and 1 % labeled data scenarios, utilizing PPT as the pretext task leads to substantial performance improvements for PStchTST and PITS compared to their original pretext tasks (mask reconstruction and contrastive learning, respectively). Notably, in the 1% labeled data setup, F1 scores improve by up to 3 percentage points for PatchTST and 11 percentage points for PITS. We hypothesize that PPT pretraining initializes the model in a more order-aware embedding space, enabling it to benefit from self supe-vue pretraining even with limited supervised training data.
5.3 SUPERVISED TRAINING
Setup. We also highlight that PPT can improve supervised training performance as an additional pretext task, with the results in Tab. 3. Here, we selectively add the patch order consistency and contrastive loss terms in Eq. (3) to the cross-entropy loss as the base loss. We evaluated the model using Original, Permuted, and Diff metrics. The Original (f) metric measures classification accuracy on the standard test set, aiming for higher values. The Permuted (L) metric assesses accuracy a previously unseen, strongly permuted (y = 40) test set, with lower scores indicating that the This method of assessing order awareness through
model considers order in its decision-makinn performance degradation has been explored in the time series (Zeng et al., 2023) and video (Yun et al., 2022) domains. Finally, the Diff (t) represents the difference between the Original and Permuted metric, with more differences signifying more effective learning of order dependencies.
Results. The results in Tab. show that incorporating PPT improves model performance (Original) across all six cases, with gains of up to seven percentage points in the PTB task for PITS. Additionally, performance on the Permuted sets decreases in all instances, indicating enhanced patch-order aware- ness provided by PPT. Notably, the performance drop occurs even when the proposed consistency and contrastive loss terms do not directly penalize logits for permuted patches, unlike methods that optimize to increase uncertainty in out-of-distribution sets (Lee et al., 2017; Yun et al., 2022). The Diff metric also shows increased scores with the addition of each loss, suggesting a synergistic effect when both losses are used. Furthermore, performance drop the Permuted set is more pronounced in the linear model compared to the Transformer-bast model (PatchTST), aligned with findings of existing work (Zeng al., 2023).
5.4 ACF-CoS METRIC FOR TIME SERIES DATASETS
We hypothesize that PPT performs optimally with time series tasks that exhibit ordered information. To test it, we conducted supervised experiment with PStchTST on tasks from the UEA repository, comparing the performance enhancement of PatchTST+PPT against the baseline of PatchTST that only applies cross-entropy (CE) minimization. As time series models are known to be sensitive to hy-
9

---

Published as a conference paper at ICLR 2025
75%
UEA Datasets
ACF-Co5 7
Acc. Win% (Wins)
Max CE/ Max PPT (Ace)
OLS
Step Function (Order 1)
0,902
Acc Win (%)
65%
RLM
Cricket
0,418
15.9% (122/162)
09.4/72.7
EigenWWrrss
0,289
61.1% (95/1622
47.1/ /4.5
NATUPS
0.216
15.356 (122/162)
70.0/71
55%
LargeKischen
0.190
(58/162)
GestureMidAii)
0.186
48.19% (78/1622
31.3
GestureMliAAirDS
0,360
37.7% (61/162)
18.2/ 16.9
45%
White Nuier (Order 1)
0.001
35%
0.0
0.1
0.3 0.4
Table 4:
for 24 tasks from the UEA Repository: Results for six subsets are shown here. "Acc. Win(%)" the ratio of the count at which PPT enhances model performance compared to cross- cotsopy (CE) at various hyperparameter setups. "Max PPT/ Max CE (Acc)." denotes the best accuracy for PPT and CE in all hyperparam- eters (hest in bold). Here, increase in ACF-CoS leads improved likelihond of PPT being beneficial. Full results in Appendix I..
The ACF-CoS
Average ACE-Cos
Figure 5:
The scatter plot for against "Acc.
the ACF-COS Win". Ordinary Least Square (OLS) and Robust Linear Method (RLM) are fitted to show their correlation
perparameters (Jin et al., 2024), we systematically conduct experiments with various hyperparameter setups; evaluations are carried out with 162 different hyperparameter configurations, each with three random seeds. To gauge the effectiveness of PPT, we measure the count in which PPT significantly outperforms CE (cross-entropy), achieving performance gain exceeding 1 % (expressed proportion, not percentage points) of accuracy. The summarized results of 978 experiments (162 configurations 3 seeds for both CE and PPT) for each task are Tab. and Fig. 5, along with the ACF-COS scores averaged with various patch lengths. The relationship between ACF-CoS scores and performance gains is explored with the Ordinary Least Squares (OLS) (Montgomery et al., 2021) and Robust Linear Model (RLM) using Huber regression (Huber, 1973), resulting in slope coefficients of +0.47 for OLS and +0.37 for RLM, both statistically significant. These findings show positive correlation between the ACF-COS score and PPT's capacity to provide additional supervisory signals, which enhances model performance. We discuss our full results in Appendix L.
5.5 ADDITIONAL EXPERIMENTS
Patch Selection Strategy. We compare various patch permutation strategies under both weak and strong permutations in Appendix C, showing that the random permutation is the most effective. Hyperparameter Sensitivity Analysis. We evaluate the performance of PPT with varying patch sizes, 8, and permutation frequencies, y in both self-supervised and supervised training in Appendix G. PPT with Time Series Foundation Models. We applied PPT to GPT4TS (Zhou et al., 2024), a time series foundation model which fine tunes GPT2 for time series (Appendix M). We show that under our patched setup, PPT can enhance performance for time series-based foundation models.
LIMITATIONS AND DISCUSSIONS
We recognize and discuss several limitations of PPT. First, as detailed in Sec. 5.4, PPT does not enhance performance for tasks where the classification of time series is independent of order information. However, ACF-CoS can help identify such tasks. Second, the credibility ACF-CoS is influenced by the size of the patch s. Although we mitigated it by averaging the ACF-COS scores in different patch sizes, a more robust metric needed. Lastly, attempts apply PPT time series forecasting tasks tend to yield minimal gains, likely because permuting patches disrupt trend patterns, which are key features in forecasting (Liu et al., 2023; Wu et al., 2021). Designing a universal pretext task for time series is our next research topic.
7 CONCLUSION
In this paper, we introduce PPT. a patch awder pretext task designed for time series classification models that utilize patch-base based representations. We first introduce several strategies to improve patch order awareness in the model by proposing channel- wise patch permutations with varying permutation strength, assisted with consistency and contrastive loss. We show that each loss component plays an important role in enhancing patch order awareness, leading to enhanced model performance. To extend beyond this gain, we develop the novel ACF-CoS metric to identify tasks where PPT can be beneficial, quantifying the importance of order information in a time series. The contribution of this work is a timely response to the growing prevalence of patch-based time series work, addressing the critical challenge of integrating order awareness for improved classification models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea(NRFF) grant funded by the Korea (ATT
and Institute of Information & communications Technology Planning & Evaluation(IITPP) grant funded by the Korea governmen(MMST)) (N0.RS2020- 11201336, Artificial Intelligence Graduate School Program(UNIIST). The authors are grateful to Seok-Ju Hahn and Yoontae Hwang for their helpful discussions during the initial ideation phase. Special thanks go to Hyunwoo Seo, Yeonjoo Kim, Gyeongho Kim, Isu Jeong, Kyu Hwan Lee, Ahin Lee, and the EAT Lab members for their feedback on the manuseript. The authors thank the anonymous reviewers for their constructive feedback during the discussion phase.
REFERENCES
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Kcogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811, 00075, 2018. 8, 19
Christoph Bandt and Bernd Pompe. Permutation entropy: natural complexity measure for time series. Physical review letters, 88( 88(17):17410)) 2002. 3, 30
Ralf Bousseljot, Dieter Kreiseler, and Allard Schnabel. Nutzung der ekg-signaldatenbank cardiodat der ptb über das internet. 1995. 8, 19
Ruichu Cai, Jiawei Chen, Zijian Li, Wei Chen, Keli Zhang, Junjian Ye, Zhuozhang Li, Xiaoyan Yang, and Zhenjie Zhang. Time series domain adaptation via sparse associative structure alignment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 6859-6867, 2021. 16
Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.0499 2023. 2
Peng Chen, Yingying Zhang, Yunyao Cheng, Yang Shu, Yihang Wang, Qingsong Wen, Bin Yang, and Chenjuan Guo. Multi-scale transformers with adaptive pathways for time series forecasting. In The Twelfth International Conference on Learning Representations, 2023. 2, 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020. 25
Mingyue Cheng, Qi Liu, Zhiding Liu. Hao Zhang, Rujiao Zhang, and Enhong Chen. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv preprint 00000:23003.003320, 2023. 7,
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive.
IEEE/CAA Journal of Automatica Sinica, ((6):12931330 2019. 2
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.X.:1811000400, 2018. 16
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp.
1422-1430, 2015. 16
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long, Simmtm:
A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36, 2024. 6, 7, 16, 26
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.1 1929, 2020. 16
11

---

Published as a conference paper at ICLR 2025
Joy O Egede, Siyang Song, Temitayo A Olugbade, Chongyang Wang, De C Amanda, Hongying Meng, Min Aung. Nicholas D Lane, Michel Valstar, and Nadia Bianchi-Berthouze. Emopain challenge 2020: Multimodal pain evaluation from facial and bodily expressions. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 849-856. IEEE, 2020. 8, 19
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv: 106, 12, 2021. 3, 7, 16, 25
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, Xiaoli Li, and Cuntai Guan. Self-supervised contrastive representation learning for semi-supervised time-series classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 3, 16
Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis,, and Marinka Zitnik, Units: Building a unified time series model. arXiv preprint arXiv:2403.0013/ 2024. 3,7
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predieting image rotations. arXiv preprint arXiv: 180.00288,, 2018. 16
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()):
1735-1780, 1997. 21
Peter Huber. Robust regression: asymptotics, conjectures and monte carlo. The annals of statistics, pp. 799-821, 1973.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. In The Twelfth International Conference on Learning Representations, 2023. 3
Ming Jin, Yifan Zhang, Wei Chen, Kexin Zhang, Yuxuan Liang, Bin Yang, Jindong Wang, Shirui Pan, and Qingsong Wen. Position paper: What can large language models tell us about time series analysis, 2024. 10
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence, 43(1 (3(11)4403744088 2020. 2
Bob Kemp, Acilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a slcep-dependent neuronal feedback loop: the wlow microcontinuity of the ceg. IEEE Transactions Biomedical Engineering. 47(9): (115))1188-111 1185-1194, 2000. 8, 19
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7499 7491, 2018.
Jaeho Kim, Hyewon Kang, Jaewan Yang. Haneul Jung, Seulki Lee, and Junghye Lee. Multi Multi-task deep learning for human activity, speed, and body weight estimation using commercial smart insoles. IEEE Internet of Things Journal, 2023. 2, 8,
Taesung Kim, Jinhee Kim. Yunwon Tae, Cheonbok Park, Jang Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference Learning Representations, 2021. 25
Dani Kiyasseh, Tingting Zhu, and David A Clifton. Cloes: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pp. 5606-5615. PMLR, 2021. 16
12

---

Published as a conference paper at ICLR 2025
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A uni- fied and generic model interpretability library for pytorch. arxiv. arXiv preprint arXiv:2009.07896, 2020. 2
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrateed classifiers for detecting out-o--distribution samples. arXiv preprint urXiv: 2017. 9, 23
Seunghan Lee, Taeyoung Park, and Kibok Lee. Learning to embed time series patches independently. In The Twelfth International Conference on Learning Representations, 2023. 2, 3, 7, 9, 21, 25
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 3
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023. 10
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint 13X1v:1301.3781, 2013. 16
Douglas C Montgomery, Elizabeth Peck, and G Geoffrey Vining. Introduction to linear regression analysis. John Wiley & Sons, 2021. 10
Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. Recofit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, pp. 2224-3234, 2014. 8, 19
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Miua-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296-23308, 2021. 16
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2022. 2, 3, 6, 7, 9, 21, 25
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pp. 69-84. Springer, 2016. 16
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 187.0:180,,03748, 2018. 5
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 32
Yale Song, David Demirdjian, and Randall Davis. Tracking body and hands for gesture recognition: Natops aircraft handling signals database. In 201 / IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp. 500-506. IEFE, 2011.
Chi lan Tang, Ignacio Perez-Pozuele Dimitris Spathis, and Cecilia Mascolo. Exploring contrastive learning in human activity recognition for healtheare. arXiv preprint arXiv22011.11542, 2020. 7, 25
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:210660005500 2021. 16
Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulié. Data augmentation of wearable sensor data for parkinson's 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 216-220, 2017. 3,
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(1)), 2008. 2, 6
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2
Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, and ZHAO-XIANG ZHANG. Droppos: Pre-training vision transformers by reconstructing dropped positions. Advances in Neural Information Processing Systems, 36, 2024a. 16
Yihe Wang, Yu Han, Haishuai Wang, and Xiang Zhang. Contrast everything: hierarchical contrastive framework for medical time-series. Advances in Neural Information Processing Systems, 36, 2024b. 19, 27
Yucheng Wang, Yuecong Xu, Jianfei Yang. Min Wu, Xiaoli Li, Lihua Xie, and Zhenghua Chen. Graph contextual contrasting for multivariate time series classification, arXiv preprint arXiv:2309.05202, 2023. 2, 3, 7, 16, 26
Zhiguang Wang. Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (LJCNN) pp. 1578-1585 IEEE, 2017. 25
Kristoffeer Wickstrøm, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. Pattern Recognition Letters, 155:54-61, 2022. 25
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term -erm series forecasting. Advances in neural information processing
systems, 34:22419-22430, 2021. 10
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022. 7, 8, 16, 25,
Sukmin Yun, Jachyung Kim, Dongyoon Han, Hwanjun Song, Jung Woo Ha, and Jinwoo Shin. Time is matter: Temporal self-supervision for video transformers. arXiv preprint S0ti:22077.00666 2022. 9, 16
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023. 9, 25
Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Yitan Cheng, Walter Talbott, Chen Huang. Hanlin Goh, and Joshua Susskind. Position prediction effective pretraining strategy. arXiv preprint arXiv:2207.07611 2022. 16
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017. 25
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems, 35:3988-4003, 2022. 7, 16, 20, 25, 27
Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024. 3, 10, 32
14

---

Published as a conference paper at ICLR 2025
A
FEATURE ABLATION
Supervised
Supervised + PPT
Sample 9 Sample 8 Sample 7 Sample 6 Sample 5 Sample 4 Sample 3 Sample 2 Sample Sample
100
200
300
400
500
100
200
300
400
500
Time
Time
Normalized Attribution
Figure 6: The figure presents randomly selected attribution results from the ShapesAll task under supervised training conditions. compare two approaches: (1) baseline method that uses only cross-entropy (CE) loss. and (2) our proposed method that combined PPT with cross-cntropy loss. We applicd the FeatureAblatio method to assess the importance of patches in model performance. The ShapesAll task was chosen for visualization due its high ACF-COS score, indicating that temporal order important in this dataset. Higher ACF-CoS suggest that order information is crucial for the task. We observe that incorporating PPT leads to better attribution scores meaningful time segments of data compared to using alone. instance, Sample we observe that the attributions are evenly distributed and the scores are high at the inflection points for PPT, while the baseline only focuses on single time segment.
We used the Captum (https: (https://captum ai library for PeatureAblation
15

---

Published as a conference paper at ICLR 2025
B EXTENDED RELATED WORKS
Order-Aware Self-Supervised Learning. Taking advantage of inherent order information within various data modalities has proven be an effective pretext task, enhancing model performance across domains. In the vision domain, structured data like pixels or patches has led to the development of order-aware tasks, such as relative position prediction Doersch al. (2015), jigsaw puzzle Noroozi & Favaro (2016), and rotation Gidaris et al. (2018) for convolutional networks. More recently, Vision Transformer (ViT) models Dosovitskiy al. (2020), which segment images into a grid of patches, have shown to be effective in modeling long-range dependencies but fail to incorporate order dependency between patches Naseer et al. (2021). As such, for ViT's, predicting the absolute positions of image patches by masking Zhai et al. (2022) and reconstructing positions of patches Wang et al. (2024a) have been proposed. Similarly, in video analysis, leveraging the temporal sequence of frames is crucial, (2022). Similarly, in natural language processing, the core assumption behind masked language modeling is that the sequential order of words in sentence contains critical cues for language understanding Devlin et al. (2018); Mikolov et al. (2013). Thus, utilizing structural order is key to designing meaningful pretext tasks across various data modalities.
it encourages models to grasp frame dynamics over static features Yun et al.
Structure-Aware Learning in Time Series. In time series analysis, several methods have their design principle based on the structural relationship within time sequences. SASA Cai al. (2021), in time series domain adaptation, focuses on extracting domain invariant representations by analyzing the causal structure among time and channel dimensions. In self-supervised learning for time series, TNC Tonekaboni et al. (2021) addresses the temporal consistency of clinical time series by defining a neighborhood function to extract positive samples for contrastive learning. Similarly, CLOCS Kiyasseh et al. (2021) deals with the temporal and spatial (channel) invariance of ECG applications based on the assumption that the information from the same temporal events is conserved in different time sequences and channels. TS2Vec Yue et al. (2022) is an universal time series representation method emphasizing contextual consistency. TF-C Zhang et al. (2022) utilizes both the time and frequency domain and proposes time-frequency consistency architectures. SimMTM Dong et al. (2024) constructs multiple masked series, where they consider these series as close neighbors of the original time series. TS-TCC Eldele et al. (2021) focuses on transformation consistency, generating two views of the time series data using weak (jittering and scaling) and strong (permuting and jittering) augmentations. Then, an autoregressive model is used to construct context vector for temporal contrasting between the two samples. This concept was expanded into semi-supervised learning (i.e. CA-TCC Eldele et al. (2023)) and graph-structured learning (i.e. TS-GAC Wang et al. (2023)). Inspired by TS-TCC, PPT introduces channel-wise patch permutation approach with varying strengths determined by the permutation frequency, termed 'weak' and 'strong' permutation. However, while the main objective of TS-TCC is to bring the weak and strong representations closer, our objective is the opposite, which is to widen the distance between weak and strong, while placing the weak closer to the non-po non-permuted 'original' set (Sec. 3.2). Moreover, unlike TS-TCC's random segment permutations Um et al. (2017) made on the fly, PPT gives more control over the strength of permutation, which enables time and feature-wise consistency and contrasting mechanism (Sec. 3.3). In addition, PPT is efficient due to its use of a permutation bank (i.e. only the permuted indices are stored) and effective because it continuously exposes the model diverse array of permuted samples. As such, while PPT utilizes a similar augmentation strategy from TS-TCC, it significantly diverges in its focus for contrasting, making it novel and effective patch-based pretext task for time series.
16

---

Published as a conference paper at ICLR 2025
C COMPARISON BETWEEN PATCH SELECTION STRATEGY
(A) Vidinity Pairs W= 1)
(B) Distant Pairs 3)
2
4
6
7
8
9
Rendom Pairs
Id
The figure illustrates the patch selection strategy utilized in the channel-wise patch permutation strategy, where two patches are selected from the same channel, and their positions are swapped. The selection method varies according to the strategy adopted, which is quantified by the distance d between the patch pairs. (A) Vicinity Pairs (d = 1): The adjacent patches with distance d are swapped. (B) Distant Pairs (d 2 3): Patches separated by least three patches swapped. (C) Random Pairs (d 1): This strategy includes combination of both vicinity and distant pairs.
Figure 7:
Step Function
Vicinity Permute
Distant Permute
Randem Permute
14 A
230
500
250
240
2.50
550
250
20.0
Time
Time
Time
Time
Figure 8: We present visualizations of the results obtained from permuting a step function using various permutation strategies: Vicinity, Farthest, and Random. The permutation frequency hyperparameter, y, was set y = 10 for the first and 40 for the second row.
Gilon HAR
Strong
Weak
PatchTST
PITS 1
PatchTST f
PITS
Average Rank |
Vicinity
Vicinity
97.2++0.55
96.0:20.76
80.2++2.89
86.2+2211
Vicinity
Distant
97.5-10.51
16.11
79.0
85.8+11.0
5.25
Vicinity
Random
97.3:10.57
96.4::082
78.4:1
1..9
4.75
Distant
Vicinity
96.7:10.84
95.7:11.01
4..79
86.1 16.00
6.00
Distant
Distant
96.7-+0.69
96.6=10,22
78.4
88.6:10.93
5.00
Distant
Random
9.75
96.7:10.41
80.6.3322
84.7114.22
5.75
Random
Vicinity
16.9+
96.0:10.71
79.6++3.83
87.8++1.44
5.50
Random
Distant
97.0++0.90
96.8
7..2
86.3++2.52
5.00
Random
Random
97.4:10.46
96.3411.19
81.4
89.5
2.50
The classification accuracy from supervised experiment with varying permutation strategics given. Here, the strong permutations The highest performing value red bold, and the second highest black bold. We calculated rank each dataset and model and averaged those ranks to obtain Average Rank, where lower score indicates better performance. Overall, the Random- Random-Random Random combination yielded the best results.
Table 5:
implemented using 7 = 40, while weak permutations used ry 1.
17

---

Published as a conference paper at ICLR 2025
D CHANNEL-WISE PATCH PERMUTATION
Algorithm Pseudocode Channel-nndependent Patch Permutation in PyTorch style.
B: batch size; C: channel number; T: Original Time Length
2. PD: Patch Dimension; PN; Patch Numbers (Number of Patches)
Print original shape of print (X. shape) # -> (B, C, T)
6
4 Reorder X into non-overlapping patches. rearrange (X, "B C (PD PN) -> В PD C PN")
10 Sample a random value (key)
11 key = torch. (0, 1000, (1,))
Permute Bank is di dictionary containing permuted indexes. shuffled_idie idx permute Barkute_bankkie [key]
15
Print shuffled idx. idx) # -> (C, PN)
print (shuffled _ix))
19
Expand the shape of idx to match the shape of X expande__shuufleddid = shuffled_idx.ext
20
(B PD C PN)
Rearrange original patch based on seraadde__suufflee idx X_shfffled
23
(x,
24
25 Print x X_sufflled shape print (X shuffffeedd. # (B PD C PN)
28
Rearrange shape back to its original
29
s_huffledd = rearrange (X, "B PD C PN -> B C PD PN")
return X s_huffledd
rearrange: rearrange elements tensor according patterm; tozch. torch.gathers gather; gather values along an axis specified by index: expand: repeat the tensor along specified dimensions match the given shape.
Implementation. The channel-wise patch permutation strategy efficiently implemented, as outlined in the PyPorc-sttyle pseudocode in Algorithm 1. Prior to model training, permutation bank is constructed, which dictionary storing permuted patch indices. For example, in the case of 32-patch permutation, instead of retaining the actual time series data for the patches, only the indices of these patches are stored. 1000 different variations these permuted indices are stored to provide wide range of permutations during model training. During model runtime, random key is generated, and the matching permuted indices is retrieved and the patches are repositioned accordingly.
Time. Building such dictionary for task with 14 channels, 32 patches, permutation fre- quency ? = 40, with 1000 possible variations takes only 2.6 seconds on an Intel(R) Xcon(R) Gold 6226R CPU @ 2.90GHz. Permuting batch of 512 instances from this task takes merely 0.018 seconds on modern GPU hardware like the NVIDIA RTX A6000. This approach ensures that during training, the model is exposed a broad spectrum of permuted orders of the dataset, while being fast, leggt- and efficient.
Weak and Strong Permute Strategy. For implementation of PPT, built multiple permutation dictionaries with varying permutation frequency to support both weak and strong permutation strategies. For weak permutations, we select a lower bound 7 = | and construct dictionaries for = [, + 1, + 2. For each training batch, one of these dictionaries from both weak and strong randomly selected for use. This method further enbances the variety of permutations the model encounters during training.
built three dictionaries for 7 2, For strong permutations,
18

---

Published as a conference paper at ICLR 2025
E DATASETS
Dataset
Task Type
#Samples
Time
#Users
# Channels
# Class
MS
HAR
14,201
200
Gilon
HAR
47,647
160
14
EMOPain
sEMG
1,143
200
30
30
PTB
ECG
62,370
300
198
15
2
Sleep
EEG
371,055
178
82
1
Table Details of datasets used in our main experiments
E RATIONALE FOR DATASET SELECTION
The selection of the five datasets used in our main experiment was based on three major reasons: (I) public accessibilityy (2) sufficient number of samples for generalization, and (3) diverse representation of time series tasks. All datasets utilized in our experiments are freely accessible, except for the Gilon task, which is available upon request and only for research purposes. Moreover, each time series task in our study includes more than 1,000 samples, with four datasets containing more than 10,000 instances each, supporting effective generalization. In addition, the chosen tasks span range of applications in time series analysis, including Human Activity Recognition (HAR), Electroenceepallggaapy (EEG) for brain wave analysis, electrocardiography (ECG) for cardiogram monitoring, and Surface Electromyography (sEMG) for muscle activity. This diversity demonstrates the applicability of PPT to a wide range of tasks. We believe that our work would be particularly beneficial in fields where sequence order is crucial, such as in biosignal analysis.
E.2 DATASET DESCRIPTION
MS HAR (Morris et al., 2014). The MS HAR task is human activity recognition (HAR) dataset using data collected from 114 users who performed 72 different gym activities, measured with a smart armband. The armband incorporates six sensor channels: three accelerometers (x, Z axes) and three gyroscopes (x, y, Z axes). To address data imbalance among the classes, we selected 10 activities from the original 72 classes. These include Bicep Curl (0), Biceps Curl with Band (1), Jump Rope (2), Plank (3). Pushups (4), Squat (5), Squat with Hands Behind Head (6), Squat Jump (7), Walk (8), and Walking Lunge (9). The raw dataset is available for access at (https://github.cm wttpiiitthbcco/mmmmmeeec wowm/mmcrosof//Eeerciiooccccocce SeearrbbleSSesoosy We will provide the preprocessed dataset through GitHub.
Recognition-fron
Gilon HAR (Kim et al., 2023). The Gilon dataset is developed for HAR and includes data collected from 72 participants performing seven different gym activities. The data was gathered using a smart insole that features 14 sensor channels, including accelerometers (x, y, Z axes) and four force-sensitive resistors (FSRs) for each foot (right and left). We used the original five-fold split given in the original paper. The dataset be accessed with request
http:/iiiuubcooiilooooniio InC/GIINNN DATASET) and is publicly available for research purposes.
EMOPain (EMG) (Egede al., 2020). EMOPain is a pain classification task using surface electromyographic (sEMG) sensor to classify different pain levels (0: Healthy, 1: Low-level pain, 2:
High-level pain). There are total of 30 sensor channels positioned strategic anatomical positions. The dataset can be accessed through the UEA time series classification repository (Bagnall et al., 2018).
PTB (ECG) (Bousseljot et al., 1995). The PTB ECG dataset comprises cardiogram signals from 294 patients, including healthy subjects. We utilized preprocessed subset of this dataset from the study by (Wang et al.. 2024b), which includes data from 198 patients. This subset redefines the original task into a binary classification problem, distinguishing between Myocardial Infarction and healthy controls. We adopted the same data splits specified in their study. The preprocessed dataset can be accessed from https://github.com/// HLL.LAAatHHHOOEEEEEE.
Sleep (EEG) (Kemp et al., 2000). The Sleep EEG is univariate brainwave signal from Electroen- cephalography (EEG) recordings, where the task to classify one out five sleeping patterns: Wake, Non-rapid eye movement (N1, N2, N3) and rapid eye movement (REM). We utilized the preprocessed
19

---

Published as a conference paper at ICLR 2025
dataset from (Zhang al., 2022) and the dataset can be accessed from (ttts://giihub.com/miis harvar//FFC-pretrainnng)
F IMPLEMENTATION OF PPT
F.1 HARDWARE AND SOFTWARE
All experiments were conducted on two types of GPU: NVIDIA RTX 3090-24GB and NVIDIA RTX A6000-48GB. We used the Py PyTorch 2.0.1 and Py Lightning 2.1.2 frameworks, All experiments can be completed on single GPU mentioned above.
F.2 TIME AND MEMORY COST OF PPT
We conducted additional experiments measure the time and memory complexity of PPT in model training. Specifically, we conducted experiments on the PTB task for self-supervised linear probing and have compared our method against mask- -ased and contrastive-bass approaches using the same backbone model (e.g. PatchTTTT).
In summary, PPT employs additional regr-segreseive models for constructing context vectors in the training process, which is the main overhead in speed and memory consumption compared to previous pretext task methods. However, the inference stage, these auto-regressive models can be discarded. In terms of time complexity, mask-based approaches took 70.3 seconds for the whole training, contrastive- based approaches took 92.9 seconds, while PPT took 139.5 seconds. This increased training time is due to the additional computations required for auto-regressive models constructing context vectors. Regarding memory complexity, the mask-based approach had a total of 79.7K parameters, the contrastive-based approach had 79.3K parameters, and PPT had 212K parameters. The higher number of parameters in PPT is primarily attributed to the use of auto-regressive models.
Despite the increased computational overhead, we believe that the improved performance achieved by PPT, as demonstrated in our experimental results, justifies the use of this technique. We acknowlede that the current implementation of PPT has room for optimization, and we propose that exploring more efficient model designs for constructing context vectors (c.g., reducing representation dimensions, and re-using the auto-regressive models) would mitigate the computation burden.
F.3 RETOEREGEESSVVE MODELS
To generate the context vectors in consistency and contrastive learning, we used a single-layer unidirectional LSTM that takes the patch sequences input. The final cell state of the LSTM serves as the context vector, effectively summarizing the sequence of patches.
For consistency learning, assigned pseudo labels: for sequences of patches from the original sequence and 0 for sequences of patches from a strongly shuffled sequence. Initially, the LSTM is unable to recognize the correct ordered sequence. However, through end-to-end training alongside the backbone patch model, the model Iearns to discriminate between the original and strongly shuffled patch sequences. Consistency learning applied along both the time and channel dimensions. For the time dimension, the module learns to recognize the correct temporal order of patches. For the channel dimension, the module learns the cross-channe relationships of patches. As shown in Fig. 1, distinct patterns co-occur along the channel dimension, and learning this cross-channel relationship has been shown to enhance model performance.
For contrastive learning, we collected the context vectors from the original, weakly permuted, and strongly permuted patch sequences. Intuitively, the weakly permuted sequence should have high similarity with the original sequence, while the strongly permuted sequence should be far from both the original and weakly permuted sequences the representation space. To achieve this, we employed the InfoNCE loss for contrastive training.
The combination of consistency and contrastive learning allows the model to capture both the temporal and cross-channel relationships of patches, leading improved performance in time series classification tasks. By learning to recognize the correct order and relationships between patches, the model can better understand the underlying patterns and structure of the time series data.
20

---

Published as a conference paper at ICLR 2025
F.4 SEQUENTIAL REPRESENTATION OF PATCHES AND HEAD TYPE
The latent data representation for patch-based models consists of encoded patch representations, differing from models that provide instance-wise representations. Previous studies have processed these representations by flattening (Nie et al., 2022) or averaging Lee et al. (2023) them to form an instance-wise format. However, these methods may not adequately capture the sequential depen- dencies between patches. To better capture these dependencies, we utilized single layer LSTM model (Hochreiter Schmidhuber, 1997) to generate instance-wise or sequential representations from patches. The patch representation is specified as R RCxPDxPM PN where C' is the number of channels, (d(d,) is the dimension size the patch, and PN is the number patches per channel. Initially, we averaged across the channel dimension to produce patch representation of shape R RDDxPN This averaged representation serves as the input to the LSTM model, treating PN as the sequence length. In all experiments, employ the context vector from the LSTM the instance- wise representation.
F.5 HYPERPARAMETERS FOR SUPERVISED LEARNING
The hyperparameter configurations for supervised training are in Tab. 7. We utilized the official implementations of both PStchTST and PITS. adopting the chaanel-innependeee outlined in its respective studies. For PatchTST, three layers of transformer encoders were used. A slightly high learning rate was found to improve the performance in PITS. Patch sizes were determined task specifically, with the aim of 10 to 40 patches per channel. No learning rate scheduling or weight averaging techniques were used for PPT. The training was terminated when the validation loss did not decrease for more than two epochs. The temperature for InfoNCE was set to 0.1.
configuration as
Model
Task
Batch Size
Permutation 7
Learning Rate
Patch Size
da
7Akeads
PatchTST
Gilon
128
10
0.002
5
128
PatchTST
SleepEEG
256
0.002
5
PatchTST
PTB
128
10
0.002
10
128
8
PITS
Gilon
128
0.02
10
128
PITS
SleepEEG
256
10
0.02
5
256
PITS
PTB
256
10
0.02
10
256
Table 7: Supervised Training Hyperparameter Configurations.
F.6 HYPERPARAMETERS FOR DELE-SUPERVISED LEARNING
The hyperparameter configuration for self-supervised linear probing is in Tab. 8. For full fine-tuning, the hyperparameters are identical except for the learning rate, which is reduced to one-tenth that used in linear probing. In self- self-supervised training, we do not separate validation set; instead, the model trained for predetermined number of steps. Subsequently, we fitted a logistic regression model using the methodology from TS2Vec (Yue et al., 2022).
Model
Task
Batch Size
Y
Learning Rate
Patch Size
dy
Train Steps
PatchTST
Gilon
32
40
0.002
5
128
1000
PStchTST
EMO
32
40
0.002
10
256
1000
PatchTST
PTB
32
0.001
64
4000
PatchTST
MS
32
40
0.001
4
256
1000
PITS
Gilon
32
40
0.002
10
128
4000
PITS
EMO
32
40
0.01
10
128
1000
PITS
PTB
32
40
0.01
6
512
4000
PITS
MS
32
40
0.002
4
512
4000
Table 8: Self-Supervised Learning Hyperparameter Configurations. We fixed temperature to
21

---

Published as a conference paper at ICLR 2025
G HYPERPARAMETER SENSITIVITY
G.1 SUPERVISED LEARNING SENSITIVITY ANALYSIS
performed a hyperparameter sensitivity analysis in both supervised and self-supervised learning. Specifically, are interested in how the size of patches and the frequency of permutations affect the performance of the model in supervised learning. We report the results for the Gilon task in Fig. Here, Gilon task has time length of 160, such used patch lengths of 5, 10, 16, and 20. We set the strong permutation frequency between 5, 10, 20, and 40. We observe that the patch size is key hyperparameter in obtaining optimal performance for PatchTST in the supervised setup, where as PITS was robust across various patch sizes. In general, increasing the permutation frequency tends to enhance model performance. However, high permutation frequency does not guarantee improved performance in all cases. We hypothesize that, beyond a certain threshold, further permutations do significantly alter the outcome.
Supervised Sensitivity Analysis
PatchTST
PITS
40
94.0
40
95.7
96.7
97.5
96.6
95.7
94.5
96.6
96.6
96.5
96.6
Permute Frequency (y)
10
96.7
96.8
95.5
94.7
10
96.8
96.3
96.3
96.5
96.4
10
20
5
10
16
20
Patch Length
Length (s)
Figure sensitivity analysis of two hyperaarameters on supervised Gilon task conducted: patch length permute frequency. The x-axis indicates the patch length and the y-axis represents intensity of permutation frequency. For PatchTST, we observe an increase in performance with a smaller patch size. In contrast, PITS displays mixed results, maintaining consistent performance across different hyperparameters.
G.2 SELF-SUPERVISED LEARNING HYPERPARAMETER SENSITIVITY ANALYSIS
We conducted a hyperparameter sensitivity analysis for the PTB (ECG) task within self-supervised learning setup, with the results in Fig. 10. Unlike in supervised learning, PITS demonstrated greater sensitivity to patch size than PatchTST. Based on these observations, we conjecture that the sensitivity to patch size and permutation frequency depend on the specific learning setup and type of task to which it is applied.
G.3 TEST ON PERMUTED DATASET
We evaluated the performance degradation of the supervised PITS model a permuted set (PTB task), detailed in Fig. 11. This figure expands upon Tab. 3, which presented results for test set permuted at a frequency of y 40. Here, we provide comprehensive results for test sets permuted at frequencies 7 = 5, 10, 20, 40. Additionally, 7 represents the unaltered original test set. The figure shows results for the three distinet permutation strategies strategies-farthest farthest, random, and vicinity- with each strategy described in Appendix C. For example, the farthest permuted set is the test set that has been permuted with the "farthest" strategy.
In general, we observe that incorporating PPT enhances model performance on the original unaltered test set (y = 0), while having a degraded performance on the permuted test sets (y = 5, 10, 20, 40). We also observe that the extent of degradation varies with the permutation strategy, being more pronounced under the farthest and random strategies. This variation intuitive, as the vicinity permutation strategy maintains more of the original sequential information, illustrated in Fig. 8.
22

---

Published as a conference paper at ICLR 2025
Self-Superviset Linear Probing Sensitivity Analysis
PatchTST
PITS
78.1
78.3
86.5
83.5
83.0
78.3
76.1
76.1
78.6
86.2
83.0
83.9
83.8
Permute Frequency (X)
10
78.1
74.5
78.0
75.3
10
86.8
85.0
83.8
83.1
79.3
75.5
78.3
86.4
83.5
84.2
20
5
20
Patch Length
Patch Length
Figure 10: A sensitivity analysis self sulf--upe PTB (ECG) task conducted. Unlike the supervised setup, we now observe that PITS is more sensitive patch size.
Supervised Model Performance on Permuted PTB Test Set
Farthest
Random
Vicinity
0.9
0.90
PPT
Test Accuracy
0.85
Truc
0.80
0.75
0.7
0.7
0.70
10
20
50
20
20
Permuted Test Set Frequency (y)
Figure 11: We trained PPT with Random-Randee Randommnnm permutation strategy with the strong permutation frequency set Ty 40. The model trained with PPT and without PPT was evaluated permuted test set, where the permutation the test set was set 7 = 5, 10, 20, 40.
A note on the drop in performance for permuted test sets. For first viewers, it is possible that the drop in model performance on the permuted test set seems trivial or insignificant. However, we emphasize that such drop occurred even though there was no direct penalization of the logits from the permuted patches, unlike related works that directly penalize logits for uncertain samples (Lee al., 2017) in out of distribution detection problems. During the training of PPT, the permuted sets were used to calculate the consistency and contrastive losses, but no logits from the patch sequences directly penalized. As such, the degradation in model performance is solely due to the learned representation, which can be considered significant.
23

---

Published as a conference paper at ICLR 2025
G.4 LEARNABLE AND FIXED LAMBDA
PatchTST
PITS
ees
e
ees
ger
88.67
learned
learned
91.24:10.72
Iearned
learned
88. 4:0.7
91-20
0.1
88. 09.399
0.5
91.00=00.79
0.1
0.1
87.63
0.5
90.99:10:46
0.1
87.46+1..5 3
17.50+1.02
1
0.2
90.99+11.0
0.5
0.2
90
87.32 5
2
0.5
90.89 90.891:0.8 86
1
87.19 1.53
0.1
90.84:11.21
0.2
87.16
0.5
90
0.1
8
87 1:93
0.1
90.80 104
0.5
2
87.04.10.00
0.2
90.78 00.78:00.36
0.5
10
86 9.71
0.2
90.76 1.18
2
10066
90.68
12
86 1.994
0.2
90.6611:027
0.2
0.5
13
86.83 +-0.8
0.5
90.63--0.42
0.1
0.2
14
86.7611.220
90. 04
15
86.68 84
0.1
90.5 2:023
0.5
0.5
86.55
90.4811077
17
86 1.65
90 90.45
0.5
0.1
00
0.5
8:.27
0.2
0.5
90.28
0.1
20
86. 16::055
2
90.04+-0.46
2
21
86. 10400.3
0.5
90.02 .05
22
86.07 1.333
2
90.02 14
0.2
23
86.07
0.1
1.095
0.2
24
85.6 1:1.2
0.1
0.1
19.23++1.38
0.1
0.5
Table 9: Performance comparison between PITS and PatchTST based on fixed loss coefficient
PatchTST
PITS
cs
c
1.2
ct
ct
A value
0.8
0.4
0.2
1000
2000
3000
4000
1000
2000 Steps
3000
4000
Steps
Figure 12: We visualized how the learnable parameters for Cs and C changes over the course of training.
To demonstrate the effectiveness of using learnable À instead of fixed X, we conducted multiple fixed-A experiments and compared them with the results of the learnable Briefly, we show that the learned Äs are the best performing compared to the fixed hyperparameter setup.
conducted grid search between LCS = 0.2, 0.5, 1.0, 2.0 and L 0.1,0.2,0.3. 1.0, 2.0, resulting 25 different combinations. We performed experiments the PatchTST (Transformer) and PITS (Lincar) model with the PTB dataset, and reported the results in Tab. 9.
We show that the learnable hyperparameters are comparable to the best performing fixed- hyperparameters that are manually searched, showcasing that the learnable hyperparameter setup can significantly reduce hyperparameter tuning while leading to optimal performance.
24

---

Published as a conference paper at ICLR 2025
H BASELINE IMPLEMENTATIONS FOR
LEARNING
For all baseline experiments in self-supervised linear probing, we standardized the batch size to 32. We conducted a grid search to determine the optimal learning rate, exploring values between [0.001, 0.002], and for the maximum number of steps, ranging between [1000, 4000], and reported the best outcomes. During full fine-tuning, we lowered the learning rate to the one-tenth of the learning rate used in linear probing. Here, we did not implement learning rate scheduling in any part of our study. For patch-based methodologies (PatchTST, PITS, TimeMAE, TS-GAC), we aimed to set patch size that ensured the number of patches per sequence fall in between 10 to 40.
PStchTST (Nie et al., 2022) is an early work utilizing patch representations in time series forecasting, employing a basic transformer architecture. We selected PatchTST as our main encoder backbone, expecting that PPT could be generalized to other transformer variants. Specifically, the work proposes to use a channel-independent encoder transformer network. We used channel- channel-independdeee independent encoder architecture with sine-cosine positional encoding from the original implementation. Across the study, we employed three transformer layers. The RevIN normalization (Kim et al., 2021) used in the original paper was not used as our focus was on time series classification. For supervised training, we utilized the representations from the last transformer layer, inputting them into an LSTM- classifier head to generate instance-wist representations. In the self-superrvsed training, we averaged the representations from all transformer layers and used a single-layered, non-bidirectional LSTM to process the patch representations. Patches were averaged along the channel dimension, where the patch indexes were considered as sequences. The final context vector was then employed in our analyses. In self-supervised training, we used patch size of 4 for MS HAR, for Gilon, 6 for PTB, and 10 for EMOPain. We used a batch size of 32 and learning rate of 0.002 for self-supervised linear probing and 0.0002 for full fine-tuning. For the mask and reconstruction pretext task, we employed http:////uuuboooo/u/uuiiii//////
the original implementation from
PITS (Lee et al., 2023). A major discussion in the time series domain is the use of MLP based models instead of the transformer architectures. Surprisingly, simple MIP based models tend to outperform transformer architectures, which has been actively discussed in (Zeng et al., 2023). We employed the MLP-based PITS architecture, following the official implementation (htips://github.co httpss//github.co///uungggh con/scunghan966ppts)) to demonstrate that PPT is also effective with MLP models. For self-supervised training, we used the following patch sizes: 4 for MS HAR, 10 for Gilon, 6 for PTB, and 10 for EMOPain. Similarly to PatchTST, we implemented an LSTM- classifier head to obtain instance-wise representations.
Mixing-up up (Wickstrøm et al., 2022) is a contrastive learning-based method for time series inspired by the mixup (Zhang et al., 2017) data augmentation strategy. This approach blends two different mini-batch samples to create an augmented sample. A contrastive loss is applied to the augmented sample, with each mini-batch treated as a positive sample weighted by the mixing coefficient. We used the official implementation from and set the hidden dimension to 64.
SimCLR (Tang et al., 2020) is an adaptation of the visual domain's 's SimCLR (Chen et al., 2020) to time series analysis. This method applies two augmentations (e.g., rotation, negation) to a time series instance, aiming maximize the agreement between latent representations from the same instances and minimize it for different ones. Instead of using the official Tensorflow-bar based implementation, we developed a custom version using ResNet- encoder (Wang et al., 2017). This encoder includes three blocks consisting of Cony1D, BatchNorm, and Maxpool1 operations, with filter sizes set to [32, 64, 128]. An average pooling layer on the final block generates 128-dimensional latent representation for each instance.
TS2Vec (Yue al., 2022) is a task-agnostic time series representation model widely recognized as a competitive baseline in numerous self-supervise studies. The model begins by randomly cropping time series to extract overlapping window segments. These segments are then projected using a linear layer, followed by random masking of the latent representations. Subsequently, both temporal and instance-wise contrastive loss is employed, where the loss is subsequently employed in a hierarcheial manner. Following the work of (Zhang et al., 2022), we employ two layers of TS2 modules, with hidden dimension of size 64 and output dimension of 128.
TS-TCC (Eldele et al., 2021) While we have outlined the difference between PPT and TS-TCC in Sec. 2, we provide further details of the TS-TCC implemention. TS-TCC begins by applying weak and strong augmentations the input time series, generating two distinct views for the raw time series.
25

---

Published as a conference paper at ICLR 2025
These views then processed by the encoder layer, which produces latent representations of the time series. These representations are treated as sequences, and a context vector is constructed using transformer model. Temporal and contextual contrasting losses applied using this context vector. We implemented TS-TCC using the official source from with the following hyperparameters: kernel size=8, stride=1, timestep= 10, and hidden dimensions of 64, Additionally, we used a jitter ratio of 0.8 and jitter scale ratio of 1.1.
SimMTM (Dong al., 2024) is a self-supervised pretraining framework that employs a multiple masked modeling strategy. Instead of creating a single masked series, the framework utilizes several masked versions of the time series for a reconstruction pretext task. The primary objective of this paper was to introduce pretraining framework, with the work primarily evaluated in transfer learning scenarios. We utilized the official implementation available at (https://github.. (ttt:://gittub.co///uu/////iin com/ttul////iMMMM/
setting the hyperparameters follows: kernel size 10, stride 5, masking ratio 0.3, masked numbers 3, and final output representation channels of 128. A significant drawback, however, is the substantial increase in computational demand proportional to the number of masked samples. As such, tested only on three masked samples. The CNN output channels were manually adjusted based on the task, with settings of 6 for Gilon, 10 for PTB, 7 for MS HAR, and EMOPain.
TAM (Cheng et al., 2023) is patch- patch-based self-supervised learning model for time series data, inspired by the masked autoencoder (MAE) approach in computer vision. The model uti- lizes transformer architectures with two pretext tasks: masked codeword classification and masked reconstruction of patches. We set the hidden dimension of the model to 64. The patch length is chosen task-spec,f,cally, ensuring the number of patches falls within the range of 15 to 40 for optimal performance.
TS-GAC (Wang et al., 2023) is an extension of TS-TCC model to graph neural network (GNN), utilizing patch representations for graph construction. The graphs are constructed by using patches as node features and patch correlations edges, where they proposes to use node and edge augmentation method for contrastive learning. Multiple graphs are formed within an instance, and these graph representations using an autoregressive model (i.e. transformer) to construct a context vector for temporal con- trasting. Additionally, the model employs wavelet-based augmentation, differentiating the intensity between weak and strong augmentations. We implemented TS-GAC using the official code from (https://itthb..o coom/rank--o Faarr/rrannaaaooooaao Wang-oss/TS TS-GAC), setting the patch length to for Gilon and 10 for PTB, EMO, and MS HAR, with a kernel size of 3 and stride of 1.
considered sequences. These sequences
then summarized
26

---

Published as a conference paper at ICLR 2025
1 TIME SERIES CLUSTERING
We performed time series clustering task with the learned time series representation from self- supervised linear probing in the works of (Zhang et al., 2022; Wang al., 2024b). We fitted K-means clustering on top of the pre-trained representations of the GL HAR task, which consists of 7 classes. The experiments were performed with 5 different random seeds to ensure robustness. We report the Silhouette Score, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) scores to evaluate the clustering performance. We also compared our works to SOTA works such as SimCLR, TS2 TF-C, TimeMAF, UniTS, and other pretext task based methods such as mask and reconstruct, and contrastive learning based method.
Model
Silhouette Score
NMI
ARI
SimCLR
0.128 +0007
0.472 +
+.289
TS2Vec
0.128
0.607
0.405
TF-C
0.143 1:0000
0.585
0.429
TimeMAE
0. 14 1:0023
0.556:10001
0.429
UniTS
0.308 100057
0.61010.108
1..75
PatchTST (Mask)
0.338 0.0007
0.548
0.384-10.018
PatchTST (CL)
0.253 +0,009
0.444
0.275 +0.22
PatchTST (PPT)
0,263 +-0023
0.644:0008
0.481 +-0.50
Table 10: Comparison of clustering metrics in GL. task
In Tab. 10, we show that PPT achieves the best performance in NMI and ARI scores, demonstrating its effectiveness in generating meaningful representations in clustering tasks. Here, NMI and ART measure the agreement between the predicted clusters and the ground truth labels. A higher NMI and ARI score suggests that the clustering assignment close to the ground truth. While we acknowledge that the mask-based pretext task used in PatchTST outperforms PPT in terms of the Silhouette Score, it is important to note that the Silhouette Score an internal evaluation metric that measures the compactness of clusters without requiring ground truth labels. A higher Silhouette Score indicates that the clusters are well-separated and compact, but it does not necessarily imply that the clustering results align with the true class labels.
The result that PPT outperforms other SOTA methods in terms of NMI and ART scores highlights its ability to capture the internal temporal structure and order relationships within the time series, leading to clustering results that align well with the true class labels. In summary, we show that PPT demonstrates its effectiveness in generating meaningful and informative representations for clustering tasks.
27

---

Published as a conference paper at ICLR 2025
J
MORE LINEAR-PROBINE RESULTS
Table 11: Linear-Probing Results. We compare our methods several self-supervised methods, The asterisk (*) denotes patch based methods. The Best and second best scores are in red and black bold.
Dataset
Models
Accuracy
AUROC
AUPRC
Fl score
Precision
Recall
Mixing-up SimCLk
74.481291
71.8011.11
52.22412.1
49.30:1:..3
51.66:1181
49.36:1-4.01
74.42-44.38
72.71
48.3446.76
44.64-66.44
47.83-47007
47.19-4-.02
TS2Vec
78.08422.9
78.00+33.58
49.07
TF-C
77.6346.18
5..
54.8447.1
53.50-16.00
56.144879
56.00:19.07
TS-ICC
+:.61+33.9
73.82-47.40
53.98:33.7
48.47::3.59
54.75
49.02-444.05
SimMTM
81.75
81.3547.0
58.70114400
53.0843.68
57.75::4844
51.62414177
TimeMAE
73.97
70.1 12:444
43.25.:2.14
42.44::2007
42.81:22.37
42.43:1:009
TS-GAC"
13.75:1166
75.92:12.30
49.29:10622
46.42:1:129
46.04:1:0.87
48.86:1:1.69
PatchTSTiMask)
78.70::0.73
82.60:11.299
55.23+2221
45.81=2200
59,40:1:532
16.35
(Mask)"
68.0011.81
63.61:1:1220
43.69:10.59
43.84:1:.87
44.77:1:0.59
46.83:1:.55
PatchTST (Ours)
81.92:10.58
8.574:1
62.81
54.19:12.38
22.96::1:.49
53.41:1242
PITS (Ours)"
75.55:12..4
68.63:1:330
45.59:1:2.28
45.75:1:.49
45.05:1:2.65
47.53:1::06
Mixing-up
83.89:1:788
67
06.6001::110
28
81.16:12..7
79.00:1..17
SimCLR
83.86:10:92
97,87:1:225
85,02:1:1.2
78.58:1 1.38
80.98:1:.88
1.13
TS2Vec
82.76++0.57
96.86-40.32
83.10:+1441
77.72-+0.67
80,24:4-084
77.20-40.22
TF-C
85.02412.1
10:10444
86.
80.34:13.59
82.51:1:.93
80.06:13.50
TS-ICC
86.67 62262
98.61 1:0.00
88.244411.44
83.48-1225
85.56-1:2.44
82.88
MS HAR
TimeMAE.
SimMTM
80.03:10042
96.04-0888
95.96-4-077
80.20
67.764
15.06=t1..0
70.51++1.44
76.58:11170
75.62-088
TS-GAC"
88.81
99.01:+0.21
92.88-1-0.81
86.12-+1.37
88.03-:1.11
85.34++147
PatchTST(Mass)
39.76
1.41111:42
64.04415.3
60.47:::..1
65.35:::430
60.94:55166
PITS (Mask)"
84.89:1:16 66
98.21::0:2
88.08:1133
80.95:12.00
83.71:22.18
80.43:1::21
PatchTST (Ours)
84.96:::.08
98.01
86.76:22.06
80.71#286
82,66:2:883
30.43:::39
(Ours)"
89.33-10.86
Mixing-up
03
89.4 2:24
95.324:
88.82:40.57
81.76::0.72
97. 07
SimCLR
79.30-1-0.99
TS2Vec
24.06-
3.256-+33.5
96.17-22.02
89.75 1.88
83.82:::2.56
96.62++055
79.58
26
93.9511.50
87.25
09
9.791
TS-TCC
79.58:11.18
19.18:1:18
03.005::1.40
86.98110.75
80,51:1:0:00
94. 14:59911:47
PTB
SimMTM
80.75
62.6662:11 1.96
94.48:11.2
88.08-20.1
79.58:20.31
98.61::0.11
TimeMAE"
43
84.72:11.30
92.97-1083
86.78.10056
78.41:1:100
1.111
TS-GAC"
83.26:1:1.97
91.18:1:2.36
95.37:1:096
89.51 1.09
81.76:1:1.95
98,90110.2
PatchTSTT(Mask'
88.94443.50
79.50:+1154
PITS (Mask)"
81,48:1:1.50
81.98:13.18
90.77:1:1.79
88.22:1:0.77
81.53:1:1.40
96.13:10.2
PatchTST (Ours)"
78.50++-1366
76.04-1-647
86.79-4:536
88:67-4-0955
78.45:::162
96.83-4-.63
PITS (Ours)"
86.48 1..40
91.83 1.36
96.26:1:0.85
91.24:110.26
85.67:10061
97,58:10.86
Mixing-up
81.76122.94
96.46-10.50
82.43:22.80
79. 19..25:3..8 3.85
79. 79.99::3.83
19:18:13:99
SimCLR
87.10:10.M
98.25
92.09:1:101
88, 0.9702-10.79
89 89,42-1:084
08.20:10.84
TS2Vec
07.69-4-0.66
98.35-4-.15
93.93
89.92440.88
90,12++0.77
89.86-1-.42
TF-C
88.23:10073
08.66.1..13
92.90:1:100
89 89.09-1 1.11
89. 30:30:1:1 1.12
89 00.000:::1
TS-ICC
GL
SimMTM
82,05 +-033-
97,08-t00
08.06-400.4
83, 53.96---0.50
84.63-+0.64
83.63-t-0.77
TimeMAE"
83.44-1:100
97.81
89.84
85.26
16.10:222000
84.7741.55
TS-GAC"
92.32-1-0.55
99.04-+-.38
96.233+064
92.79-14.54
92.67-t-065
93.23-1-.92
PatchTST(Mask)
88.43:1:044
98, 98.79440.11
94.34:10.0
89.87:1-0.3
89, 09.73:2::45
90.1 90.15::044
PITS (Mask)"
89.29:10.76
98.79.10.11
93.62:1:045
89 09.04411.40
89.72:1:.51
88.7510.35
PatchTST (Ours)"
99,28:10.10
96.83-1-0.44
93.67::-0.45
93.54-10.37
93.90-10..6
PITS (Ours)"
92.07-10.78
99,28:10.09
96.87-19.46
92.77=0003
02.99-10.70
92.83-1065
28

---

Published as a conference paper at ICLR 2025
K MORE FINE-TUNING RESULTS
Table 12: Full fine-tuning results Gilon. first perform self-supervised pretraining, followed by full fine-tuning the whole network with 100%, 10%, and 1% of training data in Gilon task. Best scores for PatchTST and PITS are in red bold and the top scoring value is underscored.
Fraction
Models
Accuracy
AUROC
AUPRC
FI
Precision
Recall
Mixing-up
93.92:1:085
98.85-00199
95.18::107
91.71::0998
91.55=1.00
92. 0:8.5
SimCLR
95.7811036
99.6420.03
98.5220.15
95.47.10.8
95.6622051
95.40
TS2Vec
91.77:1:0.5
98. 0..1
94.04-11.88
90.28:1:.26
00.52::1.29
90. 10.17::1.25
91.9841022
99.23-00:88
05.60
91.45:1:071
91.322:0.73
91.77=0.72
TS-TCC
94.58
99.39:00.06
97.38=0030
92.61+1:000
93.39=:0.92
92.61 =0.91
SimMTM
94.83-t:104 1.04
98.99-20.44 4M
1..27
93.44:::.29
93.62.111 1.28
93.57=1.044
100%
TimeMAE
95.63:10:11
05
97 0.556
94.611:0.55
94.41
94.60 24.60
TS-GAC
96.43++0552
99.71
98.38
96.20:t0.55
96.35::0.09
96. 09:4420:22
PatchTST (Random Init)
95.01:44.20
99.61 0:0.09
98.58 +.440
95.524-2.4
96.15:+1.00
95.60+2261
PatchTST (Mask)
97.12:1:.49
99.73-00.04
99,00
96.94:10.57
97.30
97.09
PatchTST (Ours)
99.76
96.70++0.71
97.03-+0.37
9..00=0.50
PITS (Random Init)
93,40:4:0:3
99.54:20.11
97.58
92.96:1:0599
92.76::0.67
9..73
PITS (Mask CI.)
91.44:1:84
99. 09
95.06111.27
90.934
90.83 1.48
9..73
PITS (Ours)
95.53-1-0.83
99.79
98.89 0-337
95.25 0.884
95.43 0.89
95.25 0.76
Mixing-up
92.85:100.69
98.74:10.26
1,14
90.44:1:091
9..59
0.99
SimCLR
84.55.10.7
98. 98.47:00.00
91. 01.74
83.60:1:.21
86.78:20.62
82, 22..56:1:1 1.16
TS2Vee
88.12:4:1.58
96.46-0.78
1..37
85.51-11.13
85, 0..86:20.69
85,97
TF-C
83.35:10448
97.96=20.0
87.87 0.336
82.73:10488
03.95
82. 10:00
TS-TOC
93.69:::.00
99.41 10.19
97,36:20.76
92.11:10.84
93.69
91. 0.883
SimMTM
51.94:10.58
98.9520.35
95.65
91.35.10533
20.44
91.40=0.65
TimeMAE
90.06-+-2.95
08.77-40.43
94.65 +.204
91.10-4:2.54
91.50:22246
90.83--2.65
10%
TS-GAC
99.50-20.05
97.98
94.20:1:160
94.47:11.77
94,25=1.41
PatchTST (Random Init)
88.92:13.54
99:25:10.09
96.43-20.59
90.44:1.:4
91.68:11.0 1.51
90.55
PatchTST (Mask)
91.61:1032
99.35-10.11
97.10:10:47
92.33:1:.899
92. 08:10.00
92.47:10:75
PatchTST (Ours)
93.26:111.7
99.50=0009
11
93.97:1:.440
94.74:: 1.23
94.27
PITS (Random Init)
87.91
98.58:20.23
91.47
88.36:42.11
87.22
1..58
PITS (Mask + CL)
85.114-3.78
98. 08.48
89.51 22663
85.67 4-222
84.61 ::2.5:
84.60--2.65
(Ours)
99.48-00112
Mixing-up
84.82442.17
91.27=0053
87.48 4:
82.08 --
83.76:+2.52
31.53
SimCLR
62.61:11.89
90 90.88:22.08
66.00 4.228
47. 17-4551-456
63.15:29.35
2..63
TS2Vec
77.41-1:133
96.17::0:45
82.84 1.67
75.17:42.85
79.04 1.10
74. 3.00
65.344:2.50
91. 19.19:11.50
11.15
52.88 98
71.92:14.11
32.95=3.65
TS-TCC
85.77:+1.08
97.82:20.25
89.85 1.19
83,02:1:1.16
86.31
83.04=1:1
SimMTM
78.44:22:00
94.9 94.93-20.87
82.75 1.34
79.48:1:.05
80.66:12.40
79.31 1:1:11100
TimeMAE
96.24-20.3
1%
TS-GAC
10.57-
97.28 1.67
87.57
12.99-119.7
82.01
74.02=11.5
PatehTST (Random Inil)
82.63-4-2.89
97.89:10.74
89.84+2202
85,16:4:2.35
82.78:14..8
83.07
PatehTST (Mask)
80,35-4:2.29
96.77=1101
86.44 2.68
83.26-422.11
81.50 50:55833.58
81.52=3.56
PatchTST (Ours)
98.08
86.92-11.46
PITS (Random Init)
74.25-:3.0
96. 15:6616
78.73
75.27-4:424
71 41.45:24774
72.08-23.65
PITS (Mask CL)
95.40=#0.60
75.92
69.83-43.77
PITS (Ours)
81.064::-136
97.68=00.00
87.26::1662
83.71:10.95
81.25 1.45
82.05=1.30 1.30
29

---

Published as a conference paper at ICLR 2025
L ACF-COS
L.1 PURPOSE OF DESIGNING ACF-COS
An autocorrelation function (ACF) calculates the correlation coefficients between time series and its own lagged versions at various intervals. Here, ACF provides an overview of the structured linear patterns within the time series, revealing specific lags where these pattems recur. For example, when applied to a time series from force sensor smart insole, the ACF shows regular peaks that may correspond to step frequency, indirectly providing information regarding the user's pace. As such, ACF provides a meaningful overview of time series's structural dependency and order patterns.
Accordingly, we designed the ACF-COS metric, which measures the cosine similarity between the ACF of time series and its patch-permuted version to quantify the order information present. To our knowledge, no existing metric systematically quantifies the amount of order information in a time series. The closest relevant metric, the permutation entropy (PE) (Bandt & Pompe, 2002), measures the frequencies of particular symbolic pattern to occur within the series. This diverges from objective of measuring "orderness" by focusing instead on pattern frequency rather than the structural order itself. The intuition behind ACF-COS is that the ACF of time series with "orderness" should strongly diverge from the ACF of its patch-permuted version, leading to smaller ACF-CoS scores. On the other hand, time series with no order information (i.e., white noise) should have similar ACF score for its patch-permuted version, leading to high ACF-COS scores.
Discussion on Weak Stationary Assumption of ACF-COS. ACF has been primarily used in the time series forecasting domain, where they analyze the absolute values of the ACF to understand trend and seasonal patterns across different time windows. Holding the weak stationary assumption (having constant mean and variance) is important for autocorrelation estimates in such scenarios. The assumption enables the comparison of the ACF against different time steps or datasets. Unlike such comparisons made with varying steps of time or datasets, our ACF-CoS does not rely on absolute values of ACF but on the similarity between patterns. Even if the data are non-stationary, our primary objective is to quantify how autocorrelation structure changes when the patches' order is shuffled. As such, we are less concerned with the stationary assumption (which may be affected when we deal with the absolute values) in calculating ACF-CoS.
L.2 FULL ACF-COS SCORES
We provide the detailed ACF-COS statistics in Tab. 13. The ACF score, calculated for 17 tasks from the UEA Repository, the average score across three different patch sizes (Tab. 14). The ACF Range provides the minimum and maximum value of ACF-CoS score. T and D is the length of the time sequence and number of channels, respectively. The AUROC Win (%) metric shows how often our method (PPT) surpassed the cross-entropy (CE) based method by at least 1 1 in the AUROC metric. Similarly, the Accuracy Win (%) measures how frequently Method outperformed the CE method in accuracy. A total of 162 different hyperparameter configurations were tested for each dataset.
L.3 HYPERPARAMETER CONFIGURATIONS FOR ACF-COS CALCULATION
We calculated the ACF-CoS score based on the hyperparameter configuration Tab. 14. We selected a patch size allowing between 10 to 40 patches per dataset. The batch size was determined by the number of training samples. The hidden dim is the size of hidden dimension in PST. Strong y is the lower bound of strong permutation frequency used in PPT training. We used two different learning rates throughout the whole training: 0.02 and 0.002. For the EigenWorma dataset, data sampling necessary due to its large size. The total combination of hyperparameters are 162.
30

---

Published as a conference paper at ICLR 2025
Table 13: ACF-COS Statistics
Dataset Name
ACF-CoS
ACF Range
T
D
AUROC (%)
Acc. Win (%)
#Acc. Wins
ShapesAll
0.449
0.239 0.660
512
44.44
62.96
102/ 162
0.423
0.262-0.58
945
35.19
46,30
75/
Cticket
0.418
0.261 0.576
1197
6
61.73
75.93
123 162
OSULeaf
37.04
CricketX
0.416
0.280-0.555 0.552
300
27.16
46,91
76/ 162
CharacterTrajecttoo
0.410
0.207
120
44.44
53,09
86/
CricketZ
0.400
0.253 0.547
300
25.31
45.06
73/ 162
CricketY
0.397
0.272 0.522
300
48.77
57.41
93/ 162
Planc
0.374
0.175
144
37.04
7
Handwrting
0.335
0.239 0.431
152
46.30
51.85
84 162
Eigen Worms
Earthquakes
0.262
0.209 0.316
512
43.21
42.59
69 162
Blink
0.223
0
510
45.06
43.21
70
Libras
0.218
0.091 0.345
45
30.25
46.91
76. 162
NATOPS
0,216
0.087 0.346
31
24
53,70
75,31
122 162
LurgeKaachenAApllianess
720
GestureMidAArrD
0,186
0.104 0.268
360
40.74
48.15
78 162
WalkingSirrng
GestureMidAir22
0,166
0.099 0,233
360
37.04
46,30
75 162
GunPointOldd Versus Young
0.105
0.028 0.188
150
3,09
7.41
12/ 162
ConnointAgeS
0.104
0.024-0.185
150
45.68
46.91
76/ 162
GunPointMale Fermssl
0.100
0.029-0.171
150
28.40
45.68
74/ 162
GestureMiJAirD33
37.65
EMOPain
0.026
0.018 0.035
200
30
40.74
33.33
54/ 162
Table 14: Hyperparameter Search Space for ACF-COS Calculation.
Name
Patch
Hidden Dim
Strong 7
learning
Data?
ShapesAll
13. 17,
4, 8, 16
16,
20,
0.02, 0.002
True
24,
32, 64, 128
16,
20, 40
0,02, 0,002
True
Cricket
30, 40,
4, 8, 16
16,
10, 20, 40
0.02, 0.002
True
OSULeaa
14,
8,
20, 40
0.02, 0,002
True
CricketX
8, 10, 15
4. 8, 16
16, 32, 64
10, 20,
0.02, 0.002
Truc
Frajectorii
128
20,
CricketZ
8. 10, 15
4.
16, 32,
10, 20
0.02, 0.002
True
CticketY
8, 10, 15
4. 8.
16,
20,
0,02, 0.002
True
Plane
3,4.7
16
16, 16,
20,
0,02, 0,002
True
Handwriting
5,
4,8,
16. 32, 64
10, 20, 40
0.02, 0.002
True
Eigen Worms
450, 600, 900
8.8,
20,
0.02, 0.002
False (3000 samples)
Eartiquakes
13. 17 26
4.8, 16
16, 32. 64
10, 20,
0,02, 0.002
True
Blink
Libras
1.2.4
8.
16, 32,
10, 40
0.02, 0.002
Truc
NATOPS
1.2,
4. B. 16
16, 32,
20, 40
0,02, 0.002
True
LungeeiicheeAppplanees
24,
32, 64,
16.
20, 40
0,02, 0.002
True
GestureMidAirt)
12, 18
4, 16
16, 32, 64
10, 20, 40
0.02, 0.002
True
WalkingSitinngStadding
5.7,
4.8. 128
0.02, 0.002
True
GestureMidAiit22
92, 18
16, 02,32, 64
10, 20,
0,02, 0.002
True
CunPooitOlddVeee Young
3.5.7
GunPointAgeSpun
3.5.7
8.
16,
40
0.02, 0.002
True
3.
8,8,
16, 32,
20,
0.02, 0.002
True
GestureMidAirD3
921 18
4, 8, 16
20,
0.02, 0.002
True
EMOPain
5.7.
32, 64. 128
16, 32, 64
10, 20,
0.02, 0.002
True
31

---

Published as a conference paper at ICLR 2025
L.4 STATISTICAL RESULTS
We provide the statistical results of fitting the ordinary least square (OLS) model and the robust linear regression (RLM) model on the correlation between ACF-COF score and Accuracy Win (%).
Table OLS Regression Results
Model:
OL.S Regression Squares
R-squared:
0.225
Method:
R-squared:
No. Observations:
24
F-statistic:
6,370
Residuals:
Prob (F (F-statistie)::
DF Model:
Log-L LigeLikellhood:
17.692
AIC:
-31.38
BIC:
22..03
Omnibus:
5,298
Prob((mniiuss):
0.071
Skew:
Kurtosis:
Durbin- Watson:
2.435
Cond. No.:
8.14
coef
std err
P>
[0.025
0.975)
const
0,3604
0.055
6.495
0.000
0,245
0.476
0,4730
0.187
2.524
0.019
Table 16: Robust Lincar Model Regression Results
Dep. Variable:
y
Model:
Ri.M HuberT HI
Dr Residuals:
22
Method:
Norm:
Scale Est.;
mad
Cov Type:
No. Iterations:
coef
std err
P>
10.025
0.9751
const
0,3870
0.039
10.022
0.000
0.311
0.463
2.819
M TIME SERIES FOUNDATION MODEL: GPT4TS
We applied PPT to GPT4TS (Zhou et al., 2024), a time series foundation model that fine-tunes specific layers of the GPT2 (Radford et al., 2019) architecture. GPT4TS employs patching the original time series sequences. However, we encountered limitations in fully exploiting PPT's capabilities with GPT4TS due to two primary factors: (1) GPT4TS, being based on a language model. assumes a univariate patch sequence. Consequently, in the original paper, patches were concatenated along the channel dimension for multivariate time series. This makes the use of feature-wise consistency and contrastive learning from PPT inapplicable. (2) GPT4TS utilizes overlapping patch setups on certain datasets, whereas PPT assumes non-overlapping patches. Overlapping patches can mix order information between patches, potentially reducing PPT's effectivenesss. Given these constraints, we reconducted the time series classification experiments on the UEA multivariate benchmark datasets using non-overlapping patch setup, described in the original paper. With GPT4TS, performed grid search using weight combinations of [0.1, 0.1). (0.1. 0.5), (0.5, 0.1), (0.5, 0.5)] to balance the losses between the consistency and contrastive terms.
32

---

Published as a conference paper at ICLR 2025
Table 17: Comparison of Mask and Reconstruction pretext task (each with 10 and 40% masking ratio) and PPT superviid linear probing setup, using GPT4TS model.
Dataset Name
Mask (10%)
Mask (40 %
PPT (Ours)
FaceDetection
67.5
67.4
66.4
SellRegulationSCCP
91.8
92.5
93.2
EthanolCooncentration
30.8
33.5
34,6
Handwriting
27.3
29.3
Japanese Vowels
97.6
97.6
98.4
Heartbeat
77.6
77.6
76,6
UWaveGestureLLibaa
85.3
84.4
85.6
SpolenArabicDigits
98.5
98.5
PEMS-SF
69.9
76.3
82.1
SeIfRegulationSCC2
57.2
56.7
56.1
Average
70.3
71.4
71.5
Tab. 17 shows the results for linear probing, where we compared PPT to mask and reconstruction pretext tasks. Here, first conducted self-supervised training with unlabeled training samples and consequently fitted a logistic regression for linear probing. We compared our work against mask and reconstruction pretext task with 10% and 40% masking ratio. We all other hyperparameters' and architectures fixed to make results comparable. We show that PPT leads to an average accuracy of 71.5% which is better or comparable to mask and reconstruct pretext task.
Table 18: Supervised Experiment of GPT4TS (CE) and GPT4TS (CE+PPT) Performance
Dataset Name
GPT4TS (CE)
GPT4TS (CE+PPT)
Performance Change (% p)
FaceDetection
67.6
68.2
+0.68
SellRegulationSCPI
92.5
92.2
-0.34
ElhanooConnenttrationn
30.0
+2.28
Handwriting
29.6
31.1
+1,41
Japanese Vowels
98.6
99.7
+1.08
Heartbeat
75.1
76.1
+0.98
UWaveGestureL.iirag
85.3
86.9
+1.56
SpokenArabicDiiitss
98.7
-0.23
PEMS-SF
81.5
86.7
+5.20
SellRegulationSCP2
54.4
58.9
+4.44
Average
71.3
73.1
+1.71
We also carried out a supervised experiment in which we incorporate PPT into the cross-entropy min- imization process. The results in Tab. 18 show that incorporation of PPT can lead to an improvement in the performance of the average accuracy from 71.3% to 73.1 %. In particular, this performance gain was achieved solely by using time-wise consistency and contrastive terms. Consequently, we anticipate further performance enhancements when PPT is integrated into time series foundation models capable considering patches in the feature dimension, such as PSt and PITS.
33


Reviews:
[{"review_id": "7EsUtGWjLO", "reviewer": "Reviewer_i3b6", "summary": "The authors believe that patch order awareness is very important in time series modeling and  introduce PPT (Patch order-aware Pretext Task) for time series classification. The authors also develop ACF-CoS metric for quantifying the importance of patch orderness and interpreting applicable to PPT scenarios. Experimental results demonstrate that PPT shows advantages in self-supervised, fully-supervised, and semi-supervised learning with sparse labels.", "questions": "- Time dimension patch order is intuitive, but channel dimension order is not easy to understand and it needs further discussion.\n- Although the author provides the consumed time of permuting a batch of instances, considering the need to store additional augmented samples and the need for additional calculation, it would be better if the author can provide the additional speed and memory consumption required by PPT.\n- Confusing content: In Tab. 4 caption: “a decrease in ACF-CoS leads to an improved likelihood of PPT being beneficial”. Should “a decrease” be modified to “an increase”? In my understanding, there is a positive correlation between the ACF-COS score and PPT's performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Work is novel and motivation regarding patch order is reasonable for time series analysis.\n- Well written, clear structure, easy to understand\n- Experiments are sufficient, and the analysis based on ACF-CoS is interesting", "weaknesses": "Rocket[1] and MiniRocket[2] are important baselines for time series classification tasks. I think considering them can make the experiment more comprehensive (including classification performance and algorithm efficiency). \n\n[1] Angus Dempster, et al. ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels. arxiv preprint arxiv: 1910:13051 \n\n[2] Angus Dempster, et al. MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification. arxiv preprint arxiv: 2012:08791", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtRgo1Z5I7", "reviewer": "Reviewer_o6fk", "summary": "The paper introduces a novel Patch Order-aware Pretext Task (PPT) for time series classification, focusing on the importance of patch order in time series data. Traditional patch-based models often overlook the critical order dependencies in time and channel dimensions, which can lead to suboptimal performance. PPT addresses this gap by generating supervisory signals through controlled channel-wise patch permutations, allowing models to learn time series' intrinsic sequential characteristics. The main contributions of the paper are as follows: (1) Patch Order-aware Pretext Task (PPT): This is the first pretext task specifically designed to enhance order-awareness in patch-based time series models. PPT leverages controlled channel-wise permutations to provide structural supervisory signals in both time and channel dimensions. (2) Two Learning Methods: Patch Order Consistency Learning evaluates the correctness of patch order by distinguishing between original and strongly permuted sequences; Contrastive Learning differentiates between weakly and strongly permuted sequences, helping the model capture finer order distinctions. (3) ACF-COS Metric: The paper proposes ACF-COS (Autocorrelation Function with Cosine Similarity), a metric to quantify the \"orderness\" in time series data, which serves as a pre-evaluation tool to determine if PPT would be beneficial for a particular dataset. (4) Performance Gains: PPT demonstrates significant performance improvements on various time series tasks, especially in scenarios where order-awareness is crucial, such as ECG and human activity recognition (HAR) tasks, with performance boosts of up to 7% and 5% respectively. By integrating order-awareness through PPT, the approach enhances the ability of time series models to utilize structural dependencies across patches, setting a new direction for patch-based time series analysis.", "questions": "1. **Is there a method that can be adapted for both classification and forecasting tasks in time series?** Specifically, is it possible to design a pretext task that effectively utilizes order information for classification while preserving trend continuity to support forecasting tasks? If so, can this approach balance prediction accuracy with the advantages of sequence dependency?\n\n2. **Does the ACF-COS metric demonstrate robust and broad applicability in measuring the orderness of time series data?** Specifically, can ACF-COS accurately differentiate between various types of time series tasks, such as high-noise sequences, non-order-dependent tasks, and strongly order-dependent tasks? Additionally, how can we ensure the stability of ACF-COS across different patch sizes, sampling frequencies, or data distributions to reliably evaluate the suitability of PPT for diverse time series datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Introduction of the Patch Order-aware Pretext Task (PPT)**: This paper presents the first order-aware pretext task specifically designed for patch-based time series models. Unlike traditional tasks such as masking and reconstruction, PPT captures essential order dependencies across time and channel dimensions, a distinctive feature of time series data. By leveraging this natural inductive bias of sequence ordering, PPT provides effective self-supervision signals that help the model better understand structural characteristics in time series classification tasks.\n\n**Implementation of Order Consistency and Contrastive Learning for Pretraining**: To achieve order-aware learning within PPT, the authors designed two learning strategies: order consistency learning and contrastive learning. Order consistency learning distinguishes between the original sequence and strongly perturbed sequences, validating the model's understanding of patch order correctness. Contrastive learning differentiates between weakly and strongly perturbed sequences, allowing the model to capture finer distinctions in order dependencies. Together, these learning approaches enhance the model's sensitivity to sequence order, helping it capture the critical ordering information embedded within time series data and improving performance on order-dependent tasks.\n\n**Significant Performance Improvement**: PPT demonstrates notable improvements across multiple time series tasks, particularly in scenarios where order information is critical. Experimental results show that PPT enhances accuracy by up to 7% in ECG classification and by 5% in human activity recognition (HAR), underscoring its effectiveness. By integrating PPT in both self-supervised and supervised settings, models are able to better leverage the structural dependencies in time series data, achieving superior classification performance. This advancement not only showcases PPT's value in practical applications but also highlights the potential of order-aware tasks in time series analysis.", "weaknesses": "**Limited Applicability to Non-Order-Dependent Tasks**: PPT is specifically designed to exploit order dependencies in time series data, which limits its applicability to tasks where sequence order is critical. For datasets or tasks that do not rely heavily on the sequential order of data (e.g., tasks with significant noise or random sequences), PPT may add unnecessary complexity without meaningful performance gains. The paper itself notes that PPT does not perform well in such scenarios, and a robust evaluation metric is required to assess its suitability beforehand.\n\n**Dependence on Hyperparameter Tuning for ACF-COS and Permutation Strength**: The effectiveness of PPT depends on the selection of appropriate hyperparameters, especially for the ACF-COS metric and permutation intensity (weak vs. strong permutations). However, the paper provides limited guidance on systematically tuning these parameters for different datasets, which could hinder reproducibility and practical implementation. Users may need to perform extensive trial-and-error testing to optimize PPT for specific datasets, which is resource-intensive and may not always yield consistent results.\n\n**Limited Success in Time Series Forecasting**: Although PPT shows promise in classification tasks, it is less effective for time series forecasting applications, where trend continuity is crucial. The patch permutation strategy disrupts these trends, making it challenging for models to capture long-term dependencies needed for accurate forecasting. The paper does not provide a modified approach for forecasting tasks, which restricts PPT’s utility and impact in a broader range of time series applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVANAz7U0k", "reviewer": "Reviewer_Z8Ek", "summary": "This paper introduces a new pretext task designed for patch-based time series models, focusing on the order of time series patches. The Patch order-aware Pretext Task (PPT) is aimed at improving time series classification performance by considering the sequential and channel-wise order of data patches. This task leverages controlled patch permutations to produce supervisory signals for training models to learn inherent order dependencies. PPT incorporates both patch order consistency learning and contrastive learning to handle weakly and strongly permuted patch sequences, achieving better model performance in various time series tasks.", "questions": "1. Could you provide more details on how the context vectors are generated by the auto-regressive model for the consistency loss? How is the architecture of this model designed, and how is it trained?\n2. Could you provide more details on how the hyperparameters $\\lambda$ are adjusted during training? What are the final results of the learned $\\lambda$ values at the end of training, and how do they affect the performance of the model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Novelty in time series pretext task: PPT introduces a novel pretext task that leverages patch order consistency learning and contrastive learning to improve time series classification performance. This is a significant contribution to the field of time series representation learning.\n- Flexibility and adaptability: By introducing both consistency and contrastive learning, PPT can be incorporated into various task settings, either self-supervised or supervised, making it a versatile method for time series classification.\n- New metric for evaluating the importance of orderness: The paper also introduces ACF-CoS, which measures the importance of orderness in time series data, providing a new metric for evaluating the effectiveness of PPT and potentially other time series models.\n- Strong empirical results: The paper demonstrates the effectiveness of PPT in improving time series classification performance across various datasets, showing significant improvements over existing methods.\n- Clear and well-structured presentation: The paper is well-written and structured, especially the explanation of the PPT task and its illustration with examples. The clarity of the presentation enhances the understanding of the proposed method.", "weaknesses": "- Unclear details on consistency loss: The paper do not provide more detail into how the context vectors are generated other than stating that they are generated by an auto-regressive model. More details into the architecture and training of this model would be helpful for better understanding the consistency loss.\n- Lack of discussion on hyperparameter $\\lambda$: The paper does not provide a detailed study or discussion on the hyperparameters $\\lambda$, which are to be \"learnable parameters\". In the paper they cited that these hyperparameters are calculated from the prediction of a separate model, but more details on how this is done would be beneficial. It would also be helpful to provide ablation studies on the effect of such dynamic $\\lambda$ on the performance of the model compared to fixed $\\lambda$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0PrYUck7Wv", "reviewer": "Reviewer_GFNC", "summary": "The paper proposes a self supervised patch order learning pretext tasks for time series classification. Experiments are shown to quantify the benefits of path order learning. An evaluation metric to measure importance of orderness is also proposed.", "questions": "See weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper offers a perspective on ordering of patches in self supervised TS learning, and presents experiments on some datasets to show benefits and drawbacks of the approach.", "weaknesses": "1. Lack of SoTA baselines, compare against and cite:\n\n- S. Gao et al, UNITS: A Unified Multi-Task Time Series Model, https://arxiv.org/abs/2403.00131, NeurIPS 2024\n- Yong Liu et al, itransformer: Inverted transformers are effective for time series forecasting, ICLR 2024\n- Shizhan Liu et al. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, ICLR 2021\n- Haixu Wu et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, NeurIPS 2021\n\nIn particular, UniTS has been shown to outperform PatchTST and other baselines in many setups.\n\n2. The authors do not consider tasks beyond classification. I'm not convinced that the method is truly better than others when considering the representations generated, for example no comparisons with SoTA is shown on anomaly detection or clustering tasks. See for example, TF-C Appendix G, which is a baseline method authors compare against.\n\n- Zhang et al, Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency, NeurIPS 2022\n\n3. Limited evaluations. Experiments are all one-to-one dataset. The authors do not consider one-to-many, or other settings to test whether the representations are generalizable to cross-dataset settings etc. \n\n4. The gains are not convincing since many other baselines outperform the proposed method in several metrics.The improvement over GPT4TS is marginal.\n\n5. It's not clear whether this method even works for diverse dataset pretraining settings as in UniTS (https://arxiv.org/abs/2403.00131). Since this method is mainly for pretext tasks, this is an important point to raise given we're in the era of foundation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)