Time Series Representations with Hard-Coded Invariances
Authors: Thibaut Germain
Chrysoula Kosma
Laurent Oudre
Venue: ICML 2025 poster (2025)
Keywords: Time Series
Invariances
Neural Networks
Convolutions
Primary area: general_machine_learning->sequential_network_and_time_series_modeling
TL;DR: Mathematical formalism of time series invariances and design of hard-coded invariant convolutions to commonly observed deformations.

Abstract: Automatically extracting robust representations from large and complex time series data is becoming imperative for several real-world applications. Unfortunately, the potential of common neural network architectures in capturing invariant properties of time series remains relatively underexplored. For instance, convolutional layers often fail to capture underlying patterns in time series inputs that encompass strong deformations, such as trends. Indeed, invariances to some deformations may be critical for solving complex time series tasks, such as classification, while guaranteeing good generalization performance.
To address these challenges, we mathematically formulate and technically design efficient and hard-coded *invariant convolutions* for specific group actions applicable to the case of time series.
We construct these convolutions by considering specific sets of deformations commonly observed in time series, including *scaling*, *offset shift*, and *trend*.
We further combine the proposed invariant convolutions with standard convolutions in single embedding layers, and we showcase the layer capacity to capture complex invariant time series properties in several scenarios.

Time Series Representations with Hard-Coded Invariances
Thibaut Germain Chrysoula Kosma
Laurent Oudre'
Abstract
2015) and dictionary-based classification (Middlehurst al., 2019), neural networks. Notable architectures are built upon recurrent layers, convolutional layers, and, more re- cently, transformers. More specifically, convolutional neu- ral networks (CNNs) consistently lead time series tasks such as classification and clustering (Ismail Fawaz et al., 2019; Tonekaboni et al., 2021). Often integrated with other modules, CNNs excel in feature extraction, while offering interpretability through kernel weight visualization.
Automatically extracting robust representations from large and complex time series data is becom- ing imperative for several woal-world applications. Unfortunately, the potential of common neural network architectures in capturing invariant prop- erties of time series remains relatively underex- plored. For instance, convolutional layers often fail to capture underlying patterns in time series inputs that encompass strong deformations, such as trends. Indeed, invariances to some deforma- tions may be critical for solving complex time se- ries tasks, such classification, while guarantee- ing good generalization performance. To address these challenges, we mathematically formulate and technically design efficient and hard-coded invariant convolutions for specific group actions applicable to the case of time series. We construct these convolutions by considering specific sets of deformations commonly observed in time series, including scaling, offset shift, and trend. We fur- ther combine the proposed invariant convolutions with standard convolutions in single embedding layers, and we showcase the layer's capacity to capture complex invariant time series properties in several scenarios.
Advances in invariance modeling for deep learning have been achieved with proper mathematical formalism of group action on data, notably images and graphs (Kvinge et al., 2022; Bronstein et al., 2021). Two strategies for incorporat- ing invariance into deep networks are learning them through data augmentation, such as contrastive learning (Antoniou, 2017) or hard-coding invariances directly within the net- work architecture. Translation in CNNs and permutation in graph neural networks (GNNs) are two prominent examples of hard-coded invariances in terms of architectural design, among others (Bietti & Mairal, 2019; Horie et al., 2020). Relevant works range across sets, images, point clouds, and graphs (Zaheer et al., 2017; Keriven & Peyré, 2019).
Incorporating knowledge on time series invariances during training is an emerging field of study. Several works focus on learning invariances, e. g., contrastive learning that lever- ages augmentations and customizable losses (Franceschi et al., 2019; Eldele et al., 2021). Yet, the selection of views within the time series collection to contrast, as well as the types of transformations to consider (e. g., scaling, shift- ing), is often arguable within the research community (Yue et al., 2022). While contrastive learning constitutes an im- plicit way of introducing invariances into learning, very few studies leverage time series hard-coded invariances. In this work, we aim to embed invariances into the network design, similar to approaches that achieve permutation invariance in graphs (Maron al., 2018) and local translation and ro- tation invariance in images (Weiler & Cesa, 2019). Such invariant networks offer improved generalization properties compared to their learned counterparts at the expense of additional computational costs (Kvinge et al., 2022).
1. Introduction
Recently, there has been a growing interest in applying machine learning time series data, further necessitated by the challenging properties of time series, including varying modalities, high noise levels, and distribution shifts.
Machine learning for time series has gradually moved from statistical methods, c. c.g.. autorcgressive models (Box et al.,
'Equal contribution Universiiéé Paris- Paris-Saclay, Université Paris Cité, ENS Paris Saclay, CNRS. SSA, INSERM, Centre Borelli, F-91190, Gif-sur-Y fif-sur-Yvette. France. Correspondence Thibaut Ger- main <tthibut,, germain waress-prr---sallyfffse sar,ay.frs,, Chrysoula Kosma Mttrysoulakkosme @ens- -ereeeerni--sseeeeeeeeeeee paris-saclay.fre
Enforcing deformation-sspecife invariances in network de- sign can significantly improve performance for time series tasks. For example, the trend in time series constitutes
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Time Series Representations with Hard-Coded Invariances
Kernel
Electrocardiogram
Cnomal
CeT
Figure I: Top right: Segment of a electrocardiogram (ECG) from the MIT-BIH dataset (Goldberger et al., 2000; Moody Mark, 2001). Top left: The convolutional kemel is the first heartbeat. Middle: With the normal convolution, individual heartheats are not identifiable as they are blurred by the trend. Bottom: With linear trend invariant convolution, all heartheat occurrences are identifiable they are positively correlated with the kernel, minimizing deformations induced by the trend.
common deformation, and removing its effects is an active research area known as baseline removal (Hippke et al., 2019; Zhang et al., 2010). Modern time series machine learning pipelines often perform baseline removal as a pre- processing step (Back et al., 2015; Yan et al., 2019; Zhang al., 2020b). However, these methods usually require tensive hyper-parameter tuning (Zhang et al., 2020a; Singhal et al., 2020). In deep learning, this challenge has been tack- led through trend modeling components (Oreshkin et al., 2019), which has also been combined with contrastive learn- ing (Liu et al., 2024; Woo et al., 2022a). Interestingly, convolution- based networks can leverage hard-coded invari- ant filters to accurately approximate trend invariance. In- deed, convolutions focus on local information, and similarly to spline functions, the trend can locally be approximated by low-degree polynomial functions, which, in its simplest form, can be seen as linear (1-degree). Figure illustrates the cross-correlation between a single heartbeat and an ECG affected by a trend assuming a standard convolution and convolution invariant to linear trend. Surprisingly, the standard convolution fails detect the correlation between the query heartbeat and the individual ones. On the contrary, the linear trend invariant convolution successfully identi- fies correlations to all underlying heartbeats and potentially offers more robustness for any ECG diagnosis (Liu et al., 202
actions and introduce invariance under these actions. We then design generic and exact hard-coded invariant convolu- tions capable of handling deformations such as trends.
We highlight the sensitivity of standard convolution to common deformations and demonstrate how deformation informaatioonnnnriaant achieving better generalization than learned invariance.
Section 4.1.
invariant convolutions mitigate this issue,
Section 4.2, 4.3 & Appendix A.5.1. We showcase the effec- tiveness of the proposed convolutions against state-of-the- baselines across multiple tasks. The results demonstrate that our hard-coded invariant convolutions offer fast, gener- alizable, and robust approach to time series representation learning.
2. Related Work
Deep Learning for Time Series. Dominant deep learning frameworks for time series leverage multi- layer perceptrons (MLPs) (Oreshkin et al., 2019), convolutional networks (CNNs) (Bai et al., 2018) and recurrent ones (RNNs) (Sali- nas et al., 2020), as well as transformer-based networks (i. e., built upon the attention mechanism) (Wen et al., 2022), Convolutional kemnls have traditionally dominated feature extraction in time series, from shapelets (Ye & Keogh, 201 1) to the recently successful ROCKET (Dempster et al., 2020), that exploits several random kernels. Additionally, convolu- tional layers of different kernel sizes, often stacked in deep architectures (Ismail Fawaz et al., 2019), with increased receptive fields, such as INCEPTIONTIME (Ismail Fawaz et al., 2020) and RESNET (Wang et al., 2017), are promi- nent for time series classification. Similarly, TIMESNET (Wu et al., 2022) model capitalizes on convolutional lay- ers to capture variations of multiple periodicities of 2D transformed multivariate time series, to solve multiple time series tasks. Beyond standard CNNs for time series, T- WaveNet (Minhao et al., 2021) is tree-structured wavelet neural network that decomposes the input signal into var- ious frequency subbands with similar energies based on
Following the previous observations, our work tackles the challenge of hard-coded spatiotemporal invariance within convolutional layers for time series accounting for deforma- tions like offset shift or linear trend and extending beyond mere time invariance. In the literature, very few works deal with hard-coded time invariance, and most are concerned with local warp-ng invariance (Shulman, 2019) or time rescaling invariance (Jacques et al., 2022). addition, our mathematical framework provides exact formulation for invariant convolutions, as opposed to approximating invari- ance in previous works. More specifically:
Section 3. We formulate time series deformations via group
2

---

Time Series Representations with Hard-Coded Invariances
the dominant frequency range. Another recent hierarchi- cal CNN-based model for time series forecasting, SCINET (Liu et al., 2022), repeatedly downsamples and convolves the input to enable information sharing at several resolu- tions. Furthermore, leveraging the success of the attention mechanism in text, transformer-basee architectures have lately proven successful capturing temporal interactions between multivariate time series inputs, mainly in time series forecasting (Zhou et al., 2021; 2022; Woo et al., 2022b; Liu et al., 202 For instance, AUTOFORMER (Wu et al., 2021) combines decomposition modules with an auto-correlation in place of self-attention, while CROSS- FORMER (Zhang& & Yan, 2022) capitalizes on 2D vector ar- ray embeddings that preserve temporal and channel informa- tion, followed by temporal and channel-wise tross-attention modules. By merging attention with masked autoencoders, UP2ME (Zhang al., 2024) applies univariate pre-training followed by multivariate cross-channel fine-tuning to gener- alize across multiple tasks. In parallel, several recent works, evaluate forecasting architectures also on the anomaly de- tection (Xu, 2021), by reformulating the task to point- wise reconstruction, with reconstruction error being the anomaly criterion. To overcome the sensitivity of transformers in overfitting, recent MLP-based architectures have showcased superior performance in forecasting, e. g.. TSMIXER (Chen et al., 2023), TIMEMIXER (Wang et al., 2024a), FRETS (Yi et al., 2024), while several studies doubt the robustness of the former in time series modeling (Zhang et al., 2022: Zeng et al., 2023). Notably, forecasting methods often per- form trend-seasonal decomposition with moving averages, such PERI-MIDFORMER (Wu et al., 2024), which applies self-attention between multiple periodic features. How- ever, moving average kernels mainly address offset shifts, contrary to our more principled and flexible approach to deformation invariance (e. g., linear or higher-order trends).
Deformed Original
[e) Amplitude scaling
(b) Offset shift
(c) Linear trend
Figure 2: Deformations applied an example series, in- cluding amplitude scaling, offset shift, and linear trend.
(Esling & Agon, 2012). For instance, local time warping invariance (Ding et al., 2008) can be tackled by dynamic time warping (DTW). Traditionally, amplitude and offset invariances are accomplished by Z-normalizing the data (Pa- parrizos al., 2020). However, while Z-normalization re- moves global offsets, it fails to address local distortions. The LT-normalized distance (Germain et al., 2024b) extends Z- normalized distance by modeling invariance to linear trends. Diffeomorphisms have also been leveraged for shift invari- ance in neural networks through a differentiable bijective function mapping between time series manifolds (Demirel & Holz, 2025). Closely related, authors in (Germain et al., 2024a) propose methods to quantify deformations like time- warping through modeling temporal deformations as diffeo- morphisms acting on time series.
3. Method
In real-world signals, constant baseline shifts, slow drifts and long-term changes can arise by external factors and dis- tort underlying signal patterns. Therefore, amplitude scaling, offset shift, and linear trends can be identified as common time series deformations, as depicted in Figure 2 on an ECG heartbeat. These deformations can serve as examples for the empirical validation of our broader theoretical framework of time series invariant convolutions. The proposed framework for learning time series representations with hard-coded invariances consists of two main components. The first component is group action that formalizes how certain deformations transform time series (Section 3. I), resulting in their deformed counterparts. Deformed time series are observable in practice, due to noise or trends. To ensure invariance specific deformations, that can be important for many applications, the second component is mapping function that constructs embeddings of deformed time series while remaining invariant to such deformations (Section 3, Finally, these embedding maps can be efficiently integrated into convolutional operations (Section 3.2) to extract robust local features to non-informative deformations.
Invariances for Time Series Modeling. Besides super- vised methods, multiple approaches emphasize unsuper- vised learning for extracting representations from time se- ries data before the downstream task (Nie et al., 2022; Dong et al., 2024). Among prominent approaches, self-super contrastive methods enforce invariances between representa- tions by applying transformations (e. g., scaling, shifting, or noise injection) to the input and training the model to map them to the same underlying representations (Chen et al., 2020). Typically, CNNs constitute basic blocks for vari- ous time series augmentation-baser contrastive frameworks, such as TS-TCC (Eldele et al., 2021) and TIMECLR (Yang augmenta-
et al., 2022). Except for transformation-base tions, samples can also be contrasted with sampled subseries (Franceschi et al., 2019), adjacent segments (Tonekaboni al., 2021), or combination of all (Yue et al., 2022). Un- like deep learning. time series invariances have long been central focus in classical time series data mining approaches
3.1. Invariant embedding for time series
We, next, construct an embedding invariant to a predefined set of deformations. Essentially, the embedding is expected
3

---

Time Series Representations with Hard-Coded Invariances
to map a geometrical object or any of its deformed versions to the same representative. In addition, we present some important properties that should verify the embedding and tailor the proposed framework to the case of time series.
Invariant embedding. An embedding invariant to group
action is expected to map any element of orbit to the same representative, and it is defined as follows:
Definition 3.2 (Invariant & orbit-injective embedding). An embedding map L M - N is said to be G-invariant, if for any (g. m) E G x M, L(g m) = L(m). Additionally, L is said to be orbit-injective if the application L [m] E M/G - L(m) E N is injective.
Deformations and group action. From a geometrical viewpoint, the notion of invariance depends on the represen- tation of deformations and the definition of the action of deformation on a geometrical object. classical approach consists of representing deformation as an element of group and its action by group action:
Note that an invariant embedding meaningful in the case of a non-transitive group action. In addition, if the embed- ding is orbit-injective, each orbit has distinct representa- tive.
Definition 3.1 (Group action). A group G with neutral e acts on the left on set M, if there exists a map G M M that verifies:
For now, we focus on the action of the finite-dimen miite-dimensional subspace H of a Hilbert space M by the usual vector addition: (h, m) E H x M + m + h E M. The following proposition exhibits a H-invariant embedding that is also orbit-injective. Proposition 1. Let PH be the orthogonal projector on H, and La be the identity map on M. the embedding, L = La PH (the projector on H is H-invariant and orbit-injective.
1) ale, m) = mm, Vm E M 2) a(,, m)) m), V(g.h) E G², Vm e M.
To simplify notation, the left action of g E on m E M denoted g m. For group G that acts on the left on set M, the orbit of
M is the set of all its deformed versions [m] {g- m 9 E G}. The set of independent orbits, denoted M/G, is called quotient space, and if this set is reduced to a singleton, the action of G on M is said transitive, and it verifies that for any m E M its orbit is the whole set: [m]] = M.
Proof. See Appendix A.1.
Remark 3.3. If (hx)sep.N1 is an orthonormal basis of the finite dimensional vector subspace H, then the orthogonal projector on H as an explicit formulation PH : m E M +
E (m, h.) E H.
group action for time series. Leveraging measure the- ory, we model the set of time series by the Hilbert space LI,,, p.) of functions defined on the closed interval I C R taking value in D and square-integrable for the Borel measure p. The inner product on IR,, (4) is de- fined as:
Invariance to amplitude scaling can easily be incorporated in an embedding defined by the previous proposition: Proposition 2. Let L M +> M be the H-invariant and orbit-injective embedding map induced by the orthogonal projector on H as defined in proposition 1. The embedding map:
(f.9)= = [U0))000)) -
where (..-) is the dot product on R., Let H be finite dimensional vector subspace of L (1, pl), we model the group of deformations as the set Ri x H with the composi- tion rule (A2, A2) x (A1,/ = A1 h2 + A211).. Finally, we model the group action by the application:
{ L(m)/IIL(m)ll if E MiH
L: m MEM-
(3)
0M
else
is (R3 Hi-invariant and orbit-injective.
Proof. (E. x H) -)-invariance is due to the linearity and H- invariance of L,, and the orbit-injectivity in induced by the linearity and orbit-injectivity of L.
(R H) LR 1)) f)
L2(1,RD,1)
+
(2)
+ Afth
This is a general group action that is not transitive as H is a finite-dimensional vector subspace (1,1, (R, (u). By convention, we refer to P H as the set of rigid de- formations. The customization of the group action de- pends on the choice of basis for the subspace H. For in- stance, the Z-normalization (Paparrizos et al., 2020) is an invariant offset shift which corresponds to the subspace of deformations {h- I c c E R²} with the ba- sis (h, : + e /// (lenthh() i E [I, where (e)).e(L....) the orthonormal basis of R.D
An example: the univariate Z-normalization. We are looking for an embedding invariant to amplitude scale and offset shift in the case of univariate discrete time series. The set of time series is modeled by L2( 1J, R, 11) where E No, 1 E/= 8, and 6; is the dirac measure at i. The set offset shifts is the subspace generated by the unit norm function e: t E [0,4] + 1/vl E R. According to Proposi- tion 2 the invariant embedding of a non-constant function f is the function: (f (f, ())//1) (f. eellz which
4

---

Time Series Representations with Hard-Coded Invariances
leads (f(1) - (1000((1)/ where n = -1- f(i) and oj
that N << L. Invariant convolutions do not consider small- size kernels (2 or 3 timestamps) but rather large kernels (30 or more). The traditional approach to convolution is not tractable in such a context. Instead, we leverage the Fast Fourier transform (FFT) (Mathieu et al., 2013), which changes the time complexity to O computational time is identical for any window size, as the computation with the FFT does not depend on the kernel size. In the experimental results, show that our proposed invariant convolutions benefit from fast computation.
3.2. Invariant convolution
CNNs have been successful in many applications related to time series, essentially becoming a key building block of the latest deep neural networks. Their success comes from their ability to capture local information in long time series. However, CNNs remain sensitive to some deforma- tions like amplitude scaling or off'st shifts (Mallat, 2016). In this section, we propose novel convolution that is invari- ant to rigid deformations at a local scale while remaining computationally efficient.
The
4. Experimental Evaluation
We present an extended experimental evaluation illustrating the use and performance of hard-coded invariant convolu- tions. As already presented in Section 3 (Figure 2), we specifically focus on convolutions invariant to constant func- tions (offset shift) HaI = {t E l - b b E IRP) or affine functions (linear trend) HLr f e at + (a, b) E RD x J.. Based on the discussion in the Introduction, these deformations are the simplest local approximation of trend deformation. In addition, we also include invari- ance to amplitude scaling, which is a common source of inter-individual variability. The experimental evaluation is organized as follows:
The formalism. Let oz (R, .14) be the set of signals, we assume that the signal in square integrable on any com- pact of IR. Let (2 14) be the set of kernels where I C E is closed interval. The classical convolution layer, named 1 ID-CNN, between signal f and kernel w is the signal:
P,,),
J / + (1((++),,00)
(4)
Let assume a group of rigid deformations G acting (1,1,, I)), and f. the G-invariant embedding map defined by Proposition 2. For any u E R. we can define the operator KS that maps the restriction of any signal / on the closed interval u- + to its G-invariant representative:
I. Robustness to Deformations. We experimentally prove the robustness of the proposed invariant convolutions com- pared to vanilla ones and contrastive-base methods on a classification task when considering deformations.
L2 R. R.P C
- L2(1.D
II. Classification Performance. We show the competitive performance of an example architecture built upon a pool of normal and invariant convolutions on classification for several benchmark datasets. We also perform ablations and computational efficiency studies for the proposed method.
(5)
f
+ Ltell f(t u))
Leveraging these operators we define the G-invariant convo- lution between a signal f and a kernel as the signal:
II.. Generalization Performance. We assess the robustness of the proposed invariant convolutions on a transfer learn- ing classification experiment against relevant contrastive learning methods.
f U.URR [ ((0))00 e (6)
Fast computation. For a group of rigid deformations (R1 H) with (he)eefNN a basis of H, thanks to Remark 3.3, the inner product between the invariant representation of f E LR ()) and w can be decomposed as follows: (L(J), w)) = ((f,w), E. (f, kw) (L)()) Assuming discrete sig- nals, the computation of (f(f), W)L requires the compu- tation of 2N + 2 dot products. However, convolving a batch of B signals of length L with the kernel, the number of inner products to compute drops from B2N + 2) to BL(N + 2) + N as the inner products between the kernel and the basis are shared across signals and subsequences. leads to the time complexity (BLNCWW) where C' is the number of channels, Wi is the kernel size and assuming
IV. Anomaly Detection Performance. We finally capi- talize on invariant filters to perform reconstruction based anomaly detection by introducing an example decoder that effectively combines different types of features to recon- struct the signal. The experimental protocol and the results are presented in Appendix 1.5.1, revealing the benefit of invariant convolutions.
Code and Experimental Details. The source code for this work is available on GitHub'. More information about the datasets and the implementation details for the proposed method and baselines can be found in Appendix A.3 and A.4, respectively.
'nttpa://githul com/ wwwe///sssssssssm//mmomemm www.://ooooos----eeeeee InvConv
5

---

Time Series Representations with Hard-Coded Invariances
Table I: Robustness study of distinct convolutional filter types with respect to invariance, e., standard ones (non-invariant) offset invariant, and trend invariant, on five UCR datasets. Models are trained on normalized data and tested on four additional synthetic deformation scenarios (random offset (off.), random linear trend (L.T), smooth random walk (RW) and their combinations). Higher better, best methods in bold, second best underlined. The self-supervised TS-TCC method is pre-trained with all deformation augmentations, and it is then fine-tuned (FT) on the raw normalized data. For the three convolutional variants, knowledge of the distribution of deformations is not incorporated during training, and thus, the model is evaluated in an out-of-distribttio setting.
CONY
INV. CONY -offset-
INV. CONV -rrend
TS-TCC (FT) (2021)
normal
-offet, trend-
Normalized
77.84 1 0.0
72.43
0.54
70.54 1 0.71
88.96 1.17
41.08
0.47
72.43
0.54
70.72 + 1.02
47.83
0.54
HandOalines
LT
37.39
0.16
71.08
1.08
70.90 1 0.82
40.87
1. 1.41
+ off.. LT
35.68 = 0.47
70.45
: 1.22
70.63 = 0.41
37,07
0.87
off.. RW
35,95 1 0.0
62.34 1.02
64.59 1: 1.24
35.82
0.15
Narmaliord
93,91 0.39
88.48
1.79
92.26 + 0.20
93.44 1.544+11.51 1,51
+ off.
38.96 0.99
88.41
1.83
92.24 + 0.73
92.56 1.31
MixedShapesRRgglarPaaa
-IT
29.33 2.16
84.63
3.92
91.55 + 0.69
82.96 4.72
+ off.. LT
28.22 1.49
84.05 1 4.08
91.64 1: 0.80
81.35 11 4,08
off. RW
4.62 76.98
90.59
68.56
Normulrued
91.35 0.05
86.53 + 0.06
85. +: 0.33
84,19 1.18
+ off.
15.32 0.44
85.11 : 0.73
83.84 +: 0.12
71.47 = 6.51
+: 0.23
42.49 0.33
82.56 +:
49.52 5.97
off.. LT
0.11
42.36
0.65
82.02 0.52
47.61 1 6.70
+ off. RW
3,48 + 0.14
0.94 37.22
73.33 + 0.05
31,55 + 6.16
Normalized
82.35 = 0.57
83.33 1 0.5
82.84 + 0.65
78.37 = 0.30
off.
60,91 3.17
81.61
1.38
83,37 1: 0.31
78.29 0.43
FordB
+ LT"
53.83 1.10
70.50 1 6.02
82.68 = 0.91
76.85 + 2.23
+ off..
53.04 0.25
69.47
7.34
82.35 +
76.49 2.83
off.. RW
50,91 0.26
68.80
8.21
80.37
0,43
75.77 + 1.67
Normalized
76.12 0.24
76.80 1 1.69
76.92 1: 0.14
80.64 1 0.63
off.
53.73
72.07
1,93
75.71
74.97 1.59
Yoga
LT
52886 0.14
67.71
2.99
74.17
:: 0.29
62.21 0.74
+ off.. LT
51.13 0.07
64.85 + 2.05
72.88 + 0.38
61.85 + 1.26
off., RW
0.74
72.58
0.19
+ off
48% 205
-1% 2%
0% 0%
-13% 1 17%
Percentage Drop (%) with respect Normalized (lower the better in abs, value)
-55%
-16%
17%
-26%
-57% + 22%
-18% 17%
-1% 2%
-28% + 20%
off. RW
-59% = 23%
-23% 16%
-6%
4%
-34% + 22%
4.1. Robustness to Deformations
ranges, as well as (iii) the addition of random trend with slope and intercept values sampled again with uniform prob- ability, (iv) combination of added random offset and trend and (v) combination of random offset shift and smooth ran- dom walk. For the last deformation, the added synthetic trend is a random walk generated from a Gaussian distribu- tion and smoothed by a rolling mean. In order to compare with learned invariances, we also include the prominent supervised contrastive learning method TS-TCC (Eldele et al., 2021). On each dataset, this model is pre-trained with contrastive losses leveraging the different deformation scenarios to capture invariance; then, it is fine-tuned for classification on the raw data. Its performances are also evaluated under the different scenarios.
Protocol. We aim to evaluate the robustness of trend defor- mations of hard-coded invariant convolutions (scaling/offe & scaling/linear trend) compared to vanilla convolutions and learned invariant representations with contrastive learning. Robustness is evaluated on classification task. We consider 5 datasets from the UCR archive (Dau et al., 2019), which, by default, are all Z-normalized. Concerning baselines, we include three versions of inception- (Ismail Fawaz et al., 2020) networks (a single convolution layer with ker- nels of four different sizes followed by linear classifier) whose architecture is presented in Appendix A.2. The first one, CONV (normal), only includes standard convolutions, the second one, INV. CONV (offset), only includes ampli- tude/offset invariant convolutions, and the last one, INV. CoNV (trend), only includes amplitude/linear trend invari- ant convolutions. These three models are only trained on the raw datasets and then tested on 5 different scenarios: (i) no udditional deformations, (ii) the addition of random offset, sampled from uniform distribution between specific
Results. Table I displays accuracy scores, which are orga- nized by scenarios of increasing magnitude of deformations, starting from (i) no additional deformations and ending with (v) offset shift and smooth random walk, the closer one to trend deformation.
6

---

Time Series Representations with Hard-Coded Invariances
In the first scenario (normalized data), there is no deformatioo-relatted distribution shift between the train and the test sets. On average, CONV (normal) and TS-TCC perform slightly better than the hard-coded invariant net- works INV. CONV (offset) and INv. CONV (trend). Outside of the HandOutline dataset, the performance difference be- tween the two groups does not exceed 5.0 points of accuracy. This slight performance difference could be attributed to small class amplitude, offset, linear trend dependencies. However, as soon as deformations are added (even small), CONV (normal) and TS-TCC performances drop signif- icantly. In contrast, the performance of the deformation invariant network INV. CONV (trend) remains constant and becomes the best performer most cases. On average, INV. CONV (trend) performs better than TS-TCC by 16.0 points of accuracy.
overcome by leveraging hard-coded invariant convolutions, which also benefit from better generalization properties than invariance learned by contrastive learning.
4.2. Classification Benchmark
Pool of convolutions. The choice of invariances is often related to the application (Yue et al., 2022), and setting the invariances by hand requires a good understanding of the nature of the signals. In the absence of such knowledge, as in classification benchmark, we decompose the space of deformations H = θ k H, in the direct sum of subspaces such that the cumulative sums, 0 C H1 C H₁ + H2 C
C H. represent sets of deformations of increasing order of complexity. We consider layer that concatenates of n (+ H:)-invariant convolutions for j E (1...., K) and no standard convolutions, In our specific experimental framework, the linear trend deformations HEr is the direct sum of offset deformations H, = (1 E 1+ b b E RD) and purely linear deformations H₂ {t 1+ at la e e Il- lustrations of the proposed pool of convolutions is provided in the Figure 4 of Appendix A.2. Other pooling strategies, like attention mechanism or reinforcement learning, are possible and left for future work.
Regarding the specificities of each model, CONV (normal) is the most affected by deformations. Its performance drops by 48% when adding minor deformations (offset) and goes up to 59% with smooth random walk trends. This sug- gests that convolutions are highly sensitive to deformations, even small ones, commonly observed in time series. Ap- pendix A.5.2 clearly illustrates the sensitivity of CONV (nor- mal) by comparing the feature maps of its convolutional filters with those of INV. CONV (offset) and INV. CONV (trend) on the same time series. When the time series un- dergoes deformations, the feature map landscape of CONv changes drastically. In contrast, its invariant counterparts (INV. CONV) maintain a consistent structure according to their invariance. TS-TCC is the second most affected model. On average, its performance drops by at most 34%, suggesting that the learned invariance is only partial and does not generalize well on new observations. In contrast, convolutions with hard-coded invariance behave accord- ing to their properties. INV. CONV (offset) performance remains constant when offset deformations are added but decreases when more complex deformations are added. In- terestingly, the performances are equivalent between the linear trend scenario and the offset linear trend one, com- forting the invariance property to offset. INV. CONY (trend) performance remains almost constant on scenarios includ- ing offset and linear trend deformations ((ii)((ii,(ii))) Note that the slight performance variations in these three scenar- ios are due to some padding effects. In the worst scenario, including smooth random walk trend, the performance remains quite constant; it drops on average by 6% compared to 59% for CONV (normal) and 34% for TS-TCC. This last scenario is the closest to any trend deformation, indicating that INV. CONV (trend) is well suited for classifying time series affected by trends.
Datasets. We consider the 26 multivariate UEA datasets (Bagnall et al., 2018), coming with a standard train/test split. We also consider 3 additional datasets, the human activity recognition UCIHAR (Anguita et al., 2013) dataset, the Sleep-EDF dataset (Goldberger et al., 2000) for sleep stage classification of EEG signals, and finally the epileptic seizure recognition Epilepsy dataset (Andrzejak et al., 2001). For these datasets, we follow the same preprocessing with (Eldele et al., 2021), deriving train/validation/eest sets of 60 : 20 20 ratio. Appendix A.3 provides additional details on data splits and preprocessing.
Baselines. We select twelve state-of-the-art models for time series classification. TIMESNET (Wu et al., 2022), PATCHTST (Nie et al., 2022), CROSSFORMER (Zhang & Yan, 2022), DLINEAR (Zeng al., 2023), (TRANSFORMER (Liu et al., 2023) and TIMEMIXER (Wang et al., 2024a) are derived from the Sine-Series-Librare (Wang et al., 2024b), Additionally, PERI-MIDFORMED (Wu et al., 2024) model is adapted from its official code repository. We also compare to the CNN-based backbone of the self-supervised method TSLANET (Eldele et al., 2024) and three powerful CNN ar- chitectures, namely INCEPTION (Ismail Fawaz et al., 2020). RESNET (Wang et al., 2017). and CNN (Ismail Fawaz et al., 2018) build upon standard stacked ID-CNNs. We also in- clude the state-of-the-art machine learning method ROCKET (Dempster et al., 2020). Finally, our model INVCONVNET combines a single inception like pool of convolutions layer of four different kernel sizes, combining balanced sets of
Conclusion. Standard convolutions are sensitive to spa- tiotemporal deformations. This sensitivity issue can be
7

---

Time Series Representations with Hard-Coded Invariances
Table 2: Classification Accuracy (%) for all considered datasets. Accuracy is averaged for datasets from UEA repository.
Higher is better, best methods in bold, second best underlined.
IocCowoNer (rund)
TIMESNET
PRiCHTTT
CRoSSHORMEA (N07))
TSLANET
TIMEMMXKK (20040)
INcUETTIN
RocKEt
(2025)
(2017)
(2018)
(20120)
UBA datasets)
66.87
1.36
66.37
64.42 2.05
60,87 3,00
1.1
61.1 #
66.00 1.09
1.56
67.37
1.59
05,67
71,294 0.90
UCIRAR
96.63 =: 8.49
91.66 0.62
55.74 2 0.50
93.43 à 0.56
94.33 a 0.08
92.64
0.57
94.71 = 0.65
57.47 0:773
84 + 1.07
95.26 a: 0.55
96.04
+ 0.48
95.78 0.20
92.06 0.15
Steep EDD
+.339
Te.4 0.73
78.53
0.28
79.22 0.89
53.31 0:2.1
63.50 1.52
54.98-044 0.43
36.15 0022
7-
+ 2.05
84.00
0.39
85.62
+ 0.13
82.41 +
33.85 0.09
Riilejay
98.43 9:84
97 0.00
98.11 + 005
98,33 0.12
97.50 0.36
47.99
0.29
14.3-2000
K2.36 0006
97.44
8.18
98.16
0.04
97.61 #.0.8
48.35 = 0.00
Table 3: Classification ablation study of filter types in INT. including solely normal ones (-N), offset-invariant (-O) ones, or trend-invariant ones (-T).
Accuracy vs Computational Cost
77.5
Model Size
75.0
InvConvNet
200K
TSLANet
Accuraci 69
72.5
In Prpoppioo 4K)
N NeTT
NET-O
N
3M
70.0
Crossformar
10M
UEA (25
71,81 0.50
67.70 1.21
2.58
DLinear
67.5
(187x 378,310 13 12.0% 320 (A11 1. 1780
DEMMAN
96.63 9.49
M5.15 6
95.75
0.35
Times [ 12000, Rd5 AK]
Ruilgpe
84.95
98.43
$4.70 0.39
98.36
83.73
0.13
65.0
CNN (2.00s. 72811)
ITranst.
625
Pari-mid 1250
(7.ds 14 (4.0
TimeMixer
1.8
2.0
222
2.4
2.6
7.7
standard, amplitude/offset and amplitude/linear trend in- variants convolutions. A linear classifier follows the pool of convolutions. Implementation details are presented in Appendix A.2.
Training Time per Epoch (sec)
Figure 3: Models' costs for Heartbeat dataset classification.
advantage in terms of training time and memory cost, as presented in Figure for the UEA Heartbeat dataset.
Results. Table 2 presents the classification performance of the proposed INVCONVNET against the twelve considered baselines, evaluated on UEA and the 3 additional datasets. All models are trained and tested for 3 runs with random seeds, and the average accuracy with its standard devia- tion is reported. Full classification results per dataset on UEA, along with a critical difference diagram based on mean ranks, are provided in Appendix A.5.3 (see also Fig- ure 10). We observe that on UEA repository, INVCONVNET has the best average test classification accuracy, followed closely by ROCKET, and both algorithms outperform the rest deep learning approaches. ROCKET'S use of thousands of random convolutional kernels enables effective feature extraction, making it highly robust on smaller UEA clas- sification datasets. Although InvConvNet has the highest mean accuracy, ROCKET is the best method in first-rank counts, but in the critical difference diagram both show sta- tistical equivalence in mean rank. Still, InvConvNet's top mean rank indicates more consistent performance, suggest- ing robustness across diverse datasets. On the additional 3 datasets, INvCON NET shows superior performance in terms of accuracy, further validating the advantage of invari- ant CNN-based approaches classification. For Sleep-EDF, RESNET slightly better than the proposed INVCONv NET, which can be attributed to the depth of the method in captur- ing complex dependencies between the time series inputs. Finally, transformer-basee competed by CNN-based ones, and the MLP-based DLIN- EAR scores the worst, failing to capture the dependee temporal dynamics. Additionally, INVCONvNET built upon a single layer of convolutional kernels shows a significant
Ablation study, In this experiment, we compare IN- VCONVNET classification performances to its standalone main components. More specifically, INvCONvNET com- bines balanced sets of standard, amplitude/offset, and amplitude/innear trend invariant convolutions. We com- pare this model to its standards INYCONVNET-N, ampli- tude/offset INVCONVNET-O, and amplitude/linear trend INVCONVNET- counterparts. In all cases, the total number of kernels remains the same. Table 3 depicts the classifica- tion performances of all four variants. INVCONVNET shows the highest performance on all datasets. INVCONVNET-N is second in most cases, closely followed by the invariant models. It indicates that features invariants to amplitude, offset, or linear trend are valuable on many classification tasks while guaranteeing robustness to the networks as seen previously in Section 4.1, and by combining them, models achieve better performances.
Conclusion. Convolutional features invariant to deforma- tions are meaningful for classification tasks, and combining them in single-layer simple and lightweight architectures, offers comparable to better performances compared to the latest and prominent methods.
methods are significantly out-
4.3. Transfer Learning Experiment.
Protocol. We assess the generalization of invariant con- volutions on a transfer learning experiment using 4 dif- ferent source and target domains of the Fault- Dault-Diagnosis
8

---

Time Series Representations with Hard-Coded Invariances
Table 4: Classification Accuracy (%) on a Transfer Learning experiment on Fault-Diagnosis (A, B, C. D sub-datasets) for the supervised INVCONVNET and INVCONVNET-N (normal) and two self-supervised methods.
1--
Arg.
(F7)
1.4
53.52 4.55
6b13 1.39
1505
=1:50:100
::00::00
222
1.33
91.4
4695
006
100. =:0
54.28 + 2.77
10.56 1.66
V82 (57)
54.1
146
5am
1 1.0
52.54
1.9
55.0
1.17
88.2 0 0.47
108.
1:0
578
1 218
25330 3.00
7841 1.39
6631 195
59.97 002
8682 1 0.54
72.18 4: 1.57
INYCONVNET (Bee))
= 641
58.9
a 0,3
53.1
0.14
78.54 2.17
99.8
6.66
70.75
à: 1.0
85.04
0.13
100.0
78.49 0:338
76.52 à 0.37
NOCCONNNGTNN (S
48.55 9,68
$5.50
50.26:
77.30
0000
93.50
0.90
6430
a 0.48
94.37
0:23
3.46
2.
0:.14:0.114
72.66 a 0.78
dataset (Lessmeier et al., 2016). Model-wise, we include the self- superviued methods TS-TCC (Eldele et al., 2021) and VSC (Yue et al., 2022) as well as INVCONVNET and its standard counterpart INVCONVNET-N N configuration as presented in Section 4.2. Models are trained and tested on different source and target domains, namely the A, B, C, and D sub-datasets of Fault-Diagnosis, with direct transfer as in (Eldele et al., 2021). The self-supervised methods pre-trained and fine-tuned (FT) on each source domain dataset leveraging contrastive learning.
Impact Statement
The work presented in this paper aims to advance the field of Machine Learning for time series analysis by introducing novel architectural design of convolutional layers that are invariant to specific group actions for time series data, built upon thorough mathematical framework. Based on the pre- sented experimental evaluation, the proposed invariant con- volutional layers have the potential to significantly advance diverse domains of time series modeling, such as healtheare, environmental monitoring, and industrial machinery. By enabling robust time series representations and insights into their invariant properties, these layers address critical challenges in real- worldworld applications and tasks, including classification and anomaly detection, among others.
Results. Table 4 presents the transferred classification accuracy performances. Interestingly, the supervised IN- vCONvNET, built on invariant convolutions, outperforms unsupervised methods and its standard counterpart (-N) by at least 4% with low variance.
References
This experiment indicates that transfer learn- ing tasks time series can benefit from convolutional in- variant features to remove the distribution shift caused by deformations and improve generalization.
Conclusion.
Abdulaal, A.. Liu, Z., and Lancewicki, T. Practical ap- proach to asynchronous multivariate time series anomaly detection and localization. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 2485-2494, 2021.
5. Conclusion & Future Work
Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C. E. Indications of nonlinear deter- ministic and dimeesiamensional structures in time series brain electrical activity: Dependence on recording re- gion and brain state. Physical Review E, 64(6))061907, 2001.
In this study, we establish a formal mathematical framework for time series invariances, and design exact hard-coded invariant convolutions that seamlessly integrate within any CNN-based model. We experimentally show their enhanced generalizability and computational efficiency in several se- tups. To showcase the robustness of our framework, we focus our invariant convolutions to simpler but common affine deformations, allowing invariance to trend when it can be assumed linear the scale of the convolution kernel size. However, plan to explore approximating the trend with non-linear functions, such as splines or higher-degree polynomials, and incorporating seasonal components with fre-frequnncy cosine bases. Finally, exploring additional merging strategies on different types of invariances and ex- tending our invariant convolutional layers to unsupervised learning settings, is agenda for future work.
Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes- Oryyss J. L.,, et al. A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pp. 3, 2013.
Antoniou, A. Data augmentation generative adversarial networks. arXiv preprint arXiv:1711.043400, 2017.
Baek, S.J., Park, A., Ahn, Y.J.,, and Choo, J. Baseline correction using asymmetrically reweighted penalized least squares smoothing. Analyst, 140(1):20-2257, 2015.
Acknowledgements
Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, E. The uea mul- tivariate time series classification archive, 2018. arXiv preprint orXiv:181 10:::81110000 1.00075 2018
This work has been partially funded by the Industrial Data Analytics and Machine Learning (IdAML) chair hosted at ENS Paris Saclay, Université Paris-Saclay.

---

Time Series Representations with Hard-Coded Invariances
Bai, S., Kolter, J. Z., and Koltun, V. An empirical evalua- tion of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0127,, 2018.
Eldele, E., Ragab, M., Chen, Z.,, Wu, M., and Li, X. Tslanet: Rethinking transformers for time series representation learning, arXiv preprint arXiv:2404.08477. 2024.
Esling, P. and Agon, C. Time-series data mining. ACM
Bietti, A. and Mairal, Group invariance, stability to deformations, and complexity of deep convolutional rep- resentations. Journal of Machine Learning Research, 20 (25): 2019.
Computing Surveys (CSUR), 45(1): 2012.
Franceschi, J.-Y, Dieuleveut, A., and Jaggi. M. Unsuper- vised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Box, G. E., Jenkins, G. M., Reinsel, G. C., and Ljung, M. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Germain, T., Gruffaz, S., Truong, C., Durmus, A., and Oudre, L. Shape analysis for time series. Advances in neural information processing systems, 37:95607-95663 2024a.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv.210.113788, 2021.
Germain, T., Truong. C., and Oudre, Linear-trent trend nor- malization for multivariate subsequence similarity search. In 2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW), pp. 167-175, 2024b. doi: 10.1109//DE6668233 20w.://..0222.002.0000
Chen, S.-A., Li, C.-L., Yoder, N., Arik, S. O., and Pfis- ter, T. Tsmixer: all-mlp architecture for time series forecasting, arXiv preprint arXiv:2303. 06053, 2023.
Chen, T., Korbllith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Goldberger, A. L., Amaral, A., Glass, L., Hausdorff, M.M Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23): e215-e220. 2000.
Dau, H. A., Bagnall, A., Kamgar, K., Ych, C.-CMM.. Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., and Keogh, E. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293-13055 2019.
Hippke, M., David, T. J., Mulders, G. D., and Heller, R. Wötan: Comprehensive time-series detrending in python. The Astronomical Journal, 158(4):143, 2019.
Demirel, B. U. and Holz, C. Shifting the paradigm: A diffeo- morphism between time series data manifolds for achiev- ing shift-invariancy in deep learning. arXiv preprint DrXiv:2502.199211 2025.
Horie, M., Morita, N., Hishinuma, T., Ihara, Y., and Mit- sume, N. Isometric transformation invariant and equiv- ariant graph convolutional networks. arXiv preprint arXh::2005.06316 2020.
Dempster, A., Petitjean, F., and Webb, G. I. Rocket: ex- ceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454 1495, 2020.
Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. Detecting spacecraft anomalies us- ing Istms and nonparametric dynamic thresholding. In Proceedings of the 24th ACM SIGKDD international con- ference knowledge discovery & data mining, pp. 387- 395, 2018.
Ding, H., Trajcevski, G., Scheuermann, P., Wang, X., and Keogh, E. Querying and mining of time series data: experimental comparison of representations and distance measures. Proceedings of the VLDB Endowment, 1(2): 1542-1552. 2008.
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, I.., and Muller, P.-.. Evaluating surgical skills from kinematic data using convolutional neural networks. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2018: 21 International Confer- ence, Granada, Spain, September 16-20, 2018, Proceed- ings, Part IV 11, pp. 214-221. Springer, 2018
Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., and Long, M. Simmtm: A simple pre-training framework for masked time-series modeling. Advances Neural Information Processing Systems, 36, 2024.
Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C., X., and Guan, C. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2/06. /4/ 12, 2021.
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.-.. Deep learning for time series classifica- tion: a review. Data mining and knowledge discovery. 33 (4):917-963, 2019.
10

---

Time Series Representations with Hard-Coded Invariances
Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., Webb, G. I., Idoumghar, I.., Muller, P.-A., and Petitjean, F. Inceptiontime: Finding alexnet for time series classification.. Data Mining and Knowledge Discovery, 34(6): 1936-1999,,,,, 2020.
Löning, M., Bagnall, A., Ganesh, S., Kazakov, V., Lines, J., and Király, F. J. sktime: A unified interface for machine learning with time series. arXiv preprint 1IXi::1909.0787,, 2019.
Mallat, S. Understanding deep convolutional networks. Philosophical Transactions of the Royal Society A: Math- ematical. Physical and Engineering Sciences, 374(2065): 20150203, 2016.
Jacques, B. G., Tiganj. Z., Sarkar, A., Howard, M., and Sederberg, P. A deep convolutional neural network that is invariant to time rescaling. In International conference on machine learning, pp.
PMLR, 2022.
Maron, H., Ben Gem-Hamu, H., Shamir, N., and Lipman, Y. Invariant and equivariant graph networks. arXiv preprint 2018.
Keriven, N. and Peyré, G. Universal invariant and equivari- ant graph neural networks. Advances Neural Informa- fion Processing Systems, 32, 2019.
Mathieu, M., Henaff, M., and LeCun, Y. Fast training of convolutional networks through ffts. arXiv preprint 2013.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445/ 2020.
Mathur, A. P. and Tippenhauer, N. O. Swat: A water treat- ment testbed for research and training on ics security. In 2016 international workshop on cyber-physical systems for smart water networks (CySWater), pp. 31-36. IEEE, 2016.
Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., and Lew, J. In what ways are deep neural networks invariant and how should we measure this? Advances in Neural Information Processing Systems, 35:32816 32829, 2022.
Middlehurst, M., Vickers, W., and Bagnall, A. Scalable dictionary classifiers for time series classification. In Intelligent Data Engineering and Automated Learning- IDEAL 2019 20th International Conference, Manchester, UK, November 14-16, 2019, Proceedings, Part I 20, pp. 11 11-19. Springer. 2019.
Lessmeier, C., Kimotho, J. K., Zimmer, D., and Sextro, W. Condition monitoring of bearing damage in electrome- chanical drive systems by using motor current signals of electric motors: A benchmark data for data-driven classification. In PHM Society European Conference, volume 3, 2016.
Middlehurst, M., Iswail-Fawaz, A., Guillaume, A., Holder, C., Guijo-Rubio, D., Bulatova, G., Tsaprounis, L.., Mentel, L., Walter, M., Schäfer, P., and Bagnall, acon: python toolkit for learning from time series. Journal of Machine Leurning Research, 25(289); 2024. URL http: 1442223311444440 html.
Liu, M., Zeng, A., Chen, M., Xu, Z., Lai, Q., Ma, L., and Xu, Q. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816-5828 2022.
Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S. Pyraformer: Low-complexity pyramidal atten- tion for long-range time series modeling and forecasting. In International conference on learning representations, 2021a.
Minhao, L., Zeng, A., Qiuxia, L., Gao, R., Li, M., Qin, J., and Xu, Q. T-wavenet: A tree-structured wavelet neural network for time series signal analysis. In International Conference on Learning Representations, 2021.
Liu, X., Wang, H., Li, Z., and Qin, L. Deep learning in eeg diagnosis: A review. Knowledge-Bard Based Systems, 227: 107187, 2021b.
Moody, G. B. and Mark, arrhythmia database. IEEE engineering in medicine and biology magazine, 20(3):45-55) 2001.
The impac of the mit-bih
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L.., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06665, 2023.
Nic, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14733 2022.
Liu, Z., Alavi, A., Li, M., and Zhang, X. Guidelines for augmentation selection in contrastive learning for time series classification. arXiv preprint arXiv:2407.09336. 2024.
Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint 2019.

---

Time Series Representations with Hard-Coded Invariances
Paparrizos, J., Liu, C., Elmore, A. J., and Franklin, M. J. Debunking four Long-standing misconceptions of time- series distance measures. In Proceedings of the 2020 ACM SIGMOD international conference on management of data, pp. 1887-1905, 2020.
Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Ets- former: Exponential smoothing transformers for time- series forecasting. arXiv preprint arXiv:2202.01381, 2022b.
Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decom- position transformers with auto-correlation for long-term series forecasting. Advances in neural information pro- cessing systems, 11-22419-224300 2021.
Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):111-1119, 2020.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang. J., and Long, M. Timesnet: Temporal 2d-variation modeling for gen- eral time series analysis. In The eleventh international conference on learning representations, 2022.
Shulman, Y. Dynamic time warp convolutional networks. arXiv preprint arXiv:1911.019000 2019.
Singhal, A., Singh, P., Fatimah, B., and Pachori, R. An efficient removal of power-line interference and baseline wander from ecg signals by employing fourier decom- position technique. Biomedical Signal Processing and Control, 57: 101141, 2020.
Wu, Q., Yao, G., Feng. Z., and Shuyuan, Y. Peri-midformer: Periodic pyramid transformer for time series analysis. Advances in Neural Information Processing Systems, 37: 13035-13073, 2024.
Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In Proceed- ings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2019.
Xu, J. Anomaly transformer: Time series anomaly de- tection with association discrepancy. arXiv preprint arXiv:21 2021.
Yan, H., Xu, T., Wang, P., Zhang. L., Hu, H., and Bai, Y. Mems hydrophone signal denoising and baseline drift removal algorithm based on parameter-optimizz varia- tional mode decomposition and correlation coefficient.
Tonekaboni, S., Eytan, D., and Goldenberg, A. Unsuper- vised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.0077 2021.
Sensors, 19(21):4622, 2019.
Yang, X., Zhang, Z., and Cui, R. Timeclr: self-supervised contrastive learning framework for univariate time series representation. Knowledge-Bir Based Systems, 245:108600 2022.
Wang, S., WU, H., Shi, X., Hu, T., Luo, H., Ma, L.., Zhang, J. Y., and Zhou, J. Timemixer: Decomposable multi- scale mixing for time series forecasting. urXiv preprinl arXiv:2405. 14616, 2024a.
Ye, L. and Keogh, E. Time series shapelets: a novel tech- nique that allows accurate, interpretable and fast clas- sification. Data mining and knowledge discovery, 22: 149-182, 2011.
Wang, Y., Wu, H., Dong, J., Liu, Y., Long, M., and Wang, Deep time series models: A comprehensive survey and benchmark. 2024b.
Yi, K., Zhang, Q., Fan, W., Wang, S., Wang, P., Hc, H., An, N., Lian, D., Cao, L., and Niu, Z. Frequency- domain mips are more effective learners in time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Wang, Z., Yan, W., and Oates, T. Time series classification from scratch with deep neural networks: strong base- line. In 20/7 International joint conference on neural networks (IJCNN), pp. 1578-1585. IEEE, 2017.
Weiler, M. and Cesa, G. General e (2)-eurrrr steerable cnns. Advances in neural information processing systems, 32, 2019.
Yue, Z., Wang, Y., Duan, J., Yang, T., Huang, C., Tong, Y., and Xu, B. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, I.. Transformers in time series: A survey. arXiv preprint ar arXiv:2202.074 1225 2022.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poezos, B., Salakhutdinov, R. R., and Smola, A. Deep sets. Ad- vances in neural information processing systems, 30, 2017.
Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S. Cost: Contrastive learning of disentangled seasonal-trend rep- resentations for time series forecasting. arXiv preprint 57555 2022a.
Zeng, A., Chen, M.. Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of
12

---

Time Series Representations with Hard-Coded Invariances
the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Zhang, F,, Tang, X., Tong, A., Wang, B., and Wang, automatic baseline correction method based on the penalized least squares method. Sensors, 20(7):2015, 2020a.
Zhang, F., Tang, X., Tong, A., Wang, B., Wang, J., Lv, Y., Tang, C., and Wang, J. Baseline correction for infrared spectra using adaptive smoothness parameter penalized least squares method. Spectroscopy Letters, 53(3):222- 233, 2020b.
Zhang. T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng. S., and Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mip structures. arXiv preprint arXiv:2207.01 2022.
Zhang, Y. and Yan, Crossformer:
Transformer utilizing cross-dimension dependeney for multivariate time series forecasting. In The eleventh international conference on learning representations, 2022.
Zhang, Y., Liu, M., Zhou, S., and Yan, J. Up2me: Univari- ate pre-training to multivariate fine-tuning as general- purpose framework for multivariate time series analysis. In Forty-first International Conference Machine Learn- ing, 2024.
Zhang, Z.-M., Chen, S., and Liang, Y.-Z. Baseline correc- tion using adaptive iteratively reweighted penalized least squares. Analyst, 135(5): 113--1 146, 2010.
Zhou, H., Zhang. S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35,
pp. 11106-11115, 2021.
Zhou, T., Ma, Z., Wen, Q. Wang, X., Sun, I.,, and Jin, R. Fedformer: Frequency enhanced decomposed trans- former for long-term series forecasting. In Interna- tional conference machine learning, pp. 27268-27286. PMLR, 2022.
Zhou, T., Niu, P. Sun, L., Jin, R., al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322 43355, 2023.
13

---

Time Series Representations with Hard-Coded Invariances
A. Appendix
A.1. Invariant embedding
Let M be a Hilbert space and H finite dimensional vector subspace of M. We focus on the action on M by the usual vector addition: (h,m) orbit-injective.
M m + h e M. The following proposition exhibits H-invariant embedding that is also
Proposition 3. Let PH be the orthogonal projector on H, and La be the identity map on M, the embedding, L - La - PH (the projector on H ) is H- --invariant and urbit-injectingg
Proof. Existence of L: As H is a finite dimension vector space, is a closed and convex subset of the Hilbert space M; the orthogonal projector on H, denoted PH. exists. Therefore, L m E M + m Ph(m) E H is well defined.
H-invariance of L: Since H is closed, M H H and for any M, we decompose mh + fHL. Thus, for any m M, and h E H:
L(m + h)
m + PH(m h)
+ h PH(mpH MH + h)
+ h (MH h) (projector closed vectorial subspace)
mH
= L(m)
which proves the H-invariance of Orbit-injectivity of L: For any E M, its orbits corresponds to:
[m] = {m h E H}
= [L(m) + + l H, N' = -PH(m) + + MM
L(m)+1 +H
Therefore, for any ([m], [m²') E M/H M/H, such that [m]] n [m']] implies that L(m) L(m') proving the orbit-inecctiitty of L.
A.2. INVCONVNET: Architectural Details
The proposed pool of convolutions is presented in Figure 4 and is built upon the concatenation of normal filters, offset shift-invariant filters, and linear trend invariant filters. The proposed convolutional layer, therefore, incorporates three distinct kernel types with respect to invariance.
1. INVARIANT EMBEDDING MODULES
After mathematically formulating the characteristics of an invariant convolutional layer, which is built upon standard (or variant) kernel, a kernel invariant to offset shift and scaling, and a kernel invariant to linear trend and scaling, we provide additional details for the design of the employed embedding modules. We present visualizations of the embedding modules used for classification and anomaly detection (i. e., reconstruction) in Figure 5.
Standard Module (Single-Layer): The simplest embedding module is single invariant convolutional layer for specific kernel size W and hidden dimensions dao for the standard convolutional part (in yellow), dn, for the convolutional part invariant to offset shift and scaling (in red), and dn, for the convolutional part invariant to linear trend and scaling (in purple).
Inception-like Module (Single-Layer): We also study inception- design by employing several kernel sizes, but without stacking the layers in increasing depths. The depth of the employed module remains equal one. As shown in Figure 5 (Left), in an inception like embedding module, we consider several kernel sizes, e. g.. W1, W2, Wy, that applied in parallel to the input series, while leveraging the three parts of the proposed pool of convolutions (including standard and invariant ones). The produced representation for the differrnt kernel sizes, i. e., Z1, Z2, are concatenated in the channel dimension producing embeddings of size (3 Σ} dny: L) for selected kernel sizes. The distinct kernel sizes as well as the hidden dimension for each part in the pool of convolutions, are hyperparameters that we need to tune, as in every CNN-based architecture.
14

---

Time Series Representations with Hard-Coded Invariances
Pool of Convolutions
Normal
Normal
Kernel Input
N
doso
nN|
ZT
Offset Shift & Scaling
Otse:
2
C
des
2
Linear Trend & Scaling
days L)
LT
W
zt
dns
Figure 4: Visualization of the different kernel types employed on an input signal inside the proposed invariant convolutional layer, including normal filters (in yellow), filters invariant offset shift (in red) and filters invariant linear trend (in purple). The produced embedding is the result of concatenations of the different representations.
Multi-Scale Module (Multi-Layer): Additionally, we examine the capacity of a multi-scale embedding module built upon invariant convolutions as presented in Figure 5 (Right), particularly for the reconstruction task. Here the employed depth is equal to two. At the first level, an inception-like layer is employed, with kemel sizes selected to be powers of two (deriving the maximum exponent from the logarithm of half the series length and setting the minimum to four). The kernels, as described above, are applied in parallel, and the produced representation for the different kernel sizes is concatenated in the channel dimension. At the second level, a standard convolutional layer applied. We similarly employ several kernel sizes of size max max(W;)/ (W,//;; matching the picked kernel sizes in the first layer W, for i {l.... K), where K the number of kernels with distinct sizes. For each distinet kernel size in this second layer, dilation factor is set as r, W. thus equal to the kernel size of the previous layer. This design enables capturing representations at different scales while employing shallow and computationally light architecture, that still benefits from invariant convolutions. Experimentallyy this module shows performance improvements for the anomaly detection task, where reconstruction can benefit from capturing dependencies at different granularities.
A.2.2.
MODULES
The embeddings derived from the modules described above are further processed by standard layers to produce the task-specific output.
I. Classification.
The embedding is passed by Global Average Pooling (GAP) layer, applied to the channel dimension, that averages over the time dimension to reduce the temporal features to single value per channel. The result of the GAP layer is followed by a linear layer that produces the final class probabilities.
II. Anomaly Detection (Reconstruction) The multi-layer embedding module presented above is used to capture dependen- cies at various scales. For the reconstruction of the input, the embedding followed by linear layers applied first on the temporal and then on the channel dimensions. The coefficients of the invariant kernels, i. c., (ay, for the offset-invariant part, passed through linear layers to be mapped to the original time and
part and avy.ay, By) for the channel dimensions and are then combined with each part (one variant and two invariant) of the embedding via addition and multiplication. More specifically, o and B/ refer to the norm of invariant embeddings of the signal, μz refers to the means, a and 3y refers to the coefficients of the linear trend. Combining the coefficients with the linearly projected embeddings adapts the level of the series to the original one, enabling enhanced temporal resolution. Figure presents details about the reconstruction module and its relation with the embedding module.
15

---

Time Series Representations with Hard-Coded Invariances
Single-Leyer (deplh 1)
Multi-Layer (depth 2)
komel sizos:
1,2,4
Wi.
W/4
W. J,W W/4
W2
W
xxl R0|las
L
L
W
(3 Σ
days L)
dap L)
F=
J-0
Pool Convolutions
(1) Pool of Convolutions
(2) Standard Convolutions
Figure 5: Left: Single-layer embedding module, i.e., of depth used in standard INVCONYNET (for chosen kemel width W) and in inception-like INVCONVNET (for several chosen kernel widths, g.. 3 visualized in the figure). Right: Multi-layer embedding module, i. e.e of depth 2, used in multi-scale INVCONVNET (for several chosen kernel widths W). In the second layer, for each kernel size, we utilize multiple kernels whose total number sums up to the larger kernel size, achieving multi-scale views.
Embedding Layer
Reconstruction
Predict Linear
Project
Channels (FC)
Normal
(FC)
Offset Shift
& Scaling
(+)
-
(+)
L
(ar
Linear Trend Scaling
nutuo
Zt
(+)
(1,)
Figure 6: Visualization of the architecture used terms of reconstruction that leverages the output of an embedding module built upon a pool of convolutions (including variant and invariant ones). The representation the embedding layer NN, zo, zr are passed from 2 fully connected linear layers (FC), from which the first operates on the temporal dimension and the second on the channels. The coefficients of each invariant operation similarly projected with linear layers and combined with the representation (with addition multiplication) to produce the output.
16

---

Time Series Representations with Hard-Coded Invariances
A.3. Datasets Details
We focus our experimental evaluation on several real-world time series datasets, including univariate and multivariate inputs, with significant applications in healthcare and medical diagnosis, wearable technology, audio processing, and transportation, among others.
Classification Datasets. Details about the 26 multivariate derived from UEA data repository (Bagnall et al., 2018), that are employed in terms of this study in the classification experiment are provided in Table More specifically, for each dataset we mention the number channels, the length of the multivariate series, as well as the number of classes and the number of instances in the predefined train and test sets.
Table 5: Details of UEA datasets used for classification.
Dataset
# TTaii
#Test
#
#Channels
Length
# #Classes
Articulary WondRecognition
275
300
9
144
25
AurialFibrillation
15
2
640
BasicMations
40
6
100
Cricket
108
72
6
1197
12
Epilepsy
137
138
3
206
4
EthlanolConcentaattonn
261
263
3
1751
4
FaceDetection
5890
3524
144
62
FingerMovements
316
100
28
50
2
HundMovemenDDinectton
320
147
10
400
4
Handwriting
150
850
3
152
26
Heartbeat
204
205
61
405
Insect Wingbeat
30000
20000
200
Japanese Vowels
270
370
12
29
Libras
180
180
24
51
L.SST
2459
2466
6
36
14
Motorlmagery
278
100
64
3000
NATOPS
180
24
51
PEMS-SF
267
173
963
144
7
PenDigits
7494
3498
2
8
10
PhonemeSpectra
3315
3353
11
217
39
RacketSports
151
ScifRegulaatinnSCPPI
268
896
SelfRegulationSCCP2
200
180
7
1152
2
Spoken/rabe Digi
6599
2199
13
93
10
StandWalJJume
12
4
2500
UWaveGestureLibbuury
120
3
315
Table 6: Details additional datasets used for classification.
Dataset
#Train
# Tesst
# #Channels
Length
# Claassss
UCIHAR
7352
2947
9
128
Slecp-EDF
25612
8910
3000
5
Epilepsy
9200
2300
178
2
Fault- Pault-Diagnosis
8184
2728
5120
3
Table contains the same details for the additional datasets used for classification, i.e., the UCIHAR (Anguita et al., 2013) dataset, the Sleep-EDF dataset (Goldberger et al., 2000), and the Epilepsy dataset (Andrzejak et al., 2001). More specifically, UCIHAR data were collected by 30 volunteers performing various activities, including laying, standing, sitting, walking, walking downstairs, and walking upstairs. Volunteers' records were captured by a waist smartphone, including distinct measurements connected to acceleration and velocity signals. The Sleep-EDF dataset from the PhysioBank database consists of PolySomnoGraphic sleep recordings containing EEG, among other measurements. We consider only the EEG signals following previous studies (Eldele et al., 2021) and performed sleep stage classification, including awake, rapid eye movement, and non-rapid eye movements. Finally, the Epilepsy dataset consists of EEG brain activity measurements for epileptic seizure classification. Following the preprocessing of (Eldele et al., 2021), we perform binary classification after merging classes referring to non-epileptic seizure. For the transfer learning experiment, we utilized the Fault- Diagnosis
17

---

Time Series Representations with Hard-Coded Invariances
dataset (Lessmeier et al., 2016), as preprocessed in (Eldele al., 2021), which comprises of measurements under 4 different working conditions, perceived as different domains, and are assigned 3 classes, including a healthy and two fault classes.
We also conduct a synthetic experiment on 5 large datasets from UCR repository (Dau et al., 2019). The selected datasets were picked from the equal-length datasets of the relevant repository that combine a large number of samples in the training and test sets, along with a large series length. Datasets with train set size and series length less than 100 were not considered in our subset. Details about the selected UCR datasets are given in Table 7. Since many UCR datasets are already preprocessed using Z- Z-nommalization to achieve zero mean and unit variance, they are not ideal for demonstrating the impact of our invariant layers on classification performance. This is the primary reason for conducting a synthetic experiment these datasets rather than conventional one with the whole repository.
Table 7: Details of the subset of UCR datasets used for the synthetic classification experiment.
Datasct
# Train
#Test
#Channels
Length
#Classes
HandOutlines
1000
370
1
2709
MixedSSapesRRggaarrIii
500
2425
1
1024
5
1800
1965
I
750
42
FordB
3636
810
I
500
Yogg
300
3000
1
426
Anomaly Detection Datasets. Furthermore, we present in Table 8 the five employed anomaly detection datasets after preprocessing them on non-overlapping subsequences of length 100, also showing the number of channels and the size of the train, validation, and test splits. The SMD dataset (Su et al., 2019) consists of data related to server machines collected at an internet company, while the MSL and SMAP (Hundman et al., 2018) datasets comprise of telemetry data from spacecraft monitoring systems. The SWaT (Mathur & Tippenhauer, 2016) dataset is collection of sensor data from the operations of critical infrastructure system. Finally, the PSM (Abdulaal et al., 2021) dataset contains measurements from application server nodes on an internet website.
Table 8: Details of datasets used for anomaly detection.
Datasct
#Train
# Val
#Test
# CChannels
Length
SMD
566724
141681
708420
38
100
MSL
44653
11664
73729
55
100
SMAP
108146
27037
427617
25
100
SWaT
396000
99000
449919
51
100
PSM
105984
26497
87841
25
100
Data Splits and Preprocessing. As mentioned already in the main paper, for the proposed method, we do not normalize the data using Z-normalization for UEA and the rest 4 datasets used in classification, while the datasets from UCR are used for the synthetic experiment are derived normalized by the data source. On the contrary, all data are normalized for classification and the baselines, as well as for anomaly detection and all considered models (including the proposed INVCONVNET). For the UEA datasets, we do validation on the whole training set since the test sets are, in several cases, quite large, and thus, a small subset of the train set picked for validation can be misleading indicator of performance. For the rest classification datasets, we perform split into train/validation/eess sets with 60 20 20 ratio, following (Eldele et al., 2021). Similarly, for the five anomaly detection datasets, we split into train/validationntest sets with a 70 : 10 20 ratio (Xu, 2021).
A.4. Implementation Details
All experiments presented in this study were conducted on an Nvidia Tesla V100 GPU, with 40 cores and 756 GB of memory. We utilized the Adam optimizer with learning rate or = 0.001 for both classification and unsupervised anomaly detection tasks. We also adopted linear cosine annealing learning rate scheduler for INVCON in classification. More specifically, the scheduler started the warmup phase with learning rate equal to 0.001, linearly increasing the learning rate over the first 10 epochs to 0.01. After the warmup, it gradually reduced the learning rate using a cosine annealing schedule, down to 0.0001 by the end of training. For anomaly detection and the rest methods, we utilized learning rate scheduler of 0.5 decrease rate per epoch. To have better estimates for the generalization performance of all models and, most importantly, our proposed shallow modules, we performed 3 runs with random seeds for all considered datasets and tasks, Additional
18

---

Time Series Representations with Hard-Coded Invariances
details for each task and the hyperparameters of the models are given below.
Classification Task: We trained the models for 100 epochs for all UEA datasets, the 5 UCR datasets and the Fault- Diagnosis dataset. We performed early stopping during training, after 20 epochs of no improvement in the validation accuracy for all models and kept the configuration of weights that correspond to the best validation accuracy during training. The standard cross entropy loss was optimized during training for classification. For the INVCONVNET model, we considered the inception- embedding module of Figure 5 (Left) for all datasets of UEA except for Epilepsy, EthanolConcentrrtionn, Heartbeat that we selected the standard embedding module of single kernel size. Finally, for FingerMovements, Handwriting, Libras and SeliRegulationSCPP datasets, we employed the multi-scale embedding layer of Figure 5 (Right). The type of the embedding layer, well the hyperparameters for the convolutional layers, e. g., kernel size and hidden dimension, were selected through random search and the best performance on the validation set. For the rest of the classification datasets, i. c., the UCIHAR, the Sleep-EDF and the Epilepsy datasets, we trained all models for 300 epochs with 20 epochs patience and considered the inception- embedding layer, since was performing better on the validation set.
Unsupervised Anomaly Detection Task: We trained the models for 10 epochs and stopped training if no improvements had been made in terms of validation loss for 3 epochs, saving the best model weights on the validation set. We optimized the models using the mean squared error (MSE) between the real input sequences and the reconstructed ones. For all five anomaly detection datasets, we used the multi-scale embedding layer of Figure 5 (Right), followed by the reconstruction module built upon linear layers in Figure 6.
We next provide more information about the selection of the kemee sizes and hidden dimensions for the different embedding modules tested in terms of INVCONVNET. For the standard pool of convolutions with one specific kernel size W, we chose the kernel size as the minimum value between the value 50 and half of the length of the time series. For the inception-like embedding module, we selected several kernel sizes, such as 50,75,101, and 125, or factors of those values for which the length of the series is proportional. Finally, for the first layer (pool of convolutions) computed the kernel sizes as powers of two, starting from 16 up maximum of 128, based on the logarithmic scaling of half the series length. For all modules, we tested hidden dimensions sizes for the pool of convolutions in {32, 64, 128, }56} doing a split that enabled almost equal contribution for the three parts, i. e., normal, invariant to offset shift and scaling, and invariant to linear trend and scaling. For instance for total hidden size equal to 32 the different parts had (12, 10, 10) hidden dimensions respectively, for 64 the split became (24, 20, 20) and so on.
Hy pererparaeeeer Selection:
of the multi-scale module,
For the common CNN- baselines INCEPTION, RESNET, CNN, we tuned the number of convolutional layers, the kernel sizes, and the hidden size of each layer. We followed a random search for value between 2 and 6 for the number of blocks and {32, 64, 128, 256} for the hidden dimensions, whereas for the kernel sizes, we used those proposed in the relevant papers (Ismail Fawaz et al., 2020; Wang et al., 2017; Ismail Fawaz et al., 2018). All baselines' implementations are derived from the Time-Series-LLbrarg (Wang et al., 2024b), with the configurations mentioned in the respective papers, and the main code resources for performing the different tasks, e. g.. classification and anomaly detection were adopted. We also used ROCKET (Dempster et al., 2020) from sktime Library (Löning et al., 2019), with 3000 random convolutional kernels. Finally, for the transfer learning classification experiment, the self-supervised contrastive TS-TCC and TS2VEC methods were trained with their default parameters for classification as proposed in the respective papers (Eldele et al., 2021; Yue et al., 2022), for 50 epochs for each phase of pre-training and fine-tuning.
A.5. Additional Results
1. RECONSTRUCTION BASED ANOMALY DETECTION BENCHMARK
Reconstruction--basel anomaly detection involves training model learn a compact representation of normal data by reconstructing the input. The reconstruction error acts the anomaly criterion, indicating whether the time series does not conform to the normal patterns based on a chosen threshold.
Datasets. For unsupervised anomaly detection, we deploy the following benchmark datasets; SMD (Su et al., 2019), MSL and SMAP (Hundman et al., 2018), SWaT (Mathur & Tippenhauer, 2016) and PSM (Abdulaal et al., 2021). We apply standard preprocessing to extract non-overlapping sub-sequences and split them into train/validationttest sets with a 70: : 10: : 20 ratio (Xu, 2021; Wu et al., 2022).
Focusing on reconstruction, we include fourteen time series (regression) models, most from Library (Wang et al., 2024b), including TIMESNET (Wu et al., 2022), PATCHTST (Nie et al., 2022), ETSFORMER (Woo
Baselines.
19

---

Time Series Representations with Hard-Coded Invariances
al., 2022b), FEDFORMER (Zhou et al., 2022), AUTOFORMER (Wu et al., 2021), PYRAFORMER (Liu et al., 2021a), INFORMER (Zhou et al., 2021), REFORMER (Kitaev al., 2020), LIGHTTS (Zhang et al., 2022), DLINEAR (Zeng et al., 2023), ITRANSFORMER (Liu et al., 2023), TIMEMIXER (Wang et al., 2024a), PERI-MIDFORMER (Wu et al., 2024) and the TSLANET (Eldele et al., 2024) backbone. For the decoder, we capitalize on the learned slope and intercept values for each of the two invariant embedding parts, i. e., the Offset and Linear Trend, to adjust the projected embedding back to the original temporal dimensions along with a linear layer. The projection to the initial channel dimension is then obtained by applying a second channel-wise linear layer. Details on the INYCONVNET embedding and reconstruction modules are also given in Appendix A.2.
Results. Table 9 shows Fi-scores (%) for the proposed model and baselines across 5 anomaly detection datasets. IN- VCONVNET performs best on the SWaT dataset and ranks third in average performance across all datasets, slightly behind TIMESNET. The latter's superior performance can be attributed to its refined CNN blocks, which capture multiple periodici- ties for finer granularity in reconstruction. TSLANET achieves the highest average Fl-score, benefiting from Fourier blocks before CNN modules to capture both short- and long-term dependencies. Other models, like the MLP-based DLINEAR and transformer- FEDFORMER, also show competitive results. Notably, INVCONVNET excels with single layer of invariant convolutions, proving the effectiveness of leveraging multi-scale invariances in shallow architectures.
Table 9: Anomaly Detection results in terms of the Fi-score (%) for all considered datasets. Higher is better, best methods in bold, second best underlined.
Tausshir
TSLAN
B1Necsetaat
ITRANGO
PaRcoMt
Lian11
TiassMia
RiSURAIK (2800)
130038:
(2022)
(3007)
(2801
(30311
SM0
$105 16
S1.63 0.56
1115 1.15
81.55 0.17
08 8:69
11:11 15
85.38 D:9
8131 - 843
1101 11:19
83.30 0/5
71 16 1:
71 11:261111.0 (1.01
71.7 DUT
71.33 D:0
MSI
65/66 1.00
81 11 1174
18 1:00
11K 678
11 96 1:511
82.86
T246
005
80.0 11.6
81:0
67.17 3.31
93 8.14
31:00 11.8
53:13
6.11
31.53 (1:8
023. 9.11
18.18
92.05
13.15
28.75
PSM
9:.34::0.0
0:0
56.20 006
05.23 6:000
89:44 8:886
05.15 014
9631 a 009
000
16.6
9401 0.77
05.25 0:00
93:06:00.33
00.35 003
00.34 00:4::000
Arp
We also evaluate the 15 models on the anomaly detection datasets using mean ranks based on Fl-score, as shown in the critical difference diagram in Figure 7. Statistical significance was assessed using the Friedman test, as implemented in the Acon Library (Middlchurst et al., 2024). Since the Friedman test did not indicate a significant difference at the a 0.1 level, no pairwise tests were applied, and all models fall within the same group, which can be attributed the small number of considered datasets. Interestingly, in terms of mean rank scores, INVCONVNET ranks second with 4.8, closely following TIMESNET, which achieved the best mean rank of 3.8, demonstrating robust performance.
0
TimeMixer
11.0000
3,8000
TimesNet
FEDformer 10.3000
InvConvNet
iTransf. 10,0000
5,0000
DLinear
Pyraformer
9.6000
6,0000
TSLANet
Reformer
9.0000
Peri-mid.
Informer
9.40d0
7.0000
PatchTST
ETSformer 9,4000
7,4000
LightTS
Autoformer
Figure 7:
Critical difference diagram based on mean ranks from F1 scores on the 5 anomaly detection datasets.
Finally, in Table 10, we perform additional comparisons, in terms of anomaly detection, including the INT model and the standard CNN- based variants, namely INCEPTION, RESNET and CNN originally proposed for classification. All models have an identical reconstruction module, with the exception of INVCONVNET, which also includes the signal decomposition coefficients the invariant basis.
As observed, our proposed method consistently outperforms all evaluated CNN-based variants, suggesting the effectiveness of combining invariances with their related signal coefficients for reconstruction.
Conclusion. Invariant convolutional features combined with their related coefficients appear to be a concise and suited representation of time series for unsupervised anomaly detection based on reconstruction.
20

---

Time Series Representations with Hard-Coded Invariances
Table 10: Anomaly Detection results for INYCONVNET and vanilla CNN-based methods. Performance mentioned in terms of the F1-score (%). Higher is better, best methods in bold, second best underlined.
INyCONvNET
INCEPTION
RES
CNN
Datasets
predicr (incur, project channels
SMD
84.05 1: 0.16
71,45 0.11
76.20
0.52
77.31 1 0.91
MSL
+
0.08
81.25
0.09
79.96
SMAP
68.29 +: 0.07
68.63 = 0.16
67.24:00.05
67.00
0.07
SWAT
92.82 + 0.19
82.60 = 0.20
0.04 50.53
80.24 + 0.95
PSM
0.35
92.30
93.30
Avg. F1 (75)
84.44 1: 0.09
79.30 1 0.28
79.58 0.42
79.56 1 0.5.4
A.5.2. ROBUSTNESS STUDY- VISUALIZATION OF FEATURE MAPS
In Figure 8, we provide visualizations of the feature maps produced from different filter types (among the ones introduced in the paper), which are later incorporated in INVCONVNET example architecture, including normal ones (CONvNET (normal)). filters invariant to offset shift (InvCoNVNET (offset)) and filters invariant to linear trend (INvCONvNET (trend)). Specifically, we consider FordB dataset from the large UCR datasets, considered in the Robustness Study of Table 1.
For the considered dataset, the convolutional filters presented below are trained on normalized raw data and tested on four additional synthetic deformation scenarios: (i) random offset (off.), (ii) random linear trend (I.T), (iii) combined offset and trend (off., LT), and (iv) combined offset and smooth random walk (off.. RW). For the last deformation, the added synthetic trend is random walk generated from a Gaussian distribution and smoothed by a rolling mean.
Extraction of Feature Maps. After passing the input series through the convolutional layer of each model and the activation function (i. e., ReLU(.)), we extract the feature map for each filter (or each hidden dimension) corresponding to the largest considered kernel size. We recall that for the synthetic experiment on the 4 larger UCR datasets, we leveraged the inception-lics embedding module of Figure 5 (Left), built upon 4 different kernel sizes with each having an equal total number of filters (or hidden dimensions equal to 128). Please note that averaging over all outputs produced by the layer for the several distinct kernel sizes produces multi-scale representations that produce similar (in terms of activated regions) but smoother maps (in terms of intensity values). The resulting feature maps, which are derived by the activated outputs for the largest kernel size, are essentially 2D representations with dimensions equal to the series length L and the 128 hidden dimensions.
Amplitude
Amplitude
Off, loffee))
LT (trenc)
Normallzeed Time Series
Normalized Time Series
200
300
405
500
100
200
300
400
500
Time
Time
(a) Input Series (+ Off.)
(b) Input Series (+ LT)
Mp
Amplitude
Amplttude
-.2
1.
offi, LT Coffoeet, trend)
Offi, RM coffiset, rw trend) Normalized Time Series
Normallzed Time Series
200
500
1000
200
300
400
500
Time
(c) Input Series (+ Off., LT)
(d) Input Series (+ Off., RW)
Figure 8: Plot of single sample from the test set of FordB dataset for UCR used to produce the example feature maps of different types of convolutional filters. The normalized time series is captured in a black dotted line, while signals in blue represent the deformed versions of the series. Specifically, (a) represents the sample with the addition of random offset, (b) represents the sample with the addition of random linear trend, (c) represents the sample with the addition of random offset and linear trend, and (d) represents the sample with the addition of random offset and random walk trend.
We provide in Figure the activated feature maps for a single sample for FordB dataset and its deformed versions (as
21

---

Time Series Representations with Hard-Coded Invariances
Feature Map of Conv (normal)
Feature Map of Inw, Conv (offset) Input
Feature Map of Inw, Conv (trend) Input
Input
Hodden Denemken
Hilden
Sequence Length
Sequence Length
Sequence Length
(a)
(b)
(c)
Input (offset)
Input
Hidden Bimeniers
Sequence Length
Seguenee Length
Sequenoe Length
(d)
Off., (offset, trand)
Input Off,, (offset, trend)
Input Off. LT (offset, trend)
Jedden Oimanon
Diwentns
(h)
Input + Off,, RW (offset,, rw trend)
Input Off., RW (offset, rw trend)
Input Off. (offset, rw trand)
Widden Dimanions
Molten Dieveiies
Sequence Length
Sequence Length
Sequence Lengh
(k)
Figure 9: Comparison of the feature maps produced the robustness study (of Table 1) on a single sample of FordB dataset by the different types of convolutional filters before average-pooling for the normalized input and the 3 scenarios of synthetic deformations (addition of random offset, addition of random offset and linear trend and addition of random offset and random walk trend). Each column represents convolutional filters (128 in total) of the same type: (d)(().(g)()) correspond to normal convolutional filters invaiinvariant,, (b) correspond to offset shift invariant convolutional filters and (b)) correspond to offset shift invariant convolutional filters and (I).
22

---

Time Series Representations with Hard-Coded Invariances
presented in Figure 8), extracted for the different considered kernel types (CONV (normal), INV. CONV (offset) and INV. CONV (trend) heatmaps. Color in the heatmap plots corresponds to the magnitude of the activation specific location of the series for each hidden dimension, with a lighter color (i. e., yellow) representing higher activations.
Deformation-Speciit Activation Maps for Different Types of Filters. The feature maps of each considered convolutional filter type with respect to invariance, ie., normal, offset shift- shift-invariant, and trend- gend-invariant, reveal distinct activation patterns under different deformation scenarios.
Normal filters are highly sensitive to both offset shifts and trends, often leading to widespread activation across the entire time series (See Figures 9d, 9g). This lack of selectivity causes the model to struggle to distinguish meaningful patterns from uninformative regions, which are better captured for the plain (normalized) data Figure 9a). For instance, when a smooth random walk trend is introduced (Figure 9j), their activations become more uniform, indicating reduced sensitivity to meaningful variations in the input.
In contrast, offset shift inft--invariant filters remain robust to local shifts in the signal, exhibiting stable activation patterns even when an offset is introduced (See Figures 9b and 9e). However, their robustness is affected slightly when a linear trend is introduced (Figure 9h), as they begin to exhibit variability in their feature maps.
Finally, wend-invarian filters consistently preserve their activation patterns across all tested deformations, demonstrating resilience to both offset shifts and trends (See all Figures 9f,9i,91). This stability ensures that relevant features are captured effectively, regardless of input distortions. Similarly, trend-invariant filters are less affected by gradual changes, maintaining consistent responses across both plain and deformed data. While normal convolutions react strongly to local variations, offset and trend-invariant filters provide stability under deformations, offering robust representation of the underlying time series patterns in the presence of deformations.
A.5.3. CLASSIFICATION
We next present Figure 10, the critical difference diagram based on accuracy for the classification of UEA datasets. Mean rank performance is also a fair indicator of the robustness of models across several datasets and can indeed reflect consistent relative performance for the proposed method against most baselines.
DLinear
9.3200
3,6200
InvConvNet
Perl-mid.
Rocket
Inception
8,4200
5,9400
TSLANet
TimeMixer
8.4000
6,1400
TimesNet
PatchTST
7,9000
ResNet
iTransf.
7,5000
6,9800
Cnn
7,0000
Crossformer
Figure 10: Critical difference diagram based on mean ranks from accuracy scores on the 26 UEA datasets.
INVCONVNET achieves the best mean rank across 26 UEA datasets and 13 methods. Following the implementation from the Acon Library (Middlehurst et al., 2024), we compute mean ranks based on accuracy and assess significance using the Friedman test, which confirms the presence of statistical differences among models. Pairwise comparisons are performed based on the Wilcoxon signed-rank test with Holm correction at a = 0.1, to identify groups of classifiers whose performance differences are not statistically significant. This procedure yields four cliques: the first includes INVCONVNET and ROCKET, while the second includes ROCKET, TSLANET, TIMESNET, RESNET, CNN and CROSSFORMER.
We also provide in Table 11 the full classification results for the 26 considered UEA datasets that correspond to the average of 3 runs for each combination of dataset and model. In the same table, we include again the already presented in the main paper, average accuracy for the whole collection of datasets as well as the number where each model scores first in the last row. The Japanese Vowels dataset is mentioned as out-of-time ('OOT") for not producing performance results since the experiment did not run within the time limits (12 hours maximum for each dataset). From the full classification results, we observe that the proposed NYCONyNET is, in several cases, slightly outperformed by the classical ROCKET method, but on
23

---

Time Series Representations with Hard-Coded Invariances
Table 11: Full Classification results for UEA datasets. Accuracy (%) is mentioned for all combinations of models and datasets. Higher is better, best methods in bold, second best underlined.
Dataact
NeTCowvNET
TIMESNET
CROSSIGIMEER
PURI-XID.
TSLANET
DLINDAR
TOMrMIXER
INCEPTION
Amiustay Wendiecognnnonn
25,00)
97.78
97.67
98.22
98.00
97.00
96.67
$4.55
58.64
97.89
AusalFFactlaion
$7.78
28.39
42.22
28.39
24.44
35.33
24.44
35.56
37.78
28.39
24.44
33.33
6,67
RacicMrutions
100.00
05.00
70.83
91.67
36.67
60.83
109.00
61.67
72.50
87.50
100.00
100.00
100,00
Cricket
98.61
03.06
94.44
92.59
$7.96
89.11
97.69
91.30
90.56
87.96
98.15
95.61
100,00
Epilepey
95.88
59.11
91.34
87.46
32.13
96.85
51.45
92.27
92.27
96.55
DeceDescction
56.59
52.75
RngerrMeement
56.33
55.00
53.33
53667
50.33
25,00
43.67
69.67
$6.35
53.00
53,00
54.00
40.99
64.11
47.75
57.21
45.95
36.94
45.50
59.01
$1,80
31.08
36.04
29.28
44.59
Handwating
53.18
28,67
26.50
26.39
22.76
12.71
45.71
18.71
25.45
17.22
57.10
$6.20
56.27
Haseboo
77,40
68.29
66.18
68.13
65.55
75.17
75.73
69.92
66.83
70.4
60.76
62.76
73.17
WaseeWa
"OO1"
'OOT
'OOT
SpunnaVowsk
79.07
91.11
LSST
55.04
12:77
49.25
1121
3.95
31.08
10.61
31.85
63.01
35.71
3.99
9.57
60.76
MotaeEmagery
39.57
52/00
50.67
55.00
51.67
55.67
47.67
50.33
49.00
51.67
51.33
$1.33
46.33
NAZOPS
95.74
93.33
75.00
$7.41
$3.15
88.70
94,63
92.78
76.48
90.74
96.67
95.74
$7.96
PEMS-SF
90.35
78.61
B1.30
$4.30
$7.86
63.39
70.96
80.15
84,01
75.53
79.38
74,37
90.15
PeuDigio
96
96.37
Phonenoelprotts
RacketSpurs
79.82
79.39
67.58
77.19
99..7
66.12
$7,601
78.34
85.32
$7.49
31.45
79.15
$3.19
83.05
$1.57
80.32
$4.64
34.55
S3.4
48.15
45.19
47.59
48.15
51.67
51.89
45.74
49.30 26
53.35
48.15
48.
54.82
gooken/rattcDiggls
99.47
98.53
97.92
98.67
98.86
97.25
99.58
95.85
97.71
95.50
99.223
98,98
99.56
StndWWak lump
28.80
33.33
$1.11
24.44
$1.11
33.35
45.89
33.33
57.78
26.67
40.00
51.11
48.50
80.94
Accuracy
61.51
65.67
average, is among the first best-competing models for most datasets, which explains its performance superiority in terms of average accuracy for the whole UEA.
In several studies (Wu et al., 2022; Zhou al., 2023), only subset of 10 UEA datasets is considered, and we also present again the results for this subset along with total the average accuracy in 12. Similar observations can be made those for Table 11, with the proposed INVCONVNET scoring the best average accuracy of 73.22%, followed by ROCKET.
Table 12: Full Classification results for a subset of 10 UEA datasets. Accuracy (%) is mentioned for all combinations of models and datasets. Higher is better, best methods in bold, second best underlined.
NeT
TIMESNET
PAICETST
CROSSPORMER
PERI-MID
TSLANET
DLINEAR
TIMIMIXUR
INCEPTSON
RESNETT
RUCKET
22.81
FaoeDrioctinn
Handorttngg
53.14
78.67
26.95
26.39
22.78
12.71
48.71
18.71
25.45
17.22
37.10
56.20
56.27
Heanbeatt
77.40
68.29
66.18
68.13
65.85
73.17
75.73
691.22
66.83
70.41
69.76
62.76
73.17
Japanese Vowels
97.66
SelRegalationSCPP
$6.12
$7.60
78.84
85.32
$3.89
81,06
79.18
83.05
81.57
80.30
84.64
84.53
SelReegslaiiss SCP2
54.44
48.15
19
47.59
45.15
$1.67
$3.80
45.74
49.26
$3.33
48.
48.
54.82
SpolenArahheiiiii
09.47
98.83
97.92
98.67
96.86
97.24
99.58
95.85
97.31
95.50
99.23
98.98
92.56
Avg. Accuracy (5)
24


Reviews:
[{"review_id": "kJlTUJbVnF", "reviewer": "Reviewer_gvG9", "summary": "The article introduces a mathematical framework for integrating invariant into convolution operators for time series. The scaling, offset shifting, and trend invariant are particularly studied. A large number of experiments are then conducted, demonstrating the advantages of these convolutions in terms of their robustness to distortions, their performance in classification, as well as their effectiveness in transfer learning and anomaly detection.\n\n## update after rebuttal\nGiven the responses provided during the rebuttal, I am increasing my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the time spent to evaluate our manuscript. We next reply to their key suggestions and comments on our work.\n\n**[Evaluation Criteria]**\n\n>\"The authors could have produced a synthetic diagram of the model's performance [...].\"\n\nWe appreciate the reviewer's suggestion and will add a critical difference diagram, along with clarifying Rocket's first-count performance in the main section. As shown, InvConvNet achieves the best mean rank across 26 UEA datasets and 9 baselines. Using the aeon library [1], we compute mean ranks based on accuracy, confirming significance via a Friedman test (critical difference value of 1.6784 at $\\alpha = 0.1$). The Nemenyi post-hoc test identifies three statistically distinct groups: 'InvConvNet' and 'Rocket' in the first, followed by 'TSLANet', 'ResNet', 'TimesNet', 'Cnn', 'Crossformer', and 'PatchTST' in the second.\n\n## Mean Rank for UEA Classification Acc.\n\n| Datasets             | **InvConvNet** | TimesNet | PatchTST | Crossformer | TSLANet | DLinear | Inception | ResNet | Cnn | **Rocket** |\n|----------------------|----------------|----------|----------|-------------|---------|---------|-----------|--------|-----|-----------|\n| *UEA* (26 datasets)  | **3.16**       | 5.38     | 6.44     | 6.10        | 4.96    | 7.44    | 6.96      | 5.12   | 5.76| **3.68**   |\n| *CD Value* = 1.6784  |                |          |          |             |         |         |           |        |     |           |\n\n[1] https://www.aeon-toolkit.org/en/latest/examples/benchmarking/published_results.html#References\n\n**[Broader Scientific Relations]**\n\n>\"Their proposed idea goes somewhat against the current trend [...] few real-world cases were invariances are so explicitly defined in my opinion.\"\n\nRecent deep learning trends for time series rely on implicit invariances from augmentations using contrastive learning, often lacking generalization guarantees (see Table 1) and incurring high computational costs. Our approach directly encodes common time series invariances within the network, ensuring both efficiency and generalization (see Figure 3, Tables 1 \\& 3). Our convolutional design is rooted in group-theoretic modeling of time series deformations, which is a novel approach in this domain. Similar formalisms have introduced robust architectures for computer vision and graph neural networks. Our layers, coupled with lightweight modules (Figures 5, 6), can achieve robust and competitive performance, even on smaller datasets.\n\n**[Other]**\n\n>\"It is unfortunate to limit the study to trend and offset [...] discussed other types of invariants.\"\n\nBaseline removal is common in applications for physiological signals, such as EEGs and PPGs, where baseline wander can mask critical variations, and removing trends can enable more accurate feature extraction [1,2]. The closest easy-to-generate deformation is a smooth random walk, which we incorporate in the robustness experiment as the most complex deformation following simple offset shift and linear trend (see feature maps visualizations in the last row of Figure 8 for smooth random walk deformation of the FordB dataset in Figure 7). Offset shifts and linear trends often arise due to sensor drift, calibration errors, or gradual measurement degradation, leading to misleading variations in the data.\nWhile z-normalization removes global offsets, it fails to address local distortions. Our model learns local invariance to offset shifts and trends, making it effective for real-world data. This justifies our choice of offset shift and linear trend as key deformations to experimentally validate the broader theoretical framework of time series invariant layers under group actions.\n\nIn the introduction (paragraphs 4–5, pages 1–2), we discuss time series invariances, including time-shift, time-rescaling, and contrastive learning-based transformations. Commonly addressed deformations include amplitude scaling, offset shifts, and trends, tackled via z-normalization or contrastive frameworks (e.g., TS-TCC). We introduce a formal mathematical framework for time series invariance to spatiotemporal deformations, providing exact formulations of invariances rather than relying on approximations.\n\nWe restrict invariant convolutions to simpler but common affine deformations, allowing invariance to trend when it can be assumed linear at the scale of the convolution kernel size. However, we plan to explore approximating the trend with non-linear functions, such as splines or higher-degree polynomials, and incorporating seasonal components with low-frequency cosine bases in future work.\n\n[1] Kaur, M., Singh, B., \\& Seema. (2011). Comparing approaches for baseline wander removal in ECG signals. Proc. Int. Conf. \\& Workshop on Emerging Trends in Technology, 1290-1294.\n\n[2] Awodeyi, A. E., Alty, S. R., \\& Ghavami, M. (2014). On filtering photoplethysmography signals. IEEE Int. Conf. Bioinformatics \\& Bioengineering, 175-178.", "claims_and_evidence": "Numerous experiments support the authors' claims regarding the advantage of incorporating hard-coded invariants into convolutions. A first series of experiments artificially introduces distortions to demonstrate the robustness of their method compared to state-of-the-art approaches (Table 1), while comparing results across different types of invariance and convolution. An ablation study is also conducted on classification results to highlight the role of each component (Table 3). Finally, transfer learning experiments are carried out to demonstrate the relevance of the approach in this context.", "methods_and_evaluation_criteria": "The approach of evaluating the proposed model in two stages is sound, first by introducing artificial distortions and then testing it in real-world conditions. The chosen datasets are standard in the community but known to be simple. The comparison of training times between the different models is also a good aspect. The authors could have produced a synthetic diagram of the model's performance compared to others, such as a mean rank, as is standard in the field. It would have certainly shown slightly lower performance of the proposed model compared to Rocket, as indicated in the exhaustive results table (Table 11) at the end of the appendix, but it could have provided clearer insight for the reader", "theoretical_claims": "The article is based on the introduction of an action group for time series. I have read the formalism in detail in the main body of the article and skimmed through the appendices. The formalism is well described and requires some mathematical concepts, not all of which are explicitly stated but remain at a modest level. The examples are well chosen to illustrate the most formal parts.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "\"To my knowledge, the most important papers in the field are well cited. The authors make a clear effort to position their work in the introduction relative to the state of the art and existing research on invariants for time series. Their proposed idea goes somewhat against the current trend—explicitly designing constraints to integrate into convolutional kernels, whereas the field generally focuses on developing models that can handle invariance in a more generic way. Consequently, the impact of this work may be limited, as there are few real-world cases where invariances are so explicitly defined in my opinion.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "It is unfortunate to limit the study to trend and offset invariants, which can be handled by other approaches. The authors could have discussed other types of invariants to broaden the relevance of their work.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZdwVWwqt6", "reviewer": "Reviewer_Bfty", "summary": "The paper proposes convolutional neural network operations explicitly designed with hard-coded invariances (e.g., scaling, offset shift, linear trends) for improved time series representation learning. By formulating invariances through group theory and embedding them directly into convolutional layers, the authors show empirically that their method enhances robustness against common temporal deformations, achieves competitive or superior accuracy across benchmark classification tasks, and offers computational efficiency advantages over learned invariances or standard CNN approaches.\n\n## update after rebuttal\nMost of my concerns have been addressed. I will keep my score to support the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's time and effort in evaluating our work. Below, we provide our responses to the main suggestions.\n\n**[Claims And Evidence]**\n\n> \"authors claim computational efficiency benefits [...], additional explicit runtime comparisons [...]\". \n\nWe have demonstrated the computational efficiency of the proposed invariant layers over baselines, with memory complexity and training time comparisons shown on the UEA Heartbeat dataset (with 61 channels and 405 timestamps) in Figure 3. Similar time comparisons on larger datasets will be included in the revised manuscript. More details on the fast computation of our convolutional layers via FFT can be found in Section 3.2 (pages 4-5).\n\n**[Methods and Evaluation Criteria]**\n\n- **Additional Experiments.**\nWe next present comparisons for classification on the 26 UEA datasets and the 5 anomaly detection (AD) datasets, evaluating the additional methods: TimeMixer, iTransformer, Peri-midFormer, and UP2ME\n\n## Additional Results Classification Acc.(\\%)\n\n| Datasets            | **InvConvNet** | TimeMixer | iTransformer | PerimidFormer | UP2ME  |\n|---------------------|----------------|-----------|--------------|---------------|--------|\n| **UEA** (26 datasets) | **71.81 ± 0.80** | 64.60 ± 1.59 | 64.42 ± 2.05 | 59.71 ± 3.03 | 54.57 ± 3.88 |\n\n## Additional Results Anomaly Detection \n\n| Datasets  | **InvConvNet** | TimeMixer | iTransformer | PerimidFormer | UP2ME (4/5 datasets) |\n|-----------|-----------------|-----------|--------------|---------------|----------------------|\n| **SMD**   | **84.05 ± 0.16** | 83.20 ± 0.06 | 82.38 ± 0.99 | 83.34 ± 0.63  | **84.34 ± 0.13** |\n| **MSL**   | **80.68 ± 0.01** | 67.12 ± 3.33 | 72.66 ± 0.04 | 80.93 ± 0.06  | 80.66 ± 0.01 |\n| **SMAP**  | **68.29 ± 0.07** | 65.55 ± 0.32 | 66.86 ± 0.08 | 67.62 ± 0.02  | 67.56 ± 0.96 |\n| **SWaT**  | **92.82 ± 0.19** | 91.77 ± 1.30 | 92.68 ± 0.01 | 92.17 ± 0.05  | OOM |\n| **PSM**   | 96.34 ± 0.01     | 94.01 ± 0.77 | 95.15 ± 0.14 | 96.31 ± 0.09  | **96.42 ± 0.01** |\n| **Avg. F1 (\\%)** | **84.44 ± 0.09** | 80.33 ± 1.16 | 81.95 ± 0.25 | 84.07 ± 0.17  | 82.24 ± 0.28 |\n\nOur method consistently retains its advantage in both classification and anomaly detection tasks.  We were unable to obtain results for the SWaT dataset using UP2ME (marked with out-of-memory), even after reducing the hyperparameters to manage memory usage.\n\n**Implementation Details:** Based on their official GitHub implementations, only iTransformer has been adapted for classification and anomaly detection. Optimal hyperparameters are provided for iTransformer on 10 UEA datasets and all AD datasets, while for TimeMixer, we used default values for non-forecasting tasks. Hyperparameters for Peri-midFormer are sourced from its GitHub for classification (on 10 UEA datasets) and anomaly detection tasks. Additionally, we test the backbone of UP2ME. We adapted TimeMixer and UP2ME for classification by reshaping the encoder outputs into a one-dimensional vector and passing them through fully connected layers for predictions.\n\n- **Visualizations of Capturing Invariances.** \nWe will definitely include additional visualizations of the invariances captured by our layers in the revised manuscript. For now, we have included visualizations of offset and trend deformations on UCR datasets, along with feature maps for normal and invariant kernels, in Figures 7 and 8 of the Appendix.\n\n**[Experimental Designs Or Analyses]** \n\nWe will incorporate the reviewer's suggestions in the revised manuscript. We already provide extensive details on hyperparameter tuning for the proposed method and the baselines in **Section A.4 Implementation Details**. While results are currently marked based on mean performance and std overlaps, we will consider paired statistical t-tests for all datasets and best methods (not provided due to time constraints). Please also refer to the critical diagram difference for UEA in the rebuttal for **reviewer gvG9**.\n\n**[Clarity]**\n\n> \"The mathematics framework is hard [...] intuitive explanations.\"\n\nTo enhance readability, we will relocate Figure 2 to an earlier section of the manuscript to clearly showcase standard deformation types. The methods section will also add a short, intuitive introduction: \"The proposed mathematical framework consists of two main components. The first is a group action that formalizes how certain deformations transform time series, resulting in their deformed counterparts. Only deformed time series are observable in practice, such as those influenced by noise or trends. However, embeddings that remain invariant to these deformations are essential for many applications. To address this, the second component is a mapping function that constructs embeddings of deformed time series while remaining invariant to specific deformations. Finally, these embedding maps can be efficiently integrated into convolutional operations to extract robust local features to non-informative deformations.", "claims_and_evidence": "The claims presented in the paper are generally supported by clear and convincing evidence. The authors provide a thorough mathematical formulation of their hard-coded invariant convolutions and validate the theoretical properties empirically on multiple tasks, including synthetic deformation robustness tests, benchmark classification datasets, and transfer learning scenarios. The experimental setup is extensive and compares against relevant state-of-the-art baselines, clearly supporting the claims regarding improved robustness and competitive accuracy. However, while the authors claim computational efficiency benefits through FFT-based convolutions, additional explicit runtime comparisons or complexity analyses against more baseline architectures (especially in larger-scale or real-world scenarios) would further strengthen this claim.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally appropriate. But I think authors should include some more stronger baselines ([1] [2] etc.). Also, I suggest authors to provide more visualization results to demonstrate the effectiveness of capturing invariances in time series.\n\n[1] iTransformer: Inverted Transformers Are Effective for Time Series Forecasting\n[2] TimeMixer", "theoretical_claims": "I checked the correctness of the theoretical claims presented, specifically focusing on the mathematical formulations related to group invariances and the definition of invariant convolution operators (Section 3, including Propositions 1 and 2). The proofs provided in Appendix A.1 (referred to clearly within the paper) appear mathematically sound and correctly derived, relying appropriately on established concepts from group theory and functional analysis. But I am not very familiar with this field, so I am not very confident about my judgement.", "experimental_designs_or_analyses": "I have checked the soundness/validity of experimental designs and analyses. The experimental designs and analyses presented in the paper appear sound and valid. Specifically, the robustness evaluation involving synthetic deformations (offset shifts, linear trends, random walks) provides clear evidence to support the claimed invariances. Benchmarking against multiple state-of-the-art baselines across diverse datasets from the widely used UCR and UEA archives ensures rigorous comparative assessment. The transfer learning experiments are also well-designed, clearly illustrating the generalization capabilities of the proposed method. One minor area for improvement could be providing additional details on hyperparameter tuning and statistical significance testing for the observed improvements to further reinforce the validity of the conclusions.", "relation_to_broader_scientific_literature": "The paper's key contributions closely relate to recent research in deep learning for time series, particularly efforts to enhance model robustness via invariant representations. Specifically, the authors build upon concepts from group theory and invariant/equivariant neural networks, aligning with prior research on hard-coded invariances in domains such as images and graphs (e.g., translation invariance in CNNs, permutation invariance in GNNs). Unlike prevalent methods that introduce invariances implicitly through data augmentation and contrastive learning (e.g., TS-TCC, TS2Vec), this paper explicitly incorporates invariances into convolutional architecture design, extending established ideas (e.g., ROCKET's random convolutions) with rigorous theoretical foundations. Thus, the presented method bridges prior theoretical findings about invariance modeling with practical CNN-based approaches widely adopted in time series classification, positioning itself clearly and convincingly within the broader scientific landscape.", "essential_references_not_discussed": "To my best knowledge, essential references are discussed.", "other_strengths_and_weaknesses": "Strengths: \n* Originality: The paper demonstrates clear originality by creatively combining established mathematical concepts (group invariances) with convolutional neural network architectures, leading to explicit and computationally efficient invariant representations for time series.\n* Significance: The contributions are significant, addressing important practical challenges in the robustness of deep learning models for real-world temporal data.\n\nWeaknesses:\n* Clarity: the mathematics framework is hard to understand. It would be better if authors could give some intuitive explanations.", "other_comments_or_suggestions": "I am willing to raise my score if authors could address my concerns.", "questions_for_authors": "I do not have other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DV8CnhbLgR", "reviewer": "Reviewer_7HtV", "summary": "This paper proposes a novel mathematical method to consider the deformation-invariance during representation learning, which is beneficial for downstream tasks such like classification. It designs a G-variant convolution model called TS-TCC to obtain deformation-invariant embeddings, which provides robustness by decoupling the key information from deformations. Theoretically, this paper proposes using group actions to represent deformations and proves that orbit-injective embedding could map the orbit (the deformations on some time series) to the same embedding, thus avoiding the adverse effects from deformations during training. Empircally, this paper constructs the Invariant & Orbit-injective embedding through a convolution and validates its performance on the classification task.\n\n## update after rebuttal\nThank you for the respnse, which clarify things. I will keep my relatively positive perspetive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and thoughtful evaluation of our work. In the following section, we address any concerns raised.\n\n**[Claims and Evidence]**\n\n> \"However, the basic assumptions about the deformation phenomenon seem not strong enough[...] While in forecasting tasks, such decomposition methods (moving average, convolutions) are widely used [...]\"\n\nFor time series forecasting, the suggested papers [1,2,3,4] assume a seasonal-trend decomposition of time series to derive suited ANN architectures. Most methods employ a (learnable) moving average kernel to infer the trend and propose different ways to deal with residual time series. The average moving kernel corresponds to a specific deformation in the proposed framework: the offset shift (see Figure 2). Assuming such offset-invariant kernels, the proposed layer offers different views of the residual time series. When focusing on trend deformations, the trend approximation could be better inferred by assuming a higher-order Taylor expansion (linear, quadratic, etc.). In the anomaly detection (Appendix A.5.1) and forecasting experiments (see rebuttal reply **Q3** to **reviewer MxbV**), we also leverage trend-residual decomposition in lightweight architectures by feed-forwarding the projections coefficients to the reconstruction layer. \nIn classification, PerimidFormer (Classification Table in rebuttal response to **reviewer Bfty**) and Dlinear (Table 2 of the manuscript) are also based on a trend-residual decomposition, yet are significantly outcompetes by the proposed invariant convolutions. For instance, on the UEA datasets, our example architecture InvConvNet, performs better than PerimidFormer and Dlinear (acc: 71.81 vs (59.71 and 61.51 resp.)), suggesting that deformation-based trend invariance is better suited than standard trend-residual decomposition for classification with neural networks. Overall, the proposed layer seems more versatile and robust compared to existing architectures on several tasks while guaranteeing state-of-the-art performances.\n\n> \"The authors seem only consider some naive linear deformations [...] are ANNs really hard to handle these?\"\n\nStandard ANNs often struggle with generalization on noisy, non-stationary time series, as illustrated in Figure 1, where a standard CNN fails to capture relevant ECG features. To test whether CNNs can learn invariance, we conduct a robustness experiment (Table 1), showing that classification performance declines as datasets are progressively deformed. Standard convolutions prove sensitive to spatiotemporal distortions (avg. acc. drop: -50\\%), whereas hard-coded invariant convolutions offer better generalization  (avg. acc. drop: -2\\%) than contrastive learning (avg. acc. drop: -23\\%). Figure 8 (Appendix) further visualizes this, revealing that standard CNN feature maps become distorted, while invariant convolutions retain structured representations (see Figure 7 for corresponding FordB dataset deformations). These findings align with (Kvinge et al., 2022) for image data. Finally, in transfer learning (Table 4), invariant convolutions surpass standard architectures and contrastive frameworks by at least 4\\%.\n\n- Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., \\& Lew, J. (2022). In what ways are deep neural networks invariant and how should we measure this?. NeurIPS, 35, 32816-32829.\n\n>The paper proposes a representation method [...] tasks such as forecasting, anomaly detection? \n\nWe conduct extensive experiments across diverse real-world applications to validate our theoretical framework of time series invariant convolutions, leveraging common offset and shift invariances in shallow (Figures 5 and 6) and lightweight example architectures (Figure 3). We have demonstrated robustness against offset and trend deformations (Section 4.1), strong performance in multivariate classification (Section 4.2), and transfer learning for classification (Section 4.3). Additionally, we extend our analysis to anomaly detection in Appendix A.5.1. Rather than proposing a general-purpose architecture, targeted to specific time series tasks, our goal was to develop versatile layers that integrate seamlessly with existing methods to enhance robustness. To illustrate this, we employ lightweight architectures—such as linear decoders for regression—that, despite their simplicity, match or surpass complex baselines. While our current focus is classification and anomaly detection, our approach naturally extends and holds promise for forecasting, as noted in our response and experiments provided to **reviewer MxbV** (rebuttal reply **Q3**).\n\n**[Experimental Designs Or Analyses]** \nWe show improved performance against the suggested baselines UP2ME [5] and PerimidFormer [6]. Please refer to the Tables in the rebuttal reply to **reviewer Bfty**. We will also include a discussion about the additional baselines in the revised version of the manuscript (not added here due to the word limit).", "claims_and_evidence": "The paper manages to solve the deformation phenomenon in time series representation learning, which hinders the capture of key information.  To achieve this, it starts from a mathematical methodology, utilizes the group theory and mesurement theory to construct a specific convolution. The theoretical aspects are correct and can support the author's motivations.\n\nHowever, the basic assumptions about the deformation phenomenon seem not strong enough. The showcase in Figure 1 conveys the information that the key information (seems to be the period) could be coupled in the trend ( a kind of deformation). While in forecasting tasks, such decomposition methods (moving average, convolutions) are widely used [1,2,3] to extract the trend. CycleNet [4] has also demonstrated that global shared periods can be easily extracted through a learnable matrix. In other words, the authors seem only consider some naive linear deformations (the equation 2 and Figrue 2), are ANNs really hard to handle these? \n\n[1]  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.\n[2] Are transformers effective for time series forecasting?\n[3] Duet: Dual clustering enhanced multivariate time series forecasting.\n[4] Cyclenet: enhancing time series forecasting through modeling periodic patterns.", "methods_and_evaluation_criteria": "Though the methodology is correct and novel, the benchmarking is somewhat weak. The paper proposes a representation method while only validating its performance on classification tasks.  Since the deformation phenomenon is ubiquitous, why not explore its performance on various downstream tasks such as forecasting, anomaly detection (should be discussed in the main text)?", "theoretical_claims": "The proofs are correct.", "experimental_designs_or_analyses": "It is recommended to evaluate the effects on several other downstream tasks such as forecasting and anomaly detection. The baselines are also somewhat outdated that only one work is published in 2024. The authors can take UP2ME [5],  Peri-midFormer [6] into consideration. \n\n[5] UP2ME: Univariate Pre-training to Multivariate Fine-tuning as a General-purpose Framework for Multivariate Time Series Analysis\n\n[6]  Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis", "relation_to_broader_scientific_literature": "Discussing some transformation problems from the perspective of group theory as in this paper is a research prospect, because mathematical methods often bring strong constraints. As long as the assumptions are correct, the probability of effectiveness will be greater.", "essential_references_not_discussed": "More recent studies should be discussed in the related works and added as baselines. See [5] - [6].", "other_strengths_and_weaknesses": "none", "other_comments_or_suggestions": "none", "questions_for_authors": "see previous sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ec9BnpqYmA", "reviewer": "Reviewer_MxbV", "summary": "This paper posits that invariances to the deformations are critical for time series tasks such as classification. The paper mathematically formulates the invariance in the language of group theory and further technically designs efficient and hard-coded invariant convolutions for specific deformations commonly observed in time series (scaling, offset shift, and trend). Experiments are conducted on time series classification and anomaly detection tasks to verify its effectiveness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's time and thoughtful feedback. In the following, we thoroughly address each identified weakness and question.\n\n**[Experimental Design]** \n\n- **Performance on UCR normalized data:** \nWe address your concern in Q1 below. To clarify, synthetic deformations are used only in test sets for the robustness study of convolutions (Section 4.1, Table 1). In contrast, in all other experiments (Sections 4.2–4.4 for classification and Appendix A.5.1 for anomaly detection), performance is evaluated on default data without added deformations.\n\n- **Significance of results:** We invite the reviewer to see the critical diagram difference for UEA in the rebuttal for **reviewer gvG9**.  \n\n**[Other]**\n\n> The presentation of the paper could be improved [...].\n\nTo enhance readability, we will move Figure 2 to an earlier section of the manuscript to better standard deformations and add a brief intuitive introduction in the method section. See our response to **reviewer Bfty** in section **[Clarity]** for details.\n\nReplies to Questions:\n\n> Q1. *InvConvNet does not exhibit optimal performance on normalized data [...]. What is the practical significance of artificially adding deformations?*\n\nIn the robustness experiment (\"I. Robustness to Deformations\", section 4.1), the performance of invariant convolutions compared to normal ones on the plain (i.e., z-normalized input without deformations) is better for 2 out of 5 plain UCR datasets. It is still close for the rest of the cases (average -4.7\\% drop for the 3 data cases). Contrary when synthetic deformations are added, the performance drops for normal convolutions are around -50\\% (on average for the 4 considered deformations), whereas for the trend invariant convolution, this drop is just around -2\\% (on average for the 4 considered deformations) showing the competitiveness and robustness of the proposed layer in comparison to its standard counterparts.\n\nThe motivation for the robustness study on UCR datasets is as follows: Most UCR datasets, including those used in this study, are already z-normalized. Their univariate nature, limited class diversity, and relative trend stability make classification easier for normal convolutions. To systematically assess the impact of synthetic deformations, we use these datasets in a controlled setting, progressively increasing deformation complexity (from offset shifts to linear trends and smooth random walks) to evaluate how different convolutional components, including those invariant to such distortions, respond.\n\n> Q2. *How can you ensure that the added deformations are justifiable for the specific dataset?*\n\nOffset and trend invariances are common in real-world applications like PPG monitoring and ECG analysis (see reply **[Other]** to **reviewer gvG9**). They are tackled by baseline wander removal and offset correction techniques that preserve physiological signals while eliminating low-frequency noise. The closest easy-to-generate deformation, a smooth random walk, is included in the robustness experiment (see Table 1).\n\n> Q3. *Can invariant convolution enhance the performance of time series forecasting?*\n\nOur invariant layers, combined with the example decoder used for anomaly detection, naturally extend to forecasting by employing the same lightweight decoder based on linear layers applied to the learned coefficients. We next provide some preliminary results (in MSE) for the ETT-small datasets (for horizon length h=96), where we use a fixed seed and compare our approach against 8 baselines (most derived from the package Time Series Library). \n\n| Datasets (h=96) | **InvConvNet** | **TimeMixer** | **PerimidFormer** | **iTransformer** | **PatchTST** | **DLinear** | **TimeNet** | **FedFormer** | **Autoformer** |\n|-----------------|----------------|---------------|-------------------|------------------|--------------|-------------|-------------|----------------|----------------|\n| **ETTm1**       | 0.342          | **0.319**     | 0.325             | 0.345            | 0.325        | 0.347       | 0.337       | 0.365          | 0.486          |\n| **ETTm2**       | 0.193          | **0.178**     | 0.180             | 0.184            | **0.178**    | 0.195       | 0.187       | 0.194          | 0.215          |\n| **ETTh1**       | 0.426          | 0.388         | **0.377**         | 0.402            | 0.380        | 0.407       | 0.394       | 0.378          | 0.463          |\n| **ETTh2**       | 0.340          | **0.289**     | 0.322             | 0.299            | 0.312        | 0.357       | 0.330       | 0.349          | 0.343          |\n\nThese results, achieved without hyperparameter tuning (in less than a week), are close to SOTA performance, highlighting their potential. Future improvements include testing hybrid architectures for the decoder (e.g., transformers) to enhance finer granularity, capture longer dependencies, and unsupervised pretraining for better generalization.", "claims_and_evidence": "The claims are supported by both theoretical and empirical evidence.", "methods_and_evaluation_criteria": "The mathematical formulation of the problem and the design of the method are clear, and the selection of evaluation criteria is well-justified.", "theoretical_claims": "The mathematical formulation is rigorous and sound, building on group theory and projection operators. Specifically, propositions 1–2 are correctly proven using orthogonal projection operators, ensuring invariance to specified deformations.", "experimental_designs_or_analyses": "The experiments are thorough, including the results of multiple tasks (classification, anomaly detection) and efficiency analysis.\nHowever, some of the results raised my concerns:\n1. In Table 1, InvConvNet is not optimal on normalized data (without additional deformation), which questions its practicality.\n2. In Table 2 and Table 11, the performance improvement is not markedly significant. For instance, InvConvNet compared to the second-best baselines:\nUEA 71.81 over 71.29, +0.52 (0.7%),\nUCIHAR 96.63 over 96.04, +0.59 (0.6%),\nEpilepsy 98.43 over 98.38, +0.05 (0.05%).", "relation_to_broader_scientific_literature": "This work connects to broader literature by formalizing time series invariances through group actions, extending principles from image invariance to the time series domain. It draws inspiration from the traditional time series mining methods' focus on invariance (e.g., warping invariance through DTW, amplitude and offset invariances through Z-normalization) and proposes invariant convolutions for deep learning, offering a new perspective for handling deformations in time series analysis.", "essential_references_not_discussed": "To the best of my knowledge, all key references have been thoroughly addressed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel approach to capturing invariances from deformations in time series, which is significant for tasks such as time series classification. Technically, a novel convolution, Inv. Conv, is proposed to keep invariant to rigid deformations. It is computationally efficient through FFT.\n2. As mentioned in Theoretical Claim, the mathematical formulation is rigorous and sound, building on group theory and projection operators.\n3. The experiments are thorough, including the results of multiple tasks (classification, anomaly detection) and efficiency analysis. Additional results in supplementary material further improve the quality of the paper.\n\nWeaknesses:\n1. The presentation of the paper could be improved; some sections are dense, making it challenging to follow for those less familiar with group theory.\n2. As mentioned in Experimental Design, in Table 1, InvConvNet is not optimal on normalized data (without additional deformation), which questions its practicality.\n3. As mentioned in Experimental Design, the performance improvement is not markedly significant in Table 2 and Table 11.", "other_comments_or_suggestions": "I suggest relocating Figure 2 to an earlier section of the manuscript, facilitating an intuitive comprehension of deformation for readers as they engage with the introduction section.", "questions_for_authors": "1. Based on Weakness 1, InvConvNet does not exhibit optimal performance on normalized data, and it only demonstrates a performance advantage when artificial deformations are introduced. However, raw data inherently possesses distribution shifts. Thus, what is the practical significance of artificially adding deformations? Furthermore, how can you ensure that the added deformations are justifiable for the specific dataset? For instance, introducing certain deformations to ECG signals might cause anomalies to the entire dataset that contravene established medical principles.\n2. Time series forecasting methods are discussed in Related Work. Can invariant convolution enhance the performance of time series forecasting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes to encode invariances to certain groups (e.g., scaling, shift, etc.) directly in a convolutional architecture, in order to endow the deep network with robustness to such prevalent deformations.

All in all, the paper was well received, and three out of four reviewers are positive. In general, most reviewers appreciated the soundness of the theoretical contribution, the experimental design, and the empirical results. Moreover, the rebuttal was effective and helped to increase scores.

The AC believes that this paper can have a positive impact in the field of time-series analysis and recommends Accept.

Remark: In the camera-ready version, the authors may want to mention other types of deformations, even if these are not included in their set of invariances. For example, several recent works in NeurIPS, ICML, and other venues focus on diffeomorphisms as a way of modeling nonlinear time warping in time-series data.
Decision: Accept (poster)