Invariant Convolutional Layers for Time Series

Authors: Thibaut Germain, Chrysoula Kosma, Laurent Oudre
Primary area: learning on time series and dynamical systems
Keywords: time series, convolution, invariances, neural network

Abstract:
Machine learning for time series has recently garnered considerable attention. Indeed, automatically extracting meaningful representations from large and complex time series data is becoming imperative for several real-world applications. Neural architectures tailored to time series are often built upon sequential modules, such as convolutional, commonly employed in text or vision. Unfortunately, the potential of standard layers in capturing invariant properties of time series remains relatively underexplored. For instance, convolutional layers often fail to capture underlying patterns in time series inputs that encompass strong deformations, such as linear trends. However, invariances to some deformations may be critical for solving complex time series tasks, such as classification, while guaranteeing good generalization properties.
To address these challenges, we mathematically formulate and technically design efficient *invariant convolutions* for specific group actions applicable to the case of time series.
We construct these convolutions by considering two sets of deformations commonly observed in time series, including (i) *offset shift and scaling* and (ii) *linear trend and scaling*.
We further combine the proposed invariant convolutions with standard (or variant) convolutions in a single embedding layer of an example architecture, the so-called *InvConvNet* method, and showcase the layer capacity to capture complex invariant time series properties.
Finally, *InvConvNet* is experimentally proven to achieve superior performance against common baselines in relevant time series tasks, including classification and anomaly detection.

Decision: reject
Reviewer scores: 5, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729664327686

thread_type:
official_review

<thread_object>
creator: reviewer_ao1z

<content>
summary:
This paper introduces a new type of convolutional neural network layer, invariant convolutional layers, which are designed for time series data. The layer can capture invariant features by taking into account common deformations in time series, such as shifts, scaling, and linear trends. This paper proposes an architecture called INVCONVNET, which combines standard and invariant convolution to improve the performance of time series classification and anomaly detection. The experimental results show that INVCONVNET outperforms existing baseline methods on multiple datasets with better generalization and computational efficiency.

strengths:
This paper contains a large number of mathematical proofs, showing that convolution can capture time series invariance, and sufficient experiments in time series classification and anomaly detection demonstrate the importance of the proposed invariant convolution.

weaknesses:
The INVCONVNET model presented in the paper, while excellent at time series classification and anomaly detection tasks, may have some limitations.Among the existing methods for capturing time series invariance, linear models and frequency-domain models （Dlinear and FITS) perform well and are not compared with them in this paper.

questions:
1.The time series itself contains variable features and time invariant features, so why design only for time invariant features?
2.As the conversion of time series to frequency domain proves to be a suitable method for capturing time-invariant features, why does this paper not consider converting time series to frequency domain for learning by convolution first?
3. I find that there are many research about Time-invariant convolution,please further clarify the contribution of this article
[1]. Chaman, A., & Dokmanic, I. (2021). Truly shift-invariant convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3773-3783).
[2].Chen, X., Cheng, Z., Cai, H., Saunier, N., & Sun, L. (2024). Laplacian convolutional representation for traffic time series imputation. IEEE Transactions on Knowledge and Data Engineering.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730129803683

thread_type:
official_review

<thread_object>
creator: reviewer_vw6u

<content>
summary:
This paper introduces a novel approach to handling time series data by developing convolutional layers that are invariant to specific types of deformations commonly found in time series, such as offset shifts, scaling, and linear trends. The author mathematically formulate invariances for time series using group theory, designed invariant convolutions that can handle these deformations. The effectiveness of the proposed method is empirically validated through classification and anomaly detection tasks.

strengths:
1. The overall paper is clearly written and easy to understand, the paper structures are well organized.

2. The mathematical formulation standardize the invariant definition for certain time series properties

3. Efficient implementation of invariant convolutions using FFT

weaknesses:
1.	My major concern is the gap from theoretical definition of time series invariants to the actual empirical improvement on time series tasks. For CNN base time series model, the major factor to improve classification task performance is tuning multi-scale convolution kernel sizes (check paper: https://arxiv.org/abs/2002.10061). This reviewed paper seems also using this “Pool of convolutions” (line 302). Without more detailed ablation studies for each of the components, it is hard to justify where the actual contribution come from.

2.	Following the above discussion, for time series classification tasks, the SOTA models are still cnn based not transformer based. Therefore, it is more meaningful to use cnn based time series models such as TimeNet or Omni-scale CNN as baselines instead of transformer based models.

3.	The invariant convolution layer implementation is not very clear. Upon checking the appendix, there still lacks a detailed description of neural network architectures such as layer number/ kernel size for each specific layer.

questions:
1.	How does the choice of kernel sizes affect the model's ability to capture different types of invariances? Is there a systematic way to select optimal kernel sizes based on understanding of “invariants”?

2.	Can you show some quantitative results that the computation efficiency gain of FFT-based convolution layer over classical kernel with longer time series?

3.	How is the proposed new invariant feature linked to/different from some standard normalization treatments in ML? i.e. the example in line 256 looks like very similar to kernel normalization.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469364824

thread_type:
official_review

<thread_object>
creator: reviewer_akru

<content>
summary:
The paper tackles the challenge of developing invariant convolutional layers tailored for time series. It begins by introducing a mathematical framework for invariant convolution in time series, followed by the design of specific architectural blocks that ensure invariance to offset shifts, scaling, and linear trends. These invariant blocks are then integrated with convolutional layers. The proposed approach is evaluated through experiments on both univariate and multivariate time series, covering tasks such as classification and anomaly detection.

strengths:
- S1. The paper addresses the compelling problem of designing invariant deep learning models for time series, which is valuable for tasks where invariance to certain properties, such as offset, is essential. This is particularly relevant in applications like classification, where only specific patterns are informative. 

- S2. Additionally, the paper explores a relatively uncharted area of geometrical deep learning for time series, providing valuable insights into this underdeveloped research area. 

- S3. The appendices are thorough and greatly enhance the reader’s understanding of the model architecture and the processes used in the supervised tasks, especially Appendix A.2 ("InvConvNet: Architectural Details"), which contains essential architectural details. 

- S4. The experimental evaluation is comprehensive, covering two key supervised tasks (classification and anomaly detection) and providing a broad perspective on the model’s performance.

weaknesses:
- W1. The positioning of the work is somewhat unclear, as it focuses exclusively on specific invariance properties for time series without connecting to the broader research in convolutional neural network invariances. Invariance is a well-studied area in deep learning (e.g., CNN shift-equivariance, see [1]), and discussing this literature could enrich the context and impact of the paper. 
- W2. The paper is challenging to follow: 
    - The extensive two-page mathematical formalism on invariant embeddings for time series may not aid reader comprehension as intended.
    - The architecture description lacks clarity; key elements are dispersed and essential components are placed in the appendices (such as in Appendix A.2), making it difficult to grasp the model’s structure. Reorganizing and detailing the architectural explanations would enhance readability. 
    - Consider moving the content of Appendix A.2 to the main body and placing the formalism of invariant embeddings in the appendix to streamline the presentation of contributions. 
 
- W3. The experimental results are not stellar. In the classification tasks, the InvConvNet model does not significantly outperform ResNet and Rocket on the UEA archive datasets (Table 1 and Table 9). Furthermore, on four selected UCR datasets, InvConvNet performs worse than its variant without invariant blocks (Table 3), despite showing some robustness to deformation. 
- W4. The practical value of the proposed InvConvNet architecture is unclear. It may be beneficial to focus on specific datasets where invariance to offset and trend is particularly desirable, which could demonstrate the model's relevance in applied scenarios.

questions:
- Q1. The description of the Transfer Learning Experiment is unclear. Could you provide a more detailed explanation of this experimental setup ?

- Q2. Could you please discuss other invariance like shift invariance?

- Q3. Could you address the concerns raised in the weaknesses section ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621959511

thread_type:
official_review

<thread_object>
creator: reviewer_wiu6

<content>
summary:
The paper introduces INVCONVNET, a novel approach addressing time series analysis through mathematically-grounded invariant convolutions. The authors formulate a theoretical framework for handling common deformations in time series data, specifically targeting offset shifts, scaling, and linear trends. The key technical contribution lies in their construction of group action-based invariant operations, which they effectively combine with standard convolutions in a hybrid embedding layer. The work bridges theoretical foundations of invariant representations with practical architectural design, demonstrating applications in both classification and anomaly detection tasks. The authors provide mathematical formulations of their invariant convolutions and validate their approach experimentally against baseline methods.

strengths:
1. Theoretical Foundation: The authors provide formal mathematical definitions of time series deformations in terms of group actions, creating a theoretically sound framework rather than just an empirical solution.
2. Structured and Explicit Design: Rather than relying on implicit learning of invariances through data augmentation or contrastive learning, they develop an explicit, structured approach to incorporating invariance properties directly into the architecture. This makes the solution more transparent and controllable.
3. Practical Problem with Clear Motivation: They demonstrate their approach through concrete examples like the ECG case, where standard convolutions fail with linear trends masking important patterns, showing clear real-world applicability.

weaknesses:
- Fixed Kernel Bias: The initial motivation using a pre-defined kernel creates a biased comparison, without adequately addressing whether standard CNNs with learned kernels could naturally adapt to these deformations.
- Missing Comparative Analysis with Contrastive Learning: The authors don't compare their architectural approach against the simpler alternative of using these deformations (offset, trend) as augmentations in a contrastive learning framework, which could potentially achieve similar invariance properties with more flexibility.
- Incomplete Novelty Justification: The paper lacks proper positioning against existing relevant work on invariant architectures, particularly:
    - Jacques, Brandon G., et al. "A deep convolutional neural network that is invariant to time rescaling." *International conference on machine learning*. PMLR, 2022.
    - "Truly shift-invariant convolutional neural networks" A comparison with these works is crucial to establish the novelty and advantages of their approach.
- Contradictory Results to Core Motivation: The paper's fundamental premise is that invariance to deformations (offset, trend) is crucial for time series analysis. However, Table 3 directly contradicts this claim, showing InvConvNet-N (without invariance) performs best in 3 out of 4 clean datasets. This raises serious questions about the paper's core motivation and the necessity of architectural invariance.

questions:
Check the Weakness Section
</content>
</thread_object>

</thread>
<thread>
creation date:
1732006799592

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #3

comment:
**[Additional Results][Including the Performance of InvConvNet-N (built solely upon normal convolutions) to the Transfer Learning Experiment (Table 4):]**
In Table 4 of the original manuscript, we present the classification results of the Transfer Learning experiment on the *Fault-Diagnosis* dataset, achieved by **InvConvNet** and self-supervised methods TS-TCC and TS2Vec. We extend Table 4 by training on the source dataset (in a supervised manner) and evaluating on the target datasets, the InvConvnet-N configuration, which is built solely on standard, non-invariant convolutions (contrary to the proposed InvConvNet that mixes variant and invariant convolutions). The performance is mentioned in terms of Accuracy (%).

| **Methods**      | **A → B**           | **A → C**           | **A → D**           | **B → A**           | **B → C**           | **B → D**           | **C → A**           | **C → B**           | **C → D**           | **D → A**           | **D → B**           | **D → C**           | **Avg. Acc. (%)**  |
|-------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|
| **TS-TCC**        | 55.33 ± 1.44       | 52.52 ± 4.55       | **62.13 ± 1.39**    | 48.05 ± 3.32       | 71.50 ± 1.83       | **100.0 ± 0.0**     | 40.76 ± 2.22       | **98.25 ± 1.22**    | **99.34 ± 0.50**    | 46.98 ± 0.65       | **100.0 ± 0.0**     | 74.28 ± 2.77       | 70.76 ± 1.66       |
| **TS2Vec**        | 54.11 ± 1.46       | 54.07 ± 1.91       | 52.54 ± 1.89       | 55.06 ± 0.17       | **88.72 ± 0.47**    | **100.0 ± 0.0**     | 57.81 ± 2.18       | 78.30 ± 3.80       | 78.41 ± 4.39       | 60.37 ± 1.95       | 99.97 ± 0.02       | 86.82 ± 0.54       | 72.18 ± 1.57       |
| **InvConvNet**    | 55.90 ± 0.42       | **55.93 ± 0.34**    | 53.41 ± 0.14       | **85.10 ± 0.63**    | 78.54 ± 0.17       | 99.05 ± 0.08       | **70.75 ± 1.32**    | 85.04 ± 0.13       | 85.12 ± 0.15       | **70.91 ± 0.73**    | **100.0 ± 0.0**     | 78.49 ± 0.38       | **76.52 ± 0.37**    |
| **InvConvNet-N**  | **60.55 ± 0.88**   | **55.50 ± 1.82**    | 53.50 ± 0.85       | 60.26 ± 2.01       | 77.30 ± 0.60       | 93.50 ± 0.90       | 64.93 ± 0.48       | 84.87 ± 0.23       | 84.46 ± 0.51       | 59.98 ± 0.98       | 99.96 ± 0.0        | 77.14 ± 0.14       | 72.66 ± 0.78       |

Again, here we confirm the superiority in terms of generalizability of the proposed InvConvNet method, which also incorporates invariant convolutions, against the InvConvNet-N configuration that considers only standard convolutions. This ablation shows here (similar to the rest of classification experiments) that invariant filters can be transferred to unseen subsets of data or different target domains of data and achieve superior performances compared to standard ones that do not incorporate invariances. When compared to self-supervised methods that are pre-training and fine-tuned on the source domain, interestingly, we observe that even without learning general-purpose representations agnostic to the task, our proposed invariant convolutions are able to learn features transferable to the unseen target domain. This can be attributed to the successful extraction of invariant features through our specifically designed layers.

We thank the reviewers once again, and based on their follow-up comments on our explanations, we aim to revise the original manuscript to better address the above aspects.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732006816024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #3

comment:
**3. [Comment 3][[Details on the Transfer Learning Experiment and Additional Results:]** For the transfer Learning Experiment, we followed the experimental setup proposed by [12] on the Fault-Diagnosis dataset [13]. This dataset has been extracted under four different working conditions, each being considered a separate domain due to distinct underlying characteristics [14]. Details about this dataset can be found in Appendix A.3, Table 7, and page 18 - paragraph 2. For the self-supervised contrastive learning frameworks (TS-TCC, TS2Vec), following the work of [12], we perform self-supervised pre-training and fine-tuning on the source domain (left side of the arrow in Table 4) and test the generalization performance by measuring the accuracy on the target domain (right side of the arrow in Table 4). Similarly, for our supervised method (InvConvNet), we perform supervised training on the source domain data and test the performance of the unseen target domain. Even with the absence of general-purpose representations agnostic to the task, our proposed invariant convolutions are able to learn features transferable to the unseen target domain, which can be attributed to the successful extraction of invariant features through our specifically designed layers.

[12] Eldele, E., Ragab, M., Chen, Z., Wu, M., Kwoh, C. K., Li, X., \& Guan, C. (2021). Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112.

[13] Lessmeier, C., Kimotho, J. K., Zimmer, D., \& Sextro, W. (2016, July). Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference (Vol. 3, No. 1).

[14] Ragab, M., Chen, Z., Wu, M., Li, H., Kwoh, C. K., Yan, R., \& Li, X. (2020). Adversarial multiple-target domain adaptation for fault classification. IEEE Transactions on Instrumentation and Measurement, 70, 1-11.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732006886614

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #2

comment:
**[Additional Results][Additional Deformation on the Synthetic Experiment (Table 3):]** Here, we extend Table 3 by considering a fifth additional synthetic deformation added to the UCR considered datasets, named *offset shift + smooth random walk*. For this deformation, we considered as a synthetic added trend a random walk that is generated from a Gaussian distribution and smoothed by rolling mean, and we also added an offset sampled from a uniform distribution. We revise below in Table 3 the classification results for the synthetic experiment on UCR, considering this additional deformation. 

**Table 3:** Robustness study of InvConvNet and its standalone declinations, i.e., -N (normal), -O (offset) or -T (trend), on the $4$ larger UCR datasets under an additional scenario of synthetic deformations: addition of random walk generated trend. 

| **Dataset**                  | **Configuration**        | **InvConvNet (ours)** | **InvConvNet-N (-normal)** | **InvConvNet-O (-offset)** | **InvConvNet-T (-trend)** |
|------------------------------|--------------------------|-----------------------:|---------------------------:|---------------------------:|---------------------------:|
| **HandOutlines**              | *Normalized*             | 74.80                 | **80.60**                 | 72.70                     | 69.82                     |
|                              | + *off.*                 | 71.20                 | 68.70                     | **71.60**                 | 70.18                     |
|                              | + *LT*                   | **74.00**             | 72.90                     | 71.70                     | 70.18                     |
|                              | + *off., LT*             | 71.70                 | 70.70                     | 70.20                     | **71.71**                 |
|                              | + *off., RW*             | 65.68                 | 61.89                     | **67.83**                 | 66.22                     |
| **UWaveGestureLibraryAll**   | *Normalized*             | 83.40                 | **85.00**                 | 75.60                     | 71.50                     |
|                              | + *off.*                 | **81.10**             | 79.00                     | 70.10                     | 69.40                     |
|                              | + *LT*                   | 82.40                 | **83.80**                 | 74.20                     | 71.30                     |
|                              | + *off., LT*             | **81.40**             | 79.00                     | 70.00                     | 68.10                     |
|                              | + *off., RW*             | **74.04**             | 68.90                     | 68.76                     | 68.79                     |
| **StarLightCurves**           | *Normalized*             | 96.00                 | **96.50**                 | 95.40                     | 93.40                     |
|                              | + *off.*                 | **97.30**             | 94.70                     | 94.10                     | 90.00                     |
|                              | + *LT*                   | **97.30**             | 95.20                     | 94.50                     | 91.90                     |
|                              | + *off., LT*             | **97.50**             | 92.30                     | 95.10                     | 90.10                     |
|                              | + *off., RW*             | **93.21**             | 81.61                     | 90.98                     | 90.80                     |
| **MixedShapesRegularTrain**   | *Normalized*             | **95.30**             | 94.50                     | 87.40                     | 92.60                     |
|                              | + *off.*                 | **94.20**             | 87.30                     | 90.60                     | 91.92                     |
|                              | + *LT*                   | **94.60**             | 92.40                     | 89.10                     | 92.10                     |
|                              | + *off., LT*             | 91.80                 | 86.60                     | 86.20                     | **92.40**                 |
|                              | + *off., RW*             | **90.89**             | 72.62                     | 88.16                     | 87.46                     |

Where: *off.* corresponds to offset, *LT* corresponds linear trend, *RW* corresponds to random walk.

We thank the reviewers for giving us the opportunity to discuss the performance of our proposed convolutional layers in the synthetic UCR experiment, which acts as a clear motivation for the invariance properties we formulated, and we are more than willing to hear their comments on the provided justification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732006957261

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #2

comment:
1. **InvConvNet-N (built solely on normal conv.) can have an advantage on the inherently normalized plain UCR data.** When it comes to the results of the different configurations of InvConvNet on the plain datasets (without added synthetic deformations), please refer to 1st line for each dataset in Table 3. The inconsistency in performance improvement of vanilla InvConvNet-N for the plain UCR datasets is implicitly explained in Appendix A.3 of the original submission (page 18, second paragraph, lines 937-942: "We also conduct [..] with the whole repository"). We next explain how this aspect influences performance in more detail. Most UCR datasets, including the four larger ones that we used for the synthetic experiment, are already z-normalized, as indicated in the paper proposing the UCR archive [10] (Section III C. page 3). This, combined with the fact that the time series datasets are univariate and have, in several cases, very few classes (e.g., 2 for HandOutlines and 3 for StarLightCurves), makes the task relatively easy for standard convolutions. Additionally, in terms of trend, the 4 considered datasets are relatively stable. Due to the inherent normalization and the described properties of these datasets, our invariant convolutions may not leverage the invariance with respect to amplitude scaling and offset shift. However, for the same reasons, the UCR datasets can serve the experimental design of showcasing how synthetic deformations added deformations can favor different components in our pool of convolutions, including mixed or those that are invariant to offset/trend deformations. 

2. **InvConvNet with mixed convolutions is significantly more robust to synthetic deformations compared to InvConvNet-N with standard convolutions.** When adding realistic deformations such as offset and smooth random walk to the considered UCR datasets, the drop in classification performances of the proposed InvConvNet with mixed convolution types (including invariant ones) remains relatively low (avg. acc.: 80.96, abs. avg. drop:-6.42 in accuracy) compared to standard convolutions (avg. acc.: 71.25, abs drop: -17.89 in accuracy). It demonstrates that standard CNNs have difficulties to correctly learn invariant features which corroborates with results observed in [11] for image data.

3. **Importance of mixing views.** The extreme configurations of InvConvNet-O and InvConvNet-T, which rely solely on invariant filters, exhibit lower performance compared to InvConvNet-N on normalized data (average accuracy: 82.77 and 81.83 vs. 89.15). However, they experience significantly smaller performance drops in scenarios like offset and smooth random walk noise (absolute average drop: -3.84 and -3.51 vs. -17.89 in accuracy). This suggests that invariant convolutions effectively handle noisy data by learning robust and meaningful features.

More importantly, by integrating all views, the InvConvNet model achieves the best performance in 11 out of 16 noisy scenarios, with relatively small accuracy drops (absolute average drop: -6.42 in accuracy in the offset + smooth random walk case). This highlights the importance of incorporating predefined invariances into convolutional layer to address the limitations of standard convolutions in learning such invariances. Furthermore, the experiment highlights the need for strategies to balance robustness and accuracy when learning filters. While the average configuration (InvConvNet), which evenly combines filters across multiple invariances, serves as a baseline strategy, it can be enhanced by exploring alternative approaches, such as attention mechanisms. The development of more refined strategies remains an avenue for future work.

[10] Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C. C. M., Zhu, Y., Gharghabi, S., ... & Keogh, E. (2019). The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6), 1293-1305.

[11] Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., & Lew, J. (2022). In what ways are deep neural networks invariant and how should we measure this?. Advances in Neural Information Processing Systems, 35, 32816-32829.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732007009512

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #2

comment:
**2. [Comment 2][Synthetic Experiment on UCR - Justifications of Results in Table 3:]** Several reviewers have posed questions concerning the synthetic experiment (page 8, lines 428-461), so in this comment, we would like to clarify the experimental setup that we followed, the specificities of the datasets concerned, as well as analyze the derived results. 

**Configurations of the Considered Networks**. As we mention in the method section (page 6, lines 302-311), "the choice of invariances is often related to the application, and setting the invariances by hand requires a good understanding of the nature of the signals." In the absence of such knowledge, we proposed an arbitrary strategy that consists of gradually including invariances of higher complexities in a single layer called the pool of convolutions. For instance, in the experiments, we considered three types of convolutional filters, namely (i) standard (variant) ones, (ii) amplitude scaling and offset shift invariant ones, and (iii) amplitude scaling, offset shift, and linear trend invariant ones. The configuration of the layer differs depending on the number of filters accorded to each type of convolution and we explored 4 configurations in the synthetic experiment consisting of the average one InvConvNet: (i,ii,iii)=(48,40,40), and the extreme cases: InvConvNet-N (128,0,0), InvConvNet-O (0,128,0), InvConvNet-T (0,0,128). Importantly, the extreme case InvConNet-N corresponds to standard convolutions and constitutes a baseline against which we can compare performances and robustness. 

**Added Deformations**. We study the performance and robustness of the above configurations of our model on a classification problem by considering the 4 largest UCR preprocessed datasets [10]. We considered 5 scenarios with increasing order of complexity by adding synthetically generated deformations (the last one corresponding to random walk-based generated trend has been added for the rebuttal period): (1) no deformations, (2) offset shift, (3) linear trend, (4) offset shit + linear trend, (5) offset shift + smooth random walk. Results are displayed in Table 3 of the initial version of the manuscript. Several key points below can benefit from additional discussion and prove our claims:
</content>
</thread_object>

</thread>
<thread>
creation date:
1732007061248

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comments #1

comment:
We thank all reviewers for taking the time to review our manuscript. We are grateful for their constructive remarks, which helped us to further improve the quality of the manuscript. In what follows, we answer several recurring remarks, and we invite all reviewers to read them as they clarify our positioning and experimental results. Individual reviewers' questions are answered in a dedicated comment. 

**1. [Comment 1][Related work \& Positioning:]**
Advances in invariance modeling for deep learning have been achieved with proper mathematical formalism of group action on data, notably images and graphs [1,2]. Two strategies are usually considered to incorporate invariance in deep networks, either by learning them through data augmentation (including contrastive learning) [3] or by hard-coding invariance within the network architecture. In this work, we focus on the second approach, which has been widely used to design networks invariant to the permutation of graphs' nodes [5] or local translation and rotation in images [6]. Such invariant networks present better generalization properties compared to their learned counterparts at the expense of additional computational costs [1]. 

In the case of time series, invariant neural networks are an emerging field of study. Many works have focused on learning invariance (Section 2, page 3, last paragraph), but very few have proposed hard-coded invariance and are solely dealing with the local time-warping invariance [7] or time rescaling invariance [8].

In this paper, we specifically address the case of hard-coded local invariance to deformations like offset or trend, which are not related to time warping. The invariance is encoded in a convolution layer, which is still a SOTA approach for several time series tasks, as mentioned by reviewer vw6U, and for which invariance improves performances [9]. Importantly, we present a general mathematical framework based on group action, which can be leveraged in future work and by other researchers to further improve invariance in deep learning for time series. Conjointly, we propose a compelling and efficient scheme to compute such invariant convolutions. Thought as a single layer that can be plugged into different network architectures, we illustrate the benefits of the invariant convolutions on several tasks, and we relegated network architectures in the appendix to further promote the mathematical framework. 

We thank the reviewers for raising questions about our positioning against related work in invariant CNNs, and we aim to improve its clarity further in the introduction section of the manuscript.

[1] Kvinge, H., Emerson, T., Jorgenson, G., Vasquez, S., Doster, T., \& Lew, J. (2022). In what ways are deep neural networks invariant and how should we measure this?. Advances in Neural Information Processing Systems, 35, 32816-32829.

[2] Bronstein, M. M., Bruna, J., Cohen, T., \& Veličković, P. (2021). Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478.

[3] Antoniou, A. (2017). Data Augmentation Generative Adversarial Networks. arXiv preprint arXiv:1711.04340.

[4] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501, 2018.

[5] H. Maron, H. Ben-Hamu, N. Shamir, and Y. Lipman. Invariant and Equivariant Graph Networks. In ICLR, pages 1–13, 2019a.

[6] M. Weiler and G. Cesa. General E(2)-equivariant steerable CNNs. Advances in Neural Information Processing Systems, 32:14334–14345, 2019.

[7] Shulman, Y. (2019). Dynamic time warp convolutional networks. arXiv preprint arXiv:1911.01944.

[8] Jacques, B. G., Tiganj, Z., Sarkar, A., Howard, M., \& Sederberg, P. (2022, June). A deep convolutional neural network that is invariant to time rescaling. In International conference on machine learning (pp. 9729-9738). PMLR.

[9] Paparrizos, J., Liu, C., Elmore, A. J., \& Franklin, M. J. (2020, June). Debunking four long-standing misconceptions of time-series distance measures. In Proceedings of the 2020 ACM SIGMOD international conference on management of data (pp. 1887-1905).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732476830152

thread_type:
public_comment

<thread_object>
creator: ~eamonn_keogh1

<content>
title:
You test on SMD MSL SWaT PSM. However it is increasingly understood that it is meaningless to test on these datasets...

comment:
You test on SMD MSL SWaT PSM. However it is increasingly understood that it is meaningless to test on these datasets, because the ground truth is mislabeled, they are trivial, and have other flaws. See 

https://www.dropbox.com/scl/fi/x6ie264xrfkl0nbdw1vtb/Irrational-Exuberance_why_most_TSAD_is_wrong.pdf?rlkey=16frcr2lo6ip5o6uf18qwdeud&dl=0

and

https://www.dropbox.com/scl/fi/cwduv5idkwx9ci328nfpy/Problems-with-Time-Series-Anomaly-Detection.pdf?rlkey=d9mnqw4tuayyjsplu0u1t7ugg&dl=0
</content>
</thread_object>

</thread>
<thread>
creation date:
1732610474388

thread_type:
official_comment

<thread_object>
creator: area_chair_74ae

<content>
title:
Encouragement to Actively Participate in the Discussion Phase

comment:
Dear Reviewers,

Thank you for your valuable contributions to the review process so far. As we enter the discussion phase, I encourage you to actively engage with the authors and your fellow reviewers. This is a critical opportunity to clarify any open questions, address potential misunderstandings, and ensure that all perspectives are thoroughly considered.

Your thoughtful input during this stage is greatly appreciated and is essential for maintaining the rigor and fairness of the review process.

Thank you for your efforts and dedication.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732717383202

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision of the Manuscript

comment:
We thank the reviewers for their valuable feedback. We have uploaded a revised version of the paper based on some comments we received (modifications are marked in **orange color**).

Here is a summary of the changes:

**1. [Based on General Comment 2][Synthetic Experiment on UCR - Additional Results in Table 3:]** We have updated Table 3 and the corresponding text to include performances for an additionally added deformation of random offset and smooth random walk-generated trend.

**2. [Based on General Comment 3][Transfer Learning Experiment - Additional Results in Table 4:]** We have updated Table 4 with the supervised normal configuration InvConvNet-N (built solely upon normal kernels) that highlights the performance gap compared to the InvConvNet (mixed types of invariances) in terms of generalization when testing on unseen target data.

**3. [Synthetic Experiment on UCR - Feature Maps Visualizations in Appendix:]** In Appendix A.5, we provide visualizations of feature maps in Figure 6, produced by InvConvNet (mixed types of normal and invariant filters) and InvConvNet-N (built solely upon normal filters) in order to show how different types of filters are activated in for a considered added synthetic deformation when the proposed normal and/or invariant convolutions are employed. Our observations are summarized below:

- **[Reply to reviewer WiU6][Weakness 1 - Fixed Kernel Bias:]** The reviewer asked whether standard CNNs with learned kernels could naturally adapt to deformations like offset shift and linear trend. In the visualizations of Appendix A.5, we provide a qualitative analysis of the activated regions in feature maps for different types of filters, including (i) normal, (ii) offset-shift invariant, and (iii) linear-trend invariant. Interestingly, it is depicted that different parts of the input signal are activated for invariant filters compared to normal ones, and the morphology of the feature maps differs among different (in terms of invariances) types of kernels. 

- **[Reply to reviewer akRu][Weakness 4 - Practical Value of Offset and Trend Invariances:]** The practical value of the proposed InvConvNet architecture is validated on the visualizations of feature maps learned for the pool of normal and invariant convolutional filters compared to the solely normal ones on the added offset and smooth random walk trend deformation. As shown in Figure 6 of the revised manuscript, normal filters alone are affected by the added trend, whereas in the pool of convolutions, trend-invariant filters produce robust feature maps when the random walk trend deformation is considered. The learned feature maps of InvConvNet on this particular case of synthetic added deformation demonstrate the relevance of the proposed invariant convolutions in applied scenarios with considered deformations that highly affect the classification performance on an example UCR dataset (as shown in Table 3).

- **[Reply to reviewer vw6U][Weakness 1 - Components Contribution Inside the Pool of Convolutions:]** The reviewer asked us to quantify the contribution of each of the components inside the proposed pool of convolutions (including normal, offset-shift invariant, and linear-trend invariant filters). In addition to our ablations studies in classification performance in Table 2 and Table 3 and our reply to reviewer vw6U in the rebuttal, we provide qualitative examples of the components' contribution of variant and invariant convolutions for an added deformation in Figure 6.

We kindly await further feedback from the rest of the reviewers before making necessary adjustments to the text regarding the positioning of the work (as outlined in our response to **[General Comment 1]**) and the additional clarifications on the experimental choices for the synthetic experiment and the transfer learning experiment (as explained in our **General Comments 2, 3**). These *changes can be implemented straightforwardly* if necessary for the final version of the manuscript. 

We sincerely thank all the reviewers for their efforts in assessing our work, and we are happy to address any additional concerns. We would greatly appreciate also their consideration in updating their scores in light of these updates.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings
The paper introduces InvConvNet, a neural architecture incorporating invariant convolutional layers designed to handle specific time series deformations such as offset shifts, scaling, and linear trends. The contributions include:
A mathematical framework for group action-based invariant convolution operations.
A hybrid embedding layer combining standard and invariant convolutions for feature extraction.
Empirical validation of InvConvNet on tasks like classification and anomaly detection using univariate and multivariate time series datasets.
The authors claim that InvConvNet provides robustness to deformations in time series data and achieves superior performance against baseline methods on synthetic and real-world datasets.

(b) Strengths of the Paper
Mathematical Rigor: The work presents a clear theoretical framework for invariant convolutions in time series, providing a solid foundation for future research.
Focus on Time Series Deformations: The paper addresses a practical problem of modeling invariances in time series, which is underexplored in comparison to image and graph data.
Broad Evaluation: Experiments cover tasks like classification and anomaly detection, with additional ablation studies to analyze robustness under synthetic deformations.
Potential Applicability: The proposed invariant convolutions can be integrated into various architectures, making them a versatile tool for time series analysis.

(c) Weaknesses of the Paper
Performance Inconsistencies:
On clean datasets, InvConvNet often underperforms compared to its standard counterpart (InvConvNet-N), raising questions about the utility of the proposed invariant layers when no synthetic deformations are present.
The proposed architecture shows limited improvement over state-of-the-art baselines (e.g., ResNet, Rocket) in several classification benchmarks.

Limited Generalizability:
The datasets used (e.g., UCR and UEA archives) are small-scale and not representative of diverse real-world time series scenarios.
Practical applications requiring invariance (e.g., ECG data) are underexplored, with only qualitative illustrations provided.

Lack of Baseline Comparisons:
The paper omits comparisons with relevant approaches, such as time-series-specific contrastive learning methods or other architectures designed for invariance.
Missing evaluations against models that incorporate learned invariances via data augmentation or adversarial training.

Unclear Practical Impact:
The robustness provided by the invariant layers is relevant only in highly specific scenarios (e.g., datasets with offset or trend deformations), limiting its broader applicability.
Hard-coded invariances may not generalize well to diverse datasets or deformations.

(d) Reasons for Rejection
Inconsistent Results: The architecture’s performance on clean datasets does not align with its stated goal of improving generalization, weakening its practical significance.
Limited Scope: The focus on specific deformations and small-scale datasets restricts the applicability and impact of the proposed method.
Theoretical and Practical Disconnect: While the mathematical framework is rigorous, its real-world utility remains unclear, especially given the modest empirical gains.
Unaddressed Reviewer Concerns: Despite the rebuttal efforts, key issues such as baseline omissions, dataset limitations, and unclear applicability remain unresolved.

While the paper demonstrates potential as a theoretical contribution to time series invariance, it falls short in terms of empirical rigor, practical impact, and broader applicability.