Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jingyun Xiao
authors: Ran Liu
authors: Eva L Dyer
reviews: 
Comments: In this paper, authors present a novel approach for learning spatiotemporal structure and using it to improve the application of transformers to timeseries datasets. A key aspect of contributions lies in the creation of a specialized group embedding scheme designed specifically for transformer architectures. This scheme enables the adaptive learning of concise grouping tokens, which encompass both channel and temporal dimensions. By incorporating group-aware structural information into the representation space, GAFormer enhances the overall understanding and encoding of data patterns.
Strengths: 1.The method proposed by the authors is intuitive and validated on multiple datasets, and the experimental results prove the effectiveness of GAFormer.

2.GAFormer achieves the adaptive discernment of channel-wise and temporal groupings without relying on any predefined structure or additional supervision beyond classification or regression tasks. Through the assignment of group embeddings to tokens, GAFormer enhances the interpretability of the model's representations.
Weaknesses: 1.The GAFormer goes through the SGE and then the TGE, what happens if the order of the two modules is reversed? Why is it modeled from the spatial point of view first?

2.Can the author explain Figure 4 again, I don't really understand the interpretability of the authors' proposal.

3.Table 4 shows that SGE is more effective than TGE, can the authors explain why?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper introduces a novel technique for improving the application of transformers to timeseries datasets by learning spatiotemporal structures. The proposed framework employs group tokens and an instance-specific group embedding layer to integrate structure into the learning process. The newly devised Group-Aware transFormer (GAFormer) demonstrates superior performance across various timeseries tasks, offering enhanced interpretability and the ability to discern latent structures in intricate multivariate datasets.
Strengths: - The idea of using *groupe mbedding*(GE) technique to learn both spatial and temporal structure in multi-variate timeseries datasets is novel.
- The proposed model achieves the state-of-the-art performance on a number of time- series classification and regression tasks
Weaknesses: Though the approach claims to offer "a more interpretable decomposition," this is a subjective claim. The degree of interpretability might vary based on the user or the specific use case.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper proposes a method to learn PEs for transformers in the multivariate temporal data setting. Specifically, the authors propose a Group Embedding (GE) to capture the group aware dynamics along the channels and also the temporal dimension. This Group Embedding is then induced into the transformer architecture called GAFormer to learn the spatial and temporal structure of the multivariate time series datasets. Experimental results on the synthetic dataset show the robustness of the proposed GE over baseline PEs. Moreover, when GE is induced in other transformer architectures for the univariate datasets the performance is enhanced. GAFormer also achieves SoTA performance on the multivariate datasets tested. The authors also claim the interpretability of the proposed method.
Strengths: - Promising PE for multivariate time-series data. The PE show improved performance over the tasks.
- The proposed GAFormer also achieves SoTA performance on the multivariate datasets tested.
Weaknesses: - Since the model is designed to learn the group structure in a data-dependent manner, it may not be able to capture these properties effectively in the low data regime as the baseline PEs would.
- The group structure exhibited in Figure 4 is claimed to provide interpretability of the model. However, these are some groups that have been learnt and the interpretability aspect is not clear as to what it means if some channels or time signals are grouped. This is similar to the groups formed in any attention mechanism and the method doesn’t seem to provide any added explanations or interpretability to the decisions. Thus it may not be right to claim interpretability based on group embedding structure.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good