Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This study addresses Multivariate time series forecasting using a transformer-based model. Namely, the authors presented Crossformer, a transformer based model which not only takes into account the temporal sequence, but also the cross dimension dependencies among the variables.
The inputs for Crossformer, is embedded using Dimension-Segment-Wise (DSW) embedding in a 2d array then it is processed by using a two-stage attention layer that  supposedly captures both cross-time and cross-dimension dependencies. 
In the cross-time stage,  a multi-head self-attention (MSA) is applied to each dimension and then an MLP is used to capture the dependency among the different segments in each dimension. 
In the cross-dimension stage, as directly applying MSA for high dimensional datasets will not be affordable, the authors proposed using router mechanism to aggregate messages from all dimensions into a low dimension space.
Then to make the all-to-all connections,  the aggregated
messages are considered as key and value and the vectors of dimensions as query.
Finally, the authors used a hierarchical  encoder-decoder structure to use the extracted information in different scales. 

strength_and_weaknesses: This manuscript is well written and organized very well. Also  the studied topic is very interesting as it is usually the case that for MTS the cross dimension dependencies is overlooked. 

The authors argue that as “For MTS, a single value at a step alone provides little information”, embedding in a way that “nearby points over time form a segment” could be beneficial. This argument seems interesting and valid as it is used and illustrated nicely (In Figure 1) by the authors.

In my opinion this is an interesting study.   I think you can complete your great work by comparing your method with “Long-Range Transformers for Dynamic Spatiotemporal Forecasting” which you can find in “https://arxiv.org/pdf/2109.12218.pdf” and also include the very simple baseline that is mentioned in this study “https://arxiv.org/pdf/2205.13504.pdf”. 

clarity,_quality,_novelty_and_reproducibility: This study is clear and pretty novel. 
summary_of_the_review: In my opinion, this study is nicely done and by adding two mentioned comparison above will be completed enough to be published in this conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Crossformer, a Transformer-based model utilizing cross-dimension dependency for Multivariate time series forecasting. Besides the time information, Crossformer considers dependency between dimension information. Specifically, Dimension-Segment-Wise (DSW) embedding and Two-Stage Attention (TSA) layer are proposed to efficiently capture both cross-time and cross-dimension dependency. Extensive experimental results show the effectiveness of Crossformer.
strength_and_weaknesses: Strength: Dimension information is an interesting topic in the time series. This paper proposes Crossformer, which performs cross-time and cross-dimension dependency. The experiment results are comprehensive.

Weaknesses: 
(i) Efficiency is my main concern about this paper. Since 2-D attention is performed in Crossformer, the complexity is much more than in the previous 1-D attention method. Can authors provide some metric of efficiency (i.e., FLOPs, number of parameters, time) in Table 1?
(ii) Interestingly, there is another paper (https://openreview.net/forum?id=GpW327gxLTF) that investigates the dependency between dimensions. But their conclusion may be different from yours. For example, the dependency between ILI is significant compared with other data sets. However, in Table 1, Crossformer is not the best compared with other baselines on ILI. Can you explain that?
(iii) The extension from 1-D to 2-D is somehow trivial in time series fields. Thus, the novelty of the paper is weak in my view.

clarity,_quality,_novelty_and_reproducibility: Clarity: fair
Quality: fair
Novelty: limited
Reproducibility: limited
summary_of_the_review: This paper proposes a cross-time and cross-dimension model - Crossformer. However, efficiency is my main concern since Crossformer should perform 2-D attention and the paper doesn't provide any efficiency metric. Thus, I think the paper is not much practical.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents Crossformer, a Transformer-based model for multivariate time series (MTS) forecasting, especially focusing on cross-dimension dependency, which has not been widely investigated so far. They propose Dimension-Segment-Wise (DSW) embedding and Two-Stage Attention (TSA) layer to address this issue and further build a Hierarchical Encoder-Decoder (HED) to combine the information at different scales for final forecasting. 
strength_and_weaknesses: [+] The idea for utilizing cross-dimension attention in multivariate time-series data seems novel and interesting. The reduced time complexity is also an advantage. 

[+] Its effectiveness is validated through extensive experiments across six datasets. Sensitivity analysis on hyperparameter setting and ablation study is also provided. 

[-] In computational efficiency analysis (Section 4.5), it would be better if the authors can also provide a comparison of actual execution time.

[-] In the ablation study (Section 4.3), the MSE of the TSA layer with and without the routers should be compared to validate the proposed router mechanism. 

[-] The authors mention that HED decreases the performance when the prediction length is short, and the possible reason is the information on different scales is helpful to long-term prediction. However, when it is combined with TSA, short-length prediction shows good performance. The explanation of how TSA overcomes the shortcoming of HED should be addressed. 

clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and the main contributions are quite clear. The experimental results demonstrate the effectiveness of the proposed method. 
summary_of_the_review: The paper is well-written overall. I think the contribution and novelty of the proposed work are clearly shown.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed `CrossFormer`, a Transformer based neural network for multivariate time series (MTS) forecasting. The author(s) claimed that cross-dimension dependency was not well utilized and developed a method to explicitly explore and utilize cross-dimension dependency for MTS forecasting. In order to achieve that, the authors employed `Dimension-Segment-Wise` embedding to aggregate segment based embedding on the time axis, and designed a two stage attention layer using multi-head self attention and MLP to capture cross-time and cross-dimension dependency of the embedded input. A hierarchical encoder-decoder is further introduced to utilize the information at different scales. In this way, long term dependency between different time steps and dimensions can be captured. The author(s) compared the accuracy of the method on six real-world datasets and demonstrated superior results in most of the tasks. They also analyzed the theoretical time complexity of the method to be $O(\frac{D}{L_{seg}^2}{T^2})$ for encoder, $O(\frac{D}{L_{seg}^2}{(\tau(\tau+T))})$ for decoder, which performed better in terms of memory occupation and computation time in empirical experiments.

strength_and_weaknesses: This paper is well organized with comprehensive introduction into the network details and extensive experiment results. The setting of multivariate time series forecasting was well-introduced. The methodology part clearly presented how the initial input was segmented and fed into the 2-stage attention layer to capture cross-time and cross-dimension dependency. The figures are very helpful in understanding how the proposed modules work. The experiment part covers quite some baseline models and datasets, demonstrating the method's superior performance. The rich ablation study covers multiple aspects of experiment settings, including padding, covariants, hyper-parameter selection and architecture designs. These experiment results proved that the modules introduced are critical to the final performance. Moreover, the author(s) analyzed the time complexity of the encoder & decoder, and further benchmarked the memory occupation and computation time of each baseline model against CrossFormer. The very detailed comparison convinced me of the model's effectiveness.

  

On the other hand, the novelty of the paper is kinda limited because the problem of MTS forecasting and use of cross-dimension dependency was explored before, and Transformer was kind of the de facto in this setting. I also have a few concerns or questions as follows:

1. In Figure 1, it's interesting to see the pattern demonstrated in the figure(a). Can you please further elaborate how that figure indicates the MTS data tends to be segmented?

2. All the segments seem to be formed based on their location and segment length. Can you please explain how the segments align with the meaningful "segments" for forecasting in MTS data?

3. In section 3.2, the author(s) proposed the design of cross-dimension stage after cross-time stage. Is there any performance comparison between the current design with the other way around?

4. In sections 3.2, is there any performance comparison on the impact of using `LayerNorm` and skip connection?

5. It's quite surprising that covariates don't improve the performance in ablation study, can you share the covariates used in your experiments?

6. In the Hierarchical encoder decoder, the final results are summarized over each layer's output, is there any difference in the impact of each layer on the final results?

  

Minor suggestions:

7. In section 3, the author(s) can mention the covariants are not used earlier in the paper to avoid confusion.

8. In section 3.2, the author(s) used the notation of `L` to represent the number of segments, where $\frac{T}{L_{seg}}$ was also used in complexity analysis. They could be unified if there's no difference in their meanings.

  
clarity,_quality,_novelty_and_reproducibility: This paper was written in high quality and clarity, the figures are very illustrative and extensive experiments were conducted to compare the forecasting accuracy, memory occupation and computation time against multiple baselines. Moreover, ablation study was well-designed to cover the choice of multiple hyper-parameters. The paper is also very clear in presenting how the projections and layers are applied to each input vector in formulas.

  

Multivariate time series forecasting is a classic problem, and previous work has explored Transformer on long cross-time dependency, and CNN, GNN on cross-dimension dependency. This paper's originality is to use two transfomer's multi-head self attention layers to capture dependency in both dimension and time axis, and proposed corresponding segmented embedding, two stage attention layer and hierarchical encoder-decoder to get it work. Therefore, the originality is kind of limited.
summary_of_the_review: 
To summarize, this paper was written in high quality. It proposed `CrossFormer`, a Transformer based neural network, to capture cross-time and cross-dimension dependency, and achieved state-of-the-art performance in most of the multivariate time series forecasting tasks benchmarked, especially in long time series. The experiments were extensive and covered aspects of accuracy, memory occupation, computation time. The ablation study was comprehensive and proved the effectiveness of architectures proposed and hyper-parameters selected. There's a bit of a lack of novelty and this method may be hard to extend to other areas of representation learning. However, the insights from cross-dimension dependency and new design of 2-stage attention layers are very important. Therefore, I still recommend this paper to be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The paper proposes a Transformer-based model that considers dependencies between variables, which the authors refer to as cross-dimension dependency, using 2-stage attention. The method is novel, the experiments are convincing and the study is well written. Reviewer pZRS identified concurrent work, and the authors were quick to run experiments to compare their method against those methods, showing Crossformer is better performing. Reviewer Zk9T pointed out the issue of efficiency, however, the authors responded with their complexity analysis, showing that Crossformer is efficient - I consider their arguments valid. While the reviewer did not respond, I'll consider this concern adequately addressed. The ablation studies in the paper were appreciated, as was the overall structure and writing of the paper. There was a question about whether cross-time or cross-dimension dependencies should be considered first. The authors responded with experiments showing that the method works well with either option.
Overall, this is an excellent paper and should be of great interest to the community.
justification_for_why_not_lower_score: All aspects of this paper are solid. The authors have answered all concerns, so I see no reason to give it a lower score.
Decision: Accept: notable-top-5%