Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on architecture design for spatio-temporal modeling for forecasting tasks. The paper first proposes three types of layers (i.e., S2T, T2S, STS) which have emphasis on modeling spatial, temporal and spatio-temporal information, respectively. The three layers are then applied in a unified encoder-decoder framework for spatio-temporal forecasting tasks. An automated architecture search algorithm is further adopted to better decide the connection configuration of different modeling layers. The final model, AutoST, achieves better results than baseline methods as well as the proposed UniST, which only involves one of the three modeling layers (S2T / T2S / STS).
strengths_and_weaknesses: Strength
1. The idea of analyzing and unifying different types of spatio-temporal modeling (i.e., spatial first, temporal first, spatio-temporal synchronous) is well-motivated and potentially valuable to the research community.
2. The paper provides clear ablation studies (in Table1) that show (1) the performance difference of the three modeling types on different datasets / temporal setups; (2) the effectiveness of introducing automated architecture search to fuse the three types of modeling layers.

Weaknesses
1. Although I like the idea of unifying different types of spatio-temporal modeling, the technical novelty and contribution of this work is not significant enough. Most of the techniques in the paper are adopted from prior works with minor modifications, for example, linear self-attention, mix graph convolution (with combination of ChebNet and AdapDC), and DARTS / PAS for automated architecture search. Even from the framework design perspective, the ideas of spatial-first / temporal-first and joint spatio-temporal modeling are widely explored in previous work mentioned as baselines in Table 1, as well as the decoupled spatio-temporal modeling design [1,2] for video action recognition. The encoder-decoder framework adopted in UniST is also widely used for both NLP and CV tasks in recent years.

In addition, although the paper provides comparison between UniST and other baseline methods in Table 1, the comparison is not totally fair due to different model complexity, computation cost and even optimization details. This also makes the contribution of the proposed UniST insufficient.

2. The major contribution of this work to me is using architecture search algorithms to unify three types of modeling layers, i.e. the final AutoST model. While the paper compares the results of using different NAS algorithms, an important baseline is missing. Instead of only using one the the three types of modeling layers (UniST), it is intuitive to have a model that includes all of the three layers at each STE and fuses all three outputs, optionally augmented with a gating module to introduce a weighted fusion mechanism. With this baseline result provided, the necessarity of using NAS to select modeling layers would be more convincing.



[1] Tran, Du, et al. "A closer look at spatiotemporal convolutions for action recognition." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.
[2] Qiu, Zhaofan, Ting Yao, and Tao Mei. "Learning spatio-temporal representation with pseudo-3d residual networks." proceedings of the IEEE International Conference on Computer Vision. 2017.
questions: Minor:
1. A comma is missing in the last equation of Eqn. 1.
2. The equation in L204 is inconsistent with the text "the outputs of each extractor are added to form the final output".

limitations: Not discussed in the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper proposes a novel universal framework for spatio-temporal forecasting. The framework combines spatial first, temporal first and spatio-temporal synchronous approaches in one network. These are implemented as S2T Layer, T2S Layer and STS Layer respectively. This is termed UniST.

Paper further proposes a novel automatic search strategy by network architecture search. 

Experimental results on 5 datasets across 9 tasks outperforms existing methods significantly.
strengths_and_weaknesses: Strengths
1. Paper is well-written and easy to understand.
2. Method is presented clearly with detailed explanations and good use of diagrams.
3. Experiments are extensive and strongly supports the main claims of the paper

Weakness
1. Minor. The labeling of Table 1 is too brief. It should contains more information to explain the Table content.
questions: No question.
limitations: Not applicable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Two Network architecture search algorithms are proposed, called Automated Spatio-Temporal
modeling approach, AutoST1 and AutoST2. They are focused on automatically crafting spatio temporal forecasting systems. Both the search algorithms operate by selecting from a layer candidate set,  formed by three types of layer: T2S Layer (Temporal-first Modeling Layer), S2T Layer (Spatial-first Modeling Layer), STS Layer(Spatial-temporal Synchronous Layer). All of these three layers have exactly same inputs and outputs,  and operate by assembling in different fashion a Time Series Linear Self-Attention Unit and a High-order Mix Graph Convolution Unit. AutoST2 is the most performing one, which creates a serial architecture of hidden layers, each one of them can be a T2S, S2T, STS layer. Results on traffic datasets support the search algorithm on five traffic forecasting datasets. 

strengths_and_weaknesses: Strengths: The idea of learning an architecture which understands whether spatial or temporal relations are more important to forecast a given signal is intriguing. Figure 2 has a very effective visualization, which makes the intuition of the authors crystal clear.



Weaknesses:
-- The experiments focus on very specific datasets, all of them representing the same problem (traffic forecasting). It would have been definitely better to have more heterogenous tasks, as for example pose forecasting, where the spatial part is the body skeleton (expressed as a graph) and the temporal part let the body joints move accordingly to anatomical costraints. An interesting comparison would have been 
Theodoros Sofianos, Alessio Sampieri, Luca Franco and Fabio Galasso  Space-Time-Separable Graph Convolutional Network for Pose Forecasting In Proc. International Conference on Computer Vision (ICCV) Virtual, October 2021 
They use a Space-Time Separable GCN (STS-GCN. Which is *not* the same as the STSGCN cited in the proposed paper as [19]). 

-- The Complexity of the AutoST1, and especially AutoST2 is not detailed. If complexity cannot be given, timings become very important.  In particular, since AutoST2 has to look for multiple optimal layers at each cell, what is the time spent overall, and for completing each layer of Fig.5.b?

-- The definition of Adaptive Diffusion Convolution is clear (ChebNet + ADC), but its motivation is not. A sentence on the rationale for this choice would be beneficial.

-- In section 3.2, it is not clear in the result connection formula what is o^{(i,j)}, and why i<j, w.r.t. which ordering?

-- The rationale why T2S, S2T, STS have been built as described in 4.2.1, 4.2.2., 4.2.3 ,with that precise architecture, is not given.

MINOR:
 
--Fig:3 S2T Layer: firstly model temporal... -->T2S Layer: firstly model temporal...
--pag.3: is automated methods --> are automated methods
--pag.3 our proposed trainning framework --> our proposed training framework
--In the objective function of section 3.2, w and alpha are undefined. 

questions: - See my question on the time complexity.
- Having five datasets of traffic forecasting, most of them having very similar structure and content, is definitely a cons. It would have been definitely better to take into account a dramatically different scenario/task (eg pose forecasting)

limitations: No limitations or potential negative impacts have been taken into account by the authors. I think it should be, since managing traffic data could be very impactful on the privacy management. ..
review_title: null

Meta-review: metareview: This paper presents a framework for spatio-temporal sequence analysis.  The framework contains spatial, temporal, and spatio-temporal building blocks and is operationalized in an architecture search approach.  The reviewers considered the authors' rebuttal and engaged in further discussion regarding the merits of the paper.

There are concerns over the overall novelty of the approach and the combination of techniques that are assembled in the method.  However, the potential of the proposed building blocks and overall empirical results were deemed to be a solid contribution.  For these reasons, this paper is recommended to be accepted to NeurIPS.
confidence: Certain
Decision: Accept