Get Rid of Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework

Authors: zhongchao  yi
zhengyang  zhou
qihe  huang
yanjiang  chen
liheng  yu
xu  wang
yang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 05:47:39

Abstract: Spatiotemporal learning has become a pivotal technique to enable urban intelligence. Traditional spatiotemporal models mostly focus on a specific task by assuming a same distribution between training and testing sets. However, given that urban systems are usually dynamic, multi-sourced with imbalanced data distributions, current specific task-specific models fail to generalize to new urban conditions and adapt to new domains without explicitly modeling interdependencies across various dimensions and types of urban data. To this end, we argue that there is an essential to propose a Continuous Multi-task Spatio-Temporal learning framework (CMuST) to empower  collective urban intelligence, which  reforms the urban spatiotemporal learning from single-domain  to cooperatively multi-dimensional and multi-task learning. Specifically, CMuST proposes a new multi-dimensional spatiotemporal interaction network (MSTI) to allow cross-interactions between context and main observations as well as  self-interactions within spatial and temporal aspects  to be  exposed, which is also the core for capturing task-level commonality and personalization. To ensure continuous task learning, a novel Rolling Adaptation training scheme (RoAda) is devised, which not only preserves task uniqueness by constructing data summarization-driven task prompts, but also harnesses correlated patterns among tasks  by iterative model behavior modeling. We further establish a benchmark of three cities for multi-task spatiotemporal learning, and empirically demonstrate the superiority of CMuST via extensive evaluations on these datasets. The impressive improvements on both few-shot streaming data and new domain tasks against existing SOAT methods are achieved. Code is available at https://github.com/DILab-USTCSZ/CMuST.

Main paper: 
 introduction: 
 Spatiotemporal learning has become a pivotal technique to enable smart and convenient urban lives, benefiting diverse urban applications from intra-city travelling, environment controlling to locationbased POI recommendation, and injecting the vitality into urban economics. Existing spatiotemporal learning solutions [19, 9, 38, 40, 51, 42, 37, 10, 8] focus on improving performances of a task-specific model independently where these methods devise various spatial learning blocks [44, 8, 43] and temporal dependency extraction modules [40, 34, 9] to model the spatiotemporal heterogeneity.Actually, urban spatiotemporal systems are usually highly dynamic with emerging new data modality, leading to serious generalization issue on both data pattern and task adaptation. As illustrated in Figure 1 , the traffic volume patterns can evolve with urban expansion and establishment of new POIs. Concurrently, with increasing attention on road safety, traffic accident prediction has become a new task in intelligent transportation that inevitably suffers from the cold-start issue. Unfortunately, traditional task-specific models usually assume that data on a single task follows independent and identical distribution and are intensively available where such assumption directly leads to failures on data sparsity scenarios and generalization to new tasks. In fact, given diverse datasets, separately training single task-specific spatiotemporal models is cost sensitive and will trap the models into isolation. To this end, we argue that a continuous multi-task spatiotemporal learning framework is highly desirable to facilitate the task-level cooperation. It is even more interesting and exciting to jointly model multi-domain datasets with multi-task learning, which empowers understanding spatiotemporal system in a holistic perspective and reinforce each individual task by exploiting the collective intelligence from diverse data domains.The key towards exploiting task-wise correlations for mutual improvement is to capture the common interdependencies across data dimensions and domains. Current multi-task learning schemes either investigate the regularization effects between auxiliary and main tasks [31, 17] , or devise loss objectives to constrain the consistency between each task [7, 32] . Actually, given a spatiotemporal domain, there must be common interdependencies across different data types and domains, which are valuable for cooperated learning. Even prosperity of multi-task learning and spatiotemporal forecasting, there are never a systematic solution on how various sourced data from different tasks reinforce a specific task with multi-task learning. More specifically, interrelations among observations in a system can be decomposed into multi-dimensional interactions, i.e., from contextual environment to respective spatial relations and temporal evolutions, spatial-temporal level interactions and relations across different data domains. Considering the multi-level semantic correlations, learning the system in a holistic perspective is required and further pose two challenges to a continuous multi-task spatiotemporal learning framework, i.e., 1) How to disentangle complex associations between data dimensions and domains, and capture such dependencies in an adaptive manner to improve spatiotemporal representation, hence facilitating the extraction of common patterns for mutual enhancement. 2) How to exploit task-level commonality and personality to jointly model the multi-task datasets, and exploit such extracted task-level commonality and diversity to reinforce respective task for getting rid of task isolation.In our work, a Continuous Multi-task SpatioTemporal learning framework, CMuST, is proposed to jointly model multiple datasets in an integrated urban system thus reinforcing respective learning tasks. Specifically, a Multi-dimensional Spatio-Temporal Interaction Network (MSTI) is first devised to dissect interactions across data dimensions, including context-spatial, context-temporal and self-interaction within spatial and temporal dimensions. MSTI enables improved spatiotemporal representation with interactions, and also provides disentangled patterns to support commonality extraction. After that, a Rolling Adaptation training scheme, RoAda, which iteratively captures the task-wise consistency and task-specific diversity, is proposed. In RoAda, to maintain task characterization, a task-specific prompt is constructed to preserve unique patterns distinguishing from other tasks by compressing data patterns via an AutoEncoder. To capture the commonality across tasks, we propose a weight behavior modeling strategy to iteratively highlight the minimized variations of learnable weights, i.e., stable interactions during continuous training, which encapsulates crucial task-level commonalities. This approach not only stabilizes learning through continuous task rolling, but alleviates the cold-start over new tasks with shared patterns. Finally, a task-specific refinement is devised to leverage commonality and fine-grained adaptation on specific tasks.The contributions of this work can be three-fold. 1) The first continuous multi-task spatiotemporal learning framework, CMuST to jointly model learning tasks in a same spatiotemporal domain, which not only reinforces individual correlated learning task in collective perspective, but also help understand the cooperative mechanism of dynamic spatiotemporal systems. 2) Technically, two learning modules, MSTI and RoAda are proposed to dissect the impacts and interactions over multidimensions, and iteratively update the task-wise commonality and generate individual personalization to continuous task adaptation in multi-task learning. 3) We construct benchmark datasets in each of three cities, where two of them consists of at least 3 types of observations within same spatiotemporal domain. The extensive experiments demonstrate the superiority on enhancement of each individual task with limited data and the interpretation of task-wise continuous learning. 
related work: 
 Spatiotemporal forecasting is an emerging technique to capture the dynamic spatial and temporal evolution for diverse urban predictions, where the methods can be divided into machine learningbased and deep learning-based. Conventional solutions rely on complex mathematical tools to simulate the dynamics including ARIMA [25] , SVR [3] and matrix-factorization learning [28] for capturing spatial correlations. With deep learning solutions flourishing, Convolution Neural Networks (CNNs) [40, 11, 46] are exploited to imitate the temporal dependencies and GNNs [43, 13, 36] are utilized to imitate spatial propagation. Meanwhile, by taking the advantage of the flexibility and interpretability of attention, Spatial-Temporal Attention [9] , and Vision Transformer (ViT) [6] are introduced to improve spatiotemporal representation. Also there are also many methods for temporal periodicity capturing [12, 20] . More specifically, DG2RNN [49] designs a dual-graph convolutional module to capture local spatial dependencies from both road distance and adaptive correlation perspectives. PDFormer [15] designs a spatial self-attention and introduces two graph masking matrices to highlight the spatial dependencies of short-and long-range views. TESTAM [18] uses time-enhanced ST attention by mixture-of-experts and modeling both static and dynamic graphs. Even so, most solutions focus on single-task intelligence, fail to deal with complex interactions between data dimensions and never extract task-level commonality patterns, resulting in inferior performances on exploiting collective intelligence over multiple tasks. In contrast, we merge the gap by disentangling learnable interaction patterns and exploring rolling task adaptations.Multi-task learning. Plenty of efforts have been made on multi-task learning (MTL) and MTL can be elaborated by two-fold, i.e., feature-based and parameter-based. Feature-based MTL [2, 24] learns a common feature representation for different tasks, but it may be easily affected by outlier tasks. To this end, parameter-based MTL [30, 16] is devised to exploit model parameters to relate different tasks, which is expected to learn robust parameters. Majority of these MTL schemes either concentrate on the diversity design and regularization effects of auxiliary tasks to main task [30, 27] , or construct loss functions to ensure task-wise consistency [47] . A pioneering work investigates a gradient-driven task grouping to realize multi-task learning [41] where the focuses are text and images using pre-trained CLIP model. For spatiotemporal learning, RiskOracle [52] and RiskSeq [53] are proposed to simultaneously predict multi-grained risks and auxiliary traffic elements. More recently, UniST [45] and UniTime [22] construct a unified model for spatiotemporal and time-series prediction. However, all researches on ST learning still never dissect task-wise correlations, especially capture explicit consistency and diversity among tasks and investigate how each task reinforce the core task, which is of great significance for performance and interpretability in MTL.Continuous learning and task continuous learning. Continuous learning (CL) usually keeps long-term and important information while updates model memories with newly arrived instances [29, 5, 35] . For spatiotemporal forecasting, Chen, et, al. [5] proposes a historical-data replay strategy, TrafficStream, to update the neural network with all nodes feeding, while PECPM [35] manages a pattern bank with conflict nodes, which reduces the memory storage burdens. Most existing CL solutions are designed for homogeneous sourced data. Then there are very few research investigating task-level continuous learning where task can be converted from one to another. A pioneering work, CLS-ER [1] realizes class-level and domain-level continuous learning with a dual memory to respectively store instances and build long-short term memories. Even so, the data input to CLS-ER is with same image-like property and scales, which is still far away from task diversity. We argue that learning continuously on task levels can increase the data tolerance over low-quality data and also generalization capacity on new tasks. In this work, by taking bonus of continuous learning, we propose to iteratively assimilate the commonality and extract personalization to reinforce each learning task and realize continuous multi-task spatiotemporal intelligence. 
preliminary: 
 Definition 1 (Spatiotemporal features.) Spatiotemporal features refer to the data points collected by sensors deployed in urban environments, such as traffic dynamics on roads. We define a spatiotemporal feature at a specific time and location as a vector X ∈ R C , where C represents the number of attributes (e.g., vehicle flow, speed) recorded by the sensor. We generalize this to define spatiotemporal data collected over a period as X ∈ R T ×N ×C , where T denotes the number of discrete time intervals, and N denotes the number of spatial nodes corresponding to sensor locations. Definition 2 (Spatiotemporal prediction.) Spatiotemporal prediction involves forecasting future values of spatiotemporal data based on historical observations. Specifically, given a historical dataset X = [X t-T , . . . , X t-1 , X t ] ∈ R T ×N ×C , the objective is to predict future observations Ŷ = [X t+1 , . . . , X t+T ′ ] ∈ R T ′ ×N ×C . Here, T ′ represents the prediction horizon, the number of future time steps we seek to forecast, which for simplicity is set equal to T in this paper.Definition 3 (Multi-task spatiotemporal learning.) Given a set of spatiotemporal tasks in an integrated urban system T = {T 1 , T 2 , . . . , T k } where each task T i is associated with a dataset D i = {X i , Y i }. Here, X i ∈ R Ti×Ni×Ci represents the input features collected over T i time steps, across N i spatial locations, and C i feature dimensions, and Y i ∈ R T ′ i ×Ni×Ci represents the corresponding targets. Multi-task spatiotemporal learning aims to learn a function f : X 0 , X 1 , . . . , X k-1 → Ŷ0 , Ŷ1 , . . . , Ŷk-1 that optimally predicts all target Y i from their respective inputs X i , leveraging shared and unique patterns across tasks to improve generalization performance.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discussed the potential positive societal impacts of our paper in Section 6. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
framework overview: 
 The CMuST is crafted to advance urban intelligence through a synergistic integration of three components in Figure 2 . We processes and standardizes diverse urban data into a harmonized format, and propose the MSTI to intricately dissect the complex interactions within spatiotemporal data, and devise a RoAda to dynamically fine-tune the model via continuous and careful updating, ensuring robust adaptability and consistent performance across fluctuating urban environments.To harness various data domains within urban spatiotemporal systems as well as data interactions between diverse dimensions, the first task is to appropriately process main ST observations, spatial indicator, and temporal indicator to create a comprehensive and integrated data representation tailored for multi-task ST learning, enabling further interactive modeling between them.To be specific, the main observation data, namely target of interest of urban datasets, are denoted as X obs ∈ R T ×N ×C , and then mapped into a spatiotemporal representation E obs ∈ R T ×N ×d obs via an MLP ObsMLP(X obs ; θ obs ). Similarly, the spatial indicator X s ∈ R T ×N ×2 , consisting of longitude and latitude coordinates, is applied with a linear layer SpatialMLP(X s ; θ s ) to produce the spatial representation E s ∈ R T ×N ×ds . Temporal indicator comprises day-of-week, time-of-day X dow , X tod ∈ R T ×N and timestamps X ts ∈ R T ×N ×6 , which are further compressed into hidden representation, i.e., E t = TemporalMLP(E ts ∥E dow ∥E tod ; θ t ) ∈ R T ×N ×dt , enabling the model to capture the periodicity and sequential features of temporal data. P τ is a task-specific prompt2 to ensure task-awareness, and is integrated into the final embedding. Given the task T k ,EQUATIONwhere ∥ denotes vector concatenation, combining spatial, temporal, and observational embeddings with the task prompt into a comprehensive representation H (T k ) ∈ R T ×N ×d h . We will use H as representation for a specific task in following sections.Spatiotemporal observations are usually complex with multiple-level interactions where such interactions and correlations play vital roles in enhancing commonality learning through different learning domains. To this end, we devise an MSTI, to intricately dissect and disentangle interactions within spatiotemporal data from spatial-temporal indicators to main observations by inheriting nice property of attention mechanisms, which all utilize transformed slices from the integrated representation H.Spatial-context cross-interaction. To quantitatively investigate how spatial indicator interact with main observations, we devise a multi-head cross-attention [14] architecture (MHCA) where spatial and observational components are alternately used as queries (Q) and key-value (KV) pairs:EQUATIONEQUATIONThe symbol ∥ denotes the concatenation of multiple attention heads, and W O is the projection matrix that aligns the output dimensions with those of H. Here, we let variables a and b indicate spatial (s) indicator and main observation (o), i.e., a = s and b = o. Then the queries, keys, and values are generated through following transformations,Q (s) h = H[. . . , slice (s) ]W (Qs) h , K (o) h = H[. . . , slice (o) ]W (Ko) h , V (o) h = H[. . . , slice (o) ]W (Vo) h(5) where W transforms input data into dimension D of attention space, slice (s) and slice (o) denote the respective slices of H for spatial and observational features. After computing attention scores, the embeddings are taken into a feed-forward network (FFN) to enhance the learning capabilities, where FFN(X) = max(0, XW 1 + b 1 )W 2 + b 2 . The final attention outputs are then normalized by,EQUATIONwhere LN is layer normalization, the resulting matrices H(s,o) and H(o,s) are then concatenated back to H at their respective dimensions as H(SCCI) ∈ R T ×N ×d h , enriching the original representation with refined features that encapsulate intricate cross-dimensional relationships.Temporal-context cross-interaction. To facilitate the attention computation with respect to temporal dimension, the representation is first transposed as H′(SCCI) ∈ R N ×T ×d h by denoting T as the sequence length for subsequent attention calculations. Then the step-wise positional encoding (refer to Appendix B.2) is introduced to allow our attention aware of specific temporal evolution.The subsequent steps closely follow those of the spatial-context cross-attention mechanism, so the temporal-context cross-interaction (TCCI) can be performed as H(CI) = TCCI( H′(SCCI) ), and final representation H(CI) ∈ R N ×T ×d h becomes the outcome of cross interactions between spatial and temporal dimensions.Self-interactions within Spatial and Temporal Aspects. We proceed to apply self-interaction across different dimensions of the representations using self-attentions [21] . Specifically, we begin with temporal dimension, as the sequence length of representation dimension has naturally been aligned as T . This setup allows direct computation of the required self-attention,EQUATIONIn this multi-head attention (MHA) configuration, the attention calculation (referred to in Equation 4, 5) involves queries, keys, and values, which is derived from the entire representation H(CI) , rather than individual slices. Then the output undergoes further processing through a FFN and non-linear transformations and LN to stabilize and enrich the feature representations similar Equation 6. The resulting H(T SI) in R N ×T ×d h signifies the outcome of temporal self-interactions (TSI). Then the tensor is transposed to H′(T SI) ∈ R T ×N ×d h . The final spatial self-interaction (SSI) computation is analogous to the temporal version, which refines spatial interactions and aggregates features across spatial nodes, as H = SSI( H′(T SI) ). The resulting tensor from this computation is H ∈ R T ×N ×d h , which represents the outcome of comprehensive multi-dimensional interactions.We then adaptively integrate the interactions by a fusion strategy and Huber loss is adopted to ensure the robustness to outliers in spatiotemporal samples. Details can be found in Appendix B.3. Our MSTI allows extracting diverse interactions, including spatial-temporal domain interactions via designing cross attentions on respective indicators, and self interactions within respective dimensions, enhancing the data relation learning and supporting commonality extraction across task domains.To ensure continuous task learning, we propose a Rolling adaptation scheme, RoAda, to model the distinction and commonality among task domains. Our RoAda is composed of two stages with a warm-up for commonality extraction and a task-specific refinement. Before the task rolling, we construct prompts to distinguish personalization of each task. Thus, the commonality and diversity can be leveraged to boost individual task adaptation.Task summarization as prompts. To capture the task distinction, we devise a task summarization by a Sampling-AutoEncoding scheme from each task. Consider task T k , main observation becomesX (T k ) ∈ R T (T k )all ×N ×C . Such data is sampled by averaging observations over equivalent times of day, yielding a periodic sample representation X(T k ) samp ∈ R L (T k ) t ×N ×C , where L (T k ) tdenotes the number of time slots for task k within a day. Since neural networks tend to fit any data regularity, the sampled features are fed into an autoencoder for extracting the compressed and distinguished data patterns. Given the encoding and decoding processes, i.e., ϕ : X samp → S and ψ : S → X ′ samp , S encapsulates the summarized core characteristics of the task. The decoding phase maps S back to a reconstructed X ′ samp , by minimizing the mean squared reconstruction error,EQUATIONwhere W s is the weight, and b s is the bias. Following the encoder, the summary features S (T k ) are transformed into the k-th task promptP (T k ) = FC p (S (T k ) ; θ p ) with dimension alignment.Weight behavior modeling. The first warm-up stage is designed via weight behavior modeling, which assimilates the regularity from task to task. This process not only adapts the model to new tasks but also solidifies its ability in generalizing across scenarios by capturing task-wise common relations with modeling of model weight behaviors.We begin with the task T 1 via independently training the model until its performance stabilizes. By denoting M as the model learned by MSTI, the training phase can be formally described as,EQUATIONwhere W init are the initialized weights, D To capture common patterns, we reflect the task-level stability and variations by the evolution behavior of weights in M, i.e., W (T2) = {W(T1) c , W (T2) 0 , W (T2) 1 , . . . , W (T2) c}. The weight set W (T2) is deliberately incorporated with finalized weight of task T 1 and evolution of T 2 , which explicitly captures the weight transition between tasks. We then introduce a collective variance σ to capture such stability, and employ a threshold δ to disentangle the stable and dynamic weights along the learning process, i.e.,EQUATIONwhere Var(w ij ) represents the element-wise variance of the across different training iterations from W .frozen(W(T2) stable ))) for -----→ each epoch W (T3) 0 , W (T3) 1 , . . . , W (T3) c (12)Similar to Equation 12, this process is repeated until the completion of task T k , resulting in the collectionW (T k ) = {W (T k-1 ) c , W (T k ) 0 , W (T k ) 1 , . . . , W (T k ) c }. The Equation 11 is similarly implemented to derive W (T k ) = W (T k ) stable ∥ W (T k )dynamic . Since T 1 is not involved with the commonality extraction, we subsequently load T 1 with W (T k ) .frozen(W (T k ) stable ) to achieve the complete rolling process. The stabilized weights across continuous tasks W ′ (T1) = W ′ (T1) stable ∥ W ′ (T1) dynamic , serves as W * can ultimately result in a robust multi-task learning parameters encapsulated with well-extracted common patterns via iterative stable weight selection, and it can also be served a collective intelligence by exploiting multiple tasks, thus enhancing the generalization for subsequent learning.Task-specific refinement phase. This phase aims to merge the gap between task-level commonality and specificity, where the process continuously train the model from the prior phase to the next one, as W * stable is frozen to maintain overall model stability and remaining weights are iteratively update with task-specific prompts. This process allows CMuST to insert the individual intelligence into integrated model and adjust itself to better suit the unique pattern of each task, i.e.,EQUATIONwhere M (Ti) is tuned to maximize performance on the task at hand, which represents the task-specific submodel after refinement. The FineTuning denotes adjustment process of model parameters.Summary. Our RoAda not only ensures the preservation of commonalities across tasks, but each model is also optimally tuned for its respective task with compressed task-level patterns, providing CMuST with opportunity to achieve peak performance with both collective and individual intelligence. More details of our methodology can be found in Appendix. 
datasets and experiment setup: 
 Data description. Given the emerging multi-task ST learning, we collect and process three realworld datasets for evaluation: 1) NYC3 : Includes three months of crowd flow and taxi hailing from Manhattan and its surrounding areas in New York City, encompassing four tasks: Crowd In, Crowd Out, Taxi Pick, and Taxi Drop. Baselines and metrics. Our CMuST model is evaluated on a widely-used baselines spatiotemporal prediction, including RNN-based models (DCRNN [19] , AGCRN [4] ), STGNNs (GWNET [40] , STGCN [44] ) for single task, and MTGNN [39] , STEP [33] PromptST [48] for multiple tasks, as well as attention-based models of (GMAN [50] , ASTGCN [9] , STTN [42] ). The performance metrics are mean absolute error (MAE), and mean absolute percentage error (MAPE), where lower values indicate higher predictive performance.Implementation details. We partitioned datasets into training, validation, and testing sets with 7:1:2 ratio. CMuST forecasts observations of next 12 time steps based on previous 12 steps, as T = T ′ = 12, . All data were normalized to zero mean and unit variance, and predictions were denormalized to normal values for evaluation. For the MSTI, embedding dimensions were d obs = 24, d s = 12, d t = 60, and the prompt dimension was 72. Dimensions for self-attention and crossattention respectively were 168 and 24, with each attention having 4 heads and FFN's hidden dimension was 256. The Adam optimizer is adopted with an initialized learning rate of 1 × 10 -3 , and weight decay of 3 × 10 -4 , where the early-stop was applied. For RoAda, the threshold δ = 10 -6 .Our model was implemented with PyTorch on a Linux system equipped with Tesla V100 16GB.1) Performance comparison among baselines. In Table 1 , we compared the predictive performance of our CMuST with other methods across various city datasets. Obviously, our CMuST significantly outperforms other baselines on all datasets across most of metrics and the multitaskbased methods improve performance by an average of 8.51% over the singletask-based methods. This result underscores the effectiveness of the cross-attention mechanism for decoupling multidimensional dependencies, which not only enhances spatiotemporal representational capacity, but also enables easy extraction of common correlations among tasks and thus empowering each individual task to benefit from well-extracted common patterns. 2) Robustness in data-scarce scenarios within multi-task framework. We constructed scenarios of data scarcity for specific tasks, i.e., we reduce part of the spatial nodes in prediction of Crowd In, and also reduce number of samples in temporal dimension by expanding time intervals on NYC, to study the robustness of CMuST under the challenging scenario of limited (reduced) data. Results shown in Table 2 indicate that assimilating common information from other tasks can help better prediction in a single task even though it is with limited data on either spatial and temporal dimension. This demonstrates that multi-task prediction relax the requirement of individual tasks on both data volumes and distributions, where the shared commonality effectively captures and delivers the consistency and diversity among tasks.To assess the effectiveness of each module in CMuST and its capability for multi-task learning, we designed a set of variants as 1) w/o context-data interaction: remove the spatial-context and temporal-context cross-interactions in the MSTI module, 2) w/o consistency maintainer: omit the separation and freezing of stable weights during the RoAda phase, instead using all weights for rolling training, 3) w/o task-specific preserver: eliminate the task-specific prompts, thus removing the task-specific diversity preservation. Figure 3 supports our hypothesis that CMuST is a cohesive and integral system, where the results of "w/o context-data interaction" deteriorate, potentially indicating multidimensional interactions from context environments to spatial relations and temporal evolution exactly make sense for prediction. The results on "w/o consistency maintainer" highlights the validity of capturing task consistency and commonality to facilitate task learning. Lastly, performances of "w/o taskspecific preserver" show that removing unique patterns distinctive from other tasks makes inferior results.1) Visualizing attention across training phases. In Figure 4 (a), we visualize the changes of attention weights of CMuST during the stage of RoAda. It is observed that as tasks are learning continuously, the relationships and interactions across various dimensions is becoming distinctive and going stable, demonstrating the consolidation process of dimension-level relations. By modeling weight behavior, such consolidated relations and interactions between context and observations can further enable the extraction of consistency in spatiotemporal interactions across tasks. 2) Performance variation along with task increasing. Figure 4 (b) shows the performance of individual tasks on NYC as the number of tasks increases. The performance of each task is improving with the addition of more tasks, which indicates that tasks are no longer isolated but gain the collective intelligence via assimilating common representations and interactive information. Figure 4 : Case studies of our proposed CMuST on NYC.We varied the dimension of the task prompt d p as {18, 36, 72, 144}, the number of attention heads in MSTI as {1, 2, 4, 8, 16}, and the threshold δ for RoAda among {10 -4 , 10 -5 , 10 -6 , 10 -7 }. Results shown in Figure 6 indicate that the optimal settings were d p = 72, head = 4, and δ = 10 -6 . Details of parameter analysis and fine-tuning can be found in our Appendix C.6. 
conclusion: 
 In this work, we enable spatiotemporal learning to get rid of isolation with proposed CMuST, which consists of two major components. In CMuST, the MSTI is devised to dissect complex multidimension data correlations, to reveal disentangled patterns. To extract the task-wise consistency and task-specific diversity, we propose a rolling learning scheme RoAda, where it simultaneously models weight behavior to enable collective intelligence, and constructs task-specific prompts by compressing domain data with AutoEncoding to empower task-specific refinement for enhancement. We believe our CMuST can not only help better understand the collective regularity and intelligence in urban systems, but significantly reduce repeated training and improve data exploitation, which is progressively approaching green computing in future cities. For future work, we will further investigate the collective intelligence in open urban systems, which can potentially generalize to wider domains such as energy and environment for human-centered computing. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our code has been provided in anonymous link.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 The Concept, Definition and Scope of Multi-task LearningActually, in our study, various domains correspond to different urban elements collected with different manners in a given city. For instance, in an integrated urban system, it includes taxi demands, crowd flow, traffic speed and accidents. We collect and organize various domain data (urban elements) in a city into one integrated dataset. The goal of our work is to explore the integrated intelligence from various domains and enhance learning of each individual urban element. To this end, the concept of multi-task here is to forecast various elements from different domains in an integrated model. Therefore, our work does not target at unifying regression or classification problems, but proposes an integrated model to iteratively establish the common intelligence among different elements and improve generalization for each element learning in succession, thus getting rid of task isolation. Noted that our experiments are performed with regression tasks, but it can easily generalize to classification task with shared representations.In this work, 'continuous' is equivalent to 'continual'. The uniqueness of our work refers to a novel continuous task learning in ST community, which collects the integrated intelligence and benefits each individual learning task. Given the D dimensions in total, we define the positional encoding as,EQUATION)where t and d are the indexes of time slots and feature dimensions, which alternately uses sine and cosine to provide a unique representation for each time slot.Predictions for future steps are influenced by representations affected by multi-dimensional interactions, where the impact of each dimension may vary. Therefore, we integrate information across various aspects effectively by a parameter matrix-based fusion strategy,EQUATION)where * denotes the convolution operation, employing 1 x 1 convolution kernels. W o , W s , and W t are the parameters within these kernels, tailored to adjust the influence of observational data, spatial locations, and temporal information on the prediction targets, respectively. We further integrate the task-specific prompt to culminate in the final prediction Ŷ = FC y (ZW z + H[. . . , slice p ]W p ; θ y ), the Huber Loss [26] is utilized as the optimization function, which is less sensitive to outliers compared to squared error loss, defined as:EQUATIONwhere Y is the ground truth, δ controls the sensitivity to outliers, which enhances the accuracy by integrating diverse data dimensions and improves robustness against data variability and anomalies.The detailed algorithmic procedure for the RoAda phase can be found in Algorithm 1.We provide detailed information about the dataset, including the number of records in the original data, the number of regions into which the data was divided, and the time intervals. These details are presented in Table 3 . The specific data preprocessing is as follows:NYC: We collect yellow taxi trip data from January to March 2016 from the NYC Open Data website. Each trip record includes information such as pickup and dropout times, locations, and the number of passengers. We filter out records with abnormal longitude and latitude values or missing data. Then we select data within Manhattan and surrounding areas, divided into 30x15 grids, and counted trips per grid, selecting those with total trips greater than or equal to 1000, resulting in 206 grids. Each grid's data is aggregated into 30-minute intervals, yielding taxi pickup counts, taxi dropout counts, and crowd in/out flows. We also include time of day (tod) and day of week (dow) as context, resulting in four tasks with input features [value, tod, dow].We collect traffic data from Suzhou Industrial Park from January to March 2017, comprising tens of thousands of records. The area is divided into nodes, and data is aggregated into 5-minute intervals. After filtering out grids with sparse data, we obtain 108 nodes, each containing traffic speed and traffic flow. We include time of day and day of week as input context, resulting in two tasks: traffic flow and traffic speed, with input [value, tod, dow].Chicago: We collect taxi trip and accident data from the Chicago Open Data platform for June to December 2023. The taxi data includes trip start, end times and locations. We divide the area into 30x20 grids and select grids with total trips greater than 100, resulting in 220 grids. Similar to the NYC dataset, data is aggregated into 30-minute intervals, yielding taxi pickup and dropout counts, resulting in two tasks with input features [value, tod, dow]. The accident data includes incident locations, times, casualty numbers, and injury severity of each casualty. We then obtain the risk score by weighting it according to each casualty and injury, mapped it to the 220 grids, and aggregated the risk score over time intervals, resulting in a risk task with input features [risk score, tod, dow].To verify whether the multi-task learning in urban systems can compete single-task learning scheme and further show the superiority of our continuous multi-task learning, our experiments are designedRequire: Model M, tasks {T 1 , T 2 , . . . , T k }, prompts {P (T1) , P (T2) , . . . , P (T k ) }, train data {D train }, learning rate γ, threshold δ 1: Warm-up Phase: 2: ▷ For all parameters in M, the parameters corresponding to their respective names ◁ 3: Initialize: W histories ← {} ▷ Dictionary with keys as names and values as parameters list 4: for each task T i in {T 1 , T 2 , . . . , T k } do Load prompt for task T 1 : M ← P (T1)8:Set learning rate as γ for each epoch j do After training, obtain the weights of M as W▷ Calculate variances and freeze stable weights ◁ 20:for each parameters' name in M do21: v ← CALCULATE_VARIANCE(W histories [name]) 22: if v < δ then 23:Freeze the parameter corresponding to the name 24:Reset W histories with W (Ti) c25: ▷ Train task 1 again since it's not involved with the commonality extraction ◁ 26: M ← P (T1) 27: The remaining steps are similar to lines 14-22 28: Finally, the weights of M are saved as W * 29: Task-Specific Refinement Phase: 30: for each taskT i in {T 1 , T 2 , . . . , T k } do 31:Load weights and prompt M ← W * , M ← P (Ti) 32:Set learning rate as γ 33:Fine-tune M on D (Ti) train and save model into both single-task learning and multi-task learning. We will further clarify how to ensure the fair evaluation and comparisons across different models.For single-task learning, we take different datasets as individual ones to respectively train and test the model [23] , where compared models and our CMuST are respectively and repeatedly trained for 5 times with different random seeds. The average results are reported in our Table . 1.For multi-task learning, since existing models have not designed the task-level continuous learning scheme, we align the features of different types of data in the same urban system into a same urban graph, where data features are concatenated for training the model. The learning objectives are followed by corresponding statement of each paper. For our CMuST, we implement the continuous and iterative model update described in Sec. 4.4. The fairness is well-incorporated by ensuring the all the data input to each model is equivalent.The experimental results find that almost all multi-task learning can compete against single-task learning, and our continuous multi-task learning is also superior to other multi-task learning without explicitly capturing the task commonality and diversity. To avoid catastrophic forgetting, we implement several strategies during the RoAda phase. Firstly, we set the learning rate for each task to 1e-5. This helps to retain more knowledge from previous tasks and prevents the model from over-adjusting to new tasks. By maintaining a low learning rate, the model can incrementally learn new information while preserving the stability of previously learned tasks. Additionally, we use task-specific weights for each task, such as task prompts. This method allows us to absorb the common features across all tasks while independently preserving and updating task-specific parameters. This approach ensures that when learning new tasks, the model does not forget the knowledge gained from previous tasks, thereby preventing catastrophic forgetting and avoiding overfitting to new tasks.We compare with unified spatiotemporal/time series learning, such as UniST [45] and UniTime [22] , to better show the generalization ability of our model for multi-task learning in the same urban system. The results of these in Table 5 .Table 4 : Explanation of table symbols of Table 5 and Table 6 . The datasets are numbered by 1 ⃝ 2 ⃝ 3 ⃝ respectively, and the specific tasks in the dataset are numbered by Roman numerals III and so on.Notation(Dataset/task) NYC Crowd In 1 ⃝/I NYC Crowd Out 1 ⃝/II NYC Taxi Pick 1 ⃝/III NYC Taxi Drop 1 ⃝/IV SIP Traffic Flow 2 ⃝/I SIP Traffic Speed 2 ⃝/II Chicago Taxi Pick 3 ⃝/I Chicago Taxi Drop 3 ⃝/II Chicago Risk 3 ⃝/IIIWe have designed the experiment of cold start. Specifically, for NYC dataset, we selected three of the four tasks of Crowd In, Crowd Out, Taxi Pick and Taxi Drop in turn for training, and calculated the adaptation time and results for the remaining one task on this basis, comparing with training a single task alone. A similar design is applied for SIP and Chicago datasets. The results are shown in Table 6 , which show that, both in terms of effect and time, it performs better than single task, indicating that our model adapts to the newly arrived task more quickly and well, which is conducive to solving the problem of cold start of urban prediction. To study the impact of hyperparameters on model performance, we varied the dimension of the task prompt d p as {18, 36, 72, 144}, the number of attention heads in MSTI as {1, 2, 4, 8, 16}, and the threshold δ for RoAda among {10 -4 , 10 -5 , 10 -6 , 10 -7 }. The experimental results, displayed in Figure 6 , are based on analysis of three datasets. 1) Initially, as the dimension of the prompt increases, both MAE and MAPE decrease, indicating that within a lower range, increasing d p benefits the model by capturing more task-specific information. However, performance significantly deteriorates after d p exceeds 72, suggesting that additional dimensions may carry redundant and non-informative elements that introduce noise and prevent the model from effectively learning useful information. 2) On the other hand, as the number of heads increases, model performance initially improves, allowing the attention mechanism to capture more diverse and dimensional information through different heads. Yet, this positive effect reverses after exceeding four heads, indicating that too many heads can lead to information overlap, reducing efficiency and adding unnecessary complexity to the model, thereby degrading performance. 3) During the rolling adaptation phase, both excessively high and low values for the δ threshold lead to performance degradation. When δ is set too low, it results in fewer stable weights being retained, which may fail to preserve essential information critical for learning stability. Conversely, setting δ too high results in more weights being frozen, which restricts the model's flexibility and adaptability, locking in potentially dynamic information and preventing the model from adapting to novel and evolving data patterns effectively.Our current work is confined to a single urban system and our experiments are solely within the transportation sector. The exploration into other domains within the same city, such as electricity, transportation, and pollution multi-task learning is still limited. Addressing these limitations will form the basis of our future research. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There are no such risks of the datasets in this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have properly cited the original papers and web pages that produced the code package and dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators.We introduce a task summarization as the prompt (refer Section 4.4) to generate the distinct prompts.https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.pagehttps://data.cityofchicago.org/browse 


Decision: The paper presents a Continuous Multi-task Spatio-Temporal learning framework (CMuST) that introduces a Multi-dimensional Spatio-Temporal Interaction network (MSTI) for capturing complex data interactions and a Rolling Adaptation training scheme (RoAda) to iteratively update the model, simultaneously maintaining task uniqueness and leveraging shared patterns across tasks.The main goal of the work (as summarized by the authors in their rebuttal) is to explore the integrated intelligence from various domains and enhance learning of each individual urban element. To this end, the concept of multi-task here is to forecast various elements from different domains in an integrated model. The framework is validated on data from 3 cities to enhance urban intelligence. 

The paper presents novel technical contributions as summarized by the various reviewers. The benchmark construction and experimental designs are also novel. Also, all reviewers have agreed that the authors have addressed their concerns. 

As the paper presents a novel framework, with a novel continual learning method for spatio-temporal data sets aimed at enhancing urban intelligence, it will be of interest to a larger audience in the domain.

Reviews: review: paper_summary: paper_summary: This work proposes a multi-task spatiotemporal learning framework that helps the model understand the relationships between multiple tasks. The specific contributions lie in proposing MSTI to model the multidimensional spatiotemporal data and RoAda to capture the commonality and personalization among multiple tasks.
strength_weakness: strength_weakness: 1.	The author mentions multi-task and multi-domain problems several times in the introduction, but these concepts are not intuitively introduced in the paper. Multi-task and multi-domain do not always have a unified consensus in the ST community. For example, does multi-task include regression tasks and classification tasks? Does multi-domain refer to different cities or different modes of transportation? The author should provide more specific scopes for these terms in the introduction.
2.	The author's method of handling ST dependencies is not novel. Using cross-attention techniques to model from the perspectives of temporal dimension, spatial dimension, and spatiotemporal relationships separately is a common processing paradigm in the ST community. The contribution of MSTI can be considered overstated.
3.	The research on ST prediction and continual learning in the related work section is insufficient, lacking analysis of advanced ST prediction models and continual learning models in recent years.
4.	The author argues that continual learning can help spatiotemporal models enhance generalization ability. The theory and experiments in the paper are insufficient to support this argument.
5.	The author neglects experiments on cold start problems.
6.	The comparative experiments did not achieve the best results on all tasks; reasons for this should be analyzed.

Overall, I believe this work proposes a very attractive challenge, but in the end with the old problem, it only solves a standard problem, i.e. the ST multi-task problem. The author proposes Rolling Adaptation to solve this problem, which is a contribution that cannot be ignored. However, this work is incomplete for many issues mentioned in the introduction are not addressed, the definition of tasks is confusing at the beginning, and there is a lack of a pseudocode algorithm to help readers understand the proposed method more accurately.
questions: questions: 1.	Is the author using "continuous learning" to replace the commonly used "continual learning" in the community to indicate the uniqueness of this work?
2.	Did the author train three models on three datasets, or train only one model and continuously update it on three datasets? In other words, the author provided definitions for temporal increment, spatial increment, and feature increment in Definition 3, but in the actual work, was the increase in spatial nodes ignored?
limitations: limitations: The limitations have been addressed.
paper_summary: paper_summary: This paper proposes a Continuous Multi-task Spatio-Temporal learning framework (CMuST) to enhance urban intelligence. CMuST introduces a Multi-dimensional Spatio-Temporal Interaction network (MSTI) for capturing complex data interactions and a Rolling Adaptation training scheme (RoAda) to iteratively update the model, simultaneously maintaining task uniqueness and leveraging shared patterns across tasks. The framework is validated through extensive experiments on datasets from three cities, demonstrating superior performance against existing methods.
strength_weakness: strength_weakness: 1. In Section 4.4, there is missing detailed description on how to avoid catastrophic forgetting during task rolling adaptation. It would be beneficial if the authors could provide more experimental details in this regard.

2. Lacking comparison baselines. More baselines which are argued for unified spatiotemporal/time series learning, such as UniST, UniTime should be added for comparisons.
questions: questions: 1. In your CMuST, how can you avoid catastrophic forgetting during task rolling adaptation?

2. Do more comparison experiments with SOTA baselines will be better.
limitations: limitations: The limitations are discussed.
paper_summary: paper_summary: This paper proposes a Continuous Multi-task Spatio-Temporal Learning Framework CMuST to facilitate the task-level cooperation in spatiotemporal predictions (mainly for traffic related tasks). The model is composed of three components . Data representation and integration module processes and standardizes diverse urban data into a harmonized format. MSTI modules reduce the complex interactions within spatiotemporal data. RoAda modules iteratively captures the task-wise consistency and task-specific diversity.

This study is generally fine with a relatively novel method for spatiotemporal multi-task learning through prompting (although prompting studies in this field is being rapidly developed). The main contribution is how the prompting is handled. The main problem I found is that the text is not that easy to follow with many jargons, coined (complicated) phrases, e.g., continuous multi-task spatiotemporal learning is a bit confusing -  is it something related to continual (lifelong) learning?
strength_weakness: strength_weakness: W1: During the model construction process, the paper does not clearly address the inconsistency issue of feature C across different task data. It is recommended to provide detailed explanations either in the data preprocessing stage or within the "Data representation and integration module" of the model.
W2: For continuous task rolling, specific operational details of each task model from training to convergence (such as the number of epochs) are not mentioned in the paper. 
W3: Typos such as "Compressedd" (line 235).
W4: Modify proprietary terms that lack detailed explanations, such as "PECPM" on line 124.

Overall, the paper falls short in its presentation. Hope that it could be revised to ease the understanding.
questions: questions: Q1: The definition of "domain" in the paper appears somewhat ambiguous. Please clarify whether "domain" refers to different types of tasks (such as "pick up" vs. "drop off") or different geographical regions (such as "NYC" vs. "SIP").
Q2: Please explain the distinction between the symbols "H[. . . ,slice(s)]" and "Es". Currently, these two parts appear to belong to the same content.
Q3. For different tasks in the same city, are the input features the same?
limitations: limitations: Yes
paper_summary: paper_summary: This work proposed a novel spatiotemporal learning framework CMuST. In CMuST, MSTI is devised to dissect complex multi-dimension data correlations, to reveal disentangled patterns. And RoAda is proposed to extract the task-wise consistency and task-specific diversity. In addition, this paper introduce a benchmark of three cities for multi-task spatiotemporal learning, and empirically demonstrate the superiority of CMuST via extensive evaluations on these datasets.
strength_weakness: strength_weakness: 1. The paper considers one of its major contributions to be the proposal of a benchmark. However, after reviewing the code, it seems that the authors did not provide details on how they processed the data.

2. The proposed MTSI module largely uses the Attention module from Transformers, but the authors did not provide any references to Transformers. Additionally, using the attention mechanism to capture relationships is a relatively straightforward design and lacks significant innovation.

3. One of the main problems this paper addresses is the cold start problem for new tasks. However, the paper still involves task-specific refinement, i.e., training is still required. Perhaps the superiority of the proposed module can be validated by comparing the adaptation time to new tasks.

4. In the experiments conducted by the authors, it can be observed that when the number of tasks is one, the model's performance is not superior to many models. This might indicate that the proposed MTSI module is not sufficiently effective.

5. The authors use a simple layer to obtain task summarization $S$. Compared to common designs in many MAEs, using just a linear layer with an activation function seems somewhat simplistic for MAE.
questions: questions: Please explain each weaknesses.
limitations: limitations: This paper adequately addressed the limitations.

Comments: title: Thanks for your time on our manuscript
Thanks for your recognition and insightful suggestions
Thanks for your constructive suggestion and positive feedback
Thanks for your support and thoughtful comments
Thanks for your positive feedback
Response
comment: Dear AC and reviewers, 

We would like to express our great appreciation of your valuable time on our manuscript. As the deadline of reviewer-author discussion period is approaching, we wonder where there are any further questions on our work and we are always willing to resolve them, thus help better understanding of this work on a novel task-level continuous ST learning. We finally thank you all can support our work during the successive stages of discussions and we promise to public all the resources of this work including both datasets and codes if accepted.  

Sincere thanks!

Authors of Paper  2077
Dear Reviewer SHU9,

Thank you for your recognition and valuable suggestions, we will carefully revise our manuscript, including adding more experimental details and baselines to further improve the quality for satisfying the high-level requirement of NeurIPS community. Thanks a lot!

Authors of Paper 2077
Thanks for the response. I have checked all the content and also my concerns are well-addressed. This work  did address a new problem and contributed new techniques in the field of spatiotemporal learning. I would like to raise my score to 7.
Dear Reviewer YiFX,

We would like to express our deeply gratitude to your professional reviews and useful suggestions for promoting our manuscript. We promise to polish our manuscript, by improving the readability, supplementing more related works, and providing additional experiments. Many thanks! Hope you a nice day!

Authors of Paper 2077
Thank you for your response. I have raised the rating.
Dear Reviewer GaXf,

We sincerely thanks for your support and valuable comments of our work, which have played a great significant role in the quality of our manuscript. We will make our best efforts to improve the presentation of the manuscript and add corresponding experimental details, as well as make the code and dataset open source for reference and availability. We would appreciate it if you could give us your kind support during the discussion phase. Thank you very much.

Authors of Paper 2077
Dear Reviewer xkVJ,

We sincerely appreciate your constructive feedback and valuable comments, which really contributed to the enhancement of our manuscript. We will try our best to improve the  quality of manuscript and make both codes and datasets open-sourced. Thanks very much!

Authors of Paper 2077
Thank you for your response. I have raised my assessment.
Thank you for your response. My concerns are addressed. And I have raise my score.