XXLTraffic: Expanding and Extremely Long Traffic forecasting beyond test adaptation

Authors: Du Yin, Hao Xue, Arian Prabowo, Shuang Ao, Flora D. Salim
Primary area: datasets and benchmarks
Keywords: spatio-temporal, traffic forecasting, time-series

Abstract:
Traffic forecasting is crucial for smart cities and intelligent transportation initiatives, where deep learning has made significant progress in modeling complex spatio-temporal patterns in recent years. However, current public datasets have limitations in reflecting the distribution shift nature of real-world scenarios, characterized by continuously evolving infrastructures, varying temporal distributions, and long temporal gaps due to sensor downtimes or changes in traffic patterns. These limitations inevitably restrict the practical applicability of existing traffic forecasting datasets. To bridge this gap, we present XXLTraffic, the longest available public traffic dataset with the longest timespan collected from Los Angeles, USA, and New South Wales, Australia, curated to support research in extremely long forecasting beyond test adaptation. Our benchmark includes both typical time-series forecasting settings with hourly and daily aggregated data and novel configurations that introduce gaps and down-sample the training size to better simulate practical constraints. We anticipate the new XXLTraffic will provide a fresh perspective for the time-series and traffic forecasting communities. It would also offer a robust platform for developing and evaluating models designed to tackle the extremely long forecasting problems beyond test adaptation. Our dataset supplements existing spatio-temporal data resources and leads to new research directions in this domain.

Decision: withdrawn
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1729964154314

thread_type:
official_review

<thread_object>
creator: reviewer_kwqa

<content>
summary:
This paper present an XXLTraffic traffic dataset spanning up to 23 years in different countries, which can be useful for the other researchers. They have also implemented the datasets with many advanced approaches to confirm the range and different experiments directions.

strengths:
1. This paper introduces an XXLTraffic dataset that spans up to 23 years from different regions, which means the dataset include sufficient information for the traffic feature.
2. This paper presents many potential conditions that the dataset could be applied to such as short-term traffic prediction, long-term multivariate traffic prediction and extremely long term prediction with gaps. All of these domains are worthwhile problem to focus.

weaknesses:
1. The dataset is valuable, but performing extremely long-term predictions is challenging, as traffic patterns can change significantly over the years, and even the road infrastructure itself may be reshaped over a 23-year span. Such factors could potentially reduce the impact of the dataset's contribution, because many open datasets are enough for normal short and long term prediction.
2. Some typos, e.g., ‘beyongd’  should be ‘beyond’ around line 130;

questions:
How is the gap dataset collected, given that "gap" typically refers to missing data? Can this dataset be directly used to address traditional missing data scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586174538

thread_type:
official_review

<thread_object>
creator: reviewer_yrbw

<content>
summary:
This paper proposed large-scale datasets, named XXLTrafic, to address limitations in existing traffic forecasting datasets. It provides extensive, real-world datasets with long timespans to support research in extremely long forecasting and adaptive model development for evolving traffic patterns.

strengths:
1.	The proposed XXLTraffic is the largest publicly available traffic dataset so far. The PEMS part spans up to 23 years and the NSW part also has around 11 years’ data.

2.	The authors evaluated the performance of existing baselines on the proposed datasets with various settings. The overall results verify the importance of introducing large-scale datasets with extremely long time spans.

weaknesses:
1.	The most significant weakness is that this paper has no novelty. Although this conference has a dataset and benchmarks area, I do not think, for a top-tier conference, it is enough to propose a large-scale dataset without any model contribution. Given the fact that there are existing large-scale datasets available with just the number of nodes or time spans fewer than XXTraffic, my concern becomes more intense.

2.	Some motivations remain unclear. In the introduction section, the authors mentioned ‘beyond test adaptation’. However, it is not easy to understand how this relates to the proposed datasets. Is the concept of ‘beyond test adaptation’ better or more realistic than test time adaptation?

3.	The setting of Extremely Long Prediction with Gaps is a bit strange to me. Here, the authors used an example of highway route planning prediction to justify the necessity of the gap. It is possible that we need to stimulate the traffic flows of uninstalled sensors to facilitate road planning. Thus, the optimal setting should be using existing sensors to predict traffic flows of uninstalled sensors, plus the gap. However, it seems the authors still used historical signals of uninstalled sensors in this setting. If we already know their historical readings, we do not have to plan anything.

questions:
The authors can address the above concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710821128

thread_type:
official_review

<thread_object>
creator: reviewer_ytp5

<content>
summary:
This paper presents a traffic dataset containing extensive data records from multiple regions over a long-term period, aimed at supporting research in ultra-long-term traffic prediction.

strengths:
(1) A new traffic dataset is proposed, comprising data from numerous regions across California and New South Wales over an extended time period.

(2) The temporal distribution evolution of selected data was visualized, revealing the evolutionary characteristics of the data.

(3) Tests were conducted on relevant prediction tasks, including scenarios with time gaps and varying input lengths.

weaknesses:
(1) Compared to existing works like LargeST, this work's contribution is insufficient as it merely expands the temporal and spatial scope of data collection. These data can be obtained through the open-source PEMS system in the same way.

(2) Although the number of regions and time span are substantial, the vast majority of data is limited to California. Including data from more cities and countries might have been a better approach.

(3) Most baselines in the experiments are specifically designed for time series prediction, lacking models specifically designed for traffic prediction/spatiotemporal prediction, such as PDFormer[1], ASTGCN[2], STGCN[3], AGCRN[4], GWN[5], etc. It is important to explore the performance of these types of baselines on traffic datasets.

[1] PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction. AAAI 2023.

[2] Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. AAAI 2019.

[3] Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. IJCAI 2018.

[4] Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. NeurIPS 2020.

[5] Graph WaveNet for Deep Spatial-Temporal Graph Modeling. IJCAI 2019.

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733741650

thread_type:
official_review

<thread_object>
creator: reviewer_6qsu

<content>
summary:
The authors present XXLTraffic, the longest available public traffic dataset with the longest timespan collected from Los Angeles, USA, and New South Wales, Australia, curated to support research in extremely long forecasting beyond test adaptation. The benchmark includes both typical time-series forecasting settings with hourly and daily aggregated data and novel configurations that introduce gaps and down-sample the training size to better simulate practical constraints.

strengths:
1. A brand-new dataset is proposed, which is the largest available public traffic dataset in extremely long traffic forecasting.

2. Numerous comparative experiments validate the differences in performance of different models on this dataset, providing a good guide for peers to understand the latest technological developments.

weaknesses:
1. The maximum number of nodes in the proposed dataset is less than the Large-ST, which seems to need improvement, after all, the real road network is very large.

2. I think it is necessary for the authors to explain in more depth the motivation for constructing such a long-term dataset, because as transportation infrastructure advances and human travel modes change, traffic data that is too ancient is not always helpful enough to understand future transportation patterns.

3.  Due to the sheer size of the proposed dataset, it requires more computational resources. Therefore, whether this dataset can be popularized and whether peers can easily adopt it in their own research is also a concern.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732498319258

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Global Rebuttal

comment:
We sincerely thank all the reviewers for the comments that have greatly improved the paper quality. We have taken all your comments seriously. We have added the new baselines, new analysis and clarify some confusing questions. The following global rebuttal provides 5 more baselines and computational statistics done during the rebuttal period.

- (1) For Reviewer ytp5’s suggestion, we have added five additional baselines shown in Table 1

**Table 1. Compasirons with different traffic forecasting baselines**
||||**STGCN**||**ASTGCN**||**GWN**||**AGCRN**||**PDFormer**||
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|**Methods**|**Metrics**||**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|
|||96|0.556|0.536|0.765|0.618|0.676|0.574|0.596|0.562|0.621|0.576|
||**1-yeargap**|192|0.562|0.539|0.764|0.626|0.581|0.546|0.565|0.543|0.545|0.536|
|||336|0.561|0.538|0.717|0.598|0.582|0.543|0.580|0.552|0.574|0.548|
|||96|1.256|0.884|1.441|0.959|1.250|0.875|1.168|0.866|1.256|0.899|
|**PEMS03\_gap**|**1.5-yeargap**|192|1.187|0.872|1.395|0.947|1.194|0.856|1.122|0.843|1.117|0.842|
|||336|1.167|0.853|1.273|0.877|1.1015|0.832|1.176|0.862|1.182|0.871|
|||96|2.059|1.176|2.236|1.234|1.987|1.148|1.950|1.150|2.068|1.189|
||**2-yeargap**|192|2.055|1.184|2.180|1.213|1.877|1.117|1.746|1.070|1.897|1.131|
|||336|2.022|1.169|1.889|1.117|1.847|1.107|1.930|1.140|1.982|1.161|
|||96|1.233|0.904|1.311|0.929|1.310|0.87|1.290|0.865|1.101|0.851|
||**1-yeargap**|192|1.398|1.025|1.443|1.023|1.255|0.879|1.318|0.907|1.078|0.816|
|||336|1.505|1.104|1.325|0.938|1.292|0.865|1.302|0.899|1.085|0.821|
|||96|1.369|1.082|1.350|0.937|1.684|1.063|1.222|0.961|1.097|0.849|
|**PEMS04\_gap**|**1.5-yeargap**|192|1.642|1.035|1.549|1.075|1.501|0.988|1.346|0.989|1.214|0.903|
|||336|1.368|0.862|1.199|0.832|1.584|0.997|0.231|0.963|1.130|0.854|
|||96|1.653|1.074|1.247|0.894|1.669|1.057|1.236|0.901|1.099|0.861
||**2-yeargap**|192|1.545|1.023|1.356|0.948|1.554|1.041|1.269|0.915|1.159|0.891|
|||336|1.556|1.011|1.174|0.842|1.128|0.869|1.314|0.947|1.032|0.816|
|||96|2.531|1.325|3.119|1.404|2.185|1.280|2.158|1.220|1.921|1.148|
||**1-yeargap**|192|2.560|1.337|2.405|1.276|2.223|1.311|2.144|1.213|2.248|1.260|
|||336|2.482|1.317|2.086|1.181|2.228|1.312|2.036|1.185|1.994|1.179|
|||96|1.772|1.084|2.370|1.192|1.309|0.896|1.166|0.808|1.664|1.055|
|**PEMS08\_gap**|**1.5-yeargap**|192|1.522|0.979|1.730|1.043|1.074|0.805|1.144|0.810|1.518|1.012|
|||336|1.495|0.966|1.444|0.948|1.533|0.999|1.078|0.784|1.049|0.801|
|||96|1.336|0.868|1.962|1.069|1.789|1.017|1.393|0.856|1.071|0.792|
||**2-yeargap**|192|1.218|0.819|1.292|0.795|1.007|0.761|1.229|0.813|1.171|0.826|
|||336|1.181|0.806|1.227|0.842|1.181|0.813|1.246|0.814|1.003|0.768|
|||96|1.176|0.776|1.404|0.874|1.451|0.871|1.283|0.824|1.070|0.756|
||**1-yeargap**|192|1.495|0.880|1.547|0.889|1.580|0.904|1.267|0.828|1.019|0.753|
|||336|1.517|0.901|1.617|0.910|1.534|0.892|1.048|0.786|0.822|0.710|
|||96|1.220|0.796|1.286|0.813|1.598|0.940|1.249|0.809|1.153|0.790|
|**TfNSW\_gap**|**1.5-yeargap**|192|1.624|0.934|1.615|0.902|1.620|0.950|1.228|0.808|1.095|0.792|
|||336|1.621|0.945|1.621|0.900|1.220|0.814|1.163|0.811|1.049|0.775|
|||96|1.201|0.785|1.331|0.823|1.414|0.862|1.124|0.749|0.969|0.702|
||**2-yeargap**|192|1.287|0.831|1.061|0.726|1.109|0.755|1.058|0.722|0.941|0.737|
|||336|1.260|0.810|0.996|0.693|1.011|0.695|0.960|0.749|0.917|0.715|

- (1) For Reviewer 6qSU’s suggestion, we have provided statistics on training times for the largest and smallest datasets shown in Table 2, demonstrating that our dataset and the settings we propose can be easily trained on one V100 GPU.

**Table 2. Training time for every epoch in the largest PEMS04_gap and smallest tfNSW**
||||**Mamba**||**iTransformer**||**DLinear**||**Autoformer**||**Informer**||**FEDFormer**||**MICN**||**PatchTST**||
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|**Methods**|**Metrics**||**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|**MSE**|**MAE**|
|||96|1.480|0.845|1.236|0.822|1.166|0.748|1.320|0.879|1.354|0.777|1.377|0.895|1.219|0.778|0.946|0.778|
||**1-yeargap**|192|1.543|0.867|1.088|0.733|1.184|0.758|1.599|0.961|1.229|0.757|1.312|0.890|1.262|0.799|1.026|0.821|
|||336|1.459|0.847|1.099|0.739|1.196|0.763|1.477|0.934|1.242|0.776|1.219|0.838|1.366|0.833|0.949|0.774|
|||96|1.745|0.944|1.355|0.877|1.372|0.846|1.658|1.016|1.615|0.879|1.440|0.919|1.204|0.778|1.025|0.807|
|**tfNSW\_gap**|**1.5-yeargap**|192|1.784|0.968|1.260|0.818|1.367|0.846|1.666|0.997|1.484|0.854|1.452|0.932|1.251|0.789|1.058|0.823|
|||336|1.628|0.939|1.284|0.835|1.385|0.854|1.515|0.945|1.500|0.883|1.346|0.871|1.203|0.770|0.995|0.788|
|||96|1.364|0.857|1.019|0.782|1.008|0.742|1.319|0.907|1.101|0.713|1.318|0.879|1.033|0.745|0.848|0.727|
||**2-yeargap**|192|1.195|0.783|0.884|0.701|1.024|0.749|1.319|0.891|1.054|0.721|1.222|0.855|1.094|0.737|0.909|0.757|
|||336|1.588|0.903|0.907|0.730|1.166|0.748|1.233|0.849|1.787|0.639|1.056|0.766|0.959|0.691|0.865|0.714|
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533775352

thread_type:
official_comment

<thread_object>
creator: area_chair_fu7j

<content>
title:
Acknowledge the author responses

comment:
Dear Reviewers,

Thank you very much for your effort. As the discussion period is coming to an end, please acknowledge the author responses and adjust the rating if necessary.

Sincerely,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797425387

thread_type:
official_comment

<thread_object>
creator: area_chair_fu7j

<content>
title:
Discussion needed

comment:
Dear Reviewers,

As you are aware, the discussion period has been extended until December 2. Therefore, I strongly urge you to participate in the discussion as soon as possible if you have not yet had the opportunity to read the authors' response and engage in a discussion with them. Thank you very much.

Sincerely,
Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents XXLTraffic with the longest timespan collected from Los Angeles, USA, and New South Wales, Australia, curated to support research in extremely long forecasting beyond test adaptation. The reviewers and I believe that providing this benchmark is a very meaningful contribution to the research community.  However, as some reviewers pointed out, the benchmark itself should have enough novelty to open or support a new research problem.   The reviewers feel that increasing the temporal range of the benchmark dataset may not be sufficient for that perspective.  I also agree with the reviewers' opinions.  When I submitted a benchmark paper to NeurIPS Datasets & Benchmarks track, I have often received similar comments.  Overall, considering the reviewers' opinions, I am sorry to recommend a reject.