A Large-scale Dataset and Benchmark for Commuting Origin-Destination Flow Generation

Authors: Can Rong, Jingtao Ding, Yan Liu, Yong Li
Primary area: datasets and benchmarks
Keywords: Commuting, origin-destination flow dataset, urban computing, weighted graph modeling

Abstract:
Commuting Origin-Destination~(OD) flows are critical inputs for urban planning and transportation, providing crucial information about the population residing in one region and working in another within an interested area. Due to the high cost of data collection, researchers have developed physical and computational models to generate commuting OD flows using readily available urban attributes, such as sociodemographics and points of interest, for cities lacking historical OD flows \textemdash commuting OD flow generation. Existing works developed models based on different techniques and achieved improvement on different datasets with different evaluation metrics, which hinderes establishing a unified standard for comparing model performance. To bridge this gap, we introduce a large-scale dataset containing commuting OD flows for 3,333 areas including a wide range of urban environments around the United States. Based on that, we benchmark widely used models for commuting OD flow generation. We surprisingly find that the network-based generative models achieve the optimal performance in terms of both precision and generalization ability, which may inspire new research directions of graph generative modeling in this field. The dataset and benchmark are available at https://anonymous.4open.science/r/CommutingODGen-Dataset-0D4C/.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730533225132

thread_type:
official_review

<thread_object>
creator: reviewer_nxhm

<content>
summary:
This paper introduces a large-scale dataset, LargeCommutingOD, which captures commuting origin-destination (OD) flows for 3,333 diverse regions across the United States, spanning 9,372,610 square kilometers and encompassing a variety of urban environments. Compared to existing datasets, LargeCommutingOD offers broader and more comprehensive coverage and is made publicly available. The authors benchmark several baseline models using a standardized evaluation protocol, uncovering that network-based generative models could be a promising avenue for future research in this domain.

strengths:
1. The authors present a large-scale and comprehensive dataset, LargeCommutingOD, which includes commuting OD flows for 3,333 regions, covering 9,372,610 square kilometers across diverse urban environments in the United States. This dataset provides more extensive coverage than existing resources and is openly accessible, promoting transparency and reproducibility.
2. By comparing LargeCommutingOD to previous datasets, the authors demonstrate that it captures a broader range of urban characteristics and types, further supporting its utility for various research purposes.
3. The authors implement and evaluate several baseline models to demonstrate the feasibility and relevance of their dataset. Notably, they identify that network-based generative models show considerable potential, which opens avenues for further investigation.
4. Upon reviewing the proposed code, I observed that the authors have implemented a comprehensive set of baselines, which could even serve as a robust library for OD flow generation. I hope the authors can commit to releasing this code if the paper is accpeted.

weaknesses:
1. Given the dataset's coverage of 9,372,610 square kilometers and its focus on approximately 3,000 regions, the OD metrics may primarily reflect macro-level commuting patterns rather than fine-grained dynamics within individual cities. This limitation may restrict the dataset's applicability for studies requiring detailed intra-city analyses.
2. The OD matrix, denoted as $\mathbf{F} \in \mathbb{R}^{N \times N}$ where $N$ exceeds 3,000, may be sparse. Providing a visual representation, such as a heatmap of the OD matrix, would clarify its density and structure, which could help readers better understand the data distribution.
3. The authors included a non-anonymized GitHub link (https://github.com/tsinghuafib-lab/CommutingODGen-Dataset) in the submission, potentially compromising anonymity. And the authors have specified that only their names need to be anonymized.

questions:
Please explain the questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631916665

thread_type:
official_review

<thread_object>
creator: reviewer_rw5x

<content>
summary:
This paper constructs a large-scale dataset (LargeCommuingOD) containing commuting OD flows for 3,333 diverse areas around the United States covering 9,372,610 km2 including a wide range of urban environments. Based on the LargeCommuingOD, the authors benchmark the existing widely used models for commuting ODflow generation. These efforts make up for the lack of comprehensive dataset and the absence of unified and systematic evaluation. Furthermore, this paper find out that network-based modeling for commuting OD flow supported by their dataset gives a promising performance. Thus, training on a large number of commuting OD networks can help the generative models to capture universal as well as distinct mobility patterns at city level, and therefore enhance the generalization ability.

strengths:
S1. This paper constructs dataset covers 3,333 areas around the United States, providing a much broader spatial scale comparing to existed datasets.
S2. The dataset constructed in this paper covers metropolitan areas, towns, and rual areas, which is more comprehensive then existing datasets the focuses on usually a single type of urban environment.
S3. Based on the LargeCommuingOD, the authors benchmark the existing widely used models for commuting ODflow generation, which makes up for the absence of unified and systematic evaluation.

weaknesses:
W1. Please clarify whether WEDAN is a newly proposed model, or the idea of it has already been proposed somewhere else.
W2. In Sec. 2.2, the link of the released dataset is given in plaintext, which may violate the anonymous policy.

questions:
Please refer to Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703787924

thread_type:
official_review

<thread_object>
creator: reviewer_kbsu

<content>
summary:
Commuting OD flows are critical inputs for urban planning and transportation, while the high data collection cost results in a lack of high-quality datasets. This paper introduces a large-scale dataset containing commuting OD flows for 3,333 areas including a wide range of urban environments around the United States. Based on this dataset, authors further benchmark widely used models for commuting OD flow generation. They find that, owing to the rich information contained in the constructed OD dataset, a generative method that considers each OD matrix collected from a county/city as a network and learns to generate it given conditional information can significantly outperform other methods. This may point to a new direction in this domain, which relies on collection of high-quality OD datasets.

strengths:
+ Commuting OD data is valuable in building smart cities, while the data is generally lacked in many areas around the world. The synthetic data generation technique is a promising solution for this problem, requiring high-quality and high-volume data for training ML models. This paper makes a good contribution towards this objective.
+ According to Table 1, the constructed dataset is more useful than existing ones, in terms of size and richness. The data analysis is sufficient, as in Figure 2-4.
+ The benchmark is designed in a reasonable manner. It covers a wide range of mainstreaming approaches in this field, and evaluate the model performance in a comprehensive way, including both flow generation accuracy and property distribution similarity.
+ The observation based on benchmark results is insightful. Based on the constructed large-scale OD dataset that is unseen in previous works, authors demonstrate the powerfulness of graph generative modeling in terms of generating OD flows for diverse cities. This may point to a new direction in this domain.

weaknesses:
+ The presented results are informative, but the explanations are rather insufficient. For example, Figures 6 and 7 should be explained in detail.
+ Although authors discuss the limitations of the constructed dataset, the spatial scale (limited to the United States) and temporal scale (one year) can degrade its usefulness in applications.

questions:
+ Can authors demonstrate that models developed based on this US dataset can manage to transfer to other countries?
+ In Figure 1, why Commuting Flows are processed to generate Regional Socio-demographics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731053265823

thread_type:
official_review

<thread_object>
creator: reviewer_zri5

<content>
summary:
This paper introduces a large-scale dataset called LargeCommuingOD for commuting Origin-Destination (OD) flow generation and provides a benchmark for evaluating models in this domain. The key contributions are:

1. Construction of a comprehensive dataset covering 3,333 diverse areas across the United States, including metropolitan areas, towns, and rural regions. The dataset spans 9,372,610 km² and contains commuting OD flows, sociodemographic data, and point-of-interest distributions for each area.

2. Development of a benchmark framework to evaluate and compare different commuting OD flow generation models, addressing the lack of standardized evaluation in existing research.

3. Benchmarking of 9 existing models, including physical models, classical machine learning approaches, and graph neural network models, using the new dataset.

4. Introduction of a preliminary adaptation of graph diffusion models called WEDAN (Weighted Edges Diffusion condition on Attributed Nodes) for commuting OD flow generation.

5. Analysis of model performance in terms of precision and generalizability across different urban environments.

The paper demonstrates that network-based generative models, particularly those leveraging graph diffusion techniques, achieve the best performance in both precision and generalization ability. This finding suggests new research directions in graph generative modeling for commuting OD flow generation.

The authors argue that their dataset and benchmark provide a valuable resource for researchers in urban planning, transportation, and related fields, enabling more comprehensive evaluation and development of commuting OD flow generation models.

strengths:
Originality:
1. The creation of a large-scale, comprehensive dataset (LargeCommuingOD) for commuting Origin-Destination (OD) flow generation covering 3,333 diverse areas across the United States is highly original. This dataset significantly expands on previous efforts in terms of scale and diversity.

2. The paper introduces a novel adaptation of graph diffusion models called WEDAN (Weighted Edges Diffusion condition on Attributed Nodes) for commuting OD flow generation, exploring a new paradigm in this field.

Quality:
1. The dataset construction process is rigorous, combining multiple reliable data sources including the U.S. Census Bureau, American Community Survey, and OpenStreetMap.

2. The analysis is thorough, examining both precision and generalizability of the models across different urban environments.

Clarity:
1. The paper is well-structured, clearly defining the problem, describing the dataset, and presenting the benchmark results.

2. Figures and tables effectively illustrate the dataset characteristics and model comparisons.

weaknesses:
1. **Limited Exploration of Model Interpretability**: The paper emphasizes model performance but lacks a thorough examination of model interpretability, particularly for the network-based generative models that achieve the best results. Understanding the reasons behind their success could offer valuable insights for urban planners and policymakers.

2. **Insufficient Exploration of Edge Cases**: The paper would benefit from a more in-depth analysis of how the models perform in extreme or atypical urban environments within the dataset. This could reveal potential limitations or areas for improvement.

3. **Lack of Discussion on Model Fairness and Bias**: Given the diverse nature of the dataset, an analysis of potential biases in the models' predictions across different urban environments (e.g., rural vs. urban, high-income vs. low-income areas) would be beneficial.

4. **Basic Benchmark Methods**: Many of the selected benchmark methods are too basic. The paper should include more advanced models based on transformers and diffusion, such as those discussed in [this paper](https://arxiv.org/abs/2411.04453) and [this one](https://arxiv.org/abs/2402.15398).

5. **Questionable Significance of Claims**: Are the claims regarding the significance of the research truly justified? Without this dataset, researchers in the field might only need about a week to conduct their own benchmarks. Additionally, there are already open-source frameworks, like LibCity, that have collected these benchmarks (see [LibCity](https://github.com/LibCity/Bigscity-LibCity)).

questions:
What do the authors see as the most promising directions for future research based on their findings? Are there specific areas where they believe the dataset and benchmark could be most impactful?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733245914419

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their valuable and constructive comments, as well as their detailed suggestions that have significantly improved the quality of our work. We are pleased that most reviewers acknowledged our **key contributions, including the introduction of the large-scale dataset LargeCommutingOD, the comprehensive benchmarking of commuting OD flow generation models, and the insights into the potential of introducing network-based generative models**.

**Contributions and Significance of the Dataset and Benchmark**

Reviewers (e.g., Reviewers Zri5, KbsU, and Rw5X) emphasized the originality and value of our dataset. We are glad the reviewers appreciated that LargeCommutingOD fills a critical gap by covering diverse urban environments across the U.S., enabling comprehensive research in commuting OD flow modeling. Additionally, as Reviewer KbsU highlighted, the benchmark offers a systematic and standardized framework for evaluating a variety of models, paving the way for more robust and generalizable techniques.

**Detailed Responses to Key Concerns**
- **Inclusion of Advanced Models (Reviewer Zri5)** We have incorporated an additional advanced model, TransFlower, into our benchmark. Results indicate that while TransFlower improves over traditional models, WEDAN still achieves superior generalization and precision thanks to the modeling from a network perspective, as shown in Table 4 of the revised manuscript.
- **Model Interpretability, Robustness, and Fairness (Reviewer Zri5)** We have conducted extensive experiments to enhance interpretability, robustness, and fairness analyses for the models in our benchmark. For interpretability, we employed SHAP and visualization from diverse perspectives to analyze model behavior; for edge cases, we evaluated model performance on extreme OD flows; and for fairness, we analyzed biases across income levels, finding that data-driven models generally performed better but still showed CPC distribution disparities. These findings underscore the need for further research in balancing performance, interpretability, and fairness.
- **Deeper dataset analysis (Reviewers NxhM)** For data granularity, we have clarified the dataset’s spatial division and accepted the suggestion to add the limitation regarding the lack of fine-grained dynamics beyond commuting into the manuscript. For sparsity, we visualize OD matrices using heatmaps and analyze variations across different city types, incorporating these insights as Figure 8 in Appendix A.1.
- **Deeper Model Analysis (Reviewer KbsU)** We provided detailed explanations for model performance analyses shown in Figures 6 and 7, highlighting how city size and structure affect model performance (Appendix C.5 and C.6).
- **Global Transferability (Reviewers KbsU)** To explore whether models trained on our comprehensive dataset have cross-country transferability, we conducted experiments using U.K. data. Results showed that models trained on U.S. data exhibit reasonable performance, and these insights are detailed in Appendix C.4.

We hope these updates address the reviewers' concerns and further clarify the significance of our contributions. Once again, we extend our heartfelt gratitude to all reviewers for their insightful feedback and constructive suggestions. 

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a large-scale dataset called LargeCommuingOD for commuting Origin-Destination (OD) flow generation and provides a benchmark for evaluating models. All the reviewers think this is an important work in this domain. The paper is mostly well written and clearly presented. The reviewers also provided some comments for the authors to further improve the paper. The authors should follow the comments and make modifications accordingly in the camera ready version.