Event Traffic Forecasting with Sparse Multimodal Data

Authors: zhenduo  zhang
yiling  wu
xinfeng  zhang
xiao  han
zhe 2024 wu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 16:35:28

Abstract: With the development of deep learning, traffic forecasting technology has made significant progress and is being applied in many practical scenarios. However, various events held in cities, such as sporting events, exhibitions, concerts, etc., have a significant impact on traffic patterns of surrounding areas, causing current advanced prediction models to fail in this case. In this paper, to broaden the applicable scenarios of traffic forecasting, we focus on modeling the impact of events on traffic patterns and propose an event traffic forecasting problem with multimodal inputs. We outline the main challenges of this problem: diversity and sparsity of events, as well as insufficient data. To address these issues, we first use textual modal data containing rich semantics to describe the diverse characteristics of events. Then, we propose a simple yet effective multi-modal event traffic forecasting model that uses pre-trained text and traffic encoders to extract the embeddings and fuses the two embeddings for prediction. Encoders pre-trained on large-scale data have powerful generalization abilities to cope with the challenge of sparse data. Next, we design an efficient large language model-based event description text generation pipeline to build multi-modal event traffic forecasting datasets, ShenzhenCEC and SuzhouIEC. Experiments on two real-world datasets show that our method achieves state-of-the-art performance compared with eight baselines, reducing mean absolute error during the event peak period by 4.26\%. Code is available at: https://github.com/2448845600/EventTrafficForecasting.

Main paper: 
 introduction: 
 Traffic forecasting plays a crucial role in Intelligent Transportation Systems (ITS), which use historical traffic signals from sensors to predict future traffic signals. Traditional statistics-based methods, such as exponential smoothing [19] and autoregressive integrated moving average [37] , have some limitations due to their reliance on stationarity-related assumptions and disregard for nonlinear relationships between traffic signals. Recent research based on deep learning captures traffic patterns from both temporal and spatial perspectives, using TCN-based [26, 39] , RNN-based [3, 22, 32] , and attention-based [11, 12, 16, 17] modules to model the temporal correlation and GCN-based modules [9, 21, 31, 40] to model the spatial correlation between traffic nodes. Currently, traffic forecasting has made substantial advancements and is utilized in several downstream applications, such as traffic management, urban computing, and automated driving [2, 14, 41] .However, various human activities that occur in urban areas have impacts on the city transportation system, leading to abrupt shifts in traffic patterns. Consequently, the traffic prediction model with excellent regular performance is weak in this case as shown in Figure . 1. Events, such as sports championships, exhibits, and concerts, are typical large-scale human activities that have the characteristic of impacting wide areas and attracting large crowds, pose serious challenges to event organizers and traffic management agencies. Inaccurate estimation of the event's influence on the traffic system may result in traffic congestion in the surrounding area. In some extreme cases, it can result in safety concerns such as traffic accidents and crowd surge. Hence, this study specifically addresses the issue of event traffic forecasting, with the goal of Figure 1 : The impact of events on traffic patterns can lead to distortions in traffic prediction models. GWNet is a powerful model that works well when no events occur (regular), but weakens when events occur.modeling the influence of events on traffic patterns and effectively predicting event traffic.In this paper, we highlight that the primary challenges of event traffic forecasting research are the diversity and sparsity of events after analyzing the real-world traffic data in Shenzhen City, China. First, there are various kinds of events held in cities, including for-profit, non-profit, official, unofficial, popular, and professional ones. Due to the diversity of events, it is challenging for researchers to construct a complete set of features for uniformly representing the events' characteristics, which makes analyzing the impact of events on traffic patterns difficult. Second, the spatial and temporal distribution of events is sparse. For example, no more than half of the days are allocated for hosting events at the Shenzhen Convention and Exhibition Center. Furthermore, the amount of data is smaller during certain time ranges, such as weekdays or special holidays, bringing a huge challenge for data-driven methods to model spatialtemporal patterns.In addition to the inherent challenges presented by the data itself, the lack of enough data also greatly limits the progress of this research. Existing public traffic forecasting datasets, such as METR-LA, PEMS-BAY, and PEMS0X, only contain the time-series modal traffic signal data and the graph-style node connection matrix, without any event-related information. Since there is no official website to directly download accurate and comprehensive event information, we can only obtain it from the Internet, where useful data is scattered in every corner. Manually retrieving relevant web pages through search engines to create datasets is time-consuming and not feasible for real-world applications. We have to solve the problem of dataset construction pipeline before developing the event forecasting method.Therefore, addressing the above issues to effectively leverage multimodal data is essential for improving the performance of event traffic forecasting. To achieve this, we first propose an efficient event description text generation pipeline with large language models (LLMs). We send specially designed prompts and retrieved related web pages to the powerful LLM for generating textual descriptions for events. We use this data generation pipeline to build two multi-modal event traffic prediction dataset at the Shenzhen Convention and Exhibition Center (ShenzhenCEC) and Suzhou International Exp Centre (SuzhouIEC). Then, we design a multimodal (Text and Time-series) event Traffic forecasting model, T3, which uses pre-trained text and traffic encoders to extract embeddings of corresponding modalities for prediction. The text and traffic encoders, which are trained on a large amount of data, have strong generalization capabilities to learn the representation of sparse data without additional training. Learnable projections are used to transform embeddings into hidden features to adjust for the prediction task, followed by two multi-layer perceptrons for outputing the prediction results together from the hidden features.We summarize key contributions of this paper as follows:• We first propose the event forecasting problem in a multimodal setting, which uses event description text and historical traffic data to predict future traffic signals.• We develop an LLM-based event description text generation pipeline, which significantly reduces the cost of obtaining event information from websites. And we build two multimodal event traffic forecasting datasets. • We propose the multimodal event traffic forecasting model, T3, which uses the generalization capabilities of pre-trained encoders to obtain representations of sparse data.SuzhouIEC to gain insight into the effectiveness of the T3. Experimental results show that our proposal is able to consistently and significantly outperform all baselines and reduce the MAE by 4.26% during the event peak period. 
related work 2.1 traffic forecasting: 
 Traffic forecasting is an important type of time series forecasting. Previous studies treated the traffic forecasting problem as a pure time series prediction task and addressed it via traditional statistic-based methods, such as exponential smoothing [19] and autoregressive integrated moving average [37] . These methods rely heavily on stationarity-related assumptions and ignore the nonlinear correlations between traffic signals, which severely limits traffic forecasting's effectiveness. Recently, deep learning-based studies have been proposed to capture the complex spatial-temporal correlations in traffic signals. DCRNN [22] and STGCN [42] were the first to apply deep learning methods to traffic prediction, using GCN to capture spatial correlation and RNN or CNN to capture temporal correlation, and have made significant progress. Subsequently, more and more methods [10, 20, 44] design exquisite spatiotemporal feature extraction modules to achieve better performance. Recently, some works about time-series or traffic fundamental models [15, 29, 47] use time-series data to fine-tune pre-trained LLMs to obtain zero-shot prediction capabilities. However, it is worth noting that these studies have not utilized multi-modal data.Multimodal machine learning aims to develop models that can process and relate information from multiple modalities, such as image, video, audio, and 3D [4] . Existing multi-modal learning research mainly focuses on vision, audio, and language modal, and a series of research works have emerged, such as SUR-adapter [46] , RTQ [36] , HAT [5] , PromptMTopic [28] , and so on [23] . And there are also some papers focus on the 3D point cloud, table, source code, graph, etc [6, 13, 33, 35] . There are many core technical challenges The output dimension of text encoder𝐷 trafficThe output dimension of traffic encoder 𝐷The hidden dimension of the T3 model surrounding multimodal machine learning, and we focus on multimodal representation and fusion in this paper. Representation involves learning how to represent and summarize multimodal data using complementarity and redundancy of multiple modalities. Pre-trained encoders [7, 18, 38] are commonly used to learn representations of inputs from relevant modalities. Fusion involves joining information from two or more modalities to perform a prediction. There are many model-agnostic approaches and modelbased approaches [1] . With the development of large language models, multimodal large models are also attracting more and more attention, such as BLIP-2 [43] , LLaVA [25] , and GPT4V. 
preliminaries: 
 In this section, we define the notions of traffic sensor, network, signal, and the event traffic forecasting problem. Table 1 shows the frequently used notation. Traffic Sensor. A traffic sensor is a sensor deployed in a traffic system that records traffic states such as the flow or speed of passing vehicles.Traffic Network. A traffic network is defined as a directed or undirected graph with the formula G = (𝑆, 𝐸), where 𝑆 is the set of |𝑆 | = 𝑁 traffic nodes and 𝐸 is the set of |𝐸| = 𝑀 edges.Traffic Signal. The traffic signal 𝑋 𝑡 ∈ R 𝑁 denotes the observation of all sensors in the traffic network G at time step 𝑡. In this paper, the traffic signals generally means the traffic speed.Event Description Text. Event description text T is a text that describes the main contents of the events in a specific time period.Event Traffic Forecasting. Given historical traffic signalX = [𝑋 𝑡 -𝑇 ℎ , 𝑋 𝑡 -𝑇 ℎ +1 , • • • , 𝑋 𝑡 -1 ] ∈ R 𝑇 ℎ ×𝑁from the past 𝑇 ℎ time steps, traffic network G and event description text T , the event traffic forecasting can be formed as:Ŷ = F (X, G, T ),where Y ∈ R 𝑇 𝑓 ×𝑁 are the nearest future 𝑇 𝑓 time steps of traffic signal, and F is the forecasting model. T can be ∅ when there is no event occurring, and this situation is equivalent to the classical traffic forecasting problem.  
dataset: 
 In this section, we will introduce in detail the construction process and analysis conclusions of the multi-modal event traffic forecasting datasets, using ShenzhenCEC as an example.Urban areas often hold a diverse range of large-scale events, including exhibitions, concerts, sports matches, charity galas, etc. These events attract large numbers of people and have a significant impact on the surrounding traffic pattern, making it different from the normal state. We try to collect comprehensive information about these events to enhance the performance of event traffic prediction. However, there is currently no official and unified way to obtain event information, and comprehensive event details are scattered in every corner of the Internet. We find that some urban venues' official websites include historical event schedule information. Nevertheless, this tabular data is limited to the event's name and period, lacking comprehensive details such as the event's introduction, theme, anticipated viewership, and so on. Although researchers can use search engines (like Google) to retrieve relevant data from massive web pages and manually construct a toy dataset, this data collection approach is time-consuming and labor-intensive, making it challenging to apply in real-world scenarios. Hence, we develop an event description text generation pipeline based on the advanced large language model as depicted in Figure . 2. There are four steps to generate a text for the event. First, event scheduling data is downloaded from the official website of venues (such as the homepage of Shenzhen Convention and Exhibition Center1 ), which only includes the name and time of historical events. Then, we create a prompt that adequately describes the requirements of our goal, where the prompt should contain the name and time of the event to alleviate hallucinations of LLM's. Next, we crawl event-related webpages from the Internet (or you can use similar tools provided by commercial services directly) as the raw material for LLMs to generate text. Finally, we send both the prompt and web data to a LLM to generate the event description text. This retrieval augmented generation (RAG) pipeline can automatically retrieve event-related information from the Internet and intelligently extract text summaries. Compared with the manual pipeline, our proposed solution is more convenient, efficient, and low-cost, which can be extended to real-life scenarios.It is crucial to acknowledge that the data generated by the aforementioned pipeline may be at data leakage risk for the forecasting task, meaning that information about upcoming incidents or emergencies may be included in the event description text. This potential risk can be resolved by limiting the publication time of crawled web pages, making the LLM only obtain web data published before the event starts. Another issue is the hallucination of LLM. Although several prompt engineering tricks, such as system prompts and few-shot prompts, can significantly reduce the occurrence of hallucinations, it is nearly impossible to entirely eliminate this problem. In order to ensure the quality of the data, we manually check all event description texts to ensure that there is no information leakage or obvious hallucinatory context. Following the above pipeline, we generate description texts for all events held at the Shenzhen Convention and Exhibition Center between the dates of February 1st, 2023, and June 5th, 2023.We select the Shenzhen Convention and Exhibition Center (Shen-zhenCEC) to build the dataset because of its high frequency of large-scale events. The statistical information is shown in Table . 2.The traffic speed data is collected at regular intervals of 10 minutes within a radius of 1000 meters surrounding the ShenzhenCEC. The time range corresponds to the duration of the event data, spanning a total of 125 days from February 1st, 2023, to June 5th, 2023.The traffic graph is constructed based on the node distances. We calculate the distance between nodes and only connect node pairs that are within a distance of less than 10 meters. The traffic data consists of a total of 18,000 time steps, 742 traffic nodes (road links), and 1277 edges. The Figure . 3 illustrates the temporal distribution of events at the ShenzhenCEC over a span of 125 days. This location hosted a total of 34 events, which were distributed across 44 days, accounting for 35.2% of the total days. It is important to note that events can last over multiple days, and it is possible for multiple events to occur within a single day. A single day can have a maximum of 5 events, and a single event can last up to 4 days in our dataset. The occurrence of events is observed on 28 weekdays and 16 weekends.  
model structure: 
 Frozen Text Encoder. We use the pre-trained text embedding model as the text encoder to obtain the text representations for event traffic forecasting. Text embedding [34] is a commonly used NLP technique that converts text data into fixed-dimensional vectors that can be processed by machine learning/deep learning algorithms. These vector representations are designed to capture the semantics and context of the words they represent. We combine the texts of multiple events that occurred in a day as the daily event description text T and send it into the text encoder to get text embedding emb text ∈ R 𝐵×𝐷 text :EQUATIONWe freeze the text encoder parameters in the training stage to keep the powerful generalization ability learned from massive text training data for sparse event text embedding.Frozen Traffic Encoder. We use the encoder layers of the traffic forecasting model, which is pre-trained on traffic data from the traffic dataset, as the traffic encoder. Common traffic forecasting models use time-series data (history traffic signals) and graph-structure data (traffic network) as inputs. These methods often employ space operations, such as graph convolutional networks (GCN) and spatial attention, to capture the spatial correlation among traffic nodes, and sequence operations, such as TCN, attention, and recurrent neural networks (RNN), to capture the temporal correlation. Our proposed T3 uses the spatiotemporal modules as the traffic encoder to extract traffic embedding emb traffic ∈ R 𝐵×𝑁 ×𝐷 traffic :EQUATIONProject, Expand, and Concat Embeddings. Although the pre-trained frozen encoders have strong generalization capabilities to extract sparse text and traffic embeddings, it is necessary for our model to maximize the utilization of the training data to adjust the embeddings to fit the event traffic forecasting task. First, we use two layers MLP to project the embeddings into the hidden space independently:EQUATIONwhere the shape of H text is (𝐵, 𝐷) and the shape of H traffic is (𝐵, 𝑁 , 𝐷). Then, we extend the size of H text to (𝐵, 𝑁 , 𝐷) and concatenate the two hidden features to get multi-modal features H mm ∈ R 𝐵×𝑁 ×2𝐷 :EQUATIONPredictor and Loss. We apply regression layers on H mm and H traffic respectively, and add their output results as the final prediction result:EQUATIONwhere FC(•) is fully connection layer and 𝜃 (•) is activation function.In the training stage, we use Mean Absolute Error (MAE) as loss function: [22] , which uses the traffic signals of the history 𝑇 𝑓 = 12 time steps to predict the next 𝑇 ℎ = 12 steps. Event peak hours are 7 a.m. to 10 a.m. and 5 p.m. to 8 p.m. during days with events.EQUATION)6.1.2 Metrics. We evaluate the performances of all baselines by four commonly used metrics in traffic forecasting, including mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and weight mean absolute percentage error (WAPE). MAE reflects prediction accuracy; RMSE is more sensitive to abnormal values; MAPE can eliminate the influence of data units to some extent; and WAPE is more robust to outliers compared with MAPE.6.1.3 Implementation Details. All traffic forecasting models are implemented using pytorch 1.10 and cudn 11.3. Models are trained and evaluated on 12th Gen Intel(R) Core(TM) i9-12900K, 128 GB RAM computing server equipped with two RTX 4090 GPUs. We employ Adam with a learning rate of 0.002 as our optimizer. We use the multistep learning rate scheduling strategy and set the decay ratio to 0.5. All models are repeated three times using fixed seed 0, 1, and 2. We use voyage-2 to get text embeddings with 𝐷 text = 1024 and GWNet to get traffic embeddings with 𝐷 traffic = 256. Other implementation details can be seen in the code.6.1.4 Baselines. We compare our proposed T3 model with the eight baseline models:• HI [8] : Historical Inertia simply uses the most recent data points in the input time series as predictions.• NLinear [45] : NLinear is a MLP-based time series forecasting model that decomposes the time series into a trend and a remainder series and employs two parallel linear layers to predict these two series. • PatchTST [27] : PatchTST is a transformer-based time-series forecasting model, which divides the input time series data into subseries-level patches which are served as input tokens to Transformer. • DCRNN [22] : Diffusion convolution recurrent neural network is an encoder-decoder structure network that combines graph convolution networks with RNN. • GWNet [39] : Graph WaveNet jointly captures spatial and temporal dependencies through the sequential integration of temporal convolution layers and graph convolutional layers. • D2STGNN [32] : This method integrates deep learning technology and signal processing theory to enhance the accuracy and efficiency of predictions.• STID [30] : STID uses learnable temporal embeddings and node embeddings to model spatialtemporal correlation. This simple yet efficient component allows it to achieve performance comparable to SOTA through multiple MLP layers. • STAEformer [24] : Spatiotemporal adaptive embedding transformer proposes a novel component called spatiotemporal adaptive embedding that can yield outstanding results with vanilla transformers.Among these methods, HI, NLinear, and PatchTST are time series prediction models, and the others are traffic prediction models. 
performance comparison: 
 To verify the generality and performance of our proposed T3 model, we compare it with eight baselines. Table . 3 shows the interval estimate results of average forecasting performance. The HI algorithm has the worst performance because it cannot capture complex traffic patterns in both time and space dimensions. Although NLinear and PatchTST are widely used in the time series forecasting task, they are significantly inferior to traffic forecasting methods due to their lack of particular modules for learning spatial correlation inside the traffic network. Among the five traffic forecasting baselines, GWNet demonstrates almost the best performance, which may be attributed to its gated structure that can learn time-varying changes in traffic speed, while STAEformer achieves the second highest ranking results, possibly contributed by its transformer structure that can learn complex traffic patterns. Compared with the above eight baseline methods, our proposed T3 model gets state-of-the-art performance across all metrics, especially during peak hours (7 a.m. to 10 a.m. and 5 p.m. to 8 p.m.) on event days, achieving a reduction in MAE of 4.26%. Furthermore, Table . 4 shows in detail the prediction performance of all methods at the third (half an hour), sixth (an hour), and twelfth (two hours) time steps. Our method demonstrates strong performance across all three time steps. In summary, our method can consistently and significantly outperform eight strong baselines and reduce the MAE by 4.26% during the event peak period.We conduct two groups of experiments and text embedding visualization to verify the effectiveness of text embedding for the event traffic forecasting task. First, we discuss whether multimodal data is necessary for the event traffic forecasting problem. We conduct a comparison between the GWNet employing handcrafted event features and our proposed approach utilizing text embeddings. The original inputs of GWNet have three kinds of features: traffic speed, time of day, and day of week. We design two extra event features: one is whether an event occurs (EO), and the other is the daily events number (EN). Experimental results in Table . 5 show that the EO feature leads to a slight decrease in prediction error during the event peak hours, while the EN feature results in an increase in prediction error throughout both periods. We hypothesize that the EN feature harms model performance because some unique values are absent in the training set but present in the test set. For example, the extreme case of five events in a single day only appears in the test set. Therefore, we can conclude that hand-crafted features struggle to represent the rich context of sparse events, and textual data is necessary for modeling the impact of events on traffic speeds. Then, we discuss whether the performance improvement comes from the semantic information of text embeddings rather than network structure. We replace the text embeddings (TE) that are output by the text encoder with two kinds of initialized embeddings: one is a random initial embedding (RE), and the other is a Gaussian initial embedding that has the same distribution as TE, namely GE. According to experimental results in Table . 7, TE presents superior performance compared to RE and GE. We can infer that the pretrained language model can extract information from event text to improve the performance of forecasting. Finally, we use t-SNE technology to reduce high-dimensional text embeddings to twodimensional formats for visual analysis. As shown in Figure . 6, there are some clusters between text embeddings, which demonstrates that the similarity between events is captured by text embeddings. Similar events will have similar impacts on traffic patterns, and our method obtains the ability to predict event traffic by learning these correlations from historical data. We conduct the ablation study and hyperparameter study about T3. In Table . 6, w/o MMP refers to T3 without the multimodal predictor, and wo TP refers to T3 without the traffic predictor. The experimental results indicate that combining two prediction heads yields the best results. We speculate that two predictors can be more effective at learning hidden patterns between events and traffic from sparse data. In addition, Table . 3 presents the impact of different hidden dimensions 𝐷 on forecasting performance, and the model achieves the best performance when 𝐷 is set to 256. 
conclusion: 
 In this paper, we propose the event traffic forecasting problem, which focuses on modeling the impact of events on traffic patterns with multimodal inputs. First, we provide an effective pipeline for generating event text using a large language model and build a multimodal event traffic dataset to describe the diversity of events. Then, to tackle the sparsity of collected data, we propose the multimodal event traffic prediction model, which uses pre-trained text and traffic embedding models to extract embeddings of corresponding modalities for prediction. Finally, we conduct experiments on the real-world dataset, and results show that our proposed method achieves state-of-the-art performance against all baselines. 
 
 Appendix: https://www.ShenzhenCEC.com/ 


Decision: The authors propose a multi-modal event traffic forecasting model that integrates textual and traffic data. The model was initially evaluated on a single dataset from Shenzhen City and this was a limitation of the work as highlighted by a couple of the reviewers. All the reviewers agree that the idea is very valuable, the work done is innovative and the integration of the two sources of information is well designed. Reading the rebuttal, I think the main objections of reviewer "WQWg" are dealt by the authors, in particular the issue related to the ablation of some components of the framework. For sure, the final version of the work should report the results on the two datasets, the current one and the one the authors have reported initial results in the rebuttal.

Reviews: review: paper_summary: paper_summary: This work entitled as "Event Traffic Forecasting with Sparse Multimodal Data" propose to leverage large language models (LLMs) to assist in the spatial-temporal analysis of traffic patterns. The key innovation lies in the integration of textual data with traditional traffic sensor data to predict traffic conditions during events such as sports games, concerts, and exhibitions. The authors utilize pre-trained LLMs to extract semantic information from event descriptions and fuse this information with traffic sensor data using a gated convolutional neural network. This approach addresses the challenge of sparse multimodal data and the nonlinear correlations between event-related traffic changes. The model is evaluated on a real-world dataset from Shenzhen City, demonstrating its effectiveness in capturing the impact of events on traffic patterns. The experimental results show that the proposed model outperforms baseline models.
strength_weakness: strength_weakness: 1. Novel Integration of LLMs on STG Learning: The paper effectively integrates large language models with traffic forecasting, demonstrating the potential of using semantic information from text data to enhance spatial-temporal predictions.
2. Addressing Sparse Data Challenge: The proposed method tackles the challenge of sparse multimodal data, providing a robust solution for scenarios where traditional data sources are insufficient.
3. Real-World Application: The model is tested on real-world data from Shenzhen City, showcasing its practical applicability and effectiveness in a real-world context.
4. Performance Improvement: The experimental results highlight significant improvements in prediction accuracy compared to baseline models, validating the proposed approach.
5. Detailed Methodology: The paper provides a comprehensive description of the model architecture and the experimental setup, ensuring transparency and reproducibility.
limitations: limitations: 1. Limited Dataset Diversity: The evaluation is conducted on a single dataset from Shenzhen City, which may limit the generalizability of the results to other geographical locations or types of events.
2. Complexity of Implementation: The integration of LLMs and the proposed model architecture may require substantial computational resources and expertise, which could limit its adoption in resource-constrained environments.
paper_summary: paper_summary: This paper focuses on the problem of urban traffic forecasting under large events such as sport games, exibitions, concerts, etc. Existing traffic forecasting methods cannot fully consider the influence of these events thus having suboptimal performance in these cases. This paper proposed to solve this problem by utilizing descriptional text of events as additional information to help deep learning model learn to forecast better under events. To this end, this paper developed an LLM-based event descriptional text generation pipeline, and built a dataset, known as SZCEC, for evaluation of urban traffic forecasting methods under large events. This paper proposed a multimodal event traffic forecasting model, known as T3, that considers both traffic signal and event descriptional text for event traffic forecasting. They conduct experiments on the proposed dataset and showed good performance of T3.
strength_weakness: strength_weakness: 1. The problem studied in this paper is important which can indeed broaden the application of traffic forecasting and satisfy real world needs.
2. It is novel to utilize LLMs to generate uniform descriptional text of events and use the texts to provide event information in a multi-modal traffic forecasting.
3. The dataset built by the authors is a contribution to the related research community and can help future research of multi-modal modeling for traffic forecasting and smart cities.
limitations: limitations: 1. It would be better if the related work section can discuss more on how is the event information is utilized in the existing methods, and maybe, how LLM-generated texts are utilized in other similar multi-model learning tasks.
1. The proposed SZCEC dataset only considers the events held at one exibition center in the city, but for large cities, there can be other events happends elsewhere in the city at the same time that also affect the traffic signal.
2. It is not clearly specified how the event text descriptions are generated, e.g., what LLM and what prompt are used?
3. Some designs in T3 are not well motivated, e.g., why the traffic encoder needs to be frozen, can it be trained end-to-end from scratch, or fine-tuned during the training process described in Figure 5?
4. The experiments mainly compared the proposed T3 model which uses both traffic signal and event information as input, with baselines only using the traffic signal, which is not so fair. Although the experiment in Section 6.3 compared T3 with GWNet with handcrafted event features, it is not comprehensive enough. It would be better to compare with all the baselines that also have certain event information, such as handcrafted event features, as input.
5. This paper needs further proofreading to avoid typos and grammar mistakes.
paper_summary: paper_summary: The authors address the event traffic forecasting problem, focusing on modeling traffic patterns influenced by events. Leveraging the rapid advancements in large language models (LLMs), this paper constructs a multimodal event traffic dataset incorporating event text generated by LLMs. To tackle this problem, the authors introduce a novel model designed to handle multimodal information, achieving state-of-the-art (SOTA) performance across three different forecasting horizons.
strength_weakness: strength_weakness: 1. The paper is well-organized and easy to follow, with clear figures and a comprehensive description of the implementation details.

2. This work introduces a new problem in the traffic forecasting domain. To my knowledge, it is the first to provide a multimodal event traffic forecasting dataset, which is highly valuable for the field. The authors reasonably use large language models to generate formatted descriptions of events.

3. The baselines presented in this paper, including PatchTST, D2STGNN, and STAEformer, are high-quality and timely, setting a high standard for future research.
limitations: limitations: 1. Table 3 appears to be redundant, especially since Table 4 provides sufficient details on the results of T3.

2. Although the superior performance of the proposed T3 model is reasonable due to its integration of additional event information and raw time series representation, T3 does not show a significant performance advantage over other baselines. For example, GWNet and STAEformer exhibit similar performance. Given that SZCEC is a dataset specifically designed for event forecasting, T3 should demonstrate a more substantial performance improvement.

3. In Figure 6, the t-SNE analysis shows the event embeddings clustered into various groups. However, the authors should clarify whether all events have been plotted. The figure shows 34 dots, while there should be 32 events in SZCEC according to the authors.

4. The proposed SZCEC dataset is one of the main novelties of this work. Therefore, more analysis on it is expected. For instance, the authors could further explore the relevance among events within each cluster, given they have already illustrated the visualization of event embeddings.

5.  More recent works could be cited or selected as convincing baselines like HimNet[1],PGCN[2], and MoST[3].
[1] Dong, Zheng, et al. "Heterogeneity-Informed Meta-Parameter Learning for Spatiotemporal Time Series Forecasting." arXiv preprint arXiv:2405.10800 (2024).
[2] Shin, Yuyol, and Yoonjin Yoon. "PGCN: Progressive graph convolutional networks for spatial–temporal traffic forecasting." IEEE Transactions on Intelligent Transportation Systems (2024).
[3] Deng, Jiewen, et al. "Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning." arXiv preprint arXiv:2405.03255 (2024).

In conclusion, I believe this paper is marginally above the acceptance standard for being the first to propose a multimodal event time series dataset. While the performance of the proposed model needs more detailed description and experimentation, the authors provide a novel method for building multimodal time series event datasets.
paper_summary: paper_summary: The paper focuses on the problem of event traffic forecasting, which aims to model the impact of various events on traffic patterns in urban areas. The authors highlight the key challenges in this problem, including the diversity and sparsity of events, as well as the lack of sufficient data. To address these challenges, the paper proposes a multi-modal event traffic forecasting model that uses pre-trained text and traffic encoders to extract embeddings from event descriptions and traffic data, and then fuses them for prediction. Additionally, the authors design an efficient LLM-based event description text generation pipeline to build a multi-modal event traffic forecasting dataset called SZCEC. Experimental results on the SZCEC dataset show the proposed method achieves state-of-the-art performance compared to baselines, reducing mean absolute error during event peak periods by 4.26%.
strength_weakness: strength_weakness: 1) The paper tackles the novel problem of event traffic forecasting, which is an important but understudied problem in the field of traffic forecasting. The authors identify the unique challenges in this problem and propose a multi-modal approach to address them.
2) The paper Designs an efficient LLM-based event description text generation pipeline to build a new multi-modal event traffic forecasting dataset（SZCEC）, which is valuable for the research community. The use of an LLM-based text generation pipeline to augment the dataset is an innovative approach.
3) The paper provides a thorough evaluation of the proposed method on the SZCEC dataset, comparing it with several baselines. The results demonstrate the effectiveness of the proposed approach, especially during the event peak periods.
limitations: limitations: 1) The evaluation is limited to a single dataset, and more comprehensive testing on other real-world event traffic datasets would be needed to better assess the generalizability of the approach.
2) The paper does not provide ablation studies to understand the individual contributions of the different components (e.g., text encoder, traffic encoder) to the overall performance. Such an analysis would help to better understand the strengths and weaknesses of the proposed model.
3) The paper does not provide extensive validation or evaluation of the quality and realism of the event description texts generated by the LLM-based pipeline.