Venue: NAACL 2025

# How LLMs React to Industrial Spatio-Temporal Data? Assessing

Hallucination with a Novel Traffic Incident Benchmark Dataset

Qiang Li, Mingkun Tanb, Dan Zhangc,* Daoan Zhangd, Xun Zhaoe, Porawit Kamnoedboonf, Shengzhao Leii, Lujun Lih, S.H. Chui\({}^{\text{a}}\)RWTH Aachen, bUniversity of Munster, \({}^{\text{Kami}}\)Technology Co..Ltd, dUniversity of Rochester eSoutheast University, fUniversity of Zurich, EEPFL, hHKUST, iColumbia University

###### Abstract

Large language models (LLMs) hold revolutionary potential to digitize and enhance the Health & Public Services (H&PS) industry. Despite their advanced linguistic abilities, concerns about accuracy, stability, and traceability still persist, especially in high-stakes areas such as transportation systems. Moreover, the predominance of English in LLM development raises questions about how they perform in non-English contexts. This study, originating from a real world industrial GenAI application, introduces a novel cross-lingual benchmark dataset comprising nearly 99,869 real traffic incident records from Vienna (2013-2023) to assess the robustness of state-of-the-art LLMs (\(\geq\) 9) in the spatial and temporal domains for traffic incident classification. We then explored three hypotheses -- sentence indexing, date-to-text conversion, and German-to-English translation -- and incorporated Retrieval Augmented Generation (RAG) to further examine LLM hallucinations in both spatial and temporal domains. Our experiments reveal significant performance disparities in the spatio-temporal domain and demonstrate the types of hallucinations that RAG can mitigate and how it achieves this. We also provide open access to our H&PS traffic incident dataset, with the project demo and code available at Website [https://sites.google.com/view/llmhallucination/home](https://sites.google.com/view/llmhallucination/home).

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 36–53

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 36–53

## 1 Introduction

Large Language Models (LLMs) such as GPT-3.5/4 Ouyang et al. (2022), and LaMDA Thoppilan et al. (2022) have substantially enhanced public access to complex information, particularly in sectors such as healthcare and public services. These models are celebrated for their capability to demystify intricate information, assisting in tasks ranging from routine inquiries to aiding clinical decision-making Brown et al. (2020). ChatGPT, a derivative of the InstructGPT model Ouyang et al. (2022), has gained widespread popularity for textual tasks due to its advanced multi-turn prompting dialog interface, refined through Reinforcement Learning with Human Feedback (RLHF) Lambert et al. (2022). However, anecdotal reports on ChatGPT have also highlighted persistent challenges Bang et al. (2023) - for instance, it struggles with specific reasoning tasks Davis (2023); Guo et al. (2023), often hallucinates facts, and produces non-factual statements, undermining its reliability Shen et al. (2023); Thorp (2023). Additionally, its language coverage remains limited and its predominant focuses on English in model training and evaluation raises issues of equitable access for non-English speakers Seghier (2023), especially given that over 82% of the global population does not speak English as their primary or secondary language Crystal (2003); Lu et al. (2022); Jiao et al. (2023).

Furthermore, substantial efforts have been directed towards developing LLMs, such as UrbanGPTLi et al. (2024), to make accurate predictions on synthetic data. Given that LLMs are trained on extensive internet datasets, it is crucial to explore how these models perform with real industrial proprietary spatio-temporal data Xu et al. (2024, 2025) and to understand variations in performance across different spatio-temporal contexts.

To address these challenges, our study originates from a real-world industrial GenAI application task, gathering lessons learned and introducing a novel, comprehensive multilingual benchmark from the industry for evaluating LLMs in sensitive sectors such as health and public services Jia et al. (2023); Li and Zhang (2022); Xu et al. (2024); Ozmermer and Li (2023) across spatio-temporal domains. Our contributions include:

* Open-source H&PS Traffic Incidents Spatio-Temporal Dataset, containing diverse traffic incidents over a decade, totaling nearly 99,869

records for investigating LLM hallucinations.
* A robust quantitative analysis of three hypotheses across multiple languages aimed at enhancing the performance of state-of-the-art (SOTA) LLMs in managing real-world generative AI applications.
* An in-depth examination using Retrieval-Augmented Generation (RAG) to assess the influence of spatio-temporal data and prompts.

## 2 Related Work

Previous studies have explored the capabilities of models like ChatGPT (Ouyang et al., 2022), suggesting various methods to mitigate its limitations. For instance, Bang et al. (Bang et al., 2023) show that ChatGPT excels in zero-shot learning across 9 of 13 NLP datasets. However, they also report a noticeable performance decline when handling non-English languages, particularly in non-Latin scripts. Manakul et al. (Manakul et al., 2023) introduced SELFCHECKQFT for hallucination response detection. However, it rely mostly on response consistency, may overlook cases where LLMs deliver consistent but inaccurate information, leading to potential false negatives responses.

The XLingEval framework (Choudhury et al., 2023) assesses LLM behavior across several languages (English, Hindi, Chinese, and Spanish), focusing on metrics like correctness, consistency, and verifiability. Their findings indicate significant performance disparities across languages, with non-English responses generally showing an 18.12% decrease in quality (Choudhury et al., 2023). However, it only investigated the influence of multilingual with state-of-the-art LLMs, without further exploring how to avoid hallucinations or in languages such as German. Additionally, it did not examine the impact of input data on these LLMs or other factors beyond language type. For example, the format of the data, the effects of prompts under different temperature. Moreover, UrbanGPT (Li et al., 2024) utilizes LLMs specifically for modeling urban environments, applying a GPT variant to zero-shot learning tasks in traffic management and public safety. The study emphasizes the critical role of high-quality, representative spatio-temporal data in training effective models (Li et al., 2024).

Moreover, widely adopted RAG techniques sometimes generate responses that are misleading, incomplete, or contextually off-target, particularly with non-English data (Siriwardhana et al., 2023). Additionally, RAG systems are latency

Figure 1: The flow chart of H&PS Traffic Incidents Dataset generation.

 

sensitive, and training local LLMs with RAG is technically more complex and costlier than methods such as prompt fine-tuning or data augmentation (Karpukhin et al., 2020; Guu et al., 2020).

## 3 Our H&PS Traffic Incidents Dataset

Due to the challenges of penalty payments, regular reporting regulations, and the complexity of analyzing over 20 types of traffic incident records, current manual and subjective legacy systems are ripe for transformation by LLMs (Large Language Models) (Brown et al., 2020). LLMs can significantly enhance the efficiency of the entire traffic incident tagging and reporting process. As shown in Figure 1, LLMs can automate the classification process, suggest tags based on dialogues between drivers and support teams, minimize subjective ambiguities, and respond swiftly to avoid costly penalties associated with reporting delays, which are particularly costly in transport systems. Moreover, LLMs can conduct additional analyses and prioritization, such as identifying problematic traffic lines or stations and enhancing human awareness.

The subsequent sections will detail our dataset creation process and GenAI workflow for analysis, including the structure of incident records. This is visually represented in Figure 1. We have queried incident records from the past ten years in the city of Vienna via API under a Creative Commons Non-Commercial 4.0 International License. The Cooperation OGD Austria (Data.gv, 2022) has developed a recommendation for publishing survey data due to the transparency obligation under the B-VG (Austrian Constitutional Law) (Data.gov, 2022) - particularly allowing for academic research. Similar platforms can also be found such as NRW ZugInfo (Zuginfo, 2023) and f59 Stoerungen (f59 stoerungen, 2023), which indicate the traffic status of Germany NRW state and Vienna in real-time.

We then select 14 categories of different traffic incidents from the data pool (as shown in Appendix Table 5), namely Faulty Vehicles, Acute Track Damages, Acute Switch Damages, Overhead Line Faults, Signal Faults, Rescue Operations, Police Interventions, Fire Brigade Interventions, Illegal Parking, Traffic Accidents, Demonstrations, Events, Delays, and Other Incidents, to track over ten years. In the end, we collect more than 99,869 unique traffic incident records of Vienna public transportation.

Each traffic record starts with an ID number indicating its index order, followed by a title that specifies the affected traffic line (bus, tram or subway) along with its ID and tag as shown in Table 1. The tag includes incident class, written in German. For example, '71 Schadhaftes Fahrzeug' signifies a faulty vehicle affecting the Bus 71 line. Subsequently, a detailed description of the incident is provided. It's important to note that all descriptions are written in German. The record concludes with the start and end times of the traffic disruption and any other affected bus or tram lines. Notably, the 'traffic start time' sometimes differs from the 'start time'; the former indicates when the traffic disruption began, while the latter denotes when the central service team received the report from the driver or reporter. All data is stored in JSON format and made publicly available.

## 4 Experimental Settings

**Robustness of LLMs on Spatial VS Temporal Domain**: we assess the robustness of major SOTA LLMs includes the GPT series (Radford et al., 2018), tinyLlama model (Touvron et al., 2023), Claude-3-Haiku(Claude-3-Haiku), Claude-3-Haiku-200K(Claude-3-Haiku-200K), Claude-3-Sonnet(Claude-3-Sonnet), Gemini-Pro 1.0(DeepMind), Mistral Medium(Medium), Mistral-8x7B(Mistral-8x7B), Llama-3-70B-T(Llama-3-70B-Inst-FW). Specifically, we

\begin{table}
\begin{tabular}{l l} \hline \hline
**Category** & **Details** \\ \hline LIM Models Converred & GPT rates include GPT+T, TinyLlama, Claudio-J, Stomset, Genini-Pro 1.0, \\  & Claude-3-Haiku, Claude-3-Soonet, Genini-Pro 1.0, \\  & Mostful Medium, Mistral-8x7B, Llama-3-70B \\  & Both Tempered and Spatib domain logical reasoning tasks. \\  & **–79.98 equal traffic incident records.** \\  & **–79.98 equal traffic incident records.** \\  & Year of Records & ** 

evaluated different SOTA LLMs output scores by examining response quality across ground truth for both spatial and temporal tasks. For temporal tasks, we analyzed responses across 10 categories, (with additional details provided in the Appendix Table 11). For spatial tasks, we assessed five scenarios across all the U-lines and selective Bus line, encompassing varying traffic conditions and routing challenges.

* 10 temporal + 11 temperature settings
* 5 spatial), comparing results across 9 different LLM models (as shown in Table 2). Here, we then carried out 66 test samples per LLM, including tests on typical LLaMA and GPT-4 models (11 temperature settings
* 2 conditions: with and without hypothesis
* 3 hypotheses (as shown in Figure 3)). We have included a table detailing architectures, hyperparameters, and prompt settings of LLMs (see Appendix Table 9). Additionally, we provide attributes of each LLMs, including cost information, energy consumption and architectural complexity.

**Would RAG Really Help and How?** we also included **RAG-driven**[7]**LLM experiments using our dataset. These experiments were conducted with DataStax (dat) and Langflow(lan), where we vectorized dataset samples as context, used Astra DB(ast) as the vector database. We incorporated spatial and temporal queries as embeddings, adhering to the allowable TPM (tokens per minute) limit of 15,000 imposed by the API rate limits. We then also made ablation studies on comparing our Dataset with existing benchmarks (see Appendix Table 8).

For primary evaluation metric, we focus on the stability and accuracy (matching to Ground-Truth) of each model's responses. To test our hypotheses, we employed Multiple Linear Regression (MLR) [10], using P-value within 95% Confidence Interval (CI) as the confidence level [10].

## 5 Main Results

In this study, we first evaluate the top nine state-of-the-art (SOTA) LLMs with the cover of mostly well-known models. We conducted over 126 sets of experiments using our dataset, which covers data from 2013 to 2023. These experiments were designed to assess the LLMs' performance in spatial vs temporal domains.

**Unbalanced Hallucinations Performance on Spatio VS Temporal Domain.** Using our pro

Figure 2: The H&PS Traffic Incidents Dataset includes 99,869 recorded incidents within the Vienna public transportation system, categorized into 14 distinct scenarios: Faulty Vehicles, Acute Track Damages, Acute Switch Damages, Overhead Line Faults, Signal Faults, Rescue Operations, Police Interventions, Fire Brigade Interventions, Illegal Parking, Traffic Accidents, Demonstrations, Events, Delays, and Other Incidents.

 

posed dataset, we qualitatively evaluate the output of SOTA LLMs and present the results in following Table 2. We observe that _almost all 9 LLMs, including the GPT-4 model, exhibit a significant number of hallucination issues, achieving an average of only 22.22% (acc.) on spatial-related questions and 5.5% on temporal-related questions_. It is crucial to note this distinct performance gap in spatio-temporal questions, which is likely due to the extensive time spans covered over a decade-long record, coupled with language ambiguities between German and English, and the inherent semantic complexity. Almost "all" nine LLMs demonstrate even poorer performance in accurately responding to these temporal questions. Even the leading GPT-4 models, while outperforming their counterparts in spatial-related tasks, struggle significantly with temporal-related questions, achieving only about 25%.

Additionally, when further examining the Table 2, the Mistral series (Mistral-8x7B) models also caught our attention in the temporal domain. Our findings further confirm that these SOTA LLMs struggle with date format calculations. Regarding hallucination output types, LLMs sometimes produce _plausible-sounding but incorrect or non-sensical answers, miscalulate durations and frequencies, provide nonsensical station names or non-existent stations, randomly order delayed subway lines_ despite using the same input data, prompt as shown in following Table 3.

Moreover, _at higher temperatures GPT tends to produce more creative answers, but this trend is not guaranteed to be linear_. Meanwhile, despite being declared as trained with 1.1 billion parameters, TinyLLama Zhang et al. (2024) performs even more poorly in logical reasoning within the German-based benchmark as shown in yellow marked station in Table 3.

**Hypothesis Evaluation via Multiple Linear Regression.** Table 4 illustrates the outcomes of multiple linear regression Yule (1897) analyses involving three variables: Original traffic incident data, Temperature, and our three Hypothesis. \(P\)-values are utilized to gauge result confidence, with the \(P\)-value summary serving as an auxiliary indicator.

For Hypothesis 1, inspired by neuroscientists Ashraf (2010) who applied the psychology of schemata theory to enhance the reading comprehension skills of Banglades shutents in English as far back as 2010, the theory Ashraf (2010) posits that schema and cognitive frameworks used to organize information in long-term memory are crucial in interpreting and understanding texts. Similarly, for lengthy conversational dialogues, we often note down key points (e.g., 1, 2, 3, ...) to retain important information and can typically recall details based on these notes. By adopting a similar approach of indexing important sentences in incident data (assigning simple tag like 1, 2, 3, ... to each sentence), we want to determine if this straightforward tagging method can assist GPT-like models in maintaining stable outputs, particularly in non

\begin{table}
\begin{tabular}{l l c c c c c c c c c c} \hline \multirow{2}{*}{**Category**} & \multirow{2}{*}{**Prompt/Questions**} & **GPT-4** & **Chude-** & **Chudre-** & **Clundes-** & **Comiti-** & **Mixtral-** & **Mixtral-** & **Llama-** & **Llama-** & **Llama-** \\  & (Oosing & **Malka** & **J Alkala-** & **3-Sonnet** & **Pro L.0** & **Medium** & **NoTk** & **Total-** & **Topo-Inst** & embedded- \\  & (Calorie, 2006) & (Calorie, 3) & (Calorie, 3) & (Day, 3) & (Meaning, 3) & (Mistral) & (Janis, 3) & **FW** & **dow** \\  & & & & & & & & & & & & \\  & & & & & & & & & & & \\  & & & & & & & & & & & \\  & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 2: Spatio-Temporal Questions & LLMs & Correctness. ✓ indicates the corresponding LLMs answered correctly with ground truth, \(\times\) means it doesn’t align with the ground truth but indeed has a conflict with the fact, and \(\sim\) shows the incomplete answer or is partly correct.

 

English scenarios and for **Spatially related tasks**. As shown in Table 4, the intercept value of 8.205 suggests that, in the absence of other influences (i.e., at the "Original" data and "Temperature" at the reference level of "0"), the expected number of answers or scores is estimated at 8.205. This estimate is highly statistically significant (\(p<0.0001\)). Temperature changes exhibit a more pronounced impact than hypothesized effects, _demonstrating a nonlinear relationship where not all lower temperatures consistently result in increased robustness_. This is evident at temperatures equal to 0.3 which its score is 8.905 (8.205+0.6), highlighting that higher temperatures generally lead to decreased scores, but this is nonlinear. In general, _adopting hypotheses 1 aids in maintaining robustness while introducing some creativity into the responses, in contrast to setting higher temperatures_ has reduced 2.35 on the score,

For Hypothesis 2, drawing from real-life experiences particularly when tasks involve date calculations, it is common practice to verbally express and spell out dates. This practice helps prevent misunderstandings and ambiguities, especially when dealing with diverse cultural date formats and time zones, such as in German (Day-Month) and English (Month-Day). Several studies have also identified that models like ChatGPT struggle with date & math calculations (Ouyang et al., 2022). Inspired by this observation, we hypothesize that standardizing date-related inputs into a uniform, human-readable sentence format. The goal is to assess whether this standardization of date input can consistently improve the LLMs' performance for **Temporal-related** tasks. As shown in Table 4, increasing the temperature leads to a significant drop in accuracy scores. However, the hypothesized data exhibited the least performance decline. This observation aligns with the aforementioned statements, suggesting that adopting Hypothesis 1&2 maintaining robustness while introducing a degree of creativity into the responses, as opposed to the effects observed with higher temperature.

What's more, for Hypothesis 3 on the spatial domain, inspired by (Choudhury et al., 2023), we aim to evaluate the effectiveness of translating non-English data, **not just limited to prompts** but particularly in context data into English. We intend to quantify the level to which translating non-English prompts & context data into English can improve the performance of LLMs, especially in terms of accurate reasoning and minimizing erroneous or

\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Hypothesis 1**} & \multicolumn{3}{c}{**Dependent 2**} & \multicolumn{3}{c}{**Hypothesis 3**} \\ \cline{2-17}
**Variable** & **Estimate** & **P value** & **V value summary** & **Variable** & **Estimate** & **P value** & **V value summary** & **Variable** & **Estimate** & **P value** & **V value summary** \\ \hline Improve(temperature) & 8.205 & \(<0.001\) & **max** & Intercept component(2) & 10.17 & \(<0.001\) & **max** & **Bosey** (temperature) & 8.079 & \(<0.001\) & **max** \\ If Uppoon(1) & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** & **[10000]** \\ Temper Temper(1) & -4.06 & 8.652 & **m** & Temperbox(1) & -1.15 & -0.043 & **-** & **Imperson(1) & -1.4 & -0.2583 & **m** \\ Temper(2) & -1.1 & 0.277 & **n** & Temperbox(2) & -1.4 & -0.012 & **-** & **Imperson(2) & -0.95 & 0.3520 & **m** \\ Temperbox(3) & **[ 

fabricated responses in **Spatio-related** tasks. Here, as shown in Table4, an estimate of 1.282 suggests that transitioning from "German" Context data to "English" is indeed linked with a performance increase in the expected number of answers by approximately 1.282. This estimate is statistically significant (\(p=0.0021\)), signifying a positive effect to generate more robust answers, even when faced with temperature variations. It serves as a valuable strategy, emphasizing that instructing GPT Radford et al. (2018) in English or simply converting context data into English, not "only asking in English" prompt significantly aids in reducing spatial hallucinations.

**Strengths and Weaknesses of RAG in Hallucination Alleviation**. As shown in the RAG experiment results in Table2 (last column) and the sample detailed output in AppendixTable7, recent studies suggest that RAG notably enhances the management of hallucination issues in domain-specific contexts Siriwardhana et al. (2023). Indeed, using RAG has made the responses more close to the topics, (e.g. Not writing non-existent station names or completely nonsensical answers), and producing more relevant, detailed answers. For instance, in the time domain, context vectorization and query embedding _have proven effective in addressing ranking and search-related questions_, like correctly pinpointing the first and last incidents, as shown in Table2 (last column).

However, while RAG improves factual accuracy, it still does _not enhance the logical reasoning required to handle more complex spatial questions or intricate temporal queries_, such as date calculations (e.g., identifying all events starting between 6 AM and 6 PM or the three incidents with the longest duration). It also did not assist in finding the shortest path (e.g U4) or incidents specifically related to the shortest line. The _output remained very general, more like matching and pairing the context_.

## 6 Conclusion

In this work, we introduce a novel industrial spatio-temporal benchmark dataset (H&PS Traffic Incidents) from industry for enabling researchers to rigorously assess hallucinations in LLMs when handling real-world spatio-temporal challenges. It features diverse scenarios requiring both temporal and spatial reasoning. And we further conclude the following interesting findings: 1) Major LLMs exhibit a significant number of unbalanced spatio-temporal hallucinations, and struggling more in the temporal domain. 2) Three useful data preprocessing techniques offers practical guidance for optimizing data workflows in generative AI. 3) While RAG improves contextual factual errors, it does not always enhance logical reasoning when handling more complex spatial problems or intricate temporal queries.

Figure 3: Comparison of original and hypothesized incident data. These hypotheses are designed to enhance hallucination detection in Spatio and temporal domains, thereby improving LLMs’ logical reasoning and accuracy of generated results. Hypotheses 1 and 3 focus on Spatio aspects, while Hypothesis 2 specifically targets temporal improvements.

 



**Limitation:** Despite being the first to release such large industrial dataset on accident information, our data still have limitations. To more effectively test the temporal and spatial awareness capabilities of LLMs, we need to manually annotate more spatial and temporal data and ground truths. Expanding to other regions or cities would require additional approvals from governments or institutions, which could further enhance our dataset.

**Future work:** To address these limitations, we will continually collect accident information from various cities. Additionally, we plan to exploring various other functionalities of LLMs beyond just hallucinations.

**Acknowledgments:** We sincerely thank Touhidul Alam, Aik Alikhanian, Christine Grimm, Kirisits Julia, Jung-Hyun Oh, Stefan Stohr, Falk-Moritz Schaefer, Martin Piskernig, Hassan Ali, and Bastian Meyerfeld for their invaluable contributions to our paper. Their support in developing GenAI applications, dataset collection, cleaning, translation, and hypothesis brainstorming was instrumental.

We strongly oppose the use of this dataset for warfare or any activities targeting human beings. We reserve the right to revoke or remove the dataset if misused and disclaim any liability for losses resulting from such actions. This dataset aims to foster responsible public discourse on how LLMs can effectively handle spatio-temporal queries, contributing to the development of robust AI systems for the community.

## References

* (1)
* Abh [2024-08][https://astra.datastax.comAccessed](https://astra.datastax.comAccessed): 2024-08.
* Datastax [2024-08][https://www.datastax.comAccessed](https://www.datastax.comAccessed): 2024-08.
* Langflow [2024-08][https://www.langflow.aiAccessed](https://www.langflow.aiAccessed): 2024-08.
* Allwein and Barwise [1996] Gerard Allwein and Jon Barwise. 1996. _Logical reasoning with diagrams_. Oxford University Press.
* Ashraf [2010] Tasleem Ara Ashraf. 2010. _Teaching of Reading Comprehension Under Psychology Schemata Theory_. Daffodil International University Journal of Business and Economics.
* Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. _arXiv preprint arXiv:2302.04023_.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_.
* Choudhury et al. [2023] De Choudhury et al. 2023. Ask me in english instead: Cross-lingual evaluation of large language models for healthcare queries. _arXiv preprint arXiv:2310.13132_.
* Claude-3-Haiku [2007] Claude-3-Haiku. Anthropic. [https://www.anthropic.com](https://www.anthropic.com).
* Claude-3-Haiku-200K [2005] Claude-3-Haiku-200K. Anthropic. [https://www.anthropic.com](https://www.anthropic.com).
* Claude-3-Sonnet [2006] Claude-3-Bonnet. Anthropic. [https://www.anthropic.com](https://www.anthropic.com).
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Crystal [2003] David Crystal. 2003. _English as a global language_. Cambridge University Press.
* Data.gov [2022]Data.gov. 2022. Cooperation oogd uustria. [https://www.data.gov.at/en/info/cooperation-ogd-austiria/](https://www.data.gov.at/en/info/cooperation-ogd-austiria/).
* Davis [2023] Ernest Davis. 2023. Mathematics, word problems, common sense, and artificial intelligence. _arXiv preprint arXiv:2301.09723_.
* DeepMind [2022] Google DeepMind. Gemini-pro 1.0. [https://www.deepmind.com](https://www.deepmind.com).
* steerungen [2023] f59 soerungen. 2023. F59 soerungen. [https://f59.at/stoerungen/](https://f59.at/stoerungen/).
* Fisher [1970] Ronald Aylmer Fisher. 1970. _Statistical methods for research workers_. Springer.
* Guo et al. [2023] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinnan Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. _arXiv preprint arXiv:2301.07597_.
* Guu et al. [2020] Kelvin Guu et al. 2020. Retrieval augmented language model pre-training. In _International conference on machine learning_, page PMLR.
* Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_

Ziyu Jia, Youfang Lin, Yuhan Zhou, Xiyang Cai, Peng Zheng, Qiang Li, and Jing Wang. 2023. Exploiting interactivity and heterogeneity for sleep stage classification via heterogeneous graph neural network. In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_.
* Jiang et al. (2023) Zhengbao Jiang et al. 2023. Active retrieval augmented generation. _arXiv preprint arXiv:2305.06983_.
* Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatept a good translator? a preliminary study. _arXiv preprint arXiv:2301.08745_.
* Karpukhin et al. (2020) Vladimir Karpukhin et al. 2020. Dense passage retrieval for open-domain question answering. _arXiv preprint arXiv:2004.04906_.
* Lambert et al. (2022) Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havvilla. 2022. Illustrating reinforcement learning from human feedback (rihf). [https://huggingface.co/blog](https://huggingface.co/blog).
* Li and Zhang (2022) Qiang Li and Chongyu Zhang. 2022. Continual learning on deployment pipelines for machine learning systems.
* Li et al. (2024) Zhonghang Li, Lianghao Xia, Jiabin Tang, Yong Xu, Lei Shi, Long Xia, Dawei Yin, and Chao Huang. 2024. Urbangt: Spatio-temporal large language models. In _Proceedings of the ACM Conference on Computer and Communications Security_.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Llama-3-70B-Inst-FW Meta ai. https://ai. facebook.com/research/publications.
* Llama-3-70B-TMeta ai. [https://ai.facebook](https://ai.facebook). com/research/publications.
* Lu et al. (2022) Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, and Furu Wei. 2022. Trip: Triangular document-level pre-training for multilingual language models. _arXiv preprint arXiv:2212.07752_.
* Manakul et al. (2023) Potsawee Manakul, Adian Liuise, and Mark JF Gales. 2023. Selfcheckgrpt: Zero-resource black-box hallucination detection for generative large language models. _arXiv preprint arXiv:2303.08896_.
* Medium (2023) Mistral Medium. Mistral ai. [https://www.mistral.ai/models](https://www.mistral.ai/models).
* Mihalyov et al. (2018) Todor Mihalyov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In _Conference on Empirical Methods in Natural Language Processing_.
* Mistral-8x7B. Mistral ai. [https://www.mistral.ai/models](https://www.mistral.ai/models).
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_, volume 35, pages 27730-27744.
* Ozmermer and Li (2023) Evrim Ozmermer and Qiang Li. 2023. Self-supervised learning with temporary exact solutions: Linear projection. In _2023 IEEE 21st International Conference on Industrial Informatics (INDIN)_.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
* Sakaguchi et al. (2019) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogradne: An adversarial winograd schema challenge at scale. _arXiv preprint arXiv:1907.10641_.
* Seghier (2023) Mohamed L Seghier. 2023. Chatgpt: not all languages are equal. _Nature_, 615(7951):216.
* Shen et al. (2023) Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, and Linda Moy. 2023. Chatgpt and other large language models are double-edged swords. _Radiology_, 307(2):e230163.
* Siriwardhana et al. (2023) Shamane Siriwardhana et al. 2023. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. _Transactions of the Association for Computational Linguistics_, 11:1-17.
* Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lambda: Language models for dialog applications. _arXiv preprint arXiv:2201.08239_.
* Thorp (2023) H Holden Thorp. 2023. Chatgpt is fun, but not an author. _Science_, 379(6630):313-313.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachataux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Xu et al. (2024a) Kunpeng Xu, Lifei Chen, Jean-Marc Patenaude, and Shengrui Wang. 2024a. Kernel representation learning with dynamic regime discovery for time series forecasting. In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_, pages 251-263. Springer.
* Xu et al. (2025) Kunpeng Xu, Lifei Chen, and Shengrui Wang. 2025. Drift2matrix: Kernel-induced self representation for concept drift adaptation in co-evolving time series. _arXiv preprint arXiv:2501.01480_ 

Wei Xu, Jue Xiao, and Jianlong Chen. 2024b. Leveraging large language models to enhance personalized recommendations in e-commerce. _arXiv preprint arXiv:2410.12829_.
* Yule (1897) G Udny Yule. 1897. On the theory of correlation. _Journal of the Royal Statistical Society_, 60(4):812-854.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In _Annual Meeting of the Association for Computational Linguistics_.
* Zhang et al. (2024) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinylama: An open-source small language model. _arXiv preprint arXiv:2401.02385_.
* Zuginfo (2023) Zuginfo. 2023. Zuginfo. [https://www.zuginfo.nrw/](https://www.zuginfo.nrw/).

 

Appendix

In this section we provide the supplementary compiled together with the main paper includes:

* Ablation study on GPT-4/TinyLlama Models on hallucination type and accuracy density map for each hypothesis on our benchmark dataset in Table 6, Figure 4;
* Ablation study on H&PS Traffic Incidents Dataset vs other LLMs Benchmark in Table 8;
* The training details and hyper-parameters of experiments in Table 9, including questions lists in Table 11, output example of SOTA (e.g., referring to our particular experiment) in Table 10;
* The illustration of how we use Multiple Linear Regression to verify our hypothesis: from raw data input, for example, in GraphPad Prism, to interpreting examples and residual plots, see Figure 5.

We provide open access to our Health & Public Services (H&PS) traffic incident dataset, with the project demo and code available at Website [https://sites.google.com/view/llmhallucination/home](https://sites.google.com/view/llmhallucination/home)

### Evaluation Metrics

**Assigned accuracy scores strategies in Table 2**

As we assembled the code and have the ground truth for each temporal and spatial question, we were able to match the output of the LLM with its corresponding answers. Since the outputs are all linguistic in nature, especially for spatially related questions, it is more reasonable to not restrict the similarity evaluation to binary values (0 for no match, 1 for a match). Instead, we propose allowing a partial score of 0.5 for partially correct or reasonable responses. This can be formulated as follows:

\[\text{Scores}_{a,g}=\frac{1}{n_{a}}\sum_{i=1}^{n_{a}}S(x) \tag{1}\]

where

\[S(x)=\begin{cases}1&\text{if $S_{a,i}=(g_{a,i})$}\\ 0.5&\text{if $S_{a,i}\in(0.5*g_{a,i},g_{a,i})$}\\ 0&\text{if $S_{a,i}<=0.5*g_{a,i}$}\end{cases}\]

where \(S\) is the similarity score, \(a\in A\) refers to an scenarios (spatial / temporal), \(g\) refers to ground truth, and \(n_{a}\) is the total number of questions for scenarios \(a\).

**Stabilize scores strategies in Table 4**

Given the presumption that a better robustness LLM should produce reproducible results and LLM-generated results should counteract the effect of different temperature parameter settings, the output should remain stable and not cause ambiguities (not vice versa). Here, in our further hypothesis verification, we used stricter binary value scores for matching. While changing LLM models and various temperature settings, the output should match the default temperature value. Here, we set the temperature to 0 as the default value. After conducting accurate ground truth experiments, here, we challenged the LLMs by observing how they altered their answers when the temperature settings were changed.

The average score metric is formulated as

\[\text{Score}_{i,g}=\sum_{i=1}^{n}S_{g,i} \tag{2}\]

where \(S\) is the similarity score, \(i\) refers to an temperature (0.0, 0,1... to 1.0), \(g\) refers to default temperature output.

Here, we restrict the \(S_{g,i}\) to binary values (0 for no match, 1 for a match) based on the default temperature output to further verify our hypothesis testing.

### Ablation study: Qualitative Results

**Our benchmark presents a challenging task for SOTA LLMs Brown et al. (2020).** We compare the existing LLMs benchmarks with our Dataset, specifically focusing on logical reasoning Allwein and Barwise (1996) and hallucination. Our H&PS Traffic Incidents Dataset proves to be significantly more complex and realistic compared to the other 6 benchmarks (see Appendix Table 8). Notably, major LLMs such as ChatGPT Ouyang et al. (2022) and Llama Touvron et al. (2023) exhibit significant spatio-temporal hallucination problems on our dataset. Instances include cases when GPT fails to identify any traffic stations or even outputs completely different responses under all the same settings resulting in 0 score, as presented by the density map of GPT-4 models in Figure 4. Additional evidences are provided as in Appendix Table 6, 10 



\begin{table}
\begin{tabular}{l|c|r l

* [15] M. A. B. Krizhevsky, S. Gopalakrishnan, and J. L. Friedman, "A survey of machine learning for image recognition in the wild: a review," _IEEE Transactions on Image Processing Systems_**20**(6), pp. 347-389.
* [16] D. C. Kingma and P. Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, vol. 13, no. 2, pp. 1373-1380, 2014.
* [17] R. H. LeCun, Y. Bengio, and E. Hinton, "Deep residual networks with applications to large scale data," _arXiv preprint arXiv:1412.6090_**, 2014.
* [18] T. Li, X. Liu, Z. Wang, and Q. Sun, "Learning deep convolutional network models from tiny images," _CoRR_**2014**, 2014.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l|p{142.3pt} p{142.3pt}} \hline LLMs Benchmark & Statistics & Dataset Example \\ \hline OpenBookQA (Mihaylov et al., 2018) & • Understanding of a subject by combining open book facts with broad common knowledge. & _Question: Which of these would let the most heat travel through?_ • a new pair of jeans \\ • & • 5.957 multiple-choice questions. & • a steel spoon in a cafeteria \\ • & • Avg question tokens is 11.46. & • a cotton candy at a store \\ • & • 1,326 elementary level simple science facts. & • a calvin klein cotton hat \\ • & • Vocabulary size (questions + choices + facts) is 12,839. & _Science Fact: Metal is a thermal conductor._ \\ \hline HellaSwag (Zellers et al., 2019) & • Commonsense Natural Language Inference. & _Sentence: A bearded man is seen speaking to the camera and making several faces. the man_, \\  & • answers or continuations. & • • then switches off and shows himself via the washer... \\ • & • Questions come from ActivityNet or Wiki. & • • then rubs and wipes down an individual’s face and... \\  & • & • • is then seen eating food on a ladder while still... \\  & • & • then holds up a razor and begins shaving his face. \\ \hline WinoGrande (Sakaguchi et al., 2019) & • Test the “common sense” of language models. & _Sentence: Kartina had the financial means to afford a new car while Monica did not, since had a high paying job._ \\  & • &



\begin{table}
\begin{tabular}{c|p{28.5pt}|p{34.1pt}|p{69.7pt}|p{69.7pt}|p{69.7pt}|p{69.7pt}} \hline Model Description & Type & Token Limit & API Price in Dollars & Hypo-parameters & Prompt Example \\ \hline GPT-4 Turbo, The latest GPT-4 model with improved instruction, reproducible outputs, parallel function calling. Returns may of 4,096 output tokens. Training data up to Apr 2023 & gpt-4.1 128\(K\) preview & Input \(0.06/K\) Output \(0.12/K\) Tkens OutPlots & Text Generation chat completion & Hypothesis 1 in German (“Du” bit ein Analyst. Aus den beretigestellten Daten antwortest du au Numerfragen, um Statistiken basierend au Bentuzeringaben zu erstellen. Dies sind die Kontext-List-Daten: “+ resulted_data + ”Im DatenContext der Wiener-Linie sind unter Tiel betorfne Linien und unter ’Beschreibung’ betorfene Stationen verzeichnet. Welche 10 Stationen sind am haufigsten betroffen? Geben Sie nur in diesem Format aus: (Stationanne, Gesamtzahl der Vorfalle). Zum Bespiel: [Rotkreuzplatz, 10].") \\ \hline Currently points to gpt-4.0613. Training data up to Sep 2021 & gpt-4.0314 & 8\(K\) & Input \(0.03/K\) Tkens OutPlot \(0.06/K\) Tkens & Text Generation chat completion & Hypothesis 2 in German (“Du” bit ein Analyst. Aus den beretigestellten Daten anwortest du zur Netzerfragen, um Statistiken betirogenesiset al want Beutrizereingaben zu erstellen. Dies sind die Kontext-List-Daten: “+ resulted_data + ”Im Datencontext list affectsed stations. Which 10 stations are most frequently affected? Only output in this format: (station name, total number of incidents). For example: [Rotkreuzplatz, 10].”) \\ \hline Currently points to gpt-4.0613. Training data up to Sep 2021 & gpt-4.0314 & 8\(K\) & Input \(0.03/K\) Tkens OutPlots & Text Generation chat completion & Hypothesis 2 in German (“Du” bit ein Analyst. Aus den beretigestellten Daten autwortest du zur Netzerfragen, um Statistiken betirogenesiset al want Beutrizereingaben zu erstellen. Dies sind die Kontext-List-Daten: “+ resulted_data + ”Im DatenContext alert 10 or Wiener-Linie sind unter (title) betoffree Linien unter (start) betroffine Startzeit und unter (end) betroffene Endzeit verzeichnet. Welche 10 Linien sand man häufigsten betroffen? Wie lange ist die insgesamt betroffee Zeit, die jede dieser 10 verzogeren Lineinen? Ghenta user nur in diesem Format aus: 1. (Linien, Gesamtzahl der Vorfalle, insgesamt betroffene Zeit in Student Minuten Sekunden). Zum Bespiel: 1. (39A, 2, 5Stunden 24Minuten 32Sekunden).”) \\ \hline
1.1B LLama model on 3 trillion tokens. Using 16 A100-40G GPUs, intermediate checkpoint trained on 503B tokens, up to date 09-16-2023, Commonsense Avg 49.57 on HellaSwag & Tiny Llamada-1.1B-trained on 503B tokens, up to chat-v1.0 & Oponsource t.llamada-1.1B-top_k=50, top_p=0.95 & Mix_new_tokens=236Hypothesis 3 in English (“You are analyst. From the data provided, you answer user questions to create statistics based on user input. This is the context list data:” + resulted_data + ”In the Vienna Line data context, affected lines are under the title, and under ‘Description’ lists affected stations. Which 10 stations are most frequently affected? Only output in this format: (station name, total number of incidents). For example: [Rotkreuzplatz, 10].”) \\ \hline \end{tabular}
\end{table}
Table 9: The backbones, hyper-parameters, and prompt settings of the SOTA LLMs Brown et al. (2020). Note: * Prompt tested on all three kinds of models and \(resulted~{}data\) is the record of the incident inserted as a dictionary form for API read.





\begin{table}
\begin{tabular}{p{42.7pt}|p{190.3pt} p{56.9pt}} \hline \multicolumn{1}{c|}{Temperature} & **Original GPT-4 generate output** \\ \hline
0 & “Basivered and den berotigestellmen Daten und die an handigten betrofissten Stationen und des Gesamt der Verfiliatzen: i.e. (Handbetischen Ordusduzte, l.) 2. (Qualitenderung, l.) 3. (Leibnizegue, l.) 4. (Schatzler Instrum ez 1, d.) 5. (Kriplaz U, 2). 6. (Kriplaz U, 1). 7. Marguergenaent [kriplaz U, 2). 8. (Heliegelensch Kentarektungsse, l.) 10. (Zippersetzate U, 1). * (Grierwermake U, 1). * (Grierwermake U, 1). * (Grierwermake Stale U, 1). 10. (Thalistraule f = Lebracketier Gamel, 1). & “Basivered and den berotigestellmen Daten und die de man hhuglichen betrofissten Stationen und des Gesamtdrude, l.) 2. (Quelitenderung, l.) 3. (Leibnizegue, l.) 4. (Schatzler Instrum ez 1, d.) 5. (Karblaz U, 1). * (Grierwermake U, 1).





\begin{table}
\begin{tabular}{l l} \hline \multicolumn{2}{c}{**Temporal Related Questions Template**} \\ \hline List the causes of disruptions per hour and return a dictionary where the hour is the key and the disruption cause along with its frequency is the value. (Note that there can be multiple disruptions in the same hour, so disruption causes should be counted based on actual occurrences.) & \\ Find the lines with the most disruptions during the morning rush hour (7 to 9 AM) and the evening rush hour (5 to 7 PM), and provide the line name and the frequency of disruptions for each period. & \\ Determine the time periods with the most disruptions. Divide the day into 3-hour intervals and calculate the total duration of disruptions in each interval. Identify the interval with the longest disruption duration. & \\ Find the first and last disruption of the day and provide their start time, duration, and type of disruption. & \\ Identify the 3 disruptions with the greatest impact on the number of affected stops and list them. & \\ Find the 3 events with the longest duration and list their titles and durations in hours and minutes (e.g., 1 hour 20 minutes) in descending order. & \\ Calculate the average duration of all events (in minutes) and find the event whose duration is closest to the average. & \\ Find all events that begin between 6 AM and 6 PM, sort them in ascending order by start time, and provide their titles and durations. & \\ If an event is completed within 1 hour, it is considered a ”short event”; otherwise, it is a ”long event.” Find all long events, list their titles, and calculate their average duration. & \\ Calculate and compare the total duration of events in the morning (6:00 AM - 12:00 PM), afternoon (12:00 PM - 6:00 PM), and evening (6:00 PM - 12:00 AM). & \\ Which 10 lines are most frequently affected? How long is the total affected time for each of these 10 delayed lines? Provide the output in this format: 1. (Line, total number of incidents, total affected time in hours minutes seconds). For example: 1. (39A, 2, 5 hours 24 minutes 32 seconds). & \\ \hline \end{tabular}

* **Spatio Related Questions Template**

Given the traffic info data 2013-2023, which 10 stations are most frequently affected? Only output in this format: (station name, total number of incidents). For example: (Rotkreuzplatz, 10). & \\ Hi, you are an agent system like Google Maps. I want to travel within Vienna city. Given the traffic info data 2013 - 2023, I want to start from Schloss Schönbrunn to Musikverein Wien using the shortest path via public transportation on 21st Nov 2023. Is my trip getting affected? & \\ Hi, you are an agent system like Google Maps. I want to travel within Vienna city. Given the traffic info data 2013-2023, I want to start from Haus des Meres to U-Bahn-Station Roßauer Lande using the shortest path via public transportation on 21st Nov 2023. Is my trip getting affected? & \\ Hi, you are an agent system like Google Maps. I want to travel within Vienna city. Given the traffic info data 2013-2023, I want to start from Theater in der Josefstadt to Naturhistorisches Museum Wien using the shortest path via public transportation on 19th September 2023. Is my trip getting affected? & \\ Hi, you are an agent system like Google Maps. I want to travel within Vienna city. Given the traffic info data 2013-2023, I want to start from Museum für angewandte Kunst to Wiener Kriminalmuseum using the shortest path via public transportation on 17th March 2023. Is my trip getting affected? & \\ \hline \end{tabular}
* **S

Figure 5: GPT Radford et al. (2018) and Tinyllama Zhang et al. (2024) response generation Multiple linear regression workflow and Example of Interpretations.