DateLogicQA: Benchmarking Temporal Biases in Large Language Models
Authors: name: Gagan Bhatia; name: MingZe Tang; name: Cristina Mahanta; name: Madiha Kazi
Published: 2024-12-17 23:25:47

Abstract: This paper introduces DateLogicQA, a benchmark with 190 questions covering
diverse date formats, temporal contexts, and reasoning types. We propose the
Semantic Integrity Metric to assess tokenization quality and analyse two
biases: Representation-Level Bias, affecting embeddings, and Logical-Level
Bias, influencing reasoning outputs. Our findings provide a comprehensive
evaluation of LLMs' capabilities and limitations in temporal reasoning,
highlighting key challenges in handling temporal data accurately. The GitHub
repository for our work is available at
https://github.com/gagan3012/EAIS-Temporal-Bias