Benchmarking the Ability of Large Language Models to Reason about Event Sequences
Authors: Svenja Kenneweg, Joerg Deigmoeller, Julian Eggert, Philipp Cimiano


Reviewer paper summaries: The paper investigates the capability of large language models (LLMs) to reason about event sequences and resolve temporal references. The research introduces a synthetic benchmark dataset with 2,200 questions designed to test LLMs in a question-answering (QA) framework, focusing on the length of event sequences and the explicitness of temporal references. The findings reveal that while LLMs perform adequately with short sequences and explicit temporal references, their performance deteriorates with longer sequences and less explicit temporal cues. The study assesses four state-of-the-art LLMs, exploring the effects of different prompting strategies on their performance.
Summary of strengths: 1. The paper tackles LLM capabilities in handling complex event sequences and temporal reasoning, which is crucial for practical NLU applications.
2. The experimental design, including creating a new benchmark dataset and systematic testing across various models and conditions, provides a framework for evaluating the claims.
3. The hypotheses are clearly stated and thoroughly tested, providing solid support for the conclusions drawn.
Summary of weaknesses: 1. Using a synthetically generated dataset might not capture the complexity and variability of real-world temporal reasoning, potentially limiting the generalizability of the findings.
2. The paper primarily focuses on two types of temporal references, which may not be enough for those scenarios requiring deeper contextual understanding.
3. The number of question categories and templates is small. They may not cover enough real-world scenarios.
Comments, suggestions and typos: 1. In Table 2, what is "lightgrayheight" in the last line?
2. If there is no appendix, please remove it.
3. It is better to plot a figure to describe the whole data generation pipeline.
4. Please plot figures about question distributions, categories, temporal expressions, etc. Figures are useful for readers to understand quickly.
Overall assessment scores: 2.5