tBen: Benchmarking and Testing the Temporal Logic Reasoning Ability of Large Language Models with DatalogMTL
Authors: Dingmin Wang, Bocheng Zou, Zhen Han, zhiqiang xu


Reviewer paper summaries: This paper introduces a synthetic benchmark of temporal logic tBen. The dataset is based on DatalogMTL, a knowledge representation language that also supports temporal logic operators.
Summary of strengths: - The authors are releasing a new benchmark.
- The theoretical framework of temporal logic is very well written and structured, showcasing the authors' knowledge on the subject.
Summary of weaknesses: A lot of crucial information is missing from the benchmark creation step and the experiments, which directly affects the quality of this work and needs to be addressed:

- W1. More details on the size of the provided benchmark should be mentioned. Data descriptives also need to be included. There is no Appendix with any related information.
- W2. The prompts used for the experiments need to be included.
- W3. In the results section, there should be also reported the human performance on the task in order to be able to understand the inherent difficulty of the task.

The analysis is limited, and some claims both on the motivation and the conclusion are not supported by the results:

- W4. The main limitation of this work is that authors claim that LLMs do not perform well in temporal logic, but the results show that LLMs might struggle with the semantics of the symbolic language of temporal logic. Understanding the semantics of temporal logic mediates the task of performing temporal logic and reasoning. This is why, to test the reasoning capability of a model (temporal: CronQuestions[1], causal: CRAB[2], commonsense: CRoW[3]), it is usually common to verbalize the logic rules or the temporal graph since models are trained in natural language. Creating the temporal logic rules automatically and then verbalizing them before passing them to the LLM will provide a better understanding of whether the models struggle with semantics or fail to perform temporal logic. From the paper's current results, it is unclear which is the case, and therefore, the claims made cannot be fully supported. 

- W5. The class imbalance or random performance scores are not reported. F1 scores should be reported, instead of accuracy. This might affect the claim that "LLMs perform nearly random performance."

- W6. To make the claims stronger and support the conclusions, it is important to perform error analysis and check how offen the generations provide the correct answer, the current reasoning, or generate hallucinations.

_[1] Saxena, Apoorv, Soumen Chakrabarti, and Partha Talukdar. "Question answering over temporal knowledge graphs." arXiv preprint arXiv:2106.01515 (2021)._

_[2] Romanou, Angelika, et al. "CRAB: Assessing the Strength of Causal Relationships Between Real-world Events." arXiv preprint arXiv:2311.04284 (2023)._

_[3] Ismayilzada, Mete, et al. "CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks." arXiv preprint arXiv:2310.15239 (2023)._
Comments, suggestions and typos: No further suggestions or typos.
Overall assessment scores: 2.5