Timo: Towards Better Temporal Reasoning for Language Models
Authors: Zhaochen Su
Jun Zhang
Tong Zhu
Xiaoye Qu
Juntao Li
Min zhang
Yu Cheng
Venue: COLM (2024)
Keywords: Temporal Reasoning; Reinforcement Learning from AI Feedback; Supervised Fine-tuning
TL;DR: Exploring generalized models for temporal reasoning, we find a time-math link. Our framework tackles 38 tasks, creating Timo in 7B/13B sizes. Timo leads base LLMs by 10.0/7.6 scores, becoming the new SOTA.

Abstract: Reasoning about time is essential for Large Language Models (LLMs) to understand the world. Previous works focus on solving specific tasks, primarily on time-sensitive question answering.
While these methods have proven effective, they cannot generalize to a wider spectrum of temporal reasoning tasks.
Therefore, we propose a crucial question: Can we build a universal framework to handle a variety of temporal reasoning tasks?
To that end, we systematically study 38 temporal reasoning tasks.
Based on the observation that 19 tasks are directly related to mathematics, we first leverage the available mathematical dataset to set a solid foundation for temporal reasoning.
However, the in-depth study indicates that focusing solely on mathematical enhancement falls short of addressing pure temporal reasoning tasks. To mitigate this limitation, we propose a simple but effective self-critic temporal optimization method to enhance the model's temporal reasoning capabilities without sacrificing general task abilities.
Finally, we develop Timo, a model designed to excel in temporal reasoning at the 7B and 13B scales. Notably, Timo outperforms the counterpart LLMs by 10.0 and 7.6 in average accuracy scores and achieves the new state-of-the-art (SOTA) performance of comparable size. Extensive experiments further validate our framework's effectiveness and its generalization across diverse temporal tasks. The code is available at https://github.com/zhaochen0110/Timo.

Published as a conference paper at COLM 2024
Timo: Towards Better Temporal Reasoning for Language Models
Zhaochen Su' Jun Zhang', Tong Zhu', Xiaoye Qu?, Juntao L Zhang', Yu Cheng²
'Institute of Computer Science and Technology, Soochow University, China;
2Shagghai Al Laboratory; The Chinese University of Hong Kong
lji.minzhang/@suda.de edu..nduuu.c... quxiayye@pjlab.org cr,go, thu19978outlook.com, chengyu@cse.cuhk.co cww..............kkkk...kk
Abstract
Reasoning about time is essential for Large Language Models (LLMs) to un- derstand the world. Previous works focus on solving specific tasks, primar- ily on time-sensitive question answering. While these methods have proven effective, they cannot generalize to a wider spectrum of temporal reasoning tasks. Therefore, we propose crucial question: Can we build universal framework to handle a variety of temporal reasoning tasks? To that end, we systematically study 38 temporal reasoning tasks. Based on the observation that 19 tasks are directly related to mathematics, we first leverage the avail- able mathematical dataset to set solid foundation for temporal reasoning. However, the in-depth study indicates that focusing solely on mathemati- cal enhancement falls short of addressing pure temporal reasoning tasks. To mitigate this limitation, we propose simple but effective self-critic temporal optimization method enhance the model's temporal reasoning capabilities without sacrificing general task abilities. Finally, we develop Timo, a model designed to excel in temporal reasoning at the 7B and 13B scales. Notably, Timo outperforms the counterpart LLMs by 10.0 and 7.6 in average accuracy scores and achieves the new state-of-the-an (SOTA) per- formance of comparable size. Extensive experiments further validate framework's effectiveness and its generalization across diverse temporal tasks. The code is available at https////gggtuub..co///aaacceeee
1 Introduction
Large Language Models (LLMs) have achieved remarkable success in various reasoning tasks (Zhao et al. 2023 Chang et 2023 Su et al. 2024), such as mathematical, common- sense, and symbolic reasoning. Despite these advances, LLMs face significant challenges in temporal reasoning (Chen al. 2021 Tan et al. 2023a), which is crucial in human per- ception. Compared to other reasoning tasks that focus solely on one specific reasoning ability, temporal reasoning an integrated task that requires arithmetic (Zhu et al. 2023a),
logic (Mishra et al. 2022a) and world knowledge (Wei et al. 2022).
Prior efforts to improve the temporal reasoning capacity of LLMs focus mainly on time- sensitive question-answering (Chen et al. 2021), and utilize methods such step-by-step reasoning (Zhu et al.. 2023b Li al. 2023) and ruled-based supervised fine-tuning (SFT) (Tan et al. 2023a; Yuan et al. 2023b). More recent studies expand the scope of temporal tasks to include basic temporal concepts understanding (e.g., duration), intricate temporal interpre- tations (e.g., relation) and computate cututiiioos (e.g., arithmetic) (Wang & Zhao 2023). Due to their task-specific nature, the aforementioned methods exhibit limited generalization across the wider spectrum of temporal tasks.
"Work was done during the internship at Shanghai Al lab.
'Juntao Li is the Corresponding Author.
1

---

Published as a conference paper at COLM 2024
Causality
Frequency
Appticatio
Shi
Ordening
90.
withenetic
$7.5
Aniigulty
Duration
27.4
soour looka
70.0
35.0
Purt Lacieg
66.8
80.0
92.0
Ambiguity
Effect
Duretton
Helkrion
Storytelling
83.0
71.0
31.3
82.0
89.5
Typical time
Canos
65.0
Ambiguity (Math)
76.5
Frequency (Math)
Artthmetic (Math)
Duration (Math)
LLame-136
Timo-138
Figure 1: The detailed classification of 38 temporal reasoning tasks. 19 tasks are di- rectly related to math (i.e., Math-time tasks).
Figure Timo outperforms LLaMA in all temporal tasks and is the current state-of- the-art (SOTA) model of comparable size.
To address these limitations, we explore a crucial question: Can we build a universal framework to handle various temporal reasoning tasks? To tackle this, we face the following challenges: (1) integrating different temporal reasoning tasks into a unified framework; (2) generating and selecting the high-quality training dataset automatically; (3) improving the comprehensive temporal reasoning abilities while maintaining its general performance.
In response these challenges, first systematically study 38 subtasks within the temporal reasoning benchmark proposed by Wang & Zhao (2023). As shown in Figure I our analysis reveals that 19 tasks are directly related to mathematical reasoning (i.e., mathematical time tasks). For example, when "identifies the next leap year following 2024", mathematical skills are required to calculate the results. The rest are categorized as pure temporal rea- soning tasks, focusing solely on temporal reasoning without using mathematical abilities. Meanwhile, mathematical reasoning stands out with its diverse and rich instruction tuning datasets compared to temporal reasoning (Cobbe et al., 2021 Mishra et al. 2022b; Yue et al. 2023). Therefore, it is intuitive to build generalist temporal reasoning tramework based math-enhanced LLMs, setting solid foundation for temporal reasoning skills. However, our in-depth study indicates that focusing solely on mathematical enhancement through supervised fine-tuning falls short of addressing pure-time tasks. To bridge this gap, we further introduce a simple but effective method to obtain comprehensive temporal reasoning abilities. Specifically, we propose self-critic method to generate and select the high-quality temporal preference pairs, which are then utilized for enhancing model tem- poral capabilities through preference optimization. Finally, we propose a unified temporal reasoning framework, namely Timo. With this framework, our model achieves superior performance among 38 temporal tasks, as depicted in Figure
In our experiments, we train LLaMA2 models at both 7B and 13B scales with our framework, which results in Timo-7B and TiBo-13B. These two models demonstrate substantial improvement of 10.0 and 7.6 in average accuracy scores over the base models, respectively. Our comprehensive analysis indicates that our framework successfully integrates substantial mathematical knowledge along with temporal information. Extensive experiments further verify the effectiveness of our method in preserving general task capabilities and maintaining robustness under different scenarios. To sum up, our contributions are shown below:
We systematically study a variety of temporal reasoning tasks and discover the inner correlation between time and mathematics, where temporal reasoning could benefit from mathematics instructions;
We make the first attempt to build a unified framework to address 38 temporal tasks. Specifically, upon mastering mathematical reasoning capabilities, we propose simple but effective self-critic temporal optimization method to strengthen the temporal reasoning capabilities comprehensively;
2

---

Published as a conference paper at COLM 2024
100
Fs. COT II MATH-COT
70.5
80
70
68.8
69.8
69.5
60
46.8
55.0
46.2
52.1
46.0
65
63.3
64.6
64.8
64.4
40
30.5
60
20
Math-time tasks
Pure-time tasks
AMB.
ARITH
DUR.
FREQ.
0k
50k
100k
150k
180k
Figure 3: Performance comparison with Math-CoT and traditional prompting methods in math-time tasks.
Figure 4: Comparisons on temporal tasks with models trained on different numbers of math instructions.
The proposed framework outperforms 10.0 and 7.6 scores over the baselines, establishing as the new SOTA model of comparable sizes. Besides, our models consistently enhance the temporal reasoning capabilities without sacrificing general task performance.
Revealing the Correlation between Math and Temporal Reasoning
2.1 Analysis on Temporal Reasoning Benchmark
Wang & Zhao 2023)
provides comprehensive collection of 38 subtasks centered around temporal reasoning tasks. It is widely observed that substantial portion of these tasks relies mathematical skills for calculating and reasoning about time. For example, within the Frequency category, the Computation subtask requires the calculation of event frequencies or intervals. In the Ambiguity Resolution task, mathematics provides standardized method of time representa tion, such as the 24-hour format and date calculation formulas, enabling different temporal expressions to be accurately understood and converted. Based these observations, we categorize temporal tasks into two categories. The specific subtasks under each category are shown in Figure Below is our classification:
Mathematical Time Tasks (Math-time tasks): These are temporal reasoning tasks that necessitate mathematical skills, such as calculating time frequencies, converting time shifts, comparing time sequences, determining time intervals, estimating durations, and so on. This category encompasses total of 19 subtasks.
Pure Time Tasks (Pure-time tasks): These tasks require only temporal reasoning abilities for resolution and include reasoning about temporal commonsense, applications in real- world scenarios, temporal natural language inference (NLI) and on. This category also contains 19 subtasks.
2.2 Bridging Mathematics and Temporal Tasks
Inspired by Wei et al. 2022), we construct Math-CoT for each temporal task to establish connection between mathematics and temporal tasks. We utilize the MathInstruct dataset Yuue et al. 2023),, which comprises a diversified collection of mathematical problems with detailed rationales. From this dataset, we select five mathematical question-CoT pairs and employ GPT-4 to generate Math-CoT rationales by mimicking mathematical reasoning. Since pure-time questions lack mathematical rationales, Math-CoT is specifically designed for math-time tasks. We compare Math-CoT with two prompting methods: (1) Few-shot, which samples five question-answer pairs per task, and (2) CoT (Wei et al., 2022), where GPT-4 is used to generate step-by-step rationales for each task. We conduct the experiments using LLaMA2-7B under the 5-shot setting and report the accuracy for each task. As shown in Figure integrating mathematical reasoning into temporal tasks leads to significant
3

---

Published as a conference paper at COLM 2024
Act Reward Model
000
Generate
Golden
Select
Correct Set
Responses
Pairs
Training
LB
Pure-time Tasks
MathLLM
Wrong Set
Preference Pairs
Temporal
Timo
Figure 5: The pipeline of our self-critic temporal task optimization method. Based on responses generated by mathematical models (MathLLM), we classify correct and wrong sets using golden answers. From these, we further select high-quality pairs with our proposed hierarchical scoring method. Finally, the chosen pairs are used for DPO training.
enhancement model performance, with Math-CoT outperforming traditional prompting methods in all math-time tasks.
2.3 Mathematical Reasoning Foundation for Temporal Understanding
Given the established correlation between mathematics and temporal reasoning, it is intu- itive to instruct models in mastering mathematical reasoning to establish solid foundation for advanced temporal reasoning abilities. This connection motivates our investigation into how varying degrees of mathematical instruction influence model performance, Specifically, select 180k mathematical CoT rationales from the MathInstruct and perform scaling experiments by fine-tur line-tuning the LLaMMA-7BB with different volumes of math instructions (i.e., 0, 50k, 100k, 150k, and 180k). We evaluate the models both math-time tasks and pure- time tasks under the 5-shot setting. The results are shown in Figure 4 After supervised fine-tuning on 50k math instruction tuning instances, the model exhibits a notable improve- ment in performing math-time tasks, with accuracy increasing from 56.4 to 63.3. However, worth noting that this enhancement in mathematical skills has a minimal impact on pure-time tasks, with a maximum enhancement of 1.9. Additionally, our analysis indicates a declining trend performance across both task categories as the volume of math instruc- tions increases. We believe this decline results from overfitting to mathematical tasks due to excessive data, adversely impacting the model's temporal reasoning capability (Mishra et al. 2022a).
3 Self-critic Temporal Task Optimization
In the previous section, discovered that focusing solely mathematical enhancement falls short of addressing pure-time tasks. To mitigate this limitation, we introduce a simple but effective self-critic optimization framework to equip the model with comprehensive temporal reasoning abilities. The pipeline of our proposed framework detailed in Figure
Given the mathematical model I,, start by generating set of N candidate responses )) = (?.......... for each input prompt x,. Given the golden label gi for each prompt x; we divide ,, into the correct response set R and the incorrect response set R,:
R {y/ Vi align (y²" .:) = true),
R = 3/ R
(1)
where align(y",,) 8:) is a function that returns true if the response y," aligns with the golden label gi, and false otherwise. Inspired by the LLM--a-a-Judde mechanism (Zheng et al. 2023 Yuan et al., 2024 Qu et al. 2024b), we utilize mathematical model L directly as a reward model to identify high-quality response pairs. Notably, we introduce novel hierarchical scoring method, which is specifically designed for evaluating responses to temporal tasks and contains five key aspects: (1) relevance and basic temporal reasoning; (2) understanding of temporal aspects; (3) application of internal temporal knowledge; (4) direct and well-organized addressing of the question; (5) insightfulness and advanced reasoning. To choose the higher quality pair from the correct set R and wrong set R,, we
4

---

Published as a conference paper at COLM 2024
prioritize the response that utilizes the model's temporal reasoning to the fullest extent. The criteria for our evaluation prompts are illustrated in Figure 10 and [1 For each criterion response meets, point is awarded. We prompt the model L to assign a score r. [0,5] to each response y/" quantifying its quality across the above dimensions.
The temporal preference pair (y, ,y; is formed by selecting the top-scoring response from the correct set R as y and from the incorrect set R as y, We then utilize these pairs to perform direct preference optimization (DPO) by optimizing the following loss function:
Te xx) (y, x)
(T x) 1et (V7 x) )
LDPO (T; reet -I (xyt DDD logo (log
Blog
(2)
where y favored over y7 and B hyperparameter.
Experiments
4.1 Experimental Setup
Training Setup We use LLaMA2 7B and 13B (Touvron et al. 2023 our base pre-trained model. For SFT, we select 100k instances from MathInstruct (Yue et al.) 2023), the most representative dataset for mathematical reasoning instruction tuning. For self-critic temporal optimization, we focus on pure temporal reasoning tasks, which encompass 19 subtasks. We reserve 100 instances for evaluation and utilize the remaining data for training. a subtask contains fewer than 5,000 samples, we maintain all of them. Otherwise, we randomly select 5,000 instances. In total, we use 35,655 instances for optimization.
Evaluation Setup We conduct a comprehensive evaluation across all temporal reasoning tasks, encompassing total of 38 tasks. Following Tan et al. (2023a), we assess the model total 3,800 instances. Consis-
performance on 100 examples for each task, amounting tent with prior work (Qu et al.) 2024a Xia et al.) 2024), we evaluate the model's temporal abilities under the 5-shot setting and utilize greedy decoding (i.e., temperature 0) for generating model's responses. We extract the prediction from the response and calculate the accuracy of each subtask.
Implementation Details We utilize four/eight NVIDIA Tesla A100 GPUs to train models. To facilitate parallel training, we employ DeepSpeed Zero-Stage 3 (Ren et al. 2021) and FlashAttention2 (Dao 2023). For SFT, use learning rate of 2e-5, a batch size of 128, and cosine scheduler with a 3% warm-up period for 2 epochs. For candidate response generation, we sample = candidate responses with temperature T 0.8, p 0.95. When evaluating candidate responses, as there is variance to these scores, in our experiments we also use sampled decoding (with the same parameters) and generate these evaluations multiple (3) times and take the average. For DPO, we follow the hyper-parameters from Tunstall et al. (2023) with a batch size 32, learning rate 5e-7, a warm ratio of 0.1 using linear warmup scheduler for 9 epochs.
4.2 Baselines
To ensure the fairness of the experiments, we select baseline models built upon the founda- tional model LLaMA2. The baselines are selected based on the following dimensions:
LLMs for Temporal Reasoning: TimeLLaMA (Yuan et al. 2023b) currently the only open-source model that is specifically designed for temporal reasoning. It is developed to make temporal predictions and generate time-related explanations.
LLMs for Mathematical Reasoning: Timo is trained through temporal optimization based mathematical models. Here, models: (1) MAmmoTH (Yue et al. 2023) is designed for general mathematics problem- solving and is trained on the Mathinstruct dataset. (2) WizardMath (Luo et al. 2023a
compare the following mainstream mathematical

---

Published as a conference paper at COLM 2024
Model
Amb.
Ari.
Dur.
Fre.
Cau.
NLI
Ord.
Rel.
Sto.
Typ.
Parameter Model
LLaMA2
61.0
52.1
64.1
72.0
93.0
44.0
43.0
54.0
66.0
72.5
TimeLLaMA
56.2
42.7
37.0
40.8
42.5
15.0
7.5
48.0
32.0
WizardCoder
57.8
51.9
63.4
68.7
86.0
43.0
38.0
45.0
54.0
65.8
lama
50.2
55.2
58.7
71.0
86.0
55.0
44.0
45.0
55.0
68.3
WizardMath
65.2
52.9
53.3
72.0
94.0
49.0
39.0
36.0
63.0
64.3
ToRA
51.2
44.8
58.1
71.5
92.0
48.0
39.5
46.0
68.0
73.3
MAmmoTH
63.0
52.3
60.3
90.5
39.0
43.0
51.0
69.0
67.3
Timo
68.8
60.8
72.1
78.0
95.0
71.5
70.0
87.0
83.8
13B
Parameter Model
LLaMA2
66.2
63.9
72.0
83.2
95.5
55.0
44.0
44.0
71.0
82.0
WizardCoder
63.2
60.1
62.1
76.5
92.0
54.0
49.0
51.0
59.0
71.3
CodeLlama
62.6
60.6
67.0
90.5
46.5
58.0
69.5
WizardMath
63.2
58.3
72.9
78.3
95.0
58.0
43.0
54.0
82.0
77.3
ToRA
61.2
50.8
67.9
77.4
97.5
56.0
38.5
64.0
80.0
79.5
TH
69.1
67.0
72.4
80.4
97.0
62.0
48.0
57.0
77.0
78.8
Timo
70.4
66.3
77.7
87.2
97.5
87.0
79.5
78.0
83.0
89.5
Table 1: Results 38 temporal reasoning tasks. The abbreviations Amb., Ari., Dur., Fre., Cau., Ord., Rel., Sto., Typ. refer to ambiguity resolution, arithmetic, duration, frequency, causality, order, relation, story, and typical time. All values are percentages. The best results are in bold and the second results are in underlined.
utilizes the proposed Reinforcement Learning from Evol-Instruct Feedback (RLEIF) (Xu et al. 2023) to enhance its mathematical reasoning capabilities. (3) ToRA (Gou et al. 2024), a series of Tool-integrated Reasoning LLM Agents, is designed to solve challenging mathematical reasoning problems.
Previous work indicates that the usage of code enhances the model's ability to solve reasoning tasks (Gao et al. 2023). We select the following popular code models as our baselines: (1) CodeLLaMA (Roziere et al. (2023), family of L.MSs for code generation and programming-reeba
LLMs for Code Generation:
tasks. (2) WizardCoder (Luo et al., 2023b is similar to WizardMath and adapts the RLEIF method within the domain of coding.
4.3 Main Results
Table presents the results of Timo across 38 temporal reasoning tasks, Results categorized by Math-time and Pure-time tasks are shown in Appendix(Glin 1'alle9 and Table 10 From the results, we observe: (1) Timo surpasses counterpart LLMs in average accuracy of 10.0 and 7.6 scores, and outperforms other competitive math-solving and code-solving models with a clear margin, achieving the SOTA results on average. We also discover that Timo-7B consistently outperforms all 13B models in average performance, achieving maximum performance gain of 7.1. (2) Mathematical models do not show significant advantages in solving math-related tasks. This phenomenon is also observed in LLMs enhanced for coding abilities and temporal prediction capabilities. It indicates that excessive training on specific abilities leads the model to overfit in task-centric enhancements, diminishing its performance in other areas (Jha et al. 2023). (3) It is worth noting that Timo underperforms MAmmoTH in the Arithmetic task (i.e., scoring 66.3 vs 67.0) when evaluated under the 13B model size parameter. The superior performance of MAmmoTH can be attributed to its advanced general math-solving abilities, which facilitate more accurate computations in time-related scenarios. However, other mathematical models like ToRA and Math do not achieve the same effectiveness handling the Arithmetic task. A detailed case study for illustration is in Appendix B

---

Published as a conference paper at COLM 2024
GPT-3.5 GPT-4
LLaMA-13B MathLLaMA-13B Timo-13B
0.8
0.7
0.6
0.5
0.4
0.3
AMB. ARITH.
DUR.
FREQ.
STORY.
DUR.
FREQ.
AMB.
CAUS.
Nu..
REL.
ORDER.
TYP.
Figure 6: Performance of GPT series and our framework's models. MathLLaMA is based on mathematical instruction tuning and Timo our final model. Math-time tasks are marked with t, while others are pure-time tasks. We highlight our model's achievements: green star (*) where Timo beats GPT-335, and a red star (+) for surpassing GP'T-.
Further Analysis on our Framework
In our framework, we initially train mathematical model, iee, MathLLaMA. Then, we optimize its pure temporal reasoning abilities to derive the final Timo model. In this section, we first compare the performance of these two stages. Then, we delve into these models through the lens of token distribution shift and detailed case analysis.
Ablation Analysis of Framework We compare the model's performance on both math- time tasks and pure-time tasks. The results are shown in Figure 6 Compared to the foundational model LLaMA, MathLLaMA demonstrates superior performance in math- related tasks and surpasses the LLaMA in the majority of pure-time tasks, achieving higher scores in 6 out of 9 tasks. This improvement is attributed to the advanced logical and reasoning skills developed through mathematical instruction tuning (Mishra et al.) 2022a). When compared to Timo and MathLLaMA, our framework demonstrates strong generaliza- tion capabilities, achieving significant improvement in pure-time tasks, with only minimal performance degradation in the arithmetic task. Additionally, it is worth noting that Timo outperforms MathLLaMA in various math-time tasks (i.e., Ambiguity Resolution, Dura- tion and Frequency). This improvement is attributed to framework's ability to learn generalized temporal features.
Token Distribution Shift Analysis To understand the learning process and the differences between the different stages of our framework, we follow the methodology proposed by Lin et al. (2024) to analyze through the lens of token distribution shift. We analyze three pairs of models at the 7B scale: LLaMA vs MathLLaMA, MathLLaMA vs Timo, and LLaMA vs Timo. The results are shown in Figure 7 Notably, we observe the largest token distribution shift when transitioning from LLaMA Timo. Furthermore, we investigate the top 200 most frequently shifted tokens, labeling math-related tokens in red and time-related tokens in green. The transition from LLaMA to LatA primarily features changes in math-related tokens. Conversely, the switch from MatA to Timo is characterized by the presence of time-related tokens. When compared to LLaMA, Timo exhibits shifts in both math-related and time-related tokens, demonstrating a profound capacity to integrate substantial mathematical knowledge along with the temporal information.
Case Analysis As shown in Table we present a case analysis to provide a clear and intu- itive demonstration of Timo's superior performance. In math-time tasks, both MaMA and Timo effectively integrate temporal knowledge with computational capabilities to give the correct CoT and answer. However, L.LMMA produces incorrect result due to the
7

---

Published as a conference paper at COLM 2024
LLaMA LaAA
MathLLaMA
Timo
LLaMA
Timo
Unahitte (8.r%)) Marginal (15.60%)
Stitte (15.1%)
Unshifted (8862%) Marginal (18.22%) Shifted (16.08%)
Unshifted (85.85% Marginal (18.225) Shilted (17.85%)
'answer', 'Therefore', 'So', 'Original' 'calculation', 'add', 'subtract', frequencyy, 'days', 'calculate', 'approximately', 'total', 'count' 'x', 'Added', 'match', 'convert', 'Sub', 'Adding', 'equals', 'exceed', '",
"calculation', 'duration', "trequency', 'align', 'count', 'total', 'hours', 'ance', 'days', 'day', 'time', 'years', "context', 'minutes', 'subtract', 'weeks'. 'AM', 'july', 'months', 'Since', 'next', 'President', 'PM' "following' 'clock'...
'answer'. 'calculation', add', 'duration'; 'frequency', 'day'. 'days', 'year, 'total', 'Since', 'hours', 'subtract', 'w', 'AM', 'approximately" weeks', 'months', 'minutes'. '-,, 'equals", 'adding', '=", 'convert', 'sum'
Figure 7: Token distribution shift on different stages of our proposed temporal task opti- mization method. The ratios of unshifted, marginal, and shifted tokens are colored (%). Frequently shifted tokens are shown below, where math-related tokens are labeled in red and time-related tokens are highlighted in green.
error in time calculation, which indicates the importance of mathematical skills in solving math-time tasks. We also In our provided case of the pure-time tasks, both MathLLaMA and LLaMA fail to grasp the sequence of events, i.e., the timing of Amy's laundry activities. On the other hand, Timo demonstrates a strong understanding and application of temporal reasoning, accurately tracking the sequence and timing of Amy's activities and giving the correct answer. Overall, these cases vividly demonstrate Timo's comprehensive capabilities in temporal reasoning across different temporal task types.
More Detailed Study
Performance Comparison between Timo and OpenAl GPT Models We compare Timo- 13B with the current most powerful LLMs, i.e., GPT-3.5 and GPT-4. Specifically, we use the gpt-3.5-turbo-111 and 8pt-4-1106-previeve and set the temperature
for consistent evaluation. The results shown in Figure [ Despite its relatively small size of 13B parameters, Timo demonstrates impressive performance on pure-time tasks, outperforming GPT-3.5 in 7 out of 9 tasks and surpassing GPT-4 in 5 out of 9 tasks. Notably, Timo exceeds GPT-4 by a significant margin of 38 accuracy scores in the Relation task. Although there has been significant improvement in pure-time tasks, the performance on math-time tasks suggests that there is still room for further enhancement. This is attributed to the foundational model's weaker mathematical reasoning capabilities. We leave as future work to further improve the model's 's temporal reasoning abilities by better integrating mathematics capabilities.
Performance Comparison among Dif- ferent Rewarding Strategies In our framework, we design a series of crite- ria to assess the standard of responses and obtain high-quality temporal pref- erence pairs. To verify the effective- ness of our criteria, we compare our prompting approach with the widely adopted self-rewarding strategy (Yuan et al. 2024) and the random selection strategy. As shown in Table our strategy outperforms others in both math-time and pure-time tasks, highlighting its superiority in evaluating the quality of generated responses across different types of temporal challenges.
Math-time
Pure-time
random LLM-Judge
61.5
79.8
61.3
80.2
Timo
63.9
81.5
Table 3: Performance different reward methods
Robustness across Mathematical Models With Timo being derived from mathematical model trained with 100k math instructions, we validate the robustness and adaptability of our framework across different mathematical models, which is achieved by implement- ing self-critic temporal task optimization in models supervised fine-tuned with different volumes of instruction dataset (i.e., 50k, 100k, 150k, 180k). The results shown in Fig-
8

---

Published as a conference paper at COLM 2024
Mathematical Time Tasks
In which week of year 1652 does the date 10-27-1652 occur?
Questions
(A) Week 43; (B) Week 39; (C) Week 46; (D) Week 30
Calculating the weeks from January 1st to October 27th, it's 27 days. Dividing 27 by 7 places October 27th in the 4th week.
LLaMA
However, depending on the week starting day and system used, can be
considered as the 39th week. The answer is (B). X
We calculate the total days from January 1st October 27th, which amounts
31 (Jan.) + 28 (Feb.) + 31 (Mar.) + 30 (Apr.) + 31 (May.) + 30 (Jun.) + 31 (Jul.) + 31 (Aug.) + 30 (Sept.) 27 (Oct.), which equals 300 days. Dividing 300 by 7, we find that October 27th falls into the 43rd week of the year. The answer is (A).v
MathLLaMA
Calculating the weeks from January 1st to October 27th, the total days are 31 (Jan.) + 28 (Feb.) + 31 (Mar.) + 30 (Apr.) + 31 (May.) 30 (Jun.) + 31 (Jul.) + 31 (Aug.) + 30 (Sept.) 27 (Oct.), summing up to 300 days. Dividing 300 by 7 places October 27th in the 43rd week of the year. The answer is (A). /
Timo
Table 2: A case study for math-time tasks to intuitively show the effectiveness of Timo with temporal direct preference optimization. The right and wrong steps are colored by blue and red, respectively. An example of pure-time tasks is shown in Table 11
ure 8 The experimental results indicate that the trained models consistently outperform
in handling time-related tasks compared to their corresponding mathematical models, highlighting our method's capability to enhance temporal reasoning across different mathe- matical training backgrounds.
General Tasks Capability Assessment To verify the model's ability to retain its original capabilities, we utilize the Im-evaluation-harnes (Gao et al. 2021] to evaluate its perfor- mance on six typical downstream tasks: 5-shot MMLU (Hendrycks et al. 2020), 25-shot ARC Challenge (Clark et al. 2018), 5-shot GSM8K (Cobbe et al. 2021), 10-shot HellaSwag (Zellers) et al. 2019), 5-shot Winogrande (Sakaguchi et al. 2021) and 0-shot TruthfulQA (Lin et a 2022). In addition to comparing with LLaMA and MathLLaMA, we introduce Timo-SFT, which mirrors our framework in all aspects except for its training methodology. Specifically, Timo-SFT is supervised fine-tuned with the chosen responses in the selected preference pairs. The results are shown in Figure 9 We surprisingly discover that Timo outperforms other baselines in the reasoning and general knowledge ability tasks. Error analysis shows that our model aligns with the base model for 97% of the correct responses. This consis- tency indicates that our Timo effectively preserves general task knowledge, demonstrating remarkable generalization capabilities.
7 Related Work
Temporal Reasoning in LLMs
Time is crucial dimension in our physical world (Lazari-
dou et al. 2021 Su et al. 2022 2023 Zhao et al. 2024). Despite the advanced capabilities of
LLMs various tasks, their reasoning abilities are still underdeveloped (Su et al. 2024 Qiao
et al. 2023 H Huang& & Chang 2023 Sun et al. Chu et al., 2023). Temporal reasoning, which
is hundamental for humans to understand the world, is an important task in reasoning and has gained substantial research focus (Pustejovsky) 2003 UzZaman et al. 2012 Huang
et al. 2024). However, existing works often specialize limited aspects of temporal rea- soning, such as frequency (Zhou et al. 2019), duration (Zhang & Choil 2021), or event-time relations (Chen et al., 2021 Ian et al. 2023a). In our work, We address a comprehensive scope of temporal reasoning, including various levels and dimensions of understanding about time (Wang& & Zhao 2023). Differing from prior approaches that rely on external knowledge (Yuan et al. 2023a Tan et al. 2023b Xiong et al., 2024) or impose temporal constraints (Li al. 2023 Zhu et al. (023b) within a narrow sub-scope of tasks, we propose unified framework designed to generalize across different temporal reasoning scenarios.

---

Published as a conference paper at COLM 2024
GSMBK
76
MathLaMA
25.3
MathLLaMA-78 Timo-7B
ARC
57.9
73
+4.8
HellaSwang
78.0
13.4
+4.1
70
H
MHLU
47.
67
46.1
TruthuuiA
243
724
28.3
64
Temporal
50k 100k
150k
180k
WiroGranade
Figure 8: Results of Timo trained on the mathematics dataset of different sizes, demonstrating consistent improvements across models.
Figure 9; Reasoning and general knowl- edge performance comparison under cur- rent mainstream benchmarks.
Preference optimization approaches typically involve training fixed reward model based on preference data, and then utilizing the reward model to train via reinforcement learning (RL) (Schulman et al. 2017 Ziegler et al. 2019
Preference Optimization for LLMs
Stiennon et al. 2020: Bai et al., 2022). To simplify this process, P Preference Optimiza- fion (DPO) Ratailov et al. 2023) is introduced to avoid training the reward model entirely, and instead directly train the LLM using preference pairs. Building on this approach, re- cent works explore automatic optimization and self-correction in LLMs (Pan et al.. 2023 Ji
et al.] 2024). This involves two key steps: instructing LLMs to self eelf-generate their training dataset (Wang et al. 2023: Taori et al. 2023; Tunstall et al. 2023) and serving LLMs as reward models (Fernandes et al. 2023 (Saha al., 2023: Dubois al. 2024) select high-quality data. The selt-generated data optimization enables models to iteratively improve their performance through self-rewarding mechanism (Yuan et al. 2024). Inspired by the above works, we introduce a self-critic temporal optimization method that leverages the inherent capabilities of the model itself to achieve significant improvements all temporal tasks.
8 Conclusion
In this paper, we consider the problem of building universal framework to strengthen the temporal reasoning capabilities of LLMs. Through systematic investigation, discover a close relationship between mathematics and temporal reasoning. Building upon this insight, introduce self-critic temporal optimization method to equip the model with comprehensive temporal reasoning abilities. The Timo model, trained within our proposed framework, indicates significant generalizability across 38 temporal tasks, establishing as the new SOTA model of comparable sizes, Extensive experiments further demonstrate the effectiveness of our framework in maintaining general task abilities.
Acknowledgement
We want to thank all the anonymous reviewers for their valuable comments. We are also grateful to Wei Liu for his insightful suggestions during the mathematical reasoning experiments. This work was supported by the National Science Foundation of China (NSFC No. 62206194), the Priority Academic Program Development of Jiangsu Higher Education Institutions, the Natural Science Foundation of Jiangsu Province, China (Grant BK20220488), and Young Elite Scientists Sponsorship Program by CAST (2023NNRC001).
10

---

Published as a conference paper at COLM 2024
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, 1lya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. https:////// http:/awwwww.ogab///////// org/abs/23122 09390
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023.
Wenhu Chen, Xinyi Wang, and William Yang Wang. A dataset for answering time- sensitive questions. In Thirty-fifth Conference on Neural Information Processing Systems Systems Datasets and Benchmarks Track (Round 2), 2021. URL
wttp://wwwwwweeffffrere//o net/forum?
id=9-LSfsU74n
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Wei- hua Peng, Ming Liu, Bing Qin, and Ting Liu. A survey of chain of thought reasoning:
Advances, frontiers and future. arXiv preprint 23X1122309.114002 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simula- tion framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André FT Martins, Graham Neubig, Ankush Garg, Jonathan H Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. arXiv preprint arXiv:2308.07220 2023.
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https://doi.
5281/zenodo. ctroodddo.37666. 5371628
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. ToRA: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on Learning Representations, 2024. httttp/s//oppmrvieeeetee////enee
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference Learning Representations, 2020.
11

---

Published as a conference paper at COLM 2024
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1065, Toronto, Canada, July 2023. Association for C 67. URL
doi: (0.18653/v1/2022 http//2222..innngsaaaa wwtpyyy.rrg//233./.. findings-acl.6)
Rikui Huang, Wei Wei, Xiaoye Qu, Wenfeng Xie, Xianling Mao, and Dangyang Chen. Joint multi-facts reasoning network for complex temporal question answering over knowledge graph, 2024.
Aditi Jha, Sam Havens, Jeremy Dohmann, Alex Trott, and Jacob Portes, Limit: Less is more for instruction tuning across evaluation paradigms, 2023.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O'Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chur -hgg Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey, 2024.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' l'Autmme, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 24229348-29363, 2021.
Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng, Shafiq Joty, and Lidong Bing. Unlocking temporal question answering for large language models using code execution, 2023.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. Urial: Aligning untuned LLMs with just the 'write' amount of in-context learning. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://openrevicee wwwpe//foormmmnddeeeeeeeeee
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicen- cio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P'apers), pp. 3214-3252, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: 10.18653/v1. /ttp://////222aalooog.g... acl
URL
httts:://al.anthoooogy.rrr////// -cl222g..21122222222
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.099:: 099583,,000 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023b.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. LILA: A unified benchmark for mathematical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5807-5832, Abu Dhabi, United Arab Emi- rates, December 2022a. Association for Computational Linguistics. doi: 10.18653/10 /v/l 2022.emnlp-main.99 URL/Inttps://aclantto/// wwtp:tthhhooooooog/2222/2 wwtpor/g///2eeeee wmw.:222emmmmmmm....m..e...m--.m 392
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. NumGLUE: A suite of fundamental yet challenging mathe- matical reasoning tasks. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational
12

---

Published as a conference paper at COLM 2024
Linguistics (Volume 1: Long Papers), pp. 3505-3523, Dublin, Ireland, May 2022b. As- sociation for Computational Linguistics. doi: 10.1655/v112202.aallorr https://aa..antoooly....///22222 acw.g///2222....1oooo... 246
URL
Linngming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.031885 2023.
J Pustejjosky. The timebank corpus. In Proceedings of Corpus Linguistics 2003, pp. 647-656, 2003.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume T: Long Papers), pp. 5368-5393, Toronto, Canada, July 2023, Association for Computational 294. URL http:://acclantholeeg..///////
Linguistics. doi: w...8663/v1/2023aaccl/ acl- acl-long.293
Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, and Jianfeng Dong, Alleviating hallucination in large vision-language models with active retrieval augmentation, 2024a. URL |R
//arxiv.org/abs/ (ataaab//2408.000 1.4405000555
Xiaoye Qu, Mingyang Song, Wei Wei, Jianfeng Dong, and Yu Cheng, Mitigating multilingual hallucination in large vision-language models, 2024b. URL /RL.:ttts///axxi..rr////////// 00550
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Seiett- Conference on Neural Information Processing Systems, 2023.
URL http::///ppeeeeeeeeeeeeee/rrr//
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. (ZeRO-Offload): Democratizing (Billion-Scale) model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp.
551-564, 2021.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:230001512 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99-106, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066 .034777,.637,...0 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Zhaochen Su, Zecheng Tang, Xinyan Guan, Lijun Wu, Min Zhang, and Juntao Li. Im- proving temporal generalization of pre-trained language models with lexical seman- tic change. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6380-6393, 2022.
13

---

Published as a conference paper at COLM 2024
Zhaochen Su, Juntao Li, Zikang Zhang, Zihan Zhou, and Min Zhang. Efficient con- tinue training of temporal language model with structural information. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 6315-6329, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 /v2//vvv//// Hhaclanthology.orr//2220 findings--mninnn
h /tt/2223..fmmee 2t1/222...nndigseemmm.. finniings--ennlp.1111 URL https
Zhaochen Su, Juntao Li, Jun Zhang, long Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, et al. Living in the moment: Can large language models grasp co-temporal reasoning? 2tw.:::220000000772 2024.
arXiv preprint
Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jiaqi Liu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. survey of reasoning with foundation models: Concepts, methodologies, and outlook.
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards benchmarking and improving the temporal reasoning capability of large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 14820-14835, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 10.18653/ v1/2023.acl-long. 828. URL
acl- aol-long. 828
Qingyu Tan, Hwee Tou Ng, and Lidong Bing. Towards robust temporal reasoning of large language models via a multi-hop qa dataset and psendo-instruction tuning. arXiv preprint arXiv:2311.000 1.0922,, 2023b.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama
2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint trXivv23307.02888 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of im alignment. arXiv preprint arXiv:2310.16944, 2023.
Naushad UzZaman, Hector Llorens, James Allen, Leon Derczynski, Marc Verhagen, and James Pustejovsky. Tempeval-3: Evaluating events, time expressions, and temporal relations. CoRR, abs/1206.552 2012. URL http:://rxxiv wttp://wwwggg/b/////ee/e/ 5333
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.
Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. As- sociation for Computational Linguistics. doi: 10.18653 /ttp//l///2222/// acl- 7544 URL https://acllantho..og org. wtg/ acl moll-lon..72 754
Yuqing Wang and Yun Zhao. Tram: Benchmarking temporal reasoning for large language models. arXiv preprint arXiv:2310.00835, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models, 2024. URL
Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.0666 06683,, 2024.
14

---

Published as a conference paper at COLM 2024
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023.
Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. Back to the future: Towards explainable temporal reasoning with large language models. arXiv preprint arXiv:2310.01074, 2023a.
Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. Back to the future: Towards explainable temporal reasoning with large language models, 2023b.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309 00653,, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational URL
Linguistics. doi:
Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7371-7387, Online and Punta Cana, Dominican Republic, November 202T. Association for Computational Linguistics. doi: w...8633/v12221emmmppp mai......mmm........66666.. URL ottps://aclanttoooggg.or//// ewm.pppp..mmnpppppmmmmm-mp emwlp--mannnn main. 586
Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hannaneh Hajishirzi, and Noah A. Smith. Set the clock: Temporal alignment of pretrained language models, 2024.
Jinman Zhao and Xueyan Zhang. Exploring the limitations of large language models in compositional relation reasoning, 2024. URL httpt/ps://aaa.oorr/aaaa////////
02615
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303.18 18223,, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going on vacation" " takes longer than "going for walk": A study of temporal commonsense understanding. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3363-3369 3369,, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ v1/D19-1332 URL
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiax- ing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4471-4483 Toronto, Canada, July 2023a. As- sociation for Computational Linguistics. doi: 10.18653 /t///v1/20000 htlpl///..aaamm acl23..cl-- URL
https://acl.antoooogy..r//////// 2rwpaanthhoooooyyo///223333 acl -log- 245 245
15

---

Published as a conference paper at COLM 2024
Xinyu Zhu, Cheng Yang, Bei Chen, Stheng Li, Jian-Guang Lou, and Yujiu Yang. Ques- tion answering as programming for solving time-sensitive questions. arXiv preprint 1222: 2023b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human prefer- ences. arXiv preprint arXiv:1909.88593, 2019.
16

---

Published as a conference paper at COLM 2024
A Prompt
Our rewarding prompts are shown in Figure 10 and 11 The prompts for different temporal
tasks can be found in our GitHub repository: https: thub.....tt.......m com//haochene 110/Timo
B Comparative Analysis of Mathematical Models on Arithmetic Tasks
We observe that task-specific fine-tuning approaches compromise the LLMs' original abili- ties. We conduct case study to delve deeper into this phenomenon. As shown in Table ToRA and WizardMath have difficulty grasping basic concepts of time. Specifically, these models demonstrate challenges in accurately converting between the 12-hour and 24-hour time formats, fundamental aspect of temporal understanding. This case study serves clear illustration of how specialized fine-tuning can compromise the LLMs' inherent ability to perform fundamental reasoning, underscoring the need for balanced approach model training.
What 02:45 PM + 10:44? (A) 2:39 AM (B) 3:41 AM (C) 1:29 AM (D) 11:20 PM
Questions
Timo
The calculation 14 10: 44 29, which is 1:29 AM. The answer is (C).
MammoTTH
The calculation 45 10: 44 29, which is 1:29 AM. The answer is (C).
WizardMath
The calculation 45
26; which is 3:41 AM. The answer (B).
ToRA
The calculation 45 10: 44 25, which is 3:41 AM. The answer is (B).
Table 4: The case chosen from the Hour Adjustment (12h) task. ToRA and WizardMath fall short in time calculation and converting between 12-hour and 24-hour formats.
C Iterative Optimization Study
Recent work [Touvron et al. 2023 Yuan al. 2024) suggests that updating preference data through multiple iterative rounds enhances the performance of preference optimization. Therefore, we explore Iterative DPO refine alignments across temporal reasoning tasks. do not observe significant improvement from iterative training. The reason might be due to the efficiency of our method, where single iteration is sufficient for robust learning, and excessive training could instead diminish performance in temporal reasoning tasks.
The results are shown in Table 5 However,
Math-time
Pure-time
iter.
63.9
81.5
iters
62.1
80.9
iters
57.8
80.1
Table 5: Comparison on different iteration settings
D Validating Timo on LLaMA3-8B
To further validate the effectiveness of Timo in enhancing temporal reasoning across different LLMs, we conducted additional experiments using the LLaMA3-8B model. The results are shown in Table 6 Compared to vanilla LLaMA3-8B, Timo shows average improvement of 5.1 scores, with 1.2 scores in math-related tasks and scores in time-related tasks. These consistent improvements across both the LLaMA2 and LLaMA3 series demonstrate Timo's strong generalization capabilities across different model series, enhancing its applicability and effectiveness in diverse settings.
17

---

Published as a conference paper at COLM 2024
Math-time
Pure-time
Average
LLaMA3-8B +Timo
81.4
79.6
80.5
82.6
88.6
85.6
Table 6: Performance Comparison of LLaMA3-8B with and without Timo enhancement
E
Evaluating the Impact of Math LLM on Temporal Reasoning
Existing work weak-to-strong generalization suggests that distilling data from weaker or equivalent LLM benefits a stronger LLM (Burns et al.) 2023). To address concerns regarding the influence of the LLM-as-Judge framework compared to the use of specialized math LLM, we conducted experiments using vanilla LLLMA2-7B and LLLMA2-7B--cll representing general SFT LLaMA models. As presented in Table our results demonstrate that incorporating a math LLM yields significant improvements in temporal reasoning tasks. Specifically, the math LLM outperforms the vanilla LLaMA2-7B and LLaMA2-7B-cha models by average of 3.6 and 7 scores, respectively. The performance gains are especially notable in math-related tasks, where the math LLM achieves scores 5.5 and 10.5 scores higher than those of the other two models. These results indicate that the math LLM component is crucial for enhancing temporal reasoning capabilities, outperforming the self-critic temporal optimization (i.e., LiM-as-Judgge framework alone. The results indicate that math-specific training plays pivotal role in reasoning over time, confirming the value of specialized LLMs in complex reasoning tasks.
Math-time
Pure-time
Average
Timo (LLaMA2-7B)
58.4
79.7
69.1
Timo (LLaMA2-7B-chat
53.4
78.1
65.7
Timo (MathLLaMA-7B)
63.9
81.5
72.7
Table 7: Comparison of temporal reasoning performance across different based LLM, with Timo applied for temporal optimization on LLaMA2-7B, LLaMA2-7B-chat, and MathLLaMA-7B.
F Further Evaluation of Timo on Temporal Reasoning Datasets
To further assess Timo's improvements temporal reasoning, we extended our evaluation to additional temporal reasoning datasets, i.e, MCTACO (Zhou et al. 2019) and TempRea- son (Tan et al. 2023a). These datasets were selected to validate Timo's effectiveness across broader range of temporal reasoning tasks.
MCTACO: This dataset evaluates a wide range of commonsense knowledge related to events, including the duration, order, stationary nature, and typical timing of events.
TempReason: This dataset emphasizes implicit temporal reasoning in structured facts, focusing on both event-time reasoning and event-event reasoning.
The results are shown in Table 8 Timo achieves scores 6.2 and 15.5 points higher than LLaMA2-7B and 7 on the TempReason task. Additionally, Timo surpasses MAmmoTH-7B by 19.3 points on the MCTACO task. These results indicate that Timo excels across various temporal reasoning datasets, demonstrating its robust general temporal reasoning abilities. In future work, We will further explore the generalization of Timo across more reasoning tasks, such as commonsense reasoning (Sakaguchi et al. 2021), and composition relations reasoning (Zhao & Zhang 2024).
18

---

Published as a conference paper at COLM 2024
MCTACO
TempReason
LLaMA2-7B
50.3
46.6
MAmmoTHE TH-7B
37.0
52.8
WizardMath-7B
12.7
37.3
Timo-7B
56.3
52.8
Table 8: Results the MCTACO and TempReason datasets. Timo-7B outperforms its counterparts, demonstrating superior general temporal reasoning abilities.
G Results for Math-time and Pure-time Tasks
In this section, present the detailed results by task category. Results for math-related tasks are presented in Table 9 Results for pure-time tasks are presented in Table I0
Model
Amb.
Ari.
Dur.
Fre.
7B Parameter Model
LLaMA2
55.0
52.1
54.0
69.5
TaMALLaMA
52.5
42.7
42.5
55.5
WizardCoder
53.8
51.9
40.5
66.0
Codel
44.5
55.2
68.0
WizardMath
63.3
52.9
45.0
74.0
ToRA
45.5
44.8
44.0
69,8
MAmmoTT
62.0
52.3
54.5
59.5
Timo
65.3
60.8
59.5
72.0
13B Parameter Model
LLaMA2
61.3
63.9
58.5
81.0
WizardCoder
58.5
60.1
55.5
72.3
CodeLlama
57.8
60.6
61.0
74.8
WizardMath
58.8
58.3
75.5
ToRA
56.8
50.8
48.0
75.8
MAmmoTH
64.9
67.0
71.0
79.8
Timo
65.0
66.3
76.5
85.5
Table 9: Results on Math-time tasks. Best results are in bold and the second results are underlined.
19

---

Published as a conference paper at COLM 2024
Model
Amb.
Dur.
Fre.
Cau.
NLI
Ord.
Rel.
Sto.
Typ.
7B Parameter Model
LLaMA2
85.0
68.2
77.0
93.0
44.0
43.0
54.0
66.0
72.5
LaMA
71.0
34.8
11.5
42.5
15.0
7.5
48.0
5.0
32.0
WizardCoder
74.0
58.6
74.0
86.0
43.0
38.0
45.0
54.0
65.8
Mama
73.0
62.0
77.0
86.0
55.0
44.0
45.0
55.0
68.3
WazardMath
73.0
56.6
68.0
94.0
49.0
39.0
36.0
63.0
64.3
ToRA
74.0
63.8
75.0
92.0
48.0
39.5
46.0
68.0
73.3
MAmmoTH
67.0
62.6
69.5
90.5
39.0
43.0
51.0
69.0
67.3
Timo
83.0
77.2
90.0
95.0
74.0
71.5
70.0
87.0
83.8
13B Parameter Model
LLaMA2
86.0
77.4
87.5
95.5
55.0
44.0
44.0
71.0
82.0
WizardCoder
82.0
69.0
85.0
92.0
54.0
49.0
51.0
59.0
71.3
CodeLlama
82.0
69.4
76.0
90.5
54.0
46.5
53.0
58.0
69.5
WathrdMath
77.2
84.0
95.0
58.0
43.0
54.0
82.0
77.3
ToRA
79.0
75.8
80.5
97.5
56.0
38.5
64.0
80.0
79.5
MAmmoTH
86.0
73.0
81.5
97.0
62.0
48.0
57.0
77.0
78.8
Timo
92.0
78.2
90.5
97.5
87.0
79.5
78.0
83.0
89.5
Table 10: Results on Pure-time tasks. All values are percentages. Best results are in bold and second best are underlined.
Pure Time Tasks
According Premise: Amy started doing her laundry early the morning every weekend. answer the question: What's the more plausible CAUSE?
Questions
(A) It allowed her to have the washing machines to herself as most people do laundry later in the day.
(B) Amy found more lost socks the laundry room.
The more plausible cause is that she found more lost socks in the laundry room. This explanation aligns with the purpose laundry, which clean and organize clothes, and a logical response to Amy's 's need to find lost socks.
LLaMA
The answer (B). X
The more plausible cause is that she found lost socks in the laundry room. This explanation aligns with the purpose laundry, which clean and organize clothes, and the additional socks found in the laundry room would be
Lath
a logical outcome. The answer is (B). X
The reason allowed her to have the washing machines herself as most people do laundry later in the day. This explanation aligns with the practicality of early morning laundry, as it minimizes the likelihood of encountering other
Timo
people using the laundry facilities. The answer is (A).
Table 11: A case study for pure-time tasks to intuitively show the effectiveness of Timo with temporal direct preference optimization. The right and wrong steps are colored by blue and red, respectively.
20

---

Published as a conference paper at COLM 2024
following 5-point scoring system review user's question and the corresponding correct answer:
Add 1 point if the response is relevant to the user's inquiry and shows some temporal reasoning ability, even if is incomplete or contains irrelevant content.
Add another point if the response correctly understands the temporal aspects of the user's question but doesn't fully resolve the query provide direct answer.
Award third point if the response accurately uses internal temporal knowledge to effectively answer the basic elements of the user's 's question.
Grant fourth point the response not only effectively uses temporal reasoning is also clearly written from an Al Assistant's 's perspective, addressing the user's question directly and comprehensively and is well-organized.
Bestow fifth point for response that excellently applies temporal reasoning, reflecting expert knowledge in time-based queries, and demonstrates high-quality, engaging, and insightful answer.
User: f prompt)
<response> response </response>
After examining the user's instruction and the response:
Conclude with the score using the format: "Score: <total points>".
Figure 10: The prompt for our LLM to act as chosen reward model
Use the following 5-point scoring system to review the user's question and the corresponding incorrect answer that attempts to use temporal knowledge but fails to correctly solve the problem:
Add point if the response attempts to relevant the user's inquiry and shows attempt at temporal reasoning. even the information incorrect.
Add another point if the response partially correctly attempts to handle the temporal aspects of the user's question but includes errors or misconceptions.
Award a third point if the response clearly tries to use temporal knowledge but fails to accurately address the basic elements of the user's 's question.
Grant a fourth point if the response, while attempting to reason temporally from Al Assistant's 's perspective, shows some logic but contains errors misses key information. Bestow fifth point for those answers that make effort in temporal reasoning but are incorrect, potentially misleading the user but not completely deviating from the topic of the question.
User: { prompt <response> response </response>
After examining the user's instruction and the response: Conelude with the score using the format: "Score: <total points>".
Figure 11: The prompt for our LLM to act rejected reward model
21


Reviews:
[{"review_id": "MYpaiKD973", "reviewer": "Reviewer_Q4nZ", "summary": "This paper proposes a model, Timo, achieving new SOTA at 7B and 13B scales, which leverages the mathematical datasets to set a solid foundation model for temporal reasoning and enhance with self-critic temporal optimization method.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Q4nZ, \n\nThanks for your feedback and insightful suggestions!\n\nSince temporal reasoning is an integrated task that requires arithmetic, logical, and commonsense reasoning abilities [1], our designed self-critic framework effectively handles these aspects and achieves significant improvements in temporal reasoning.\n\nTherefore, we believe our framework may also enhance other reasoning tasks that rely on the above three kinds of abilities. We will continue to explore the generalization of our framework across more reasoning tasks in the future.\n\n[1] TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models, arxiv-23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. For temporal reasoning, it leverages the mathematical dataset as the foundational SFT data and uses DPO with temporal reasoning datasets for further improvement.\n2. Experimental results demonstrate the performances of Timo are even better than GPT-4 in some areas. The analysis of the framework is insightful, and can deeply inspire other reasoning tasks.", "reasons_to_reject": "1. More generalization of the framework should be validated in other reasoning tasks.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dVDnZYIWVt", "reviewer": "Reviewer_QQVe", "summary": "Comments after Author Discussion\nReasoning about time is essential for Large Language Models (LLMs) to understand the world. In this paper, the authors propose a simple but effective self-critic temporal optimization method (TIMO) to enhance the model’s temporal reasoning capabilities without sacrificing general task abilities. It’s designed to excel in temporal reasoning at the 7B and 13B scales and outperforms the LLaMA2 by 10 and 7.6 in average accuracy scores on 38 Temporal reasoning tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QQVe,\n\nThanks for the helpful comments! Hope the following responses address some of your questions:\n\n> Concern 1: Lack of analysis on comparison with other models in the ‘Main Results’ section.\n\nThanks for your suggestion. We will add a detailed analysis of comparisons with LLMs for temporal reasoning, mathematical reasoning, and code generation in the next version.\n\nFor example:\n\n**Comparison with LLMs for temporal reasoning:** TimeLLaMA is the only open-source model specially designed for making temporal predictions and generating time-related explanations. TIMO's performance is far better than TimeLLaMA (i.e., scoring 72.7 vs 38.6 on average), indicating that focusing solely on enhancing a single aspect of temporal capabilities does not lead to a comprehensive improvement.\n\n> Concern 2: Lack of comparison with GPT-4 in the ‘Ablation Analysis of Framework’ section.\n\nThanks for your review. \n\nActually, we have detailed the comparison between TIMO and GPT-4 in Section 6 \"Performance comparison between TIMO and OpenAI GPT models\".\n\nTIMO excels in pure-time tasks due to its ability to generate and select high-quality temporal preference pairs. However, TIMO still falls short in math-related tasks due to LLaMA's weaker mathematical reasoning compared to GPT-4.\n\n> Concern 3: In table 1, the result ‘83.0’ is not the best result.\n\nSorry for the typo. In the revised Table 1, the best results will be bold, and the second results will be underlined.\n\n> Concern 4: In case study, there is a certain ambiguity in the first case (Math-CoT lacks prerequisites ‘1652 Jan 1st is Monday’). Suggest the authors to replace other case.\n\nThanks for your suggestions! For a better demonstration, we will analyze the following case:\n\n```markdown\nWhat is 13:01 + 14:43? (A) 5:50   (B) 0:43   (C) 3:44   (D) 4:48\n```\n\n> Question1: Conducting experiments on other LLMs to verify TIMO's effectiveness.\n\nAs shown in the table below, we conduct experiments on LLaMA3-8B to verify TIMO's effectiveness.\n\nCompared to vanilla LLaMA3-8B, TIMO shows an average improvement of **5.1** scores, with 1.2 scores in math-related tasks and 9 scores in time-related tasks.\n\nThese results are consistent with the experiments conducted on LLaMA2-7B in our paper,  indicating TIMO's capability to enhance temporal reasoning abilities of different LLMs.\n| | Math-related task | Time-related task | Average  |\n| - | - | - | - |\n|LLaMA3-8B|81.4|79.6|80.5|\n|TIMO (LLaMA3-8B)| **82.6**| **88.6** | **85.6** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. In this paper, the authors discover the inner correlation between time and mathematics, where temporal reasoning could benefit from math instructions by systematically studying diverse temporal reasoning tasks.\n\n2. The authors make the first attempt to build a unified framework to address 38 temporal tasks. They propose a simple but effective self-critic temporal optimization method to strengthen the temporal reasoning capabilities comprehensively upon mastering mathematical reasoning capabilities.", "reasons_to_reject": "1.In section ‘Main Results’, there is lack of analysis on comparison with the other models.\n2. In section ‘Ablation Analysis of Framework’, there is lack of analysis on comparison with the GPT4 model. Why do the performance differences between TIMO and gpt4 differ on different tasks? The author should make appropriate analysis .\n3. In table 1, the result ‘83.0’ of column ‘Pure-time tasks-AMB.’ is not the best result. Suggest the authors to highlight the second results in the table as well.\n4. In case study, there is a certain ambiguity in the first case (Math-CoT lacks prerequisites ‘1652 Jan 1st is Monday’). Suggest the authors to replace other case.", "questions_to_authors": "None", "extra_scores": {}}, {"review_id": "lCzJBuqKkq", "reviewer": "Reviewer_P1tK", "summary": "This paper presents a framework to improve the general temporal reasoning capabilities of LLMs. The authors empirically verify the intuition that temporal reasoning is closely related to math reasoning, which motivates the proposed approach that adopts the LLM-as-Judge scheme using a math LLM and temporal reasoning instructions to produce temporal-related pairwise preference dataset, which is later adopted for training a DPO model called TIMO. The authors conduct experiments on TRAM (a temporal reasoning benchmark consisting of 10 categories) and show that TIMO outperforms the math LLM under most cases.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer P1tK,\n\nThanks for your feedback, which is crucial in improving this work.\n\nWe hope the below experimental results and explanation can mitigate your concerns and answer your questions.\n\n> Concern 1: Lack of comparison with non-math LLMs (e.g., vanilla LLaMA or general SFT LLaMA). The improvements shown by TIMO might result from the LLM-as-Judge framework rather than the math LLM.\n\nThanks for your suggestions!\n\nWe conduct experiments on vanilla LLaMA-7B and LLaMA2-7B-chat (as a general SFT LLaMA) to validate the importance of math LLM in our framework.\n\nAs shown in the table below, utilizing the math LLM outperforms vanilla LLaMA-7B and LLaMA2-7B-chat with average improvements of **3.6 and 7 scores**, respectively.\n\nSpecifically, in math-related tasks, our model achieves 5.5 and 10.5 scores higher than the other two models.\n\nThe above results indicate that using a math LLM plays a more important role in reasoning over time compared to self-critic temporal optimization (LLM-as-Judge framework).\n\n| | Math-related task | Time-related task | Average  |\n| ----| -| ---| ----|\n| TIMO (LLaMA2-7B)      | 58.4              | 79.7              | 69.1     |\n| TIMO (LLaMA2-7B-chat) | 53.4              | 78.1              | 65.7     |\n| TIMO (MathLLaMA-7B)   | **63.9**      | **81.5**          | **72.7** |\n\n> Concern 2: Evaluations on other temporal reasoning datasets could be more convincing.\n\n\nFollowing your comments, we evaluate TIMO on two temporal reasoning datasets to validate the effectiveness, i.e., MCTACO [1] and TempReason [2].\n\nAs shown in the table below, TIMO outperforms LLaMA2-7B and WizardMath-7B by **6.2 and 15.5 scores** on the TempReason task, respectively, and exceeds MAmmoTH-7B by **19.3 scores** on the MCTACO task.\n\nThese results indicate that the improvements of TIMO over math-LLM result from better general temporal reasoning rather than specialized adaptation to TRAM.\n\n|  | MCTACO   | TempReason |\n| -| -| --|\n| LLaMA2-7B     | 50.3     | 46.6       |\n| MAmmoTH-7B    | 37.0     | **52.8**   |\n| WizardMath-7B | 12.7     | 37.3       |\n| TIMO-7B       | **56.3** | **52.8**   |\n\n[1] Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models, ACL-23\n\n[2] “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding, EMNLP-19", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work empirically shows that temporal reasoning benefits from better math reasoning abilities. Such an assumption has been indirectly studied before, but no prior work has really proved it.\n\n2. The produced TIMO checkpoint, upon being publicly released, could be a good option for direct usage on temporal reasoning tasks.", "reasons_to_reject": "1. The final DPO step uses the data distilled and evaluated by the math-LLM through the LLM-as-Judge framework. However, there is no comparison with adopting a non-math LLM (e.g., vanilla LLaMA or generally-SFT LLaMA) for generating such data. Existing work on weak-to-strong generalization suggests that distilling data from a weaker or equivalent LLM benefits a stronger LLM. Thus, it is possible that the improvements shown by TIMO results from the LLM-as-Judge framework rather than the math LLM (which is considered as the main contribution of this work).\n\n2. Among all models that are compared against TIMO, TIMO is the only one that has been trained on the instructions from TRAM. So one concern is whether the improvements of TIMO over math-LLM result from better general temporal reasoning or specialized adaptation on TRAM. Some evaluations on other held-out temporal reasoning datasets could be much more convincing.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper has value and would contribute to the program in the conference. There were some concerns by one reviewer about not having experiments on held-out temporal reasoning datasets. The authors provided a proper rebuttal with new experiments that made the reviewer reconsider their position.
Decision: Accept