Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation
Authors: Qiang Zhang
Jason Naradowsky
Yusuke Miyao
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue system
time-aware dialogue model
long-term conversation generation
multi-session dialogue
TL;DR: We explore the idea of making dialogue models aware of time, and present a multi-session dialogue dataset in which the time between each session varies.

Abstract: Knowing how to end and resume conversations over time is a natural part of communication, allowing for discussions to span weeks, months, or years.  The duration of gaps between conversations dictates which topics are relevant and which questions to ask, and dialogue systems which do not explicitly model time may generate responses that are unnatural.  In this work we explore the idea of making dialogue models aware of time, and present GapChat, a multi-session dialogue dataset in which the time between each session varies.  While the dataset is constructed in real-time, progress on events in speakers' lives is simulated in order to create realistic dialogues occurring across a long timespan. We expose time information to the model and compare different representations of time and event progress.  In human evaluation we show that time-aware models perform better in metrics that judge the relevance of the chosen topics and the information gained from the conversation.

Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation
Qiang Zhang, Jason Naradowsky, Yusuke Miyao
Department of Computer Science
The University of Tokyo
(qiangzhang714, narad, yusuke| @is.s.uttokyo.ac.j
Abstract
Hey, how are you doing?
good. just busy with my doctorate thesis. It's tough, but could make progress on the literature review.
Knowing how to end and resume conversations over time is a natural part of communication, al- lowing for discussions span weeks, months, or years. The duration of gaps between con- versations dictates which topies are relevant and which questions ask, and dialogue sys- tems which do not explicitly model time may generate responses that are unnatural. this work we explore the idea of making dialogue models aware of time, and present GapChat, a multi-session dialogue dataset in which the time between each session varies. While the dataset constructed in real-time, progress on events in speakers" lives is simulated in order to create realistic dialogues occurring across expose time information the model and compare different represen- tations of time and event progress. In human evaluation we show that time-aware models per- form better in metrics that judge the relevance the chosen topics and the information gained from the conversation.
Speaker
Oh, it does sound like tough work, I'm glad that you can make some progress. Do you want to have lunch together?
Oh, maybe not today. I'll need to join a book reading event from now
Speaker
2 hours later
How the thesis writing, any progress?
Time unaware
told you had some progress on the literature review. It's been only 2 hours...
Speaker
How was the book reading event?
T Time aware
Yeah, was great. People shared lot of interesting books. You should also go there next time!
a long timespan.
Speaher
An illustration of the time aware dialogue models we propose. The time aware model talks about events that informative and effective tion considering the duration of the event and the gap between sessions.
Figure I:
conversa-
1 Introduction
As language models scale to unprecedented sizes, so too has their ability to generate reasonable and fluent responses to human dialogue. However, one limitation of large language models (LLMs) to- wards creating human-like like dialogue lies in the fun- damental distinction that human cognition is em- bodied, while LLMs are not. Any considerations which stem from the human physical experience may not be well-represented explicitly in the train- ing data dialogues of LLMs, as reiterating infor- mation known to both parties would violate the Gricean maxim of quantity (Grice, 1975), and may not be effectively utilized during generation.
lows human speakers to more accurately gauge which topics and questions will result in new infor- mation, leading to conversations which are more informative and appear more natural. For exam- ple (Figure 1), dissertation is time-consuming endcavor, if given two-hour gap between con-
versations, it is unlikely that significant progress has been made. In the context of potential infor- mation gain, asking for a progress update may be less useful than asking about an event with more expected progress within that time frame, such as lunch or a meeting. To enable dialogue models to behave similarly, we propose incorporating awareness of time it pertains (1) how much time has passed since the previous conversation. (2) what events were previously discussed, and (3) what progress of each event is expected over that duration.
In this paper we focus on one aspect of embod- ied experience time and the role that awareness of time plays in shaping realistic dialogue in multi- session conversations. We argue that an awareness of the passage of time between conversations al-

---

Previous work introduced the concept of multi- session chat (MSC) and dataset for the task where the discussion between two speakers is divided with gaps between sessions (Xu et al., 2022a). However, gaps between sessions are relatively short (1-7 hours, or 1-7 days), and event topics are de- rived from basic persona attributes (I have six cats) which prevents any discussion of long-term events event progress. Moreover, annotators vary be- tween sessions, which may have further reduced consistency.
et al., 2021) conversations. To assess the quality of follow-up questions posed by artificial agents, Ge et al. (2022) proposed measures based on Gricean maxims to capture aspects such as relevance, in- formativeness, truthfulness, clarity, and coherence. Motivated by this line of work, we explore the role that time awareness plays in gaining information ef- fectively by avoiding discussing non-mentionable topies.
Long-term Dialogue Model Research on long- term dialogue generation based on neural networks has primarily focused on means of storing and ac- cessing long-term information beyond the bounds of a limited context window. One approach to this problem is selecting valuable information for long- term storage, such as conversation summaries (Xu et al., 2022a) and persona attributes (Xu et al., 2022b). This approach involves training model which retrieve from storage information which is relevant to the current topic. An alternative retrieval, attention mechanisms can fulfill similar role (Zhang et al., 2022).
To remedy these issues and support long-term MSC research, we present GapChat, a dataset which extends the MSC dataset with additional sessions ! However, while chat in MSC is open- ended, conversations in GapChat are based on sim- ulated timelines. This design choice allows for two participants to create realistic and consis- tent conversation about long-term events, with gaps between sessions on the scale of days, weeks, or months, in a comparatively shorter period of time.
We explore multiple ways of making dialogue models time-aware, using information such as the ongoing events of speakers, the expected duration of the events, and the duration of time gap between sessions. Our contributions are as follows: (1) the creation of a new MSC-style dataset that simu- lates explicit gaps in communication, across vari- ous time-scales, to enable research into long-term MSC dialogue generation, (2) the exploration of different time-aware dialogue models, which incor- porate time information into their contexts to able varying degrees of time-based reasoning about discourse topics, and (3) we demonstrate via human evaluations of generated dialogues that the inclu- sion of time information improves the naturalness, informativeness, and relevance of conversations.
Dialogue Models on Topic Selection A poten- tial benefit time-awareness topic selection that results in more informative conversations. Previous approaches have used semantic relevance as the ba- sis for topic selection (Somasundaran et al., 2020; Xu et al., 2021), have selected based topic transition patterns (Xie et al., 2021; Ling et al., 2021). While such factors are shown to be effective in general discourse, in the context of multi-session discourse with gaps between conversations, com- monsense about previous topics may help rule out topics that are unlikely to result in new information being discussed. If this is a guiding principle in hu- man discourse, conversations that utilize temporal information should appear more natural.
2 Related Work
GapChat: A Time-Aware MSC Dataset
Pragmatics Theory of Communication Under- lying this work is the cooperative principle of com- munication (Grice, 1975), in which communica- tion goal-oriented, and where effective conver- sations are those which adhere to basic principles, including the desire to communicate only infor- mation which requires it. The cooperative prin- ciple has been used a means to evaluate both human-human (Eskritt et al., 2008; Kleinke, 2010) and human-computer (Lan et al., 2020; Langevin
The MSC dataset contains gaps between sessions and their durations are annotated. However, the gaps were short, and the annotators were changed between sessions. They were not instructed to pay attention to the relation between topics, events and the duration between gaps. T hus temporal infor- mation in MSC has been shown have minimal effect on model performance (Xu al., 2022a). However, we hypothesize that the use of a dataset with discussions focusing on more realistic long- term events may have a more significant impact
'https://g ttub.ccom///ooo..tt cttp///////////////o 2tt/ /MI (Thheimmm tree/main

---

Session 1
Session
Session
Session 4
Session
months
Build (3 wks)
Learrnngg Spanish (6 manths)
Watch movie (2 hrs)
Move new apartment (2 wks)
Take a basic cookery couree (2 wks)
Have short trip (2 days)
Argantina won the championship of World Cup
O A huge avalanche happened in Austria
FTX went bankrupt
An example of the simulated timelines for a speaker GapChat, Each speaker will engage in specific events that take certain period of time finish. Yellow rectangles and circles indicate life events and grey circles indicate world events.
Figure 2:
on model performance. To test this hypothesis. we create new conversation dataset, GapChat. We follow the general settings in MSC and simi- larly collect dialogue data via crowdsourcing. Thus GapChat can also be used as additional data for any task utilizing MSC. However, unlike MSC, conver- sations in GapChat are not completely open-ended, but are grounded to the hypothetical lives of the annotators, who are each given procedurally generated timeline of events in their lives. This notion of simulated life provides speakers with updates to events in their lives which are realis- tic with respect to the time which is said to have passed since the last chat session. Sessions are then scheduled to take place randomly at various posi- tions along the timeline, which may occur before, during, or after any particular event.
Reddit) using the query "How long will it usually take finish <event>?". Up to 5 answers are se- lected, and the average estimation used as the duration.
To include more diverse range of events in our dataset, we also use ChatGPT to generate addi- tional life events that vary in duration and domain. When generating with ChatGPT, we prompt the model "Generate a list of events or daily ac- tivities that require around <duration> finish," where <duration> represents the time needed to complete the event (e.g., 1-3 weeks). Manually and generated, a total of 50 life events with varying durations, including hours, days, weeks, months, and years are collected, with 10 events for each duration.
In order reflect changes in the topic over time, longer events are further subdivided into series of steps denoted as an event schedule. To do that, we ask ChatGPT to generate the steps towards fin- ishing each event via prompting (see Figure 11), For instance, in the case of attending a 3-day ba- sic cookery course, the introduction may require day to complete, learning knife skills may take another day, and learning cook basic dishes may take more days. And these three steps make up a schedule for the event of "attendinn basic cook- ery course". dialogue session could randomly be assigned to occur at any place within this event, or not at all. We create event schedules for each event that is with a duration that is longer than "1 hour". Each schedule of event consists of maximum of steps. For events longer than month, we generate two schedules with varying sets of steps, each representing different approach complet- ing these long-term events. Examples life events and their corresponding schedules can be found in Appendix A.
3.1 Events and Timelines
We define a timeline as a linear sequence of (pos- sibly overlapping) events, where each event is rep- resented by a string label (watch a movie) together with expected duration (2 hours). When two annotators are selected, series of events is sam- pled, and timelines created for both speakers by randomly ordering the sampled events. There two types of events: life events and world events.
Life Events A life event is an event that is said to happen in the person's life and they are exclu- sive to each speaker. We use life events to ensure the continuity in the multi-session conversations. Life events are crafted in two ways: manually, and generated with ChatGPT When crafting events manually, we collect example life events from on- line resources* 3, The durations of these events are estimated by searching online forums (e.g., Quora,
Potttss://iattee form. openai. con/ docs/ models/gpe-3-1
World Events Not all discussion topics pertain to the lives the speakers, and it is common to also have discussions on current events. To account for
11 life-events; nttps: www. dud eycourtpress. con/50-1ife-en
for for cor

---

#Session
Dialogues
Utterances
the following conversation, speaker and speaker 2 are updating their daily lives. In the conversation, both speakers might have mentioned some events. The events might have been finished are currently going on and just started. Extract only the events that speakers are currently going on and just started with the following conditions.
3
150
8,235
4
300
26,651
5
200
21,368
Total
650
56,254
#Conditions: Summarize events as nouns or noun phrases. such as "going for a tirp", "starting a MBA program", "laking an online course". "building swimming pool". Describe the events as brief as possible using the shortest summary.
Table 1: Data statistics of GapChat.
this and add further realism the conversations, we also include world events. We define a world event as a newsworthy event which takes place in the world, and is thus contained in the timelines of both speakers, such as "Argentina wins the World Cup championship" We collected 78 world events by extracting news article titles from Google News between November 2022 and January 2023. When used in timelines, world events happen in the same order as in the real world.
Generate the answers in the format "speaker <event 1>, <event 2> pp.Inspeaker__ 2: <event_1> <event 2>".
4. "speaker_1 and "speaker 2" must be in lowercase, and there must be a "'n' before "speaker_2 2".
5. If the speaker did not mention any events, generate:"
<speaker>: Not mentioned."
#Conversation:
(speaker_ 1: Hi, how are you doing?
speaker 2: I'm doing great. How about you? ...)
Figure 3: The prompt we use to extract events from a given conversation during inference.
3.2 Data Collection
We collect data via Amazon Mechanical Turk (AMT), beginning with the easier task of 3-session conversations and inviting reliable workers to take part in the collection of 4 and 5 session conver- sations. A total of 48 workers were selected to participate in the final data collection. We require fluency in English for all participants. Workers are paid $7 per hour average. The instructions and interface used for data collection can be found in Appendix C.
full duration of the event, speakers will be noti- fied that the event has been completed and new life events will be provided according to the timeline.
Session details & statistics A typical MSC ses- sion contains 12- 14 utterances, and this may make it difficult for speakers to engage in in-depth discus- sions before transitioning to new session. We ob- serve that many sessions in MSC immediately tinue the conversation of the previous session, es- sentially making the data no different than a longer single session. To address this, we design each session in GapChat to have at least 20 utterances and to start and end in natural way (e.g.. greetings and closures), and we encourage sessions to con- tinue past this minimum. Additionally, we instruct speakers to be mindful of the time gap between sessions as it may influence their behavior in the subsequent session. Table shows the statistics of GapChat. total we collect 56,254 utterances across 2,650 sessions. Each conversation consists of 3~55 sessions. Examples of 4-session conver- sation is available in Appendix B. For evaluation we split the data to Train/Valid/ with ratio of 0/0.0.1/0.2.
Initial session At the beginning of the initial ses- sion, each speaker is provided the first event in their timelines (e.g., "You just started attending basic cookery course, which would take about days"") as an initial event share in the conversa- tion. The initial events are randomly selected from the life events. Once the minimum session length is reached, either speaker can end the current session, move time forward, and create new session with a randomly generated time gap ranging from 10 minutes to year.
Subsequent sessions For all subsequent sessions, speakers are provided with updates on the progress of their ongoing events based the duration, steps of the events, and the time gap. If the time gap is shorter than the minimum required time to reach the next step in the schedule of the event, speak- ers will receive a message stating "No significant progress." Conversely, if the time gap covers the
4 Modeling Time
We propose a method of modeling time in multi- session dialogue systems. As shown in Figure 4, this consists of three procedures: 1) extracting on-

---

Dialogue History:
Progress label:
A
Hey, how are you doing?
B: writing doclorate thesis > significant progress.
RAG Model
book reading event -> finished.
I'm good. I'm just busy with
my doctorate thesis...
Computing Events
ChatGPT
New session:
join book
reading event today.
Progress
Prompt:
How was the book reading
Estimate duration
Yeah, it was great. People shared lot of interesting books. You should also go
Prompt:
Extracted events:
Extract events the
ChatGPT
B: writing doctorate thesis, reading event.
conversation ...
Extracting Ongoing Events
Time-aware Dialogue generation
Figure 4: The illustration of the process of modeling time with progress label. When using schedules, an additional step of prompting LLM for schedule information added.
going events, 2) computing events progress, and 3) time-aware dialogue generation.
Given of events provided by two people, speaker_ and speaker 2, estimate a typical time duration to finish each event. For each event, label it with a time duration tag in the following steps.
4.1 Extracting Ongoing Events
#Steps:
1. select a base time range from (hour, day. week, month, year) using the commonsense knowledge.
At training time, dialogue events are observed, since they pre-defined in timelines that are as- signed to the speakers. However, at inference time, it is necessary to extract ongoing events on-the-fly fly from the dialogue history. We phrase this as a text generation task where the input is the preceding dialogue history and the output a list of events discussed in the history. The events to extract are defined as in Section 3.1. For instance, a speaker mentioned "I'm currently working on research project until the end of the semester", the event "working research project" is extracted as life event. We extract events in prompt-based manner using ChatGPT, Different styles of prompts are ex- plored, as shown in Table 12~ 14 Appendix D. The prompt which resulted in the highest extraction performance is shown in Figure 3, and is used in the following experiments.
select a number that is associated with the base time range to form the final time duration tag.
3. generate the answer with <speaker beesker_dddd:ooo <event> >>
<number><base time range>.
4. use N/A the events for speaker are not provided. The generated text contains only the answer.
# Example:
List of ("ppekk 1": just started a one-year collage program. "speaker_ 2": getting driving license"]]
Answer: speaker 1: just started one-year collage program. -> yearlnspeaker 2: getting driving license
> months
#Question:
List of events:
("speaker 11:
Figure The prompt used to estimate event durations.
GPT as a knowledge base, and prompt it to provide typical duration for each event. Figure 5 shows the prompt we for estimating event duration.
4.2 Computing Events Progress
During training, we compare the event duration against the session gap to determine the progress of events. At inference time, we first estimate the duration of the events extracted in the previous procedure through prompting. We two styles of methods to represent the progress of the events: the progress labels and event schedules.
Using progress labels When using progress la- bels to represent the event progress, we formulate event progress as a discrete labeling task, where each extracted event in the previous procedure is labelled with one of five progress labels, (No sig- nificant progress, 1/4 finished, half finished, 3/4 finished, finished). Finer real-valued estimation of progress is possible, but was not pursued due to potential sparsity issues. Event durations are then compared with the time gap to calculate the progress labels for each event. For instance if the
Estimating event duration Event duration is es- timated by querying a knowledge base that has the temporal commonsense knowledge as the MC- TACO dataset (Zhou et al., 2019). We utilize Chat-

---

duration of the event "getting driver license" is "2 months" and the time gap is "6 weeks", , the event is given a progress label of "3/4 finished".. Although it is possible to prompt ChatGPT directly to gener- ate the progress labels, we calculate these labels to prevent potential inaccuracies or hallucinations that ChatGPT might produce regarding mathematical tasks (Bang et al., 2023).
A: Hey, how you doing?
B: I'm good. I'm just busy with my doctorate thesis.
B: I'll need to join a book reading event today. Events
B: writing doctorate thesis, book reading event.
Progress
Using event schedules As an alternative to the more numerically oriented progress labels, the schedules collected in Section 3.1 contain steps that required to finish event, and thus can also represent the progress of the events. We lever- age the schedules and split a schedule into two lists "finished" and "to-do". The "finished" part contains those steps that can be completed during the session gap, and the "to-do" part contains the remaining steps. For instance, a schedule of the "getting the driver license" contains 5 steps of "one week for learning rules, 2 weeks for practicing, 2 weeks for passing exams, one week for road check, one week for getting license." If the time gap is "two weeks," the "finished" list includes "one week for learning rules," and the remaining steps are added to the "to-do" list. The "finished" part of the schedule represents the current progress of the event.
B: writing doctorate thesis [no significant progress], book reading event [finished]. Gap hours
Table The sample input time-aware models.
MSC-RAG 2.7B 1024 as the initial model. A base- line model, RAG (FT), is also trained with only the dialogue history and extracted events but without time-aware information. All models trained the ParlAT platform in seq2seq style, with the dialogue context as the input and the response the label. The models are trained with NVIDIA A100 GPUs for 72 hours.
5 Experiments
4.3 Time-aware Dialogue Generation
Models are separated into two groups, time- unaware models and time-aware models. Human evaluation is conducted to compare all models against RAG (FT).
We RAG Retrieval-Auumeenter Generation) 2.7B model (Lewis et al., 2020) for dialogue gener- ation. RAG model utilizes retriever to retrieve related contexts stored as documents memory, making it effective at handling large collections of text. The truncation of this model is set to 1024, enabling it to encode more context.
5.1 Time-unaware Models
We include comparisons to a number of models which do not explicitly consider time gap and event duration information as baselines:
Documents and retriever Following MSC, we save the dialogue context as documents and retrieve with a DPR model (Karpukhin et al., 2020). Dif- ferent sessions are saved as separate documents in the memories encoded by the DPR model. When provided with the dialogue context, the top-5 docu- ments are retrieved for response generation.
MSC-RAG The RAG 2.7B model proposed by MSC (Xu et al., 2022a), where the model saves previous dialogue sessions and retrieves relevant information during generation.
RAG (FT) MSC-RAG model fine-tuned on GapChat. No time-aware information is provided in this model.
Training the model During training time, events and different types of time-aware information (progress labels and schedules) are combined with dialogue history context in manner similar to Persona-Chat (Zhang al., 2018) (Table 2).
5.2 Time-aware Models
TA-RAG Time-aware models based on the MSC-RAG model, fine-tuned on GapChat, and us- ing various time-aware information described in Section For the TA-RAG (progress) model, the
We train different time-aware RAG models (TA- RAG) by providing various time-and aware informa- tion (gaps, progress labels and schedules) using the
hhttps:/ppar...

---

Model
Human Ratings against RAG (FT)
Naturalness
Informativeness
Relevance
Time- Awareneess
Total
Time-
MSC-RAG
-3177
1.12
44.68
44..0
12.71
unaware
ChatGPT
7.64
-0.50
1.00
20.28
7.11
TA-RAG:
progress 9.02
5.62
13.75
44.82
18.30
schedule
9.16
4.06
15.22
54.84
20.82
Time- aware
both 15.15
6.32
20.78
50.56
23.20
ChatGPT:
gap only
10.52
2.18
3.12
16.66
8.12
progress 18.08
5.12
15.38
38.78
19.34
schedule 15.84
4.54
12.30
47.78
20.12
both 18.26
3.80
17.30
52.58
22.99
Table Human evaluation results of different models when compared against RAG (FT). Negative numbers indicate that the model performs worse than RAG (FT).
time-aware information provided in the style of progress labels, and for the TA-RAG (schedule) model, is in the style of event schedules. TA- RAG (both) indicates that both styles of time-aware information are provided.
Conversations are generated session by session, where the first utterances of the first session are seeded from manually crafted scripts to ensure the models start the conversation by sharing some events. In subsequent sessions, randomly selected time gaps and new events are provided together with the dialogue history of the previous session to the models. All models the same time gaps and events to ensure consistent experiment setting. Evaluation follows ACUTE-Eval (Li et al., 2019), where annotators are asked rate the con- versations generated by various models in com- parison to the baselines. In our experiments, we compare the models RAG (FT) and evaluate the conversations session by session. The annota- tors are provided the dialogue history of previous sessions, the event information and the time gap. They are then asked to select the conversations generated by one model over another. For each model, collect 150 multi-session conversations spanning different session lengths (50 for 3-session, 4-session, and 5 5-session scenarios, respectively).
5.3 ChatGPT
Besides the fine-tuned models, we also add Chat- GPT into our experiments to explore time- awareness. We consider both time-unaware and time-aware scenarios and prompt ChatGPT to gen- erate responses. In the time-unaware case, only di- alogue history and extracted events provided to the model. In the time-aware case, explore dif- ferent types of ChatGPT settings. In the "with gap only" type, in addition to the dialogue history and events, time gaps between sessions are explicitly given, enabling ChatGPT to recognize the engage in a multi-session conversation. In the "progress", "schedule" and "both" types, progress of events are also provided as in TA-RAG models.
5.4 Evaluation
Collecting dialogues Models are evaluated using the human evaluation by comparing the conversa- tions they generate. Conversations from all RAG models are generated in a self-chat style, where the RAG models engage in conversation with common BlenderBBot3B model. Self-chat has been shown perform comparably human-mon model con- versations (Smith et al., 2022) for evaluation pur- poses. the case of ChatGPT, we utilize prompts to obtain responses (Figure 13, Appendix E).
Human evaluation questionnaire In the eval- uation, annotators are asked a total of 11 ques- tions, grouped as 4 attributes: naturalness, infor- mativeness, relevance and time-awareness. Natu- ralness refers to the ability of the model to gen- erate conversations that feel like two friends up- dating each other their daily lives. Informa- tiveness evaluates whether the model frequently asks frustrating questions about event progress, affecting the information gain (Ge et al., 2022).

---

Session 2
Model Events: finding new apartment
Gap: 2 months
Events: wriling a grant proposal, prepare presentation.
Speaker Events: wrlling thesis.
Speaker Events: watch movie, take photography course.
(both)
(FT)
Hi, how are you doing?
Hey, how are you?
Hey, how are you?
I'm doing great. How about you? What you up
good. It's been while.
Hey, I'm doing great. like horror movies, what kind of you like?
How are you doing?
recenily starled taking photography course.
Have you finished your
photography course?
like action movies.
sounds looking for new apartment.
Not yet. preparing for the final lest.
Figure 6: A sample conversation generated by RAG (both) and RAG (FT), The models generate the conversations with the same events and time gap.
Annotators are asked, "Which speaker asks an- noying questions about events with no significant progress?" Relevance measures the model's s abil- ity to generate follow-up sessions related to men- tioned events by asking annotators "In which dia- logue does the speaker ask/talk more about relevant events?" Time-awareness assesses the model's abil- ity to identify time gaps and the progress of events by asking annotators "In which dialogue does the speaker identify time gaps more accurately?"
Between different types of time information, we find that progress labels contribute most to the enhancement of informativeness, but schedule information has relatively minor impact on the informativeness. This may be attributable the more direct manner in which this approach pro- vides information the model, as the progress towards an event represents the outcome of a time- based reasoning process that performed outside of the model. Using the progress representation would closely align with desired discourse ac- tion (e.g., do not select events with labels of 'no significant progress"), whereas the schedule repre- sentation requires the model to perform additional reasoning.
We conduct the evaluation AMT. Questions are phrased differently for each attribute and aver- age ratings are calculated (Appendix F). After pilot tests, 66 annotators are chosen for the final evalua- tion task. annotators are hired for each task of model comparison. We also request explanations for choices to validate responses. During analysis, we filter out evaluations with short working time (< 200 seconds) and unreasonable explanations (e.g., single-word, repetitive, unrelated to the conversa- tion, or copied from other text). The Fleiss' Kappa across all annotators is K 73.1%.
6.2 Analysis
Various gap duration We analyze the perfor- mance of the models with regard to specific gap durations, but do not find a consistent relationship between gap length and the evaluation measures (Appendix H). In some models (TA-RAG (both)) we find that naturalness and time-awareness are more stable different session gaps, whereas informativeness and relevance shows larger devi- ations (see Figure 7), but these deviations do not change the overall ranking of systems on each at- tribute.
6 Results & Analysis
6.1 Main Findings
Results of the human evaluation are shown in Ta- ble Our main finding is that time information improves overall naturalness of the conversa- tions, which supports the hypothesis that time- based reasoning of discourse topics is an important part of long-term multi- session dialogues. Figure 6 shows example in which TA-RAG (both) is able produce more natural conversations (more ex- amples in Appendix G). TA-RAG with one type time information performs worse than ChatGPT on naturalness, however, overperforms ChatGPT when both types of time information are added.
Time-awareness in ChatGPT ChatGPT con- siderably larger and trained on more data than any other model in our study, and may have induced better understanding of time. In fact, rely on ChatGPT as source of temporal common- sense when obtaining event durations. is then no surprise that ChatGPT exhibits certain level of time-awareness, however it does not score as high on this metric as TA-R TA-R models. Surprisingly,

---

naturalness informaiveness relevance imne-wwrenesss
Model
#Correct Events
75
60,87
49.43
45.78
RAG (FT)
17
50
TA-RAG (progress)
46
Reiiin
18.16
17.32
26.90
19.22
19.19 9.990:
TA-RAG (schedule)
42
13.76
0.48
TA-RAG (both)
44
0:92
3.86
-25
Table 4: Number of session-pairs in which the model selects correct events subsequent session. Total number of tested case is 60.
minutes
hours
days
waeks
months
Sersion gap
Figure 7: Performance of TA-RAG (both) over different session gaps.
formation, results in selection of more appropriate events in subsequent sessions (where the more de- sirable events are ones which are most likely cause discussion of new information).
we found that while adding information about the gap between sessions improves naturalness, infor- mativeness and relevance, it does not enhance the time-awareness of ChatGPT. Rather, we observed improved greetings and conclusions, and the inclu- sion of gaps appears have primarily improved the overall structure of the dialogue.
Conclusion
This work is a study of the role that temporal rea- soning plays in shaping dialogues over multiple sessions, and the extent to which temporal infor- mation is useful when integrated into existing dia- logue systems. We demonstrate via human evalua- tions that time-aware models generate text with im- proved naturalness, informativeness, and relevance, resulting better topic selection and improved text generation quality overall. The results emphasize the importance for future dialogue models to con- sider "time-awareness" achieving natural conversations. Additionally, our analysis shows that incorporating time-aware infor- mation also enhances the performance of existing LLMs such as ChatGPT.
ChatGPT outputs lengthy responses resulting in relatively low informativeness. To explore the impact of time information ChatGPT's perfor- mance, we use similar prompts, as depicted in Fig- ure 13, albeit with time information incorporated into the prompts. The inclusion of time information improves iime-awareness, demonstrating that this information is useful and not implicitly utilized to the same extent by the original model. In terms of informativeness, we observe that adding both types of information results in lower performance than only utilizing progress labels. We find that this is due to the longer length of responses, which anno- tators naturally consider less informative. When schedule information added, ChatGPT exhibits a tendency to incorporate schedule details into the reply, which increases the reply length.
an important factor in
Limitations
We rely existing LLMs (ChatGPT) to serve knowledgebases for temporal commonsense. The extracted event durations agreed with author intu- itions, but as they are data-driven they may not reflect reality and may give a false expectation of the time necessary to complete events. The events used in our dataset represent a small set of pos- sible events, and we do not know what coverage this system would have on events in real conversa- tions. Further, using this system requires ChatGPT in the loop to extract events during inference time, and while accuracy was high and the extracted text was sensible the events used in this study, there are inherent risks present when using an LLM for text generation, and incorporating that information downstream without human oversight.
Adding time information helps the model select events in subsequent sessions. We conducted an analysis of the generated conversations examine the extent to which the TA-RAG models effectively select appropriate events as topics in subsequent sessions. We manually review 60 randomly se- lected consecutive session-pairs (20 for each of 3-session, 4-session and 5-session conversations) and mcasured the quality of topic selection as the extent the subsequent session avoided addressing events that have a label of "no significant progress". Table 4 shows the results of the conversations gen- crated by different models. We observe that adding time information, especially the progress label in-

---

Acknowledgements
Yanxiang Ling, Fei Cai, Xuejun Hu, Jun Liu, Wanyu Chen, and Honghui Chen. 2021. Context-controlles topic-aware neural response generation for open- domain dialog systems. Information Processing & Management, 58(1:102239.
This work was partially supported by JSPS KAK- ENHI Grant Number 19H05692 and Moonshot R&D Program Grant Number JPMJMS2237_4.
Yao Lu, Max Bartolo, Alastair Moore, Sehastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
References
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning. hallucination, and interactivity, arXiv preprint 2xXv:2332.04022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022. Reframing instructional prompts to GPTk's 's language. In Find- ings of the Association for Computational Linguistics: ACI. 2022, pages 589-612, Dublin, Ireland. Associa- tion for Computational Linguistics.
Michelle Eskritt, Juanita Whalen, and Kang Lee. 2008, Preschoolers can recognize violations of the gricean maxims. British Journal of Developmental Psychol- ogy, 26(3):335-44,
Yubin Ge, Ziang Xiao, Jana Diesner, Heng Ji, Karrie Karahalios, and Hari Sundaram. 2022. What should i ask: A knowledge-driver approach for follow-up questions generation in conversational surveys. arXiv preprint ar2.::2205.1090:00
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Herbert Grice. 1975. Logic and conversation. In Speech acts, pages 41-58. Brill.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3769-6781.
Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston, 2022. Human evaluation of conversations is an open problem: com- paring the sensitivity of various methods for eval- uating dialogue agents. In Proceedings of the 4th Workshop on NI.P for Conversational AI, pages 77- 97, Dublin, Ireland. Association for Computational Linguistics.
Online. Association for Computational Linguistics.
Sonja Kleinke. 2010. Speaker activity and grice's max- ims of conversation at the interface of pragmatics and cognitive linguistics. Journal of pragmatics, 42( (222)333453336
Swapna Somasundaran et al. 2020. Two-level trans- former and auxiliary coherence modeling for im- proved text segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 7797-7804.
Tian Lan, Xian-Ling Mao, Wei Wei, Xiaoyan Gao, and Heyan Huang. 2020. Pone: A novel automatic eval- uation metric for open-domain generative dialogue systems. ACM Transactions Information Systems (TOIS), 3911:1-337
Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, and Ann Copestake. 2021. Tiage: A benchmark for topic-shift aware dialog modeling. arXiv preprint 209.:2210..4556.
Raina Langevin, Ross Lordon, Thi Avrahami, Ben- jamin R Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic evaluation of conversational agents. In Pro- ceedings of the 2021 CHI Conference Human Factors in Computing Systems, pages 1-15.
Jing Xu, Arthur Szlam, and Jason Weston. 2022a. Be- yond goldfish memory: Long-term open-domain con- versation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5180-5197, Dublin, Ireland. Association for Computational Linguistics.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, al. 2020. Retrieval-augmented generation for knowledge-intensivee nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng- Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 20226. Long time no see! open-domain conversation with long-term persona memory. In Findings of the As- sociation for Computational Linguisties: ACL 2022, pages 2639-2650, Dublin, Ireland, Association for Computational Linguistics.
Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with opti- mized questions and multi-turn comparisons. arXiy preprint arxiv: T00i:190000000 1909003087.

---

Xu, Hai Zhao, and Zhuosheng Zhang. 2021. Topic- aware multi-ture turn dialogue modeling. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 14176-14184.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? Proceedings the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Tong Zhang, Yong Liu, Boyang Li, Zhiwei 7eng, Peng- wei Wang, Yuan You, Chunyan Miao, and Lizhen Cui. 2022. History-awan hierarchical transformer for multi-session open-domain dialogue system. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 3395-3400 53407, Abu Dhabi, United Arah Emirates. Association for Computational Lin- guistics.
Ben Zhou, Daniel Khashabi, Qiang Ning. and Dan Roth. 2019. "going on vacation" takes longer than "go- ing for a walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3363-3369, Hong Kong, China, Association for Computational Linguistics.

---

A Samples of events in simulated timelines
D Prompts for modeling time
D.1 Prompts we used in our experiment settings
Table and are examples of the life events. An event can have different schedules that lead to dif- ferent consequences/ends. We use these schedules to increase the diversity of the directions of devel- oping the ongoing events. For each life event, pre-define the duration and use ChatGPT to gen- erate steps of a schedule for the event within the duration via prompts. An example prompt can be found in Figure 12. Table shows an example of the world event.
Figure 11 is the prompt we use to generate the steps (schedule) towards finishing life event in our training data.
Figure 12 the prompt we use generate short schedules for given events. These prompts are used as they show best performance in our tests.
D.2 Prompts we tried with different styles
We try different types of prompts following pre- vious works researching on the factors that could affect the performance of prompting results (Schick and Schütze, 2021; Lu al., 2022; Mishra et al., 2022). Only few-shot prompting method is ex- plored because we require the generated events be in certain format (Table 12~144).
Samples of GapChat
Table 8~~11 show a 4-session sample conversation we collect. For each session in the dataset, we pro- vide the progress of events and the utterances. For subsequent sessions, we also provide the session gap since the previous session.
For the cuestion-ansseriigg style. we only fix the first two questions and dynamically select the questions based on the answer to previous question. For instance, if the answer to Question I is "Yes", will choose "What are the events that speaker A is engaging?" as Question 3. When asking question, all the question and answers previous questions are included part of the prompt.
C Instruction and interface for data collection
Figure 8 andd9 show the instructions we use for data collection. We explicitly ask speakers to follow events and timelines the instruction. We also raise the point that any offensive, abusive content will not be allowed and the speakers are able to report and stop and task any time. We also provide positive and negative examples to demonstrate how the data should look like for the speakers.
To generate the conversations, we use a match- ing system to first randomly match two participants. After matching, the participants are redirected to the chat room as shown in Figure 10. At the be- ginning of the conversation, each participant will be shown their initial events for the participants to talk about. When the session reaches the maximum length, either participant could move the time for- ward for a random time gap by clicking the "End current session" button. And for the new session, we show different types of information to both par- ticipants. Finished progress indicates the life events they were engaging in the previous session. The progress is represented as the schedule steps based on the time gap. We also show some random life events and world events defined in the timeline. Fi- nally, also provide some future plans for the participants. The future plans are either some fu- ture schedule steps for unfinished progress or new events in the timeline the previous events are finished.

---

Life event
You just started preparing and executing social media marketing campaign for your company, which would take about months.
Duration: months
Schedule
Steps:
weeks for researching the audience and markets, one week for creating engaging content that aligns with the campaign goals, one week for designing and setting up the campaign, weeks for executing and optimizing the campaign, one week for analyzing the campaign data, one week for making adjustments to the content and strategies as necessary.
Table An example the schedules simulated timelines.
Life event
You just started writing your doctor thesis, which would take about one year.
Duration: year
Schedule
Steps:
one month for reviewing the guidelines and outlining the structure of the thesis, months for writing the introduction, months for writing the remaining chapters, months for revising the thesis based on feedback from other colleagues, one month for preparing the formatting and citations, one month for final tough and submitting to committee, one month for addressing comments requested by the committee members and defending the thesis.
Schedule 2
Steps:
one month for reviewing the guidelines and outlining the structure of the thesis, months for writing the introduction, months for writing the remaining chapters, months for revising the whole thesis for clarity, coherence and flow, one month for revising the thesis based on feedback, 2 months for preparing and submitting a conference paper, one month for preparing the extension procedures of the doctorate program.
Table 6: An example of multiple schedules that lead to different results of the same life event.
World events
Argentina won the championship of World Cup. Prince Harry reveals whether he's circumcised in bombshell 'Spare' memoir
US closes in Bankman-Fiin Fried Inner Circle with probe of FTX chief engineer.
Table 7: Examples of world events.

---

Session
Events:
Speaker 1: You just started organizing the storage room, which would take about 2 days.
Speaker 2: You are about to have short vacation to the beach, which would take about days.
Speaker 1: Hi! How are you doing? I heard you' re going on a short vacation to the beach. That sounds exciting!
Speaker 2: Hey! Yes, I'm really looking forward to it. It's been while since I've had a break. Speaker I: Definitely, it's always nice to take some time off. Do you have any plans for what you'll do while you're there?
Speaker 2: Not really, just planning to relax and soak up some sun. Maybe take a dip in the ocean.
Speaker 1: That sounds like the perfect way to unwind. Have you packed everything you need for the trip?
Speaker 2: Yes, I have everything sorted. I just need to doublc-check that haven't 't forgotten anything important.
Speaker 1: That's great to hear. On my end, I've been organizing my storage room. It's been a bit of mess, I've been taking some time to sort through everything.
Speaker 2: Oh wow, that sounds like quite the task. How's it going so far?
Speaker 1: It's definitely been timecconummnn, but it's starting look much better. I've been able clear out a lot of clutter and create more space.
Speaker 2: That's great! It always feels good to have more organized living space.
Speaker 1: Absolutely. Speaking of which, have you been keeping with your own home organization?
Speaker 2: Yeah, I've been trying to keep things tidy. It's a never-ending process though!
Speaker 1: Tell me about it. It always feels like there's something to clean organize. Any other plans for when you return from your vacation?
Speaker 2: Not really, just getting back into the routine of things. How about you?
Speaker 1: I'm planning work on some writing projects when finish organizing the storage room. I've been meaning to get back into for a while.
Speaker 2: Oh, that's interesting! What kind of writing do you do?
Speaker 1: Mostly fiction, but I've been wanting to try my hand at some non-fiction pieces as well.
Speaker 2: That sounds like lot of fun. Let me know if you need any help or feedback.
Speaker 1: Thanks, appreciate it! How about we check back in after your vacation and see how everything's been going?
Speaker 2: Sounds good me! Have good one.
Speaker 1: You too, enjoy the beach! Talk to you soon.
Speaker 2: Thank you..
Table 8: Session of sample conversation collect.

---

Session 2
Gap: week.
Events:
Speaker 1: You just started to prepare for marathon context, which would take about one year.
Speaker 2: You are about to visit the nearby town for about two days.
Speaker 1: Hey! It's been a week since we last talked. How have you been?
Speaker 2: Oh, hi! I've been great, thanks. How about you?
Speaker 1: I'm doing well, thanks for asking. By the way, how your beach vacation?
Speaker 2: was amazing! The beach was so beautiful and the weather was perfect. even got nice tan.
Speaker That's s awesome to hear! So, finished organizing the storage room, and now I've started preparing for a marathon contest that will take about year. What about you? Did you have any progress in your recent events?
Speaker 2: That sounds interesting! Well, I've been studying for certification exam, and also planned and executed fundraising campaign for my local community center.
Speaker 1: Wow, you' 've been busy! How did the fundraising campaign go? Speaker 2: went really well! We managed to raise a good amount of money, and the community center was very grateful for our help. Speaker 1: That's s great hear!
Speaker 2: Same great to hear!
Speaker 1: By the way, have you heard about ChatGPT's Al making puzzles that will make you want to throw brickbats?
Speaker 2: No. I haven't.
Speaker 1: Well, apparently they' re really challenging and frustrating. but also addictive.
Speaker 2: What are these puzzles about?
Speaker Some people are saying they can't stop playing them.
Speaker 2: Oh, I see. That sounds like fun, but also kind frustrating.
Anyway, what are your future plans for the next 6 months?
Speaker I: I'm planning to work on personal project and pursue passion that I've been neglecting for while. How about you?
Speaker 2: I'm planning to spend Monday and Tuesday working presentation that I have give on Wednesday.
Speaker 1: Good luck with that! Speaker It's going to be a busy couple of days, but I'm looking forward to it.
Speaker Let me know how it goes.
Table 9; Session 2 of sample conversation collect.

---

Session 3
Gap: 18 hours.
Events:
Speaker 1: You watch TV dramas.
You play video games.
Speaker 2: Tour the town's main attractions.
Have lunch at local restaurant.
Speaker 1: Hey, how's s it going?
Speaker 2: I've been busy with lot of things
Speaker 1: It's been almost day since we last talked. What have you been up to?
Speaker 2: Nothing really exciting to talk about. How about you?
Speaker 1: Well, I've been working my storage room organization project.
Speaker That's great!
Speaker 1: finished it yesterday, and now I'm looking forward to starting
something new.
Speaker 2: Congratulations on finishing it. So, what's your next project going to be?
Speaker 1: I'm planning to participate in marathon that's going to happen next year, so just started preparing for it. Speaker 2: Wow, that sounds like big challenge.
Speaker 1: Yes, my goal is complete it within specific time frame.
Speaker 2: Do you have any specific goals for the marathon?
Speaker 1: I'm also planning set smaller goals for myself, like improving my running time and endurance.
Speaker That's impressive. Good luck with your training. By the way, did you hear about the latest news on Microsoft and Activision? Speaker 1: No, what happened?
Speaker 2: Apparently, they backed off from their aggressive claim in the FTC case. It's quite interesting
how these big companies handle legal issues.
Speaker I: Hmm, that's interesting indeed. haven't 't been keeping up with the news much lately, but did watch some TV dramas and play some video games during my free time.
Speaker That's cool. I also spent some time assembling furniture that bought last week. It was quite challenging, but I'm happy with the end result.
Speaker 1: Oh, nice! What kind of furniture did you assemble?
Speaker 2: It was bookshelf and cabinet for my study room. It took me a while to figure out the instructions, but I managed to get it done.
Speaker 1: Good for you! By the way, I'm planning to set aside some time to review and reflect on my personal and professional goals for the year and make any necessary adjustments. Do you have any plans for the next few days?
Speaker Yes, have work on a proposal that's due on Friday, so I'm going to spend Wednesday and Thursday on it. How about you? Speaker 1: I'm planning to choose the marathon and make a commitment to participate. I'm also planning to set realistic goals for myself and develop a plan to achieve them. Speaker 2: Sounds like you're re making progress. Keep up the good work!
Table 10: Session 3 of an sample conversation we collect.

---

Session 4
Gap: 3 hours.
Events:
Speaker 1: You do some housecleaning.
You play with your pet.
Speaker 2: You take a bath.
You do board game with your family.
Speaker I: Hey, it's good to talk to you again!
Speaker 2: Hey! It's been an interesting day.
Speaker I: How have you been spending your time since last talked?
Speaker 2: I've been watching some TV dramas, playing some video games, and assembling some furniture. What about you?
Speaker Sounds like a nice day. I've been working on my presentation, and it's going pretty well so far.
Speaker 2: Thanks for letting me know. On my end, I'm planning to set aside some time to review and reflect on my personal and professional goals for the year. Do you have any tips how to do that effectively?
Speaker 1: That's great idea! One tip have is to make sure you set aside enough time for reflection, and really focus on what's most important you. You might also want consider setting some specific, measurable goals to help you achieve your overall objectives.
Speaker 2: That's good point. I'm also planning to choose a marathon and make a commitment to participate, as well as set some realistic goals for myself and develop plan to achieve those goals. Do you have any experience with marathon training? Speaker I: Not personally, but have friends who have trained for marathons before. It can be challenging but rewarding experience.
Speaker 2: I'm excited to take on the challenge! By the way, also wrote a short story this week. Would you like hear about it?
Speaker Of course! I'm always interested in hearing about your creative projects. Speaker 2: It's suspenseful story about a woman who becomes trapped in an elevator with a stranger who may or may not be dangerous. Speaker 1: That sounds really interesting!
Speaker 2: had a lot of fun writing it.
Speaker 1: Do you plan doing anything with the story, like submitting it for publication sharing with friends?
Speaker 2: I'm considering submitting some literary magazines, but I haven't decided yet.
Speaker That sounds like a good plan.
Speaker 2: might also share it with some friends to get their feedback. Speaker I'm sure will be well-received.
Table I: Session 4 of sample conversation collect.

---

Task Description
lask, you natural convanution specific events.
the events are some act vetae//eeets
enganiing at the curent time, lng, writng novel haking a
conversatio suut--esston
be subsequent stosions you roscive ovents you need
beginning first
progress these
Due to the various time duration between the semsinn, some eventa might have significant progresa shere, and some may not. and what about naving Du NOT walk about cstalls about this unk leap. Tris HIT inimssiing
skillc knowledge while close attention your
Please
check negetive ecamples.
Instructions
Step 1: Match
When you are ready for the uslk hi the Start Matching button at the botton of this page.
Step 2: Confirm your initial progress events
Hour
Post
Enl Corrent Sociann
Sestions you howe Freshech
Ieital Been Varjint stabed planinng vilowo russ woild take about
Bolone chatling will read to confim initial event
wotting your
year
nnverration, you can raity share this evert with ooother partioipant planning to reletie musk othm nost
Eg. Oh,
Step Have the conversation
Clat Heckor
You: hells mes medt you.
How ane doing?
Part
End Comment Scuaion
fnathed
Ingut
to
message.
con check the sistory any during the took
Your messages oau NOT be stteted ence sent, You can calk freeys to make the conversation naturt
Howerer, remomber trutyou
progress your
2. chare the progress of your events you're asked.
Step 4: End current session and move the time forward
That
Part
Eril Current Secsion
Sartini you have
loot We rocomt mend all partidoonts to ene the vess on ds ratura es possible
ttogother with speaker)
button becomes enabled
rood
Per iispanta provise high cialoque data be rewarded.
By di dicking zhe End Current Seasion button, you will movve the dins farward with pariod tima (oq. weok movdss ste) and start sossion
Figure 8: The instruction we use for data collection.

---

Step Confirm the progress and events
month has/have passed During this time, the following events happened.
Finished Progress:
Think about
album.
Life Events:
speak Spanish.
You launch buisness and
venture.
You have a wedding anniversary.
Argentina wan the championship of World Cup.
Fatal winter storm leeves millions without power North America with -I5C tempenetures. Coming For 4010k0 Accounts.
Retirement: New
Future Plans:
Begin writing recording songs.
Select the 10 best
the album.
Begin producing the tracks.
planning
your
Mondey
progress project
You are planning take Friday off and spend the day with your family and friends.
new session will provide you the fellewing info mation:
Progress describes the progress your inttial evend.
events are some events you could have done durinng
(something heppen to you).
World
happen world (news).
Future plans somelhing will do from
Select what events to talk about based on your social sidlls each session.
every
Instoac, conversation natural acssibie while using events smartly.
Repeat step 3-5 until the end of the multi-session conversation. You need to finish at least 4 sessions to complete this HIT.
Evaluation Criteria (Important)
events, than having frea chitchats. however, you responsible make your Fluent English, Your must communicatte in English fluently.
events/ progress.
Naturalness.
Non-offersive, You are not allowed post any offensive, abusive and harmiful messages in the conversation.
Examples
What we want:
progress of appropriate
Session 1: talked ebout thesis writigg and watching movie..
Time gap: months
Session::
Hi its been quite long time, hows everything?
i'm doing great. how about you?
Great, thanks acking How your wriling? progress?
Don't ask about the movic, because it's been months
What we do NOT want:
What's your topic?
My topic about having a vocation.
Hi how you?
Il. have trip.
deing great.
3. Filler utterances:
Hi.
Hi.
HL
Hey, de have beyfriend?
QR
your name?
Figure 9: The instruction use for data collection.

---

matched with partner. You the conversation the and Post button.
Chat history
Your partner:: And really enjoy the processes. think that's great experience.
hear thee,, grmal the hunoee inillding
Your Thanks, you luck your campaign.
You thanks, nice talking with you, see
Your partnnr:: : See you.
year ago)
Your message...
Lnd Current Session
Sessions you have linished:
year
During this the events
Finished Progress:
You finished the previous progress and started the following new event
You just started to write a new business plan, which would take about 2 weeks.
Events:
get divorced.
You finish the training process
World Events:
Nintendo Slashes Prices Switch eShop Games New Year Sale (North America)
megalive COVID mandslorr countries.
Future Plans:
You are planning spend time volunteoring local charity organization once week throughout the month.
You are planning attend professional conference the midsle of the month learn about the latest industry developments.
Figure 10: interface for speakers to have conversations.

---

Instruction description
In the following conversation, the speakers are engaging in some events that take a certain amount of time. Extract such events and estimate the expected time to finish these events.
Instances +N
Conversation:
A: Hi how are you?
B: Yes I am fine and how are you doing today?
A: Doing good. What is the plan for tonight?
B: Not yet planed for something. just started with preparing and executing a social media marketing campaign for my company.
A: Oh are you busy in that?
Events:
executing social media marketing (about months)
Question
A: Hi, how are you doing?
B: I'm doing great, how about you?
A: I'm also doing good. I'm just busy with my paper writing the deadline is approaching.
Events:
<extracted events>
Table 12: An example of few-shot prompting with information explaining
Instances *N
Conversation:
A: Hi how you?
B: Yes I am fine and how are you doing today?
Doing good. What the plan for tonight?
Not yet planed for something. I just started with preparing and executing a social media marketing campaign for my company.
A: Oh are you busy in that?
Events:
In the above conversation, speakers talked about the events they are engaging. A is engaging in something is not mentioned. B is engaging in executing social media marketing,
which takes about 3 months.
Question
A: Hi, how are you doing?
B: I'm doing great, how about you?
A: I'm also doing good. I'm just busy with my paper writing the deadline is approaching.
Events:
In the above conversation, speakers talked about the events they are engaging. is engaging in
engaging in
Table 13: An example of few-shot prompting for slot filling, where the LLM required to fill in the blanks according to given instances.

---

Instances *N
Conversation:
A: Hi how are you?
B: Yes am fine and how are you doing today?
A: Doing good. What is the plan for tonight?
B: Not yet planed for something. just started with preparing and executing social media marketing campaign for my company.
A: Oh are you busy in that?
Questions
Question 1:
Did speaker A mention any events that speaker A is engaging? Answer with Yes or No (No)
Answer:
Question 2:
Did speaker B mention any events that speaker B is engaging? Answer with Yes No
Answer: (Yes)
Question 3:
What are the events that speaker B is engaging? Answer the the content of the event and an estimated time to finish that event.
Answer: Speaker B is engaging in executing social media marketing, which takes about 3 months to finish.
Question 4:
Give a rough schedule of the events that B is engaging within the estimated time.
Answer:
Week 1-4: Research the target audience and their social media habits. Identify the platforms
that the target audience uses most.
Week 5: Create high-quality, engaging content that aligns with the campaign goals and appeals to the target audience.
Table 14: An example of question answering style prompt to extract the events and their duration.

---

Given an event and rough duration finish this event, generate short schedule for finishing this event. it requires more information get the schedule, roughly estimate one.
Generate the answer with the following requirements.
#requirements: 1. The generated schedule should be finished within duration of event.
2. The format should be the same as Answer shown
in #Example.
3. Each schedule has at least steps.
#Example:
event: getting driving license
duration: 2 months
Answer: one week for learning rules, 2 weeks for practicing, 2 weeks for passing exams, one week for road check, one week for getting license # Question:
event: (...}
Figure I1:
generate the steps towards finishing life event.
The prompt
Given list of events, generate short schedule for finishing each event in JSON forma.. If it requires more information to get the schedule. roughly estimate Generate the answer with the following requirements. #requirements:
1. Must be valid json file that be parsed python
json package. Pay attention the commas.
2. The format should be the
the Answer shown
in the #Example.
3. Each field the Answer list.
#Example:
events: ('speaker 1": [']ust started a one-year collage
program. 'speaker 2": ["getting driving license"]]
Answer: ("speaker 1": ['1 month initiating, 2 months basic courses, months for main courses, months selecting thesis topics. months for finishing thesis, month preparing defense. "J, "speaker_ 2": ("one week for learning rules, weeks for practicing. 2 weeks passing exams, one week for road check, one week getting license"). }}}
Question:
List events: ("speaker_ 1": .
Figure 12: The prompt use to get schedules for the pre-defined events during training and extracted events during inference.

---

First
the reason of their choices. The justifications are used for both filtering out invalid answers and ana- lyzing users feedback. We consider the following two types of annotations as invalid.
You are having multi-session conversation your friend with the following conditions example.
# conditions:
updating your
You are aware the rough time estimation to finish different events.
First, we remove those annotations that are with an extremely short working time. Given the length of the dialogues and questions, we consider it im- possible to finish the annotation within 200 seconds, Therefore, any annotations that are with work du- ration less than 200 seconds are directly filtered out. Second, we remove those annotations that are with unrcasonable justifications. For instance anno- tators may give I'good", "good", "good", "good"] as justifications, which are obvious signs of the worker not engaging in the task. We perform pilot tests to select good annotators for our tasks. The annotators required to have finished more than 2000 tasks, with an approval rate over 97% and be fluent in English.
# your events:
You are currently working on research project, which takes about
3 monthe.
weeks. You are planning to have short vocation to the beach side during this weekend.
finding apartment
...
Subsequent session:
It's been (two weeks) and friend having another conversation
with you. Continue conversation with the following conditions.
# conditions:
updating your
2. You are aware of the rough time estimation to finish different
your events:
(1. You started taking an online course to learn programminng which takes around months. You are planning watch movie which takes around hours.)
Figure 13: Prompts for generating conversations from ChatGT (with gap). Content in ( replaced with
same events and time gaps in the collection process other RAG models.
G Sample Conversations
Table 16~18 18 show an example of the conversations generated by TA-RAG (both) (Speaker B).
E Prompt for collecting conversations from ChatGPT
Figure 13 is the prompt we use for collecting con- versations from ChatGPT. We use two prompts to generate the first session and subsequent sessions. For the first session, we prompt ChatGPT to have a multi-session conversation with the same event settings as in other TA-RAG models. This prompt contains the conditions and events information. Af- ter generating the first session, we use a different prompt for generating the subsequent sessions. In the time-aware case of this prompt, we explicitly tell ChatGPT that there is time gap between this session and the previous session and previous di- alogue history is included as part of the prompt. First several utterances will be removed when the conversation is longer than the maximum input of ChatGPT. When generating time-ane aware ChalGPT conversations, we add another section of progress labels or schedules into the prompt we use for gen- erating subsequent sessions.
F Human Evaluation Questionnaire
We ask the annotators in total 11 questions (Ta- ble 15) and ask them choose one system over another one. For question Q3, Q6, Q9 and Q11 we ask annotators to provide justifications to explain

---

Questions
Naturalness
QI. Which dialogue do you think is more natural like two friends updating their daily life?
Q2. Which dialogue do you think is more like a dialogue between normal friends?
Q3. In which dialogue do you think the speaker B talks more naturally?
Informativeness
Q4. Which dialogue do you think provide more information about the speakers' daily events?
Q5. In which dialogue do you think the speaker B asks annoying questions more frequently about events that have no significant progress?
Q6. In which dialogue do you think the speaker B cares more about speaker A's daily events?
Relevance
Q7. Does the speaker asks/talks about relevant events?
Q8. Which dialogue do you think sticks to the topics of updating events?
Q9. In which dialogue do you think the speaker chooses natural, relevant events to talk about?
Q10. which dialogue do you think speaker can identify a time gap?
Q11. In which dialogue do you think speaker B is aware of the progress of relevant events?
Table 15: The questions we ask annotators for the human evaluation.

---

Events:
A: You watch movie.
You are going to visit your grandparents.
B: You plan to go music festival.
A: am doing well. love the outdoors as well. What beach did you go to?
B: do you have any plans for the upcoming weekend? I'm planning go to festival.
A: That sounds like fun. am going concert this weekend. What kind of music do you
listen to?
What about you? What do you do for fun? I like dance and go to concerts.
A: like lot of different kinds of music. My favorite band Led Zeppelin.
B: what is your favorite thing to do the evening? I love to go dancing with my friends.
A: like going out to eat with my family. We like to eat out a lot. Do you have favorite
food?
B: That's great. also like to go out with friends. like dancing and dancing.
A: Dancing is great way get out and have fun. What type of dancing do you enjoy?
B: love dancing, but I don't 't dance much. I'm more of outdoor person.
A: Outdoor dancing great too. It is so much fun be out in nature. love that movie. It's one of my all time favorites. Have you seen it?
A: No, haven't t seen it. have heard good though. What is it about?
B: love watching movies and listening to music. What's 's your favorite genre of music to
dance to?
A: I love pop music. I listen to it all the time while I work out at the gym.
B: That's nice. What about your family? Are you planning to do anything special for the holidays?
A: My family is great. We are going visit my grandparents. They live in Florida.
Do you like watch movies or tv shows? am a big fan of the genre.
like movies and tv shows too. What are some your favorite shows to watch on tv?
I'm going to watch a movie with my brothers and sisters. will be fun.
A: What movie are you going to see with your siblings? hope it's a good movie.
Table Session of an conversation generated by TA-RAG (both).

---

Gap: week
Events:
A: You watch tv dramas.
B: You plan to go hiking.
B: what about you what are your plans for this weekend? What are you up to? plan to go hiking.
A: Hiking sounds like good idea. I might go for a hike too. I've never been hiking before.
B: how were your grandparents? are they doing great?
A: Thanks for asking, they are doing great. And I had great time with them. We watched
Game of Thrones.
B: that's great, I'm glad hear that. How about you, what's going on?
A: just finished watching the last season House of Cards. It was great show.
B: yes, like too, it is a romantic comedy and romance comedy.
A: It really is. I can't wait for the new season to come out 2019.
B: do you know any good romance movies? I really like romance and thrillers movies.
A: do! I love The Notebook. think it's my favorite movie of all time.
B: love romantic movies, especially romantic ones. just watched a movie about romance.
A: That sounds interesting. I'll have to check it out. What did you think of the movie?
like romantic drama and thriller movies, do you also like those genres of movies?
A: do like those types of movies. especially like the thrill of a good thriller.
B: yes, love romance, action, adventure, thriller, mystery, suspense.
A: Have you ever seen the movic "The Sixth Sense" by David Suchet?
Table Session 2 of an conversation generated by TA-RAG (both).
Gap: 2 months
Events:
A: You prepare for the coming tests.
You just started to learn to play piano, which takes about 5 months.
I'm doing great, thanks for asking. How is your day going? What have you been up to lately?
A: My day is going well, thank you! I've been studying for a test have tomorrow.
B: what are you planing to do for the weekend? am planning to see movie.
A: I plan on going to the movies as well! I'm looking forward to seeing the new Star Wars movie!
B: do you want to go with me? can take you to the theater to see the movie.
A: That would be great! I'll bring my boyfriend well, he's the only one I've ever gone with.
B: like watching movies, tv shows, documentaries, documentarys, series and documentaries.
A: love documentaries! watch them every night before go bed. Whats your favorite?
do you have any other hobbies hobbies? just started learning how to play the piano.
A: That's so cool! play the drums. I started when was a teenager.
B: I'm planning to take trip to the beach in the next few months. hope you can join me.
A: would love to! I have never been to beach before. Where will you be going?
B: It is place somewhere near to the south east coast.
A: That sounds like lot of fun. I hope could go together with you.
Table 18: Session 3 of an conversation generated by TA-RAG (both).

---

H TA-RAG model's performance over different session gaps
Table 19 shows the numbers of different session gaps in evaluation experiments. Figure 14 and I5 are the performance of TAG models with differ- ent time-aware information over different session gaps. Although we are able to observe more stable naturalness for TA-RAG (both) model, that does not hold TA-RAG (schedule). Informativeness is also more stable in TA-RAG (progress), however shows larger deviation for TA-RAG (both) and TA- RAG (schedule).
raturainess
informeliverees
relevance
Immeowareness
41.54
28772
45.46
29.32
4422
14
20026
-1444
200
Session gap
#sessions
40
-23.8
minutes
hours
days
wooks
manths
year
minutes
82
Sension gho
hours
90
Figure 14: Performance of TA-RAG (progress) over different session gaps.
days
65
weeks
78
months
62
year
73
Table 19: Number of different session gaps in our ex- periments.
time-awereneen weeeeeses
75
57
50
Rainn
17.28
28,99
-2. 9.
2.61
2.61
wooks
year
Figure 15: Performance of TA-RAG (schedule) over different session gaps.


Reviews:
[{"review_id": "dBIT455S38", "reviewer": "Reviewer_j5Mt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews and your appreciation of the strengths of this paper.  Regarding the typos, thank you for pointing them out.  We will perform further editing and seek proofreading by native speakers for a final draft.\n\nRegarding the prompts, we did explore many prompt strategies in support of this research, and reported what we believe to be the most important design decisions.  These are presented in Appendix D.2, with Table 11-13 showing the different types of prompt formats explored. To be more specific, we extracted 51 conversations from our dataset and manually extracted the events as gold labels. We then evaluated the accuracy of a total of four different types of prompts (indicated as `instruction only`, `slot filling`, `conditions`, and `QA`). We assessed: 1) whether the response contains correct event information, and 2) whether the response is in the specified format, as given in the prompt. The final prompt we chose for our paper (`conditions`) achieved the highest accuracy of 76.19 for event information and 100.00 for format. The complete results are as follows:\\\nFor content:\\\nInstruction only: 38.10, Slot filling: 71.43, Conditions: 76.19, QA: 74.41\\\nFor format:\\\nInstruction only: 95.24, Slot filling: 71.43, Conditions: 100.00, QA: 96.13", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides a fresh perspective on multi-session conversation research.\n\n\n2. The process behind the construction of the GapChat dataset is meticulously detailed and explained.\n\n\n3. The approach for integrating temporal information into multi-session conversation models is both well-defined and interesting.\n\n\n4. The results from the human evaluation substantiate the authors' claims.", "reasons_to_reject": "1. I noticed a minor typo in line 329: \"a the progress...\". While I did not spot other errors, I recommend the authors conduct thorough proofreading as there may be additional typos.\n\n\n2. While ChatGPT is an impressive model, the current state of this paper seems heavily reliant on ChatGPT's accuracy in event extraction and duration estimation. Did the authors undertake empirical studies specifically related to prompt tuning?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel multi-session dialogue dataset called GapChat, which incorporates time-related information, such as the intervals between sessions and the progression of speaker-related events. To emulate real-life situations, the paper distinguishes between life events and world events. Additionally, the paper introduces a method for integrating temporal modeling in models utilizing the GapChat dataset.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XGF5XMa6h3", "reviewer": "Reviewer_b1Jg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews and your appreciation of the strengths of this paper.  We agree that an automatic evaluation would be a great inclusion in the paper, but unfortunately we had difficulty finding a metric which can automatically capture time-awareness in a way that is reasonably accurate.  We believe the best automatic metric would itself use GPT, and since GPT is used in the identification of events and durations used in generation, we fear this would likely result in something that is far more biased and preferential to our models specifically.  We are certainly open to suggestions.  But in that sense we believe the human evaluation is more fair and more reflective of overall quality and time awareness.  In our human evaluations, we did control for a variety of priming, order, and presentation biases. We performed pilot studies to select trustworthy annotators and we had 10 annotators for each question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel and challenging task of time-aware MSC, which has not been well-studied before.\n2. The paper presents a reasonable and effective pipeline to construct GapChat, a high-quality MSC dataset with time-aware information.\n3. The paper proposes TA-RAG, a time-aware dialogue model that exploits GapChat to generate more relevant and informative responses.\n4. The paper conducts extensive experiments to demonstrate the superiority of GapChat and TA-RAG over existing MSC datasets and models.", "reasons_to_reject": "1. The human evaluation may be subjective and influenced by various factors. It would be desirable to have some automatic metrics that can measure the time-awareness of the dialogue model.\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of multi-session chat (MSC), where a conversational agent needs to maintain coherent and engaging dialogues with a user across multiple sessions with time gaps. Previous MSC datasets have short time gaps and do not consider the impact of time on topic transitions and information updates. To fill this gap, the paper proposes GapChat, a novel MSC dataset that incorporates time-aware information and simulates realistic scenarios of MSC. The paper also proposes a time-aware dialogue model, TA-RAG, that leverages GapChat to generate more appropriate and informative responses. The paper evaluates the proposed dataset and model using both automatic and human metrics, and shows that they outperform existing baselines.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2qjmNaOjsU", "reviewer": "Reviewer_hTeo", "summary": "", "questions": "- Section 3.1 (p.3): Were past real-world event also ordered chronologically as they happened in the real world? Or are they shuffled randomly as the life events?\n\n- l.230-1: \"$7 per hour on average in accordance with 231 living wage standards\" -> How do you know that these are standard living wages, if you recruited crowd-worked through Amazon Mechanical Turk and the only requirement was a fluency in English. Aren't living wage standard higher in many English speaking countries?\n\n- l.244-6: \"For all subsequent sessions, speakers are provided with updates on the progress of their ongoing events based on the schedules and the time gap.\" -> How did you decide on these updates? Randomly or manually?\n\n- l.271-2: train/val/test split\n(a) How did you split the data in train/val/test? Randomly or stratified across session length? \n(b) Moreover, did several conversation about the same timeline occur? (Not specified in paper as far as I could see.) If so, did you take this into account while making the train/val/test split? Did all conversations on a specific timeline occur in only in 1 unique set (either train, val or test)?\n(c) Is the test set actually used, as your models are evaluated by humans?\n\n- l.285: Could you in your formulas distinguish between ground truths and estimates/generations close to this ground truth?\n\n- l.306-7: \"The prompt which resulted in the highest extraction performance(...)\" -> How did you calculate this performance? What if spans didn't correspond? Was it calculated in an end-to-end or pipeline setup?\n\n- l.349-51: \"During training time, events and different types of time information (progress labels and schedules) are combined with dialogue history\" -> What happened when this was too long to fit into the model? How did you truncate?\n\n- l.353-5: \"When generating a subsequent session, the gap information is also added as part of the context following the method in MSC.\" -> Could you provide some more details to clarify this sentence.\n\n- Table 2 (p.6): What metric is it? What scale are these scores on?\n\n- l.365-6: Could you provide more details on the fine-tuning here?\n\n- l.373: The ChatGPT \"with gap\" model seems to also be a time-aware model. Although it doesn't have a lot of information about the times, the model does know between which instances there was/wan't a gap. Why did you decide to frame it as a time-unaware model?\n\n- p.6: It is very difficult to judge this section, without having been given more details on the MSC-RAG model. How does this model work? Does it use prompt-based learning or is the input encoded in an embedding to fine-tune the model?\n\n- l.437-448: By how many annotators was each instance annotated?\n\n- l.447-8: How many annotators were involved in the Fleiss' Kappa study? How many instances did you use to calculate this score?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and many questions.  We would like to discuss them in two parts: (1) aspects of the procedure that may affect the presented results, (2) other aspects which are for more documentary purposes.\n\n----------\n(1) \n----------\n`Q:` How did you split the data in train/val/test? Did several conversation about the same timeline occur? Is the test set actually used?\n\n`A:` Regarding the train/valid/test split, we split the data randomly based on session length. Based on the sheer number of variables in constructing a timeline (choosing of the many events, the time gaps, how those events relate to world events, where in the conversation those events are introduced) it is extremely unlikely to have significant overlap between any two timelines.  Indeed, examining the dataset reveals that no timelines have identical event sequences, which is true both between train/test sets, but also within any set. We did not save certain events for the evaluation, and we acknowledge the concerns regarding the potential leakage of events from the training set to the test set. In our dataset, we found that 1.54\\% of life events sequences and 2.31\\% of world events sequences in the test set also appear in the training set, though they are presented as part of distinct timelines in the two sets.\n\n`Q:` More details on the MSC-RAG model? More details on the fine-tuning of RAG (FT)? What happened when this was too long to fit into the model? How did you truncate?\n\n`A:` The MSC-RAG model is a transformer model that stores past dialogue history as documents and employs a retrieval system to select certain history portions for inclusion in the final encoding. It has demonstrated effective performance in handling extensive context. Various methods for storing documents exist (such as storing them as DPR weights or using a FAISS index). We adopt the approach that yielded the best performance in the MSC paper [1] (the DPR approach) for all our RAG models. \n\nTo obtain the RAG (FT) model, we fine-tune the MSC-RAG model on GapChat. The fine-tuning involves two steps:\nCalculate new DPR weights using GapChat available at \\url{https://github.com/facebookresearch/DPR}.\nContinue training the MSC-RAG model with the new DPR weights using the ParlAI platform.\nRegarding the training settings, we also follow the approach from MSC. In our time-aware models, we concatenate different time-related information (progress labels, gap information, schedules) as raw text with the dialogue history in the encoding. If the dialogue history becomes excessively long, we truncate several utterances from the beginning of the conversation.\n\n`Q:` How many annotators was each instance annotated? How many annotators were involved in the Fleiss' Kappa study? How many instances did you use to calculate this score?\n\n`A:` As stated in Lines 440-441, during the evaluation process, we employed 10 annotators to annotate each instance, resulting in a total of 66 annotators. Fleiss' Kappa was calculated across 150 conversations (600 instances: $50\\times3 + 50\\times4 + 50\\times5$) involving all 66 annotators.\n\n[1] Beyond Goldfish Memory: Long-Term Open-Domain Conversation (\\url{https://aclanthology.org/2022.acl-long.356/})\n\n\n----------\n(2) \n---------- \nRegarding other questions of a more documentary / reproducibility standpoint, we want to first reiterate that the dataset and all code used in its creation is available upon acceptance, and answers to these questions are observable in these resources.  In addition we will incorporate the answers to the following questions into the appendix or into the main content where appropriate:\n\n`Q:` Were past real-world event also ordered chronologically as they happened in the real world? \n\n`A:` Yes.\n\n`Q:` How did you decide on these updates of life events?\n\n`A:` We prompted ChatGPT to provide plausible multi-step decompositions for long-duration life events, and to estimate their durations.\n\n`Q:` Aren't living wage standard higher in many English speaking countries?\n\n`A:` We limit the locale of workers to North America.  We mistook minimum wage for living wage which is now estimated to be much higher in these countries (\\$14-20), and we will correct this.  We strive to accommodate workers fairly, and offered \\\\$7 based on minimum wage and the average MTurk rate (\\\\$3), but unfortunately the grants supporting this work are not from these countries which makes it difficult to further increase the compensation per annotator.\n\n`Q:` How did you calculate this performance of the prompts? What if spans didn't correspond?\n\n`A:` Regarding the prompt selection, please refer to the response to reviewer j5Mt. We check the generated content with different prompts manually in a pipeline setup, in which for some prompt templates we send multiple prompts and the consequent prompt depend on the answers to previous prompts. We check both the content and format3  If the spans didn't correspond or the answers are not the expected format, they will be labelled as format errors.\n\n`Q:` Why did you decide to frame ChatGPT (with gap) as a time-unaware model?\n\n`A:` Thank you, it is a good point and we understand how it could be a point of confusion.  It is time-aware in the sense that it is given information about gaps and their durations.  The model does not receive event and event progress information, which is the distinction we were making with \"time-aware\", but we see how the terminology could be improved and clarified.\n\n`Q:` What metric is in Table 2? What scale are these scores on?\n\n`A:` Table 2 scores are from human evaluation using ACUTE-Eval, which is a preference based task. We display two sessions generated by two different models (one target model and one baseline model) and ask annotators to rate one model over another one. We calculate the human ratings to get the numbers in Table 2, where negative numbers indicate that annotators rate the target models as being worse than the baseline model and positive numbers indicate the opposite. In our experiments, we use RAG (FT) model as the common baseline and compare all other models against it to get Table 2. The scores are calculated in percentages. For instance $13.75$ means for all evaluation instances, $13.75$ more annotators voted for the target model being better than RAG (FT) on the specific attribute.\n\n`Q:` Typos in the text and formula?\n\n`A:` Thank you for pointing out some minor typos.  We will perform further editing and seek proofreading by native speakers for a final draft. And regarding the writing style in the formula, we will represent predicted/generated results as \n$ \\hat{y} = \\\\{ \\hat{y_1},... \\hat{y_i} \\\\}$ to distinguish from ground truth. That makes the formula:\n\n$$ p\\(\\hat{y_i}|H, progress, t \\) =  \\prod P(\\hat{y_i}|\\hat{y_1},...,\\hat{y_{i-1}, H, progress, t})$$\n, where $H$ represents the dialogue history $\\{x_1,...x_{i}\\}$, $progress$ is the corresponding progress of the events and $t$ is the time gap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors present a very interesting novel dataset that can become a valuable research for embodied and real-life conversational applications.\n\n- The merit of this dataset is demonstrated through ML experiments. Systems that receive time-aware information perform generally better than their time-unaware counterparts.", "reasons_to_reject": "- Although the paper starts very promising and there is definitely a lot of potential in the GapChat dataset, there are currently too many crucial details missing (on the dataset, implementation of models and results) to reproduce the results (see \"Questions For The Authors\").\n\n- The formulas in the task definition seem to be wrongly formulated.\n\n- In terms of modelling, there seems to be relatively little innovation. The authors give prompts to ChatGPT and use an existing model on their dataset. Nevertheless, it was difficult to properly judge this section as quite some implementation details were missing, so my judgement might be wrong here.", "questions_to_authors": "- Section 3.1 (p.3): Were past real-world event also ordered chronologically as they happened in the real world? Or are they shuffled randomly as the life events?\n\n- l.230-1: \"$7 per hour on average in accordance with 231 living wage standards\" -> How do you know that these are standard living wages, if you recruited crowd-worked through Amazon Mechanical Turk and the only requirement was a fluency in English. Aren't living wage standard higher in many English speaking countries?\n\n- l.244-6: \"For all subsequent sessions, speakers are provided with updates on the progress of their ongoing events based on the schedules and the time gap.\" -> How did you decide on these updates? Randomly or manually?\n\n- l.271-2: train/val/test split\n(a) How did you split the data in train/val/test? Randomly or stratified across session length? \n(b) Moreover, did several conversation about the same timeline occur? (Not specified in paper as far as I could see.) If so, did you take this into account while making the train/val/test split? Did all conversations on a specific timeline occur in only in 1 unique set (either train, val or test)?\n(c) Is the test set actually used, as your models are evaluated by humans?\n\n- l.285: Could you in your formulas distinguish between ground truths and estimates/generations close to this ground truth?\n\n- l.306-7: \"The prompt which resulted in the highest extraction performance(...)\" -> How did you calculate this performance? What if spans didn't correspond? Was it calculated in an end-to-end or pipeline setup?\n\n- l.349-51: \"During training time, events and different types of time information (progress labels and schedules) are combined with dialogue history\" -> What happened when this was too long to fit into the model? How did you truncate?\n\n- l.353-5: \"When generating a subsequent session, the gap information is also added as part of the context following the method in MSC.\" -> Could you provide some more details to clarify this sentence.\n\n- Table 2 (p.6): What metric is it? What scale are these scores on?\n\n- l.365-6: Could you provide more details on the fine-tuning here?\n\n- l.373: The ChatGPT \"with gap\" model seems to also be a time-aware model. Although it doesn't have a lot of information about the times, the model does know between which instances there was/wan't a gap. Why did you decide to frame it as a time-unaware model?\n\n- p.6: It is very difficult to judge this section, without having been given more details on the MSC-RAG model. How does this model work? Does it use prompt-based learning or is the input encoded in an embedding to fine-tune the model?\n\n- l.437-448: By how many annotators was each instance annotated?\n\n- l.447-8: How many annotators were involved in the Fleiss' Kappa study? How many instances did you use to calculate this score?", "paper_topic_and_main_contributions": "The authors of this paper introduce a new dataset, called GapChat, of multi-session dialogues. This way, they want to investigate how generation models can be made time-aware. They extend the existing MSC dataset by (1) looking at long-term gaps (instead of short) and (2) paying attention to the relation between topic events and the duration of gaps.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l.32: \"may not be well-represented explicitly\" -> may not be wel represented vs. may not be represented explicitly\n- l.118: \"coversation\" -> conversation\n- l.143-4: \"to pay attention the relation\" -> pay attention to\n- l.271: \"split the data to\" -> split into\n- l.280: the final x between {} should be x_n. Especially if you talk about all utterances as x_i\n- l.285: Make a distinctions between ground truth (e.g., E) and estimation of ground truth (by circumflex, e.g., Ê)\n- l.285: y = argmax P(y|H, progress, t) -> this formula does not make sense. See previous comment.\n- l.329: \"a the\"\n- l.449-460: Would it not be better to place these training details in the section on models instead of the section on evaluation?"}}]

Author rebuttal:

Meta-review: This paper presents a multi-session chat dataset where each session is separated by a time interval. During this time interval, speaker-related or world-related events may change. The authors also propose a model to handle such dialogues. The reviewers appreciate the novelty of this dataset as well as its thorough presentation and evaluation but also raise some concerns, including confusion in some formulas, lack of novelty of the proposed model (not the dataset), as well as some issues with the evaluation. The authors address the main concerns and therefore I recommend accepting this paper.
Decision: Accept-Findings