Time-Considerable Dialogue Models via Reranking by Time Dependency
Authors: Yuiko Tsunomori
Masakazu Ishihata
Hiroaki Sugiyama
Venue: EMNLP 2023 Findings (2023)
Keywords: dialogue model
dialogue system
response generation
time information

Abstract: In the last few years, generative dialogue models have shown excellent performance and have been used for various applications. As chatbots become more prevalent in our daily lives, more and more people expect them to behave more like humans, but existing dialogue models do not consider the time information that people are constantly aware of. In this paper, we aim to construct a time-considerable dialogue model that actively utilizes time information. First, we categorize responses by their naturalness at different times and introduce a new metric to classify responses into our categories. Then, we propose a new reranking method to make the existing dialogue model time-considerable using the proposed metric and subjectively evaluate the performances of the obtained time-considerable dialogue models by humans.

Time- Time-Considerable Dialogue Models via Reranking by Time Dependency
Yuiko Tsunomori', Masakazu Ishihata', Hiroaki Sugiyama'
'NTT Communication Science Laboratories (yuiko. tsunomori, masakazu. ishihata, conatta,iirrkeee hiroaki. Jogiyammaennt.com
Abstract
Utterance
Spoken time
8
the last few years, generative dialogue mod- els have shown excellent performance and have been used for various applications. As chat- bots become more prevalent our daily lives, more and more people expect them behave more like humans, but existing dialogue mod- els do not consider the time information that people are constantly aware of. In this paper, we aim to construct a cimeiconsieerable dia- logue model that actively utilizes time informa- tion. First, we categorize responses by their naturalness at different times and introduce metric classify responses into egories. Then, we propose new reranking method make existing dialogue model time-considerable using the proposed metric and subjectively evaluate the performances of the obtained consideraee dialogue mod- els by humans.
I' hungry.
Day
Response
(1)
What do you want to eat?
AN
(2)
Didn't you have breakfast?
TN
(3)
You will set fat if you eat at this time.
TU
(4)
I don't like doing sports.
AU
Figure 1: Conversation examples depending time.
the most basic and important external information, still does not seem to be considered important in dialogue models.
1 Introduction
In the last few years, generative dialogue models have achieved outstanding performance (Ziegler al., 2019; Adiwardana et al., 2020; Ouyang et al.. 2022: Thoppilan al., 2022) and have been used in various applications, including search engines, recommendations, healthcare, finance, and more (Ling et al., 2023). As chatbots permeate our daily lives, more and more people expect chatbots to be- have in a human-likk manner. Examples of research to make chatbots more human-like include the in- troduction of common sense (Wang et al., 2020), empathy (Ma et al., 2020), personas (Zhang al., 2018), and so forth. The common point among these studies is that they have achieved richer dia- logue by actively utilizing not only internal infor- mation obtained through the conversation but also external information that does not appear in the cur- rent dialogue. On the other hand, time information,
Humans are basically always aware of time in conversation, whether explicitly or implicitly, be- cause the naturalness of utterances and responses may change based on their spoken times (e.g.. time of the day, day of the week, and season). Given an utterance and its spoken time, a response can be cat- egorized into the following four types by focusing on the time variations of its naturalness:
AN: always natural
TN: temporarily natural at the spoken time
TU: temporarily unnatural at the spoken time
AU: always unnatural
Figure I shows examples of the above four types of responses. Assume two time periods, day and night, where responses (1)-(4) have different levels of naturalness at different times. Responses (1) and (2) (N & AN U TN) are natural to a given utterance at a spoken time, although not (3) and (4)
'The detailed information about our dataset is available at https://githuu htt.ttuu..ccmm/tttllbbb//// time-cone coonsiderable-dee wearsiideealll-allgguueee

---

( A AU). On the other hand, the natural- ness of (2) and (3) (T & TN TU) changes with the spoken time (day or night), and that of (1) and (4) (A & AN U AU) remains unchanged. Here- after in this paper, we refer to such categorization of responses as the NUTA categories.
and AN and a new mechanism generate TN responses.
In this paper. we propose a new reranking method that is post-processing method to make existing dialogue models more time-considerable. first verify our
As a preliminary analysis, sumption users lend to prefer TN AN by hu- man evaluation in Section 3. We next formally define the NUTA categories and propose an auto- matic metric for the NUTA categorics in Section 4. In Section 5, we propose a new reranking method using the proposed automatic metric to make exist- ing dialogue models time-considerable and subjec- tively evaluate obtained coniidebseee models to verify whether our reranking method improves response qualities.
For a dialogue model to achieve natural conversa- tion, it is expected to generate natural responses N and to avoid generating unnatural responses U. If a dialogue model correctly evaluates the naturalness of responses considering the spoken time, we call it time-aware; otherwise, we call it time-unaware. Many existing dialogue models are time-unaware because they are trained on datasets without time information. As a result, a Lime-unaware dialogue model may consider TU responses as natural as N responses because they are natural at some time. In other words, it may generate TU response, which is inappropriate at the spoken time. The sim- plest way construct time-aware dialogue model is to train a general dialogue model with time in- formation (Sato et al., 2017). Recently, the output large language model (LLM) was adjusted by giving an appropriate prompt (Lester et al., 2021; Bac et al., 2022; Liu et al., 2022), and prompting the time information to an LLM is another promis- ing way to achieve time-aware dialogue model. The difference between time-unaware and time- aware dialogue models is that the former considers a response as natural if it is natural at some time, while the latter does if it is natural at its spoken time; namely, time-aware dialogue models avoid generating TU responses.
Related Work
Previous studies have pointed out that general dia- logue models trained on large-scale datasets tend to generate neutral (bland, generic, and hackncyed) responses (Li et al., 2016; Serban et al., 2016). To tackle this issue, some dialogue models utilizing external information generate interesting and in- formative responses have been proposed, where examples of external information include sys- tem/user persona (Zhang et al., 2018; Roller et al.. 2021; Lu et al., 2022), knowledge graphs (Zhang et al., 2020), knowledge sources (Parthasarathi and Pineau, 2018; Majumder et al., 2022), interpersonal relationships (Utami and Bickmore, 2019), and sit- uated environments (Misu, 2018). Some studies have empirically shown that generating responses specific external information improved users' impression of the dialogue models (Vinyals and Le, 2015; Li et al., 2016; Zhou et al., 2021). this paper, we introduce the time information as external information and consider that T responses, whose naturalness varies with time, are specific to time information. Furthermore, to verify whether the time information improves model performance as well as other external information, we evaluate the qualities of T responses by human evaluations.
Prior research has shown that users' impressions of dialogue models are improved by actively uti- lizing external information (Vinyals and Le, 2015; Li et al., 2016; Zhou et al., 2021). Referring to this fact, we assume that users' impressions will be similarly improved by actively utilizing time information; namely, users prefer TN AN (we empirically verify this assumption in Section 3). Under this assumption, we aim to realize a time- considerable dialogue model that actively outputs more TN than AN. The difference between time- aware and time-considerable models is that the former only considers time information to eval- uate the naturalness of responses at given spoken time (time-aware naturalness), while the latter ac- tively generates responses whose naturalness varies with time. To realize time-considerable models, we need a new criterion to distinguish between TN
In this paper, we define a dialogue model as time- aware if the model evaluates the naturalness of re- sponses considering their spoken times. Sato et al. (2017) proposed time-aware dialogue model that is an encoder-decoder model based on Long-Shon Short Term Memory (LSTM) (Zaremba et al., 2015) in- spired by Johnson et al. (2017) and trained on interance-responee pairs with their timestamps ex-

---

tracted from Twitter. Another recent technique to achieve a time-aware dialogue model is prompt- ing LLMs, where prompting a technique to guide LLMs in generating high-quality and rele- vant responses by providing detailed descriptions and/or input-ouiput examples of the target task as input (Brown et al., 2020). Various prompting- based methods for utilizing external information have been proposed, and their examples include system/user persona information (Kasahara al., 2022; Lee et al., 2022), knowledge sources (Liu et al., 2022), and fictional character's style (Han et al., 2022). In this paper, we aim to construct a time-considerable dialogue model, which actively utilizes the time information, and propose new reranking method to make the existing dialogue models more consideaeeee using time-aware naturalness represented by time-aware dialogue model. While time-aware dialogue models only consider time to evaluate the naturalness of re- sponses, cine-considerable dialogue models ac- tively generate responses whose naturalness varies with time.
Lancers a Japanese crowdsoureing service. We assigned one crowd worker each utterance and asked them to perform the following tasks to create responses:
A.NN to that is natural at 1. Create response T' any time.
2. Select two time periods No and t (tN, t E T) and create response TN to u that is natu- ral at N but not at e
"It seems the train is stopped," the crowd worker selected two time periods 22 and t' = and created two responses AN = "Really? wonder when it will start moving." and r "Wow, it's almost the last train. wonder what's going to happen?" Finally, we constructed NUTA dataset by cre- ating the following four tuples for each utterance U, where u' was randomly chosen from Ut so that o u/:
For instance, given utterance
(u, A., ru AN AN), (u, TN TU), (u, N AN AU).
(u, N IN TN); Fu
3 Preliminary Analysis
Thus, the dataset consists of 2,560 (= 1 x 4) tuples. Table I shows examples of four created tuples for the same utterance.
Through this paper, we assume users prefer TN to AN, and in this section, we verify this assump- tion by human evaluations. We first constructed a NUTA dataset consisting of the tuples of utter- ances, their spoken times, their responses, and their NUTA categories: AN, TN, TU, and AU. Then, we conducted a subjective evaluation to determine which category of responses users found most in- teresting and informative.
3.2 Subjective Evaluation
We conducted subjective evaluation to determine which response category is the most interesting and informative for humans.
We introduced new metric to measure the qual- ity of responses considering time information. The metric is in the range [0, 1] and based on the Sen- sibleness, Specificity, Interestingness (SSI) met- ric (Thoppilan et al., 2022) for evaluating responses based on context. Our metrie, denoted by SSI-t, averages the following four scores:
3.1 The NUTA Dataset
We constructed a NUTA dataset, which is collec- tion of tuples t A (u, r, r). where u is an utter- ance, t E T A {0, 1, 1,,,,,....) 23} is its spoken time in 24-hour time format, r is response u, and e (AN, TN, TU, AU) is its NUTA category.
Sensibleness for the utterance (SU): If its spo- ken time is ignored, is the response reasonable to its utterance?
We first prepared set of utterances 4. We ex- tracted Japanese tweets posted between May and December 2022 with filtering rules described in Appendix A. and randomly selected 1,000 tweets. We manually deleted tweets containing discrimi- natory, violent. or other inappropriate expressions. As a result, we obtained 640 appropriate tweets and used them utterances U.
Sensibleness for the spoken time (ST): If its terance is ignored, the response reasonable its spoken time?
Specificity to time (S): Is the response specific any time regardless of its spoken time?
Interestingness (I): Is the response interesting or informative?
We next obtained responses to the prepared utterances 2/ by crowdsoureing, where we used
hhttps://www.laccers..t

---

Utterance u = seems the train is stopped.
Time f
Response
Category
22
Really? wonder when will start moving. ( 12/2222
AN
Wow, it's almost the last train. wonder what's going to happen? (A 1
22
TN
1E E, 2 23. D ?)
Wow, almost the last train. wonder what's going to happen? (ž7 カビッ
6
TU
12 2. Z &g. T2 ?)
Understood! I'll decide on the character as soon as possible. (7 | ワ 3e 5 tン)
22
AU
Table I: Four tuples t = (u, T; e) the same utterance. crowd worker selected N 22 and 1 = 6 and generated two sentences A and IN,, The utterance and all responses were originally written in Japanese and translated into English by the authors.
For example, score SU of 1.0 indicates is perfect response to if is ignored, and a score of 0.0 indicates the naturalness of r never changes over time.
Time-aware naturalness We assume that the time-aware naturalness (TAN) of and r at t is implicitly defined by conditional probability dis- tribution p(u,r () = p(u 1) p(r u, 1), where p(u t) and p(r u. t) indicate the TANs of u at and r given u at f. We consider N responses of the NUTA categories natural at spoken time t; i.e. response is classified as N (resp. U) iff p(r u, I) is high (resp. low). Since it is very difficult to know the true TAN p. throughout this paper, assume that TAN p given time-aware dialogue model that allows us to evaluate p(r u, t) for any u, r, and t.
We randomly selected 25 tuples for each NUTA category (100 tuples in total). For any tuple t t= (u, 4,,, e) and any score SC E (SU, ST, S, l]. we asked two expert annotators, who are in-house workers specialized in annotating dialogues and have worked in their positions for at least five years, rate SC of with either SC value of t as the average of two obtained rates and the SSI-t value of as the average of all SC values of t. Finally, for any NUTA category e and any score SC' E (SU, ST, S, I, SSI-t), we ob- tained the SC value of e by averaging those of all tuples whose categories are Table shows four examples of tuples and their obtained values.
1. We defined the
Change of naturalness Using TAN p, we define the change of (log) naturalness (CN) from l' to I
CNe:euu, r) - p(u, r [t)
(1)
p(u, r (')
Table shows the SSI-t scores for each NUTA category and indicates that TN achieved the high- est quality (SSI-t): more specifically, the highest ST and scores. We believe that the results sup- port our assumption that users prefer TN to AN, and based on this assumption, we will propose method to realize time-consine coneidonsieeeable dialogue model that actively outputs more TN than AN.
CNra(u) 2 p(u t)
(2)
p(u E')
CNea(r 1 2 In p(r [u,
(3)
p(r u,)")
By definition, the following equation must hold:
CNva(u, CNea(u) + CNea(r u), (4)
where the CN of conversation (u, r) can be factor- ized into the CNs of and r given u. For instance, CNe (u, r) > holds iff the conversation (u, r) more natural t than at
4 Automatic Metric for NUTA Categories
We propose a new automatic metric for classify- ing the NUTA categories of given responses and experimentally show that our metric can correctly categorize the responses of the NUTA dataset.
Time dependency Using the above CN, we de- fine time dependency (TD) of r given u as
4.1 Definition
TD(r u,,t) _ max CN,u,(r u), PUTT
(5)
We mathematically introduce the time-aware natu- ralness and the time dependency of responses and define the NUTA categories using those quantities. Let u, r, and é be an utterance, a response to u, and the time at which the conversation took place.
TD ()) A max TD | u, N.)
(6)
NCT
(r u, t), denoted by TD@t, is the CN of r given u from the most unnatural time ! to spoken

---

Utterance u
Time
Response r
Cat.
SU
ST
SSI-t
If gummics are within reach, can't stop cating them endlessly, ( E. T-O E 12 Bb MES
also love gummics, especially the ones with lots of fruit juice.
10
( + t TT. #
AN
1.0
0.0
0.75
<BEL L 5.
Does anyone want play Apex Leg-
Maybe we can play a game to- gether tonight. (# EE
ends ranked match with time? (I< 5 / z & # F2 2 A G to ?...)
17
TN
1.0
1.0
1.0
1.0
1.00
5 5 o C E
Oh! just realized I have over 1000 followers! (A !! 4 31
Congrats! I'm glad to hear the good news before go bed. (35 2 & 5 ! ti & BEW = X ( T Li do Autumn goes by so fast. (OXLE 2. 1 T +2)
10
TU
1.0
0.0
1.0
0.5
0.63
i 2 b / 0000W 2 o
Maybe I'll curry today. (9 お
AU
0.0
0.0
0.5
0.38
L f (2)
Table 2: Four example tuples (u, r, c) and their obtained SU, ST, S, I, and SSI-t values. All utterances and responses were originally written Japanese and translated into English by the authors.
Category
SU
ST
S
I
SSI-t
Category
p(r |u,
TD(r I
AN
0.98
0.86
0.14
0.66
0.66
AN
High
Low
TN
0.97
1.00
0.80
0.76
0.89
TN
High
High
TU
0.93
0.19
0.88
0.71
0.68
TU
Low
High
AU
0.06
0.87
0.13
0.66
0.43
AU
Low
Low
Table 3: SU, ST, S. I, and SSI-t scores of responses of each NUTA category
Table 4: Definition of NUTA categories using TAN
p(r u, () and TD@all TD(r u).
time f, which evaluates whether r is specific to spoken time t. Consequently, has a high TD@t if it is natural at i but unnatural at another time and a low TD (i.e., near zero) if its naturalness remains unchanged time changes. On the other hand, (D(r u), denoted by TD@all all, the CN from the most unnatural time t to the most natural time N which evaluates whether r specific to time or not. So, r has a high TD@all it is natural at some time but not at another time. Since we consider the naturalness of T responses of the NUTA categories varies with time, is classified as T (resp. A) iff TD(r u) is high (resp. low).
Related criteria In natural language processing. various criteria have been proposed for measur- ing the dependency between two sentences. Li et al. (2016) proposed pointwise mutual informa- tion (PMI) to choose appropriate response given utterance u.:
p(r u) p(r)
PMI(r u A log
(7)
As an extension of PMI, Paranjape and Manning (2021) proposed pointwise conditional mutual in- formation (PCMI) to cope with additional external information other than utterance evaluate appropriate response to Given utterance u, sponse r, and time information as external infor- mation, PCMI @1 is defined as
NUTA category By the definitions of TAN p(r u, and TD@all all TD(r w), we define each NUTA category by Table 4. For instance, re- sponse r classified as TN iff both p(r u. t) and (D u) are high, and r is classified as AU iff both p(r u. t) and TD(r u) are low. Strietly speaking, to use this definition, two thresholds must be set that distinguish between the high and low of p(r) u, and (D u): however, since we be- lieve that determining these thresholds in advance is difficult, all the methods proposed in this pa- per are designed so that they do not require such thresholds.
p(r u,t) p(r u)
PCMI(r u,t) A log
(8)
where let p(r |u) be the time-unaware naturalness (TUN). While TD@t t in Eq. (5) is the CN of given from the most unnatural time & to the current time PCMI @ is the CN of given u when the naturalness changes from TUN to TAN similar manner as TD@all, we define PCMI@all as PCMI(r u) A max,Ne7 PCMI(r u, u,,N).
In a

---

We believe our TD is more appropriate met- ric for time information than PCMI. PCMI con- siders the presence or absence of time informa- tion, not its change; however, time information always exists and changes, unlike such common external information as user persona and knowl- edge graphs. Therefore, PCMI is expected to be more blurred in its evaluation than TD. For in- stance, suppose two time ranges, &1 and 12, such that p(t1) - p(f2) 0.5 where (u, r) is strongly time-specific response such that p(r | (u,1)) (1) (i.c., r cannot be a response to u at 61) and p(r | u, (2) = (i.e., r is a perfect response to u at 12). Then, p(r u) 0.5 since p(r u) Li p(t)((r t). By Eqs. (3), (5) and (6), TD(r u) = where @D@all considers r strongly time-specific response. On the other hand, PCMI(r u, In2 x 0.69 where PCMI @all considers as not time-specific. Thus, using TUN p(r u) blurs the evaluation of time depen- dency, and our TD@all all is expected to detect time- specific responses more clearly than PCMI@al all.
Naturalness
TD
PCMI
TUN
TAN
@all
@all
@!
N
0.60
0.68
0.47
0.58
0.46
0.35
0.54
0.54
0.68
0.60
0.53
0.48
Table 5: Accuracy of N/U and T/A classifications of each quantity. Best scores are indicated by bold.
PCMI
@all
@1
@all
@t
AN
0.54
0,43
0.43
0.38
TN
0.48
0.60
0.35
0.51
TU
0.44
0.19
0.34
0.14
AU
0.45
0.44
0.32
Ave. 0.48
0.42
0.36
0.34
Table 6: Accuracy of each NUTA category of each
combination, Best scores are indicated by bold.
4.2.2 Experimental Results
We first checked whether TAN p(r u,()) and @D@all TD(r u) correctly classified N and T. Using high/low labels obtained by each quantity, we categorized tuples with high (resp. low) labels as N (resp. U) and computed the accuracy of the N category of each quantity. Similarly, we also computed the accuracy of the T category. Table 5 shows the accuracies of each quamtity and indicates that TAN and TD@ @all achieved the highest accu- racy of N and T. Consequently, TAN and TD @ all are appropriate quantities for evaluating the natu- ralness and time dependency of responses.
4.2 Experiments
We conducted experiments show that our auto- matic metric for the NUTA categories correctly or- ders responses of the NUTA dataset of Section 3.1.
4.2.1 Experimental Settings
For each utterance u E U, ranked four tuples containing in the NUTA dataset by each quantity: TUN p(r w). TAN p(r u, u. TD@all TD(r u). TD@t TD(r u. t), @CMI@all PCMT(r w), and PCMI PCMI(r u, l). We attached a label high (resp. low) to the top (resp. bottom) of two tuples in each ranking for each quantity and used the ob- tained labels to classify NUTA categories.
Next, we checked whether the combination of TAN and TD@al all correctly classified each NUTA category. We categorized tuples into one of AN, TN, TU, and AU according to Table 4 using the combination of high/low labels obtained by TAN and those obtained by one of TDT@alll @ll, TD@1. PCMI@all all, and PCMI@t. Table 6 shows the curacy of each NUTA category of each combina- tion and indicates that TD@ achieved the best average accuracy; however, for the TN category, TD@: achieved the best. Since TD@t evaluates whether response r is specific to the current time t but not to other times, it is more effective to detect TN responses than TU responses. Be- cause our original motivation for using these quan- tities as automatic metrics for the NUTA categories is to detect TN responses that users prefer than AN, we conclude that TD@: TD(r u, t) is the most appropriate metrie for constructing a time- considerable dialogue model.
As TUN p(r u). we used the Transformer- based Japanese dialogue model (TJD) (Sugiyama al., 2023) with 1.6B parameters trained on more than two billion tweet-reply pairs: (u, We con- structed TAN p(r] "u, 1) by fine-tuning the above model using tweet-reply-time triplets: (u, r, t), where the fine-tuning dataset was obtained simi- larly as (Sato et al., 2017): 7): we collected Japanese tweets with replies from August 2021 1 to April 2022 with filtering rules described in Appendix A. and obtained 470.255,625 triplets. We denote the fine-tuned time-aware TJD by TJD-t and used TJD p(r u) and TJD-t p(r u. f) to evaluate TD@all/t and PCMI all/t. Detailed implementational set- tings are shown in Appendix A.2.

---

Model
len
distinct-1
distinct-2
SU
ST
S
SSI-t
TJD
17.22
0.36
0.57
0.86
0.84
0.16
0.40
0.56
TC-TJD
12.50
0.55
0.78
0.86
0.85
0.19
0.45
0.59
TJD-t
16.74
0.33
0.48
0.78
0.83
0.12
0.36
0.52
TC-TJD-t
14.44
0.49
0.75
0.86
0.86
0.20
0.48
0.60
GPT-3.5
40.50
0,45
0.76
0.80
0.89
0.36
0.60
0.66
TC-GPT-3.5
34.92
0,44
0.74
0.74
0.90
0.53
0.54
0.68
GPT-4
37.79
0,40
0.73
0.86
0.96
0.89
0.62
0.83
TC-GPT-4
30.88
0.37
0.69
0.83
0.96
0.93
0.59
0.83
Table 7: Average length (len), distinct-N (N = 1). SU, ST, S, I, and SSI-t scores of each model. TC-M is time-oosiierrbble M achieved by our proposed reranking method. White and gray rows correspond to M and TC-M. TC-C'''s exceeded M' one, is indicated by bold.
5 Time-Considerable Dialogue Models
base dialogue model M is required only to gener- ate multiple responses for the same utterance and TAN p(r t) only to be evaluable. Namely, our method can be applied to dialogue models whose architectures and parameters not publicly avail- able but are provided as APIs. Of course, if M is time-aware and evaluable, can also be used TAN p(r u, t).
We propose a new reranking method to make exist- ing dialogue models comeeconniieeable We applied our method to various existing models, including GPT-4, which is ofte--of-theart LLM, and eval- uated the time-considdeaabe dialogue models to verify whether they improved response qualities.
5.1 Proposed Reranking method
5.2 Experiments
Let M be any dialogue model that generate multiple responses to the same utterance. Our pro- posed reranking method extends M to be time- considerable. Given base dialogue model M,, TAN p(r u. !). positive integer N, and probability we obtain time-considerable response r" to utter- ance u at time by the following manner:
We applied our proposed reranking method to exist- ing dialogue models and gauged their performance by human subjective evaluations.
5.2.1 Experimental Settings
We briefly explain our experimental settings, and the detailed settings shown in Appendix A.2.
1. Generate N candidate responses to u at I, de- noted by R & {ra i [N]], from base dialogue model M,
Base dialogue models We used the following four dialogue models as base dialogue model M of our proposed reranking method:
2. Evaluate TAN p(r; u, for all ri R and delete from R if r has sufficient natural- ness; ie.. p(r: u. 5,
1. TJD is transformer-bas Japanese dialogue model with 1.6B parameters trained over two billion tweet-reply pairs (Sugiyama et al., 2023) described in Section 4.2.
3. Evaluate TD@: TD(r: u,t) f) for all r; R and find the most time-specific response r* max TD u, u,
2. TJD-t is time-aware TJD obtained by fine- tuning described in Section 4.2.
4. Return obtained r* as a time-considerable re- sponse to at
3. GPT-3.5 is an extension LLM of GPT-
Since Step 2 removes candidate responses with lower naturalness than threshold 8, the filtering mechanism may improve the naturalness of the final response y* when the response generation model is weak.
3 (Brown et al., 2020) with 355B parameters and supports various tasks many languages. gpt-3. 5-turbo specialized GPT-3.5 for dialogue tasks and provided as an API.
Our proposed reranking method is simple but strong because we can create various time- considerable dialogue models by combining ex- isting base dialogue models and TANs, where
4. GPT-4 is large-scale multimodal model that extends GPT-3.5 (OpenAI, 2023). gpt-4 is a specialized GPT-4 for dialogue tasks and is provided as an API.

---

All models can generate multiple responses for the same utterance by top-p sampling (Holtzman et al., 2020), where p hyperparameter and set to 0.9 through the experiments. Since GPT-3.5 and GPT-4 do not treat time information as input, they are originally time-unaware however, in our experiment, we used them with prompts generate time-aware responses. We created prompts based a sample prompt for dialogue tasks provided by OpenAl and shown in Appendix A.2.
Evaluation criteria For each dialogue model, we computed the average output response length (len) and distinct-N (N 1, 2), which the ratio of the number of unique --grams to the total words generated by the dialogue model, for evaluating its diversity (Li et al., 2016). We also obtained the ST, SU, S, I, and SSI-t scores defined in Section 3.2 by human subjective evaluations. We assigned five crowd workers for each response and asked them to evaluate its SU, ST, S, and scores with a five- point Likert scale (1 5) and normalized them into [0, 1] scale. We obtained the SU, ST, S, and values of each model by averaging the annotated values of responses generated by the model. Finally, we obtained its SSI-t value as the average of those values.
Settings on proposed reranking method For any base dialogue model M, we denote time- considerable M achieved by our proposed rerank- ing method by TC-M (e.g., TC-TJD, TC-GPT- 4). Throughout the experiments, we used TJD-t TAN p(r t, and set N 20 and 8 = 0, where N is the number of candidate responses to be generated and 8 is a threshold for filtering candi- date responses by their naturalness. We set 0 because we used sufficiently strong models for re- sponse generation and did not aim to improve their naturalness.
5.2.2 Experimental Results
Table 7 shows the values of each model for the eight indicates: the average output response length (len), distinct-N (N = 1,2), SU, ST, S, I, and SSI- t scores. For any score except len, higher value denotes better result. In the table, if the score of TC-M exceeds the original M, we highlighted it in bold; in other words, bold scores were improved by our proposed reranking method.
Time intervals Since TD@ is the maximum of Cr, u) for all possible U E T, U could be very close to spoken time f, and in such cases, the time dependency of r might be not interpretable for humans since it is too short-term. To avoid detecting such short-term time dependency, we di- vided 24 hours into three intervals, morning (3 to 8), noon (9 to 17), and night (18 to 2), and de- fined their representatives as 6, 13, and 22, with reference to Yamamoto and Shimada (2019). The reason of adopting the above intervals is that morn- ing/noon/night is defined based on the human life- cycle; thus, this interval is more intuitive for human understanding. We evaluated TD@t t by Eq. (5) with T as representatives except for one of spoken time (e.g., t 10 = T {6, 22)).
TC-M improved the ST, S, and SSI-t scores for almost all M; namely, it was more time- considerable than the original M. TC-M tended to prefer short response to M because the len of TC-M is smaller than M for each model. For GPTs, the distinct-N, SU, and I scores of TC- M were slightly lower than M. This is because those scores of M were already sufficiently high, and our reranking method aims to improve time- considerability but not diversity and general sensi- bility of naturalness. Consequently, we confirmed that our reranking method improved the quality of the existing dialogue models.
One interesting observation from this experiment is that TJD-t, which we used as TAN p(r u, t,. had lower quality than TJD. This fact suggests that fine-tuning explained in 4.2 was not enough and that our reranking method improved the per- formance of the existing dialogue models even if TAN's performance is somewhat low. However, of course, using TAN with low performance is not desirable. For GPTs, TC-M achieved higher score but lower SU and I scores than the origi- nal M. We believe this deterioration of SU and scores can be easily avoided using a more accurate model as TAN, such as GPT-4.
Evaluation dataset As an evaluation dataset, we prepared 100 utterance-time pairs in the same man- ner as U in Section 3.1, where we excluded the same tweets as the NUTA dataset. Given utterance u and its spoken time f, for each base dialogue model M E (TJD, TJD-t, GPT-3.5, GPT-4), generated the best response of M and time- considerable response of TC-M, denoted by T.M and rjM- Thus, the evaluation dataset consisted of 100 tuples of u, t, and responses FM and M'M for each. M.

---

Tomo-san, you can get up feeling refreshed after drinking; I really like you. ( 1 # (و 2 @ a.m (daytime)
Utterance
Tomo-san! Tomo-san! Tomo-san! Tomo-san! (2- 1-------2------2----------------
TJD
전 -!!! - - v A!!!)
TC-TJD
I overslept and missed my first class. (1 (T
TJD-1
I also get up feeling refreshed after drinking, (b L b
TC-TJD-t
I'm sure that I'll get back to bed. (= X ta )())
Yes, I have trouble getting after drinking, but I slept well last night, so I feel fine. ( お
GPT-3.5
A SE [I F. ENr t < f3 b) & F 70 L- - > 1 C &
Yes, you feel better. But was drinking last night, so I am a little hungover today. ( ta <
TC-GPT-3.5
x8 L $ & 12, E, Hi < a E e L H 2 f 0 C 4 H L E
Oh, that's great Being energetic in the morning has very positive effect the start of your day, doesn't - HO ta a P K & C é R 1 D
GPT-4
it? (2
E # J E C ta 1 to 6 Ju /
1. t2,
Oh, that's great! Being energetic in the morning has a very positive effect on the start of your day, doesn't マク - tul 3 - 1 v 12 & T t WE A #サ
TC-GPT-4
it? (x
32
Utterance
I still love a big chorus!!!! ( X
p.m (nighitime)
TJD
Thanks a lot! (3' 2 - (t,)
TC-TJD
Yay! Thanks for today! It was fun! (1t== Thanks a lot! I had the best time!!!! 2 (< "
TJD-t
TC-TID-1
It was great.... Thank you so much for today! (1
That's right! That song is real crowd-plcaser when everyone sings it together. But what about the next song? ( 5 #2 12 do o ti 3 & c 5 2 * 12 R% b E & 22, Cb.. N &
GPT-3.5
Yes, big chorus is really exciting, isn't it? Are you going out for drink afterward? ( 1 di f2 5 5 1 & 2 I t k 7 3 &
TC-GPT-3.5
I really agree! It's even more exciting to hear a big cherus at this time of night, is!'t (4 12 7 5 Ā ta Ø Z D M A & & H < k. 2/ S tr ts 2 4 J J: 22.
GPT-4
I really do! But it's almost time for dinner, so why don't I we head home? S
TC-GPT-4
3 f 0B: f of UVt S. hi 3 5 カ-?)
Table 8: Examples of generated responses by each model. TC-M is time-considerable M achieved by our proposed reranking method, White and gray rows correspond to M and TC-M. Time-specific phrases are indicated by bold. All utterances and responses were originally written in Japanese and translated into English by the authors.
Table 8 shows example responses FM and r'M generated by each M and TC-M, where FM and "M are shown in white and gray rows. The result indicates I'M contained more time-specific expres- sions than r.M e.g., "I overslept" in the morning and "thanks for today" at night. The detailed anal- ysis described in Appendix A.3, For the first utterance, the original GPT-4 already generated time-specific responses, and TC-GPT-4 selected the same response as its output; namely, in this example, TC-GPT-4 considered that the original response was sufficiently coneeconsieerable.
method improved the qualities of existing models. A promising future study is to control the de- gree to which a time-considerable dialogue model actively uses time information in different situa- tions. Not only time information but all external information is valuable when used appropriately, but excessive use may harm the users' impression. Therefore, we plan to develop a mechanism to es- timate an appropriate TD@ I value that a response should have in the current situation and to output or generate a response with the estimated value.
A Acknowledgements
6 Conclusion
This work was supported by JSPS KAKENHI Grant Number 19H05693.
We proposed a new reranking method to construct time-considerable dialogue models that distinguish between always natural responses AN and tem- porally natural responses TN and actively output TN. We verified the assumption users prefer TN AN by human evaluations and introduced new metric to classify the NUTA categories of re- sponses. We proposed new reranking method to make existing dialogue models cine-considerable using the metric and empirically showed that our
Limitations
Effect of TAN's quality In our proposed rerank- ing method, we used TJD-t, which is time-aware transformer-based Japanese dialogue model ob- tained by fine-tuning, as TAN, but our experimental result showed the quality of TJD-t was lower than the other models. One of our contributions is that we empirically showed that our reranking method

---

successfully improved the S (specificity to time) score of each model even though the quality of TAN (TJD-t) is somewhat low. However, would be desirable to investigate how the effectiveness of our proposed method changes as the quality of TAN changes.
2022. Meet your favorite character: Open-domain chatbot mimicking fictional characters with only a few utterances. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACI-HLLT), pages 5114-5132.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. https: hrpp:s//arxxvorr www.:///g//////////e//. 1904/ 09751.
Different types of time information In this pa- per, we used the time of day as time information. However, there are other types of time information that have longer periods, such as day of the week and seasons. It is promising future research to in- vestigate how the quality of dialogues changes with the use of such longer periodic time information.
Melvin Johnson, Mike Schuster, Quoe V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google' s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Cultural differences in time information In this
paper, we investigate the effect of the use of time in- formation on dialogues in Japanese; however, it has been shown that the time-specific expression varies depending on the country and culture (Shwartz, 2022). Therefore, it is desirable to investigate whether the proposed method can produce time- considerable responses for different languages.
Tomohito Kasahara, Daisuke Kawahara, Nguyen Tung, Shengzhe Li, Kenta Shinzato, and Toshinori Sato. 2022. Building a personalized dialogue system with prompt-tuning Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies (NAACL-HLT) Student Research Workshop, pages 96-105
Ethics Statement
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 66- 71.
In this paper, we employed workers using a crowd- sourcing service. We made sure that the workers were paid above the minimum wage. It applies to all crowdsourcing experiments in this paper.
Young-Jun Lee. Chae-Gyun Lim, Yunsu Choi, Ji-Hui Lm, and Ho-Jin Choi. 2022. PERSONACHATGEN: Generating personalized dialogues using GPT-3. Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge, pages 29 29.48.
References
Daniel Adiwardana, Minh-Thang Luong, David R So. Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open- domain chatbot. httpp:///on 0r.5/10.45/5 arXiv. 099777.
Brian Lester, Rami AI Af-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference Em- pirical Methods Natural Language Processing (EMNLP), pages 3045-3059.
Sanghwan Bac, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang. Sang- Woo Lee, and Woomyoung Park. 2022. Building a role specified open-domain dialogue system leveraging large-scale language models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)) pages 2128-201 2250.
Jiwei Li. Michel Galley, Chris Brockett, Jianfeng Gao. and Bill Dolan. 2016. A Hiversity-promootee ob- jective function for neural conversation models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies (NAACL- HLT), pages 110-119.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877-1901.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng. Can Zheng, Junxiang Wang, Tanmoy Chowdhury. Yun Li, Hejie Cui. Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang. Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Carl Yang, and Liang 7hao. 2023. Beyond one-model-fiee specialization for large language models. https: //arxiv. org /aaa//b//...111
A survey of domain
Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang.

---

Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro, 2022. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Compulational Linguistics: ACL, pages 1317-1337.
Stephen Roller, Emily Dinan, Naman Goyal. Da Ju. Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott. Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the Conference of the Eu- ropean Chapter of the Association for Computational Linguistics (EACL), pages 300-325.
Hongyuan Lu, Wai Lam, Hong Cheng, and Helen Meng- 2022. Partner personas generation for dialogue re- sponse generation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)) pages 5200-5212.
Shoetsu Sato, Naoki Yoshinaga, Masashi Toyoda, and Masaru Kitsuregawa. 2017. Modeling situations in neural chat bots. In Proceedings of the Anmual Meet- ing of the Association for Computational Linguistics (ACL) Student Research Workshop, pages 120-127.
Iulian Serban, Alessandro Sordoni, Yoshua Bengio. Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierar- chical neural network models. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), volume 30.
Yukun Ma. Khanh Linh Nguyen. Frank Z Xing, and Erik Cambria. 2020. A survey on empathetic dialogue systems. Information Fusion, 64:50-70.
Bodhisattwa Prasad Majumder, Harsh Jhamtani, Tay- Ior Kirkpatrrik, and Julian McAuley. 2022. Achieving conversational goals with unsupervised post-hoe knowledge injection. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 3140-3153.
Vered Shwartz. 2022. Good night at 4 pm?! time pressions in different cultures. In Findings of the As- sociation for Computational Linguistics: ACL, pages 2842 2853.
Teruhisa Misu. 2018 Situated reference resolution us- ing visual saliency and crowdsourcing- priors for a spoken dialog system within vehicles. Com- puter Speech 'I8' Language, 48:1-14.
Hiroaki Sugiyama, Masahiro Mizukami, Tsunehiro Ari- moto, Hiromi Narimatsu, Yuya Chiba, Hideharu Nakajima, and Toyomi Meguro. 2023. Empirical analysis of training strategies of transformer-based japanese chit-chat systems. In Proceedings of IEEE Spoken Language Technology Workshop (SLT), pages 685-691.
OpenAI. 2023. Gpt-4 technical report. https:// arxiv. www.orgg//b//////eee 08774.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for se- quence modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (NAACL-HLT).
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. https:///ol org/10. /0/. 405.48850//aa/ att.v.rxivv...///. 2201. 08239.
Dina Utami and Timothy Bickmore. 2019. Col- laborative user responses in multiparty interaction with couples counselor robot. In Proceedings of ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 294-303.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Informa- tion Processing Systems (NeuriPS), volume 35, pages 27730-27744.
Oriol Vinyals and Quoe Le, 2015. A neural con- versational model. https: //trxiv..vvv///////// org/abs//506. 05869.
Jian Wang, Junhao Liu, Wei Bi, Xiaojiang Liu, Kejing He, Ruifeng Xu, and Min Yang. 2020. Improving knowledge-awee aware dialogue generation via knowledge base question answering, In Proceedings of the AAAI conference artificial intelligence, volume 34:05, pages 9169-91766
Ashwin Paranjape and Christopher Manning. 2021. Human-like informative conversations: Better ac- knowledgements using conditional mutual informa- tion. In Proceedings of the Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (NAACL-HLT) pages
Kohei Yamamoto and Kazutaka Shimada. 2019. Ac- quisition of knowledge with time information from twitter. In Proceedings of International Conference Asian Language Processing (IALP), pages 148- 153.
Prasanna Parthasarathi and Joelle Pineau. 2018. Ex- tending neural generative conversational model using external knowledge sources. In Proceedings of the Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 690-695.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. https://doi.org: 480.0.8850///// ar Xiv.
1.V.11180 1409. 2329..

---

Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020. Grounded conversation genera- tion as guided traverses in commonsense knowledge graphs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 2-31-2043.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). pages 2204-2213.
Wangchunshu Zhou, Qifei Li, and Chenle Li. 2021. Learning from perturbations: Diverse and informa- tive dialogue generation with inverse adversarial training. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) and the International Joint Conference on Natural Language Processing (IJCNLP), pages 694-703.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences. https: / //arxiv.org////// hwtp:/ooggg/bb////eee /tt.b/s//99....0 08593.

---

A Appendix
language modeling, and other generation tasks, In fine-tuning, we used SentencePiece (Kudo and Richardson, 2018) to tokenize utterances and re- sponses written in Japanese. Table shows the hyperparameters we set to in fine-tuning. We used the computational resouree of AI Bridging Cloud Infrastructure (ABCI) provided by the National Institute of Advanced Industrial Science and Tech- nology (AIST).
A.1 Dataset Construction
We describe the filtering rules used to obtain the NUTA dataset in Section and the fine-tuning dataset in Section 4.
NUTA dataset In Section 3, we constructed the NUTA dataset which is a collection of tuples of utterances, their spoken times, their responses, and their NUTA categories. As utterances, we extracted Japanese tweets posted between May and Decem- ber 2022 that satisfied the following conditions:
Configurations
Values
Model Architecture Pretrained Model
Transformer
TJD (Sugiyama al., 2023)
Devices
Nvidia V V100 GPU
Do not contain URLs, usernames, other tweets, parentheses,
Max tokens
4,000
Optimizer
Adafactor
Learning rate
le-04
Consists of to 30 characters,
Learning scheduler
inverse sqrt 10000
Not posted by users whose names contain "bot",
Warmup
weight decay
0.0
Loss Function
label smoothed entropy
Not Replied to another tweet.
Table 9: parametrs for fine-tuning
We randomly selected 1,000 tweets from the ex- tracted tweets and manually deleted tweets contain- ing discriminatory, violent, or inappropriate expres- sions. As result, obtained 640 appropriate tweets and used them utterances U.
GPT-3.5/4 We used GPT-3.5 and GPT-4 (Ope- nAI, 2023) as statc-of-the-art dialogue models. Since architectures and parameters of GPT-3.5/4 were not publicly available, we used OpenAl APIs gpt-3.5 5.5ttrbbbb and gpt-4 generate responses of GPT-3.5/4. We created the following prompt based on a sample prompt for dialogue tasks pro- vided by OpenAl and gave the prompt as input to GPT-3.5/4 to generate a time- aware response.
Fine-tuming dataset In Section 4, we con- structed fine-tuning dataset in the same manner Shwartz (2022) to obtain time-aware TJD by training TJD on the obtained dataset. We collected Japanese tweet-reply pairs with their timestamp posted between August 2021 to April 2022 that do not contains URLs or other tweets. As a result, we obtained 470,255,625 triplets.
The current time is [hour], and A and B are having a conversation. Taking into account the current time, generate the following B's s's response to A's utterance. However, avoid expressions like "it's [hour] o o'clock now." A: [utterance] B:
A.2 Implementational and Experimental Settings
We here describe the detailed settings of our imple- mentation and experiments.
TJD and TJD-t For a time-ume unawaee dialogue model, we used TJD which is transformer- based Japanese dialogue model with 1.6B pa- rameters trained on over two billion tweet-reply pairs (Sugiyama et al., 2023). We downloaded the trained TJD and obtained time-aware TJD, denoted by TJD-t, by fine-tuning TJD on Fairseq (Ott al., 2019), which is sequence modeling toolkit to train custom models for var- ious takes including translation, summarization,
TD@ in our proposed reranking method As shown in Eq. (5).TD@t is the maximum of CN,u,(r1 u) for all possible U T. and U could be very close to spoken time In such cases, the time dependency of might be difficult for humans to understand since its naturalness varies in too short-term. To avoid detecting such short- term time dependency, we divided 24 hours into three intervals, morning (3 to 8), noon (9 to 17),
Phttps://gii thw.//ttttttt..eeee wwt.///oollleeeeeeeeppii...

---

and night (18 2). We used 6, 13, and 22 as the representatives of morning. noon, and night, re- spectively. The division and their representatives were determined with reference to Yamamoto and Shimada (2019). We computed @D@t by Eq. using the above T excluding the representative of spoken time I. For instance, for ( = 10, we use T = (6, 22} to evaluate TD @t because the rep- resentative of 10 is defined as 13. We conducted the same experiments as Section 4 using the above TD@t, and Table 10 and I1 show the results. The results indicate that the accuracy TN classifi- cation was slightly improved by introducing the above time intervals but not for the average accu- racy. Therefore, it cannot be said that either TD with intervals is better than the original TD.
Naturalness
TD
PCMI
TUN
TAN
@all
@all
@1
N
0.60
0.68
0.48
0.61
0.48
0.55
T
0.54
0.54
0.65
0.58
0.51
0.48
Table 10:
Accuracy of N/U and T/N classifications the morning/day/naight quantization
TD
PCMI
@all
@t
@all
@f
AN
0.50
0.39
0.41
0.38
TN
0.48
0.61
0.39
0.51
TU
0.41
0.15
0.30
0.14
AU
0.46
0.45
0.34
0.32
Ave.
0.46
0.40
0.36
0.34
Table 11: Accuracy of NUTA classification with the moming/day/naihht quantization
A.3 Time-specific Expressions
We conducted subjective evaluation to count the number of time-specific expressions by authors. As a result, our method increased their occurrences by 80% and 8% for TJD-t and GPT-4, respectively. These results on occurrences of time-specific ex- pression are consistent with the improvements in S shown in Table 7.


Reviews:
[{"review_id": "8ppzx7b2Sv", "reviewer": "Reviewer_Ytjk", "summary": "", "questions": "- Regarding L.257-263, what was the degree of agreement among the annotators?\n- I agree with the advantage of the reranking method (e.g., easy to apply to existing models). On the other hand, how much does the reranking method improve the accuracy of the task compared to existing methods that straightforwardly train to generate time-aware responses, such as the method of Sato et al. mentioned by the authors in L153?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback and comments. \nWe appreciate the insights and suggestions of the reviewer.\n\n# Questions\n\nQuestion 1:\n> Regarding L.257-263, what was the degree of agreement among the annotators?\n\nTwo annotators evaluated 100 responses for the four scores (SU, ST, S, and I) with either 0 or 1.\nThe degree of agreement among the two annotators was as follows: SU at 0.97, ST at 0.86, S at 0.83, and I at 0.71.\nThis agreement rates high enough to trust the annotated scores.\nThe agreement rate for I (interestingness) is slightly lower than for the other indicators because interestingness judgments are more subjective than others.\n\n\nQuestion 2:\n> I agree with the advantage of the reranking method (e.g., easy to apply to existing models). On the other hand, how much does the reranking method improve the accuracy of the task compared to existing methods that straightforwardly train to generate time-aware responses, such as the method of Sato et al. mentioned by the authors in L153?\n\nTJD-t is a model trained in the same manner as Sato et al., an existing method that straightforwardly trains to generate time-aware responses. TC-TJD-t is a model obtained by applying our reranking method to TJD-t.\nTable 7 shows that TC-TJD-t improved all scores (ST, SU, S, I, and SSI-t) compared to the original TJD-t.\nThese results indicate that our reranking method successfully improved a model that was straightforwardly trained to generate time-aware responses.\n\n\n# Reasons To Reject\n> The experiments were conducted using only Twitter-sourced data. Due to the difficulty of redistributing the tweet data and the fact that the Twitter API is currently unavailable, it would not be easy to reproduce or follow up on this paper. Lack of reproducibility is a serious problem in scientific papers, and I am concerned about this point.\n\nAs the reviewer pointed out, the free Twitter API is unavailable, making it difficult to collect tweets without incurring costs.\nHowever, Twitter Inc. offers a paid Twitter API to collect tweets, including those we used in this paper.\nConsequently, everyone can collect the same dataset by the paid API with our filtering rules described in Appendix A.1.\nFurthermore, all base models used in our experiments are publicly available.\nGiven these points, it is NOT difficult to reproduce or follow up on this paper. \nWe will consider releasing our code If the reviewer's concern is reproducibility; however, we are not allowed to release the Twitter dataset we constructed in this paper due to its terms of use.\n\n\n# Typos Grammar Style And Presentation Improvements\n\nThank you for your constructive comments. \nThey are valuable suggestions for improving the readability and organization of the paper, and we will incorporate them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The challenge to develop the time-considerable dialogue models is interesting.\n- The NUTA category, consisting of four types (AN, TN, TU, AU), proposed to discuss the validity of responses in a time-aware manner, which will be useful for future research.\n- The careful notation and mathematical explanation make it easy to follow the description of the proposed method and the discussion.", "reasons_to_reject": "- The experiments were conducted using only Twitter-sourced data. Due to the difficulty of redistributing the tweet data and the fact that the Twitter API is currently unavailable, it would not be easy to reproduce or follow up on this paper. Lack of reproducibility is a serious problem in scientific papers, and I am concerned about this point.", "questions_to_authors": "- Regarding L.257-263, what was the degree of agreement among the annotators?\n- I agree with the advantage of the reranking method (e.g., easy to apply to existing models). On the other hand, how much does the reranking method improve the accuracy of the task compared to existing methods that straightforwardly train to generate time-aware responses, such as the method of Sato et al. mentioned by the authors in L153?", "paper_topic_and_main_contributions": "This paper addressed generating more attractive responses by using external information beyond the past context of the dialogue, specifically, using time information to generate time-aware responses. After experimentally confirming the plausibility of the hypothesis that \"people prefer responses that are natural only at specific times to those that are natural always,\" they proposed a reranking method for preferentially generating \"responses that are natural only at specific times.\" Experiments showed the usefulness of the proposed method when applied to a standard generative model (e.g., GPT-4) by automatic and human evaluation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- If I understand correctly, the actual data the authors have created is only in Japanese. If that is the case, including the actual Japanese data as examples (especially figures and tables) in the paper would be accurate. Then, it would be desirable to include an English translation to benefit many readers.\n - L.517: Please define N, \\delta, p, respectively."}}, {"review_id": "sRM1zva72U", "reviewer": "Reviewer_eL5Z", "summary": "", "questions": "\nl. 259: how were the \"experts\" selected, what was their expertise?\nl. 512: which prompts were used for GPT models? Can you add them?\nl. 524: can you assess the impacts that this usage of time intervals (instead of single hours) had? \nl. 541: why did you measure lengths and n-grams? What does diversity have to do with the goal of this paper? \nl. 574ff: this holds only for the weaker models, but NOT for GPT-4, wich is out of the box already excellent (SSI-t 0.83). Do you agree?\nl. 593: did you count the number of time-specific expressions (if so: where is the data?), or how did you come to this conclusion?\n\nwhy do you not publish your dataset?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback and comments. We appreciate the insights and suggestions of the reviewer.\n\n# Questions\n\nQuestion 1:\n> l. 259: how were the \"experts\" selected, what was their expertise?\n\nThe \"experts\" are in-house workers specialized in annotating and evaluating dialogues and have worked in their positions for at least five years.\n\n\nQuestion 2:\n> l. 512: which prompts were used for GPT models? Can you add them?\n\nThe template prompts used for GPT models are shown in Appendix A.2.\nThe detailed prompts were created by replacing the [utterance] and [hour] placeholders in the template with concrete values.\nWe omitted the detailed prompts since the number of detailed prompts is the same as the dataset size.\n\n\nQuestion 3:\n> l. 524: can you assess the impacts that this usage of time intervals (instead of single hours) had?\n\nThere are various ways to determine time intervals: hours, morning/noon/night, seasons, and more.\nWhereas a single hour is just a unit of time, morning/noon/night and seasons are defined based on the human lifecycle. \nThus, the latter is more intuitive for human understanding and used as meaningful intervals in other studies: e.g., Yamamoto et al. used morning/noon/night, and Sato et al. used seasons.\nWe used morning/noon/night as a time interval, similar to Yamamoto et al., to provide a more intuitive analysis.\n\n\nQuestion 4:\n> l. 541: why did you measure lengths and n-grams? What does diversity have to do with the goal of this paper?\n\nLength and distinct (n-grams) are metrics commonly used to measure the diversity of dialogue models.\nLow-diversity models tend to generate monotonous responses and are often considered low-quality.\nOur paper aims to generate responses with high time dependency while avoiding the side effects of sacrificing diversity.\nThus, we calculated lengths and distinct (n-grams) to ensure our method maintained the diversity of the base models, and the experimental results showed that our method did not cause a reduction in diversity.\n\n\nQuestion 5:\n> l. 574ff: This holds only for the weaker models, but NOT for GPT-4, which is out of the box already excellent (SSI-t 0.83). Do you agree?\n\nAs the reviewer pointed out, applying our proposed method may not enhance the naturalness of stronger models such as GPT-4.\nThis paper aims to increase responses' \"specificity to time\" while maintaining their naturalness, and our method successfully increased S (specificity to time) of all models, including GPT-4, without decreasing their naturalness.\nThe result implies that our method is effective not only for weaker models but also for stronger models like GPT-4.\n\n\nQuestion 6:\n> l. 593: did you count the number of time-specific expressions (if so: where is the data?), or how did you come to this conclusion?\n\nAs shown in Table 7, our method improved S scores by 66.7% and 4.5% for TJD-t and GPT-4, respectively.\nWe conducted a subjective evaluation to count the number of time-specific expressions by authors.\nAs a result, our method increased their occurrences by 80% and 8% for TJD-t and GPT-4, respectively.\nSince these results on occurrences of time-specific expression are consistent with the improvements in S shown in Table 7, we omitted the results in our paper for brevity.\nGiven the reviewer's comment, we consider adding the results in Appendix.\n\n\nQuestion 7:\n> why do you not publish your dataset?\n\nThe Twitter API policy does NOT allow us to redistribute the dataset constructed by the Twitter API.\nThus, we cannot publish the dataset we used in this paper.\n\n\n# Reasons To Reject\n> the results show that only weak models benefit from the introduced methods, but not the strongest model, GPT-4\n\nAs mentioned in the answer to Question 5, our method improved S not only for weak models but also for strong models such as GPT-4.\nConsequently, GPT-4 would also benefit from our reranking method to improve the \"specificity to time\" of its responses.\n\n\n# Typos Grammar Style And Presentation Improvements\n\nThank you for your constructive comments. \nThey are valuable suggestions for improving the readability and organization of the paper, and we will incorporate them in the revision.\n\n\nPresentation Improvement 1:\n> l. 103: an example what time-aware vs time-considerable is would be helpful here\n\nFigure 1 shows four examples of AN, TN, TU, and AU responses.\nThe reviewer pointed out that providing another example that compares explicitly AN and TN is beneficial for readers to understand our idea.\nWe plan to add such an example to our paper.\n\n\nPresentation Improvement 2:\n> l. 457: how is delta chosen?\n\nDelta is a threshold to filter generated responses based on their naturalness; we can remove responses with lower naturalness by setting a higher value to the delta.\nTherefore, when the response generation model is weak, this filtering mechanism can improve the naturalness of the generated responses. \nWe set the delta value to 0 because we used sufficiently strong models for response generation and did not aim to improve their naturalness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting research question\n- introduced metrics are reasonable and helpful for assessing time appropriateness\n", "reasons_to_reject": "- the results show that only weak models benefit from the introduced methods, but not the strongest model, GPT-4\n", "questions_to_authors": "\nl. 259: how were the \"experts\" selected, what was their expertise?\nl. 512: which prompts were used for GPT models? Can you add them?\nl. 524: can you assess the impacts that this usage of time intervals (instead of single hours) had? \nl. 541: why did you measure lengths and n-grams? What does diversity have to do with the goal of this paper? \nl. 574ff: this holds only for the weaker models, but NOT for GPT-4, wich is out of the box already excellent (SSI-t 0.83). Do you agree?\nl. 593: did you count the number of time-specific expressions (if so: where is the data?), or how did you come to this conclusion?\n\nwhy do you not publish your dataset?\n", "paper_topic_and_main_contributions": "This paper investigates how time information can be incorporated into dialogue systems. It introduces corresponding notation and metrics, and shows how to re-rank outputs from 4 dialogue systems for a Japanese dataset to improve time-appropriate responses.  ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 101: add remark that you will verify this assumption lateron in the paper\nl. 103: an example what time-aware vs time-considerable is would be helpful here\nl. 149: evaluated-> evaluate\nl. 172-181: could be skipped, was already said before\nl. 249: definition what (S) means is unclear\nl. 259: use other character (not S) for scores, since S is already used for Specifity of Time, and italic font is hard to distinguish \nl. 275: insert \"more\" before TN\nl. 325ff: sentence seems corrupted\nl. 355: remove \"let\"\nl. 457: how is delta chosen?\nl. 565: there is something missing before M, probably \"all\"\ntable 8: grey/white lines are not visible (at least in my black-and-white printout)\n\noverall, the paper is sometimes hard to follow since a huge amount of abbreviations is introduced. Maybe check if some abbreviations are really necessary"}}, {"review_id": "9zfJDsmL0G", "reviewer": "Reviewer_GNUY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing valuable feedback and comments. \nWe appreciate the insights and suggestions of the reviewer.\n\n# Reasons To Reject\n> Too narrow. It seems to be that their proposal can be extended to other contextual information (beyond time of day) and that their algorithms should be extended to consider other context. They should evaluate how their approach varies in accuracy when considering different contexts.\n\nWe consider that our method is NOT narrow.\nOne of our contributions is that this paper distinctly separated contextual information into \"persistent information\" and \"optional information.\"\nWhereas optional information, such as user information, sometimes exists but is sometimes absent, persistent information, such as time information, always exists.\nThis paper primarily focused on time information, an essential example of such persistent information. \nWe proposed a new metric for evaluating dependency between utterances and persistent information, while existing metrics are for optional information. \nMoreover, we introduced the NUTA category for responses based on time dependency, which helps analyze utterances with time information.\nUsing our new metric and the NUTA category, we proposed a method to generate appropriate responses according to time information without modifying the base model.\nWe validated the efficacy of our method through experiments.\nDue to space constraints in the paper, we used the time-of-day interval in our experiments; however, we can apply our method to other time intervals and persistent information.\nAs acknowledged by other reviewers, our proposed method must be helpful for future research.\n\n\n# Typos Grammar Style And Presentation Improvements\n\nThank you for your comment. \nIt is a valuable suggestion for improving the readability and organization of the paper, and we will incorporate it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Interesting proposal to consider additional context when ranking LLMs' responses. ", "reasons_to_reject": "Too narrow. It seems to be that their proposal can be extended to other contextual information (beyond time of day) and that their algorithms should be extended to consider other context. They should evaluate how their approach varies in accuracy when considering different contexts. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a method to rerank an LLM's responses based on the time of day. They propose a method to classify responses based whether they are always (un)suitable independent of the time of day or temporarily (un)suitable.  ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 4, 2nd col. header: \"Chage of naturalness\" should be \"change\"?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method to add time awareness to conversational models. The reviewers generally like the approach and the importance of the problem. However, the reviewers also point out that improvements using this method are only found in smaller models, that the data used is proprietary, and that the contribution may be too narrow. The authors respond to most of the reviewers concerns and I think this would be a good Findings paper.
Decision: Accept-Findings