This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

1

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

2
3
4
5
6
7
8
9
10
11
12
13
14

RYOICHI SHIBATA, Keio University, JapanSHOYA MATSUMORI, Keio University, JapanYOSUKE FUKUCHI, National Institute of Informatics, JapanTOMOYUKI MAEKAWA, Keio University, Japan
MITSUHIKO KIMOTO, ATR, Japan and Keio University, JapanMICHITA IMAI, Keio University, Japan
When people are talking together in front of digital signage, advertisements that are aware of the context of the dialogue will work themost effectively. However, it has been challenging for computer systems to retrieve the appropriate advertisement from among the15

many options presented in large databases. Our proposed system, the Conversational Context-sensitive Advertisement generator

16(CoCoA), is the first attempt to apply masked word prediction to web information retrieval that takes into account the dialogue

17

context. The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then18

CoCoA automatically generates a context-sensitive expression as a complete search query by utilizing a masked word prediction

19

technique that adds a word related to the dialogue context to one of the prepared Core-Queries. This automatic generation frees

20

the advertisers from having to come up with context-sensitive phrases to attract users’ attention. Another unique point is that the

21
22
23

modified Core-Query offers users speaking in front of the CoCoA system a list of context-sensitive advertisements. CoCoA wasevaluated by crowd workers regarding the context-sensitivity of the generated search queries against the dialogue text of multiple

24

domains prepared in advance. The results indicated that CoCoA could present more contextual and practical advertisements than other25

web-retrieval systems. Moreover, CoCoA acquired a higher evaluation in a particular conversation that included many travel topics to26which the Core-Queries were designated, implying that it succeeded in adapting the Core-Queries for the specific ongoing context

27

better than the compared method without any effort on the part of the advertisers. In addition, case studies with users and advertisers

28
29
30
31

revealed that the context-sensitive advertisements generated by CoCoA also had an effect on the content of the ongoing dialogue.
Specifically, since pairs unfamiliar with each other more frequently referred to the advertisement CoCoA displayed, the advertisements
had an effect on the topics about which the pairs spoke. Moreover, participants of an advertiser role recognized that some of the32

search queries generated by CoCoA fitted the context of a conversation and that CoCoA improved the effect of the advertisement. In

33

particular, they assimilated the hang of designing a good Core-Query at ease by observing the users’ response to the advertisements

34

retrieved with the generated search queries.

35
36

CCS Concepts: • Human-centered computing → HCI theory, concepts and models; User studies.

37
38

Additional Key Words and Phrases: dialogue, context, advertisement, query generation, mask prediction

39
40
41
42
43
44
45
46
47
48
49
50
51
52

1

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

53

Core-Queries

54

Enjoy the spectacular view

55
56

Summer family travel

57:

Search results of“ Enjoy the spectacular view the U.S. ”

58
59

H mm

…Iw

sell…
ant toSearch query

60
61
62
63

Advertiser

64
65

I want to travel to the U.S.

66

CoCoA

Core-Query+Context

Advertiser’s
database“ Enjoy the spectacular view + the U.S. ”

DBSoundsnice!

Great! I’d like a placewith a beautiful view.

67
68
69
70
71

User

72
73
74
75
76

Fig. 1. Example of CoCoA in use. CoCoA refers to the user’s dialogue context and selects “Enjoy the spectacular view” from several
Core-Queries created by the advertiser because the user’s dialogue contains “a beautiful view”. In addition, the term “the U.S.” in the
dialogue is appended to the Core-Query to generate a search query. Then, the database prepared by the advertiser is searched using
the search query. The search results relevant to the dialogue context are then presented to the user.

77
78
79
80
81
82

1

INTRODUCTION

Intelligent communication systems such as smartphones, wearable devices, and voice assistants are becoming more and83

more a part of our daily lives. These systems make it possible to present ads in a context-sensitive manner. Existing84

systems have utilized time- or user-dependent information as the context, such as weather [7], location information

85[21, 35], and video viewing history [30]. On the other hand, to the best of our knowledge, there are no systems that86
87

utilize the contents of real spoken dialogue as the context or present advertisements relevant to the context in real time.

88

Conversational dialogues contain a variety of topics, and the topic often changes in a relatively short period of time,

89

which makes it extremely difficult to design advertisements that match the content of the dialogues in advance. For

90example, even in a dialogue limited to the travel domain, various topics such as destination, budget, duration, season,

91
92

etc. may be discussed as the dialogue progresses. Our key idea in this study is that advertisements can be optimized for93

individual users if we dynamically transform the advertisements according to the context of the dialogue between two

94

users. To this end, we propose CoCoA, a system that dynamically presents advertisements that match the context of the

95
96
97

dialogue. An example of CoCoA in use is shown in Fig. 1. First, the advertiser prepares several Core-Queries, which
represent the core messages he wants to convey. When users talk with each other in front of a digital signage on which

98

CoCoA is running, CoCoA selects the appropriate Core-Query given the context of the dialogue and then appends

99

an appropriate word from the dialogue to it to generate a search query that is used to extract advertisements from an100

advertisement database. This search query is used to advertise the corresponding product from the advertiser’s product

101
102

database. In this way, CoCoA is expected to present effective advertisements by generating search queries that fit the103

dialogue. Let us consider a scenario in a restaurant to illustrate the benefits CoCoA offers to users. Recently, many104

2This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

105

restaurants have started using tablet devices on tables to display menus and handle customers’ orders. Our system can

106

attract customers’ interest by displaying context-aware advertisements on the tablets. From a user’s perspective, the

107
108
109
110

advertisements related to the current conversation can be fruitful information to expand their conversation at dining. In
addition, offering restaurant coupons, which are funded by arbitrary advertisers, can encourage the use of our system.
The novelty of this paper lies in the following two points.

111
112

Defining the search query. We define the search query as the combination of “the phrase that the advertiser

113

wants to convey” and “the keywords in the dialogue”. This definition overcomes the difficulty of generating a114

search query that fits the dialogue. The phrase is called the “Core-Query”, since it contains the core message of

115
116

the search query, and the keyword is referred to as the “complementary word”, as it depends on the context in117

the dialogue. Since there are a variety of candidates for the complementary word, it is difficult to prepare a118

complete set of search queries applicable for any dialogue in advance. Therefore, we divide the search query

119
120
121

into two parts and consider them separately. The appropriate Core-Query according to the context is selectedfrom a set of several Core-Queries prepared in advance, and the appropriate complementary word is selected

122

from the dialogue history by using a Query Completer. The search query is then generated by concatenating123

the Core-Query and the complementary word. This is how the Core-Query is transformed into a practical

124
125
126
127

search query. Even though the Core-Query itself is an abstract phrase, the generated search query contains theconcrete complementary word that is in line with the dialogue. Thus, the search query can come up with anadvertisement that effectively attracts the user.

128

Applying mask prediction to information retrieval. The second point is that the extraction of keywords in

129

the dialogue, which are complementary words, is achieved by using mask prediction. Mask prediction, which

130
131

predicts words that are hidden in a sentence, is known to be effective in learning language models such as

132

BERT, but it has not been used much except for the purpose of acquiring language models. Our research is the133

first to implement the mechanism of mask prediction into a framework for information retrieval that takes into

134
135
136
137
138

account the dialogue context. As a result, CoCoA can continue to dynamically present advertisements that are
relevant to the dialogue from a small number of Core-Queries as the dialogue progresses.
We conducted two evaluations to investigate the design of CoCoA. The first was a crowdsourced evaluation utilizing

139

600 workers to examine the appropriateness of the proposed function of finding context-sensitive complementary words

140

against target Core-Queries in terms of the subjective interpretation of participants. The appropriateness was evaluated141

from two perspectives. First, we conducted a context sensitivity test to determine whether CoCoA could generate

142

appropriate search queries for a given context. We found that CoCoA outperformed two other methods that either did

143
144

not have a complementary word or had a word selected by Google Suggest. Second, we conducted a robustness test145

to evaluate the system’s robustness by having each participant create a query to see if the system could adequately

146

generate advertisements for all ad creators and all Core-Queries. We found that it was able to do so regardless of the147

variations of Core-Queries.

148
149

The second evaluation was a descriptive study [48] consisting of two case studies. The objective here was to clarify

150

how pairs of participants interacted with CoCoA while conversing and how the advertisers evaluated the effect of the151

Core-Queries they created on the actual conversation between users. Usability tests in a laboratory setting provided

152

us with insights into the effect of the advertisements generated by CoCoA on both the conversation and behaviors

153
154

of the participants. We first conducted a user case study with six people (three pairs) in which we observed how the155

search query generated by CoCoA in real time during user conversations affected the dialogue. We found that the156

3

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

157

frequency of system usage changed depending on the level of familiarity between the individuals in a pair, and that158

the domain of the dialogue changed depending on the system. Second, we conducted an advertiser case study with

159
160

three groups of one advertiser and two users in which we asked the advertiser to observe the interaction between

161

individuals who were using CoCoA to evaluate its usefulness in actual conversations from the advertiser’s viewpoint.

162

The advertiser created several Core-Queries in advance and CoCoA then used them to generate Search queries. These

163

queries were displayed during the conversations of the users and then evaluated by the advertiser. We found that, from

164
165

the advertiser’s perspective, CoCoA generated search queries that were more contextually relevant and effective than166

the compared method, which was prepared in the same way as the crowdsourced evaluation. This article constitutes an

167

extension of our original paper [38] published in the Proceedings of the 2022 Conference on Intelligent User Interfaces.

168

The main addition is the “advertiser case study” in Section 5.

169

This paper is organized as follows. Section 2 describes related work in terms of advertising methods and query

170
171

expansion. Section 3 proposes our system named CoCoA and explains in detail how it generates search queries from172

dialog history. Section 4 describes two experiments to evaluate whether CoCoA generates appropriate search queries

173

for given dialogues and discusses the results. Section 5 reports two case studies of face-to-face human conversations

174
175
176

using CoCoA and discusses the results. Section 6 discusses future work on our research, and Section 7 concludes thepaper.

177
178

2

RELATED WORK

179
180
181
182

2.1

Interactive advertising

Digital displays have been increasing in public spaces over the past several years. One of the advantages of the digital

183

displays is that the content can be changed in real time and therefore information can be presented in a way that is184

more impactful to the user. Such displays are now frequently being used to present advertisements [34]. Digital displays

185

are attractive to advertisers because they can present video images that are more likely to draw user interest than

186
187

traditional paper-based, unchanging text and images [16, 31]. Moreover, by introducing interactive advertisements

188

that allow users to not only look at an ad but also touch it, appropriate advertisements can be presented in response to189

specific user desires, thereby increasing the likelihood of a sale [4, 15]. The interactive ads that have been implemented

190

on TV [5] have demonstrated an improved advertising effect [42, 43]. [12] proposed an interactive mobile ad framework

191
192
193

that allows users to filter out ads they are not interested in. [39] developed an interactive coupon-giving robot system
for a shopping mall and demonstrated the usefulness of social robots for advertising.

194
195

2.2

Context-aware advertising

196
197

Systems that utilize the user’s context information to present an appropriate advertisement are known as context-aware

198

advertising. The contextual information used by such systems includes the weather [7], location information [21, 35],

199
200
201
202

and video viewing history [30]. However, to the best of our knowledge, a system that uses dialogue content as context
and presents advertisements in real time has not been proposed. However, there is another type of system similar tocontext-aware advertising that “provides appropriate advertisements to users”: the recommender system.

203

Recommender systems estimate user preferences and then use them as a basis for recommending appropriate

204

content and products. They have been used for recommending video content such as in-vehicle systems [14], tourist

205
206

attractions [53], and even YouTube [8] and Netflix [13] shows. A system that provides higher quality and more accurate

207

recommendations through interaction with the user is called a conversational recommender system (CRS) [6, 22, 41].

208

4

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

209[17] created the goal-driven recommendation dialogue dataset (GoRecDial), which can be utilized to estimate the210

user’s preferences and recommend movies by repeatedly asking questions in the dialogue. [54] proposed a new task

211
212

topic-guided conversational recommendation that focuses on active guidance from the system side, as opposed to213

the traditional user request-driven dialogue, and built a large high-quality dataset that incorporates topic threads for214

topic control. In the dialogue, BERT was used for topic estimation, and GPT-2 was used for sentence generation. [26]

215
216
217

proposed a new task, dataset, and baseline that can actively and naturally lead from open-domain non-recommendation
dialogs such as chitchat to recommendation dialogs in user-system interaction. These studies require interaction and218

questions between the system and the user to estimate preferences and make recommendations, which is different from219

our work, in which advertisements are continuously presented on the basis of user interactions without questions.

220
221

2.3

Query expansion

222
223

Dialogue is a unique event, so a pre-prepared advertisement may not always be a good fit. If the advertisement can be224

appropriately transformed to fit the context of the dialogue, the effectiveness of the advertisement can be enhanced.

225

The idea of transforming sentences to fit the context (in a broad sense) has been attracting particular attention in the

226
227

field of information retrieval. This has led to the concept of query expansion (QE), where search results more desired228

by the user are presented by transforming the search query he or she entered. Consider the case where the user’s

229

input query is “senior citizens”. In this case, since the synonym “the elderly” is more frequently searched, the user’s

230

query is replaced with the latter. Thus, in query expansion, the user’s initial query is reformulated as needed. Several

231
232

methods of query expansion use named entity recognition [9, 49], which classifies named entities such as “Japan” or

233“Albert Einstein” into appropriately named entity tags such as “Location” or “PersonName”, respectively. [9] used the234

features of named entities and links to Wikipedia and freebase knowledge bases containing structured named entity

235

tags and text for query expansion. However, because the types of queries that can be processed depend on the number

236
237

of named entity tags, named entity-based query expansion methods have a problem in that the domain is limited when238

the number of named entity tags is small. In [33], the use of BERT [10] was shown to improve the accuracy for longer

239

queries. These methods utilizing query expansion are aimed at appropriately transforming queries to improve search

240

performance, which differs from our goal of appropriately completing words for Core-Queries.

241
242
243

There are relatively few studies on query expansion methods referring to conversation history. Among them, [50]proposed a method that appropriately extends a query by referring to the conversation history between users when a244

user inputs a query, and evaluated it using the CAsT dataset. In the CAsT dataset, the queries become more ambiguous

245with each turn of the conversation. However, the dialogue history included in the dataset is short (one sentence) and

246
247

does not cover long dialogues where the topic changes dynamically.

248

Query auto completion (QAC) is a method that takes a part of a query from a user and generates multiple completion249

candidates in addition to the query. A typical example of query auto completion is Google Suggest, which is used in

250

Google Search: when a query is entered, automatically completed query candidates are displayed at the bottom of

251
252

the search bar. [2] proposed most popular completion (MPC), which generates a complemented query based on the253

frequency of searches along with the words entered by the user as a query. [18] achieved faster query completion254

by referring to a database created in advance. [23, 40] introduced a personalization feature to achieve more accurate

255

query auto completion that adapts to each individual by utilizing user information. Instead of using a database, [47]

256
257

uses a deep network-based language model to find the most likely complementary words for the user’s current input.

258

However, the methods discussed here perform query auto completion using only the user’s input query or only the259

query and user information. Our approach differs in that we perform query auto completion with reference to the260

5This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

261

conversation history. Research on automatic query completion using dialogue history includes [45], where the authors

262

collected dialogue data and search history data in a conversational task using Web search and then used it as a basis to

263
264

create a topic model utilizing Dirichlet-Hawkes processes (DHP). When the user performs a Google search, the search265

history and dialogue history are used to properly re-rank the query candidates in Google Suggest, thereby reducing

266

keystrokes and other search efforts.

267
268
269

[11] has the advertiser create the item keywords in advance, and then extends them. These item keywords are similar
to the Core-Queries in our work, but since they are prepared as words, the range of expression is limited. In our system,270

we enable advertisers to create Core-Queries more intuitively by allowing them to be entered as sentences. Furthermore,271[11] does not handle dialogue history, whereas we aim to provide real-time advertisements to users who are talking to272

each other.

273
274
275

2.4

Mask language models for information retrieval

276

We focus on BERT [10] as a critical mechanism for developing our system. The reasons for employing BERT are that it277

is a model built on a large corpus to handle open domains. Moreover, BERT has outperformed existing methods in tasks

278
279

such as document retrieval [29, 52] and question answering [32, 51]. [1] developed a system that integrates the open

280

source information retrieval toolkit Anserini with BERT to perform end-to-end document retrieval for large document281

collections. In [37], BERT was utilized to calculate the similarity of unlabeled query-question pairs to increase the QA

282
283
284

data size and achieved high-performance retrieval. [19] improved the BERT-based architecture and created a modelthat allows for real-time document ranking with improved speed and accuracy, and implemented it in a search engine.

285

However, few studies have focused on the function of predicting masked words in BERT to extend queries and perform

286

information retrieval. Because predicting masked words can be utilized to extend queries in any domain, we focus on287

this function.

288
289
290

2.5

Chat dialogue system

291

The most basic chat dialogue system uses a sequence-to-sequence framework [44], which can generate simple con-292

versations when given a large conversational training dataset. Although it is suitable for short textual conversational

293
294
295

tasks, it is known to generate generic responses such as “I don’t know” in open-domain dialogues because it assumes a
one-to-one context-response relationship. To avoid this problem, methods to apply objective functions for one-to-many

296

problems [24] and to control the training of natural language generation from the training data by adding a bias to the297

corpus during training [27] have emerged. Models that achieve a specific task-oriented dialogue during chatting include

298
299

those that acquire task-specific knowledge through pre-learning [3], those that extract knowledge from the outside300

and input it with queries during inference in the context of restaurant reservations [25], and those that learn language

301

processing and recommended entity selection in an end-to-end manner [28]. Although these approaches work well302

for certain tasks and domains in chat dialogues, they require large datasets to fit the models to those tasks and also

303
304

demand expertise on the part of the developer, making it difficult to design the tasks. In this paper, we use advertising305

as a task and aim to enable advertisers without computer science backgrounds to appropriately insert advertisements

306

into open-domain dialogues with only the simple task of preparing phrases.

307
308

2.6

Problem setting

309
310

We assume that the generated search query is used to search the database prepared by the advertiser to present the

311

information sought by the user. Under this assumption, it is necessary to generate the search query used in the search

312

6This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337

Fig. 2. System configuration diagram

338
339
340
341
342

according to the context of user dialogue. In this research, we evaluate whether the context of the user dialogue isproperly taken into account when creating advertisements.

343

We focus on human-human conversations rather than human-computer conversations because we wanted to develop344

a method that would work well in a variety of contexts. A conversation between a human and a chat system restricts345

the context to just one specific topic that the system can handle. The specific topic is not appropriate for the target of

346
347

our method. Furthermore, if the advertisement is presented in face-to-face human communication, it can be shared by348

both speakers and can facilitate the conversation. These are the reasons we focus on human-human interaction.

349

Toward the development of technology to generate advertising text in response to user interaction, we will build a350system to present advertisements to users during interaction. Figure. 2 shows a diagram of the conversation context-

351
352

sensitive advertisement presentation system. The system allows advertisers to create Core-Queries and store them in353

the Core-Query DB. The user’s dialogue is recognized by the speech recognition module through the microphone, and354

the dialogue text is input to the CoCoA proposed in this research through the dialogue history management module.

355

CoCoA selects a Core-Query appropriate for the dialogue text from the Core-Query database, converts the Core-Query

356
357

into a search query appropriate for the topic of the dialogue, and outputs the search query. Based on the search query358

generated by CoCoA, the search module retrieves a digest of relevant advertising information from the advertising359

database or the Internet, creates an advertising list with touch-clickable buttons. The search-result-display module

360
361

presents the advertising list on the display to the user. The user can view the presented list of ads during a conversation.

362

If the user finds an advertisement in the list that interests him or her, he or she can tap on it. Then the search module363

retrieves the details of the advertisement information and displays them on the screen for the user. The advertisement364

7This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

365

lists based on search queries and ads selected by the user with a tap may reversely have an effect on the user’s366

conversation.

367

In this research, we study the CoCoA mechanism based on the conversation context-sensitive advertisement

368
369

presentation system, aiming at the development of a technology that generates advertisements in response to user

370

interaction. In addition, by using this system with CoCoA incorporated, we will support users to actually engage

371
372
373

in face-to-face conversation and, at the same time, view advertisements presented in accordance with the topic ofconversation. Furthermore, the system will contextualize Core-Queries created by advertisers without effort to consider

374

individual conversational situations according to the content of the user’s interaction, and present the user with375

advertisements that are appropriate to the conversational situation.

376
377
378
379
380

2.7

Scope of this research

In this section, we clarify the technical issues that this research aims to address. This study focuses on the mask

381

prediction functionality enabled by the large-scale language model, and challenges the question of how far advertising382

queries can be created by converting Core-Query text, which is advertising text, within the range of capabilities that can

383
384
385
386

be achieved by contextualizing the text using a language model. The CoCoA module, the core of the system, receives
texts of the dialogue and the Core-Query, and then performs a technique to contextualize the Core-Query text using alarge-scale language model.

387

When aiming to present advertisements that appeal to users, it would be highly effective to serve advertisements

388

according to the emotions and preferences of users during the conversation. It is also possible to estimate the mood

389
390

and preferences of the conversation by conducting sentiment and emotional analysis of the user’s conversational text.

391

However, the effect of extending search sentences with a large-scale language model itself is still unclear, so this paper

392

will focus on that aspect of the research.

393
394
395
396
397

3

COCOA

In this paper, we propose CoCoA, a method that dynamically selects the Core-Query from a list of candidates prepared398

in advance in accordance with the context of the dialogue and then appends the complementary word from the ongoing399

dialogue to the Core-Query. CoCoA combines the “Core-Query Selector”, which selects the appropriate Core-Query

400

from a list of candidates, and the “BERT-Query Completer (B-QC)”, which determines complementary words, to enable

401
402

advertisers to generate context-sensitive and effective advertisements in real time in any context with only a few

403

Core-Queries. Since we assume that the generated search query is used to perform a database search, this sentence is404

considered a search query and does not need to be grammatically correct. The flow of the CoCoA process is shown in

405

Fig. 3.

406
407
408

3.1

Create the Core-Query

409
410

The Core-Query is an abstract sentence that forms the core of a recommendation, and is prepared in advance by the411

advertiser. Examples are “Enjoy the spectacular view” or “Summer family travel” for travel agencies. We chose to412

make Core-Queries abstract sentences because they are more extensible as advertisements. In the above example, by

413

making the sentence abstract without limiting the destination, it is possible to respond to any context by supplementing

414
415
416

keywords from the dialogue in later steps.
8This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

417
418
419
420
421
422
423
424
425

Time
Dialogue sentences
1 I haven’t traveled abroad in several years.
2
Me neither. Where do you want to go?
3
The U.S. or somewhat.
4
Good.
5
I’d like a place with a beautiful view.
ID
1
2
3
:

CoCoA
Search query

Core-Query
Selector

Core-Queries
Enjoy the spectacular view
Summer family travel
Travel for the arts:

Enjoy the spectacular view + the U.S.

B-QC

426
427
Sentence-BERT

!"!

429

Dialoguesentence

430

Core-Query

Sentence-BERT{!"" ⋯ !# }

428

431

Dialoguesentence

Core-Queries
Similarities
0.8
Enjoy the spectacular view
Summer family travel
0.6
Travel for the arts
0.3
:
:

432
433
434
435

Extract nouns
in dialogue

Filter&$%&%$%&Nouns Confidences
the U.S.
0.9
place
0.5
years
0.1
:
:

[MASK] + %$%&BERT

Sarch query!!"# + "!"#

Mask Prediction

436
437

Fig. 3. Process flow of CoCoA.

438
439
440
441
442
443
444

3.2

Select the Core-Query

Let the set of Core-Queries 𝑞 be the Core-Query list Q. The Core-Query Selector selects the Core-Query within thecontext of the current dialogue by calculating the sentence vector similarity between the current dialogue and the445

Core-Query. Specifically, the similarity 𝛾𝑞 is calculated from the sentence vectors, 𝑒 𝑐 and 𝑒 𝑞 , of the last three dialogues446

𝑐𝑡 −2:𝑡 and the Core-Query 𝑞 ∈ Q at time 𝑡,

447
448
449

𝑒 𝑐 = 𝑓𝑠 (𝑐𝑡 −2:𝑡 ),(1)

𝑒 𝑞 = 𝑓𝑠 (𝑞),(2)𝛾𝑞 = sim(𝑒 𝑐 , 𝑒 𝑞 ),

(3)

450
451
452
453
454

where 𝑓𝑠 (·) is a function to calculate the sentence vector and sim(·) is a function to calculate the cosine similarity.

455

We use Sentence-BERT [36] to calculate the sentence vector similarity. Then, for all similarity results Γ𝑄 = {𝛾𝑞 |𝑞 ∈ Q},

456
457

the maximum 𝑞 max is output as the Core-Query suitable for the current context.

458
459
460

𝐹 QS (ΓQ ) =

461
462
463
464
465




 arg max 𝛾𝑞 ,if ∃𝑞 0.5 ≤ 𝛾𝑞



𝜑,


otherwise

𝑞 ∈Q

(4)

Here, if all 𝛾𝑞 are below the threshold value of 0.5, it is determined that there is no Core-Query suitable for thecurrent context, so no Core-Query is output and no final search query is output. This prevents the output of search

466

query at every turn, which would probably irritate the user, and makes it possible to output search query only in

467

contexts close to the Core-Query.

468

9This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

469
470

3.3

Select complementary word

471

B-QC takes as input the Core-Query 𝑞 max selected by the Core-Query Selector and all dialogue histories 𝑐 1:𝑡 up to time 𝑡

472

and then selects the word 𝑤 max from the dialogue history that matches the Core-Query. The final search query is output

473

by adding the word to the Core-Query. Since the Core-Queries are abstract sentences and lack context-sensitive words,

474

they need to be supplemented with a keyword contained in the dialogue history. We felt that the masked language

475
476

model, which predicts the words in masked areas on the basis of the context, would be effective. Therefore, B-QC uses

477

BERT’s mask word prediction to predict context-dependent keywords. Since BERT includes masked language models in478

its pre-training, it can accurately complete words in sentences where some words are missing. For the Core-Queries, we479

assume that the context-dependent word is missing, and designed the B-QC to estimate the missing word using BERT.

480

In B-QC, the complementary word is determined by the following procedure, where the Core-Query selected by the

481
482

Core-Query Selector is denoted as 𝑞 max and the part of BERT that predicts words is denoted as [MASK]. We assume

483

that “+” denotes the conjunction of letters and sentences (e.g., this+is+an+apple → this is an apple), and let the input

484

sentence 𝑆 to BERT be 𝑆 = [MASK] + 𝑞 max . Then, the words are predicted in [MASK] by

485
486
487

𝑃 = 𝑓B (𝑆),

488
489
490
491

where 𝑃 = {𝑝 𝑤 |𝑤 ∈ W} denotes the predicted probability for any word 𝑤. From among the nouns 𝑁 𝑤 (𝑐 1:𝑡 ) in the
dialogue history 𝑐 1:𝑡 , the word 𝑤 max with the maximum 𝑝 𝑤 is obtained as the complementary word.

492

𝑤 max = arg max 𝑝 𝑤

495
496

(6)

𝑤 ∈N𝑤 (𝑐 1:𝑡 )

493
494

(5)

Finally, the complementary word 𝑤 max is combined with the Core-Query 𝑞 max to output the final search query{𝑤 max, 𝑞 max }.

497
498
499
500
501
502
503
504
505
506
507

4

EVALUATION EXPERIMENTS

4.1

Overview of experiments

We conducted two experiments: a “context sensitivity test” and a “robustness test”.
In the context sensitivity test, the system generated search queries using core queries that we prepared in advance,
and the participants answered the questionnaire by seeing them inserted in the dialogue.
In the robustness test, the system generated search queries using core queries created by each participants, and theparticipants answered a questionnaire by seeing them inserted in the dialogue.
The purpose of each is described below.

508
509

Context sensitivity test Conduct experiments using the common Core-Queries predetermined by the partici-

510

pants, and evaluate whether the Core-Query selected by the system and the complementary word are appropriate

511

for the context.

512
513
514
515
516
517

Robustness test Evaluate the robustness of the system by having each participant create a query to see if thesystem can properly generate advertisements for all ad creators and all Core-Queries.
The quality of the advertising system by CoCoA depends on the performance of the part of the system that transforms
Core-Query. In this experiment, we evaluated the performance of text processing related to core queries in a cloud518

experiment. In particular, since the purpose of CoCoA is to make it easier for advertisers to prepare the Core-Query

519

itself, the evaluation of the contextualization of the Core-Query is important.

520

10

In addition, the type of search results

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

Table 1. Details of the dialogue dataset.

521
522

Conversation

Trip

Food

SportsOpen-domain

No. of data
No. of turns in dialogue
Average no. of words

5
32.4
492.6

5
43.2
622

5
33.2
475.2

5
107.6
1789.8

523
524
525
526
527
528
529
530
531
532
533

that can be obtained from the Core-Query itself depends on the results of the Internet search, and since the uncertaintyof the results is expected to be large, only the text was evaluated in the cloud experiment.

534
535

4.2

Datasets

536
537

We used the Osaka University Multimodal Dialogue Corpus (Hazumi) provided by Osaka University via the IDR Dataset

538

Service of the National Institute of Informatics [20] as the dialogue data to present advertisements in both experiments.

539
540
541

Hazumi is a multimodal corpus of dialogues between agents and people using Wizard-of-Oz. Each dialogue lasts about
15 minutes and contains data for 89 people. The dialogues of each experimental participant contain conversations on542

multiple themes. From these, we collected dialogues in three domains: “trip”, “food”, and “sports”. In the dialogue data,543

the agent starts a dialogue by specifying a theme, such as “Let’s talk about a trip!”, so this is what we used to determine544

the domain. This resulted in 67, 72, and 29 sets of dialogs for the “trip”, “food”, and “sports” domains, respectively. The

545
546

five dialogues with the longest number of turns were selected as “trip conversation”, “food conversation”, and “sports547

conversation”, respectively. Next, from the dialogues of the experimental participants that were not selected for the548

above dialogues, we selected the five dialogues that contained multiple domains and had the longest number of dialogue549

turns, and designated these as “open-domain” dialogues. We obtained a total of 20 dialogue datasets containing five

550
551

dialogs each and four types. Finally, the agent’s utterance “Let’s talk about XX!” indicating a change in the dialogue552

theme was removed to prevent the system from easily guessing the context of the dialogue and to make the dialogue553

more natural. The details of the obtained dialogue data are shown in Table 1.

554
555
556
557

4.3

Experimental setup

4.3.1

Common setup for the two experiments.

558
559
560

Design: In the two crowdsourced experiments, we presented information in a web form for crowd workers to evaluate

561

online whether the search queries generated from Core-Queries were appropriate for the context of the conversation.

562

Specifically, Core-Queries, conversational text, and system-generated search queries were presented in text form, and563

the crowd workers were asked to rate the system-generated search queries using a web form questionnaire.

564
565
566

Experimental environment: Since the crowdsourcing experiment employed a text-based evaluation, we could not dis-567

play advertisements dynamically. Thus, the generated search query was evaluated by the participants as an “advertising568

sentence”. This evaluation method enables evaluation of the text processing that CoCoA transforms Core-Queries into

569
570
571
572

contextualized search queries. We assumed that web search would be used as a general-purpose database instead of thedatabase prepared by advertisers.
11

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

573

Since we recruited Japanese participants, CoCoA had to deal with Japanese texts. CoCoA can be used with574

any language for which a masked language model is available (English, Spanish, French, German, Chinese, Japan-

575
576

ese, etc.), but in this experiment, we used Japanese. In order to generate advertisements in Japanese, the follow-

577

ing processes were performed in B-QC. We define the post-positional particle in Japanese as 𝑗 ∈ 𝐽 , where 𝐽 =578{“𝑔𝑎”,“𝑤𝑜”,“𝑛𝑖”,“ℎ𝑒”,“𝑡𝑜”,“𝑦𝑜𝑟𝑖”,“𝑘𝑎𝑟𝑎”,“𝑑𝑒”,“𝑦𝑎”,“𝑛𝑜”}. The Core-Query selected by the Core-Query Selector is 𝑞, the579part of BERT that predicts the word is [MASK], and the input sentence 𝑆 to BERT is 𝑆 =[MASK]+𝑗 + 𝑞, after which

580
581

the word is predicted in [MASK]. The confidence of the predicted word is 𝑝, and the word with the largest 𝑝 among582

the nouns in the dialogue history is denoted as the candidate 𝑤 ( 𝑗) for the complementary word predicted using the583

particle 𝑗. The above operations are performed for all 𝑗. 𝑤 ( 𝑗) is added to the Core-Query as a complementary word in584

the order of frequency of occurrence among the set of 𝑤 ( 𝑗), and the result is output as the final advertisement.

585
586

Experimental conditions: Three experimental conditions were set up for the two crowdsourcing experiments. The

587

following three methods were used to generate search queries from Core-Queries, and we investigated the superiority

588
589

of the CoCoA condition through evaluation of the search queries.

590

1. CoCoA condition: Create a search query using the CoCoA method. Select a Core-Query that is appropriate for591

the dialogue, and attach a complementary word that is appropriate for both the context of the conversation and592

the Core-Query.

593

2. Core condition: Use only the Core-Query Selector. Select a Core-Query appropriate for the dialogue and use it

594
595

as the search query. No complementary word is attached.

596

3. Google condition: Use Google Suggest to select a complementary word. Select a Core-Query that is appropriate

597
598

for the dialogue, and attach a complementary word that is the most frequently searched word among the words

599

suggested by Google Suggest.

600
601

Since the Core condition uses a model that outputs only Core-Query as a search query without a complementary word,602

the output search query should be less context-dependent than the CoCoA condition. The Google condition uses Google

603

Suggest for the complementary word, so the output search query is more specific than the Core condition. But the604

complementary word is not related to the context of the dialogue, so it is expected to be less context-dependent than605

the CoCoA condition.

606
607

Participants: All participants in both experiments were recruited through a crowdsourcing website.

608
609

4.3.2

Context sensitivity test.

610
611
612
613
614
615

Design: Using the web form prepared for the crowdsourcing experiment, we prepared to investigate which method
is better by presenting search queries made under the CoCoA, Core, and Google conditions, along with dialog text.
We conducted a context sensitivity test in which the Core-Query was the same for all participants. We extracted
catchphrases consisting of two or more words related to “trip”, “food”, and “sports” from travel sites, restaurant search616

sites, and sports facility sites, respectively, and randomly selected ten catchphrases each for use as the Core-Queries in617

the “trip”, “food”, and “sports” domains. The Core-Queries used for each domain are listed in Table 2.

618
619
620
621

Experimental environment: To prevent bias in the evaluation of Core-Query by conversational domain, we prepared
two different corpus: one is a conversational corpus with the same domain as the domain of Core-Query. The other is622

an open-domain conversation corpus, independent of the Core-Query domain. The first one was the advertisements

623

generated in the “trip” Core-Query for “trip conversation”, the “food” Core-Query for “food conversation”,

624

12

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

625

and the “sports” Core-Query for “sports conversation”. The average number of advertising sentences generated626

by each dialogue was 24.8. The other was the advertising sentences generated in the “open-domain dialogue” in the

627
628

Core-Query for each domain. Participants were able to see the dialogue data and the advertising sentences created by629

each system that were inserted in the dialogue data. The order in which the systems were displayed was randomly

630

switched for each participant.

631
632

Participants: We used crowdsourcing to recruit 300 participants for the experiment: 128 men and 172 women aged

633

between 18 and 73 years with a mean age of 40.7 years. We divided the participants into six groups of 50 people

634
635

each. The first three groups were asked to respond to search queries where the topic of the Core-Query corresponded636

to the topic of the conversation. The other three groups were asked to respond to search queries for open-domain

637

conversational text.

638
639
640

Procedure: In the evaluation, the following instructions were given. The variable {DOMAIN} contains the values“trip”, “food”, and “sports” according to the above domains.

641
642

An advertiser decided to run an ad that would encourage {DOMAIN} after COVID-19 converged.

643

So, she came up with catchphrases that would make users think “What kind of things are there?”,644

search for it, and then want to go to {DOMAIN}. Evaluate the advertising sentences created by the645robot based on the catchphrase.

646
647
648

Observation and data analysis: We adopted a within-subjects design between CoCoA, Core, and Google condition

649

because dynamically providing advertisements according to a conversation is a novel experience for users, which leads

650

to large individual differences in their impression and makes it difficult to draw a clear conclusion. The within-subjects

651

design enables participants to evaluate the results in a unified measure. The questions used in the evaluation are shown

652
653

in Table 3. The survey results from the experiments with conversational data from the same domain as Core-Query and654

the open-domain conversational text were mixed and the mixture of data eliminated the effects of differences in the655

conversational domain.

656
657

4.3.3

Robustness test.

658
659

Design: The second experiment was a robustness test in which participants created the Core-Queries. At the beginning660

of the web form, participants were asked to create ten Core-Queries. After input, the web form proceeded to the web

661
662

form screen prepared for the crowdsourcing experiment, where search queries created based on the Core-Queries

663

created by the participants were presented along with conversational text, using the CoCoA, Core-Query, and Google

664

conditions. Participants were then surveyed to determine which method they preferred.

665
666
667

Experimental environment: As in the Context sensitivity test, we prepared conversational corpus of the same domain
as the Core-Query for one group and the open-domain dialogue for the other.

668
669
670
671
672
673

Participants: We recruited 300 participants through crowdsourcing: 136 men and 164 women aged 15 to 73 years
with a mean age of 39.1 years.
Procedure: The participants were divided into three groups of 100 people each, and each group was asked to create674

ten Core-Queries for each of three domains (“trip”, “food”, and “sports”). In the evaluation, the following instructions

675

were given. The variable {DOMAIN} contains the values “trip”, “food”, and “sports” according to the above domains.

676

13

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

677

Table 2. Core-Queries used in context sensitivity test (originally in Japanese; translated into English for inclusion in this paper).

678

Trip

Food

SportsA must-visit tourist destination
A trip to experience something extraordinary
Relaxing trip to an open-air bath
Enjoy the spectacular view
Experience nature
Summer family travel
First cruise trip
Tour to enjoy local food
Women’s travel by bullet train
A trip to enjoy the arts

A place where we can drink craft beer
A place for entertaining
Cheap BBQ restaurant
A place to eat sweets
Famous Italian restaurants
Bars with all-you-can-drink
Recommended for lunch
Enjoy local cuisine
Popular sports bar
All-you-can-drink course for more than 3 hours

Health maintenance
Increase basal metabolism and lose weight
Beginners are welcome
Helps to combat lack of exercise
Continue to have fun
Regain your strength
Fitness in your spare time
Become an athlete for the weekend
Comfortable tiredness
Enjoy with friends

679
680
681
682
683
684
685
686
687
688
689
690
691

You work for an advertising agency, and you’ve been assigned to run an ad that encourages people692

to {DOMAIN} after COVID-19 converged. So, you need to come up with catchphrases that would693

make users think “What kind of things are there?”, search for it, and then want to go to {DOMAIN}.

694
695
696
697

In addition, we instructed them to pay attention to the following two points when creating Core-Queries, and allowed
them to use a web search if necessary.

698• The catchphrase should support various genres of {DOMAIN} in various regions.

699• The ten catchphrases should not be too similar to each other.

700
701

Each group was divided into 50 participants, and each participant evaluated the advertising sentences generated by702

each system using the same procedure as the context sensitivity test, using the same domain as the Core-Query for703

three groups and the open-domain dialogue for the other three groups. The questions used in the evaluation are shown

704
705
706
707

in Table 4.
Observation and data analysis: The questions used in the evaluation are shown in Table 4. Here, as in the Context

708

sensitivity test, the questionnaire results from the experiment with conversational data from the same conversational709

domain as Core-Query and the experiment with open-domain conversational texts were mixed.

710
711
712

4.4

Experimental results713

4.4.1 Context sensitivity test. The evaluation of each system by the experiment participants is shown in Fig. 4. We

714

used ANOVA to compare each system and found that the main effect of the system was significant for all questions.

715

Q1: [𝐹 (2, 897) = 33.3, 𝑝 < .001, 𝜂 2 = 0.069], Q2: [𝐹 (2, 897) = 22.2, 𝑝 < .001, 𝜂 2 = 0.047], Q3: [𝐹 (2, 897) = 15.6, 𝑝 <716.001, 𝜂 2 = 0.034], Q4: [𝐹 (2, 897) = 30.4, 𝑝 < .001, 𝜂 2 = 0.063], Q5: [𝐹 (1, 598) = 239, 𝑝 < .001, 𝜂 2 = 0.285], Q6:

717
718

[𝐹 (2, 897) = 17.6, 𝑝 < .001, 𝜂 2 = 0.038], Q7: [𝐹 (2, 897) = 18.4, 𝑝 < .001, 𝜂 2 = 0.039]. Q8, which was an open-ended

719

question, is described in the Discussion.

720
721

4.4.2 Robustness test. The evaluation of each system by the participants is shown in Fig. 5. We used ANOVA to722

compare each system and found that the main effect of the system was significant for all questions. Q1: [𝐹 (2, 897) =723

3.26, 𝑝 = 0.039, 𝜂 2 = 0.007], Q2: [𝐹 (2, 897) = 6.33, 𝑝 = 0.002, 𝜂 2 = 0.014], Q3: [𝐹 (2, 897) = 4.14, 𝑝 = 0.016, 𝜂 2 = 0.009],

724

Q4: [𝐹 (2, 897) = 4.33, 𝑝 = 0.013, 𝜂 2 = 0.010], Q5: [𝐹 (2, 897) = 8.88, 𝑝 < .001, 𝜂 2 = 0.019], Q6: [𝐹 (2, 897) = 3.09, 𝑝 =

725
726

0.046, 𝜂 2 = 0.007]. The average creation time of the Core-Queries was 569 seconds, and we obtained 2955 different

727

queries after duplicates were removed. The average number of advertisements output in each dialogue was 12.8.

728

14

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

Table 3. Questions for the context sensitivity test. A and B are the two characters in the dialogue.

729
730

Metric731
732

Questionnaire items (Q1–Q7 were responded with seven-point scales.)Understanding

Q1. Did the robot have a clear understanding of the conversation?

734

EffectivenessQ2. Is it likely that the advertisement will have an effect on A and B?

735

Appropriate timing

Q3. Was the robot able to present the catch phrases createdby the advertiser at the appropriate time?

Ad context awareness

Q4. Were the advertising sentences created by the robot based on the catch phrases
created by the advertiser appropriate for the context?

Word context awarenessQ5. Are the words added by the robot to the catch phrases createdby the advertiser appropriate for the context?

Expected timingQ6. Did the robot display the advertising sentences at the expected time?

Unexpectedly good timing

Q7. Were there any advertising sentences that you did not anticipate
when you saw the catchphrase, but which were displayed at the appropriate time?

Best ad

Q8. Which advertising sentence did you think was the best? (free description)

733

736
737
738
739
740
741
742
743
744
745
746
747
748

Table 4. Questions for the robustness test. A and B are the two characters in the dialogue.

749
750
751

MetricQuestionnaire items (each is rated on a 7-point Likert scale)752

Understanding

Q1. Did the robot have a clear understanding of the conversation?

754

EffectivenessQ2. Is it likely that the advertisement will have an effect on A and B?

755

Appropriate timing

Q3. Was the robot able to display the appropriate advertising sentence at the appropriate time?

Expected timing

Q4. Did the robot display the catchphrase at the timing you had in mind when you created it?

758

Unexpectedly ad

Q5. Did you receive an advertising sentence that you did not anticipate when creating your catchphrase?

759

Good timingQ6. Did you see any advertising sentences that were displayed at the appropriate time?

753

756
757

760
761
762
763

4.5

Discussion764

4.5.1 Context sensitivity test. First, we compare the proposed method with the other two methods. In all questions,765

the proposed method outperformed the others. This demonstrates that the proposed method generates appropriate

766
767

advertising sentences in accordance with the context of the dialogues, which makes the advertisement more effective.

768

Next, we discuss the Google Suggest’s overall lower ratings for Q1–Q7 compared to the Core-Query Selector-only769

model, even though Google Suggest was used to complete the words. A Core-Query is an abstract advertising sentence

770

that forms the core of what the advertiser wants to recommend. Because of its abstraction, the minimum requirements

771
772
773

for an advertisement can be satisfied as long as the Core-Query that matches the context is selected by the Core-Query
Selector. Google Suggest, on the other hand, makes queries concrete by performing completion on the Core-Queries. In

774

this case, Google Suggest is not able to refer to the dialogue history, so if the complementary word is inappropriate, it

775

will result in a specific advertisement that does not match the context, and the evaluation will be low.

776
777

Finally, we discuss Q8, where the respondents were asked to select the best advertising sentence. Only those selected778

as the best created by the proposed method are discussed. First, for “trip”, there are two Core-Queries included in779

the advertising sentences that are considered to be significantly better only in the proposed system: “A must-visit

780

15

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

788

5

4
3

6

4
3

6

5
4
3

7
p < .001
p < .001
p < .001

5
4
3

789

2

790
791

1

792

2

Core

CoCoA Google

Model

1

2

Core

CoCoA Google

1

Model

CoCoA Google

1

Model

p < .001

6

2

Core

7
6

5
4
3
2

Core

CoCoA Google

Model

1

7
p = 0.373
p < .001
p < .001

5
4
3
2

CoCoA

1

GoogleModelp = 0.154
p < .001
p < .001

6

Unexpectedly good timing

787

5

7
p = 0.531
p < .001
p < .001

Expected timing

786

6

Effectiveness

785

Understanding784

7
p = 0.086
p < .001
p < .001

Word context awareness

6

783

7
p = 0.007
p < .001
p < .001

Appropriate timing782

Ad context awareness

7

781

5
4
3
2

Core

CoCoA Google

Model

1

Core

CoCoA Google

Model

793
794
795
796

Fig. 4. Analysis results for context sensitivity test. The system with only Core-Query Selector is called “Core”, the proposed system iscalled “CoCoA”, and the system with Google Suggest is called “Google”. As for Q5, the data are only for “CoCoA” and “Google”, where
the word is completed.

797

803
804
805
806
807

5

5

4
3
2

6

4
3
2

7
p = 0.999
p = 0.037
p = 0.032

5

6

4
3
2

7
p = 0.759
p = 0.086
p = 0.013

5

6

4
3
2

7
p = 0.616
p = 0.006
p < .001

65

p = 0.961
p = 0.111
p = 0.060

5Good timing

802

Understanding

801

6

7
p = 0.747
p = 0.002
p = 0.022

Unexpectedly ad

6Effectiveness800

7
p = 0.880
p = 0.131
p = 0.042

Expected timing

7799

Appropriate timing

798

4

4

3

3

2

2

808
809
810

1

Core

CoCoA

Model

Google

1

Core

CoCoA

Model

Google

1

Core

CoCoA

Model

1

Google

Core

CoCoA

Model

Google

1

Core

CoCoA

Model

Google

1

Core

CoCoA

Model

Google811
812

Fig. 5. Analysis results for robustness test.

813
814
815

tourist destination” and “Enjoy the spectacular view”. For these, the advertising sentences with complementary words

816

that specify the location received good evaluations. Specifically, participants mentioned the advertising sentences “A

817

must-visit tourist destination Barcelona” in the context of a dialogue about Barcelona, “A must-visit tourist destination

818
819

foreign country” in the context of a person who has not been abroad recently, and “Enjoy the spectacular view a hot

820

spring” in the context of a dialogue about a person who wants to go to a hot spring. Next, for “food”, there are two

821

Core-Queries included in the advertising sentences that are considered to be significantly better only in the proposed

822
823
824

system: “Enjoy local cuisine” and “Popular sports bar”. In the former case, the advertising sentence “Enjoy local cuisine
Hokuriku” (Hokuriku is a place name in Japan), which was complemented with place names as in the case of “trip”, was825highly evaluated. In the latter case, sports-related words were complemented instead of place names, and advertising826

sentences such as “Popular sports bar Olympics” when the topic was the Olympics or “Popular sports bar snowboarders”

827

in the context of snowboarding as a hobby were highly evaluated. Finally, for “sports”, the Core-Query included in the

828
829

advertising sentences that was significantly better only in the proposed system was “Fitness in your spare time”. This830

was highly appreciated when the word “housework” was complemented, which is interesting because, unlike with831

“trip” and “food”, it is not a place name or a named entity. Specifically, in the context of the fact that housework is hard

832

16

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

833
834

Table 5. Core-Queries with the highest number of occurrences in each domain (originally in Japanese; translated into English forinclusion in this paper.)

835
836

Trip

Food

837

Did you know that traveling is fun?
Popular hot spring resorts
Enjoy famous sightseeing spots
Hot spring lovers!
A tourist spot full of attractions
A chance to visit popular tourist spots
Rediscovering the charms of Japan
Travel to enjoy the hot springs
Treat yourself to a trip of luxury
This is the best time of year to visit

Drink, eat, and be happy!
Daily lunch specials are recommended
Delicious snacks to go with drinks
A place with good drinks
A place to eat and relax in peace
Cafe with great sweets
A Japanese restaurant so delicious you’ll want to keep coming backCafes with lots of delicious sweets
Women’s favorite sweets shop
A cheap and tasty restaurant

838
839
840
841
842
843
844
845
846
847
848

Sports849

Recommended sports for hobbies
If you want to eat what you love, play sports! Keep moving your body and eating what you like!
Continue to enjoy sports more easily and happily
If you watch the competition and want to do the same
It can be fun
Fun sports
Let’s experience it! Olympic Games
Those who were inspired by the Olympics and want to play sports
If you want to lose weight, you need to do it in a fun and sweaty way.
It could be an Olympic sport

850
851
852
853
854
855
856
857
858
859
860
861

Table 6. Complementary words and number of times completed per domain for the Core-Queries (originally in Japanese; translated
into English for inclusion in this paper.)

862

Trip

863

Food

Sports864

Word

Number of completions

Words

Number of completions

Words

Number of completions

865

Japan
trip
world
foreign countries
convenience store
Hokkaido
Hokurikusea
sake
hot spring

173
95
56
55
28
26
24
23
23
22

Japan
cake
like
chocolateconfectionerysakeconvenience store
department store
food
coffee

126
118
101
100
94
91
73
70
69
67

sportshealthJapan
Olympics
TVtennisfriends
movie
food
baseball

215
130
75
51
47
24
24
21
19
19

866
867
868
869
870
871
872
873
874
875

because there are many things to do, the advertising sentence “Fitness in your spare time: housework” was generated876

to promote physical fitness by making efforts during housework.

877
878

4.5.2 Robustness test. First, we compare the proposed method with the other two methods. In all questions, the879

proposed method outperformed the others. Although the Core-Query in the robustness test was different for each

880
881

participant, unlike in the context sensitivity test, we found that the system was effective regardless of the variation of882

the Core-Query. At the same time, we found that the evaluation from the Core-Query creator side, i.e., the advertiser

883

side, was also high. In addition, although the Core-Queries alone can only present advertisements within the expected

884

17

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

885

range, the responses to Q5 demonstrate that word completion using B-QC produced advertising sentences that were886

not expected by the advertisers.

887

Next, we discuss the obtained Core-Queries. Table 5 shows the Core-Queries with the highest number of occurrences

888
889

in each domain. The total number of Core-Queries obtained was 2955, of which 1133 were displayed at least once as890

advertising sentences. The average number of characters in the Core-Queries that were displayed at least once was891

12.1, and the average number of characters in the Core-Queries that were never displayed was 10.7. The results of

892

ANOVA testing showed that there is a significant difference [𝐹 (1, 2953) = 75.9, 𝑝 < .001, 𝜂 2 = 0.025], which means that

893
894

even in this limited dataset, the Core-Queries with a large number of words have a large number of occurrences. The

895

reason may be that the sentence-vector could not be calculated accurately for the Core-Queries with a small number of896

characters, i.e., short sentences, because Sentence-Bert was used to select the Core-Queries.

897

Finally, we consider complementary words. Table 6 shows the complementary words with the highest number of

898
899

occurrences in each domain in the advertising sentence created by the proposed system. In the trip domain, words

900

describing destinations such as “Japan”, “world”, “overseas”, “Hokkaido”, “Hokuriku”, “sea”, “hot spring”, etc. were901

the most complemented. In the food domain, words indicating specific foods or drinks, such as “cake”, “chocolate”,

902
903

“confectionery”, “sake”, and “coffee”, were frequently supplemented. In the sports domain, a wide range of complementary904

words related to sports were complemented, such as “tennis”, “baseball”, “Olympics”, “health”, “friends”, and so on.

905

These findings, along with the evaluation results of the experiment, demonstrate that the system was able to complete906

the words appropriately.

907
908
909
910

5

CASE STUDIES

5.1

Purpose of case study

911
912
913
914
915

We conducted two case studies to examine how participants use CoCoA in an actual conversation by providing apreliminary CoCoA system to advertisers and users. These studies provide insights into the effects of CoCoA thatcannot be obtained from the crowdsourced experiment alone. We have the following objectives in this case study.

916

1. How often do users refer to the advertisements CoCoA generates while conversing?

917

2. Do the advertisements from CoCoA have any effect on the topics of the users’ conversation?

918
919

3. How did advertisers evaluate the generated search queries?

920

4. Can the advertisers create the Core-Queries easily?

921
922

The first is a “user case study” and the second is an “advertiser case study”. The user case study investigates objectives923

1 and 2 by having two participants acting as users conduct a conversation in front of the experimental system displaying

924
925
926

generated advertisements during the conversation in real-time. The advertiser case study mainly investigates objectives
3 and 4. It consists of three participants acting as an advertiser and two users. The advertiser creates original Core-

927

Queries and evaluates how the experimental system selects and modifies them to generate the search queries during

928

conversations between the two users. As this case study includes participants playing user roles, we also investigate

929
930

objectives 1 and 2 here.

931
932
933
934
935
936

5.2

Setup of case study

This subsection describes the policy, system and environment, experimental participants, procedures, observation
methods, and data analysis utilized for the user case study (5.2.1) and advertiser case study (5.2.2).
18

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

937
938

那須フラワーワールド

939
940

【営業時間】9:30〜17:00

941【定休日】月曜日【アクセス】〜〜〜942
943
HPはこちら

944
945
946
947
948
949
950
951
952
953
954
955
956
957

Web page titles958

Web page contents

959
960

Fig. 6. System screen used in experiment.

961
962
963
964
965
966
967

5.2.1

User case study.

The policy of the user case study design: We prepared an interaction environment in which participants acting as
users have a casual conversation in front of CoCoA, which allows us to observe how the system is utilized during

968

conversation. The system, interaction environment, and experimental procedures were designed to satisfy objectives 1

969

and 2 discussed in the previous subsection.

970
971
972
973
974

System and environment for the user case study: We conducted the experiments for this case study in our laboratory.
A screenshot of the system is shown in Fig. 6. The proposed system was implemented using the Core-Queries for“Trip” that had been derived from the context sensitivity test (Sec. 4). In these experiments, we used web search975

as a general-purpose database. Specifically, search queries were generated along with the dialogue input, and after

976

performing a Google search using the search query, the titles and thumbnails of the top five pages of the search results

977
978

were displayed as a set of buttons on the left side of the screen. Participants could touch one of the buttons to display979

the page contents on the right and then freely scroll and explore. A microphone placed on the desk was used to collect980

the dialogue audio. Figure 7 shows a photograph of the actual experimental environment, where participants were981

seated in pairs in front of a CoCoA-equipped display.

982
983
984
985

Participants for the user case study: A total of six participants (two men and four women, average age 23.0 years)
took part in this experiment. They were divided into three pairs (Pairs A, B, and C), and each participant in the pair had

986

an individual number (A-1 and A-2, B-1 and B-2, and C-1 and C-2) that we utilized when creating transcripts of their987

interaction.

988

19

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009

Fig. 7. Experimental environment.

1010
1011
1012
1013

Procedure for the user case study: We asked participants to sit in pairs in front of the CoCoA-equipped display, gave

1014

instructions on how to use the system, and then encouraged them to chat for ten minutes. We did not provide any topics

1015

for the conversation. We told the participants they could use the system if they needed to, but we did not force them to

1016
1017
1018
1019
1020

do so. During the conversation, participants were free to view the advertisements on the display, which were updated
in real time. After the experiment, we interviewed the participants to determine how they felt about using CoCoA.
Observation and data analysis for the user case study: We recorded the interaction between participants on a video to1021

count how often they referred to advertisements appearing on the display and to observe any effect the advertisements

1022

from CoCoA had on the topics of conversation. We also created transcripts of the conversations from the video to grasp

1023
1024
1025
1026
1027
1028
1029
1030

how participants behaved with reference to the advertisements and how they changed the topic of conversation in
response to the generated advertisements.
5.2.2

Advertiser case study.

The policy of the advertiser case study design: We prepared the same system as the user case study, where twoparticipants acting as users have a casual conversation. There was also an additional participant acting as an advertiser,

1031

who we asked to provide original Core-Queries and to evaluate the effectiveness of the search queries generated from1032

them. This design allows us to observe how the advertiser creates Core-Queries as well as how the users utilize CoCoA

1033

during the conversation. We designed the system, interaction environment, and experimental procedures to mainly

1034
1035
1036

investigate objectives 3 and 4, as discussed in subsection 5.1. We also acquired some knowledge related to objectives 1
and 2.

1037
1038

System and environment for the advertiser case study: The system for the participants playing user roles was the1039

same as that in the user case study. They had a conversation in the same environmental configuration shown in Fig. 7.

1040

20

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1041

As for the participants playing advertisers, we prepared a web-based form for them to input the Core-Queries they1042

created. This form was separated from CoCoA so that we could conduct the session of preparing the Core-Queries

1043
1044
1045

independently from the session of users’ interaction. Before starting the users’ interaction session, we input the created
Core-Queries into CoCoA.

1046
1047
1048
1049

Experimental conditions for the advertiser case study: We set two experimental conditions: CoCoA and Core. The

1050

CoCoA condition generated search queries using the CoCoA model, which selected a Core-Query appropriate to the1051

current users’ conversation and found a complementary word to make the search query more suited to the context. The

1052
1053

Core condition determined a Core-Query using the Core-Query Selector only and did not add any complementary1054

word when outputting a search query. The displayed advertisements included the search queries created under both1055

conditions at the same time. Since both search queries were based on the same Core-Query, the experimental system

1056

might show similar advertisements, which could confuse the users, so we attempted to mitigate this by having the

1057
1058
1059

system randomly shuffle the advertisement list. Ultimately, the users had conversations while watching advertisements
generated by two conditions, and the advertisers could compare the effects of both.

1060
1061
1062
1063

Participants for the advertiser case study: A total of nine participants split into three groups took part in this experiment.

1064

In each group, one person played the role of advertiser and the other two played the role of users. The advertisers were

1065
1066

participants from the user case study (one from each pair) and included one man and two women with the average age

1067

of 23.7 years. The users were six men with the average age of 22.5 years. The three groups were named D, E, and F,

1068

where D-1, E-1, and F-1 were the advertisers and D-2 and 3, E-2 and 3, and F-2 and 3 were the users.

1069
1070
1071
1072

Procedure for the advertiser case study: The advertisers chose their favorite domain from among “Trip”, “Food”, and1073“Sports” and then created ten Core-Queries using the same procedure as the Robustness test in Section 4. We sent the

1074
1075

Core-Queries to CoCoA. Advertiser D-1 selected “Trip”, E-1 selected “Food”, and F-1 selected “Trip”. We asked the users

1076

to converse in front of the CoCoA-equipped display in the same manner as the user case study. Also, the users in each1077

group were not informed of the presence of the advertiser role. After the experiment, we interviewed the users to

1078
1079
1080
1081

determine how they felt about using CoCoA. Advertisers were also interviewed after the end of each experiment. We
asked the advertisers to evaluate the generated search queries after we had finished preparing the recorded video andtranscripts of users’ interactions (i.e., on another day).

1082
1083
1084
1085

Observation and data analysis for the advertiser case study: We recorded user interactions on a video that the advertisers1086

could watch to assess their satisfaction with the search queries (objective 3). Advertisers were requested to respond

1087
1088

to a questionnaire while referring to the user interaction video, dialogue transcript, generated search queries, and1089

visited websites. The generated search queries and visited websites were aligned with the transcribed dialogue so that1090

advertisers could follow the timing of each.

1091
1092

21

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

Table 7. Items evaluated by advertisers.

1093
1094
1095

MetricQuestionnaire items (Q1–Q2 were answered using 7-point Likert scale).

Context awarenessQ1. Did the robot choose the appropriate catchphrase from among the prepared candidates,

EffectivenessQ2. How effective was the advertisement?

1101

Clicked website

Q3. Why do you think users clicked on the web site at this time? (free description)

1102

Good catchphrase

Q4. Which of the catchphrases you created was the best, and why? (free description)

1103

Bad catchphrase

Q5. Which of the catchphrases you created was the worst, and why? (free description)

Other Core-Query

Q6. Based on the results, what catchphrases other than the ones you prepared

1096
1097
1098
1099
1100

1104
1105

and/or transform the catchphrase into a more contextualized form?

might would have generated more effective advertisements? (multiple answers allowed)

1106
1107

Best ad

Q7. Which was the best search query the system generated? (free description)

1108

Impression

Q8. Please provide your overall impressions. (free description)

1109
1110
1111
1112

The items evaluated by advertisers are listed in Table 7. Q1 and Q2 were assessed in terms of each search query1113

generated by the CoCoA condition and the Core condition, respectively, in the dialog. Q3 was answered only if the1114

users visited the website. We utilized descriptive statistics to grasp the overall tendency of the responses from the1115

relatively small number of participants: specifically, the median, maximum, and minimum values, the interquartile

1116
1117
1118

range, and the smoothed probability density of the data of the answers. The advertisers also responded to items Q4 to
Q8 in Table 7. We interpreted the free descriptions and multiple answers to these questions to determine whether the1119

advertisers were able to easily create the Core-Queries (objective 4). To cover objectives 1 and 2, we also counted how

1120

often users referred to advertisements in the advertisement case study and observed the effects of the search queries on

1121
1122

the topics of the users’ conversations.

1123
1124
1125
1126

5.3

Results

5.3.1 Overview of results. Subsection 5.3 reports the case study results to clarify the effectiveness of CoCoA in terms of1127

the four objectives listed in subsection 5.1. We discuss two main results: the interaction that occurs between the pair of1128

users and CoCoA, and the quality of the Core-Queries.

1129
1130
1131

One notable result regarding the interaction between users is that CoCoA could make advertisements influence theuser’s conversation by presenting ads that match the context of the dialogue. In particular, the less familiar users were1132

with each other, the more frequently they used CoCoA to search for information (see 5.3.2). The other results suggest

1133

that the advertisements CoCoA continued to display influenced the user’s topic of conversation, even if the user did not1134

tap on the ad (see 5.3.3).

1135
1136

The results regarding the quality of Core-Queries yields two findings. The first is that the search queries generated

1137

by CoCoA reflected the dialogue context to a remarkable extent. The other is that advertisers could create Core-Queries

1138

quite easily. The advertiser case study confirmed that the search queries generated by CoCoA were highly context-aware

1139

and accurately reflected the users’ dialogues (see 5.3.4). Moreover, the proposed way of providing Core-Queries was easy

1140
1141

for advertisers to learn and did not require them to have any specific knowledge about CoCoA (see 5.3.5). Advertisers

1142

could learn how to create more effective Core-Queries after just one trial of creating initial Core-Queries for CoCoA

1143

and observing how users behaved in response to the CoCoA-generated advertisements.

1144

22

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1145

5.3.2 How often do users refer to advertisements generated by CoCoA while conversing? The number of times users

1146

tapped on the system screen during a ten-minute interaction was broken down as follows: case with the most taps (Pair

1147
1148
1149
1150

A: nine times), cases with few taps (Pair C: one time, Group D: one time, and Group F: two times), and cases with no taps
(Pair B: zero times (three times when the experiment was extended for another ten minutes) and Group E: zero times).

1151

Case with the most taps: Pair A consisted of two casual acquaintances who had never chatted before. The fact that

1152

they tapped on the advertisements nine times in ten minutes indicates that they heavily relied on the system to keep1153

the conversation going. We obtained essentially the same insight when we interviewed them: after the experiment,

1154

they stated that they had not been able to think of what to say, so they used the system to find topics for conversation.

1155
1156
1157
1158
1159

Cases with few taps: Pair C consisted of two individuals who were meeting for the first time. They began theconversation with self-introductions, and after four minutes and fifty seconds, C-1 mentioned that she had studied in
France. The system displayed a search result related to France and the remainder of the dialogue continued as they1160

scrolled through the search results. Although the number of taps was just one, they used the system the entire time.

1161

D-2 and D-3 were friendly acquaintances who had conversations daily. Transcript 1 indicates that they tapped on the

1162
1163
1164

advertisement generated in response to a conversation about a TV travel program. The generated search query was “No
Worries about Rain! Enjoy your holiday with indoor activities on a one-day tour” “definitely (a complementary word)”.

1165
1166

Transcript 1 from Group D

1167

D-2

The system is all about rainy day outings.

1168

D-3

Maybe it’s because it’s raining right now. You want to see something about a rainy day out?

1169

tapped query“No Worries about Rain! Enjoy your holiday with indoor activities on a one-day tour” “definitely”

1170

Web site“https://www.jalan.net/news/article/231633/”

Q1: 3

Q2: 3

1171
1172
1173

“Q1: 3 and Q2: 3” on the right side of the tapped query in Transcript 1 indicate the advertiser’s answers to questionnaire1174

items Q1 and Q2 in Table 7. The value of 3 given to both indicates that advertiser D-1 felt the tapped search query was

1175

inappropriate to the conversation context and did not have enough of an effect for the advertisements. This opinion

1176
1177
1178

was confirmed in the interview when we asked D-1 to explain the reason for the tapping. D-1 answered that the users
were surprised to see a website with content different from the context of their conversation at that time.

1179

Users in Group F were friendly acquaintances who had conversations daily. In the case study, when they were1180

talking about lack of exercise, they tried to tap on the search query consisting of “mountain climbing trip for exercise”

1181“hiking”. However, the system updated the query at the moment of the tap, and they accidentally tapped the query of

1182
1183
1184
1185
1186

“Round-the-Prefecture Bus Tour” “sport” by mistake.
Transcript 2 from Group F
F-2

I’m sure I’ll get good at it if I do it a lot,but I don’t have the opportunity to do it a lot. It’s not something I go to every week.

1187
1188

F-3

I know what you mean. People who score over 150 points are amazing.

1189

F-2

The system was showing something like “hike for exercise”, but it’s changed.

1190

tapped query“Round-the-Prefecture Bus Tour” “sport”

website“https://www.hatobus.co.jp/”

1191

Q1: 1

Q2: 1

1192
1193
1194
1195
1196

After that, they talked about cooking and tapped the search query “Summer swimming tours to beat the heat”“processing”.
23This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1197
1198

Transcript 3 from Gropu F
F-3

1199
1200
1201
1202
1203
1204

Washing is the most annoying part.
Every time I worry about processing the grease that remains after cooking meat, I put it off.

F-2

You have a lot of dishes to wash.

F-3

By the way, I make curry a lot.

tapped query“Summer swimming tours to beat the heat” “processing”

web site“https://www.city.fukui.lg.jp/kyoiku/kyusyoku/ncenter/natuyasaikare-.html”

Q1: 1

Q2: 1

1205
1206
1207

In this scene, the tapped query was not related to the conversation topic of cooking curry. However, the search

1208

resulted in the web page “Blow off the heat! Nutritious [Summer Vegetable Curry]”, and since the webpage matched the

1209
1210
1211
1212

curry conversation by chance, users F-2 and 3 tapped on it.
The low values of Q1 and Q2 indicate that advertiser F-1 did not recognize the appropriateness of the generated
search query. We explain the reason in subsection 5.3.4.

1213
1214
1215
1216

Cases never used: The individuals in Pair B were acquainted with each other, had a chatting relationship, and did not
use the advertising system at all during the first ten minutes. The reasons given were “I thought it would be rude to the1217

other person because I had to move my eyes a lot to look at the screen while the dialogue was going on.” There were1218

also comments such as “It would be easier to use a screen the size of a computer on a desk” and “I was curious but

1219
1220
1221
1222

didn’t want to look away from my partner.” After that, Pair B wanted to use the system a little more, so the experiment
was extended another ten minutes and observation was continued. They checked the advertisements three times during
the latter ten minutes of the interaction.

1223

Users E-2 and E-3 were acquainted with each other and had a chatting relationship. The topic of conversation

1224

began with E-3’s hobby of houseplants and ended with E-2’s hobby of golf. They never tapped on the advertisement

1225
1226

throughout their interaction. User E-2 mentioned in the interview that “I knew that when we were talking about health,1227

there were web pages about health-conscious eating, but the rest of the time we weren’t talking about restaurants, and1228

I thought the topic of the conversation was different.”

1229
1230
1231
1232

The system used for Groups D, E, and F generated two search queries based on the conditions of CoCoA (our method)
and Core. Although the number of taps was small, all tapped queries came from the CoCoA condition.

1233

5.3.3 Do the advertisements from CoCoA affect the topics of the users’ conversation? The case studies featured some1234

scenes where the topic of the users’ conversation changed in accordance with the displayed advertisements. We discuss

1235

here all groups except Group E, as the users in this group did not look at the display during their conversation.

1236
1237
1238
1239
1240

Pair A. The system displayed a search query about the hometown of one of the participants, and from there, theusers started a dialogue about tourist attractions in that town. Since they did not have a topic for the conversation at
first, this advertisement was able to initiate their interaction.

1241
1242

Pair B. Users did not use the advertising system at all during the first ten minutes. When the dialogue about going to1243

Disneyland stopped, B-2 looked at the travel advertisement on the left half of the screen and then asked, “If you were

1244
1245

going on a trip, where would you go?” In other words, they changed the conversation topic to a trip in response to the1246

advertisement. In the latter ten minutes, the conversation proceeded in accordance with the contents of the displayed1247

web page, and even when the conversation turned into a chat that had nothing to do with travel, an advertisement for a

1248

24

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1249

cruise trip as the Core-Query was displayed. The context was then changed to a discussion about the price of going on

1250

a cruise.

1251

Pair C. When one of the participants mentioned he had studied in France, the other asked about famous places

1252
1253
1254

there. This dialogue led to the generation of the search query “A must-visit tourist destination” “France” and a button

1255

about the beautiful scenery of France was displayed, which the user tapped. This would not have been possible if the

1256

system did not refer to the context of the dialogue. The button was only tapped once, but after that, the dialogue was

1257

inspired by the displayed web page until the end of the experiment. Specifically, the participant who had gone to France

1258

explained the details of the displayed tourist attractions, while the other participant asked several questions. In addition,

1259
1260

there were times when tourist attractions that were not known to either of them were displayed, which eventually led

1261

to the utterance “I want to go to France.”

1262
1263

Group D. D-2 began by talking about his hobby of strength training, and from there, the topic shifted to exercise and1264

then to travel with CoCoA as the trigger. D-2 commented on the topic shift that “I was talking about muscle training,1265

but only ads about travel were shown. I found it hard to connect with the conversation.” D-3 said in the interview, “The

1266
1267

system gave a good opportunity to change the topic and initiate a conversation.”

1268
1269
1270

Group F. User F-2 had a conversation about not getting enough exercise. However, F-2 tried to tap on the search query“Round-the-Prefecture Bus Tour” “sport”, which was different from the topic of the ongoing conversation. Advertiser

1271

F-1 gave the following reasons for the tap: “The user had been feeling a lack of exercise recently, and wanted to see an

1272

advertisement for a trip that would help him be more active. As a result, the ad changed to a different ad, so it is not1273that they tapped on the ad they actually wanted to see, but I think their interest in the ad was strengthened.” In this

1274
1275

situation, the user was interested in a search query about hiking and tried to tap on it, but the moment he did so, it

1276

changed to another search query, and as a result, he tapped on a search query that was not the one he wanted. However,1277

at the time of the tap, the user’s interest in the ad was piqued and the conversation continued on the basis of the web1278

site he opened. They began talking about tourist attractions in Tokyo that were listed on the web site.

1279
1280
1281

7

7

7

7

7

7

6

6

6

6

6

6

5

5

5

5

5

5

4

4

4

score

4

score

1286

score

1285

score

1284

score

1283

score

1282

4

4

1287

3

3

3

3

3

3

1288

2

2

2

2

2

2

1289
1290
1291

1

1
CoCoA

Core

D-1's answer for Q1

1
CoCoA

Core

D-1's answer for Q2

1
CoCoA

Core

1
CoCoA

E-1's answer for Q1

Core

E-1's answer for Q2

1
CoCoA

Core

F-1's answer for Q1

CoCoA

Core

F-1's answer for Q2

1292
1293
1294

Fig. 8. The violin plots express the median, maximum and minimum values, interquartile range, and the smoothed probability densityof the data of the answers to Q1 and Q2 obtained from Advertisers D-1, E-1, F-1.

1295
1296
1297

5.3.4 How did advertisers evaluate the generated search queries? The violin plots in Fig. 8 depict the descriptive statistics

1298

of each advertiser’s answers to Q1 and Q2 in Table 7. This formatting is intended to clarify the overall tendency of how

1299

the advertisers perceived the effect of the generated search queries.

1300

25

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1301

We found from D-1’s and E-1’s answers to Q1 that the advertisers (except for F-1) felt the search queries generated

1302

by CoCoA were more suitable for the conversation context than the search queries without complementary words.

1303
1304

These results are compatible with those of the crowdsourced evaluations.

1305

In contrast, D-1’s and E-1’s answers to Q2 indicate a difference from the results of the crowdsourced evaluation (Q2:1306

Effectiveness). Advertisers in the case study (except for F-1) did not identify a difference in the effect of two queries on1307

the advertisement to the users, even if they recognized the impact of the complementary word to make the search query

1308
1309

contextual. On the other hand, there were significant differences between the methods in the crowdsourced evaluations.

1310

Two typical interactions we observed help us to infer why there was no tendency of difference. In the first, the query1311

generated by CoCoA had the same weak effect on the advertisement as the one without a complementary word, even if

1312

CoCoA’s query was appropriate to the conversation context. In the second, a selected core query was enough for giving

1313
1314
1315
1316

context to the advertisement regardless of whether there was a complementary word.
The following transcript shows a case where the contextualized search query had only a weak advertisement effect(the same as the one without a complementary word).

1317
1318

Transcript 4 from Group E

1319
1320

E-2

I bought a flowerpot at IKEA and bought soil at Home Depot.

1321

E-3

It’s authentic.

1322

E-2

How many do you have?

1323

E-3

About 10 now.

E-2

What size?

E-3

Small sizes.

1327

CoCoA

“Fully private rooms for peace of mind” “size”

Q1: 5

Q2: 1

1328

Core

“Fully private rooms for peace of mind”

Q1: 2

Q2: 1

1324
1325
1326

1329
1330
1331

Users E-2 and E-3 talked about growing houseplants. Since CoCoA gave the Core-Query the complementary word

1332

“size”, it acquired a higher Q1 score (5 points) than the Core-Query without it (2 points). However, the selected Core-

1333
1334

Query “Fully private rooms for peace of mind” did not have enough attractiveness as an advertisement regardless1335

of having/not having the complementary word. The Q2 scores of both generated search queries also received the1336same low point value because the selected Core-Query “Fully private rooms for peace of mind” did not have enough

1337

attractiveness as an advertisement for the current conversation, even though it contained an appropriate contextual

1338
1339

word. The following transcript demonstrates that the Core-Query had a sufficient advertisement effect regardless of

1340

whether it contained a complementary word or not.

1341
1342
1343
1344

Transcript 5 from Group D
D-3

Yes, the overall concept was to create something like the bathhouse in “Spirited Away”.
So it’s like a festival with lots of food stalls and stuff,

1345

so the kids are basically never bored.

1346
1347

D-2

So it’s family fun there then.

1348

D-3

That’s the impression I got when I went there.

CoCoA“Three-day trip for you and your children” “family”

Q1: 7

Q2: 6

Core

“Three-day trip for you and your children”

Q1: 6

Q2: 6

1349
1350
1351
1352

26

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1353

CoCoA chose the Core-Query “Three-day trip for you and your children” in response to the conversation about a trip

1354

to a hot spring. Since the selected Core-Query fit the conversation topic, advertiser D-1 gave high scores for Q1 to both

1355
1356
1357
1358
1359
1360
1361

search queries: 7 for the query with a complementary word and 6 for the one without. In addition, the complementary
word “family” improved the Q1 value of the search query.
Advertiser D-1 gave the same evaluation (6) for Q2 for both search queries. The complementary word “family” did
not give new contextual information to the selected Core-Query, which included the phrase “your children”, but thecase had a sufficient advertisement effect due to the initially suitable design of the Core-Query.

1362

Although the violin plots indicate that the advertisers seemed to consider the effect of complementary words on the1363

advertisement to be insignificant (D-1’s and E-1’s answers for Q2 in Fig. 8), there was one search query among those

1364
1365
1366

generated by CoCoA that advertiser E-1 gave 6 points to in contrast to 5 points for the one without the complementary
word. A partial transcript from this interaction is as follows.

1367
1368
1369
1370
1371
1372
1373

Transcript 6 from Group E
E-2

That’s good. I’d like to try it in the hot summer too.

E-3

I want to go to the beach in the summer.

CoCoA“Bar with beautiful night view” “beach”

Q1: 6

Q2: 6

Core

“Bar with beautiful night view”

Q1: 5

Q2: 5

1374
1375
1376

CoCoA added the complementary word “beach” to the Core-Query “Bar with beautiful night view”, which was

1377

related to the conversation between the users. Advertiser E-1 mentioned the difficulty of giving advertisements to1378

the users who talked about topics utterly different from the target domain that the Core-Query designated. Since the

1379
1380

complementary word in the above example had the effect of contextualizing the Core-Query, the 6 points awarded to1381

Q2 against the search query indicates that advertiser E-1 recognized the effect of the advertisement.

1382

The violin plots for advertiser F-1’s answers indicate that she evaluated the search queries severely. She stated during

1383

the interview that it was difficult to cover a lot of conversation topics by utilizing Core-Queries prepared for a single

1384
1385

domain. Moreover, she was extremely strict when evaluating the search queries in terms of whether the users had been1386

talking about a trip. This style of evaluation resulted in lower scores for Q1 and Q2.

1387
1388
1389
1390
1391

5.3.5 Can the advertisers create the Core-Queries easily? All advertisers in these experiments were creating CoreQueries for the first time. The violin plots in Fig. 8 indicate that these Core-Queries included both “good” and “bad”
ones. Although relatively few queries were created overall, advertisers D-1 and E-1 felt they could provide appropriate

1392

queries on average, and F-1 also created three good Core-Queries. We investigated whether the advertisers were able to1393

make good Core-Queries by analyzing how well they grasped the created queries from questionnaire items Q4–6 and1394

the interview responses.

1395
1396

The responses of the advertisers to Q4 (good catchphrase) and Q5 (bad catchphrase) after the experiment indicated1397

they were able to judge the Core-Queries as good or bad simply by watching ten minutes of conversations conducted1398

on the basis of the Core-Queries. The specific answers to Q4 and Q5 are as follows.

1399
1400
1401

For Q4 (good catchphrase), advertiser D-1 selected “A little luxury! All-you-can-eat local gourmet tour.” The reason
for this answer was “because it can be used in a variety of situations and is very versatile.” Advertiser E-1 chose “Bar1402

with beautiful night view” and explained her reasoning by stating that “If the topic includes a place or scenery, users

1403

can relate to it even if they are talking about something other than food.” Advertiser F-1 chose “Tour to enjoy seafood,”

1404

27

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1405

commenting that “it would be easy to advertise because it can be displayed not only in trip conversations but also in1406

meal conversations.”

1407
1408

For Q5（bad catchphrase), advertiser D-1 selected “Enjoy a slightly different kind of holiday on a sleeper train.”1409

The reason given was “the limited number of places that can be reached by sleeper train and the lack of versatility.”

1410

Advertiser E-1 chose “Fully private rooms for peace of mind” because “while it was generic, the fact that it did not

1411
1412
1413
1414

refer directly to the restaurant made it a catchphrase with unintended connotations, depending on the words that
followed.” Advertiser F-1 selected “Summer swimming tours to beat the heat.” The reason given was that “it would besummer-only or it would not appear unless it was a sea-related conversation.”

1415

Watching the users’ 10-minute conversations helped the advertisers get the hang of creating Core-Queries. They1416

evaluated Core-Queries as good if they did not restrict information and could be handled in conversations in other

1417
1418
1419

domains, while Core-Queries that could only be used in more limited situations were judged as less versatile andtherefore as less desirable catchphrases.

1420

In Q6, we asked the advertisers to create additional Core-Queries based on their experience in the advertiser case1421

study. Advertiser D-1 came up with “For the active! Full-day activity tours”, “No worries, even for an indoor person! The

1422
1423
1424

hottest indoor tourist spot”, and “Student Discount: Cheap 2-day tours”. E-1 came up with “Stop by and enjoy gourmet
food”, and F-1’s answer was “Luxurious hot spring trip to enjoy seasonal ingredients”. These responses indicate that the1425

advertisers were able to learn how to create more sophisticated Core-Queries after just one experience of using the1426

CoCoA system.

1427
1428
1429

5.4

Discussion

1430
1431

5.4.1 Overview. Subsection 5.4 discusses the usage scenarios of CoCoA, suggestions for improvement, and limitations

1432

of the present setup based on the results of each case study.

1433
1434
1435

We can clarify the actual usage scenarios of CoCoA by taking into account the relationship between CoCoA users,
which also helps us to consider ways of improving CoCoA (see 5.4.2). From the user interviews, we found that there1436

was a clear advantage in terms of dialogue support in that it was possible to provide conversation topics to users with1437

low familiarity with each other. Furthermore, it is possible to improve the system to attract a broader range of users

1438

by adjusting the frequency of displaying advertisements based on the familiarity and relationships between users.

1439
1440

In particular, we found that the system had an effect on user interaction even when the topic of the dialogue was

1441

beyond the scope of what the Core-Query and the complementary word could cover (see 5.4.3). This means that if

1442

the advertisements displayed by CoCoA are in domains unrelated to the conversation, the unexpectedness of the ads

1443

could prompt users to tap them, thus changing the dialogue to the domain of the advertisements and activating user

1444
1445

conversation.

1446

We also investigated appropriate complementary words, what a better Core-Query is, and how to create a Core-Query.

1447

We found that proper complementary words were related to the context of the user’s dialogue, as is the goal of CoCoA.

1448

The discussion referring to the transcriptions of actual interactions reveals that advertisers gave high evaluations to

1449
1450

search queries that CoCoA generated by contextualizing the Core-Queries with appropriate complementary words

1451

from the users’ dialogues (see 5.4.4). In addition, the case study results indicate that better Core-Queries covered a1452

wider range of topics (see 5.4.5). Moreover, based on the results of the questionnaires for advertisers, we identified three1453

standard features that better Core-Queries should possess. The criteria provide a guideline for advertisers in creating

1454
1455
1456

Core-Queries in the future (see 5.4.6).
28

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1457

We emphasize the need to improve the user interface of CoCoA when implementing it in actual stores (see 5.4.7).

1458

The critical points of improvement are the frequency of advertisement presentation and the user’s gaze behaviors. The

1459
1460

reason for choosing these points is that the frequency of advertisements provided by the UI utilized in the case study1461

was too high for users to read, and the users felt it was inconvenient to turn their gaze to check the display during the1462

conversation.

1463

In addition to the above points, we also discuss possible biases and limitations in the case studies (see 5.4.8 and 5.4.9).

1464
1465

5.4.2 Effect of the relationship between users on the usage of CoCoA. We found that the less familiar the relationship

1466

between two users, the more they used the system. Once they started using the system, they continued to interact by

1467
1468

keeping the displayed domains open and did not leave. For example, the number of conversational turns, displayed

1469

queries, and taps was 42, 44, and 3, respectively, for Pair B, which indicates a high degree of familiarity between users.

1470

In contrast, these numbers were 32, 37, and 9, respectively, for Pair A, which indicates a low degree of familiarity. Thus,1471

the lower the familiarity, the more difficult it is to find a topic for conversation. The system thus played a key role in

1472
1473

providing a topic of conversation for people who were not particularly close to each other.

1474

For example, A-1 commented after the experiment, “I couldn’t think of a topic, so I used the system to find out”,

1475

and E-3 stated, “it may be useful in assisting communication between people who meet for the first time”. F-2 also

1476
1477

commented that “I found this helpful when there was a pause in the conversation. It might be even more helpful if it1478

was a first meeting.” Advertiser F-1 commented on the usefulness of CoCoA, saying “users who didn’t know each other1479

well could use this system to expand the conversation.”

1480

Since the low-familiarity pairs had fewer dialogue turns, they had more time to use the system to find topics when1481

both users were silent, which increased the number of taps. In contrast, since the close-familiarity pairs had more

1482
1483

dialogue turns, they paid less attention to the system. This is the reason for the low attention to the system in the1484

close-familiarity pairs.

1485

The case study results indicate that CoCoA has a distinct advantage in terms of supporting dialogue by giving a1486

topic of conversation to users with low familiarity. Furthermore, we believe the system can adjust the frequency of

1487
1488

displaying advertisements depending on the user’s relationship to include a broader range of users.

1489

Finally, we should point out that even close-familiarity pairs found it difficult to find a conversation topic when talking

1490

for a long time. The results from the low-familiarity pairs indicate that CoCoA can also support the close-familiarity

1491

pair when they run out of topics.

1492
1493
1494
1495

5.4.3 Advertisements for domains unrelated to the conversation trigger new topics of conversation. The domain of theongoing dialogue is not necessarily the same as the domain of the Core-Query that the advertiser wants to advertise.

1496

For example, users in Group E did not tap the advertisement for Food generated from the Core-Query related to the1497“Food” domain because the content of their dialogue was not in the Food domain. Furthermore, if the domain of the1498

Core-Query differs from the domain of the dialogue, CoCoA may not select the appropriate complementary word.

1499
1500

For example, if the user is talking about travel and CoCoA selects a Core-Query related to sports, CoCoA chooses a

1501

complementary word that is a place name, which is not necessarily appropriate for a sports Core-Query. As a result,1502

the generated search query may not be able to draw users’ attention.

1503
1504
1505

On the other hand, these search queries may be eye-catching to the user. Users in Group D selected a search query
unrelated to the dialogue because of the unexpectedness of the search results by tapping it. Also, users in pair B, Group

1506

D, and F tapped on the search query unrelated to their conversation. While they were looking at an advertisement1507

shown by the search query, the users shifted the topic of their conversation to the domain that the advertisers had

1508

29

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1509

originally wanted to advertise by the Core-Query. Advertiser E-1 also commented, “The catchphrases created by Robot

1510

1 (CoCoA) were often strange word combinations, but I found them to be interesting for the accidental ways they went

1511
1512

together.” The results of the case study suggest that CoCOA has the potential to attract the user’s interest by continuing1513

to present information even in domains that are irrelevant to the context. Furthermore, once the topic of conversation

1514

shifts to the same domain as the Core-Query over the course of continuous presentation, CoCoA can more accurately

1515

select complementary words and present more effective advertisements.

1516
1517

An earlier study on creativity support utilized a method of presenting information unrelated to the topic to activate

1518

user interactions and stimulate new ideas [46]. The usage scenarios of CoCoA include similar interactions of giving

1519

users topics they may not expect and activating new conversations. For example, F-3 stated, “I enjoyed learning about

1520
1521
1522
1523

new things. For example, I have never been to Tokyo Sky Tree, but it appeared on a web site I happened to be looking
at. I thought that was good in that it was not 100% related to what we were talking about, but slightly off, leading to a
new topic. It made me want to go to places I didn’t know existed.”1524

The main purpose of CoCoA is to advertise in a way that is relevant to the topic. However, the results of the case1525

study suggest that CoCoA can also benefit users by activating their conversation even beyond the scope of topics that

1526
1527
1528

Core-Query and complementary words can cover. We did not envision this advantage when we originally designed
CoCoA.

1529
1530
1531

5.4.4 Advertiser’s evaluation of contextualizing the Core-Queries. This section discusses the context awareness and1532

effectiveness of the advertisers’ responses to the search queries generated by CoCoA. A close examination of the search

1533

queries that the advertisers cited in Q7 (best ad) of the questionnaire demonstrates the effectiveness of CoCoA, since1534

the search queries that the advertisers in each group cited here were all generated by CoCoA. For example, the best ad

1535
1536
1537

in Group D, “A little luxury! All-you-can-eat local gourmet tour” “Okinawa”, was generated by CoCoA. The adjacentconversation is as follows.

1538
1539
1540
1541
1542

Transcript 7 from Group D
D-2

Yes, I went to Okinawa.

D-3

What were you doing there? Did you just want to go to Okinawa?

1545

CoCoA

“A little luxury! All-you-can-eat local gourmet tour” “Okinawa”

Q1: 6

Q2: 4

1546

Core

“A little luxury! All-you-can-eat local gourmet tour”

Q1: 5

Q2: 4

1543
1544

1547
1548
1549
1550
1551

The reason this was considered the best ad is presumably that words representing place names have a high affinity1552

with “Trip”, the domain of Group D’s advertisements, and the word “Okinawa”, which represents a place name that

1553
1554

appeared in the conversation immediately before the word, was complemented, thus increasing its suitability as the1555

search query. Therefore, in the trip domain, ratings were often higher when a word representing a place was chosen

1556

as a complementary word in the dialogue. For example, the conversation about sports continued and turned to E-2’s

1557

hobby of golf. In transcript 8, the place name “Saitama”, which appeared in the conversation as a place where a golf

1558
1559
1560

course is located, was chosen as a complementary word.
30

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1561
1562

Transcript 8 from Group E
E-2

1563
1564
1565
1566

There are not many golf courses near here to begin with,
but golf courses in Kawasaki and Tama are really expensive.

E-3

I see.

E-2

But if you go to the Chichibu area in Saitama Prefecture, or Yamanashi, Ibaraki,or Tochigi, it is a little cheaper.

1567
1568

CoCoA

“Old-fashioned retro coffee shop” “Saitama”

Q1: 6

Q2: 5

1569

Core

“Old-fashioned retro coffee shop”

Q1: 3

Q2: 3

1570
1571
1572
1573
1574
1575
1576
1577
1578
1579

This narrowed down the location of the Core-Query “Old-fashioned retro coffee shop”, which is why the CoCoA
search query was rated higher in terms of both Context awareness and Effectiveness than the one by Core.
As the above examples show, CoCoA is able to generate good search queries by contextualizing the Core-Queries
with contextual complementary words.
While CoCoA can generate good search queries, the overall results of the advertiser case study indicate that advertisers
often did not perceive a clear difference in the advertising effectiveness (Q2: Effectiveness) of CoCoA’s contextualized1580

search queries from the original Core-Queries. Unlike the crowdsourced experiment in Section 4, where the search

1581

queries were evaluated on the conversational text, in the advertiser case study, where the search queries were evaluated

1582
1583

on the live interaction, there were multiple factors that may affect which advertisements appeal to users, and this was1584

the main reason there was little difference in the evaluation in Q2. Specifically, in Section 4, we found that CoCoA was

1585

significantly superior to the other systems with respect to Q2 in both the Context sensitivity test and the Robustness

1586

test (Effectiveness in Figs. 3 and 4).

1587
1588
1589

Although the advertiser case study cannot be directly compared with these results because they are descriptive
statistics of the individual evaluations by each advertiser, there is no noticeable difference among all of the violin plots

1590

for Q2 in Fig. 8, and the results of the advertiser case study reveal a different trend from the Effectiveness results of the1591

context sensitivity test and robustness test.

1592
1593

The most significant reason unique to the case studies is that advertisers may rate the system less highly if they use

1594

it less frequently. For example, advertiser E-1 responded in Q8 (Impression) that “I felt it was difficult to advertise when1595

we were having a conversation with a completely different domain,” and the evaluation was lower overall than that of

1596

Group D. In fact, while the domain of Group E’s Core-Query was “Food”, there was no dialogue in which “Food” was

1597
1598

the domain, and no taps were made, so Q2 (Effectiveness) was also rated low overall. On the other hand, the maximum

1599

score of Q1 may have been higher for CoCoA than for Q2 in terms of the combination of contextualized complementary

1600

words. Thus, it seems likely that Q2 was lower overall because it was used less frequently owing to the dialogue not1601

always transitioning to the dialogue in the domain of the Core-Query in a short 10-minute dialogue.

1602
1603
1604
1605

5.4.5 Appropriate breadth of topic domain for designing Core-Queries. The results in subsection 5.3.3 and corresponding
discussion in 5.4.3 and 5.4.4 suggest that Core-Queries should have a wide domain that can cover a variety of topics.

1606

Since CoCoA selects Core-Queries that are appropriate for the user’s conversation and at the same time selects a

1607

complementary word that is appropriate for applying the Core-Query to the context of the conversation, even if the1608

domain of the Core-Query is broad, we can expect the search query to be contextualized and on-topic. To increase the

1609
1610
1611
1612

chances of appropriate Core-Queries appearing and thereby increase the effectiveness of advertising, it is important todesign Core-Queries with an awareness of the breadth of the domain.
31This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1613

In this case study, each group created ten Core-Queries with one domain, so there were situations where the domain

1614of the Core-Query did not match the domain of the dialogue, making it difficult to display the advertisement. In fact, as

1615
1616

advertiser D-1 stated, “I felt there were situations that could not be handled with the ten catchphrases I had prepared.”

1617

Advertiser E-1 similarly stated that “I found it difficult to advertise when users were having a completely different

1618

domain conversation.” She then added, “I felt that I needed to either create catchphrases that connect the different

1619

domains or prepare ads for multiple domains.” This suggests that, if multiple advertisers were simultaneously creating

1620
1621
1622
1623
1624
1625
1626
1627

Core-Queries for different domains, it would be possible to present appropriate advertisements based on the domain of
the conversation and on the user interests.
As indicated by the above comments from the participants playing the role of advertisers, a method of preparing
Core-Queries for each of multiple different domains should be considered. At the same time, considering CoCoA’sability to contextualize Core-Queries, it would also be effective to design Core-Queries that can span multiple domains
to some extent.

1628
1629

5.4.6 How to prepare Core-Queries. In the advertiser case study, participants in the advertiser role gained a better

1630

understanding of how to design Core-Queries after reflecting on the Core-Queries they created in a single experiment.

1631

This relatively easy learning curve suggests that only a few cycles of creating Core-Queries and using the CoCoA are

1632
1633

required for helping advertisers figure out the appropriate way to create Core-Queries. Specifically, advertisers were1634

able to understand that it is important to create Core-Queries that clearly represent the domain they want to advertise1635

but that can also be used for a wider range of domains. The three advertiser participants in this case study created1636

Core-Queries for the first time, and many of the search queries created on the basis of these prepared Core-Queries

1637
1638

were found to be effective for users, as indicated by the violin plot in Fig. 8. We found that even advertisers with no1639

experience in creating Core-Queries could present more effective advertisements to users by using CoCoA. In the1640

following, we present some basic considerations for how to create effective Core-Queries.

1641
1642
1643

The responses to Q4, Q5, and Q6 (subsection 5.3.5 Can the advertisers create the Core-Queries easily?) indicate thatthe following three points are common to effective Core-Queries.

1644

1. Core-Queries that contain the elements of the domain you want to advertise.

1645

2. Core-Queries that can be used for other domains.

1646
1647

3. Core-Queries that are not too specific.

1648

Regarding 1, advertiser E-1 chose the domain of “Food”, created Core-Queries, and then indicated after the experiment

1649

that the Core-Query “Fully private rooms for peace of mind” was least effective (Q5: bad catchphrase). As a reason,

1650
1651

E-1 stated that “this Core-Query did not explicitly include the domain of food that I wanted to advertise, and as a

1652

result, search queries for the domain of food were not created.” This comment indicates that if the Core-Query does1653

not adequately represent the target advertising domain, it may generate search queries that are generic but do not1654

accurately advertise.

1655
1656
1657

Regarding 2, advertiser D-1 selected “A little luxury! All-you-can-eat local gourmet tour” as a good catchphrase(Q4), and advertiser F-1 selected “Tour to enjoy seafood”. In both cases, it was possible to take a place name as a

1658

complementary word in the trip domain, and since the Core Query also included an element of “Food”, advertisements

1659

in a wider range of domains could be presented. Another search query, “A morning menu to energize you for the day”,

1660
1661

which was created by E-1 and displayed during a conversation about sports, was similarly an effective Core-Query1662

for both “Food” and “Sports”. Thus, Core-Queries that can be associated with multiple domains have the potential to1663

generate appropriate search queries in a broader domain dialogue.

1664

32

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1665

Regarding 3, for Q5 (bad catchphrase), advertiser D-1 selected the Core-Query “Enjoy a slightly different kind of

1666

holiday on a sleeper train”, which included a very specific mode of transportation (“sleeper train”). While this specific

1667
1668

Core-Query represented the travel domain well, it was less versatile and actually used only once as a Core-Query.

1669

We believe that by presenting the above three characteristics of effective Core-Queries to advertisers as design guide-1670

lines, they will get the hang of preparing Core-Queries more quickly. We should point out of course that characteristics

1671

1 and 2 are contradictory and neither can be fulfilled perfectly. However, by designing Core-Queries while at least

1672
1673
1674
1675

keeping 1 and 2 in mind, advertisers can be motivated to adjust the size of the domain to the extent possible.
5.4.7 How to display search results and where the monitor should be. The proper placement and UI of the display

1676

showing advertisements based on the search query created by CoCoA needs to be improved. Our objective in the case1677

study was to automatically generate search queries tailored to the content of the user’s dialogue, but if the system is to1678

be used in a real environment, it needs to be provided in a form that is more user-friendly and that places less of a1679

cognitive load on the user. We therefore asked participants for their opinions about the UI after each experiment. User

1680
1681

A-2 commented that “there were so many buttons to be updated that they would disappear the moment we tried to tap

1682them”, so we changed the number of buttons to be updated from five to two in subsequent experiments.

1683
1684
1685
1686

Regarding Group D, when we asked if they had noticed the search query about “Okinawa”, as they did not seeminterested in it when they were talking about “Okinawa”, D-2 said, “I hadn’t noticed. The screen was right next towhere we were talking, so I didn’t have a chance to see it. We were both looking right at the other person, so I knew

1687

something was on the side screen, but it was hard to see.” D-3 mentioned, “I think it would be better if the web site was1688

displayed automatically, instead of just having a button waiting on the left screen to open the web site. Although the1689

button displays the title of the web site, it is just a list of characters and has a lot of information. It would be better

1690
1691

to have the page open suddenly.” In addition, D-2’s comment that “It is also difficult to see that the same things are1692

being displayed” suggests that the same advertisement with similar queries increases the cognitive load on the users. In

1693

Group E, E-2 commented that “It would be easier to see the system if it were placed on the table between us, because

1694

the direction of eye movement would be up and down.” E-3 said “I would like to see animations that move up and down

1695
1696

when the list on the left side of the screen is updated. This would make it easier to see which button is the new one,1697

and the movement would draw the eye.”

1698
1699
1700
1701
1702
1703

Thus, our future work will focus on determining the appropriate display positions and how to display search queries
in ways that facilitate eye guidance during dialogue.
5.4.8 Bias in the use of CoCoA. In any case study conducted in a laboratory setting, biases that encourage participants
to use the target system naturally exist, so care must be taken in interpreting the results regarding whether participants

1704

used the system or not. In our case study, participants sat across from each other at a desk in a laboratory equipped1705

with a microphone and chatted with each other while interacting (or not) with a display. This environment entails a

1706

possible bias in that it is designed for usage of the system.

1707
1708

We attempted to minimize this potential bias by telling participants that they could use the system if they wanted to1709

but by no means were they required to. In fact, some of the participants never used the system, so we believe the bias

1710

was effectively minimized.

1711
1712
1713

The instructions we provided to users brought them closer to the actual situation in which they use applications on
websites and mobile phone, i.e., with the understanding that advertisements will be displayed on them. Specifically, when

1714

using such applications, it is up to the user’s free will whether or not to click and view the presented advertisements. In

1715

this experiment, we tried to create a usage scenario that mimics this situation as closely as possible.

1716

33

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

Overall, although there was some degree of bias regarding CoCoA use in this case study, we believe it was fairly

1717
1718

close to the actual use scenario of the system and that the bias was minimized as much as it could be.

1719
1720

5.4.9

Limitations of the case study. This case study has the following limitations

1721

Dependence on BERT model: Since the processing of dialogue text depends on the BERT model, CoCoA may not

1722
1723
1724
1725
1726

perform correctly for words that were not included when the BERT model was trained, for dialogue texts in completely
new domains, or for grammatically incorrect sentences. CoCoA may not have performed as designed depending on the
topic of the conversation, the user’s utterances, or Core-Query.

1727
1728

Limitations of speech recognition: Since CoCoA utilizes speech recognition technology to convert user dialogue into

1729

dialogue text before processing it, this case study was conducted within the performance limits of speech recognition. In

1730

our case studies, some search queries were rated lower due to speech recognition errors in which the words uttered by the1731

user were not recognized correctly, resulting in words that were not included in the dialogue becoming complementary

1732
1733

words. In addition, in speech recognition, failure to detect speech segments and misrecognition can occur due to1734

surrounding noise. Since this case study was conducted in a laboratory setting, the effects of environmental noise were1735

eliminated.

1736

Limitations of search databases: The database used in this experiment was not a dedicated database for advertisements,

1737
1738

so there may be no search results at all, or it may happen to show something like it in an online search.

1739
1740
1741

6

FUTURE WORK

1742

Although the experimental results demonstrated that CoCoA can generate advertisements in accordance with the1743

conversational context, challenges still remain in applying this system to the real world.

1744
1745
1746

Resolving the privacy issue is critical if people are going to accept our approach. At the very least, we need to ask
users to provide their consent prior to use. Then, after obtaining consent, there should be some sort of benefit involved,

1747

such as the user getting a coupon or using the facility at a lower price, as in the scenario described in Section 1. At

1748

present, our research is focused on demonstrating the application of masked word prediction and we have not addressed

1749
1750

the privacy issue yet. In putting CoCoA to practical use, it is also important to resolve the privacy issue on a technical1751

level. For example, we need to identify proper nouns extracted from dialogues that lead to personal information or

1752

domain words that users feel uncomfortable with, and to improve the system so that effective advertisements can be

1753

provided while protecting privacy at the same level as other personalized advertisements.

1754
1755

As pointed out by the experiment participants in the case study, the current method of presenting advertisements

1756

in CoCoA requires various minor improvements. On the other hand, the interaction scenes that can be supported

1757

by the CoCoA technology are not limited to human interaction. For example, it is possible to extend the technology

1758
1759
1760

to present advertisements linked to text chats such as Messenger, or to present information on special offers. Whenactually developing applications, support considerations and user interface designs that are different from those of1761

CoCoA will be necessary. Further development of the method constructed in this paper will be possible through new1762

research and development.

1763
1764
1765

One significant development issue for the practical application of CoCoA in the future will be the resolution ofproblems related to speech recognition. Besides improving speech recognition technology, we believe that CoCoA’s

1766

ability to present advertisements can be improved. To solve the problems caused by misrecognition of speech, consider

1767

first that the attributes of words that are compatible with a Core-Query will vary from domain to domain. For example,

1768

34

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1769

in the travel domain, complementary words are compatible with words that describe places. Thus, when the advertiser

1770

creates a Core-Query for “Trip”, if the system can automatically select only words with the attribute of location as

1771
1772

complementary words, it can create search queries for the domain the advertiser wants to advertise. At the same time,1773

if the system can filter out complementary words that are irrelevant to that domain or are generic (e.g., “regular”),

1774

advertisers will be able to provide more effective advertisements, and users will see advertisements that are relevant to1775

the context of the interaction.

1776

Improving the language model used in CoCoA is also important for system improvement. The BERT used by CoCoA

1777
1778

is based on an existing Japanese model trained using the Japanese Wikipedia corpus, but it is also possible to fine-tune

1779

the BERT model with dialogue data, to fine-tune it with the assumption that Core-Queries are used, and to use other

1780
1781
1782

natural language models to generate effective advertisements that are more appropriate to the dialogue context from awider range of Core-Queries. Furthermore, CoCoA currently adds only one complementary word to a Core-Query,1783

but by increasing the number of complementary words, more specific search queries can be generated. Furthermore,

1784

instead of combining Core-Queries and complementary words, Core-Queries can be converted into a single sentence

1785

that matches the context based on the Core-Query, and that sentence can be presented as an advertisement. This could

1786
1787

then be presented to the user in a way that feels more intuitive for them.

1788
1789
1790
1791

7

CONCLUSION

In this paper, we proposed CoCoA, a method for dynamically generating search queries for generating context-sensitive1792

advertisements by utilizing mask prediction. CoCoA consists of a Core-Query Selector and a BERT-Query Completer

1793(B-QC). The Core-Query Selector calculates the similarity of sentence vectors between user utterances and Core-Queries,

1794

abstract advertising sentences prepared by the advertiser, and selects one that fits the context of the dialogue best.

1795
1796
1797

In the B-QC, we used masked word prediction to generate concrete search queries in real time by complementing
Core-Queries selected by the Core-Query Selector with an appropriate word from the dialogue history. We can extract

1798

context-sensitive advertisements by utilizing word-complemented search queries. We conducted experiments in which

1799

users evaluated the advertisements presented by CoCoA against dialogues in several domains prepared in advance.

1800
1801

The results showed that CoCoA was able to present more contextually relevant and effective advertisements than

1802

Google Suggest or a method without word completion. In addition, we found that CoCoA could generate high-quality

1803

advertisements that advertisers had not expected when they created the advertisements. In addition, we conducted

1804

case studies involving users and advertisers in using CoCoA. The results indicated that pairs unfamiliar with each

1805
1806

other referred to the advertisement CoCoA showed on display more frequently, which increased the chance for the1807

advertisement to affect their conversation topics. Moreover, the advertisers highly evaluated the quality of the search1808

queries. They easily grasped the way of designing a good Core-Query by observing the response of the users to the1809

advertisements retrieved with the generated search queries.

1810
1811
1812

ACKNOWLEDGMENTS

1813

This work was supported by JSPS KAKENHI Grant number JP21J13789 and JST CREST Grant number JPMJCR19A1,

1814

Japan.

1815
1816
1817

REFERENCES

1819[1] Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Applying BERT to Document Retrieval with
Birch. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference

1820

35

1818

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1871

on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations (Hong Kong, China). Association for Computational Linguistics, 19–24.
https://doi.org/10.18653/v1/D19-3004[2] Ziv Bar-Yossef and Naama Kraus. 2011. Context-Sensitive Query Auto-Completion. In Proceedings of the 20th International Conference on World
Wide Web (Hyderabad, India) (WWW ’11). Association for Computing Machinery, New York, NY, USA, 107–116. https://doi.org/10.1145/1963405.
1963424[3] Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning End-to-End Goal-Oriented Dialog. In International Conference on Learning
Representations. https://openreview.net/forum?id=S1Bb3D5gg[4] Salem Mohamed S Busen and Che Su Mustaffa. 2014. The role of interactive advertisements in developing consumer-based brand equity: A
conceptual discourse. Procedia-Social and behavioral sciences 155 (2014), 98–103. https://doi.org/10.1016/j.sbspro.2014.10.263[5] Verolien Cauberghe and Patrick De Pelsmacker. 2006. Opportunities and Thresholds for Advertising on Interactive Digital TV. Journal of Interactive
Advertising 7, 1 (2006), 2–23. https://doi.org/10.1080/15252019.2006.10722122[6] Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. Towards Knowledge-Based Recommender
Dialog System. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint
Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China). Association for Computational Linguistics, 1803–1813. https://doi.org/10.18653/v1/D19-1189[7] Keith Cheverst, Nigel Davies, Keith Mitchell, and Adrian Friday. 2000. Experiences of Developing and Deploying a Context-Aware Tourist Guide:
The GUIDE Project. In Proceedings of the 6th Annual International Conference on Mobile Computing and Networking (Boston, Massachusetts,USA) (MobiCom ’00). Association for Computing Machinery, New York, NY, USA, 20–31. https://doi.org/10.1145/345910.345916[8] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM
Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA,
191–198. https://doi.org/10.1145/2959100.2959190[9] Jeffrey Dalton, Laura Dietz, and James Allan. 2014. Entity Query Feature Expansion Using Knowledge Base Links. In Proceedings of the 37th
International ACM SIGIR Conference on Research & Development in Information Retrieval (Gold Coast, Queensland, Australia) (SIGIR ’14). Association for Computing Machinery, New York, NY, USA, 365–374. https://doi.org/10.1145/2600428.2609628[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language
Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers) (Minneapolis, Minnesota). Association for Computational Linguistics, 4171–4186.
https://doi.org/10.18653/v1/N19-1423[11] Siyu Duan, Wei Li, Jing Cai, Yancheng He, and Yunfang Wu. 2021. Query-Variant Advertisement Text Generation with Association Knowledge.
In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (Virtual Event, Queensland, Australia)(CIKM ’21). Association for Computing Machinery, New York, NY, USA, 412–421. https://doi.org/10.1145/3459637.3482290[12] Yong-Yi Fanjiang and Yu-Yu Wang. 2017. Context-aware services delivery framework for interactive mobile advertisement. Computer
Communications 99 (2017), 63–76. https://doi.org/10.1016/j.comcom.2016.08.002[13] Carlos A Gomez-Uribe and Neil Hunt. 2015. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on
Management Information Systems (TMIS) 6, 4 (2015), 1–19.
[14] Maria Guinea, Isabel Litton, Rigel Smiroldo, Irma Nitsche, and Eric Sax. 2020. A Proactive Context-Aware Recommender System for In-Vehicle Use.
In Proceedings of the 2020 4th International Conference on Vision, Image and Signal Processing (Bangkok, Thailand) (ICVISP 2020). Association
for Computing Machinery, New York, NY, USA, Article 32, 8 pages. https://doi.org/10.1145/3448823.3448852[15] Nor Hazlina Hashim, Normalini, and Norhazlina Sajali. 2018. The Influence Factors Towards Mobile Advertising Message Content on Consumer
Purchase Intention. Global Business Review 19, 5 (2018), 1187–1206. https://doi.org/10.1177/0972150918788746[16] Elaine M. Huang, Anna Koster, and Jan Borchers. 2009. Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at
Public Displays?. In Proceedings of the 6th International Conference on Pervasive Computing (Sydney, Australia) (Pervasive ’08). Springer-Verlag,
Berlin, Heidelberg, 228–243. https://doi.org/10.1007/978-3-540-79576-6_14[17] Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul Crook, Y-Lan Boureau, and Jason Weston. 2019. Recommendation as a Communication
Game: Self-Supervised Bot-Play for Goal-oriented Dialogue. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China). Association for
Computational Linguistics, 1951–1961. https://doi.org/10.18653/v1/D19-1203[18] Dimitrios Kastrinakis and Yannis Tzitzikas. 2010. Advancing Search Query Autocompletion Services with More and Better Suggestions. In
Proceedings of the 10th International Conference on Web Engineering (Vienna, Austria) (ICWE’10). Springer-Verlag, Berlin, Heidelberg, 35–49.
[19] Matěj Kocián, Jakub Náplava, Daniel Štancl, and Vladimír Kadlec. 2022. Siamese BERT-Based Model for Web Search Relevance Ranking Evaluatedon a New Czech Dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 12369–12377.
[20] Kazunori Komatani and Shogo Okada. 2019. Osaka University Multimodal Dialogue Corpus (Hazumi). https://doi.org/10.32130/rdata.4.1[21] Kai Li and Timon C Du. 2012. Building a targeted mobile advertising system for location-based services. Decision Support Systems 54, 1 (2012),
1–8.
[22] Raymond Li, Samira Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations.
In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS’18). Curran Associates

1872

36

1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Conversational Context-Sensitive Ad Generation With a Few Core-Queries

1923

Inc., Red Hook, NY, USA, 9748–9758.
[23] Yanen Li, Anlei Dong, Hongning Wang, Hongbo Deng, Yi Chang, and ChengXiang Zhai. 2014. A Two-Dimensional Click Model for Query AutoCompletion. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (Gold Coast,
Queensland, Australia) (SIGIR ’14). Association for Computing Machinery, New York, NY, USA, 455–464. https://doi.org/10.1145/2600428.2609571[24] Yahui Liu, Wei Bi, Jun Gao, Xiaojiang Liu, Jian Yao, and Shuming Shi. 2018. Towards Less Generic Responses in Neural Conversation Models: A
Statistical Re-weighting Method. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (Brussels, Belgium).
Association for Computational Linguistics, 2769–2774. https://doi.org/10.18653/v1/D18-1297[25] Zhibin Liu, Zheng-Yu Niu, Hua Wu, and Haifeng Wang. 2019. Knowledge Aware Conversation Generation with Explainable Reasoning overAugmented Graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International
Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Hong Kong, China). Association for Computational Linguistics, 1782–1792.
https://doi.org/10.18653/v1/D19-1187[26] Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. Towards Conversational Recommendation over Multi-Type
Dialogs. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Online). Association for Computational
Linguistics, 1036–1049. https://doi.org/10.18653/v1/2020.acl-main.98[27] Nurul Lubis, Sakriani Sakti, Koichiro Yoshino, and Satoshi Nakamura. 2019. Positive Emotion Elicitation in Chat-Based Dialogue Systems. IEEE/ACM
Transactions on Audio, Speech, and Language Processing 27, 4 (2019), 866–877. https://doi.org/10.1109/TASLP.2019.2900910[28] Wenchang Ma, Ryuichi Takanobu, and Minlie Huang. 2021. CR-Walker: Tree-Structured Graph Reasoning and Dialog Acts for Conversational
Recommendation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Online and Punta Cana, Dominican
Republic). Association for Computational Linguistics, 1839–1851. https://doi.org/10.18653/v1/2021.emnlp-main.139[29] Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized Embeddings for Document Ranking. In Proceedingsof the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Paris, France) (SIGIR’19). Association
for Computing Machinery, New York, NY, USA, 1101–1104. https://doi.org/10.1145/3331184.3331317[30] Rishi Madhok, Shashank Mujumdar, Nitin Gupta, and Sameep Mehta. 2018. Semantic understanding for contextual in-video advertising. In
Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[31] Jörg Müller, Florian Alt, Daniel Michelis, and Albrecht Schmidt. 2010. Requirements and Design Space for Interactive Public Displays. In Proceedingsof the 18th ACM International Conference on Multimedia (Firenze, Italy) (MM ’10). Association for Computing Machinery, New York, NY, USA,
1285–1294. https://doi.org/10.1145/1873951.1874203[32] Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019).
[33] Ramith Padaki, Zhuyun Dai, and Jamie Callan. 2020. Rethinking query expansion for BERT reranking. In Advances in Information Retrieval: 42ndEuropean Conference on IR Research (Lisbon, Portugal). Springer, 297–304.
[34] Terry R. Payne, Esther David, Nicholas R. Jennings, and Matthew Sharifi. 2006. Auction Mechanisms for Efficient Advertisement Selection on Public
Displays. In Proceedings of the 4th European Workshop on Multi-Agent Systems.
[35] Ajax Persaud and Irfan Azhar. 2012. Innovative mobile marketing via smartphones. Marketing Intelligence & Planning 30, 4 (2012), 418–443.
[36] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084 [cs.CL][37] Wataru Sakata, Tomohide Shibata, Ribeka Tanaka, and Sadao Kurohashi. 2019. FAQ Retrieval Using Query-Question Similarity and BERT-Based
Query-Answer Relevance. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval(Paris, France) (SIGIR’19). Association for Computing Machinery, New York, NY, USA, 1113–1116. https://doi.org/10.1145/3331184.3331326[38] Ryoichi Shibata, Shoya Matsumori, Yosuke Fukuchi, Tomoyuki Maekawa, Mitsuhiko Kimoto, and Michita Imai. 2022. Utilizing Core-Query forContext-Sensitive Ad Generation Based on Dialogue. In Proceedings of the 27th International Conference on Intelligent User Interfaces (Helsinki,
Finland) (IUI ’22). Association for Computing Machinery, New York, NY, USA, 734–745. https://doi.org/10.1145/3490099.3511116[39] Masahiro Shiomi, Kazuhiko Shinozawa, Yoshifumi Nakagawa, Takahiro Miyashita, Toshio Sakamoto, Toshimitsu Terakubo, Hiroshi Ishiguro, andNorihiro Hagita. 2013. Recommendation Effects of a Social Robot for Advertisement-Use Context in a Shopping Mall. International Journal ofSocial Robotics 5, 2 (2013), 251–262. https://doi.org/10.1007/s12369-013-0180-4[40] Milad Shokouhi. 2013. Learning to Personalize Query Auto-Completion. In Proceedings of the 36th International ACM SIGIR Conference on
Research and Development in Information Retrieval (Dublin, Ireland) (SIGIR ’13). Association for Computing Machinery, New York, NY, USA,
103–112. https://doi.org/10.1145/2484028.2484076[41] Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In The 41st International ACM SIGIR Conference on Research &Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR ’18). Association for Computing Machinery, New York, NY, USA, 235–244.
https://doi.org/10.1145/3209978.3210002[42] Shiree Treleaven-Hassard, Joshua Gold, Steven Bellman, Anika Schweda, Joseph Ciorciari, Christine Critchley, and Duane Varan. 2010. Using theP3a to gauge automatic attention to interactive television advertising. Journal of Economic Psychology 31, 5 (2010), 777–784. https://doi.org/10.
1016/j.joep.2010.03.007[43] Paraskevi Vennou, Evangelia Mantzari, and George Lekakos. 2011. Evaluating Program-Embedded Advertisement Format in Interactive Digital
TV. In Proceedings of the 9th European Conference on Interactive TV and Video (Lisbon, Portugal) (EuroITV ’11). Association for Computing
Machinery, New York, NY, USA, 145–154. https://doi.org/10.1145/2000119.2000149[44] Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 (2015).

1924

37

1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922

This is a preprint of an article published in ACM Transactions on Interactive Intelligent Systems.
The final authenticated version is available online at: https://doi.org/10.1145/3588578

Ryoichi Shibata, et al.

1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947

[45] Tung Vuong, Salvatore Andolina, Giulio Jacucci, and Tuukka Ruotsalo. 2021. Spoken Conversational Context Improves Query Auto-Completion in
Web Search. ACM Transactions on Information Systems (TOIS) 39, 3 (2021). https://doi.org/10.1145/3447875[46] Hao-Chuan Wang, Dan Cosley, and Susan R. Fussell. 2010. Idea Expander: Supporting Group Brainstorming with Conversationally Triggered Visual
Thinking Stimuli. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (Savannah, Georgia, USA) (CSCW ’10).
Association for Computing Machinery, New York, NY, USA, 103–106. https://doi.org/10.1145/1718918.1718938[47] Po-Wei Wang, J. Zico Kolter, Vijai Mohan, and Inderjit S. Dhillon. 2018. Realtime query completion via deep language models. In The 2018 SIGIR
Workshop On eCommerce. https://openreview.net/forum?id=By3VrbbAb[48] Kirsty Williamson. 2018. Research concepts (second edition ed.). Chandos Publishing. 3–25 pages. https://doi.org/10.1016/B978-0-08-1022207.00001-7[49] Chenyan Xiong and Jamie Callan. 2015. Query Expansion with Freebase. In Proceedings of the 2015 International Conference on The Theoryof Information Retrieval (Northampton, Massachusetts, USA) (ICTIR ’15). Association for Computing Machinery, New York, NY, USA, 111–120.
https://doi.org/10.1145/2808194.2809446[50] Jheng-Hong Yang, Sheng-Chieh Lin, Chuan-Ju Wang, Jimmy Lin, and Ming-Feng Tsai. 2019. Query and Answer Expansion from Conversation
History. In Proceedings of the Twenty-Eighth Text REtrieval Conference (Gaithersburg, Maryland, USA) (NIST Special Publication, Vol. 1250).
National Institute of Standards and Technology (NIST). https://trec.nist.gov/pubs/trec28/papers/CFDA_CLIP.C.pdf[51] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-End Open-Domain Question
Answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics(Demonstrations) (Minneapolis, Minnesota). Association for Computational Linguistics, 72–77. https://doi.org/10.18653/v1/N19-4013[52] Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Simple applications of BERT for ad hoc document retrieval. arXiv preprint arXiv:1903.10972(2019).
[53] Xiaoyao Zheng, Yonglong Luo, Liping Sun, Ji Zhang, and Fulong Chen. 2018. A tourism destination recommender system using users’ sentimentand temporal dynamics. Journal of Intelligent Information Systems 51, 3 (2018), 557–578.
[54] Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020. Towards Topic-Guided Conversational Recommender
System. In Proceedings of the 28th International Conference on Computational Linguistics (Barcelona, Spain (Online)). International Committee
on Computational Linguistics, 4128–4139. https://doi.org/10.18653/v1/2020.coling-main.365

1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976

38