ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
Authors: Haoqin Tu
Yitong Li
Fei Mi
Zhongliang Yang
Venue: EMNLP 2023 Main (2023)
Keywords: visual dialogue
multimodal dataset
knowledge-enhanced dialogue
pre-trained language model
TL;DR: Two currently the most fine-grained visual dialogue datasets with entity&turn-level images, and a unified multimodal dialogue system with either shared or separate encoder-decoder setup.

Abstract: Incorporating visual knowledge into text-only dialogue systems has become a potential direction to imitate the way humans think, imagine, and communicate. However, existing multimodal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To address these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (ReSee-$\texttt{WoW}$, ReSee-$\texttt{DD}$). We propose to explicitly split the visual knowledge into finer granularity ("turn-level" and "entity-level"). To further boost the accuracy and diversity of augmented visual information, we retrieve them from the Internet or a large image dataset. To demonstrate the superiority and universality of the provided visual knowledge, we propose a simple but effective framework ReSee to add visual representation into vanilla dialogue models by modality concatenations. We also conduct extensive experiments and ablations w.r.t. different model configurations and visual knowledge settings. Empirical, encouraging results not only demonstrate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model ReSee outperforms several state-of-the-art methods on automatic and human evaluations. By leveraging text and vision knowledge, ReSee can produce informative responses with real-world visual concepts. Our code is available at https://github.com/ImKeTT/ReSee.

RESEE:
Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
Haoqin Tu', Yitong L Fei Mi², Zhongliang Yang**
University of Chinese Academy of Sciences, HHaawei Noah's Ark Lab
Huawei Technologies Ltd. "Beijing University of Post and Telccommunications
caisaact 630gma com, (1 (1ryiteng3, mi fei2) @huawel com, yangz. @bupt wbwpt.edu.c.. on
Abstract
Sessione avel Image
Turn-Leve Image
Incorporating visual knowledge into text-only dialogue systems has become a potential direc- tion to imitate the way humans think, imagine, and communicate. However, existing multi- modal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To ad- dress these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (RESEE-WOW, RESEE- DD). We propose explicitly split the visual knowledge into finer granularity ("turn-level" and "entity-levell". racy and diversity of augmented visual infor- mation, we retrieve them from the Internet or large image dataset. To demonstrate the supe- riority and universality of the provided visual knowledge, we propose a simple but effective framework RESEE add visual representation into vanilla dialogue models by modality con- catenations. We also conduct extensive experi- ments and ablations w.r.l different model con- figurations and visual knowledge settings. Em- pirically, encouraging results not only demon- strate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model RESEE out- performs several state-of-the-an methods automatic and human evaluations. By leverag- ing text and vision knowledge, RESBE can pro- duce informative responses with real-world vi- sual concepts. Our code available https: / //github. com/ TT ReSee.
Ammleaiinii jasts on tiuunie places lazz (icin)).
Can you brook the
Eacity oaayy-Laaalle lmnaps
It Dalmston
Vooi, Fye really cnpyat niong my kind The Appabbcianltaii was one my Anoeiis'quuale long 12. miled
You &b.
Eaiity Lend Iinege
Figure Traditional visual dialogue (left) is grounded on single given picture, while the proposed multimodal dialogue (right) provides both Turn-Level and Entity- Level images based text-only dialogue data.
further boost accu-
communications often requires real-world experi- ences shared between speakers (Bisk et al., 2020). Text alone may fall short in accurately conveying rich world knowledge (Harnad, 1990), where visual signals are essential to share experiences and conduct high-quality conversations. As humans converse day to day, it is common and natural for them to group information into smaller chunks of memory through images. That explains why incorporating visual perceptions in dialogue systems can potentially bring the conversation quality to a higher level.
Visual dialogue (Das et al., 2017) was proposed to learn to communicate with users based on one simple image, making the visual knowledge very limited for multi-turn dialogue session. order to enhance the dialogue quality by providing larger capacity and flexibility of visual information, re- cent works have considered employing multiple images and image searching processes to better align with the dialogue context. Even so, they are confined to retrieving images on coarse-grained dialogue concept (e.g., session-level) or leverage inaccurate visual knowledge searched from inad- equate image resources (Liang al., 2021; Shen et al., 2021). To sum up, current works have two main issues that may compromise the performance of multimodal dialogue. (1) Coarse-grained visual
1 Introduction
With the availability of large-scale datasets (Li et al., 2017: Dinan et al., 2018) and pre-trained language models (Radford et al., 2019; Raffel et al., 2020), dialogue generation develop rapidly in recent years. Conducting effective linguistic
"Corresponding Author

---

search corresponding images from the internet retrieve them from a large image corpus with over 150K images.' We further conduct a human eval- uation to quantify entity-image relevance. Images searched from the internet outperform and tie re- trieved ones in 52% and 12% cases respectively. Based on the above-mentiones two concepts of vi- sual knowledge, we take step forward and come up with novel framework to automatically con- struct multimodal dialogue data.
Ikebana
flower
Searching Internet (Ours)
flower Ikehana
To verify the efficiency of provided visual infor- mation, we present RESEE, a generative conver- sational framework powered by real-world visual experiences. Our framework follows the encoder- decoder paradigm with either shared or separate encoder-decoder setup. We handle multimodal dia- logue context by concatenating these information into the encoder, then the model generates plau- sible responses using its decoder. Three types of token embeddings are considered in the encoder module to sink in the knowledge from different modalities. To prove the effectiveness of RESEF, we further compare our dialogue model with sev- eral strong baselines, including four task-oriented pre-trained models and two similar multimodal di- alogue systems. RESEE outperforms most base- lines on both automatic and human evaluations. We also conduct comprehensive ablation experi- ments to demonstrate (1) the model performance gains brought by different visual knowledge, (2) the model performance with increased visual knowl- edge volumes, and (3) the relation between the proposed visual knowledge and the conventional document knowledge.
Retrieve from Flickr30k (VisAD)
Figure 2: Samples of entity images of Ikebana and flower from searching the internet v.s. retrieving from limited image-caption data. Images from the internet are more accurate and diverse compared to the counterpart.
knowledge: existing multimodal dialogues mostly follow the framework of image-grounded conversa- tion, which inherently provides insufficient visual knowledge (one image) and leaves lots of details unexploited for complete conversation. (2) Poten- tially inaceurate visual knowledge: though recent explorations come up with using fine-grained im- ages, they are limited in searching from small-scale image caption datasets (e.g., Shen et al. (2021) em- ploys Flickr300 30k (Young et al., 2014) for this process). These defects will introduce knowledge bias into the system (e.g., entity images retrieved from F1 Plickr30p may be wrong monotonous w.r.. given entities in Figure 2) and impair the conversational skills of dialogue agent.
Contributions. (1) We provide to construct multimodal dialogue data and two datasets based on it. A comparison between ours and other multimodal dialogue datasetss is Ta- ble 1. (2) We propose a simple yet effective multi- modal dialogue framework RESEE, which utilizes visual knowledge to generate informative and plau- sible responses. (3) Extensive experiments and promising results two constructed datasets jus- tify the effectiveness of our dialogue framework.
paradigm
To overcome the above two shortcomings, believe: (1) Compared with session-level visual knowledge, fine-graine fine-grained visual knowledge such as entity-level image is more competent to help models build a comprehensive understanding of on- going conversations. We thus propose to explicitly divide the visual standard of a dialogue session into turn-level and entity-level. (2) Instead of matching photos from existing image sets, search images on the internet for every entity obtain accurate and diverse visual representations accordingly. To justify the advantage of our approach in obtaining pictures with higher quality, we randomly sample 50 entities from existing dialogue data and either
Multimodal Dialogue Datasets
In this section, we introduce our framework for con- structing multimodal dialogue datasets. The overall
We employ COC02017 (Lin al., 2014) combined with E' c1ickr30k as an image pool in this process.
"The Cohen's 's kappa score (Cohen, 1960) is 0.493, indicat- ing two annotators reached a moderate agreement.

---

Datasets
#Dialogues #Utters
Domains
Auto Construct Avg, Img.
Img. Level
VisDial (Das al., 2017)
123K
2.4M
Image-based QAs
x
Turn
GuessWhat (De Vries al., 2017)
155K
L66
Image-based QAs
x
Turn
1GC (Mostafazadeh al., 2017)
4K
25K
Image-based QAs
X
Turn
MMD (Saha 2018)
150K
6M
Fashion Search
X
28
Entity Entity
MMCony (Liao al., 2021)
5.1K
39.7K
Conv. Search
X
22.32
RESEF-WON (Ours)
22.3K
100.4K
Know,-based Conv.
24.19
Tum&Fnnitty
RESEE-DD (Ours)
13.1K
49.2K
Daily Conv.
13.83
Tum&Ennity
Table I: Statistic and comparison of our ReSee datasets comparing existing multimodal dialogue datasets. "Avg. Img." is the averaged number of images per dialogue session and "Img. Level" is image granularity.
similarity between captions and dialogue turns for image retrieval. Using similarity from only the language domain helps us mitigate biases caused by using multimodal similarity measurement from various image domains (Liang et al., 2021).
Ouline Scarching...
THO reen nighe skydising and smiling Sky backgmund....
Qwant pixabay 257
Testual Similarily
For the image set to be searched, we group four image-caption datasets, i.e., COCO2017 (Lin et al., 2014), Flickr30k (Young et al., 2014), NoCaps (Agrawal et al., 2019) and Localized Narrati ves (LN) (Pont-Tused et al., 2020) with 826,539 image-caption pairs total. Then we use the following steps for turn-level image retrieval: (1) Turn Summarization: To avoid information discrepancy between dialog turns and image cap- tions arising from different sentence lengths. We first summarize the dialog turns into shorter ver- sion. (2) Texual Representation: To fully leverage caption descriptions of images, we use pre-trained sentence BERT (Reimers and Gurevych, 2019) get the textual representation of both summarized dialog turns and image captions. (3) Image Re- trieval: Finally, we employ processed textual rep- resentations of dialogue turns queries and repre- sentations of captions as keys to index the most relevant image every dialogue turn from the image-caption database. And we further present the percentage of turn-level images retrieved from each image-caption dataset Table 2.
Apaling parsotiating
Woold try day diving, Phe hoan Jydiving entire and dlways
Entiyy Extraction
Turn Summariration
Apprentice: would like diving. Have you done it?
Wizard:
I sure have! I've boen skydiiing my entire life, I always bring
strung purachute with me! Would you be interested in learning more about Parachuting?
Dislogus Turns
Figure 3: Data processing and construction of dataset RASBR-WOR using one example from WoW.
data flow for dataset construction is in Figure 3. A dialogue session should consist of two aspects of vi- sual information, namely the turn-level outline and entity-level details. We search for both visual con- cepts from either a very large image pool or the in- ternet. In detail, we construct multimodal datasets extended from Wizard of Wikipedia (MoM) (Dinan et al., 2018), a knowledge-groundee dialogue dataset, and the commonly used Daily Dialogue (DD) (Li et al., 2017).
2.1 Turn-level Visual Knowledge
2.2 Entity-level Visual Knowledge
One dialogue turn is a single exchange of conversa- tion between two speakers (e.g., a question and an answer). Intuitively, turn-level visual knowledge is helpful when there are more than one topic re- lated to a dialogue session with multiple turns, and the turn-level visual knowledge should be highly relevant to the current ongoing conversation turn. Since one complex dialogue is generally long and diverse, instead of being restricted to one spe- cific data domain, we gather a relatively large group image-caption data and propose to use sentence
The turn-level knowledge alone not competent to provide full visual details for long and knowl- edgeable conversations. We thus propose to use entity-level images to empower the dialogue agent with insights into details. Specifically, entity-level visual knowledge involves images of both nouns and named entities from every dialogue. We use the following steps for entity extraction and their corre- sponding images acquirement: (1) Named Entity: We use a pre-trained RoBERTa model (Liu et al., 2019, 9) to extract named entities in every dialogue

---

Shared Encoder Decoder
Decoder
Encoder
Decoder
Encoder
Separate Encoder Decoder
Segment
Position
Image Encoder
Word Encoder
Decoder
Tum-level Vinual
ISEPN
Eniit--ovvll Visual
(EPI)
(SEP2]
[EOS]
Knowledgg (T.V.)
Knowledge (E.V.)
Fniiiies (F.)
Dialag
Response (R.)
Figure 4: Model structure of the proposed multimodal dialogue system. We consider three types of embeddings (Token, Position, and Segmenty) as the model input. Our dialogue framework split into two paradigms, namely model with shared encoder-decoder (RESBE (SHAReF) and model with separate encodler-decoder (RESEE (SEP.)).
Dataset
Cap-Img Now (%)
DD (%)
RESEE Methodology
COCO2017 (2014)
123,287
2.67
2.99
We consider a simple approach to concatenate and to infuse multimodal information into plain dia- logue models. As shown in Figure 4, we apply this approach to two transformer models with shared or separate encoder-decoder for dialogue responding. Formally, we define our modeling task as: given the dialogue information (C, E, Vr, VE), where C is the dialogue context, E is the extracted entities from C, V, = (Vyl V22 :.,, Vp] is set of turn- level images from C and V = (VE. V2. V() is set of entity-level images from C. We aim to learn appropriate response R with given infor- mation by modeling p(R.| C, E, C,E,V, V,, Vg).
Flickr30k (2014)
31,014
2.67
2.32
NoCaps (2019)
4,500
0.05
0.02
LN2020 (2020)
671,469
94.61
94.67
Table 2:
Image-caption pairs in four existing datasets for turn-level image retrieval, and the percentage (%) of retrieved images from used datasets.
instance. (2) Regular Nouns: We then extract
all nouns from dialogues using the public toolkit Stanza (Qi et al., 2020). (3) Image Searching: Fi- nally, we use two online search engines² to search images for the entity-level visual knowledge. Since we leverage two searching engines i.e., Qwant, Pix- abay this process, we make sure that there is at least one valid image for every extracted entity.
3.1 Model Input
We employ different encoders for different modal- ity encoding. In concrete, we utilize transformer blocks (Vaswani et al., 2017) for word encoding, which projects word tokens to a continuous word embedding space. For image encoding, we utilize CLIP encoder (Radford et al., 2021) to capture the global information of picture and then use MLP functions to transform it into the same embedding space as the word. To distinguish different modality information and to identify dialogue contexts from responses, we employ three kinds of token-wise embeddings and sum them up as the input to our transformer- basedd dialogue systems, namely token embedding, position embedding, and segment embedding.
2.3 Overall Dataset
The proposed datasets are advantageous in compar- ing prior works by providing fine-grained and more accurate images related to the dialogue context. This is because (1) we explicitly split the visual knowledge into turn-level and entity-level; (2) large image pool as well online searching engines to acquire images. We additionally present examples and detailed statistics RESEF-WOW and RESEE-DD Appendix B. Note that, since turn- level information is conveyed through sentences, whose semantic information may not be fully cap- tured through conventional word matching, we did not employ online searching for turn-level images.
Token Embedding: The token embedding is the concatenation of VTw: VEw: Eg Cw, Rw, which denote the word embedding of turn-level and entity-
www.. qwant com and www. www.pixabay.. com

---

level visual knowledge, extracted entities, dialogue context and response respectively. We addition- ally add special token [SEP] between different modalities and content from distinet speakers in the dialogue. Note that, we separate response em- bedding R., from this concatenation for the model with separate encoder-decoder setting.
Inspired by recent progress in language genera- tive methods (Dong et al., 2019; Wang al., 2021), for both types of models, we process the encoder input with bi-directional attention, while giving the decoder output causal attention masks. This masking strategy makes sure our models fully un- derstand dialogue contexts and autoregressively generate tokens with learned knowledge.
Position Embedding: Since the transformer model itself cannot learn the token position, we employ position embedding to encode signals of the token order in the input sequence.
3.1.2 Response Generation
For the separate encoder-dccoder model, we feed multimodal information X the model encoder and autoregressively generate responses from the decoder. As for the shared encoder-decoder model, we first encode X with a special token [BOS] J behind it. Then, the model starts to generate by ap- pending a [MASK) token to the input and samples word from the predicted distribution over vocab- ulary. The [MASK] token is then replaced by the generated token and new [MASK] is appended to the input sequence for next word prediction. Both generation processes terminate when the model predicts [EOS] token or reaches the max length.
Segment Embedding: Segment embedding is em- ployed to differentiate which segment (turn-level entity-level visual knowledge, textual entities, dialogue context or response) the token is in.
3.1.1 Model Training
Separate Encoder-Decoder Model (RESEE (SEP.)): Dialogue model with separate encoder de- coder employs different sets of model parameters for context understanding and response generation respectively. We apply cross-attention (Vaswani et al., 2017) between the encoder output and the de- coder input to bridge the gap between multimodal dialogue context learning and response generation. We first initialize it with T5 (2020) parameters. For the training objective, the model is optimized to recover the response R with the given multimodal knowledge X [Vr, VE, E, Cl:
Experimental Setup
4.1 Evaluation Metrics
Automatic Metrics. We employ automatic met- rics to assess the model performance: (1) Flu- ency: perplexity (PPL) measures the confidence of the generated responses; (2) moken-based Rele- vance: BLEU (Papineni al., 2002) and Rouge- L (Lin, 2004); Embedding- Relevance: (Ser- ban al., 2017): Embedding Average cosine sim- ilarity (Avg.), Vector Extrema cosine similarity (Ext.), and Embedding Greedy Matching score (Gre.). (3) Diversity: Distinct-1 (Dist-1) and Distinct-2 (Dist-2) (Li et al., 2016) measure the number of distinct uni-grams and bi-grams divided by the total grams.
SSop (R
Σ WVR
log (wx X).
Shared Encoder-Decoder Model (RESEE (SHARE)):: Dialogue model with shared encoder decoder integrates the understanding and gener- ation process with the same set of parameters. We take masked response prediction main training task to make the model aware of appropriate responses with multimodal dialogue context. In detail, we first initialize with UNILM (2019). During training, 70% the responses are replaced by a special token [MASK] or another token in the vocabulary. The masked response is denoted as R. In detail, we use the unmasked dialogue information X, RR predict R:
the
Human Evaluation. We perform human evalu- ation over the generated responses. We consider three conventional criteria: fluency (Flue.), infor- mativeness (Info.), and relevance (Rely.) following Song et al. (2021). Also, we consider Sensibleness and Specificity Average (SSA) metric (Adiwardana et al., 2020), evaluating whether response makes sense and is specific. We strictly obey double- blind procedure, where the annotators know noth-
Lshare ( - Σ log (w:) X, R
w.c
All these metrics are calculated by first excluding sen- tence punctuation and then running the public NLG evaluation script from gi sithub. ccm uuba quuba/nl inlg-vval
Besides, we also follow Liang et al. (2021) I) to con- sider entity knowledge bias when decoding.

---

Model
PPL
BLEU f Rouge-L 1
Avg. T
Ext.
Gre.
Dist-1
Dist-2 t
RESEE-WOW
DIALOGPT (Zhang et al.. 2020)
13.38
0.0987
0.1467
0.805
0.423
0.705
0.163
0.403
GPT-2 (Radford et al., 2019)
9.64
0.1296
0.1345
0.793
0.422
0.677
0.151
0.371
UNILM (Dong al., 2019)
18.86
0,1088
0,1215
0.770
0.373
0.582
0.125
0.335
T5 (Raffel al., 2020)
18.35
0,1361
0,1411
0.830
0.412
0.620
0.160
0.392
MSDP (Liuu et al., 2022)
0.1154
0.1905
RESEE (SHARE)
17.94
0.1193
0,1255
0.790
0.369
0.595
0.124
0.354
RESEE (SHARE) E.
18.36
0.1183
0,1192
0.783
0.370
0.585
0.119
0,332
RESEE (SHARE) E.V.
17.39
0,1167
0,1253
0,768
0.366
0.587
0.095
0.250
RESEE (SHARE) B. T.V.
18.54
0.1148
0,1238
0.778
0,376
0.593
0.099
0.269
RESEE (SHARE) E. B.V.
18.05
0.1143
0,1230
0.793
0.377
0.590
0.144
0.393
RESEE (SBR)
17.46
0,1508
0.1599
0.844
0.426
0.632
0.162
0.416
RESEE (SEP.) -2
17.47
0,1426
0.1509
0.837
0.422
0.627
0.166
0.430
RESEE (SEP.) -...
17.58
0,1479
0,1521
0.841
0.422
0.629
0.171
0.440
RESEE (SEP.) -2. T.V.
17.58
0,1460
0,1480
0.839
0.421
0.628
0.168
0.429
RESEE (SEP.) E.V.
17.52
0.1370
0,1456
0.833
0.420
0.626
0.161
0.410
RESEEDDD
DIALOGPT (Zhang al., 2020)
5.95
0.1132
0.1345
0.734
0.467
0.658
0.186
0.466
GPT-2 (Radford al., 2019)
5.83
0.1183
0.1519
0.692
0.434
0.657
0.134
0.520
UNILM (Dong al., 2019)
6.57
0.0871
0.1031
0.723
0.424
0.565
0.115
0,333
TS (Raffel al., 2020)
8.11
0.1102
0.1392
0.729
0.469
0.578
0.183
0.503
VISAD (Shen al., 2021)
17.81
0.1247
0.642
0.451
0.526
0.097
0.332
RESEE (SHARE)
8.94
0.1111
0.1497
0.773
0.446
0.607
0.164
0.446
RESEE (SHARE)
9.51
0.0991
0.1444
0.756
0.451
0.600
0.168
0.459
RESEE (SHARE) V.V.
10.10
0.0924
0.1301
0.748
0.434
0.593
0.126
0.354
RESEE (SHARE) -.. T.V.
9.53
0.0873
0.1276
0.753
0.444
0.593
0.165
0.437
RESEE (SHARE)-
10.02
0.0919
0.1332
0.749
0.443
0.594
0.165
0.447
RESEE (SRP)
7.39
0.1215
0,1582
0.786
0.475
0.628
0.196
0.535
RESEE (SEP.) -E
7.50
0.1204
0.1567
0.786
0.,71
0.625
0.205
0.544
RESEE (SBP.) E.
7.31
0.1197
0.1564
0.783
0.474
0.624
0.206
0.560
RESEE (SBP.) T.V.
7.66
0.1174
0.1523
0,767
0.469
0.618
0.193
0.528
RESEE (SBP.) V.V.
7.47
0.1159
0.1548
0.733
0.467
0.616
0.184
0.500
Table Results of the proposed dialogue models RESEE-WOR WoW (test unseen set) and RESEE-DD (test set). Models without textual entity, turn-level entity-level visual knowledge the inputs appended with * "."," "- T.V.V. and E. V. respectively. Our dialogue framework RESEE with shared and separate encoder-decoder are appended with (SHArE) and (SEP.) respectively. We mark the best result with bold face and the second best with underline.
ing about the models. We sample 100 instances across each model for human evaluation."
which considers words extracted from dialogue context and their corresponding images into gen- eration. Note that, RESEE (SHARE) is similar to MARIA (Liang et al., 2021), which considers simi- lar training paradigm. However, MARIA takes only one image per dialogue session, thus consider our RESEE (SHARE) as an extension of MARIA. See Appendix A.2, C for more model details.
4.2 Baselines
To verify the advantages of the proposed frame- work in dataset construction and multimodal dialogue generation, we take competitive Di- ALOGPT (Zhang et al., 2020), GPT-2 (Radford et al., 2019), UNILM (Dong et al., 2019) and T5 (Raffel et al., 2020) as traditional dialogue baselines, all of which consist of 24 transformer layers. On Wow dataset, we additionally consider recent method: MSDP (Liu et al., 2022), dialogue model that leverages prompt tuning, multi-stage refinement with the GPT-2. On DD dataset, we incorporate a strong multimodal dialogue system VISAD (Shen et al.. 2021).
5 Results and Analysis
Main Results. We present evaluation results of models with separate or shared encoder-decoder two datasets in Table 3. (1) Our model with separate encoder-decoder (RESEE (SEP.)) per- forms better than the model with shared encoder- decoder (RESEF (SHARF)). This may be explained as models with separate encoder-decoder explicitly divide the understanding process of multimodal in-
Detaais of human annotutors are listed in Appendix E.

---

Model
PPL 1
BLEU f
Rouge=L f
Avg. 1
Ext.
Gre.
Dist-1 f
Dist-2 t
RESEE (SEP.)
7.39
0.1215
0.1582
0.786
0.475
0.628
0.196
0.535
RESEE (SER.) E.
7.40
0.1216
0.1345
0.787
0.473
0.624
0.196
0.534
RESEE (SER.) V.
7.62
0.1220
0.1603
0.787
0,472
0.624
0.200
0.536
RESEE (SEP.) 4 V..V.
7.75
0.1246
0.1610
0.790
0.475
0.626
0.208
0.555
RESEE (SEP.) 5 5E.V.
7.81
0.1230
0.1502
0.785
0.469
0.622
0.211
0.565
Table 4: Model performance with varied image number per entity during training ("n R. ") over RESBEEDD.
Model
Flue. 1 Info. t Relv. t SSA
ble 5, which further justify the outcomes and find- ings from automatic metrics above.
RESEE-WoW
GPT-2 (2019)
3,495
3.140
2.033
47.50%
5.1 Visual Knowledge
DIALOGPT (2020)
3,830
3.660
2.260
59.25%
UNILM (2019)
3.745
3.113
1.800
41.17%
We conduct extensive ablation experiments over variants of the input information better understand their respective roles the dialogue generation task. (1) The performance improvement on our model benefits from both aspects of visual knowledge in providing external information. (2) Fine-grained visual information (i.e., entity-level), plays more important role in improving the generation performance than turn-level visual knowledge, which explains the necessity to find and utilize fine-graid tne-ggriied visual clues. (3) Turn-level images also prompt model performance (i.e., ". E. v.s. " E. -..V.") which is consistent with findings from the traditional visual dialogue. (4) However, textual entities bring more performance gain comparing entity- visual knowledge. We ascribe this to the model pre-trane pre-training stage that is originally on the language domain, which makes it harder for dialogue models to understand visual in- formation than to acquire knowledge from texts. (5) Introducing visual knowledge improves the quality of generated responses, but generally degenerates the diversity. This is attributed to the constraints brought by fine-grained visual inputs. These inputs enlighten the model with explicit visual clues, making it compelling to specific knowledge but leading tolerable sacrifice of text diversity.
T5 (2020)
3,920
3.495
2.237
57.50%
RESEE (SHARE)
3,900
3.143
1.870
46.00%
RESEE (SBP.)
3,930
3.650
2.267
65.75%
RESEE-DD
GPT-2 (2019)
3,916
3,237
2.760
50.67%
DIALOGPT (2020)
4.023
3.547
2.727
56%17%
UNILM (2019)
4,168
3.340
2.585
47.25%
T5 (2020)
4.153
3.567
2.713
52.50%
RESEE (SHARB)
4.187
3.460
2.587
48.50% 50%
RESEE (SEP.)
4.190
3.690
2.833
58. 58.50%
Table 5: Human evaluation results.
formation and the generation of textual responses using different model parameters. This makes the model devote more to each learning phase. (2) On both constructed datasets, RESEE (SEP.) with full visual knowledge achieves the best or com- petitive performance terms of relevance metrics Le., BLEU, Rouge-L, even comparing models with task-oriented pre-training (DIALOGPT) or external document knowledge (MSDP). This observation demonstrates the effectiveness of our model lever- aging representations from both text and vision. (3) When considering embedding-bamee based metrics, method is better than baselines in Avg. and Ext., but is slightly inferior two GPT models in Gre.. That is to say, though RESEE may not reach the similarity upper bound compared to pre-trained GPTs, it is still advantageous in the averaged sen- tence similarity comparing strong baselines.
5.2 Multiple Entity-level Images per Entity
Since we provide a one-to-many mapping between entities in the dialogue context and their corre- sponding images, we conduct experiments with varied numbers of entity-level images as input. In Table 4, (1) increasing the number of entity-level images can further boost the dialogue model per- formance by generating more relevant responses. We ascribe this to a larger information capacity
We also observe that finetuned GPT-2 and Dr- AT.OGPT perform better than our method in PPL on both datasets. This is attributed to their pre- training stage which dedicates in directly optimiz- ing model generation ability. However, our model can achieve better diversity compared with base- lines, especially our model variants without textual entity input and/or entity-level visual knowledge.
"The average Fleiss's 's kappas (Fleiss and Cohen, 1973) of RESEF-WOR and RESER-DD are 01.199 and 0.492 respectively, indicating three annotators reached a moderate agreement.
We also present human evaluation results Ta-

---

Model
PPL 1
BLEUT f
Rouge-L 1
Avg. 1
Ext.
Gre.
Dist-1
Dist-2 t
T5 (Raffel al., 2020)
18.35
0.1361
0.1411
0.830
0,412
0.620
0.160
0.392
IS w/ Know.
17.66
0.1411
0.1488
0.839
0.423
0.631
0.157
0.403
RESEE (SER.)
17.46
0.1508
0.1599
0.844
0.426
0.632
0.162
0.416
RESEE (SEP.) w/ Know.
17.68
0.1528
0.1561
0.847
0.429
0.631
0.174
0.464
E.
17.53
0,1463
0,1582
0.840
0,420
0.628
0.161
0.426
F. T.V.
17.76
0,1443
0,1485
0.839
0.424
0.629
0.168
0.427
-E. E..V.
17.56
0,1419
0.1445
0.836
0.419
0.627
0.157
0.399
Table Model performance with external document knowledge w// Know ") on RESEEE-WWW (test unseen set).
provided by extra visual knowledge. (2) However, giving too many entity-level images can be a show- stopper for the model, i.., the model with 5 images per entity generally performs worse. This might be attributed to the plain multimodal infusion method considered, where the model may confuse different images that belong the same or another entity, (3) More entity-level images jeopardize the model's output confidence with lower PPL. yet make gener- ated responses more diverse with consistently more distinct n-grams (i.e., higher Dist-1/2).
Contexts:
know what you mean. ought to wear right clothes the right time.
You got it. Only in this way can you gain the respect of the interviewer and confidence your judgement.
Turu-tevel Visual Knowledge
Entity-level Visual Knowledge (Entity)
(clothes)
(interviewer)
(udgement)
5.3 External Document Knowledge
Reference: not be true, but the first and lasting impression of you determined by your clothes and behavior.
Is the visual knowledge complement of existing textual knowledge? To answer it, we conduct ex- periments over RESER-WOW with provided topic passages appended to the input. Table 6, we observe that (1) our visual knowledge can fur- ther boost model performance even with document knowledge, demonstrating the evidence provided by visual knowledge is complementary to existing textual knowledge. But the performance gain of adding documents the visual models is not as sig- nificant as models without visual knowledge (T5). This indicates that there exist certain intersections between information provided by two modalities. (2) Bringing document knowledge to the model greatly improves diversity. Because abundant tex- tual information helps models understand dialogues comprchensively and generate responses diversely.
Generated Responses:
the job.
DIALOGPT: think think I'll wear suit and tic.
UNILM l'lll try to make sure that you don't lose your interviewers. TS
:1 agree with you.
RESEE (SHARE): Yes, illl try best to make good impression on the
interviewers
RESER (SEP-) think should be comfortable in your clothes make good impression on the interviewer.
An example of responses generated by our models and baselines. Highlighted words overlap enti- ties in the dialogue context the response reference.
Figure 5:
models can produce words that pay more attention to "interviewer" and "clothes". which are missing in the unimodal counterparts. These demonstrate that datasets provide useful visual information, which the proposed multimodal dialogue system captures and subsequently leverages to generate better responses that are relevant to the reference. Please refer to Appendix for more examples.
5.4 Case Analysis
We exhibit an example of generated responses in Figure 5. As this conversation is talking about the importance of dressing code in interviews, our dataset provides one turn-level image showing professional person with suit and tie as well as three entities and their corresponding images. Com- pared with models without visual enhancement, our two models focus more on the provided visual con- texts and generate responses that are highly relevant to dialogues and the reference. For example, our
Related Works
Visual Dialogue Dataset. Images serve dif- ferent purposes in a dialogue. Visual dialog (or vi- sual question answering, VQA) is task answer questions about the factual contents of the image in a multi-turn manner. VisDial (Das et al., 2017) was constructed of one image and about 10 indepen-

---

dent question-answer pairs grounded on the given image. De Vries et al. (2017) introduced image grounded QA dataset with pixel-Ievel object loca- tion of the image. IGC (Mostafazadeh et al., 2017) was constructed based on Twitter conversations with (image, description, samples. In visual-enhanced conversational rec- ommendation, MMD (Saha et al., 2018) was multi- modal dataset under shopping situation and aimed at providing applicable recommendations based on textual conversations as well as images of poten- tial shopping items. MMConv (Liao et al., 2021) was applied in tourism scenarios across 5 real sit- uations, it also provided knowledge base and a photo gallery about recommended items. Recently, MMDialog (Feng et al., 2022) was proposed with massive multimodal open-domain conversations and associated images derived from social media. IMAD (Viktor and Denis, 2023) was constructed using massive amount of dialogues, with the last utterance to be replaced with collected images.
et al., 2020), LLAMA (Touvron et al., 2023a.b). more and more works start incorporating LLMs as their text generative framework and get excep- tional performance in the open-domain dialogue tasks (Zhu et al., 2023a; Liu et al., 2023; Ye et al., 2023; Dai et al., 2023).
triplet
7 Conclusion
In this paper, we present a paradigm for multimodal dialogue construction with two novel datasets and a multimodal dialogue responding framework RE- SEE. We explicitly separate the visual knowledge into two aspects, using online searching or retriev- ing from large image corpora to construct accurate and diverse visual knowledge. Transformer-bie based dialogue models with shared and separate encoder- decoder verify that provided visual knowledge pro- motes model capacity. Further, we explore feeding multiple entity-level images and external document knowledge into models. By providing fine-grained visual knowledge on dialogues, we demonstrate dialogue models can substantially achieve better performance across different setups and domains.
Open-domain Dialogue. Open-domain dialogue models aim at responding to general human- conversations various circumstances. While di- alogue generation has a rich history, the area has made significant progress with the rising of pre- trained models in varied linguistic domains (Zhang et al., 2020; Mi et al., 2022; Zhu et al., 2023b; Tou- vron et al., 2023b). The introduction of external knowledge in traditional models plays vital role in leading them intellectual dialogue agents. For example, Wu et al. (2021) leveraged three domains of knowledge to enhance the model performance in Chinese contexts. Wang et al. (2022) employed an extra retrieval process to find knowledgeable evidence as input to enlarge dialogue model capac- ities. Recent works focus on efficient knowledge integration like retrieval-free approaches (Wang al., 2023a) and few-shot prompting (Wang et al., 2023b). Moreover, visual knowledge has also been recently considered to boost the performance of dialogue models. Multi-Modal BLENNEER (Shuster et al., 2021) was pre-trained on large-scale visual question-answer datasets for image- grggedeeo con- versation. Liang et al. (2021) introduced a method to allocate conversations with a picture exter- nal knowledge. Shen et al. (2021) extended the visual augmentation the token-level, providing versatile visual information to the model. Most recently, as the emergence and wide spread of large language models (LLMss, such as GPT-3 (Brown
A Acknowledge
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFC3303301 and in part by the National Natural Science Foundation of China un- der Grant 6230071708 and Grant 62172053. The authors would like to thank Qiyu Wu, Haoyue Dai, and Kangwei Liu for their insightful discussions and contributing the human evaluation process.
Limitations
(1) The provided datasets are auto-constructed, meaning visual biases brought by online search- ing are inevitable. We plan to take next step make the dataset more accurate and to include more visual knowledge (e.g., visual knowledge from ex- ternal document knowledge in WoW) in our mul- timodal dialogues. (2) For now, we did not con- sider a one-to-one mapping between the textual entity and entity images in the model input, more sophisticated relations can also be introduced for better modal interaction and modeling.
(3) Our framework offers novel way to enhance text-only dialogue system performance by adding extra infor- mation from a multimodal perspective. However, this comes at the cost of extra computational over- head brought by learning visual knowledge.

---

9 Ethics Statement
Amanda Cereas Curry and Verena Rieser. 2018. # metoo alexa: How conversational systems respond to sexual harassment. In Proceedings of the second acl workshop on ethics natural language processing, pages 7-14.
We are aware that automatic dialogue generation may create deceptive, harmful, or objectionable content due to their internal biases (Curry and Rieser, 2018; Gchman al., 2020). These biases are usually inherited from the training data itself. We observe that since our dataset construction is totally based on existing text-only dialogues, our RESEE framework can be used to mitigate those bi- ases easily. For instance, one of our future work di- rections is to employ the proposed multimodal data collection method on detoxification dialogues (e.g.,
Wenliang Dai, Junman Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint 0rXiv:2305.06500
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh. Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE conference on computer vision and patter recognition, pages 326-335.
The Moral Integrity Corpus (Ziems et al., 2022)) for building safer and better dialogue agents.
We are well aware that the online searching pro- cess of entity-level images may cause biases (e-g., gender, race) in our constructed dataset. To mit- igate the bias, we collect multiple images on the internet for one entity in dialogues (see Appendix B for statistical details of our datasets), so that the model can choose more than one specific image during model training. For licenses of images, other employed dialogue data, and the constructed datasets that are about to be released, please refer to Appendix A. 1 for more details.
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi- modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 5503-5512.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston, 2018. Wizard of wikipedia: Knowledge-powent conversational agents. arXiv preprint 1..0011
Li Dong, Nan Yang. Wenhui Wang, Furu Wei. Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. 2019. Unified language model pre- pra-training for natural language understanding and generation. Advances in Neural Information Process- ing Systems, 32.
References
Daniel Adiwardana, Minh-Thang Luong, David R So,
Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, al. 2020. Towards a human-like open-domain chat- bot, arxiv preprint arXiv:2001. 2001:2001.09911 099977.
Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao. Dongyan Zhao, and Qing- wei Lin. 2022. Mmdialog: A large-scale multi- turn dialogue dataset towards multi-modal open-domain conversation. arXiv preprint arXiv:22 2ttp:2211.001111 1.05719.
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. No- caps: Novel object captioning at scale. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 9948-8957.
Joseph L Fleiss and Jacob Cohen. 1973, The equiva- lence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-61)
Samuel Gehman, Suchin Gururangan. Maarten Sap. Yejin Choi, and Noah A Smith. 2020. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lap- ata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020. Experience grounds language. arXiv preprint arXiv: 10151.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Stevan Hamadd, 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-366
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.696
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educattionll and psychological mea- sarement, 20 1):37446.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart:

---

Denoising sequence-to -equuenee pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-
Fei Yitong Li, Yulong Zeng. Jingyan Zhou, Yasheng Wang, Chuanfei Xu, Lifeng Shang. Xin Jiang, Shiqi Zhao, and Qun Liu. 2022. Pangubot: Efficient gener- ative dialogue pre-training from pre-trained language model. arXiv preprint arXiv:2203. 17090.
Jiwei Li. Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promootng promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computu- tional L Linguistics: Human Language Technologies, pages 110-119.
Nasrin Mostafazadeh, Chris Brockett, William B Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende. 2017. image-grounded con- versations: Multimodal context for natural ques- tion and response generation. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing ( (Volume 1: Long Papers). pages 462-472.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume 1: Long Papers). pages 986-99 986-995.
Kishore Papineni. Salim Roukos. Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meefing of the Association for Computa- tional Linguistics, pages 31 311-318.
Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xi- ubo Geng, Yining Chen, Fan Liang, and Daxin Jiang. 2021. Maria: A visual experience powered conver- sational agent. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1tth International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 5596-5611.
Jordi Pont- Tusee, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In European conference on computer vision, pages 647- 664. Springer.
Peng Qi. Yuhao Zhang. Yuhui Zhang. Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations.
Lizi Liao, Le Hong Long, Zheng Zhang, Minlie Huang. and Tat-Seng Chua. 2021. Mmconv: an environment for multimodal conversational search across multiple domains. Proceedings of the 44th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages $75-684.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark. et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European confer- ence on computer vision, pages 740-755. Springer.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tniigg.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1692.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese nett-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shocybi, and Bryan Catanzaro. 2022. Multi-stage prompting for knowledgeable dialogue generation. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 1317-1337.
Amrita Saha, Mitesh Khapra, and Karthik Sankara- narayaman. 2018 Towards building large scale multi- modal domain- conversation systems. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 32.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.0510..

---

Iulian Serban, Alessandro Sordoni, Ryan Lowe, Lau- rent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Yihe Wang. Yitong Li, Yasheng Wang. Fei Mi, Pingyi Zhou, Xin Wang, Jin Liu, Qun Liu, and Xin Jiang. 2022. Pan more gold from the sand: Refining open- domain dialogue training with noisy self-retrieval generation. arXiv preprint arXiv22201.13300
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision. In International Conference on Learning Representations.
Lei Shen, Haolan Zhan, Xin Shen, Yonghao Song. and Xiaofang Zhao. 2021. Text is not enough: Integrating visual impressions into open open-domale domain dialogue gener- ation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4287-4296.
Sixing Wu, Ying Li, Minghui Wang. Dawei Zhang, Yang Zhou, and Zhonghai Wu. 2021. More bet- ter: Enhancing open-domain dialogue generation via multi-source heterogeneous knowledge. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2286-2300.
Kurt Shuster, Eric Michael Smith, Da Ju, and Jason Weston. 2021. Multi-modal open-domain dialogue. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 4863-4883.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou. Junyang Wang, An- wen Hu, Pengcheng Shi, Yaya Shi, mplug-owl: Modularization empowers large lan- guage models with multimodality. arXiv preprint urXiv:2304. 14178.
Haoyu Song, Yan Wang. Kaiyan Zhang, Wei-Nan Zhang. and Ting Liu. 2021. Bob: Bert over bert for training persona-based dialogue models from limited personalized data. arXiv preprint arXiv:2106 06169.
2023,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anry Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metries for semantic in- ference over event descriptions. Transactions of the Association for Compatational Linguistics, 2:67-78.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large- scale generative pre- pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics:
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
System Demonstrations, pages 270-278.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need, Advances in neural information proceemssiipppocssiiig systems, 30.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023a. Minigpt-4: Fahhaccing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592.
Qi Zhu, Fei Mi, Zheng Zhang, Yasheng Wang. Yitong Li, Xin Jiang, Qun Liu, Xiaoyan Zhu, and Minlie Huang. 2023b. Kpt: keyword-guicee for grounded dialog generation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pages 14065-14073.
Moskvoretskii Viktor and Kuznetsov Denis. 2023. Imad: Image-augmented multi-modal dialogue. arXiv preprint 2305.:335..0111
pre-training
Rui Wang, Jianzhu Bao, Fei Mi, Yi Chen, Hongru Wang, Yasheng Wang, Yitong Li. Lifeng Shang, Kam-Fai Wong, and Ruifeng Xu. 2023a. Retrieval-free knowl- edge injection through multi-document traversal for dialogue models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6608-661
Caleb Ziems, Jane Yu, Yi-Chia Wang. Alon Halevy, and Diyi Yang. 2022. The moral integrity corpus: A benchmark for ethical dialogue systems. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). pages 3755-3773
Yihe Wang, Yitong Li, Yasheng Wang, Fei Mi, Pingyi Zhou, Jin Liu, Xin Jiang, and Qun Liu. 2023b. His- tory, present and future: Enhancing dialogue genera- tion with few-shot history-future prompt. In ICASSP 2023-2023 IEEE International Conference Acous- fies, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.

---

mizer (Loshchilov and Hutter, 2017) with the learn- ing rale linearly increasing from 0 to 0.005 for the first 20% training steps, then linearly decreasing to trained BART (Lewis et al., 2020) model to sum- marize the dialogue turns. After we have access representations both dialogues and captions encoded by sentence BERT, we employ FAISS? for indexing speedup. As for entity-level image online searching, we use Qwant and Pixabay" to search at least one valid image for every extracted entity, As for licences of images we employed in our datasets, Pixabay images are all royalty-free -free. Images from Qwant follow one of five protocols for reproduction, sharing and modification: Public do- main; Non-commercial reproduction and sharing; Reproduction and sharing; commmmmmeraaal repro- duction, sharing and modification; Reproduction, sharing and modification. And our datasets will be released under Non-commercial reproduction and sharing license to ensure proper usage.
A Implementation Details
A.1 Dataset Construction
For turn-level image retrieval, we employ pre-
O. We train the model until has no progress on validation set (valid unseen set for RESEF-WOW). All experiments are conducted on two NVIDIA TITAN GPUs with 24G memory in total, it takes around 12 hours for RESLE-WOW training and 7 hours RESEE-DD,
B Dataset Details
B.1 Dataset Statisties
First of all, for two text-only datasets we em- ployed, Wow dataset is under an MIT License, and is publicly available at https://parl.a. pro dataset is licensed under CC BY-NC-SA 4.0, and the dataset can be obtained from http: We present detailed dialogue dataset information, including unique turn-level image number, unique entity- level image amount, turn and entity level images averaged on a dialogue session and average number of images that belong to one entity in Table 7. We also show the relationship between entity number per dialogue session and dialogue session number in Figure 6, the data distribution of how many ex- amples are there for each (n entity-level image, turn-level image) setting in Figure 7. From these four distribution figures, we can tell that the Ru- SEE-NOW dataset has more concentrated turn-level image number and entity-level image number pairs. while the range of entity-level image number of RESEL-DD is wider.
DD
A.2 Dialogue Models
We initialize parameters of RESEE (SEP.) and RE- SEE (SHare) using T5 (Raffel al., 2020) and UNILM (Dong et al., 2019) respectively. Note that, we only add the segment embedding to the shared encoder-decoder model to separate their respect in- puts. On the RESLE-WOW dataset, we truncate the context input (i.e., dialogue context, entities and visual knowledge) to fixed length of 190, and the response to 35. We exclude the most frequent and uncommon nouns (words that appears less than times and more than 100 times) to accelerate model training. The cleaned nouns in RESEE-NOW takes around 68% of the original extracted words. We make sure that for every training data, the entity- level visual knowledge as well as the entity input is no more than 8 and the turn-level image is no more than 5. To make the model fully understand knowledgeable conversations in RESEE-WOW, we split every dialogue session into smaller conversa- tional chunks with maximum of 2 turns for training. For RESEE-DD dataset, the encoder input was set to 185 with 35 to be the response. Every training data has no more than 6 entity-level images and 5 turn-level images. Also, we reduce the entity- level to around 80% of the original entity-level image accelerate training. We use AdamW opti-
B.2 Multimodal Examples
We present sampled examples from our constructed datasets RESLE-WOW and RESEE-DD in Figure 8, From these examples, we can clearly tell the visual enhancement for dialogue understanding from both knowing named entities and enlarging impressions of regular nouns. For instance, the noun Ikebana is proper noun in the dialogue, the model would never know what it looks like from just reading the dialogue contexts. However, the entity-level image provides the model with a straightforward approach to access related visual knowledge. Another exam- ple shows that images corresponding to abstract nouns such as love can provide an ambiance of ro- mance for models, which may strengthen model's 's
'ithhub. womm//acccborreercttttaa.... www. www.gwaattccmm com p1 www.xaay.coo

---

RESEB-ROR
RESEE-DD
Dataset
train
valid (seen/unseen)
test (seen/unseen)
train
valid
test
Dialog Session
18,430
981/967
963/968
11.118
1,000
1,000
Turn-level Image
46,319
8,896/7 ,944
8,851/6,705
32,399
7,966
7,626
Entity (Image)
14,618
3,699/2,862
3,748/2,762
6,204
2,298
Avg. Turn Image
4.52/4.53
4.49/4.51
3.85
Avg. Ent. Image
19.87
18.97/18.53
19.00/18.0m 8.66
9.96
10.16
10.14
Max. Ent. Image
44/47
50/48
76
Min. Ent. Image
5/4
5/5
0
Table 7: Statistics of two constructed multi-modal dialogue datasets. We present unique entity-level image count well as unique image count of the most similar image on turn-level visual data retrieval. The average, maximum and minimum number of images hased one dialogue session. We also present the average number of searched valid pictures for every entity at the last row.
1400
1000
Senvion Number
Session
600
200
0
Entity Number per Session
Entty Number per Session
(a) RESEF-WON
(b) RESEE-DD
Figure 6: Data distribution of entities of dialogue session datasets. The X axis represents entity number, while the Y axis represents dialogue session number.
Entty-eveel Image per Session
64
56
10a
100
Entoyleevel mage por
48
25-
40
20
15-
10a
10
5
0
10
10a
10
12 Turn-level Image Session
100
14
16
Turn-level Image per Session
(a) RESEE-WON
(b) RESEE-DD
Figure 7: Distribution of turn-level image and entity-level image numbers of two datasets. We use logarithm function to normalize the number of samples with varied turn-level and entity-level images and indicate their values using color bar.
understanding of dialogue histories and further as- sist it to produce high-quality responses.
from Huggingface for G0 and DIALOGPT!!
model. For two models, we used their 24-layer ver- sion to make fair comparisons with rest methods.
Baseline Details
whttpe//augiingieeecco// 2tpp://taaccccpppaa/appppa -endiu.. hhttps://huggingg aagecec co/m crooott/ DialoGPT-mediis
We present the implementation details of sev- eral baselines. We took the pre-trained weights

---

THEORY
Contexts:
The best snother 30 dollars off. That's definitely the lowest go lawves gap dallars.
Let's each other half awwf again and split the difference I chink price we can
A:Ckebana), it's kitt fowwrr and it's quite different from Western style arrangetieet because in decörate one flower, only with
Ikehana's (theory one flower
Turn-level Visual Knewledge
B: What's s another Japaneelaaa
Entity-level
Visual Knowledge
Opli)
(prixe
(Eatity)
Reference: we can meet hatffwwy again.
Generated Responses:
(a) Multimodal DD (RESEB-DD)
GFT-2:
I'm sorry, but I can't do it. I'm afhaid can't make it. lhink
DIALOGPT:
T5:
I'm afraid we can't afford to that anymore.
again lowlymway
happets.
Figure 9: Additional example for RESEE and baselines.
Apprentioe:[lamburgee the best.for dinner because's it is versatile.
Wizurd: I partially agree with that. But Jove tuste) of cooked patties of ground meat placed inside sliced bread(rol]arr bun)
Contexts:
Well, we've had cut down the amount of solvents we use and we 're allowed lead products
What happens you exceed the limito ?
Turn-level
Visual Knowledge
(b) Multimodal WoW (RESEE-WON)
Entiry-level
Visual Kaowledge (Entity)
(solvent)
(lead)
(product)
Figure 8: Dataset sample one dialogue turn on our multimodal datasets. Pictures pointed by dashed lines are entity-level images, while the one pointed by solid line turn-level image for one instance.
(lindts)
Reference: try not you might to pay quite henvy
fine.
Responses:
GPT-2
We have to be curofal about what we use.
We used Adam (Kingma and Ba, 2014) optimizer with learning rate increases from 0 to 0.001 for the first 20% iterations for both GPT-2 and Di- ALOGPT. We truncate input data to a fixed length 250 and make sure that the length of every gen- erated response is no more than 30. We train two models on two datasets until they have no progress on validate sets, which takes around epochs. All baselines are trained on the same machine as RE- SEE with two NVIDIA TITAN GPUs.
DIALOGPT:
We have lot lead.
TS:
I'm affid we oun't afffndd to do that anymore.
RESEE (SHARE): You'll pay moet
fine, and we will be fined 20 each
(SEP.):
Figure 10: Additional example RESEE and base- lines.
models in Figure 10, and 11. From these quali- tative results, we can draw the conclusion that our RESEF method can better understand given dia- logue contexts with enhanced visual knowledge, hence, generating responses with higher quality.
D Additional Qualitative Results
We also present more generated examples of our RESEE models as well as several baseline dialogue

---

Informativeness:
Sorry, but this is the bost price You can "I get any chesper than here.
1. The system' 's result is dull, repetitive, and does not have new information.
pay travlerr'
Turn-level
2. Choose this score when you hesitant be- tween score and score 3.
Visual Knowledge
3. The system's 's result contains some new infor- mation and it displays certain level of diver- sity.
(pay)
Entity-level Visual Knowledge
(prize)
(Entity)
4. Choose this score when you hesitant be- tween score and score 5.
(chockss)
5. The system's result is very informative and contains novel content. In addition, it displays a high level of diversity and it is enjoyable read.
Reference: affaid you can yt. 're welcome to pay cask.
Generated Responses:
GPT-2:
you DIALOGPT: Yes, but you'll have to pay the extra fee.
accept
Relevance:
you pay by credit card.
RESEE (SEP.): I don't think you can, we accept cash only.
1. The system's 's result is completely irrelevant to the given reference.
Figure 11: baselines.
Additional example for RESEE and
2. Choose this score when you are hesitant be- tween score and score 3.
E Human Evaluation
3. The system's result is partially related to the reference and some of its content can be found in the reference.
For annotators, we hire three undergraduate stu- dents from America China with fluent English reading skills. Each annotator is assigned 100 (instances)x6 6 (models) x (aspects)
400 ing tasks, resulting 2, 100 (tasks) x3 (annotators)
4. Choose this score when you hesitant be- tween score and score 5.
7, 200 human ratings in total. The annotators have acknowledged the use of annotated data sets and are paid an average annotation salary. All an- notators were aware of the potential risks or ethical concerns of machine-generated texts.
5. The system's 's result is very related to the given reference and contains diverse set of con- cepts in the reference.
Make Sense:
Annotation Instruction Here present the hu- man evaluation standard: Fluency:
YES: the response is completely reasonable
in context.
the response is confusing, illogical, out of context, or factually wrong.
NO:
1. The system's result does make sense and unreadable.
2. Choose this score when you are hesitant be- tween score and score 3.
Being Specific
YES:
the response is specific to the given con- text.
3. The system's result contains minor errors but they do not affect your understanding.
NO: the response could be used in dozens different contexts.
4. Choose this score when you are hesitant be- tween score 3 and score 5.
5. The system's result is human-like like, grammati- cally correct, and very easy to understand.


Reviews:
[{"review_id": "Z7CDG45F5h", "reviewer": "Reviewer_cBSK", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback. For your concerns:\n\n- Visual biases: thank you for pointing out this limitation. We recognize that dataset biases are a critical issue and we do not take them lightly. In fact, we explicitly acknowledge the presence of certain noises in the auto-constructed datasets and intend to take steps to make them more accurate and inclusive of visual knowledge, such as external document knowledge (L513-519) as the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Two new dataset RESEE-WoW and RESEE-DD\nA simple but effective method RESEE to add visual representation into vanilla dialogue models by modality concatenations\n", "reasons_to_reject": "The datasets are auto-constructed. Maybe there are visual biases.\n", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "1) Providing a new paradigm of constructing multimodal dialogues\n2) Two new dataset extended from text-only dialogues RESEE-WoW and RESEE-DD\n3) RESEE, adding visual representation into vanilla dialogue models by modality concatenations\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "BNvvGWI96p", "reviewer": "Reviewer_oqrU", "summary": "", "questions": "What is the purpose of extending the dialogue datasets with images? i.e., What is the role of the images in the dialogues?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback. For your concerns:\n\n- Weak novelty: our main contributions are centered on two sides: 1. We propose a new paradigm for constructing multimodal dialogue data and **two new multimodal datasets**; 2. a framework to incorporate multimodal inputs and generate dialogue responses with **two new multimodal dialogue models**.  Our dataset construction framework is unique by considering both fine-grained entity-level and turn-level images as knowledge augmentation to text-only dialogues. On the model side, only our ReSee (share) is regarded as an extension of Maria with finer-grained visual input (L370-372). Note that (1) our proposed ReSee (Sep.) stands out from other existing dialogue methods due to its use of encoder-decoder architecture with designed multimodal input and training. (2) Both of our models are specifically designed to handle visual knowledge on entity- and turn-level in the dialogue, which further verify the efficacy of the proposed visual knowledge even with a simple modality integration method.\n\n- Significant contribution of larger datasets and online searching: First of all, the benefit of larger multimodal datasets is to improve model training with larger scale. Secondly, online image searching is one initial step of our data construction paradigm (L93-101), while building upon online searching, we have further developed a comprehensive end-to-end multimodal data construction framework to ensure the data quality and more fine-grained image-dialogue data. Through this framework, we demonstrate the effectiveness of our approaches of dataset crafting and two dialogue models.\n    \n- Insignificant improvement in model: Table 3 illustrates the superiority of our proposed models with visual enhancement over vanilla UniLM and T5 models (the dialogue model upon which our ReSee models are initialized) by up to 33% and 18% on ReSee-DD dataset using the Rouge-L metric, respectively.  This significant improvement highlights the effectiveness of incorporating the proposed visual knowledge, even through simple integration methods, which has surpassed some SOTA dialogue models (*e.g.*, MSDP on BLEU, VisAD on embedding-based metrics).\n\nFor your questions:\n\n- The purpose of extending the dialogue with images: the primary purpose of extending the text-only dialogue data with images is to enhance model performance by incorporating external knowledge. As we are motivated in the Introduction section: Visual knowledge can accurately convey rich world knowledge beyond text, providing commonsense knowledge or related information that may not be acquired by specific nouns or entities through text alone (L38-44). In practical situations, it may be difficult to classify a named entity solely based on words at the first glance. For example, models may not understand what an \"ikebana\" is by just looking at the word, but an image of \"ikebana\" can help text-only dialogue systems quickly link this concept with flowers and arts (Figure 2), allowing these models to make more informed associations and better understand underlying meaning of novel concepts. Additionally, in Section 5.3, we demonstrate that our provided visual knowledge can still enhance dialogue systems even with rich textual knowledge.\n    \n- The role of images: the fundamental role of images in our work is a knowledge augmentation of existing dialogue contexts. In particular, turn-level and entity-level images serve to provide the comprehensive information (*e.g.*, the skydiving event in Figure 3) of a conversation and depict details or associated scene in the chat (*e.g.*, the sky, the parachutes), respectively. \n\nIn addition, thanks for pointing out the missing references, we will check and improve them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes better ways to collect high-quality images for multimodal conversation\n- The papers collect finer-grained images for improving accuracy.", "reasons_to_reject": "- The novelty of the proposed approach seems weak. As also mentioned in the paper, it appears to be merely a simple extension of previous work. Moreover, this extension itself does not seem particularly interesting or innovative. The reliance on larger datasets and online image search might not constitute a significant contribution.\n\n- Although the paper, as claimed, proposes an effective model, the performance improvement doesn't seem significant.", "questions_to_authors": "What is the purpose of extending the dialogue datasets with images? i.e., What is the role of the images in the dialogues?", "paper_topic_and_main_contributions": "This paper proposes a new approach to collect images for multimodal dialogues. Specifically, two types of images (turn-level and entity-level) are collected for higher fidelity. Simple, but effective baseline model is also presented. Extensive experiments demonstrate the effectiveness of the new datasets and model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Line369: Maria; It would be beneficial to cite the first introduction of a model or approach, even if it has been previously cited in a different context."}, "extra_text": {}}, {"review_id": "hvS3o9BO0h", "reviewer": "Reviewer_ks6W", "summary": "", "questions": "1. At the beginning of this paper, I think Figure 1 could use a bit more explanation. It's essential because the figure really helps demonstrate the novelty and significance of the authors' work.\n\n2. I'm curious about how well this approach would work with less informative prompts.\n\n3. I am also curious about this approach's efficiency. Though the work shows a competitive quantitative performance, it is also important to benchmark this work's computational cost. \n\n4. This work offers a strong performance but I think to reimplement it might concurs a few challenges due to its complexity. Would authors be making this work along with the dataset open-accessible in future?  ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and feedback. For your concerns:\n\n- Qualitative results: thank you for your advice, due to page limits, we were unable to present more qualitative examples in the main part; we will append more results in the appendix to show the limitations and other capacities of our models in our final version.\n- Cold-start performance: thank you for bringing up the important discussion of cold-start performance. However, this work may not directly aim to address this issue; we acknowledge its importance and plan to consider it as a future step in our research. We believe that investigating cold-start scenarios will enhance the practical applicability of our work, and we look forward to exploring this area in future studies.\n- Reproducibility: we have provided the training/inference code in the Supplementary Materials. We will release the proposed datasets and codes, and we also plan to maintain and improve the datasets by eliminating potential biases in the data.\n\nFor your questions:\n- About Figure 1: thanks for your advice, we will add more explanations to Figure 1.\n- Less informative prompts: in our work, the prompt we feed into the dialogue system is the dialogue context, entity phrases, and two levels of images. In Table 3, we present comprehensive ablation results for different information inputs. With less informative prompts (*i.e.*, without images or entity texts), the dialogue models perform worse on both datasets. Moreover, we investigate the model performance gain brought by an increased number of entity-level images per entity in the dialogue in Table 4, which are considered to represent detail information. And there exists an upper bound of adding such information, *i.e.*, more than 4 entity-level images inversely impairs model capability.\n- Efficiency: we have listed the total training time of each of our models in Appendix A.2, since our models are initialized using UniLM and T5 severally, our dialogue models can be trained at a relatively small cost (no more than 12 hours using two GPUs with 48G memory in total).\n- Accessibility: we will release the datasets, codes, and model checkpoints for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors' work represents a significant and exciting contribution to the fields. The article is well-structured and written, demonstrating a clear motivation, detailed methodology explanations, and a well-designed experimental setup.\n\nThe study focuses on addressing a prevalent limitation in contemporary text-only dialogue systems. These limitations encompass restricted scale and quality of available datasets, as well as insufficient integration of visual knowledge. The authors propose a novel paradigm for constructing a multimodal dialogue system that effectively incorporates visual knowledge into various granularities.\n\nFurthermore, the authors make a valuable contribution to the field by introducing two datasets that extend the scope of text-only dialogues. These datasets hold promise for advancing research in this domain.\n\nThis work offers extensive ablation studies, examining different variations of input information to gain deeper insights into their respective roles in the dialogue generation task. Additionally, the study presents human evaluation results, further reinforcing and justifying the findings obtained from automatic metrics. Such human evaluation yields a potential for real-application, such as multi-modal based E-commerce recommender system. ", "reasons_to_reject": "Authors should enrich the paper with qualitative results that offer a comprehensive and illustrative assessment of the model's performance, enabling a deeper understanding of its capabilities, limitations, and areas for enhancement.\n\nFurther, the authors have not addressed the cold-start performance of their work. Given the likelihood of real-world applications encountering user prompts beyond the model's learned response domain distribution, investigating cold-start scenarios becomes imperative.\n\nAdditionally, a bit concern of its work's reproducibility as this is a fairly complex approach. Thus, incorporating a clear and concise algorithmic box, illustrating the learning algorithm employed for training and inference, would be beneficial in providing readers with a more transparent view of the methodology.", "questions_to_authors": "1. At the beginning of this paper, I think Figure 1 could use a bit more explanation. It's essential because the figure really helps demonstrate the novelty and significance of the authors' work.\n\n2. I'm curious about how well this approach would work with less informative prompts.\n\n3. I am also curious about this approach's efficiency. Though the work shows a competitive quantitative performance, it is also important to benchmark this work's computational cost. \n\n4. This work offers a strong performance but I think to reimplement it might concurs a few challenges due to its complexity. Would authors be making this work along with the dataset open-accessible in future?  ", "paper_topic_and_main_contributions": "The authors present a novel approach that enhances text-only dialogue systems by integrating visual knowledge, addressing limitations in the existing solutions. These limitations include the restricted scale and quality of available datasets, as well as less aware of the visual knowledge integration. To overcome these challenges, the authors introduce a new paradigm for constructing multimodal dialogues. Their method involves a deliberate and explicit division of visual knowledge into finer granularities at both the entity and turn levels. These finer-grained visual representations are obtained from either the Internet or a large image dataset. The proposed framework, named RESEE, seamlessly incorporates visual representations into vanilla dialogue models. Despite its straightforward nature, RESEE proves to be highly effective in enhancing dialogue systems. Extensive experiments and ablations are conducted, demonstrating the efficacy of integrating visual knowledge and confirming the superiority of the RESEE model over alternative approaches. Additionally, the authors contribute to the field by providing two datasets that extend text-only dialogues. These datasets hold promise for further advancements in this area of study.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper proposes a "simple but effective" method to add visual knowledge into vanilla dialogue data. Two new visually augmented datasets are provided as an outcome of the method.

**Pros**: Most reviewers agree the contributions of the work, and in particular, the datasets provided by the work will be a good resource to the community. One reviewer comments explicitly on the experimental study, citing components (e.g., ablation and human evaluation) that provided confidence in takeaways and potential real-world application. Multiple reviewers find the work to be fairly exciting/interesting.

**Cons**: One reviewer has concerns about the presence of visual biases - given that the dataset is automatically constructed. Another provides examples of experiments that would improve the work. Authors make some promises to ease concerns, but leave some challenges as future work. Reviewers appear to generally consider these technical concerns as minor after rebuttal. 

With regards to novelty, there is some disagreement. While two reviewers find the work to be exciting/interesting - one of which takes care to highlight many of the strengths of the work - another reviewer finds the approach to be more incremental, emphasizing that the reliance on larger data sources may not be a significant contribution. Given the disagreement, I looked into the rebuttals and some paper details as well. From a methodological standpoint, I agree "larger data source" is not a significant contribution. *But,* some novelty is also owed to the use of refined visual knowledge (entity + turn level). Indeed, this refinement in approach leads to two new datasets which are well positioned as novel by the authors (Table 1).
Decision: Accept-Main