A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models
Authors: Jaewook Lee
Seongsik Park
Seong-Heum Park
Hongjin KIM
Harksoo Kim
Venue: EMNLP 2023 Main (2023)
Keywords: multimodal agent
multimodal dialogue agent
multi-turn dialogue agent
warm-up task

Abstract: Most research on multimodal open-domain dialogue agents has focused on pretraining and multi-task learning using additional rich datasets beyond a given target dataset. However, methods for exploiting these additional datasets can be quite limited in real-world settings, creating a need for more efficient methods for constructing agents based solely on the target dataset. To address these issues, we present a new learning strategy called vision-language warm-up tasks for multimodal dialogue models (VLAW-MDM). This strategy does not require the use of large pretraining or multi-task datasets but rather relies solely on learning from target data. Moreover, our proposed approach automatically generate captions for images and incorporate them into the model's input to improve the contextualization of visual information. Using this novel approach, we empirically demonstrate that our learning strategy is effective for limited data and relatively small models. The result show that our method achieved comparable and in some cases superior performance compared to existing state-of-the-art models on various evaluation metrics.

A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models
Jaewook Lee, Seongsik Park, Seong-heum Park, Hongjin Kim, Harksoo Kim
Konkuk University
(benecia428, a163912, tjdgma95, jin3430, nlpdrkim @ @konkuk.ac.kr
Abstract
Most research multimodal open-domain dialogue agents has focused on pretraining multi-task learning using additional rich datasets beyond given target dataset. How- ever, methods for exploiting these additional datasets can be quite limited in real-world tings, creating need for more efficient meth- ods for constructing agents based solely on the target dataset. To address these issues, present new learning strategy called vision- language warm-up tasks for multimodal dia- logue models (VLAW-MDM), This strategy does not require the use of large pretraining or multi-task datasets but rather relies solely on learning from target data. Moreover, our pro- posed approach automatically generates cap- tions for images and incorporates them into the model's input improve the contextual- ization visual information. Using this novel approach, empirically demonstrate that learning strategy is effective for limited data relatively small models. The result show that our method achieved comparable and in some cases superior performance compared to existing state-of-the-ar models on various eval- uation metries. The code is available at https: //gi thub. com/ httn/Bnncciaeee/V.... MLW
Image
m worried about this race (Anxious)
Turn
All these bike races are happening near my street. (Dull)
Turn 2
I'd be worried ahout accidentally hitting one of them (Aaxious)
Turn 3
Table I: IMAGE-CHAT dataset
conversation, which is more consistent with how humans communicate. This approach has been shown to be effective in generating conversations that users find more engaging(Hu et al., 2014). To build such multimodal open-domain conver- sational agents, most previous studies have sidered multi-task learning have utilized pre- trained models using large-scale data beyond the target data(Shuster al., 2020b, 2021). This al- lows models trained with only text information to accept visual information. However, the require- ment of collecting additional datasets is restrictive, and pretraining with additional datasets is ineffi- cient in terms of time and resources. In this study, propose method align text and images us- ing only the target data address these issues. We experimentally evaluated the effectiveness of the proposed method with limited data or smaller mod- cls.
1 Introduction
Developing artificial intelligence (AT) that can con- verse naturally with humans is a primary goal of AI research. In particular, open-domain conversational agents that are not restricted specific domain have attracted considerable attention. Many studies have adopted a pretraining approach using large datasets to improve the performance of these open- domain conversational exeens(Adiwarde) et al., 2020; Roller et al., 2021).
Recent works have focused multimodal open- domain conversational agents that consider visual information in addition textual information for dialog generation. This approach utilizes visual information to help understand the context of a
The proposed method utilizes only target data align images and texts. By automatically generating captions for the images and adding them as input to the model, our approach enables a text-based pre-

---

trained model to process image information more effectively. This differs notably from existing mul- timodal open-domain models that only receive im- age and utterance information. We propose vision- language warm-up tasks for multimodal dialogue models (VLAW-MDM) to effectively integrate in- formation from images, captions. and context data. To construct the framework, we incorporated four warm-up tasks based on existing multimodal pre- training models(Chen et al., 2020; Yu et al., 2021; Wu et al., 2022). These tasks include generation captioning (GCP), image swapping (ISP), masked region modeling (MRM), and masked language modeling (MLM). They can be applied using only target data without any additional data required. These warm-up tasks enable the model to learn the associations between images and utterances.
tally evaluated how they each affected the per- formance of the model. Our results show that the best performance was achieved when all four warm-up tasks were utilized together.
We analyzed how automatically generating and utilizing captions affected the perfor- mance of the model. Our results showed that our proposed framework incorporating cap- tion information was effective for training a multimodal open-domain dialogue model.
We also evaluated the warm-up tasks in the absence of caption information. The results show that the proposed method is effective even environments where captions are not available or are difficult to create.
The construction of this framework was inspired by the generative framework used by Ling et al. (2022), which we reformulate into a multimodal ar- chitecture suitable for the purposes of this research. We apply our warm-up framework to the popular suueenee-to-seqennee models BART and Blender- Bot. For BlenderBot, we used smaller model (400M) than that (2.7B) adopted in the multi-modal blenderBot (SMus(Shuster et al., 2021). This al- lowed us to explore the performance of our pro- posed framework on smaller versions of the model and to validate the framework against a relatively large model with additional training data.
Related Work
Vision and Language Tasks. The integration of language and vision is topic of active research that traditionally includes tasks such image caption- ing and visual question answering (VQA)(Devlin et al., 2015; Xu al., 2015; Fang et al., 2015; Don- ahue et al., 2015; Antol et al., 2015; Ray et al.. 2016; Malinowski and Fritz, 2014). Image cap- tioning tasks focus on generating appropriate de- scriptions of a given image. Major datasets include COCO Captions (Chen et al., 2015) and Flickr30k (Young et al., 2014). These datasets of images cov- ering various topies provide ideal benchmark for assessing model's ability to understand com- plex content in an image and express it in natural language.
We used the Image-Chat dataset(Shuster 2020a) to evaluate the effectiveness of our proposed framework. The data were structured as shown in Table 1. Image-Chal comprises conversations orga- nized into a series of turns with utterances based on the speakers' styles. We used Image-Chat for warm- up tasks first, and then performed fine-tuning to evaluate the effectiveness of our proposed method in constrained learning environment.
al.,
-o-suuuee---- structures are the most common method to process these datasets (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018).
The VQA task (Antol et al., 2015) requires im- age recognition and factual verification of text con- tent. It evaluates the ability of model generate accurate answers to questions related to given im- age. As a natural extension this work, method has been proposed to generate questions that can capture a user's attention based on a given image. However, these methods involve the limitation that the conversation usually ends in single turn. To address this, visual dialog(Das et al., 2017) extends this with continuous question-answer scenario. However, this approach does not provide a di- rect way to evaluate whether users experience conversations as interesting and engaging. To ad- dress this, multimodal multi-turn dialog datasets
The main contributions of this study are summa- rized as follows:
We propose framework for vision-language warm-up tasks in multimodal dialogue mod- els, called VLAW-MDM, and deseribe the pro- cess of warming up the model using only data from the target task. We experimentally eval- uated the performance of this framework as described above.
We introduce four different warm-up tasks (MLM, ISP, MRM, and GCD) and experimen-

---

Vision-Language Warm-up Tasks
Captioning
Image- Swapping
Region Modeling
Language Modeling
poyers
<eot>
positive <ROR
depp
Blenderior Decoder
BlenderBot Decoder
Blender Bot Decoder
Blenderior Decoder
<bos
sged> players
<bos>
<ispo
positive
<bos?
SHimP
<feat>
<zero>
<boss
smimm
tall
BlenderBott Encoder
Modality segment Embedding
Embedding
Emedding
Embedding
Token Embedding
dimp>
CLI Image Encoder
</Ime>
Caption
Style & Utterance
Style Utterance
Style Utterance
Objs: Generation lon Captioning
Objy Masked Modaling
Obji Rugion Modeling
Caption Generation
Ohl Jeane Swappios
Fale snower Image
Figure Overview of the Proposed Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models.
(Mostafazadeh et al., 2017; Shuster et al., 2020a) have been proposed. In particular, Image- Chat (Shuster et al., 2020a) supports multi-turn dialog based single image. Each utterance contains style information allow model to learn differ- ent styles of conversation. Hence, Image-Chit -hatt an ideal dataset for training multimodal open-domain agents and enables more engaging dialog genera- tion.
2020a; Chen et al., 2020). However, multimodal pretraining approach has generally not been per- formed with data in the target domain. One poten- tial solution is to employ domain aadative pretrain- ing by using data related to the domain of the target data(Shuster al., 2021). This method enables a model adapt more effectively a specific do- main. However, Nomain-adaptinn pretraining also utilizes data related to the target domain and does not provide a pretraining methodology for specific target data. As solution, Ling et al. (2022) pro- posed task-specific vision- language pretraining framework for multimodal aspect-based sentiment analysis. which realizes target data-specifie pre- training on multiple tasks instead of single task.
Multimodal Representation Learning. Utiliz- ing the weights of existing models trained on single modality and fusing them together is com- mon strategy for multimodal learning(Kiela et al., 2019; Le and Hoi, 2020; Chen et al., 2020; et al., 2021; Zhang et al., 2021). Many studies have adopted this approach to reuse models pretrained single modality for multimodal representation learning.
3 Methodology
In the present work, we adopt BlenderBot as a back- bone model. The overall architecture is illustrated in Figure 1. In this section, we describe the oper- ation of the entire framework. First, describe how process the images used input the model. In particular, discuss our approach tract features from an image and generate captions. Then, we describe how the encoder is extended to handle multimodal inputs and how the decoder
To process images and text together, Shuster et al. (2020b) uses multi-tae talli training, in which a model learns by bundling multiple tasks that are related to a given target task. For this purpose, Shuster et al. (2020b) includes 12 subtask sets, which allows it to perform multiple tasks on large datasets.
In some cases, multimodal representations have been learned from multimodal datasets(Li et al.,

---

generates utterances from the information received from the encoder. Finally, we describe the warm-up tasks that comprise the framework.
the caption created earlier. There is no special to- ken for the caption; rather a (sep) token is simply added to the end of the caption. The caption is fol- lowed by the style and utterance. An additional special token such as (sty) is appended at the end of styles to distinguish them from utterances. Styles are followed by utterances, and the difference be- tween the warm-up task and fine-tuning phases becomes relevant here. In the warm-up task phase, the styles and corresponding utterances are input the encoder together. However, in the fine-tuning phase, the style is not followed by an utterance be- cause the model needs to predict an utterance for the style. Therefore, in the warm-up task, styles and utterances are combined and followed by an (cos) token. However, during the fine-tuning stage, the (eos) token is appended immediately after the (sty) special token representing the style.
3.1 Image Encoder
Our proposed method uses pretrained models to extract visual features from images. Previous stud- ies(Shuster et al., 2021; Ling et al., 2022) have mainly used Faster R-CNN(Anderson et al., 2018 8). Recently, patch-based models have shown better performance in image encoding(Shen et al., 2021; Wu al., 2022). Based on these findings, we adopt a patch-based method based on CLIP in the pro- posed approach(Radford et al., 2021). In image en- coding, single image is divided into nine patches used as input to the model to obtain visual features. We denote the visual features as R = (ri, .... rg), where E 1252 is the visual feature of the i-th patch. The obtained visual features do not have the same number of dimensions the textual represen- tation, so an additional linear transformation layer is used to put the visual features along with the textual representation as input to the multimodal encoder. This linear transformation layer projects the visual features of each patch to d-dimensional vector denoted as V E Rd
Decoder. As shown in Figure 1, all warm-up tasks are processed through a single decoder. To distinguish between warm-up tasks, we add spe- cial token at the beginning of the decoder's 's input, following prior work (Yan al., 2021; Ling al., 2022). The input of the decoder starts with (bos), followed by (gcp), (isp). (mrm), and (mlm), de- pending on the warm-up task. The input format is followed by the label values according to the warm-up task.
3.2 Caption Generation
Image captions are textual descriptions of objects, situations, and other information provided by an im- age. We used captions as bridge between images and text. Because a caption textual represen- tation of the information in an image, we assume that aligning the image with the utterance text is beneficial. However, because there are no separate captions for images in the existing dataset, we use an image captioning model (Li al., 2023) to gen- erate captions. The generated captions provide description of the image and are used input to the multimodal encoder along with the image and utterance.
3.4 Warm-up Tasks
In this study, we introduce VLAW-MDM to effi- ciently integrate multimodal information. This is a warm-up task that strengthens the connections be- tween images and text before the fine-tuning phase to improve the model's ability handle complex multimodal information more effectively. During this warm-up task, the model trained to under- stand and strengthen the relationship between im- ages and text by utilizing data on the target task. This improves the model's ability to process mul- timodal input, which in turn improves its perfor- mance by utilizing only data the target task with- out any additional data for pretraining. These en- hanced connections between images and text play an important role in improving performance on the target task during the fine-tuning phase, allowing for more effective utilization of multimodal data at no additional cost.
3.3 Multi-Modal Architecture
Encoder. The encoder receives different kinds of modality information. To separate the modal- ity information, we add segment embedding that separates the image from the text. We also add special tokens such (img). (/img) before and after the extracted image features following Xing et al. (2021). As shown in Figure the images are entered in the order they appear first in the modal- ity information. The image feature is followed by
Generation Captioning (GCP). The GCP task replaces all captions with masking tokens and re- stores them to the original captions. In the GCP

---

task, the model interprets the context of the im- age based on other multimodal information such as image or utterance data without any caption in- formation and generates caption accordingly. This helps the decoder not only analyze and understand information from each modality independently but also acquire the ability comprehensively under- stand and appropriately integrate information from other modalities such images and utterances.
certain probability. Images are then fed into the en- coder along with caption and dialog. The encoder
processes this multimodal information and passes the results to the decoder. Based on this informa- tion, the decoder determines whether the image is an original or an altered image. The results are expressed "positive" or "negative." "positive" means that the decoder recognizes the original im- age, whereas "negative" means that the image has been altered.
The target sequence for the GCP task Y = Kged)). ....... CN. (eos)]. where c represents cap- tion tokens and is the number of caption tokens. Traditional training methods such as maximum like- lihood estimation (MLE) have problem in that they mainly generate high-frequeney responses that exist in the dataset. Therefore, to control these high- frequency responses, we adopt unlikelihood train-
The target sequence for the ISP action is Y [boo). (isp) S, (eos)], where S indicates "posi- tive" or "negative. " The loss function for the ISP task is as follows:
= -Bs==-ExD..EE
log (e vee), (4)
ing (Roller et al., 2021; Li et al., 2020b; Welleck et al.). The formula for MLE is as follows:
V1 E Y, which refers to text token or special token in Y. Let yoe be the tokens before the I-th utterance of yt-
In the loss function, we consider two cases: X and X. X an instance from a data distribution that retains the original image, caption, and utter- ance. In contrast, X represents a case where only the image changes while the caption and utterance remain the same. I denotes the combined input in- formation for these cases, encompassing both X and X.
CULE = - Coope(ud yer),
(I)
where y/ E Y and y is caption token or special token for Y. Let V<r be the tokens before the th utterance of yr- The X mcans that the caption input to the encoder is masked, where the rest of the multimodal information except for the caption token is in its normal form. The formula for the unlikelihood loss function is as follows:
Masked Region Modeling(MRRM). We adopted the MRM method used by Xing et al. (2021); Ling et al. (2022). We masked random positions in the patches. The masked image is passed to a multimodal encoder, and the decoder estimates the masked part(Ling et al., 2022). For the MRM task, the inputs the decoder are (feat)) and (zero). Here, the (Seal) token is used for the unmasked normal image region and the (zero) to- ken used for the masked part, which feeds the value at position (zero) into the MLP layer. The MLP layer is trained to match the output repre- sentation to the original image representation. The final loss value of MRM is as follows:
CUL
(2) log(1 - pe Pe(yelX, vee)),
MeEC,
The negative candidates C₁ are the set of tokens that we do not want to generate at each time step. This is controlled by assigning a penalty if the to- ken generated by the model belongs to Ct. Like- lihood is used to increase the probability of the next fired token, Vt. while unlikelihood is used to decrease the probability of ye. The final loss value for GCP is as follows:
LOCp CULE + aLuL
(3)
EMRM = -Ex- DEL (90.))lPoP))) (5)
a is the weighting representing how much to reflect LUL-
The representation predicted by the model is p(u,) and the actual input image representation is (v,). The model was trained by minimizing the KL divergence, where R is the number of masked regions.
Image-Swael ( The ISP task serves to train the model's ability to determine whether an image is the original or an altered image. The main process of an ISP task is as follows. An image is replaced with another image in the batch with a
Masked Language Modeling(MLLMD. For MLM, we masked the tokens in an utterance at

---

worrled
about
this
moce
Train
Valid
Test
Number of Images
186,782
5,000
9,997
Dlander Bot Decoder
Number of Utterances
355,862
15,000
29,991
<bou>
am
worried
about
this
rece
Style Types
215
215
215
Vocabulary Size
46,371
9,561
13,550
Blenderibo Encoder
Token per Utterance
12.3
12.4
12.4
Modullty Suginent Embbtdinn
Table 2: IMAGE-CHAT dataset statistics.
Token Embedding
Image Emedding
Tokeo Embedding
Takes Embedding
CLIP Image Encoder
Caption
Style
Fine-tuning the Dataset. We used the Image- Chat dataset to verify the effectiveness of the pro- posed framework. The data consisted of an image, style attributes of two speakers (A and B), and dialogue between the two speakers. It also in- cluded set of 215 possible style attributes from Shuster al. (2019), which are categorized pos- itive, neutral, and negative. The style attributes are used to define the speakers' personalities in the conversation. The images in the dataset were se- lected from the YFCC100M dataset (Thomee et al., 2016). Some statistics on the data included in the Image-Chat dataset are shown in Table 2.
Caption Generation
Figure 2: Example turn-by-turn utterance generation for Image-Chat
certain rate. In this case, masking was not per-
formed for the entire turn but rather only for
certain percentage in each turn(Devlin et al., 2018).
The target sequence for the MLM task is Y [mlm), .d ........ d (sty), dit ...., di (cos)]- The sequence consists of the style of the utterance and the turns that represent the utterance. It begins with a start token (bos) and (mlm), followed by N tokens d' indicating the style, and ending with the token (sty) token to indicate the end of the style. This is followed by M tokens dl representing the utterances in that turn, with the turns separated by (sep) tokens. this example, we indicate the end of the turn through the (eos) token directly, without separate (sep) token, because it indicates that we sampled for Turn 1.
The utterance generation for the first turn using Image-Chat is shown in Figure 2. The data input the encoder in the first turn were the image, caption, and style of the first turn. The encoder processes these inputs and passes the information to the de- coder. The decoder generates an utterance for the first turn based on the information from the encoder. In the second turn, the encoder receives an image along with caption and style from the first turn, and utterance of the first turn, and style from the second turn. The decoder generates the second round of utterance from these inputs, and the third round proceeds in the same manner.
To calculate the MLM loss value, the loss value is calculated in the GCP operation, and the final loss value is follows:
4.2 Main Results
LMLM = LMLLE + aoou
(6)
Impact of Each Warm-up Task. Table 3 shows the contribution of each warm-up task to perfor- mance. We compared the performance of models trained with and without warm-up tasks.
Full Warm-up Task Loss. The final objective function is as follows:
Acoor deCIsP + AgEMAR +Aerus (7)
To validate the scalability of the warm-up task. we applied it to two different sequence- sequencc-to--sqquenc models, including BART and BlenderBot. First, in terms of BlenderBotts performance, an improve- ment may be observed in all measures except the BLEU-4 score at Turn 3 when the MLM task was introduced compared to no warm-up task. This likely due the word prediction ability learned through the MLM helping with utterance gener- ation. Next, when we added the ISP task along- side MLM, we observed additional performance
The values given above are adjustable hyper- parameter. For this experiment, all values were fixed to 1.
4 Experiment
4.1 Experimental Setup
Evaluation Metrics. We used the F1, BLEU-4, and ROUGE-L metrics to evaluate the performance of the proposed model.

---

Turn 1
Turn
Turn 3
IC
Model
Warm-up Task
F1
B
R
Fi
R
F1
B
R
F1
B
R
w/o warm-up task
8.97
0.44
10.41
12.34
0.60
10.37
14.68
0.79
11.70
12,00
0.61
10.83
MLM
11.16
0.61
9.90
12.92
0.65
9.96
14.39
0.75
11.12
12.82
0.67
10.33
BART
MLM-ISP
11.76
0.70
10.32
0.66
10.32
14.41
0.78
11.32
13:10
0.71
10.65
MLM+ISP+MRM
11.77
0.67
10.08
13.26
0.65
10.17
14.61
0.77
11:42
13.21
0.70
10.56
MLM++SP+ MRM+GID
12.22
0.69
10.63
13.70
0.75
10.79
14.68
0.85
11.60
13,53
0.76
11.00
w/o warm-up task
15.36
0.81
11.47
16.53
0.79
12.31
16.53
0.71
12.50
16.14
0.77
12.09
MLM
15.72
0.90
11.69
16.75
0.86
12,43
16.88
0.70
12.70
16.45
0.82
12.27
BlenderBot
MLM-ISP
15.81
0.89
11.81
16.89
0.88
12.59
17.25
0.81
13.04
16.65
0.86
12.48
MLM+ISP-MRM
15.91
0.91
11.90
17.02
0.87
12.70
17.24
0.78
13.09
16.72
0.85
12.56
MRMMGGD
15.90
1.04
12.00
17.05
1.03
12.73
17.30
1.00
13.17
16.75
1.02
12.63
Table 3: Ablation study results presenting the performance of models across various warm-up tasks. Each model's performance was evaluated at three distinct interaction turns (Tum 1, Turn 2, Turn 3), each with their respective F1. BL)E--(B), and ROUGE-L((R) measurements. The IC column represents the average these metrics across interaction turns.
Turn I B
Turn B
Turn
Caption
Warm-up Task
F1
R
F1
F1
B
R
w/o warm-up task MLM + ISP + MRM
14.85
0.91
11.20
16.16
0.89
12.12
16.42
0.84
12.48
w/o caption
15.25
1.02
11.61
16.38
0.97
12.27
16.76
0.97
12.75
w/o warm-up task MLM ISP + MRM + GCD
15.36
0.81
11.47
16.53
0.79
12.31
16.53
0.71
12.50
w/ caption
15,90
1.04
12.00
17.05
1.03
12.73
17.30
1.00
13.17
Table 4: Performance Evaluation based on caption information. Each model's performance is evaluated at three distinct interaction turns (Turn 1, Turn 2, Turn 3), each with their respective F1, BL.EU44 (B), and ROUGE-L (R)
measurements.
gains on all scales except for the BLEU-4 score on Turn 1. ISP determines the appropriateness of a given image based its caption and dialog. This allows the model to learn to align the image with the dialog, which is likely the reason for the per- formance improvement with the addition of ISP.
of the model, and the best performance was achieved when all the warm-up tasks were com- bined. This demonstrates the effectiveness of the framework proposed in this study.
Table 3 shows that applying warm-up task im- proved the performance of both BART and Blender- Bot. Compared to BlenderBot, BART is smaller model. These results demonstrate that the proposed method is effective even for small models, ini- tially assumed.
Third, when we added the MRM task to MLM and ISP, the results showed a further increase in per- formance. MRM helps the model understand the image features given in the form of patches. This is important for multimodal open- open-domain agents that perform conversations based on images and seems to have helped with utterance generation. Fi- achieved when
Impact of Caption. Table 4 compares the per- formance of the models tested with and without caption. In Table 4, row shows the performance of the backbone model without caption and row 3 gives the performance of the backbone model with caption. Comparing the two, it may be observed that simply providing caption helped the backbone model generate utterances. This was likely the case because the caption effectively acted as bridge between the image and the utterance. Of note, the improvement on Turn 3 smaller than that on Turns 1 and 2. When generating an utterance for Turn 3, the input was an utterance from Turns and 2. As with the caption, the utterances in Turns and 2 provide information about the image in the form of text, which made easier for the language
nally, the highest performance the GCD task was added to ML.M, ISP, and MRM. The GCD task generates captions based on a given conversation and image. Through the GCD task, the model learns the caption and its relationship to given conversation and image. This process allows the model to quickly learn the relationship between dialog, image, and caption. In particular, the information provided by the image during utter- ance generation appears to help the language model recognize and generate the correct utterance.
These experimental results show that each warm- up task contributed to improving the performance

---

model understand the image and generate an ut- terance. This was most likely the reason that the two performances were similar.
IC
Model
F1
DialoGPT
6.2
0.1
5.2
(Zhang al., 2020)
The results with and without the warm-up task when captions were provided are shown in rows and 2. We excluded the GCD from the warm- up task because they were not captioned. The re- sults show that the warm-up task without captions helped improve performance. These results show that the proposed method is effective even in envi- ronments where captions are not available or can- not be automatically generated.
Dodeca (Shuster et al., 2020)
12.9
24.6
2AMMC (Ju et al., 2019)
9.3
0.1
11.0
BlenderBot (Roller al., 2020)
9.2
0.1
12.3
MultiModal BlenderBot
13.1
0.4
18.0
(Shuster et al., 2021)
VLAW-MDM
16.8
1.0
12.6
The performance on the warm-up task when captions were provided is shown in rows and 4. Because captions were provided, we ran all the warm-up tasks, including GCD. The results showed that applying all warm-up tasks significantly im- proved performance. In particular, the performance for Turn row 4 shows that we achieved suffi- ciently high performance improvement compared with the other methods. This suggests that warm- ing up the model with GCD is more effective than naively entering textual information (captions and turn-by-turn utterances) for images.
Table 5: Comparison results with existing models for Image-Chat.
utterance generation by utilizing additional caption information.
Conclusion
We have proposed VLAW-MDM as a methodol- ogy for training multimodal open-domain agents using only target data to obviate the need for large amounts of data for pretraining or multiple tasks. The experimental results have shown that even with limited data, a model pretrained from single modality effectively process multimodal infor- mation. Furthermore, our proposed approach out- performed existing models in terms FI score on the Image-Char dataset and outperformed MMB terms of Fl and BLEU-4 scores despite its smaller size. In future work, we plan to explore extensions to this framework.
4.3
Experimental results compared to baseline
To evaluate the performance of the proposed method, we conducted comparative analysis of Image-Chal with various existing models used in the experiments. The results of the comparison are listed in Table 5. Most of the compared models use additional datasets other than the data of the target task perform pretraining align text and image information, or apply various multi-task techniques. This differs from our model, which utilizes only data on the target task. See Appendix A for a de- scription of models compared.
Limitations
While the methodology presented in this study pro- vides meaningful results, it also involves num- ber of limitations. To demonstrate the performance of the proposed framework, we only used Image- Chat, which is characterized by combining image and style information perform multi-turn conver- sations. Therefore, differences style and dialog format may have affected the findings of this study.
Table shows that BlenderBot with our frame- work exhibited performance comparable to that of state-of-the-ard methods. In particular, the highest performance was achieved in terms of Fl score. The proposed model is smaller than the Blender- Bot model used by MMB. Nevertheless, our model outperformed the F1 score of MMB, which was previously the highest-scoring model, and also per- formed better in terms of BLEU-4 score. Conse- quently, these results show that framework is able incorporate image information into model pretrained using only existing text data. They also demonstrate that our framework can be effectively applied to small models and can further improve
Because our model utilizes image captions additional input, it is highly dependent on the accuracy of the generated captions. Captions play an important role in the learning and performance of the model because they serve as a textual rep- resentation of the image. However, errors in the caption generation process or captions that do not

---

accurately reflect the key content of an image can affect the model's ability to generate utterances. Finally, we quantitatively evaluated the perfor- mance of the proposed model. However, although this quantitative evaluation is useful for measuring overall performance, it involves some limitations in capturing qualitative aspects such as the user experience. These qualitative factors such user satisfaction, convenience, and understanding are useful more accurately evaluate the actual per- formance of a learning model. They also play an important role in improving models based on user feedback.
captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 6077-6086. IEFE Computer Society.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages
Xinlei Chen, Hao Fang. Tsung- Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 100.v0500.00022
In this study, we have focused on a quantita- tive evaluation to clearly demonstrate the perfor- mance of our proposed methodology. However, acknowledge that this does not comprehensively cover the qualitative factors. In future work, we plan to perform a qualitative evaluation based on user feedback to further evaluate the performance of the model and user satisfaction.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EEI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, 2020. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part XXX, pages 104- 120. Springer.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1080-1089, IEEE Computer Society.
Acknowledgements
This work was supported by Institute of Infor- mation& & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2023-00216011 Devel- opment of artificial complex intelligence for con- ceptually understanding and inferring like hu- man). And this research was supported by the MSIT(Ministry of Science and ICT), Korea, under the IIRC(Infonnation Technology Research Cen- ter) support m pervised by the IITP(inttituee for Information & Communications Technology Planning & Evalua- tion). And this work supported by Institute of Information & communications Technology Plan- ning& & Evaluation (IITP) under the metaverse sup- port program to nurture the best talents (0TTP-2023- RS-2023-00256615) grant funded by the Korea gMVerMmen((MSITT)
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1rXiv:1810.04800.
Jacob Devlin, Hao Cheng, Hao Fang. Saurabh Gupta, Li Deng. Xiaodong He, Geoffrey Zweig, and Mar- garet Mitchell. 2015. Language models for image captioning: The quirks and what works. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference Natural Language Pro- cessing (Volume Short Papers), pages 100-105.
su-
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadar- rama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recogni- tion and description. In Proceedings of the ITE. conference computer vision and pattern recogni- tion, pages 3625-2634.
References
Hao Fang, Saurabh Gupta, Forrest landola, Rupesh Sri- vastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xi- aodong He, Margaret Mitchell, John Platt, et 2015. From captions to visual concepts and back. In CVPR 2015.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, al. 2020. Towards a human-like open-domain chat- bot. arXiv preprint 2xXiv:2001.09977
Yuheng Hu, Lydia Manikonda, and Subbarao Kamb- hampati. 2014. What we instagram: A first analysis of instagram photo content and user types. In Pro- ceedings of the international AAAI conference on web and social media, volume 8, pages 595-598.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image

---

Da Ju. Kurt Shuster, Y-Lan Boureau, and Jason We- ston. 2019. All-in-one amage-grounde conversa- tional agents. arXiv preprint arXiv:1912 1234.
Arijit Ray, Gordon Christie. Mohit Bansal, Dhruv Batra, and Devi Parikh. 2016. Question relevance in vqa: Identifying non-visual and false-premise questions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 919- 924.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. 2019. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv: 1909, 0290000
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Eric Michael Smith, Y-Lan Boureau, et al. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistix:: Main Volume, pages 300-325.
Hung Le and Steven CH Hoi. 2020. Video- greeo-grounded dialogues with pretrained generation language mod- els. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal en- coder for vision and language by cross-modal pre- training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336- 1 11344.
Sheng Shen, Liunian Harold Li. Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2021. How much can clip benefit vision-and-langen language tasks? arXiv preprint arXiv:2107.066 06383.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping seggaag--mage pre- training with frozen image encoders and large lan- guage models. arXiv preprint ar arXiv:2301.1 12597.
Kurt Shuster, Samuel Humeau, Antoine Bordes, and Ja- son Weston. 2020a. Image-chat: Engaging grounded conversations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2414-2429.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020b. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 12516-12526.
Yan Ling, Jianfei Yu, and Rui Xia. 2022. Vision- language pre- pre-training for multimodal aspect-based sentiment analysis. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2149 2159.
Kurt Shuster, Da Ju. Stephen Roller, Emily Dinan, Y- Lan Boureau, and Jason Weston. 2020b. The di- alogue dodecathlon: Open-domain knowledge and image grounded conversational agents. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2453-2470.
Nya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.05101.
Kurt Shuster, Eric Michael Smith, Da Ju, and Jason Weston. 2021. Multi-modal open-domain dialogue. In Proceedings of the 2021 Conference Empiri- cal Methods in Natural Language Processing, pages 4863-4883.
Mateusz Malinowski and Mario Fritz. 2014. A multi- world approach to question answering about real- world scenes based on uncertain input. Advances in neural information processing systems, 27.
Leslie N Smith and Nicholay Topin. 2019. Super- convergence: Very fast training of neural networks using large learning rates. Artificial intelligence and machine learning for muliidomain operations applications, volume 11006. pages 369-386. SPIE.
Nasrin Mostafazadeh, Chris Brockett, William B Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende. 2017. Image grounded conversations: Multimodal context for natural ques- tion and response generation. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume 1: Long Pupers), pages 462-472.
Bart Thomee, David A Shamma, Gerald Friedland, Ben- jamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2)-64-73
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8788 8763. PMLR.
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Dumitru Erhan, 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 3156-3164.

---

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In Interna- tional Conference on Learning Representations.
response generation from the GPT-2 model. This model can only take textual information as input. Dodeca(Shuster et al., 2020b): The multi-task learning approach was used to train multiple tasks at once, and for this purpose, the dodecaDialogue dataset was built. The dodecaDialogue dataset con- sists of 12 tasks, and the model was trained on these tasks. For image feature extraction, the ResNeXT- IG-3.5B model (Mahajan et al., 2018) was used.
Te-Lin Wu, Alex Spangher, Pegah Alipoormolabshhi, Marjorie Freedman, Raiph Weischedel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 4525-4542.
2AMMC(Ju al., 2019): The model was con- structed by combining ResNeXt-IG-3.B with Faster R-CNN image feature extraction. 2AMMC is uti- lized as a search model that references various transformers to blend ResNeXt-1G-3.5B and Faster R- R-CNN image features.
Yiran Xing, Zai Shi, Zhao Meng, Gerhard Lakemeyer, Yunpu Ma, and Roger Wattenhofer. 2021. Km-bart: Knowledge enhanced multimodal bart for visual com- monsense generation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 525-535.
BlenderBot(Roller et al., 2021): BlenderBot is model with sequence- -o--nc------suuenee structure. It was also pretrained on large dataset. only takes single modal representation, text, as input.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015, Show, attend and tell: Neu- ral image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR.
Multi-Modal SSthutte et al., 2021): This is a multimodal extension of the BlenderBot model. MMB used BlenderBot's 2.7B model and domain-adaptive adaptive training to allow a model trained single modality to receive multimodal infor- mation.
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021. A unified generative framework for aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 24 2416-2429.
B Implementation Details.
BlenderBot was used as the backbone model. Un- like MBB, we did not use the 2.7B BlenderBot model, but rather adopted smaller 400M Blender- Bot model. The default hyperparameter values of the model were used without modification for com- parison with MBB. The warm-up up task was trained for 20 epochs, batch size was set to 16, and number of accumulation steps was set to 126. The model trained with the warm-up task is fine-tuned for the target task, Image-Chat. For fine-tuning, set epoch to 10, batch size to 32, and accumulation step to 8. For BART, the warm-up task runs for 10 epochs, the batch size is set to 32, and the accumulation step is set to 2. For fine-tuning, we set the epoch to and the batch size to 64. We used Adam Adam ((Loscchllo and Hutter, 2017) as the optimizer and addition- ally OneCycleLR and Topin, 2019). All im- plementations for this experiment were done via Pytorch.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Fei Yu, Jiji Tang, Weichong Yin. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208- 3216.
Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multi- modal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347-14355.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation.
A Compare with existing models
DizloGPTTZhang et al.): Additional social media data was used to perform conversational neural


Reviews:
[{"review_id": "BhAyMQ6eNR", "reviewer": "Reviewer_CkdE", "summary": "", "questions": "Q1. Can you provide human evaluation for a small dataset with your approach?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper. I greatly appreciate your constructive comments and suggestions, which I believe have significantly improved the quality of my work. I have addressed your queries below:\n* * *\n> (R) 2.The model is dependent on the quality of the generated caption which might affect the performance in real situations.\n\nThe VLAW-MDM framework introduces a new strategy for the efficient training of multi-modal dialogue models. Our initial objective was to quantitatively evaluate and validate the algorithm's performance. While we've confirmed the multi-modal learning capabilities of the model through quantitative evaluations, we acknowledge that these metrics alone do not provide a comprehensive validation of the model's interpretability and clarity.\n\nThis framework not only integrates image and text information but also incorporates automatically generated caption data. If the captions are inaccurate, they could adversely affect the quality of the generated dialogue and the overall user experience. As a result, qualitative assessments are essential and will be a primary focus in the next research phase.\n\nWe plan to include a section in the paper's appendix that 'analyzes and categorizes error types associated with inaccurate caption information.'\n\nAdditionally, to understand the dependency on caption quality in our proposed model, we'll examine the dialogue generation results, differentiating between instances where captions are well-generated and those where they are not.\n\nWe commit to sharing all analysis results transparently with the research community. We believe that this will spotlight areas requiring attention in future studies.\n* * *\n> (Q) 1. Can you provide human evaluation for a small dataset with your approach?\n\nWe aim to introduce criteria for qualitative assessment based on frequent questions from reviewers. To this end, we are designing a framework for qualitative evaluation. Using this framework, we will carry out qualitative assessments. The evaluation criteria will focus on the naturalness of utterances, the image's reflection in the dialogue, and the accurate representation of style. We intend to score the model based on each of these criteria. We believe that this qualitative assessment will address many reviewers' concerns, and we will incorporate the results in our paper.\n\nWe will transparently share the code, datasets, and experimental results for all models with the research community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The following are the reasons to accept the paper -\n\n1. The paper is well-written and easy to follow.\n\n2. It provides detailed study for the impact of each of the warm-up tasks.\n\n3. The paper also provides a comparison for the Image-Chat with the existing model and their approach. ", "reasons_to_reject": "The following are the weakness of the paper -\n\n1. It is missing human evaluation which is important for the evaluation.\n\n2. The model is dependent on the quality of the generated caption which might affect the performance in real situations.\n ", "questions_to_authors": "Q1. Can you provide human evaluation for a small dataset with your approach?", "paper_topic_and_main_contributions": "The paper presents a new learning strategy called vision-language warm-up tasks for multimodal dialogue models. This strategy doesn't require the use of large pre training or multi-task datasets but rather relies solely on learning from target data. Moreover, the approach automatically generate captions for images and incorporate them into contextualization of visual information.\n\nThe following are the main contributions of the paper -\n\n1. A framework for vision-language warm-up tasks in multimodal dialogue models, called VLAW-MDM, and describe the process of warming up the model using only data from the target task.\n\n2. The paper introduces four different warm-up tasks (MLM, ISP, MRM, and GCD) and experimentally evaluated how they each affected the performance of the model.The results show that the best performance was achieved when all four warm-up tasks were utilized together.\n\n3. The results show that the framework incorporating caption information was effective for training a multimodal open-dialog model.\n\n4. It also evaluates warm up tasks in the absence of caption information.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kwvxY1hhaT", "reviewer": "Reviewer_v7if", "summary": "", "questions": "(Q)A. Does your method also improve performance on the larger version (2.7B) of Blenderbot? \n\n(Q)B. Could you explain the large difference in the BLEU and ROUGE scores of the Dodeca model compared to your method (shown in Table 5)? \n\n(Q)C. Regarding Table 3, it looks like the increment caused by the MRM task is incremental (or in some cases even degrading) compared to the other warm-up tasks. Could you report how much improvement the MRM task causes if no other warm-up tasks are used?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper. I greatly appreciate your constructive comments and suggestions, which I believe have significantly improved the quality of my work. I have addressed your queries below:\n* * *\n> (R)A. The authors claim that their warm-up process aligns text and images using only the target data without any pretraining.\n\nThe primary objective of this paper is to construct an effective multi-modal dialogue model utilizing solely the target domain dataset. A pivotal step in achieving this involves extracting robust representations from images for multi-modal tasks. Conventionally, this representation extraction relies on pre-trained models familiarized with the relevant domain. In line with this tradition, our study introduces the pre-trained CLIP and BLIP-2 models, effectively harnessing them for image representation and caption information extraction.\n\nThe deployment of these pre-trained models is instrumental in elevating the performance of another significant innovation presented in this research: the warm-up tasks and the integrated framework. This design choice grants us the flexibility to seamlessly upgrade the model as more advanced pre-trained models emerge.\n\nConsequently, this research proposes a technique to enhance the performance of a multi-modal dialogue model using only the target data, without dependence on other pre-trained datasets or multi-task data. Incorporating pre-trained models is a key strategy to realize this ambition, and it aligns harmoniously with the overarching aim of our investigation.\n* * *\n> (R)B. The authors do not show any examples of captions generated by their model. \n\nThe VLAW-MDM framework presents a new strategy for efficiently training multi-modal dialogue models. The initial goal of this research was to quantitatively evaluate and validate the performance of the algorithm. While we have verified the multi-modal learning capabilities of the model through quantitative evaluation, we recognize that these numbers alone do not provide sufficient validation for the model's interpretability and clarity.\n\nIn particular, this framework integrates not only image and text information but also automatically generated caption information into the model. If the accuracy of such captions is lacking, it could negatively affect the quality of the generated dialogue and the overall user experience. Therefore, qualitative evaluation is also necessary and will be an important consideration in the next stage of research.\n\nTo conduct an in-depth analysis of this issue, we plan to add a section in the paper's appendix that 'analyzes and categorizes error types associated with inaccurate caption information.' Additionally, we also intend to include a qualitative evaluation section in the paper.\n* * *\n> (Q)A. Does your method also improve performance on the larger version (2.7B) of Blenderbot?\n\nOur paper emphasizes the use of the VLAW-MDM strategy to boost the performance of small models, especially in resource-limited settings. This focus, however, does not imply the method's inapplicability to larger models. The primary objective of the VLAW-MDM strategy is to enhance performance by efficiently leveraging data. Therefore, in theory, its benefits should extend to models of any size.\n\nIn the scope of our current research, we limited our experiments to smaller models due to constraints in our environment: warm-task learning and fine-tuning with BlenderBot 2.7B were not viable. Future research will delve into the strategy's effectiveness on larger models. Given the notable performance boost observed in the 400M BlenderBot versus the 110M BART model, as demonstrated in Table 3, we anticipate similar enhancements with the 2.7B version of BlenderBot. We hope that by releasing our model code, the broader research community will be better positioned to evaluate and adopt our methodology.\n\n* * *\n> (Q)B. Could you explain the large difference in the BLEU and ROUGE scores of the Dodeca model compared to your method (shown in Table 5)?\n\nThe Dodeca model's performance enhancement primarily stems from an exhaustive hyper-parameter search tailored for the Image-Chat target dataset, as well as specific token generation settings. These findings are detailed in the MultiModal BlenderBot paper[1]. In particular, we found that a shorter minimum generation length boosts the BLEU-4 score, while a longer minimum generation length (50 tokens) is favorable for the ROUGE-L score.\n\nConversely, in our research, we aligned with the MultiModal BlenderBot study[1], adopting BlenderBot's default settings and implementing our VLAW-MDM strategy (referred to as \"Our approach\") on top of that foundation. This means our framework enhances the model's performance through our proposed warm-up tasks, without the necessity for the parameter search that Dodeca undertook. Remarkably, we witnessed a performance surge even in a more compact BlenderBot version (400M) in comparison to the MultiModal BlenderBot (2.7B) — a claim supported by the data in Table 5.\n\nConsequently, the variances in BLEU-4 and ROUGE-L scores can be attributed largely to the differing research methodologies and settings between Dodeca and our approach. It's worth noting that these differences arise naturally from the distinct objectives and scopes characterizing each methodology.\n\n[1] Shuster, Kurt, et al. \"Multi-Modal Open-Domain Dialogue.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n* * *\n> (Q)C. Regarding Table 3, it looks like the increment caused by the MRM task is incremental (or in some cases even degrading) compared to the other warm-up tasks. Could you report how much improvement the MRM task causes if no other warm-up tasks are used?\n\nAs shown in Table 3, MRM (Masked Region Modeling) sometimes exhibits reduced efficacy when combined with other warm-up tasks. This could indicate that the MRM task might be less crucial for enhancing Image-Chat performance compared to other tasks. Nevertheless, this observation doesn't undermine the intrinsic value of the MRM task.\n\nWe intend to further probe the effects of MRM in isolation, without integrating other warm-up tasks, through subsequent experiments. These will provide clearer insights into the standalone impact of MRM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(A)A. The authors show improved performance on the Image-Chat dataset despite using a smaller model than the current SOTA on 2/3 metrics. \n\n(A)B. The work tries to introduce a method that will be useful in improving smaller models in resource-constrained settings. \n\n(A)C. The authors have conducted detailed ablation studies explaining the impact of each warm-up task on overall performance. \n\n(A)D. The authors have clarified the impact of the caption generation task, which is the most significant contributor amongst the warm-up tasks.", "reasons_to_reject": "(R)A. The authors claim that their warm-up process aligns text and images using only the target data without any pretraining. However, they utilize the BLIP-2 model for generating captions from target data which they then use in the Generation Captioning Warmup task as well as in the ablation experiment shown in Table 4. The BLIP-2 model is a large model which has been pretrained on a large amount of data (Both the Visual and Language components). They also show in Table 4 that adding captions has a larger improvement in performance (F1-score) compared to the increment created by the other three warmup tasks (MLM, MRM, and ISP). This means that the greatest improvement is caused by using captions generated by a pretrained model. Thus I think the claim that they do not use any pretraining or data sources is not entirely valid. \n\n(R)B. The authors do not show any examples of captions generated by their model. They only report quantitative findings based on the F-1, BLEU, and ROUGE metrics. The authors should include a few examples of good and bad dialog utterances generated by their method on the image-chat dataset. It might also be a good idea to include some captions that were automatically generated for GCP. \n\n(R)C. There is no explicit mention of what happens when their method is used with the large version of Blenderbot. Is this method only useful for smaller models? It stands to reason that their method should also improve the performance of the 2.7B version of Blenderbot. The authors should clarify whether this is the case.", "questions_to_authors": "(Q)A. Does your method also improve performance on the larger version (2.7B) of Blenderbot? \n\n(Q)B. Could you explain the large difference in the BLEU and ROUGE scores of the Dodeca model compared to your method (shown in Table 5)? \n\n(Q)C. Regarding Table 3, it looks like the increment caused by the MRM task is incremental (or in some cases even degrading) compared to the other warm-up tasks. Could you report how much improvement the MRM task causes if no other warm-up tasks are used?", "paper_topic_and_main_contributions": "In short, the authors of this paper have proposed a set of warm-up tasks to prime encoder-decoder models for generating multimodal open-domain dialog. The warmup tasks are used before fine-tuning the model. They introduce the following warmup tasks - Masked Language Modeling (MLM), Masked Region Modeling (MRM), Image Swapping (ISP), and Generation Caption (GCP). The authors state that utilizing these tasks before training relatively smaller models improves performance and makes it comparable to the levels of SOTA while only utilizing the target data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A minor grammar error in the abstract - \n\nMoreover, our proposed approach automatically generate captions for images and incorporate them into the model’s input to improve the contextualization of visual information. -> Moreover, our proposed approach automatically generates caption*s* for images and incorporate*s* them into the model’s input to improve the contextualization of visual information."}}, {"review_id": "3TiWF3GTLv", "reviewer": "Reviewer_twFK", "summary": "", "questions": "1. Could you please provide a proper rationale for the lack of qualitative justifications in this research?\n2. It would be highly beneficial if you could share valuable insights into how this research could be extended to tackle domain adaptation problems effectively.\n3. While the work exhibits a competitive quantitative performance, it is equally important to assess its computational efficiency. Could you provide some benchmarks for the computational cost?\n4. Although this work showcases strong performance, it appears that reproducibility might pose some challenges due to its complexity and lack of sufficient explanation. Can you confirm whether the authors plan to make this work, along with the dataset, accessible to the public in the future? This would be of great interest to the research community.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper. I greatly appreciate your constructive comments and suggestions, which I believe have significantly improved the quality of my work. I have addressed your queries below:\n* * *\n> Could you please provide a proper rationale for the lack of qualitative justifications in this research?\n\nThe VLAW-MDM framework introduces a new strategy for the efficient training of multi-modal dialogue models. Our initial objective was to quantitatively evaluate and validate the algorithm's performance. While we've confirmed the multi-modal learning capabilities of the model through quantitative evaluations, we acknowledge that these metrics alone do not provide a comprehensive validation of the model's interpretability and clarity.\n\nThis framework not only integrates image and text information but also incorporates automatically generated caption data. If the captions are inaccurate, they could adversely affect the quality of the generated dialogue and the overall user experience. As a result, qualitative assessments are essential and will be a primary focus in the next research phase.\n\nWe plan to include a section in the paper's appendix that 'analyzes and categorizes error types associated with inaccurate caption information.'\n\nAdditionally, to understand the dependency on caption quality in our proposed model, we'll examine the dialogue generation results, differentiating between instances where captions are well-generated and those where they are not.\n\nWe commit to sharing all analysis results transparently with the research community. We believe that this will spotlight areas requiring attention in future studies.\n* * *\n>  It would be highly beneficial if you could share valuable insights into how this research could be extended to tackle domain adaptation problems effectively.\n\nWhile this paper emphasizes multimodal dialogue models, we posit that our methodology can be applied to other multimodal tasks as well.\n\nThe VLAW-MDM approach relies solely on the target dataset specific to a domain for its warm-up phase, negating the need for distinct pre-training data or multi-task datasets. This strategy is expected to more accurately capture the nuances of the target domain.\n\nDistinct from other research methodologies, ours does not incorporate learning from data outside the designated target set. Essentially, our results can be achieved without external data. We anticipate our approach to be versatile, potentially benefiting other domains. To substantiate the efficacy of our methodology across various domains, we intend to release both the code and dataset to the broader research community. This initiative not only augments the transparency of our research but also showcases the practicality and dependability of our approach.\n* * *\n> While the work exhibits a competitive quantitative performance, it is equally important to assess its computational efficiency. Could you provide some benchmarks for the computational cost?\n\nOne of the big advantages of VLAW-MDM is that it only utilizes the target dataset. This is expected to significantly save computational costs, but clear benchmarks for this have not been included in other comparative papers. However, the points raised by the reviewer could serve as important indicators for evaluating the model's scalability and efficiency. Therefore, we plan to add measurements for computational efficiency to the paper.\n* * *\n> Although this work showcases strong performance, it appears that reproducibility might pose some challenges due to its complexity and lack of sufficient explanation. Can you confirm whether the authors plan to make this work, along with the dataset, accessible to the public in the future? This would be of great interest to the research community.\n\nWe agree that the paper lacks a sufficient explanation of the methodology and results presented. This is a critical element for ensuring reproducibility, so we plan to address this in the future. \n\nTo enhance transparency, we will release the code and dataset to the research community. Furthermore, we aim to supplement this with a tutorial or guide detailing the model architecture, hyperparameter settings, and warm-up tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Authors of this work address a prominent machine learning challenge in multimodal dialogue modeling, the scarcity of available training data. Many existing approaches require a substantial amount of fine-tuning data, which is often unavailable in practical application scenarios. The proposed framework by the authors is of significant value and practical importance to the field.\n\nTo tackle this issue, the authors introduce four distinct warm-up tasks (Masked Language Modeling, Image-Text Matching, Multimodal Retrieval Matching, and Guided Conversational Decoder) and conduct a comprehensive evaluation to determine their impact on model performance. Through extensive experimentation, the authors identify the most robust learning algorithm for their work. The empirical findings suggest that optimal performance is achieved when all four warm-up tasks are utilized in conjunction. Additionally, incorporating caption information proves effective in training a multimodal open-domain dialogue model. \n\nThe proposed solution has been thoroughly validated through well-structured experiments, showcasing superior performance compared to existing state-of-the-art solutions in the field.\n\n\n", "reasons_to_reject": "Although this study has been quantitatively demonstrated as effective and superior compared to existing solutions, it is crucial to emphasize the significance of assessing the interpretability and clarity of its responses, given the nature of the dialogue model. Therefore, the authors should enrich their discussions with sufficient qualitative results that provide a comprehensive and illustrative evaluation of the practical performance of this work. This additional qualitative analysis is essential to ascertain the validity of the claimed performance.\n\nMoreover, the domain adaptation ability of this work or mainly the model's generalization ability requires further justification. Many previous studies have demonstrated effectiveness by utilizing additional data for fine-tuning purposes. As this work's primary contribution lies in addressing data insufficiency issues. Therefore, it becomes important for the authors to substantiate the effectiveness of their proposed learning strategies when using a limited amount of data in other domains. This validation will enhance the credibility and practical applicability of the proposed approach.\n\nLastly, the methodology section lacks comprehensive explanation. To enhance reproducibility and clarity, the incorporation of an algorithm box is recommended. Such an addition would succinctly outline the detailed workflow of the study, empowering readers to replicate the experiment with precision and ease.", "questions_to_authors": "1. Could you please provide a proper rationale for the lack of qualitative justifications in this research?\n2. It would be highly beneficial if you could share valuable insights into how this research could be extended to tackle domain adaptation problems effectively.\n3. While the work exhibits a competitive quantitative performance, it is equally important to assess its computational efficiency. Could you provide some benchmarks for the computational cost?\n4. Although this work showcases strong performance, it appears that reproducibility might pose some challenges due to its complexity and lack of sufficient explanation. Can you confirm whether the authors plan to make this work, along with the dataset, accessible to the public in the future? This would be of great interest to the research community.", "paper_topic_and_main_contributions": "Authors introduce a novel learning algorithm, VLAW-MDM, for constructing robust multimodal dialogue models. Unlike existing approaches that rely on extensive pretraining or multi-task datasets, VLAW-MDM solely leverages the target dataset for learning. Furthermore, it automatically generates image captions, enhancing the contextualization of visual information in the model's input. The proposed solution is extensively evaluated through empirical studies, demonstrating its superiority over state-of-the-art methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5xhEJBwsIY", "reviewer": "Reviewer_qMM1", "summary": "", "questions": "1. What is the performance with the proposed method on semantic similarity based metric such as BERT-score or other trained similarity metric. Lexical similarity based metrics are not reliable for evaluating open-ended text generation tasks, especially in this task were the dialogue needs to follow a given style.\n2. All the proposed methods are from 2020, it would be interesting to see comparison to more recent works given this field is exploding.\n3. The ROGUE-L score seems to be quite low compared to other models, what can be the potential reason.\n4. Try to present some error analysis, it is especially useful for open-ended generation tasks.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper. I greatly appreciate your constructive comments and suggestions, which I believe have significantly improved the quality of my work. I have addressed your queries below:\n* * *\n> What is the performance with the proposed method on semantic similarity based metric such as BERT-score or other trained similarity metric. Lexical similarity based metrics are not reliable for evaluating open-ended text generation tasks, especially in this task were the dialogue needs to follow a given style.\n\nMetrics rooted in semantic similarity, such as BERT-score, are believed to offer valuable insights when evaluating the quality of generated utterances. We concur that leveraging such measures could enrich our evaluation process. For this study, our primary emphasis was on traditional evaluation metrics, aiming to ensure straightforward comparisons with extant research and outcomes. Earlier studies employed evaluation techniques like F1, BLEU-4, and ROUGE-L scores for Image-Chat [1, 2, 3]. Consequently, we assessed the efficacy of our framework through these well-established metrics. Nonetheless, as highlighted by the reviewer, these traditional methods may fall short in capturing stylistic nuances. We echo these sentiments and are poised to incorporate findings from semantic similarity metrics, such as BERT-score, into our paper.\n\n[1] Shuster, Kurt, et al. \"Image-Chat: Engaging Grounded Conversations.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n[2] Shuster, Kurt, et al. \"The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n[3] Shuster, Kurt, et al. \"Multi-Modal Open-Domain Dialogue.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n* * *\n> All the proposed methods are from 2020, it would be interesting to see comparison to more recent works given this field is exploding.\n\nRecent methodologies frequently employ large LLM models, such as LLaMA, as their foundational backbone. Given that LLM models were originally designed to accept solely text inputs, they necessitate extra training to accommodate images. This is addressed through additional preprocessing, as detailed in [4]. Such strategies harness the vast knowledge encapsulated in the LLM models. However, they still mandate substantial resources for the LLM model itself. Compared to the model in our study, these methodologies continue to consume a considerable amount of resources. Regrettably, our environment posed constraints, preventing us from experimenting with these large-scale, state-of-the-art models. Moreover, the latest model designed for the Image-Chat dataset in the realm of multimodal models is the MultiModal BlenderBot [3], against which we have drawn comparisons.\n\n[4] Zhu, Deyao, et al. \"Minigpt-4: Enhancing vision-language understanding with advanced large language models.\" arXiv preprint arXiv:2304.10592 (2023).\n* * *\n> The ROGUE-L score seems to be quite low compared to other models, what can be the potential reason.\n\nWe posit that the observed discrepancy stems from our model's incorporation of captions. As illustrated in Table 5, our proposed model significantly outperforms the baselines in the F1 score, while its ROUGE-L score lags behind. We attribute the elevated F1 score to the model's adeptness in pinpointing objects or entities within the image via the generated captions, subsequently weaving this information as nouns into the generated response. On the other hand, the subdued ROUGE-L score might arise from variations in portraying actions or states associated with these nouns, particularly in terms of verbs or adjectives, which might not align perfectly with the reference.\n\n* * *\n> Try to present some error analysis, it is especially useful for open-ended generation tasks.\n\nIn response to recurrent queries from several reviewers, we are devising a framework for qualitative evaluation, which we will detail in a new section. This assessment will focus on the naturalness of utterances, the model's ability to reflect images, and the appropriate representation of style. For each criterion, we will assign scores to gauge the model's performance. We are confident that the results from this qualitative analysis will address the concerns expressed by reviewers, and we will integrate these findings into the paper.\n\nFurthermore, we pledge full transparency by releasing all codes and datasets for each model, accompanied by our experimental results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Underlying idea of using prediction from an external system to improve the performance of the target system is interesting.\n3. Proposed method shows improvement over some larger baseline models.\n3. Ablation study is thorough.", "reasons_to_reject": "1. The performance gains are not very high, more most of the metrics the different between the baseline (w/o caption + w/o warmup) and best approach (with caption + warmup) is less than 1%.\n2. The paper lack error analysis and model output examples. ", "questions_to_authors": "1. What is the performance with the proposed method on semantic similarity based metric such as BERT-score or other trained similarity metric. Lexical similarity based metrics are not reliable for evaluating open-ended text generation tasks, especially in this task were the dialogue needs to follow a given style.\n2. All the proposed methods are from 2020, it would be interesting to see comparison to more recent works given this field is exploding.\n3. The ROGUE-L score seems to be quite low compared to other models, what can be the potential reason.\n4. Try to present some error analysis, it is especially useful for open-ended generation tasks.", "paper_topic_and_main_contributions": "Paper proposes multiple auxiliary warm-up tasks from the target data in order to reduce the reliance on external pre-training data, for the task multimodal multi-turn dialogue generation. For a given image + multi-turn dialogue instance, the proposed method first generates an image caption from an external system. Then they use the image, image-caption, multi-turn dialogue instance to create four different warm-up tasks: \n(i) Generation Captioning - generate the caption conditioned on the image and dialogue utterances \n(ii) Image-Swapping - predict if a focus patch is original or fake\n(iii) Masked Region Modeling - predict the feature of a masked image patch\n(iv) Masked Language Modeling - predict the masked tokens of the dialogue utterances\n\nAfter the wam-up training the model is fine-tuned on the target task. \n\nExperimental results show that training on warm-up task created from the target dataset, helps improve model performance without the warm-up based training.\n\nMy hypothesis is that the introducing caption is indirectly leading to distillation/transfer of some knowledge from the caption generation model, which might be complimentary to the target task model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Include Error analysis.\n2. Include semantic based similarity metric based performance."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel learning strategy termed vision-language warm-up tasks for multimodal dialogue models (VLAW-MDM). Remarkably, this strategy avoids dependence on large pretraining or multi-task datasets and instead focuses on learning directly from the target data. The authors propose four distinct warm-up tasks. Furthermore, VLAW-MDM can autonomously generate captions for images, integrating them into the model's input to better contextualize visual information. Within constraints of limited data and smaller model configurations, experimental results indicate that VLAW-MDM offers comparable, and occasionally superior, performance against existing state-of-the-art models on the ImageChat benchmark.

**Soundness Scores**: (3, 3, 3, 3)
The presented work exhibits moderate soundness. While the authors have a detailed ablation study for the four warm-up tasks, the experiments are confined to a single benchmark. Notably absent are a case study and error analysis. Additionally, the methodology section lacks a thorough exposition, potentially impeding reproducibility.

**Excitement Scores**: (3, 3, 3, 3)
The excitment towards the paper is moderate. The majority of reviewers appreciate the novel direction of relying solely on target data for pretraining and recognize the uniqueness of the techniques introduced. However, the performance improvements observed are marginal. The general applicability of the proposed techniques remains questionable due to an absence of supporting justification. This gap makes it challenging to ascertain the broader utility of the introduced methods.

In summary, this work is **moderately sound and moderately exciting** .
Decision: Accept-Main