Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models (2024)

Abstract: Recent advancements in dialogue systems have highlighted the significance of integrating multimodal responses, which enable conveying ideas through diverse modalities rather than solely relying on text-based interactions. This enrichment not only improves overall communicative efficacy but also enhances the quality of conversational experiences. However, existing methods for dialogue-to-image retrieval face limitations due to the constraints of pretrained vision language models (VLMs) in comprehending complex dialogues accurately. To address this, we present a novel approach leveraging the robust reasoning capabilities of large language models (LLMs) to generate precise dialogue-associated visual descriptors, facilitating seamless connection with images. Extensive experiments conducted on benchmark data validate the effectiveness of our proposed approach in deriving concise and accurate visual descriptors, leading to significant enhancements in dialogue-to-image retrieval performance. Furthermore, our findings demonstrate the method’s generalizability across diverse visual cues, various LLMs, and different datasets, underscoring its practicality and potential impact in real-world applications.1

heading: 1 Introduction
text: In recent years, the landscape of online conversations has undergone a significant transformation thanks to the proliferation of instant messaging tools. Unlike the past, when these exchanges were confined to text alone, today’s conversations have evolved into a multimodal experience, incorporating elements like images and speech. The various communication modes not only enhance engagement but also prove invaluable for conveying complex information that can be challenging to communicate solely through text. Sun et al. (2022) highlighted the advantages of integrating images
1https://github.com/MiuLab/VisualDialog
into conversations. For example, when discussing a topic with someone who may not grasp the concept, sharing an image can provide visual clarity for better comprehension. Additionally, when precision is required to convey specific details about a subject, relevant images can be a more effective means of communication than text alone (Hsu et al., 2023). Consequently, the ability to generate responses using images is a crucial area of research in enhancing automatic dialogue systems. To equip these systems with the capacity to respond using images, a common method involves text-to-image retrieval, as demonstrated by previous work (Liao et al., 2018; Zang et al., 2021). In this approach, a model selects an appropriate image from a preconstructed image repository based on the context of the ongoing conversation.
As storage costs decline and computational power advances, vision foundation models pretrained on large-scale, open-domain image-text pairs have emerged (Radford et al., 2021; Jia et al., 2021; Yuan et al., 2021). These models have demonstrated outstanding performance in text-toimage retrieval tasks, excelling in both zero-shot and fully-trained scenarios. However, despite their impressive capabilities, these pre-trained visionlanguage models (VLMs) still come with some limitations. One significant limitation is their suboptimal design for handling complete dialogue contexts effectively. Often, they suffer from extracting key information comprehensively from the entire conversation. Table 1 presents an illustrative example, where a dialogue-to-image model fine-tuned from CLIP (Radford et al., 2021) fails to correctly interpret the dialogue’s intent. This highlights the challenge of dialogue comprehension, a task for which pre-trained VLMs may not be adequately equipped. Additionally, most existing VLMs typically impose input text length constraints during their pre-training stages, preventing them from processing the entirety of the dialogue context directly.
11777
This constraint can lead to the loss of crucial contextual information, potentially undermining the model’s overall performance.
Inspired by Menon and Vondrick (2022), we leverage the reasoning capabilities of large language models (LLMs) to generate the visual descriptor for the dialogue context. These descriptors encapsulate speculations about the image that the speaker intends to share, aiming to provide concise and precise cues for better text-to-image retrieval. Our objective is to address the aforementioned limitations and enhance task performance. Given that most vision models excel at identifying objects, scenes, and other visual elements in images (Kuznetsova et al., 2020; Zang et al., 2021), we employ a set of visually-focused queries, such as main subject and background scene, to bridge the gap between the ongoing dialogue and the pool of potential image candidates. These queries serve as templates for the LLM to predict corresponding visual cues based on the dialogue context. We then utilize these queries and their resulting answers as dialogue-associated visual descriptors, as illustrated in the bottom part of Table 1. Our ex-
periments on the benchmark dataset showcase the exceptional performance of our approaches, surpassing all previous results. In addition to demonstrating the effectiveness of our LLM-generated visual descriptor, we compare it with other descriptor creation methods and conduct an in-depth analysis to evaluate the efficacy of each proposed query.
Our contributions can be summarized as 3-fold: • This paper introduces a novel approach for
retrieving associated photos in dialogue systems, leveraging the reasoning capabilities of LLMs to generate visually-focused cues for improved image retrieval. • We design a series of visually-focused queries based on common image features, employing them to construct conversation descriptors. Our experiments validate the effectiveness of these designed queries. • The proposed approach achieves state-ofthe-art performance on multiple benchmark datasets.

heading: 2 Related Work
text: Multimodal Dialogue Systems Recent years have witnessed a notable shift in research towards multimodal dialogues, moving beyond the confines of text-only interactions (Liu et al., 2022). While the exploration of image-grounded conversations, where textual dialogues are generated from images, has gained traction (Yang et al., 2021; Shuster et al., 2021; Chen et al., 2023; Gong et al., 2023; Zhu et al., 2023; Liu et al., 2023), an increasing number of studies are delving into the incorporation of multimodal responses within dialogue systems. This multimodal evolution enables human-machine conversations to reflect real-life human-human interactions and communicate concepts that are difficult to convey through text alone. For instance, Liao et al. (2018) introduced a task-oriented multimodal dialogue system featuring a taxonomy-based learning module that captures nuanced visual semantics and employs reinforcement learning to ensure response coherence. Further advancements in robotic technology now enable robots to more effectively understand and respond to human requests in life support scenarios. This requires the interpretation of both the visual context of the environment and the user’s verbal communication, integrating multimodal capabilities into dialogue scenarios (Tanaka et al., 2024; Tsai et al., 2024). Moreover, Sun et al. (2022) introduced a frame-
work capable of directly generating multimodal responses via a text-to-target-modality generator. In contrast, rather than directly generating multimodal responses, Zang et al. (2021) achieve multimodal responses by employing image retrieval models to select appropriate images from a pre-existing image repository. For better practicality, our paper centers on the same task—recommending a suitable image from the user’s image repository based on the ongoing dialogue context.
External Knowledge of LLMs for Visual Tasks Many studies have showcased that the commonsense knowledge and reasoning capabilities in large language models (LLMs) can significantly augment the performance of visual tasks. For instance, Tsimpoukelli et al. (2021) confirmed that by projecting image encodings into the embedding space of an LLM, it becomes possible to harness the rich knowledge contained within the LLM for few-shot visual question answering (VQA) tasks. Similarly, Zeng et al. (2022) introduced Socratic Models, which leverages multiple pre-trained large models trained on data from diverse domains. By translating non-language domain information into textual prompts, Socratic Models achieve state-ofthe-art results in zero-shot image captioning and video-to-text retrieval tasks. Furthermore, Menon and Vondrick (2022) took a novel approach by obtaining visual features for different categories through queries to GPT-3 (Brown et al., 2020) using category names. These textual descriptors are then employed as internal representations for
zero-shot visual classification and text-to-image retrieval tasks. Our work centers on harnessing the reasoning capabilities of LLMs to derive contextually relevant visual descriptions for shared photos within the dialogue context. Different from the prior work based on non-language domains or single sentences, our approach focuses on the nuanced domain of photo sharing within conversations, which presents unique challenges due to its reliance on commonsense knowledge and an understanding of human-human interactions.

heading: 3 Methodology
text: Our objective is to select an image from a preconstructed photo set {(vj , oj)}mj=1 given a dialogue context D, where vj represents an image candidate and oj lists the objects appearing in vj . Note that the object lists can be obtained through object detection in the pre-processing stage, and we treat this object information as given data.
Figure 1 illustrates the proposed framework, which introduces an innovative approach to estimate retrieval scores for each image candidate within a dialogue context. These scores are based on two criteria: scene-aligned and vision-aligned scores, both relying on visual descriptors. The scene-aligned score assesses whether the speculated visual cues align with the image-associated objects in a textual format. In contrast, the visionaligned score evaluates the alignment between the visual description and the image using visionlanguage models.

heading: 3.1 Dialogue-Associated Visual Descriptor
text: Considering that visual descriptors can significantly enhance the understanding of visual content (Menon and Vondrick, 2022), we focus on generating dialogue-associated visual descriptors to improve image retrieval capabilities. To create high-quality visual descriptors that can connect with visual elements in the photo, we define a set of visually-focused queries, denoted as Q = {qi}. These queries encompass various visual attributes related to an image, such as main subject and background scene, which are instrumental in linking the target photo to the dialogue.
Drawing from prior work (Kuznetsova et al., 2020; Zang et al., 2021) and our common experiences, we assume that photos shared in online messaging typically contain components such as main subjects, prominent foreground objects, background scenes, events, and materials and attributes, as detailed in Table 2. Note that we do not expect all answers to these queries to be perfectly extracted from the dialogue context or found in the ground-truth image. Instead, our goal is to leverage automatically inferred visual descriptors to bridge the gap between the image and the given dialogue context.
Leveraging the powerful reasoning capabilities of large language models (LLMs) (Touvron et al., 2023), we construct a prompt comprising the dialogue D and the set of queries Q and input it into the LLM. This process yields a set of dialogueassociated visual descriptors in a zero-shot manner:
desc = LLM(D,Q). (1)
For instance, a generated visual descriptor regarding the main subject might read, “The main subject of the photo is a group of friends.” The used prompts can be found in Appendix A.

heading: 3.2 Image Relevance Estimation
text: To measure the relevance of each image candidate in the context of a given dialogue D, we calculate two retrieval scores based on their gener-
ated visual descriptors desc: Sscene(oj , desc) and Svision(vj , desc). The former score assesses if the objects in the photo candidate align with the inferred visual descriptors in their text-only forms, referred to as the scene-aligned score. The latter score evaluates if the photo candidate matches the visual descriptions through multimodal methods, termed the vision-aligned score.

heading: 3.3 Image Retrieval Learning
text: Our task involves retrieving the target image from a pre-constructed photo set, and it can be approached in two settings: 1) zero-shot and 2) training with contrastive leanrning.

heading: 3.3.1 Zero-Shot
text: Using the descriptor desc derived from the dialogue context D, we employ a pre-trained vision language model (VLM) for zero-shot text-to-image retrieval. This process yields two scores through its text encoder and image encoder, as illustrated in Figure 1. The final retrieval score is calculated as:
Sscene(oj , desc) + λ · Svision(vj , desc), (2)
where λ is a weighting parameter. The image with the highest score is selected in a zero-shot manner.

heading: 3.3.2 Contrastive Learning
text: To further enhance retrieval performance, we finetune the VLM model using the training set. Following the pre-training stage outlined by Radford et al. (2021), we apply contrastive learning to optimize our dialogue-image retriever. During training, we randomly sample a minibatch of dialogueassociated descriptors and photo pairs, designating (desc, v∗, o∗) as the positive example, while the remaining (b − 1) examples within the minibatch serve as negative examples. The contrastive losses are calculated separately for the scene and vision components, focusing on aligning dialogue-
associated visual descriptors and the target photo.
Lscene = − log exp(Sscene(o ∗, desc)/τ)∑ j∈b exp(Sscene(oj , desc)/τ) ,
Lvision = − log exp(Sscene(v ∗, desc)/τ)∑ j∈b exp(Sscene(vj , desc)/τ) ,
where τ is the trainable temperature parameter. The final training loss is a combination of these contrastive losses:
L = 1 b
∑ j∈b (Lscene + λ · Lvision), (3)
where λ is a weighting parameter. This approach optimizes our dialogue-image retrieval model through contrastive learning.

heading: 4 Experiments
text: For the LLM in (1), we utilized well-established LLMs with instruction tuning and reinforcement learning from human feedback (RLHF), including LLaMA-2-7B-Chat and LLaMA-2-13B-Chat (Touvron et al., 2023). We employed greedy decoding for generating descriptors to ensure the correct format and reasoning capability. Our pre-trained vision-language model (VLM) backbone is CLIP ViT-B/32, and VLM training is executed on a single NVIDIA GeForce RTX 2080 Ti GPU with a batch size of 56. We utilize the ADAM optimizer with an initial learning rate of 1e-5. The weighting parameter λ was set to 1 to strike a balance between scene-alignment and vision-alignment.

heading: 4.1 Data & Metrics
text: Our approach is evaluated on a benchmark multimodal dialogue dataset called PhotoChat. PhotoChat dataset (Zang et al., 2021) is characterized by open-domain, high-quality multimodal dialogues and comprises 10,917 images paired with 12,286 dialogues. Specifically, the dataset is divided into 10,286 instances for training, 1,000 for validation, and another 1,000 for testing. Each image in the dataset is accompanied by an associated object list presented in textual form. In each data instance, one photo is shared within the context of the conversation.
Given that this task can be formulated as an image retrieval task, we employed Recall@k (R@k) as our evaluation metric. During the training phase, we select the final model based on the highest avg(R@1, R@5, R@10) score on the validation
set. In the testing phase, for each dialogue instance, the trained models retrieved images from the candidate photos in the testing set.

heading: 4.2 Baselines
text: We compare our approach against several established baselines:
• VSE++: Faghri et al. (2018) incorporated hard negatives in the ranking loss to learn visualsemantic embeddings for text-image retrieval. • SCAN: Lee et al. (2018) utilized stacked cross attention to align image regions and words in a sentence and calculate image-text similarity. • Dual Encoder (DE): Previous work (Parekh et al., 2021; Zang et al., 2021) employed a dual encoder architecture, where one encoder processes the image and its object list using CLIP ViT-B/32 for images and FFNN for object features. For the dialogue encoder, two different text encoders were experimented with: CLIP ViT-B/32 Text and BERT (Devlin et al., 2019) with an additional projection to ensure consistent dimensions. The retrieval similarity between the image and dialogue encodings is measured using dot product.

heading: 4.3 Descriptor Variants
text: In addition to the query-based descriptors, we conduct experiments using the following descriptor variants for in-depth analysis:
• Desc - Diag (whole dialogue as descriptors): All utterances are concatenated to form the descriptors, allowing the image retriever to utilize complete cues within the dialogue. • Desc - Caption (caption as descriptors): Inspired by Li et al. (2023), we performed zeroshot image captioning on images in the training set using BLIP-2. We then trained a text generator to create image captions as descriptors based on a given dialogue. • Desc - Summary (summary as descriptors): Descriptors are generated by LLMs based on a dialogue summary, offering a more concise representation of the conversation. • Desc - Guessing (visually-focused guessing as descriptors): LLMs are allowed to speculate about the features of the upcoming shared photo from the dialogue without being constrained by a specific query. • Desc - Queries (visually-focused query descriptors): Utilizing our designed visuallyfocused attributes as descriptors.

heading: 4.4 Results
text: Table 3 provides a comprehensive overview of the results for both zero-shot and fully-trained settings on PhotoChat dataset. In zero-shot scenarios, Desc - Guessing emerges as the top-performing method among all results. Notably, Desc - Queries outperforms Desc - Summary, indicating that visuallyfocused queries and guessing contribute valuable information for linking the desired images. We hypothesize that the poor performance of Desc - Summary is because the LLM summarize all content of the dialogue, rather than focusing solely on the visual information. Consequently, Desc - Summary often includes sentences that do not aid in text-toimage retrieval. As for why Desc - Queries did not perform better than Desc - Guessing, we hypothesize that it is because we construct our descriptors using the templates from 3.1. These sentences tend to be less fluent and are not the types of sentences CLIP was familiar with during pre-training stage. In contrast, Desc - Guessing generates more fluent descriptors without any constraints, making them closer to the sentences seen by CLIP during pre-training stage. This gives Desc - Guessing an advantage in zero-shot scenarios.
In the fully-trained setting, the descriptor-based results (Desc - Summary, Desc - Guessing, Desc - Queries) with LLaMA-2-13B-Chat exhibit similar performance, with Desc - Queries achieving the highest average performance. These results validate the effectiveness of our proposed approach, demonstrating that the generated visual descriptors successfully facilitate the connection between associated images through the LLM’s understanding of dialogue. Additionally, it is evident that LLaMA-

heading: S + G 47.32 69.62 77.63 64.86

heading: S + Q 47.78 68.81 77.61 64.73

heading: G + Q 47.44 68.90 77.15 64.50

heading: S + G + Q 48.79 70.01 78.44 65.75

heading: S + G + Q + C 48.84 70.20 78.74 65.93
text: 2-13B-Chat outperforms LLaMA-2-7B-Chat due to its stronger reasoning abilities for understanding dialogues. When compared to the fully-trained baselines, our proposed descriptor-based methods achieve superior performance even in zero-shot settings, establishing a new state-of-the-art performance achieved by a single model.

heading: 4.5 Ensemble
text: We further conduct experiments on ensemble learning using all descriptor-based results based on the validation set. The results in Table 4 demonstrate that ensemble learning consistently improves performance. Even in cases where the caption model performs poorly in a fully-trained setting, ensemble learning benefits other models. These findings highlight the efficacy of combining various types of descriptors, leading to the best overall performance and establishing a new state-of-the-art for PhotoChat. This suggests that the generated descriptors focus on diverse patterns that can complement each other and enhance scores.

heading: 5 Analysis

heading: 5.1 Effectiveness of Two Alignment Scores
text: Our proposed method incorporates two scores: scene-aligned (text-only) and vision-aligned (multimodal) scores. We conduct an ablation study to assess the impact of each score. Table 5 presents the experimental results. The results show that models trained solely on the scene-aligned score (text-only) perform better in terms of R@1, whereas models trained on the vision-aligned score (multimodal) perform better for R@5 and R@10.
In addition, we conducted experiments with different weighting parameter λ in (2). Figure 2 illustrates the results obtained for various λ values in the zero-shot scenarios. It can be observed that the best performance is achieved when λ approaches 1. This indicates that excessive bias towards either the scene-aligned score or the vision-aligned score leads to a decrease in performance. Interestingly, our findings reveal that the optimal zero-shot performance occurs at λ = 1.2.

heading: 5.2 Object Detection Sensitivity
text: Given our experimental assumption of having available object lists for the images, a degradation in system performance may occur when object detection is poor. Consequently, we explored the sensi-
tivity of error propagation in our proposed method by simulating various error rates in detected objects. Specifically, we randomly removed objects from the gold object list or substituted them with others from the training set to simulate scenarios of “missing” or “incorrect” objects. As shown in Table 6, it is evident that with increasing error rates, regardless of the error type, performance tends to degrade. This suggests that our method is still influenced by error propagation. Moreover, the impact of incorrect objects is particularly severe since they may mislead the model in image selection. Nevertheless, even with a 35% error rate, our results outperform relying solely on the vision-aligned score, indicating a relative robustness to missing objects. Therefore, precision in object detection during object list extraction is more important for our proposed approach.

heading: 5.3 Visually-Focused Query Impact
text: To assess the influence of different visually-focused queries on our results, we conducted experiments by systematically removing individual queries from the original query set (see Table 7). Remarkably, the query concerning the main subject emerges as the most pivotal feature for bridging the dialogue
context and the target image, as its removal leads to a significant decrease in scores. Following in descending order of impact are the queries related to foreground objects, events, background scene, and materials and attributes.
Additionally, we augmented the original query set with two common photo features: atmosphere or mood and lighting.
• Atmosphere or mood: Research by Sun et al. (2022) suggests that photographs convey not only object details but also emotions and ambiance, evoking feelings like happiness, boredom, and coziness. • Lighting: As highlighted by Hunter et al. (2021), the presence and quality of light are fundamental in photographic composition. Our investigation explores whether an LLM can accurately predict lighting conditions within a photo solely based on dialogue context and whether this predictive information enhances text-image retrieval capabilities.
Results indicate that atmosphere or mood improves performance at R@5, while lighting enhances results at both R@1 and R@5 compared to the original set. This suggests that these abstract and challenging-to-predict queries have varying impacts on performance.

heading: 5.4 Generalization to Other Datasets
text: To explore the generalizability of our approach, we conducted experiments on other benchmark
datasets, VisDial (Das et al., 2017) and MMDialog (Feng et al., 2023), with identical settings.
• VisDial (Das et al., 2017): Each instance in VisDial comprises a single image from the COCO dataset (Lin et al., 2014) and a dialogue context consisting of question-answer pairs. During testing, the model’s task is to retrieve the correct image from a pool of 2,064 images in the testing set. • MMDialog (Feng et al., 2023): This dataset comprises multimodal open-domain dialogues gathered from a global social media platform. Instances in the testing set are linked with 999 corresponding negative images. MMDialog uniquely allows dialogue context to contain both text and images. In our experiments, we employed BLIP2 (Li et al., 2023) for image captioning within the dialogue context. These resulting captions replaced the images, enabling textual dialogue context input into the LLM. Our experiments were conducted on a random sample of 1,000 instances from the testing set. The average performance of LLaMA-2-13B-Chat in zero-shot scenarios across different datasets is summarized in Table 8, with detailed scores available in Appendix B. Our findings indicate that our proposed approach consistently outperforms Desc - Diag baseline and even recent work (Koh et al., 2023), showcasing its robust generalization capability across diverse datasets.

heading: 5.5 Qualitative Study
text: To further explore the quality of our generated descriptors, we conducted a qualitative analysis across various models and descriptors. We present an example of a testing dialogue using different LLMs and descriptor generation methods in Table 9. A comparison between LLaMA-2-7B-Chat and LLaMA-2-13B-Chat in the Desc - Queries domain revealed that LLaMA-2-7B-Chat often delivers inaccurate or unrelated answers (highlighted in red). In contrast, LLaMA-2-13B-Chat tends to produce more precise answers and demonstrates an ability to infer information not explicitly present in the dialogue context (e.g., medieval architecture). This discrepancy is likely attributable to differences in reasoning capabilities between the two models.
Both Desc - Summary and Desc - Guessing can accurately describe the features of the photos. However, Desc - Summary occasionally incorporates additional details not directly related to the
photos, such as “A is willing to share these pictures with B,” likely because we instructed the LLM to summarize information about the users sharing the photo rather than solely focusing on the photorelated information. Our analysis suggests that our visually-focused approaches are particularly effective for this task, especially when coupled with the powerful reasoning capabilities of LLMs.

heading: 6 Conclusion
text: This paper presents a novel approach to enable multimodal dialogue systems for seamless photo sharing. By harnessing the reasoning capabilities of LLMs, our method generates precise visual cues from ongoing dialogue contexts. Addressing challenges encountered in prior methods, such as accurately understanding extensive dialogue contexts and handling input length constraints, our approach exhibits clear superiority in experimental results. Additionally, our comprehensive ablation study confirms the effectiveness of text-only vi-
sual descriptors, indicating a promising pathway for bridging intricate dialogues and images through deep dialogue understanding via LLMs. This work not only advances photo sharing within dialogues but also lays groundwork for future sophisticated multimodal dialogue systems.
Limitations
Our method assumes the availability of object detection capabilities during pre-processing to extract object lists associated with the images. This reliance on object detectors may limit the method’s applicability in scenarios where object detection is challenging or unavailable, potentially affecting its performance.
Lastly, our method assumes that the shared images align with the given dialogue context. In cases where users share images that are intentionally misleading, unrelated to the conversation, or dependent on users’ personal information, our method may struggle to retrieve appropriate images, leading to
potential accuracy issues in such scenarios.

heading: Acknowledgements
text: We thank the reviewers for their insightful comments. This work was financially supported by the National Science and Technology Council (NSTC) in Taiwan, under Grants 111-2222-E-002013-MY3, 111-2628-E-002-016, and 112-2223- E002-012-MY5. We thank to National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.

heading: A Prompts
text: The designed prompts for all descriptor-based approaches are shown as follows.
A.1 Desc - Summary
Please read the following dialogue context: <dialogue_context>
Based on the dialogue context, please summarize the information of speaker A.
Answers:
A.2 Desc - Guessing
Please read the following dialogue context: <dialogue_context>
Based on the dialogue context, please describe the photograph shared by speaker A.
Answers:
A.3 Desc - Queries
Please read the following dialogue context: <dialogue_context>
Based on the dialogue context, please describe the photograph shared by speaker A. List the answer in JSON format. - main subject: {simply list the answer by ','} - prominent objects in the foreground: {simply list the answer by ','} - background scene: {one background scene} - events: {simply list the answer by ','} - materials and attributes: {simply list the answer by ','}
Answers:

heading: B Detailed Results for Other Datasets
text: Table 10 presents detailed scores on the VisDial and MMDialog datasets using LLaMA-2-13B-Chat in zero-shot scenarios. It is worth noting that we sampled 1,000 instances from the MMDialog testing data for our experiments, limiting direct comparison with previous work due to the discrepancy of testing sets.

heading: Future Work
text: This work not only advances photo sharing within dialogues but also lays groundwork for future sophisticated multimodal dialogue systems. Limitations
Our method assumes the availability of object detection capabilities during pre-processing to extract object lists associated with the images. This reliance on object detectors may limit the method’s applicability in scenarios where object detection is challenging or unavailable, potentially affecting its performance. Lastly, our method assumes that the shared images align with the given dialogue context. In cases where users share images that are intentionally misleading, unrelated to the conversation, or dependent on users’ personal information, our method may struggle to retrieve appropriate images, leading to
potential accuracy issues in such scenarios