Large Language Models can Share Images, Too! (2024)

Abstract: This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the PHOTOCHAT++ dataset, which includes enriched annotations (i.e., intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve (DRIBER) framework. With extensive experiments, we unlock the image-sharing capability of DRIBER equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance. Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of DRIBER. We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available 1.

heading: 1 Introduction
text: People often share a variety of images during interactions via instant messaging tools. In practice theory, this is referred to as photo-sharing behavior (Lobinger, 2016), which is interpreted as a communicative practice. 2 This behavior operates internally through a two-stage system (Zang et al., 2021): (1) when to share and (2) what to share. For example, as shown in Figure 1, we first discern the appropriate moment (i.e., decision) for sharing an image with certain intent based on one sentence that invokes the image-sharing behavior. Then, we
1https://github.com/passing2961/DribeR 2From now on, we refer to this as image-sharing behavior, given that “image” is a broader concept than “photo,” thereby providing more flexibility to language models.
share a relevant image at that moment, either by searching the internet or using photos taken on our mobile devices. This work primarily focuses on unlocking the image-sharing behavior capabilities of Large Language Models (LLMs) in a zero-shot manner.
However, recent studies related to multi-modal dialogue exhibit two significant limitations; (1) Over-Simplification. As shown in Figure 1, we humans decide to share images with an underlying intent, such as visual clarification, due to a certain triggering sentence (e.g., “What did you eat?”) in the previous dialogue context. Nevertheless, existing studies (Zang et al., 2021; Feng et al., 2022) have reduced this complex behavior to a binary format (i.e., “yes” or “no”), which oversimplifies the complexity of image-sharing behavior. This sim-
692
plification limits a comprehensive understanding of image-sharing behavior. (2) Limited Understanding of Dialogue. Since the key evidence is scattered across the entire dialogue context (Chae et al., 2023), understanding the linguistic cues underlying dialogue context is critical (Wang et al., 2023a) for retrieving images relevant to the dialogue. Nevertheless, existing systems (Lee et al., 2021; Zang et al., 2021; Feng et al., 2022) have primarily leveraged a dialogue-image matching score based on the cosine similarity between the two modalities (i.e., dialogue and image) to retrieve the relevant image at the image-sharing moment. This approach leads to a lack of capacity to comprehend the dialogue context due to the limitation of dual encoder structures (e.g., CLIP) in fully grasping the dialogue context (Yin et al., 2024).
This work explores whether LLMs contain the image-sharing capability, primarily focusing on a zero-shot performance. To this end, we introduce Decide, Describe, and Retrieve (DRIBER), a gradient-free, extensible, and generalizable framework to unlock this image-sharing capability of LLMs through in-context zero-shot learning. Broadly, DRIBER consists of three stages: (1) deciding the image-sharing behavior with the intent, (2) describing the image description relevant to the previous dialogue, and (3) retrieving the relevant image to the image description. The overall pipeline is illustrated in Figure 3. In addition, we introduce PHOTOCHAT++, an extended version of PHOTOCHAT. PHOTOCHAT is a multi-modal dialogue dataset constructed via crowdsourcing for human-human interaction. PHOTOCHAT++ contains six intent labels, a triggering sentence, and salient information (e.g., “words” or “phrases”) to invoke the image-sharing behavior. In our experiments, we successfully unlock the image-sharing capability of LLMs in a zero-shot setting with the aid of DRIBER, with ChatGPT achieving state-ofthe-art performance. Using the PHOTOCHAT++ dataset, we demonstrate that image-sharing is a challenging task for both humans and LLMs even if we apply the few-shot setting and Chain-ofThought reasoning. Our extensive experiments further confirm that our framework is effective and versatile in real-world applications, specifically in (1) human-bot interaction dialogue and (2) dataset augmentation.
In summary, our main contributions are as follows: 1) We introduce the Decide, Describe, Retrieve (DRIBER) framework, designed to evalu-
ate the image-sharing ability of LLMs in a zeroshot setting. 2) For a comprehensive assessment of LLMs’ image-sharing capabilities, we present the PHOTOCHAT dataset, enriched with additional information such as intent, triggering sentences, image descriptions, and salient information. 3) Compared to the existing method, Experimental results show that DRIBER with LLMs achieves competitive zero-shot performance, even without additional training. 4) To the best of our knowledge, this is the first study to test the image-sharing capability of LLMs through zero-shot prompting.

heading: 2 Overview of PHOTOCHAT++
text: The PHOTOCHAT++ provides additional information (i.e., intent, triggering sentence, image description, and salient information) related to the imagesharing behavior by annotating the original PHOTOCHAT test set. The purpose of this dataset is to thoroughly assess the image-sharing capability of LLMs based on the internal operation system of humans. We describe the details of the human annotation procedure in Appendix E.

heading: 2.1 Preliminary: PHOTOCHAT
text: The PHOTOCHAT (Zang et al., 2021) is a humanhuman multi-modal dialogue dataset constructed through crowdsourcing. This dataset contains 10k multi-modal dialogues, where each dialogue D = {(u1, s1), ..., (ut−1, st−1), (it, st), (ut+1, st+1), ..., (uN , sN )} in the dataset contains only one image it to be shared at turn t. The N and sj ∈ {0, 1} denote the number of dialogue turns and speaker information, respectively. In addition, they define two tasks by decomposing the image-sharing behavior — a photo-sharing intent prediction task and an image retrieval task. The formulations are described as follows.
Task 1: Photo-Sharing Decision Prediction. Given the dialogue history (uj)t−11 and the corresponding speaker information (sj)t−11 , this task aims to predict whether it is appropriate to share the image at turn t in the binary classification formulation, where the label y ∈ {0, 1}. 3
Task 2: Image Retrieval. Given the dialogue history (uj)t−11 and the corresponding speaker information (sj)t−11 , this task aims to retrieve most
3Originally, this task is called as “photo-sharing intent prediction”. However, we consider that the “decision” term is more suitable than the term “intent” in this task because this task just predicts “yes” or “no”.
Intent Purpose
Information Dissemination To convey important information Social Bonding To strengthen social relationship Humor and Entertainment To amuse or entertain Visual Clarification To clarify complex situations Topic Transition To change the topic of dialogue Expression of Emotion or Opinion To express emotions, opinions, or reactions
Table 1: Intent Category for Image-Sharing Behavior.
0
0.1
0.2
0.3
0.4
0.5
0.1 0.3 0.5 0.7 0.9
% o
f C ou
nt
Relative Turn Ratio (%)
Sent Salient
12.98
29.41
30.16
11.78
12.19
3.49
Figure 2: Analysis of PHOTOCHAT++. (Left) the distribution of triggering sentence and salient information. (Right) the intent distribution.
appropriate image at turn t from the image candidate set.

heading: 2.2 Intent Category
text: As shown in Table 1, we design six intent labels for image-sharing behavior: Information Dissemination, Social Bonding, Humor and Entertainment, Visual Clarification, Topic Transition, and Expression of Emotions or Opinions. The detailed explanation is described in Appendix B.

heading: 2.3 Collecting Annotations from Humans
text: We collect additional information through the human annotation process based on the following considerations. (1) Intent. Recognizing that the predefined intents for image-sharing behavior are not always mutually exclusive and can intersect based on the context and content of the image, we instruct annotators to select all intents that are applicable. (2) Triggering Sentence. Annotators are asked to identify and highlight the most significant sentence (i.e., only one sentence) from the preceding dialogue that contributed to the decision to share an image. (3) Image Description. We request annotators to write only one image description, beginning with prefix phrases such as “An image of” or “A photo of”. This format aims to standardize the descriptions for ease of analysis. (4) Salient Information. We ask annotators to highlight all words or phrases they focus on in generating the image description. This step is crucial for understanding the key elements that influence the description process in the human internal mind.

heading: 2.4 Analysis of PHOTOCHAT++
text: We analyze PHOTOCHAT++, focusing on intent distribution and the distribution of triggering sentences and salient information.
Intent Distribution. Visual clarification is the most prevalent, indicating that images are often used in social conversations to aid understanding. Social bonding was also common, likely because the PHOTOCHAT is designed for social bonding. The least common is Topic transition, which is logical since topic transitions are more likely in longer conversations. However, the PHOTOCHAT typically has only 12 utterances per conversation, shorter than many long-term dialogue datasets. Topic transition is interesting and related to proactiveness, suggesting that the future creation of longterm multi-modal dialogue datasets could be beneficial. Distribution of Key Information. We analyze which utterances in the dialogue just before image sharing trigger this behavior and where in the previous dialogue people focus when creating image descriptions. It’s evident that image-sharing behavior is often triggered by utterances immediately preceding the image-share. Notably, the words or phrases that people cognitively focus on when creating image descriptions are not only distributed in the immediate preceding utterance but also throughout the early and middle parts of the conversation. This shows the importance of a model’s ability to understand the entire dialogue when performing dialogue-to-image retrieval.

heading: 3 Methodology

heading: 3.1 Input Prompt Template
text: To elicit the image-sharing ability of LLM in a zero-shot setting, we manually construct a QAstyle prompt template for our framework. The prompt template consists of four main parts: [instruction], [dialogue], [restrictions], and [question]. For each stage, we use different sentences for [instruction], [restrictions], and [question]. To explicitly control the model’s output, we add [restrictions] with Restrictions:. In addition, to make the [dialogue] more natural, we replace sj with Top1K common names of US SSN applicants from 1990 to 2021 4, followed by a previous work (Kim
4https://catalog.data.gov/dataset/baby-names -from-social-security-card-applications-nationa l-data
et al., 2022). We present the prompt template in Appendix.

heading: 3.2 Stage 1: Deciding Image-Sharing
text: Given the dialogue history (uj)t−11 , the corresponding speaker information (sj)t−11 , prompt p1 and model M, this stage generates an output ŷo about whether it is appropriate to share the image: ŷo = M(p1). Previous studies (Zang et al., 2021; Li et al., 2023b) ask the model M to predict the decision, through a simple binary response (yes or no), whether sharing an image is proper or not. In this work, we consider additional information, intent and triggering sentence, to investigate the image-sharing capability of the model M thoroughly. Specifically, the model M predicts all possible intents among the multiple-choice options and generates one sentence that invokes the imagesharing behavior for the same dialogue. When the predicted decision is “yes”, then we generate the relevant image description for the same dialogue in stage 2.

heading: 3.3 Stage 2: Describing Relevant Image
text: Next, stage 2 uses the same model M to generate the image description ŷdesc conditioned on the previous dialogue history (uj)t−11 : ŷdesc = M(p2). Then, instead of using the dialogue (uj)t−11 to retrieve the image, as in previous works, we use ŷdesc as a query to retrieve the relevant image. Therefore, the result of retrieved image is bound to depend on the quality of the corresponding ŷdesc.

heading: 3.4 Stage 3: Retrieving Relevant Image
text: This stage aims to retrieve the relevant image based on the generated image description (ŷdesc) from stage 2 by leveraging the vision-and-language pre-trained model (VLM). In this work, we use VLM as CLIP (Radford et al., 2021), which is a well-generalized and widely used model in various multi-modal tasks. However, our framework could work with any VLMs, such as BLIP (Li et al., 2022) or ALIGN (Jia et al., 2021).

heading: 4 Evaluating Image-Sharing Capability
text: To verify the performance for each stage, we measure the performance on various evaluation metrics. Each metrics are described as follows.

heading: 4.1 Evaluation for Stage 1
text: To understand how well the model M predicts the decision, intent, and triggering sentence, we report three types of metrics:
(1) DECISION[Y/N]. The model should predict the decision with “yes” or “no”. We measure the macro F1 score between the ground-truth decision label yD and the predicted one ŷD. (2) INTENT[CHOICE]. The model should choose all possible intents among the multiple-choice options. Given the ground-truth intents yI and the predicted intents ŷI, we measure the F1 score between yI and ŷI. (3) SENTENCE[DIST.]. The model should generate one most contributed sentence to the imagesharing behavior in a free-form response. We measure the distance between the ground-truth sentence yS (in PHOTOCHAT++) and the predicted response ŷS by using the token F1 score. Specifically, we get the averaged token F1 score between ŷS and the ground-truth sentences in PHOTOCHAT++.

heading: 4.2 Evaluation for Stage 2
text: Based on the theoretical view (Jaimes and Chang, 1999; Santurkar et al., 2022), we evaluate the image description in terms of descriptiveness and completeness. In addition, we evaluate whether the model has a similar cognitive process to a human with respect to salient information. We report three types of metrics: (1) DESCRIPTIVENESS (Santurkar et al., 2022). This measures the inter-modal consistency: how much the image description (ŷdesc) can replace the ground-truth image provided by PHOTOCHAT. However, measuring the descriptiveness is infeasible. Thus, previous work (Santurkar et al., 2022) approximates it with the help of an image-text
matching model (e.g., BLIP (Li et al., 2023a)). Here, we use CLIPScore (Hessel et al., 2021). (2) COMPLETENESS (Santurkar et al., 2022). This involves how much the generated image description (ŷdesc) represents the object in the image. We measure the intersection ratio between the object ŷobj in ŷdesc and the object yobj in the groundtruth image is |yobj∩ŷobj||yobj| . While yobj is provided by the PHOTOCHAT, we extract the object ŷobj in ydesc by prompting ChatGPT to extract the object in the given object categories from PHOTOCHAT. The prompt we used is presented in Appendix.
(3) CONSISTENCY. This measures the intramodal consistency between the generated image description (ŷdesc) and the human-written image description in PHOTOCHAT++. We measure the averaged sentence similarity using SentenceBERT (Reimers and Gurevych, 2019) 5.
(4) SALIENT INFORMATION. To understand whether LLMs show similar cognitive processes when they generate image descriptions, we ask the model M to generate the salient words or phrases that are used to generate the image description. We measure the average token F1 score between the model prediction and ground-truth salient information in PHOTOCHAT++.

heading: 4.3 Evaluation for Stage 3
text: We evaluate how the generated image description retrieves relevant images using the CLIP model. We use a standard metric Recall@{1, 5, 10} and mean reciprocal rank (MRR).

heading: 5 Experiments

heading: 5.1 Large Language Models in DRIBER
text: The primary objective is to assess the imagesharing capability of DRIBER equipped with LLM in terms of zero-shot performance, which necessitates complex reasoning. To achieve this, it is inevitable to leverage instruction-tuned large language models. For proprietary LLMs, we evaluate 2 models in total: 1) ChatGPT (OpenAI, 2023a), and 2) GPT-4 (OpenAI, 2023b). 6. For open-sourced LLMs, we evaluate 3 models in total: 1) VICUNA 13B (Chiang et al., 2023), 2) FALCON INSTRUCT (40B; (Wang et al., 2023c)), and 3) LLAMA2 CHAT 70B (Touvron et al., 2023). We
5We use sentence-transformers/all-roberta-large-v1 model. 6We conduct experiments with all language models by calling the OpenAI API between November-2023 and December2023.
present the hyperparameter settings for each stage in Appendix J.

heading: 5.2 Results
text: DRIBER unlocks the image-sharing capability. Table 2 shows the zero-shot results of various models of DECISION[Y/N] on PHOTOCHAT. Compared to the fine-tuned models, we mainly show that most LLMs can share images by understanding the given dialogue context without additional training on PHOTOCHAT. This is notable when compared to the performance of models that have been finetuned. Such a finding suggests that DRIBER can be effectively utilized in social dialogues requiring an understanding of, and an ability to imagine, interactions between multiple individuals, which is benefited from the power of instruction-tuned LLM. Notably, DRIBER with ChatGPT 0613 establishes a new state-of-the-art performance on DECISION[Y/N], surpassing the PaCE (Li et al., 2023b). Interestingly, DRIBER with GPT-4 1106 and LLaMa-2-Chat 70B exhibit lower performance, highlighting that the task of image sharing remains a challenging one for these models.
Few-shot examples confuse DRIBER in deciding the image-sharing behavior. As shown in Table 3, we evaluate the few-shot performance of DRIBER with ChatGPT 1106 by varying the number of few-shot examples. Interestingly, increasing the number of few-shot examples does not improve but rather decreases the DECISION[Y/N] performance. Specifically, there is an inverse relationship between the number of few-shot examples and the performance of DECISION[Y/N]. This finding contrasts with previous research on factual reasoning, such as in (Kojima et al., 2022), where few-shot examples are shown to improve the capabilities of LLMs. To thoroughly investigate this phenomenon, we measured the ratio of instances (e.g., “I’m sorry, I cannot assist with that request”) where DRIBER refuses to make a decision (Refusal Ratio). As indicated in Table 3, the refusal ratio significantly increases as performance decreases. This indicates that the presence of diverse decisions within few-shot examples confuses DRIBER in making image-sharing decisions.
Chain-of-Thought is not effective in the imagesharing capability. Table 4 shows the zero-shot performance of DECISION[Y/N] when we apply Chain-of-Thought (CoT) reasoning, specifically using the prompt “Let’s think step by step.”, as followed by (Kojima et al., 2022). Generally, we observe that CoT does not enhance the performance of DECISION[Y/N]. This suggests that inducing the model to think about the possibility of imagesharing behavior sequentially often leads to a conclusion in favor of sharing images. In fact, it is noted that the proportion of FP increases. These results confirm that the image-sharing behavior has significant subject property.
DRIBER enhances the dialogue-to-image retrieval task. Table 5 presents the zero-shot results of the dialogue-to-image retrieval task. Notably, DRIBER, combined with various LLMs, significantly outperforms comparative models by a large margin of approximately 7.1% (an abso-
lute value). In addition, we observe that even large-scale pre-trained vision-and-language models (ViLT, PaCE, CLIP) underperform compared to human performance. This suggests that these models have limitations in comprehending dialogues and summarizing relevant image descriptions accurately. The image-sharing behavior also presents a considerable challenge to humans, underscoring its complexity.
Furthermore, we assess the image-sharing capability of recent large multi-modal models such as LLaVA v1.5, MiniGPT-4, and GPT4-V, which show remarkable performance on various visualgrounded language tasks (e.g., VQA). Given that these models are not specifically designed for crossmodal retrieval tasks, we evaluate them by measuring the text similarity score between the generated image descriptions and dialogue history, using Sentence-BERT (Reimers and Gurevych, 2019), as proposed in prior works. Among these models, GPT4-V achieves the best performance. However, their performance still falls short of that achieved by CLIP-large. This result indicates that, despite their versatility in visual-grounded language tasks, these models are not the optimal solution for tasks involving image-sharing behavior, likely due to their structural limitations.
DRIBER has visual imagination ability better than humans. Table 6 summarizes the zeroshot results from stage 2, focusing on DESCRIPTIVENESS, COMPLETENESS, CONSISTENCY, and SALIENT INFORMATION. Contrasting with the DECISION[Y/N] results, DRIBER with GPT-4 1106 outperforms DRIBER (w/ ChatGPT) in describing images, even surpassing human performance in DESCRIPTIVENESS and COMPLETENESS. These findings support the notion that DRIBER GPT-4 1106 possesses notable visual imagination capabilities (Lu et al., 2022; Zhu et al., 2023b; Lu et al., 2023). However, DRIBER ChatGPT 0613 demonstrates superior attention to SALIENT INFORMATION compared to DRIBER GPT-4 1106. This suggests that the task of image sharing remains a significant challenge for these models.
Zero-shot Results of INTENT[CHOICE]. Figure 4 shows the zero-shot results of INTENT[CHOICE] and SENTENCE[DIST.] on PHOTOCHAT++. Our findings reveal that DRIBER with ChatGPT 0613 exhibits a lower performance in INTENT[CHOICE] than other models. However, regarding performance across ALL categories, DRIBER with ChatGPT 0613 significantly outperforms the others. This observed decrease in ALL category performance among the other models indicates a selective impact on varying aspects of DECISION[Y/N], INTENT[CHOICE], and SENTENCE[DIST.].

heading: 5.3 Applications
text: We demonstrate the applicability and extensibility of our pipeline in two distinct applications: (1)
Human-Bot Interaction, and (2) Dataset Augmentation.
Application 1: Human-Bot Interaction. To validate the effectiveness of our pipeline in practical scenarios, we evaluate our pipeline on real humanbot interaction datasets based on VisDial (Das et al., 2017) dataset, which is introduced in the prior work (Levy et al., 2023). This specific scenario does not necessitate the decision-making process regarding image sharing. Therefore, DRIBER is tailored to include only stages 2 and 3, showcasing its flexibility. As shown in Figure 5, DRIBER significantly outperforms the recent ChatIR system (Levy et al., 2023), indicating its strong generalization performance. In addition, we find that the performance of DRIBER is enhanced when provided with a more extensive dialogue context. These results indicate a strong and robust understanding of the dialogue context within DRIBER, benefiting from the enormous ability of LLM.
Application 2: Dataset Augmentation. Previous studies (Lee et al., 2021; Kim et al., 2022; Chun et al., 2022) have reported that machineannotated or generated datasets significantly enhance generalization performance. Drawing inspiration from these works, we augment PHOTOCHAT with DRIBER equipped with ChatGPT 1106 by providing the full dialogue context to the LLM. The motivation for this is that given that the dialogue context in PHOTOCHAT is already fixed, it is crucial to find image-sharing moments that wouldn’t disrupt the existing conversation flow after inserting relevant images at the image-sharing turns. Thus, we slightly modify the input prompt, as detailed in Appendix. Additionally, we prompt
( ) by prompting predicted image descriptions from LLM (i.e., ChatGPT 1106), which are highlighted in orange, pink, and green boxes.
the LLM to generate a rationale for the act of image sharing, thereby gaining insight into the LLM’s judgment. Once an image-sharing moment is identified, we align a relevant image to this moment using Stable Diffusion (Rombach et al., 2022)7. This step is crucial as the LLM sometimes overgenerates image descriptions, making it challenging for CLIP to align the relevant image using existing source image datasets like Conceptual Captions 3M (Sharma et al., 2018). We refer to this augmented dataset as aug-PHOTOCHAT .
To assess if aug-PHOTOCHAT improves generalization performance on unseen multi-modal dialogue datasets, we implement simple text and image retrieval models (details in Appendix K). We train these models on both PhotoChat and augPHOTOCHAT and evaluate them on PhotoChat and MMDialog (Feng et al., 2022), a million-scale multi-modal dialogue dataset. Table 7 demonstrates that models trained on aug-PHOTOCHAT outperform those trained on the two multi-modal dialogue datasets. Notably, there is a significant performance boost on MMDialog with augPHOTOCHAT , highlighting the effectiveness of our pipeline in constructing aug-PHOTOCHAT . An example from aug-PHOTOCHAT is presented in Figure 6.
7https://huggingface.co/stabilityai/stable-d iffusion-2-1-base

heading: 6 Related Work
text: Multi-Modal Dialogue Dataset. Existing studies predominantly fall into two categories, depending on whether the image in the dialogue is grounded or sharing. Image-grounded dialogue tasks are designed to answer questions (Antol et al., 2015; Das et al., 2017; Kottur et al., 2019) or generate natural conversations (Shuster et al., 2018; Meng et al., 2020; Wang et al., 2021b; Zheng et al., 2021) about given images. Nevertheless, it is common to share images pertinent to dialogue contexts in everyday conversations for the purpose of reinforcing social bonding, as well as enhancing engagement and interest. Recent studies have proposed datasets that encapsulate this image-sharing behavior. This has been achieved by collecting a human-human dialogue dataset (PhotoChat) via crowdsourcing (Zang et al., 2021), a large-scale dataset (MMDialog) from social media (Feng et al., 2022), or constructing datasets automatically using vision-and-language models (Lee et al., 2021). In this work, our focus is exclusively on the PhotoChat dataset to gain a deeper understanding of the image-sharing capabilities of LLMs. We do not include automatically constructed datasets or the MMDialog due to the considerable expense associated with conducting experiments using LLMs. Prompting Large Language Models. Recent studies have witnessed the success of large language models, such as Instruct GPT-3 (Ouyang et al., 2022), ChatGPT (OpenAI, 2023a), GPT4 (OpenAI, 2023b), in a zero-/few-shot performance. The use of these models, in conjunction with “prompt engineering,” has unlocked the abilities of LLMs, even emergent ones (Wei et al., 2022a), across various tasks. These tasks range from dialogues (Lee et al., 2022a; Kim et al., 2022; Lee et al., 2022b), complex reasoning tasks (Wei et al., 2022b; Kojima et al., 2022), and theory-ofmind (Sap et al., 2022; Kosinski, 2023), to image classification(Yang et al., 2022; Pratt et al., 2022; Menon and Vondrick, 2022; Zhang et al., 2023) and multi-modality (Lu et al., 2023; Han et al., 2023).

heading: 7 Conclusion
text: In this paper, we explore the image-sharing capabilities of LLMs in a zero-shot prompting by introducing a three-stage framework: Decide, Describe, Retrieve (DRIBER). Our extensive experiments demonstrate the effectiveness of our framework in enhancing zero-shot performance across
both stages, with ChatGPT achieving state-of-theart performance. We also reveal that the imagesharing ability is an emergent ability in the zeroshot prompting. With extensive experiments, we reveal the effectiveness of our framework on two useful applications. In future works, we will assess this capability in a few-shot setting using additional multi-modal dialogue datasets. We will construct a personalized multi-modal dialogue dataset using our framework.
Limitations
Here, we highlight some limitations of our work. Firstly, our prompt template is rather lengthy, which complicates expansion into the few-shot setting. We anticipate that conducting few-shot prompting to utilize the image-sharing capability of LLMs would result in better performance compared to zero-shot prompting. Secondly, LLMs tend to over-generate image descriptions even in the absence of specific demographic information such as age or appearance. For instance, in the description, “An image of a woman with long, brown hair wearing a flowy white dress and brown boots,” there is no reference to long hair in the given dialogue. Providing additional information (e.g., persona) can enhance the relevance of image descriptions generated by LLMs.
Ethical Considerations
We report several potential issues with our proposed framework. First, generated image descriptions may propagate social bias because GPT-3 can still produce harmful content, including social bias and offensiveness(Baheti et al., 2021; Hartvigsen et al., 2022). Second, this issue has resulted in the inclusion of problematic descriptions in the constructed aug-PHOTOCHAT dataset, leading to socially-biased images generated using Stable Diffusion (Rombach et al., 2022). As a result, when vision-and-language models like CLIP (Radford et al., 2021) and DALL-E (Ramesh et al., 2021) are trained on this augmented dataset, they may exhibit social biases. As reported in (Wang et al., 2021a), even if we give the gender-neutral query to CLIP (Radford et al., 2021) model, the CLIP model sometimes retrieves images causing gender-bias issues. We are concerned that this problematic issue may exist in the augmented dataset. Therefore, the image retrieval model trained on this dataset may sometimes retrieve biased images. In addition,
text-to-image generative models learn social biases from the augmented dataset, as reported in the prior work (Cho et al., 2022). We should consider this problem important when building a multimodal search model.

heading: Acknowledgement
text: This work was supported by a grant of the KAISTKT joint research project through AI Tech Lab, Institute of convergence Technology, funded by KT [Project No. G01230605, Development of Task-oriented Persona-based Dialogue Generation Combining Multi-modal Interaction and Knowledge Modeling].

heading: A Additional Related Work
text: Large Multi-Modal Models. Recent studies have proposed large multi-modal models that demonstrate the surprising generalization performance on various visual-grounded language tasks, such as Flamingo (Alayrac et al., 2022), LLaVA (Liu et al., 2023), MiniGPT-4 (Zhu et al., 2023a), Qwen-VL (Bai et al., 2023), CogVLM (Wang et al., 2023b), and GPT4-V (OpenAI, 2023c). In this work, we assess the imagesharing capability of these models in a zero-shot setting.

heading: B Details of PHOTOCHAT++
text: We describe each intent category as follows:
(1) Information Dissemination. This intent involves sharing images that convey important information, such as current news, economic updates, educational content, or plot summaries. These images are intended to inform or educate the recipient.
(2) Social Bonding. This category encompasses sharing personal photographs, memories of recent encounters, recollections of past events (i.e., memory recall), or images that connect to one’s own experiences. Such image sharing is primarily used to strengthen social ties.
(3) Humor and Entertainment. This intent involves sharing images that aim to amuse or entertain, like humorous pictures or memes. The primary focus is on sharing light-hearted content to bring joy or laughter.
(4) Visual Clarification. Here, images serve as supplementary material to clarify complex concepts or situations, especially when textual conversation alone is insufficient. Images can significantly enhance understanding in such contexts.
(5) Topic Transition. In this category, images are utilized to change the topic of conversation or modify the mood subtly. This can be a strategic approach to redirecting a discussion or lightening the atmosphere.
(6) Expression of Emotion or Opinion. This intent includes the sharing of images to express emotions, opinions, or reactions that are challenging to communicate through text alone. It involves the use of emotive photography, art, or reaction images that effectively convey feelings or viewpoints.

heading: C Comparison to Previous System
text: Figure 7 illustrates the overview of previous systems (Zang et al., 2021; Feng et al., 2022) and DRIBER.

heading: D Additional Experiments
text: Figure 8 shows the zero-shot results of INTENT[CHOICE], SENTENCE[DIST.], and ALL when we apply zero-shot chain-of-thought (CoT) reasoning. For CoT, we follow (Kojima et al., 2022) and use the prompt “Let’s think step by step”. We observe that CoT does not improve the performance of INTENT[CHOICE], SENTENCE[DIST.], and ALL. These results suggest that CoT selectively improves the image-sharing capability of LLMs.

heading: E Details of Human Annotation Procedure
text: E.1 Preparing Human Annotation We first prepare 968 dialogues from the test set of PHOTOCHAT to collect additional information crucial for understanding image-sharing behavior in a manner akin to human comprehension. Then, we ask 12 human annotators to annotate various elements at the ground-truth moments of imagesharing in PHOTOCHAT. These elements include intent, triggering sentence, salient information, and image description. Notably, we intentionally do not show the ground-truth images from PHOTOCHAT during this annotation process. This process is adopted to explore the complexity and subjectivity inherent in image-sharing behavior.

heading: F Human Evaluation Questionnaire
text: This section lists the questions and notes used for human evaluations.
F.1 Questions • What is the intent behind sharing the im-
age? Options: Information Dissemination / Social Bonding / Human and Entertainment / Visual / Clarification / Topic Transition / Expression of Emotion or Opinion
Notes: You can select all of the following options that are possible. If the answer you have in mind is not listed, please write it in the space. If you write more than two sentences, then please divide them using the separator “;”.
• Which sentence invokes the image-sharing behavior? Notes: You should highlight the most contributed sentence (i.e., only one sentence) using “Sentence”
• What kind of image would be appropriate to share at the [Sharing Image] turn? Notes: You can write only one image description, starting with “An image of” or “A photo of”.
• What words or phrases do you focus on to write the image description? Notes: You can highlight all words or phrases that you focus on using “Word/Phrase”

heading: G Human Evaluation System
text: We show a screenshot of the human evaluation system in Figure . We implement this system using Label Studio (Tkachenko et al., 2020-2022).

heading: H Details of Human Evaluation
text: We recruited 12 individuals, unknown to us, who are either graduate or undergraduate students. Prior
to participating in the experiment, they were provided with comprehensive instruction on the task, an overview of the multi-modal dialogue dataset, and a detailed explanation of the evaluation criteria. This preparatory phase lasted approximately one hour.

heading: I Discussions
text: Domain Generalizability. Towards Artificial General Intelligence (AGI), we need to show the proposed methods’ generalization capability across different domains, such as medical imaging. We recognize the significance of this aspect and have designed our methodology to be both extensible and generalizable. Our approach’s extensibility allows us to tailor DRIBER according to specific domains. For instance, in a specialized domain like medical, we could incorporate a domain-specific LLM, such as Med-PaLM 2 (Singhal et al., 2023b),
or even GPT-4, which has demonstrated superior performance in tasks requiring medical knowledge (Nori et al., 2023), including outperforming domain-specific models like Med-PaLM (Singhal et al., 2023a) in the United States Medical Licensing Examination (USMLE). For the image retrieval aspect, a specialized model such as MedCLIP (Wang et al., 2022) could be employed to enhance retrieval accuracy in the medical context.
Towards Better Image-Sharing Ability. As shown in our experiment, the likelihood of performance improvement is high as the model’s size increases or when it is trained with alignment to human preference. This suggests that the imagesharing ability is subjective and resembles humanlike tasks. Therefore, receiving real-time feedback through interactive mode (a form of human-AI collaboration) and further training the model using the RLHF method could lead to better performance, aligning the model’s actions favorably with imagesharing ability.
Furthermore, understanding conversational context is essential, and imbuing the model with the ability of perspective-taking, understanding situations from the user’s point of view, could lead to performance enhancement. For instance, when a user is feeling down due to poor test results, the model could not only provide empathy through text but also share a picture of a dog based on the user’s fondness for dogs and the current context of struggling with test scores, thereby offering multifaceted empathy.
In addition, unlike image-grounded dialogue, image-sharing scenarios might lack explicit information from previous conversations. For instance, understanding what “it” refers to in “I love it” requires considering the preceding conversational context. Thus, it’s important to consider coreference resolution. Moreover, while sharing images, incorporating information about significant utterances from previous dialogues or using keywords and keyphrases could likely improve performance.
As depicted in Figure 6’s orange-generated results, the language model might sometimes overgenerate due to excessive creativity. For instance, if the conversation only contains information about a coffee shop without mentioning “French-style,” the model might still produce the word “French.” Such cases could pose challenges in practical applications where inappropriate images could be retrieved.
In practical applications, it’s beneficial to consider the user’s satisfaction and share images that account for their personal information. For example, if a user mentions, “I work in a hotdog stand,” and their friend, who also works there, has a picture related to selling hotdogs in their phone album, it would be more suitable to share an image depicting the user themselves selling hotdogs rather than an image with the friend. Of course, obtaining explicit consent for sharing personal information is crucial.
Additionally, beyond improving the imagesharing ability, at the application level, using videos could enhance user engagement. Exploring this avenue could be a promising direction for future research.
Intrinsic Properties of LLMs. We believe that the intrinsic properties of LLM, which have been experimentally proven in various studies, have influenced image-sharing ability.
• Understanding the dialogue context: It’s essential to grasp the conversation topic holistically, emotional shifts between users, and general knowledge. Recent research results have shown that language models possess these abilities.
• Understanding the interlocutor’s mental state: It is important to comprehend the interlocutor is situation to determine whether sharing an image is appropriate. For instance, if the interlocutor is upset, it might be better to respond empathetically rather than share an image. This ability is highly related to the Theory-of-Mind (ToM). Recently, LLMs have achieved competitive performance in Theoryof-Mind (ToM) tasks, which may influence image-sharing ability.
• Understanding the intent: From the model’s perspective, sharing an image can be seen as intent. Many language models have demonstrated good performance in task-oriented dialogue tasks.
• Visual imagination ability: To share an appropriate image, one must imagine which image is best. This capability has been empirically proven in various recent studies. We investigated the C4 dataset, a representative pretraining dataset for LLMs, to analyze why this capability is manifested. The data discovered in C4 consists of pairs of images and
their corresponding captions. These captions contain words/phrases related to visual imagination ability, such as “depict” and “photo of.” Moreover, on blogs, images often appear consecutively along with stories. Due to these elements, the LLM learned an inherent visual, and imaginative capability during its pretraining phase.
J Implementation Details of LLMs
To evaluate the image-sharing capabilities of LLMs, we call ChatGPT (OpenAI, 2023a) and GPT-4 (OpenAI, 2023b) by calling OpenAI API. All experiments are conducted on two A100 (40GB) GPU. For each stage, the generation configuration is as follows:
• For Stage 1, we set maximum tokens to 1024, temperature to 0.0, frequency penalty to 0.0, presence penalty to 0.0, top_p to 1.0, and stop tokens to \n\n.
• For Stage 2, we set maximum tokens to 1024, temperature to 0.9, frequency penalty to 0.0, presence penalty to 0.4, top_p to 0.95, and stop tokens to a default setting.

heading: K Details of Experimental Settings
text: To explore how our dataset affects both text and image retrieval tasks, we implement two simple and standard baseline retrieval models for text-toimage and image-to-text settings.
K.1 Task Definition Follwing (Lee et al., 2021; Zang et al., 2021), we explain the formulation of two main tasks - next response prediction and image retrieval. Let us assume that we have a multi-modal dialogue D = {(uj , ij , cj)}N1 where N denotes the number of dialogue turns, and j = t is the turn that an image sharing behavior occurs. Then, each task is formulated as follows.
Next response prediction is to predict the next utterance at turn t + 1 given the dialogue history ({uj}t1) and image it. Image retrieval is to retrieve relevant image at turn t given the dialogue history ({uj}t−11 ).
Following (Shuster et al., 2018; Lee et al., 2021), we set the number of retrieval candidates to 100 and use Recall@{1,5,10} and mean reciprocal rank (MRR) for the evaluation metrics.
K.2 Baseline Models
As illustrated in Figure 10, we present the architecture of baseline models: the text retrieval and image retrieval models. We provide a detailed description of baseline models below.
Text Retrieval Model. The text retrieval model consists of three primary components: the dialogue encoder, the response encoder, and the image encoder. The dialogue encoder processes the entire dialogue history into a fixed-size representation using the BERT model (Devlin et al., 2018). The dialogue history includes up to three turns preceding the current turn, with each turn separated by the [SEP] token. The response encoder also converts responses into fixed-size representations, utilizing a different version of the BERT model than the dialogue encoder. After BERT processes the text, mean pooling is applied to the text representations for both encoders. These pooled representations then pass through a linear projection layer followed by the ReLU activation function (Nair and Hinton, 2010). The image encoder uses the CLIP-base model (Radford et al., 2021) to extract feature vectors from images. The dialogue and image feature vectors are then combined using element-wise addition. The loss is computed by taking the dot product between the response feature vector and the resulting summed vector.
Image Retrieval Model. The image retrieval model is composed of two main components: the dialogue encoder and the image encoder. The dialogue encoder uses the BERT-base model to convert the dialogue into a representation, followed by mean pooling of the text representations. For the image representation, the CLIP-base model is utilized. After encoding, the image and dialogue vectors are passed through their respective linear projection layers, each followed by a ReLU activation function. The loss is determined by calculating the dot product between the image feature vector
and the dialogue vector.
K.3 Implementation Details We implement baseline models based on PyTorch Lightning. All experiments are conducted on two A100 GPUs (40GB). To accelerate the training time, we apply distributed training to baselines. We follow the hyperparameter settings similar to the previous works (Lee et al., 2021; Zang et al., 2021), which are described as follows:
Text retrieval. In our experiment, we set the batch size to 256, the learning rate to 5e-5, and the gradient clipping value to 2.0. We use the AdamW optimizer with a cosine learning rate scheduler. We set the warm-up ratio as 0.1% and weight decay as 0.2.
Image retrieval. We set the batch size to 256. We also use the AdamW optimizer with an initial learning rate of 2e-5 and decaying 0.1%.
Training. Since our dataset contains several images per utterance, we randomly choose one image in each batch. We do not update the parameter of the image encoder.

heading: L Rationale Distribution
text: We present the rationale distribution as shown in Table 8.

heading: M Prompt Templates
text: Here, we present all prompt templates used in our work, such as restriction-based prompt templates for each stage, and several prompt templates for the ablation studies.
M.1 Prompt Templates We present a prompt template for dataset augmentation in our proposed framework, as shown in Figure 11.
N More Examples of aug-PHOTOCHAT
We provide more examples of aug-PHOTOCHAT .
dataset using our proposed framework with LLM (i.e., ChatGPT 1106) and Stable Diffusion( ).

heading: Future Work
text: Exploring this avenue could be a promising direction for future research. Intrinsic Properties of LLMs. We believe that the intrinsic properties of LLM, which have been experimentally proven in various studies, have influenced image-sharing ability. • Understanding the dialogue context: It’s essential to grasp the conversation topic holistically, emotional shifts between users, and general knowledge. Recent research results have shown that language models possess these abilities. • Understanding the interlocutor’s mental state: It is important to comprehend the interlocutor is situation to determine whether sharing an image is appropriate. For instance, if the interlocutor is upset, it might be better to respond empathetically rather than share an image. This ability is highly related to the Theory-of-Mind (ToM). Recently, LLMs have achieved competitive performance in Theoryof-Mind (ToM) tasks, which may influence image-sharing ability. • Understanding the intent: From the model’s perspective, sharing an image can be seen as intent. Many language models have demonstrated good performance in task-oriented dialogue tasks. • Visual imagination ability: To share an appropriate image, one must imagine which image is best. This capability has been empirically proven in various recent studies. We investigated the C4 dataset, a representative pretraining dataset for LLMs, to analyze why this capability is manifested. The data discovered in C4 consists of pairs of images and
their corresponding captions. These captions contain words/phrases related to visual imagination ability, such as “depict” and “photo of. ” Moreover, on blogs, images often appear consecutively along with stories. Due to these elements, the LLM learned an inherent visual, and imaginative capability during its pretraining phase. J Implementation Details of LLMs
To evaluate the image-sharing capabilities of LLMs, we call ChatGPT (OpenAI, 2023a) and GPT-4 (OpenAI, 2023b) by calling OpenAI API. All experiments are conducted on two A100 (40GB) GPU. For each stage, the generation configuration is as follows:
• For Stage 1, we set maximum tokens to 1024, temperature to 0. 0, frequency penalty to 0. 0, presence penalty to 0. 0, top_p to 1. 0, and stop tokens to \n\n. • For Stage 2, we set maximum tokens to 1024, temperature to 0. 9, frequency penalty to 0. 0, presence penalty to 0. 4, top_p to 0. 95, and stop tokens to a default setting