SCREEN: A Benchmark for Situated Conversational Recommendation

Authors: dongding  lin
jian  wang
tou  chak
  leong
wenjie  li
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 07:51:26

Abstract: Engaging in conversational recommendations within a specific scenario represents a promising paradigm in the real world. Scenario-relevant situations often affect conversations and recommendations from two closely related aspects: varying the appealingness of items to users, namely $\textit{situated item representation}$, and shifting user interests in the targeted items, namely $\textit{situated user preference}$. We highlight that considering those situational factors is crucial, as this aligns with the realistic conversational recommendation process in the physical world. However, it is challenging yet under-explored. In this work, we are pioneering to bridge this gap and introduce a novel setting: $\textit{Situated Conversational Recommendation Systems}$ (SCRS). We observe an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To this end, we construct a new benchmark, named $\textbf{SCREEN}$, via a role-playing method based on multimodal large language models. We take two multimodal large language models to play the roles of a user and a recommender, simulating their interactions in a co-observed scene. Our SCREEN comprises over 20k dialogues across 1.5k diverse situations, providing a rich foundation for exploring situational influences on conversational recommendations. Based on the SCREEN, we propose three worth-exploring subtasks and evaluate several representative baseline models. Our evaluations suggest that the benchmark is high quality, establishing a solid experimental basis for future research. The code and data are available at https://github.com/DongdingLin/SCREEN.

Main paper: 
 introduction: 
 Building a Conversational Recommendation System (CRS) [4, 14, 22] that can communicate with people in multimodal situations is an attractive goal for the AI community. Existing multimodal CRSs [7, 27, 38, 40, 44] integrate textual and visual product information in various ways to enhance recommendation processes. Chen et al. [3] , Nie et al. [29] enrich the multimodal context by incorporating item images and dialogue histories. Meanwhile, Zhang et al. [42] employs a multi-attribute graph model to capture diverse item attributes. Additionally, Du et al. [6] advances item representations through a multimodal transformer, capturing both global and local perspectives of items. Multimodal CRSs are targeted to integrate textual and visual information to model user preferences and item representations, expecting the system to provide precise and appropriate recommendations.Despite considerable advancements, existing multimodal CRSs still face challenges in understanding dynamic user preferences and accurately representing real-world items. This is mainly due to two key aspects: (1) User Preference Modeling: current methods [7, 38, 44] narrowly leverage a user's historical profiles, general interests, and conversational histories to model user preferences. These approaches typically overlook the dynamic nature of user interests and choices, which can fluctuate significantly due to situational factors, such as product location and the current season's climate. This leads to what can be termed as situated user preference -a concept that requires grounding the user's underlying interests to the current environment and situational context. (2) Item Representation: conventional modeling in CRSs often represents items through intrinsic and static attributes [6] . It fails to account for the variability in an item's attraction, which can change with situational factors like the spatial layout and daily weather, leading to what can be termed as situated item representation. This concept extends beyond traditional static attributes by adapting to environmental contexts, providing a more context-sensitive item representation. As demonstrated in Figure 1 , considering the situation (e.g., spring) beyond the intrinsic attributes of an item (e.g.,Hi, I want to buy a dress that I can wear to a party. Do you have any recommendations?Of course, do you prefer long skirts or short? I usually wear short skirts. But I want to try long skirts for this party.What do you think of this light-red dress? It appears very elegant and is suitable for wearing to party. It is also quite appropriate for the spring season and matches well with your temperament.Looks great, thanks for the recommendation, please add it to my shopping cart.A clothing store Location: Late afternoon Time:The climate is warm and pleasant, with a gentle breeze and the essence of springThe user is in a relaxed and cheerful mood User Mood:The user is dressed in a light, casual button-down shirt and khaki chino pants, with long, soft hair. brand or appearance) leads to a more accurate representation of the item in the given context. Moreover, by analyzing the user's outfit (e.g., dressed in a light shirt), emotion (e.g., cheerfulness), and dialogue history, the system captures user preferences more accurately within the current scenario. Afterward, the system is able to recommend a personalized product (e.g., a light-red dress) that is more appealing to the user.User Appearance:In this work, we extend the traditional multimodal CRS to a more realistic paradigm: Situated Conversational Recommendation System (SCRS). It requires the system to consider the inherent connection between users and items under specific situational contexts, thereby conversing with higher engagement and providing more appropriate recommendations to users. While the advancement of SCRS is essential, the absence of a high-quality benchmark remains a significant obstacle to its development. Thus, we raise the question: how can we utilize minimal human efforts to construct a high-quality SCRS dataset?Inspired by the powerful human-mimicking capabilities of large language models (LLMs) [10, 19] , we construct a high-quality SCRS benchmark, namely Situated Conversational REcommENdation (SCREEN), based on a role-playing approach. We use carefully designed instructions to prompt LLMs as agents to play the roles of users and recommenders in a simulated environment. To let them "see" the co-observed scene, we use multimodal LLMs to extract visual features given specific situations. In the end, we obtain over 1.5k scenes with 20k recommendation-oriented dialogues. Moreover, we delineate three critical subtasks to evaluate SCRS comprehensively: system action prediction, situated recommendation, and system response generation. These subtasks measure the system's performance in accurately interpreting user intentions, modeling situated item representations, capturing situated user preferences, and generating responses that actively engage users. Additionally, we employ multiple representative baseline models and evaluate their performances on the SCREEN benchmark.Our contributions are summarized as follows: (1) We expand the scope of traditional multimodal CRS to SCRS. This under-explored yet promising paradigm incorporates situational context into the recommendation reasoning process, furnishing users with more engaging and contextually appropriate recommendations. (2) We construct a comprehensive and high-quality benchmark named SCREEN to facilitate exploration in this nascent field. (3) We identify and articulate three essential subtasks for evaluating SCRS. We further present baseline results on the SCREEN, establishing a solid experimental basis for future research. 
related work 2.1 conversational recommendation systems: 
 Conversational recommendation systems (CRS) have become a major research focus, delivering superior recommendations through natural language interactions [14, 22, 34] . Most CRS datasets, including REDIAL [20] , TG-REDIAL [46] , INSPIRED [11] , and DuRecDial [23, 24] , rely heavily on text, using dialogue histories and item attributes but neglect the crucial role of visual information associated with items. To address the need for multimodal CRS, the introduction of the MMD benchmark dataset [33] marked a significant advancement, initiating tasks that cater to multimodal, domainspecific dialogues. The MMConv dataset [21] further expanded this by covering multiple domains. Despite these advancements, existing datasets fail to fully capture the diverse expressions of users' subjective preferences and recommendation behaviors in real-life scenarios, a gap the SURE dataset [25] seeks to fill. The SIMMC-VR dataset [36] also enhances the system's comprehension of spatial and temporal contexts. However, integrating situational context into CRS-adapting recommendations based on users' environments and activities-remains underexplored, presenting a promising direction for developing more context-aware systems.Recent advancements in situated dialogues have emphasized the importance of embedding interactions within specific contextual situations, driving interest in training agents for multimodal actions grounded in dynamic multimodal input and historical dialogue context. To facilitate this research, Crook et al. [5] , Moon et al. [28] developed the SIMMC dataset, establishing a foundation for situational, interactive multimodal conversations. Despite its significance, the SIMMC dataset faced criticism for its simplistic and unrealistic multimodal contexts. To this end, Kottur et al. [17] introduced SIMMC 2.0, enhancing multimodal dialogue capabilities but focusing primarily on immediate, local topics, which limited its support for more dynamic, forward-looking conversations.To address this limitation, Otani et al. [31] presented SUGAR, a dataset to improve agents' proactive response selection. Despite these advancements, integrating situated conversations into CRSs presents untapped potential. It demands a deep understanding of user preferences and item representations in specific contexts. Our work incorporates situational factors into recommendation reasoning, enabling the system to deliver more accurate, contextually relevant recommendations that align with the user's situational context and preferences, potentially enhancing user engagement. 
scrs dataset construction 3.1 problem formulation: 
 We consider a SCRS dataset D = {(S 𝑖 , I 𝑖 , U 𝑖 , C 𝑖 )} 𝑁 𝑖=1 , where 𝑁 is the total number of dialogues. In the 𝑖-th dialogue, S 𝑖 represents the situational information, which includes the user-system co-observed scenario (e.g., scene snapshot), spatiotemporal information (e.g., time), and environmental information (e.g., climate). I 𝑖 represents all items in this situation S 𝑖 . U 𝑖 denotes the user's personalized information (e.g., user profile, user state).C 𝑖 = {C 𝑖,𝑡 } 𝑁 𝑇 𝑡 =1is the dialogue content, with a total of 𝑁 𝑇 turns. The task of SCRS is formalized as follows: given the situational information S, all items I in this situation, a set of user's personalized information U, and a dialogue context C, the objective is to select and recommend the most appropriate item in the scene S to the user and generate a natural language response that matches the scene content. Compared with traditional CRS, SCRS requires that the recommended items and responses closely relate to the situational context.This section describes a role-playing framework for constructing an SCRS dataset by integrating multiple LLM agents, inspired byImagine yourself as a consumer viewing a scene screenshot. Several boxes are drawn on the image, and each box encloses a piece of clothing or furniture, marked with a corresponding number. Describe them according to the number sign, focusing on the color, type, and pattern. Below is the name of each piece of clothing or furniture. It is auxiliary information to help you identify them: 1. <NAME_OF_ITEM 1> 2. <NAME_OF_ITEM 2> … Figure 3 : Instruction template for the scene pool generation. [35] . As depicted in Figure 2 , the framework comprises two key components: generating a Scene Information Pool and establishing a Role-Playing Environment. First, scene item metadata only describes its intrinsic attributes, lacking the subjective descriptions common in real-world user-salesperson interactions [25] . To address this, we use multimodal LLMs to generate subjective descriptions, enriching item information (e.g., describing a clothing item as "enthusiastic and bold" rather than just "very red"). Second, the role-playing environment includes three agents: user, system, and moderator, each following meticulously designed instructions to facilitate effective communication and interaction. Our work leverages VR snapshots from the SIMMC 2.1 dataset [16] , encompassing diverse scenes from 140 fashion and 20 furniture stores, with detailed metadata covering nine attributes per item, including type, color, pattern, material, price, brand, size, sleeve length, and consumer reviews.When making conversational recommendations in scenarios, users often prioritize a product's situational attributes (e.g., appearance and location) over intrinsic ones (e.g., price, brand), leading to more intuitive decision-making. Users tend to prejudge products based on these situational attributes, which can vary depending on the scenario and be influenced by external factors like lighting or item placement. For example, pants may appear more vibrant under soft lighting than under bright lighting. While existing research typically describes products using specific referring expressions (e.g., "red clothes"), non-expert users often use subjective descriptions (e.g., "clothes designed for young women"), which are generally missing from conventional product meta-databases. To address this, we enhance item metadata with situational attributes and subjective descriptors, providing a more nuanced item representation. Leveraging recent advancements in multimodal LLMs, particularly the Set-of-Mark technique [39] , we improve item recognition and description generation. Using spatial data from the SIMMC 2.1 dataset, which provides precise coordinates for products within scene snapshots, we create bounding boxes and assign unique identifiers for each item. These annotated snapshots are processed by GPT-4V (gpt-4-1106-vision-preview version), tasked with elucidating situational attributes and subjective descriptions based on the prompt shown in Figure 3 . The output is integrated into existing product metadata to form a comprehensive scene information pool.Our role-playing environment is crafted to provide a global environment description that prompts all LLM agents. To engender a realistic and multifaceted setting, it incorporates three principal dimensions: (1) Temporal phases, which are delineated into morning, noon, afternoon, and evening; (2) Spatial settings, which encompass both fashion and furniture retail spaces; and (3) Climate, which is represented by the quartet of seasons: spring, summer, autumn, and winter. To augment the diversity within the simulation, we employ ChatGPT (gpt-3.5-turbo version) to generate succinct narratives for each seasonal context. For example, in the scenario "afternoon, fashion store, spring," the ambiance is vividly depicted as "It is the afternoon, and you find yourself in a fashion store. A gentle breeze wafts through, heralding the arrival of spring." Such tailored descriptions are appended to the beginning of each agent's instructions, ensuring a coherent framework for interaction.The user agent primarily aims to simulate consumers' shopping behavior across diverse scenarios, generating responses based on their preferences, profiles, and personalities. To this end, we set user information through the following aspects: User Preference. In the given scenario, we catalog the attributes of all products and allocate user preferences (favor, aversion, or neutrality) to each attribute randomly. This approach facilitates the generation of a wide range of personalized preferences. To enrich the expression of user preferences and inject it with a more natural and diverse vocabulary, we employ ChatGPT to refine this structured information into fluent natural language. Figure 4 illustrates the instruction template used for this transformation. Subsequently, structured user preferences are described more naturally, such as, "You exhibit a preference for red, an aversion to white, and display no particular inclination towards purple;..."The following is a structured expression of user preference. Please refine this structured information in natural and fluent language. Be careful to start with "you," and the length should not exceed 50 words. Color: red (favor), white (aversion), purple (neutral)… Style: jacket (favor), shirt (favor), sweater (aversion)… …The following is a structured expression of a user profile. Please refine this structured information into natural and fluent language. Be careful to start with "you," and the length should not exceed 50 words. Name: John; Age: 18 years old; Gender: Male; Profession: Doctor; Emotional State: Joyful; Upper Body: White shirt; Lower Body: Jeans; Hair Style: Short; …The following is a structured expression of a user's personality. Please refine this structured information into natural and fluent language. Be careful to start with "you," and the length should not exceed 50 words. Openness: Intellectual; Conscientiousness: Efficient; … User Profile. Leveraging the user information in the DuRecDial dataset [23] , we developed a structured pool of personal profile attributes, including but not limited to name, age, gender, and profession. For instance, a typical user profile in this structured pool might be described as follows: "Name: John; Age: 18; Gender: Male; ...". Additionally, we enriched these profiles with emotional states (e.g., joy, cheerfulness, excitement, sadness, worry, and grief) and appearance descriptions based on items captured in another scene snapshot to mirror real-user scenarios. An example could be "Emotional State: Joyful; Upper Body: White shirt; Lower Body: Jeans; ...". It is crucial to highlight that, similar to how a salesperson makes recommendations based on the user's appearance in real life, the system agent can also observe the user's appearance to infer the user's preferences and make appropriate recommendations. We use ChatGPT to refine this structured information into fluent natural language similar to processing user preferences information. The instruction template is shown in Figure 4 .User Personality. To further reflect the user's personality and increase the user agent's diversity, we also use the Big Five personality traits [9, 41] to simulate user personalities. These traits provide a framework for the assignment of attributes representing positive and negative aspects along five dimensions: openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N). Combining these characteristics allows for creating a nuanced and comprehensive user personality model, enriching diverse interactions. As shown in Figure 4 , we leverage ChatGPT to refine such structured information into fluent natural language, similar to how user preferences are processed.In the end, we use natural languages to express the simulated user and prompt the user agent to play the role of a customer. Figure 5 shows the complete instruction template.Imagine yourself as a consumer shopping at a clothing store | furniture store. This image is a snapshot of the store. Here are your details: 1. <Generated User Preference> 2. <Generated User Profile> 3. <Generated User Personality> You need to judge whether the system's recommendations align with your criteria. Your response should be concise, no more than 50 words. You do not need to recommend anything but feel free to express your interests.Imagine yourself as a salesperson in a clothing store | furniture store. This image is a snapshot of the store. The following is the customer's appearance: <Generated User Appearance>.You have metadata and subjective description for all items in the store: <Item Metadata>; <Generated Item Subjective Description>. You need to first choose one of the 6 actions (<Predefined Actions>) to decide your next action, and generate a corresponding response based on your action. Please output in the format of [Action]: [Reply] . Your response should be concise, no more than 100 words.You are the moderator of a conversation. You need to decide whether the conversation should end immediately. The conversation should be terminated in the following three situations: (1) The system completes the recommendation, the user accepts the recommendation based on the preset preferences, and the system action is not <Topic Transfer>. (2) The user rejects the system's recommendations multiple times (more than three times) (3) The conversation between the user and the system reaches the maximum number of rounds limit (30 rounds). Should the following dialogue <Ongoing Conversation> be ended? Answer yes or no. 3.3.2 System Agent. The system agent aims to serve as a humanlike salesperson, such as a clothing salesperson in a fashion store. Its primary objective is to recommend the most appropriate items based on the user's preferences expressed during the conversation. To realize this vision, we design the system agent with predefined actions: (1) Describe Item Information. The system agent proactively offers the user comprehensive details of the items, including intrinsic attributes, situational attributes, and subjective descriptions. ( 2) Inquire About Preferences. The system agent gathers user preferences by querying their opinions on specific items within the scene or clarifying ambiguities in the user's requests to ascertain their needs accurately. (3) Address User Queries. The system agent provides the requested information upon user inquiries about an item, ensuring that user inquiries are promptly and effectively addressed. (4) Topic Transfer. When the user accepts an item the system agent recommends, the system agent determines whether to introduce another item or to delve deeper into the current selection, thus guiding the conversation strategically. (5) Make Recommendations. When the system agent deems sufficient information on user preferences has been collected, it will decide which item to recommend. ( 6) Add to Cart. When the user accepts a recommendation, the system agent inquires whether the user wishes to add the item to their shopping cart. It is worth noting that in each interaction, the system agent is required to identify the action it intends to execute initially. Subsequently, it generates a response that aligns with the specified action.In addition, similar to real-life shopping experiences, the salesperson can observe the customer's appearance but cannot obtain the customer's profile (e.g., name, profession). In the simulated conversation between the user and the system, the system can get the user's appearance but not the user's private profile. Therefore, we convey information about the user's appearance to the system agent, aiding in understanding and capturing user preferences. In practice, we further enhance system agents with self-augmented instructions, where the agent's prompts will be repeated in each conversation round to avoid forgetting the items' details. The specific system agent instruction template is shown in Figure 5 .The moderator agent is designed to automatically manage whether the conversation between the system agent and the user agent should be terminated. It also tracks whether the user agent accepts or rejects the recommended items based on their preset preferences. To ensure that the constructed data meets the desired characteristics, we set certain natural language conditions to terminate the conversation. These conditions are summarized as follows: (1) The system agent completes the recommendation, the user agent accepts it, and the recommended item aligns with the agent's predefined preferences. In addition, the system action is not topic transferred. (2) The user agent rejects the recommended items by the system agent multiple times (e.g., more than three times). (3) The interaction is deemed concluded once the conversation between the system agent and the user agent hits the maximum number of turns. Note that the synthesized conversation terminated under the first condition is accepted as valid data, while those that end under the second and third conditions are categorized as invalid and discarded. Figure 5 describes the specific moderator agent instruction template.In this study, the unique multimodal context of our conversational scenario integrates both visual (i.e., scene snapshots) and textual (including dialogue history and instructions) elements. To accommodate this complexity, the user and system agents are powered by GPT-4V (gpt-4-1106-vision-preview version), a variant of ChatGPT specially enhanced for multimodal tasks. Conversely, the moderator agent, which functions without reliance on visual cues, utilizes GPT-4 (gpt-4-1106-preview version) to navigate its decision-making processes effectively. The dialogue initiation occurs as the system agent greets the user agent, triggering a sequence of interactions that evolve through numerous dialogue rounds. These interactions are concluded ultimately with an intervention from the moderator agent. Collectively, these agents can collaborate to construct large-scale, high-quality dialogues rapidly, significantly reducing the need for human intervention.Our role-playing framework is built upon the open-source library ChatArena [37] . We have standardized the response generation across all agents by setting a temperature of 0.8. The maximum generation tokens are also tailored for each agent type, with limits set at 120, 80, and 20 for the system, user, and moderator agents, respectively. This structured approach ensures a balanced and efficient dialogue generation process, catering to the distinct needs of each agent's role in the conversational architecture. 
screen dataset: 
 Based on our dataset construction framework, we build a highquality SCRS dataset named SCREEN. Compared to related multimodal CRS datasets, our SCREEN uniquely targets situated recommendations. We first provide a comprehensive overview of the SCREEN dataset, then propose three essential sub-tasks to measure SCRS, including the task formulation and evaluation metrics.Table 1 presents a comparative analysis between the SCREEN dataset and other related datasets. To our knowledge, the SCREEN dataset is the first dataset within the SCRS domain designed to facilitate recommendations in distinct scenarios. In deviation from conventional text-based CRS datasets, such as REDIAL, SCREEN incorporates visual elements, thus enabling a more comprehensive modeling of item representations. While datasets like MMD and SURE also integrate visual information, they do not consider user preferences and item representations in specific scenarios. SIMMC 2.0 serves as a task-oriented dataset geared towards situational dialogues, while the SCREEN dataset distinguishes itself by concentrating on situated conversational recommendations. Moreover, the SCREEN dataset's inclusion of detailed, personalized information-namely, the Big-5 personality traits-in creating user agents ensures that the generated utterances are more natural and realistic. Table 2 presents a comprehensive analysis of the SCREEN dataset. As the table delineates, the dataset is divided into training, validation, and test sets, adhering to an 8:1:1 ratio. A notable observation is that sentences generated by the system agent are longer than those generated by the user agent. This discrepancy stems from the system agent's necessity to introduce detailed information regarding the items to the user. On average, each dialogue mentions approximately four distinct objects, while each conversational scene involves around 20 objects. A particular feature of the SCRS, as opposed to traditional CRS, is that each dialogue within SCRS is associated with a unique list of recommendation candidates, diverging from the conventional approach where all conversations access a communal candidate list. Consequently, the SCRS framework is required to model the representation of items within the conversational scene to provide appropriate recommendations.We delineate three sub-tasks to explore the performance of SCRS: system action prediction, situated recommendation, and system response generation. These subtasks are essential to validate whether an SCRS comprehends the situation, user intent, and conversational history nuancedly, which are critical for delivering accurate and contextually relevant recommendations. The system action prediction measures the ability to generate guided actions to satisfy user needs. The situated recommendation measures how well the situational context is utilized to tailor recommendations to user preferences. The system response generation focuses on crafting natural, coherent, and context-aware responses crucial for sustaining user engagement and satisfaction.As delineated in Section 3.3.2, the system agent determines its subsequent actions based on information derived from the dialogue history and the contextual scenario involving the user. This necessitates the system's capability to comprehend the user's intent, capture the user's preferences, and incorporate the attributes of items present within the scenario to decide the next step (e.g., make recommendations). The system's performance is quantitatively assessed by calculating the aggregate precision, recall, and F1 scores of the system's action predictions.Building upon the foundation laid by [17] , we extend the traditional tasks in CRS to encompass the situated recommendation task as a principal task within SCRS. This pivotal task requires the system to align items' attributes with the user's situated preferences, leveraging the scenario, dialogue history, and detailed item information to deduce the aptest recommendation for the user. It is important to note that a recommendation Table 3 : Automatic evaluation results of representative baseline models on three proposed subtasks based on the SCREEN dataset. The best results are highlighted in bold (𝑡-test with 𝑝-value < 0.05).System Response Generation Model Precision Recall F1 R@1 R@2 R@3 PPL (↓) BLEU-2 BLEU-3 DIST-1 DIST- is not mandated at each interaction, as users might seek insights into the item's attributes or other related information. Recommendations are thus made only when the system's action is explicitly to "make recommendations. " Following existing studies [20, 46] , we adopt a widely recognized metric for assessing the performance of recommendations. The automatic metric, Recall@𝑘 (R@𝑘, where 𝑘= 1, 2, 3), evaluates the accuracy of the top-𝑘 items recommended by the model against the ground truth items provided by the system agent.Response Generation. The objective of this subtask is to generate responses in natural language. The system is required to generate responses based on its decided actions, the historical context of the dialogue, snapshots of the scene, and information of items within the given scenario. Similar to [17] , our evaluation metrics include Perplexity (PPL) [15] , BLEU-2,3 [32] , and Distinct 𝑛-gram (DIST-𝑛, 𝑛 = 1, 2) [2] , to assess the quality of the responses generated by the system. Perplexity serves as an indicator of natural language fluency, wherein a lower Perplexity value signifies a higher degree of fluency. The BLEU-2,3 metric evaluates the concordance of word sequences between the generated responses and the reference responses, with higher BLEU scores indicating closer approximation to the reference responses. The DIST-𝑛 measures the diversity of the generated responses at the sentence level, with higher scores denoting a broader variety in sentence construction. 
experiments 5.1 baseline models: 
 We implemented and assessed several multimodal baseline models on our proposed SCREEN dataset: (1) SimpleTOD+MM Model [5] : It is an extension of the SimpleTOD model on the SIMMC dataset, supporting multimodal inputs. It views system action prediction as a causal language modeling task and finetunes the pretrained GPT2 language model to generate both system actions and responses.(2) Multi-Task Learning [18] : It utilizes multitask learning techniques to train a GPT2-based model, demonstrating robust performance across all tasks on the SIMMC dataset. (3) Encoder-Decoder [12] : It is an end-to-end encoder-decoder model based on BART for generating outputs, achieving first place in the overall ranking in the SIMMC competition. (4) Reasoner [26] : It employs a multi-step reasoning method and performs exceptionally well in the SIMMC 2.0 competition. (5) MiniGPT4 [47] : For this widely used multimodal LLM, we concatenate the dialogue history and scene snapshot as input for the model and view all three subtasks as response generation tasks to generate results. ( 6) GPT-4o [30] : It is a state-of-the-art multimodal LLM developed by OpenAI. To ensure a fair comparison, we follow the same setting as MiniGPT4 and adopt official configurations during inference.The automatic evaluation results for the three subtasks on the SCREEN dataset are presented in Table 3 , with the best metrics highlighted in bold. GPT4o received the highest score, as expected. Among the open-source models, MiniGPT4 outperformed other models across all subtasks, benefiting from its advanced language understanding and generation capabilities based on an LLM. In contrast, SimpleTOD+MM and MultiTask Learning, based on GPT2, showed weaker performance. The Encoder-Decoder and Reasoner models performed similarly, though the Reasoner had a slight edge due to its dual-system mechanism. Notably, all models struggled with situated recommendations, underscoring the challenge of capturing user preferences in specific scenarios. Even GPT-4o, while accurate in system action prediction, faced difficulties in recommending items and generating responses, two critical tasks in SCRS.We engaged three well-educated annotators to manually evaluate the system-generated responses. To assess the relevance of these responses to the contextual scene, we introduced a novel metric, Situated Relevance. This metric evaluates whether the responses accurately reference items in the scene and consider the user's appearance and climate conditions. Additionally, we employed the criteria from [43, 45] to assess Fluency and Informativeness. Each indicator was scored on a scale from 0 to 2, where 0 indicates no relevance, informativeness, or fluency, and 2 signifies high relevance, rich information, and smooth fluency. To determine interannotator agreement, we calculated Fleiss's kappa [8] and aggregated the scores to derive the average human-evaluated results. The human evaluation results in Table 4 show that Fleiss's kappa scores are within the [0.4, 0.6] range, indicating moderate agreement among annotators. These findings closely align with the results from automatic evaluations, supporting the effectiveness of the three designed subtasks in assessing SCRS performance. Notably, GPT-4o and MiniGPT4 outperform other models in generating more situation-relevant, fluent, and informative responses. Although the Reasoner and the Encoder-Decoder models demonstrate comparable levels of situation relevance and fluency, the Reasoner's outputs are more informative to its multi-step reasoning process that gathers necessary elements for generation.We conducted a human evaluation to verify the reliability of the SCREEN dataset. We randomly selected 50 dialogues each from SCREEN and SIMMC 2.1, forming dialogue pairs, and asked five human evaluators to assess these pairs based on the following criteria: "Situation Relevance, " which determines which dialogue is more relevant to the scene; "Personality," which evaluates which dialogue better reflects the user's personality; "User State, " which assesses which dialogue considers the user's mood and appearance more; and "Rec. Appropriateness, " which judges which dialogue's recommendation is more appropriate. The comparative results, presented in Figure 6 , indicate that the SCREEN dataset achieves higher win percentages than the artificially generated SIMMC 2.1 dataset. This outcome demonstrates the reliability of our dataset.To demonstrate the quality of responses generated by those baseline models on the SCREEN dataset, we present an illustrative case in Figure 7 . We observe that Reasoner and MiniGPT4 can successfully utilize contextual information from the scene (e.g., climate: summer) and conversational history (e.g., playing basketball) to make appropriate recommendations. While SimpleTOD+MM also attempts a recommendation, it fails to specify the clothing recommended. The MultiTask Learning and Encoder-Decoder models limit their outputs to mere descriptions, omitting recommendations. Significantly, MiniGPT4 demonstrates an enhanced ability to generate responses enriched with informative content, underscoring the advanced capabilities of LLMs. Nonetheless, we draw a conclusion that there is still a large room to improve these baseline models to fully address situated recommendations, remaining huge research potential in the future.We also identify some limitations of this work as follows. Utilizing LLM agents to simulate predefined roles in developing SCRSA clothing store Location: Late afternoon Time:The weather is relatively dry, and the hot air indicates summer.The user is in an excited and joyful mood.The user wears a blue long-sleeved shirt and sports pants with a crew-cut hairstyle.This brown pair of pants looks very good.Yes, it looks great.The weather is so hot. I believe you will like this black top.Of course, I recommend this black short-sleeved T-shirt to you. It is very suitable for playing basketball.Yes, the weather is hot. I would recommend this black top with a flame pattern in the middle. It makes you look cool and is suitable for playing basketball.Hello, I want to buy a sportswear suitable for playing basketball. Do you have any recommendations?Of course, do you prefer long sleeves or short sleeves. System: I usually prefer to wear long sleeves, but the weather is too hot, so I want to buy a short-sleeved one this time. still poses some challenges. Despite efforts to increase variability through controlled settings, LLMs occasionally generate responses with hallucinations [1] . In the future, post-processing measures such as verification and corrections by multiple moderators will be designed to enhance dataset quality. Moreover, rigorous ethical considerations are paramount, especially in preventing the generation of harmful content and ensuring no sensitive and private information should be involved. To some extent, this can be alleviated through manual sampling inspection. 
conclusion: 
 This work proposes a novel problem setting named Situated Conversational Recommendation System (SCRS) that enhances traditional multimodal conversational recommendations by integrating situational factors. To facilitate advancements in this field, we construct a comprehensive, high-quality benchmark named SCREEN using an efficient role-playing approach based on multiple LLM agents. Furthermore, we define three essential subtasks for SCRS and evaluate several representative baseline models, moving to a new research direction that narrows the gap between traditional and real-world conversational recommendations. 
 
 Appendix:  


Decision: The reviewers have different opinions on their evaluation of this article. After reading this paper, I think the paper meets MM standards, and lean towards acceptance.

Reviews: review: paper_summary: paper_summary: This paper construct a new benchmark dataset for a novel task of situated conversational recommendation. It first proposes this new task formulation, then use LLMs as agents to simulate and build the dataset. Multiple baseline methods are implemented on this dataset.
strength_weakness: strength_weakness: 1. The motivation of introducing situation into the task of conversational recommendation makes sense. 
2. The idea of leveraging LLMs and agents to simulate and study the problem is reasonable and promising. 
3. The overall formulation and system design of the role-playing (agent) system make sense.
limitations: limitations: 1. The dataset is constructed using more advanced LLMs such as GPT-4V and gpt-3.5-turbo, while the strongest LLM-based baseline is MiniGPT4, which is much weaker than the LLMs used for dataset construction. I am wondering how such advanced LLMs perform in this dataset. I am worrying about that the user actions in the dataset are generated by LLMs, and a consistent and integrated LLM can predict the actions very well especially when the actions are generated by itself. 
2. A follow-up question is there is no direct and systematic evaluation of the quality of the dataset. It is unclear to what degree that the generated dataset is aligned with real world scenarios. If such dataset deviates much from real use cases, it would be less meaningful or reliable to conduct study on this dataset. 
3. There should be a formal definition of the situation. Since situation is the main focus of this work, it is necessary to give formal definition of this key concept, for example, what characterstics, properties, or functions should be conveyed by situation. Even though there are examples such as temproal, spatial, climate, etc, it lacks an overall comprehensive definiton.
4. Using agent to study the conversation recommender system is a good idea, however I respectfully disagree to use it as a dataset construction tool. Because LLMs are sufficiently powerful to solve such problems, and we can directly use them as solutions rather than restrict them to build datasets.
paper_summary: paper_summary: In this paper,  the author pioneering to address this gap and introduce a novel setting: Situated Conversational Recommendation Systems (SCRS). We observe an emergent need for high-quality datasets, and building one from scratch requires tremendous human effort. To this end, they construct a new benchmark, named SCREEN, via a role-playing method using large language models. This benchmark
comprises over 20k dialogues across 1.5k diverse situations, providing a rich foundation for exploring situational influences on conversational  recommendations. Based on the SCREEN, the author propose three worth-exploring subtasks and evaluate several representative baseline models. The evaluations confirm that the benchmark is high quality, establishing a robust experimental basis for future research in situated conversational recommendation.
strength_weakness: strength_weakness: The paper aims to construct a benchmark dataset for recommendation tasks that is more in line with real-life scenarios, as there is limited situational information in existing recommendation tasks. This dataset has certain data value for recommendation tasks.
limitations: limitations: 1 Is the dataset that integrates scenarios compatible with real scenarios and is it universal for recommendation tasks.
2 The paper proposes multiple baseline methods for experimental validation of the dataset, proving that the dataset can measure the advantages and disadvantages of the methods, as well as data quality analysis. But has this dataset been publicly evaluated through evaluation competitions?
3 What are the advantages of analyzing the size of the dataset compared to existing recommendation task datasets, only by integrating scenarios? The comparative experiments conducted in the paper are not sufficient.
paper_summary: paper_summary: This paper introduces Situated Conversational Recommendation Systems (SCRS) to address the impact of situational factors on recommendations. The paper employs large language models to perform multi-role simulations, thereby establishing a novel benchmark dataset based on the SIMMC dataset. The paper has a good structure and is easy to read. This paper provides a detailed description of the experimental procedure. Particularly, Figure 2 clearly illustrates the main steps involved in the dataset construction, facilitating the reproducibility of this study. Furthermore, the paper offers a comprehensive analysis of various tasks based on the proposed benchmark, including both model evaluation and human evaluation.

However, I have two concerns:
1. Since all the utterances in the dataset, including the Ground Truth, are generated by GPT-4, could there be an inherent bias favouring the ChatGPT series of large models when using this benchmark for evaluation? 
2. Although the authors mention in Section 5.3 that the results of human evaluation are consistent with those of automatic evaluation, I think merely observing the ranking of models is insufficient. It is necessary to further validate the consistency between automatic evaluation metrics and human annotations on the new benchmark. For example, calculating the correlation between metric and human scores, or using Predictive Power, may help to assess the reliability of this benchmark.
strength_weakness: strength_weakness: 1. This paper provides a detailed description of the experimental procedure.
2. Extensive analyses are conducted.  
3. The paper has a good structure and is easy to read.
limitations: limitations: Please see the summary.
paper_summary: paper_summary: The authors introduce a new Benchmark for Situated Conversational Recommendation Systems. Compared with the existing datasets, they leverage LLM agents via role-playing approach to simulate the environmental and situational context for both the user and system such as the climate, user personality, etc. They also propose three subtasks to evaluate the performance of the existing baselines.
strength_weakness: strength_weakness: 1.	This is the first dataset specifically designed for situated conversational recommendation systems, addressing the gap in the current research landscape.
2.	The dataset incorporates a wider range of situational factors that influence user behavior in SCRS scenarios, providing a richer context for evaluation.
3.	The paper is well-organized and easy to follow.
limitations: limitations: 1.	While the dataset includes various situational factors like climate and personality traits, it is unclear if these factors significantly influence users' purchase behaviors. The practical applications and the potential impact of the dataset for real-world scenario should be further discussed. 
2.	There is a lack of detailed description regarding the measures taken to ensure the quality and accuracy of the generated dialogues. Additionally, more data statistical analysis is needed to help users understand the dataset comprehensively
3.	Most of the baseline models used are from the SIMMC dataset. It would be beneficial if the authors could design a simple, specific baseline tailored to the proposed dataset to better evaluate its unique aspects.
4.	It is not specified whether the authors will make the dataset publicly available.