LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play

Authors: li-chun  lu
shou-jen  chen
tsung-min  pai
chan-hung  yu
hung-yi  lee
shao-hua  sun
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 09:43:40

Abstract: Large language models (LLMs) have shown exceptional proficiency in natural language processing but often fall short of generating creative and original responses to open-ended questions. To enhance LLM creativity, our key insight is to emulate the human process of inducing collective creativity through engaging discussions with participants from diverse backgrounds and perspectives. To this end, we propose LLM Discussion, a three-phase discussion framework that facilitates vigorous and diverging idea exchanges and ensures convergence to creative answers. Moreover, we adopt a role-playing technique by assigning distinct roles to LLMs to combat the homogeneity of LLMs. We evaluate the efficacy of the proposed framework with the Alternative Uses Test, Similarities Test, Instances Test, and Scientific Creativity Test through both LLM evaluation and human study. The results show that our proposed framework outperforms single-LLM approaches and existing multi-LLM frameworks across various creativity metrics. The code is available at https://github.com/lawraa/LLM-Discussion.

Main paper: 
 introduction: 
 Large language models (LLMs) have emerged as highly efficient tools in addressing daily challenges across various applications, demonstrating exceptional capabilities in natural language processing (Lappuin, 2024; Yang et al., 2023b) . Specifically, LLMs achieve excellent performance in language comprehension tasks, such as sentiment analysis (Zhang et al., 2023a) and question answering (Qin et al., 2023) . Also, LLMs are widely adopted in content generation tasks, ranging from writing articles to composing poetry, by producing coherent and contextually relevant text that closely mimics human-written content (Qin et al., 2023; Yuan et al., 2022) . Nevertheless, recent studies suggest that LLMs demonstrate limited creativity in answering open-ended questions and often fail to produce original responses (Ippolito et al., 2022; Chakrabarty et al., 2023; Mohammadi, 2024) .Psychological studies suggest that engaging in discussions with participants from diverse backgrounds, perspectives, and experiences can significantly enrich individual creativity (Han et al., 2022; Paulus & Nijstad, 2003; McGrath, 1984; Sutton & Hargadon, 1996; Karwowski & Soszynski, 2008) . Can LLMs likewise strengthen their creative capabilities through collaborative discussions with diversified peers? While Du et al. (2023) ; Liu et al. (2023b) ; Sun et al. (2023) explore collaboration among LLMs, most existing works are limited to improving the performance on close-ended tasks, such as Massive Multitask Language Understanding (Hendrycks et al., 2021) , Mathematics (Cobbe et al., 2021) , and Code Generation (Huang et al., 2023) , leaving discussions on creativity largely under-investigated. On the other hand, prior works researching LLM creativity are mostly restricted to a single LLM agent (G ómez-Rodríguez & Williams, 2023; Stevenson et al., 2022) .Our goal is to boost the creativity in LLMs by designing a discussion framework. Subsequently, two main challenges arise due to the nature of LLMs. First, most modern commercialized LLMs are not specifically trained or reinforced to engage in multi-turn conversations (Ouyang et al., 2022; Ding et al., 2023) , which results in unengaging discussions among LLMs that quickly converge. Second, the high homogeneity of LLMs renders it difficult for discussions to diverge and produce creative outcomes (Ouyang et al., 2022; Padmakumar & He, 2024) .Let's start by analyzing it from some of its features!Sure! I think forks that are made of metal are reusable and durable.Agreed. How about using it for outdoor installation art for higher value? To address these issues, we develop a role-play enhanced LLM discussion framework, dubbed LLM Discussion, as illustrated in Figure 1 . To facilitate vigorous and diverging exchanges throughout rounds of discussion and ensure convergence to creative answers, we arrange a three-phase discussion and design prompts specialized in the initiation phase, the discussion phase, and the convergence phase. Then, to tackle the LLM homogeneity, we adopt the role-playing technique, an established practice in Design Thinking (Camacho, 2016) . Specifically, we propose to assign a diverse set of roles with distinct backgrounds and perspectives to LLM agents participating in a discussion, such as Visionary Millionaire and Futurist, and roles specialized for various facets of discussion from Six Thinking Hats methodology (Setyaningtyas & Radia, 2019) . 
approach: llm discussion: 
 Our key insight is to boost creativity in LLMs by designing an effective discussion framework and employing role-play techniques. To this end, we propose LLM Discussion, a three-phase discussion framework that incorporates role-playing, allowing multiple LLM agents to build on each other's thoughts and collectively produce creative outcomes. Specifically, to combat the inability of LLMs to conduct engaging multi-turn discussions, we propose to arrange three different phases, the initiation, discussion, and convergence phases, as described in Section 3.1. In each phase, a short-term objective is provided to each LLM with a specialized prompt, e.g., actively discuss or efficiently converge, facilitating both divergent and convergent thinking. Then, to alleviate the LLM homogeneity issue, i.e., all LLMs think alike, Section 3.2 presents our proposed role-play mechanism, which assigns diverse backgrounds, perspectives, and personalities to LLMs.We aim to design a discussion framework that hosts rounds of dynamic exchanges of ideas, progressively fostering novel solutions and deeper insights. However, most modern commercialized LLMs are not specifically trained or reinforced to engage in multi-turn conversations (Ouyang et al., 2022; Ding et al., 2023) , resulting in unengaging discussions among LLMs that quickly converge. To facilitate meaningful interactions among LLMs, Du et al. (2023) introduce LLM Debate, which requires multiple LLMs to propose and debate their individual responses with other LLMs over multiple rounds to arrive at a joint answer agreed upon by all the LLMs. Yet, LLM Debate is devised to improve LLM's performance in answering closed-ended questions, such as question answering (Hendrycks et al., 2021) and mathematical reasoning (Cobbe et al., 2021) . In particular, the mechanism employed in LLM Debate requires each LLM to maintain its own answer while observing answers produced by other LLMs, which is effective in resolving factual questions but can fall short of promoting divergent thinking building upon others' responses.To encourage LLMs to discuss with others and inspire others actively to engender collective creativity, we devise a three-phase discussion framework that explicitly requires each LLM to build upon others' responses. We arrange an initiation phase, a discussion phase, and a convergence phase, which are described in detail below. In each phase, each LLM aims to fulfill an objective, e.g., actively discuss or efficiently converge, as instructed by a specialized prompt. The proposed discussion framework is illustrated in Figure 2 .Initiation Phase. This phase serves as the groundwork for the discussion, where each LLM is introduced to the topic and the objectives of the dialogue. In this phase, we aim to explicitly inform the LLMs about the situation and declare the goal. Specifically, we notify each LLM that it is in a group discussion setup by stating "you are in a group discussion with other teammates" in the prompt. Moreover, we stress the importance of teamwork spirit with the instruction "Initiate a discussion with others to collectively complete the following task." Then, we provide a description of the task that we want the LLMs to solve, e.g., "come up with creative uses for forks." Discussion Phase. The goal of this phase is to induce multiple rounds of discussions carrying meaningful idea exchanges among LLMs. In each discussion round, each LLM receives responses from all other agents in the previous round, resembling the idea of active listening in human discussions. Additionally, we aim to encourage each LLM to compose and diverge its answers. For each LLM to balance between generating its own answers and following up on others' thoughts, we experiment with various prompts as discussed in Section 4.5.Convergence Phase. After rounds of diverging and creative discussions, the convergence phase aims to summarize the ideas brought in discussions and converge to a collective conclusion. To this end, we inform each LLM by prompting it with the instruction, "This is the last round of the discussion; please finalize and present a list of creative answers." Note Discussion PhaseInitiate a discussion with others to collectively complete the following task: + {Task} + You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can.These are the solutions to the problem from other agents: One agent solution```{sol. 1}``` + … + One agent solution```{sol. N}``` + {Task} + You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can. Repeat N-2 Rounds Final Answers 2 nd -(N-1) th Rounds 1 st Round N th Round These are the solutions to the problem from other agents: One agent solution```{sol. 1}``` + … + One agent solution```{sol. N}``` + {Task} + You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can. This is the last round of the discussion; please finalize and present a list of creative answers.Figure 2 : Discussion Framework. We propose an LLM discussion framework that induces collective creativity by bringing divergent and convergent thinking together. The initiation phase informs LLMs of the discussion setup and the task. The discussion phase allows LLMs to build on the ideas of others as well as diverge and generate their own answers. The convergence phase summarizes the discussed ideas and draws a collective conclusion. that the term "finalize" in our convergence prompt serves as an explicit prompt to conclude, while "present a list" facilitates parsing.Full Discussion Procedure. Putting everything together, our proposed discussion procedure with N rounds is described as follows.• 1 st Round: In the first discussion round, we initiate the discussion by prompting LLMs using the prompt described in the initiation phase.• 2 nd to (N -1) th Rounds: To encourage multiple rounds of discussions, we repetitively prompt each LLM and provide it with responses from other LLMs using the prompt presented in the discussion phase.• N th Round: The goal of the last discussion round is to summarize the ideas and converge to a collective conclusion. Hence, the LLMs are prompted with the convergence phase prompt. Note that we also provide the context by first prompting LLMs with the discussion phase prompt.A five-round discussion qualitative result is shown in Section K.We empirically observe that even with engaging discussions, LLMs often fall short of producing diverging answers given the same prompts due to their homogeneity, which aligns with recent findings reported by Ouyang et al. (2022) ; Padmakumar & He (2024) .To address this issue, we propose to employ role-play, a technique commonly used in the ideation stage of Design Thinking (Camacho, 2016) . This technique aids people in coming up with innovative ideas for various purposes, unlike problem-oriented roles, which focus on solving specific issues within distinct fields.Role Generation. To generate a set of roles with diverse backgrounds, perspectives, and experiences and detailed descriptions of these roles, we propose an automated pipeline that uses GPT-4 (OpenAI et al., 2023) to produce a list of {role, speciality, role prompt}, e.g., {Visionary Millionaire, Financial success and forward-thinking, As a Visionary Millionaire, your mission is to leverage your financial insight and forward-thinking approach to inspire groundbreaking ideas ...}. The detail process and generation prompts are presented in Section J. We also adopt Six Thinking Hats methodology (Setyaningtyas & Radia, 2019) to give LLMs different perspectives, e.g., the red hat represents emotions, and the green hat stands for innovation. Role-Play Enhanced Discussion. To incorporate the roles into our discussion framework, we assign different roles to different LLMs before the initiation phase. Specifically, we adopt an autoregressive way to select the roles for each member in a discussion. Then, at the beginning of each discussion round, we remind LLMs of their roles and responsibilities and ask them to declare their roles. This ensures that other members of the discussion are aware of each speaker's identity, enabling a more structured and coherent exchange of ideas. Note that the roles of LLMs stay unchanged throughout the entire discussion. An illustration of role assignment and examples of roles are shown in Figure 3 . 
benchmarks: 
 Creativity encompasses several distinct aspects, including divergent thinking, connection-making, and practical innovation. To measure LLM creativity, we utilize four existing renowned benchmarks for human creativity in our evaluation, each responsible for assessing distinct aspects of creativity. We adopt three benchmarks from the widely-used Wallach-Kogan Creativity Tests (Wallach, M. A., and Kogan, N., 1965) : Alternative Uses Task (AUT), Instances Test (INSTANCES), and Similarities Test (SIMILARITIES). We also include the Scientific Creativity Test (Hu & Adey, 2002) (SCIENTIFIC), which measures scientific creativity and scientific process skills.AUT assesses the ability to develop practical and innovative use cases for objects like a fork or a shoe. INSTANCES requires participants to list things following specific criteria e.g., things that move on wheels or things that are square. SIMILARITIES evaluates creativity in making connections, challenging participants to brainstorm and link ideas through commonalities, e.g., how a book and a magazine are alike. Lastly, SCIENTIFIC evaluates creativity in scientific contexts and the ability to devise practical and creative solutions to scientific problems. Some sample tasks of these benchmarks are presented in Table 1 .Task augmentation. Due to the limited public availability of the tasks from these benchmarks, we generate additional tasks using GPT-4 (OpenAI et al., 2023) . Specifically, for each benchmark, we provide GPT-4 with publicly available task samples and ask it to generate 30 additional tasks (Liu et al., 2023a; Ubani et al., 2023; Fang et al., 2023; Peng et al., 2023) . Then, we carefully review the generated tasks to ensure their relevance and alignment with the benchmarks. We aggregate the original and generated tasks to form each benchmark for evaluation. We will release the full task sets to facilitate future research studying LLM creativity.We compare our proposed framework, LLM Discussion, with the following methods.• Single Agent involves querying a single LLM agent to solve the task on its own. We experiment with various prompting techniques, such as few-shot prompting (Brown et al., 2020) , zero-shot-CoT (Kojima et al., 2022) , LLM Stimuli (Li et al., 2023b) , and the take-a-deep-breath prompt (Yang et al., 2023a) , and the results are shown in Table 5 . Since no particular prompt consistently outperforms others, we select the zero-shot prompt, i.e., with only task description, as our main baseline.AUT This benchmark requires participants to come up with as many creative uses as possible for a simple object.What are some creative uses for a fork?This benchmark requires participants to list as many creative answers as possible that fit a specific criterion.Name all the round things you can think of.This benchmark asks participants to creatively explore similarities between two things.Tell me all the ways in which a book and a magazine are alike.This benchmark asks participants to generate innovative ideas, envision hypothetical scenarios, propose improvements, and design experiments, all scientifically.Please think of as many possible improvements as you can to a regular bicycle, making it more interesting, more useful, and more beautiful. It includes a selection phase where potential solutions are judged against specific criteria to identify the most original responses. Since it is designed for AUT, we only present the AUT results of this method. Also, BTS is a single-LLM framework, unlike our proposed framework, which encourages iterative enhancements through multi-round discussions and dynamically evolving the brainstorming process rather than filtering static responses. • LLM Debate (Du et al., 2023) is a multi-LLM framework aiming to improve factuality and reasoning ability of LLMs by requiring LLMs to debate their responses and reasoning processes over multiple rounds. Specifically, the LLM Debate asks each LLM to examine responses from other LLMs and verify the correctness and reasonableness instead of following up on the ideas brought up by others. In contrast, our LLM Discussion framework focuses on developing collaborative discussions with LLMs assigned diverse roles, allowing for the emergence of collective creativity.More details of these methods can be found in Section A.We use LLMs and humans to quantitatively evaluate the creativity of generated responses.We employ the metrics established by the Torrance Tests of Creative Thinking (TTCT) (Torrance, 1966) , recognized for their robustness in measuring creative capabilities and are commonly applied in assessing human creativity (Cramond et al., 2005) . The TTCT framework comprises four distinct metrics:• Originality: Considering novelty responses, not familiar and unusual, but relevant.• Elaboration: The amount of details used to extend a response. LLMs have demonstrated the ability to resemble human judgment and reasonably evaluate contents, achieving results comparable to human (Chan et al., 2024; Wang et al., 2023a; Adlakha et al., 2023; Kocmi & Federmann, 2023) . As a result, we adopt LLMs to evaluate the creativity of generated responses. Specifically, we use ChatGPT (Wang et al., 2023a) for the LLM evaluation.In our LLM evaluation, responses are assessed individually on Originality and Elaboration and collectively for Fluency and Flexibility. That said, the response to "Creative uses of a fork" might be [Eating Pasta, Eating Pancake, Art projects, Drilling]. For Originality and Elaboration, each response receives an individual score, for example, [2, 2, 3, 1] and [2, 2, 2, 1], respectively. For Fluency and Flexibility, the entire list of responses is collectively evaluated three times, receiving an average score, for instance, 3.33 and 2.33, respectively.We also conduct human evaluation on the responses generated by our method and the baselines. Human evaluation is conducted using the same metrics as those used in LLM evaluation. To ensure consistency across both setups, the same rubric used in the LLM evaluations is also presented to human evaluators, as shown in Section C. In total, 1,406 responses from 42 distinct annotators are collected online for evaluating the creativity of generated responses. Additionally, we investigate the correlation between human and LLM evaluations in Section 4.4.2.Our experiments and evaluation both utilize the gpt-3.5-turbo-0125 model.We present the LLM evaluation results across four benchmarks in Table 2 . Our proposed framework, LLM Discussion, outperforms the baselines (Single Agent, Brainstrom, then select, and LLM Debate) in Originality on four benchmarks and Elaboration on three benchmarks. In the following, we discuss the qualitative results observed in the chat logs provided in Section K from three perspectives.Collaborative Dynamics. Our three-phase discussion framework stimulates a more collaborative tone; an example sentence is "Building on those ideas, I believe we can combine..."This results in more distinctive responses generated in each round, reducing the LLM homogeneity issue. In contrast, LLM Debate tends to be more discriminative, merely collecting or correcting previous responses without following up on ideas brought up by others.Role-Specific Responses. Our proposed role-play technique encourages responses specifically aligned with the assigned roles, diversifying the creativity from the perspectives of various fields rather than being limited to general purposes. For example, in an AUT scenario of an umbrella, the role, Futurist, proposes integrating it with VR technology, while the role, Environmentalist, suggests using it as a shelter for wildlife. In contrast, Single Agent and LLM Debate tend to suggest more general uses, such as re-purposing the umbrella as a tray or as a quirky headpiece for a costume. The qualitative results from the AUT and SCIENTIFIC benchmarks presented in Figure 4 show that our LLM Discussion received higher Originality scores by producing more creative responses compared to other frameworks. The impact of temperature on creativity score is further discussed in Section H.A creative use for "Towel"How can you test which of two smartphones has a longer battery life? Explain your testing approach, the necessary tools, and the procedure.Use a towel as a picnic blanket in the park.Turn a towel into a cozy seat cover for long car rides. Repeat the test multiple times under varying conditions to ensure the results are consistent and reliable.Integrating AR Technology: Develop an AR application that overlays real-time battery usage data onto the smartphones, enabling participants to visually compare battery life while engaging in everyday tasks, creating an immersive and interactive testing experience. Human evaluation results presented in Table 3 show that our proposed LLM discussion outperforms the baselines in Originality and Elaboration across three out of four benchmarks, aligning with the LLM evaluations.We investigate the correlation between the LLM evaluation and human evaluation by calculating Kendall's τ correlation coefficient between LLM and human evaluations, and each human evaluator against the average ratings of the others human evaluators, as shown in Table 4 . (Botsch, 2011; Chiang & yi Lee, 2023; Zhang et al., 2023b) . The results show a strong correlation between LLM and the average human evaluation, except for Elaboration, where only a moderate correlation is observed.In particular, for the SIMILARITIES, the Elaboration scores from human evaluations exceed those from LLM evaluations. Consider the response to the prompt, "Tell me one way in which a donkey and a horse are alike", where the answer provided was: "Interstellar Companions: Looking towards interstellar exploration, these animals might pioneer new frontiers in space travel, accompanying humans on missions and serving as invaluable companions and assistants in uncharted realms of the cosmos." This response was scored 1 point on Elaboration by LLM evaluators but received an average of 3.88 from ten human evaluators. The potential reason is that human evaluators tend to favor lengthier responses, as shown in Section E, whereas LLM evaluations focuses on whether the response addresses the question and prioritize the development of ideas over mere verbosity. We also analyze the average length of the answers and their impacts on each benchmarks for Elaboration and Originality in Section F.Interestingly, the correlation coefficients of LLM to the average human scores were higher than those among human evaluators, suggesting that human scores have a higher variance. This indicates that LLM evaluations are more correlated with the average human score than individual human evaluations are with the average human score.We conduct ablation studies in AUT on different prompts, the number of discussion rounds, and the number of LLM agents involved in discussions to determine the settings for the main experiments. Also, we verify the effectiveness of role-play and 3-phase discussion framework respectively.Role-Play and 3-Phase Discussion Framework. We study the effect of role-play and discussion prompts individually. Section I shows the results of three different conditions of multi-LLM frameworks on each task: role-play without 3-phase discussion prompts, 3-phase discussion prompts without role-play, and both 3-phase discussion and role-play prompts (LLM Discussion).Solely role-play or 3-phase discussion received better scores in Originality and Elaboration compared to single agent and LLM Debate. LLM Discussion, which features both discussion framework and role-play, has the best performance among the three, outperforming the other two across four out of eight Originality or Elaboration of each benchmark.Prompt Design. We design various specialized prompts for our proposed three-phase discussion framework, aiming to explore how the prompts affect creativity in a discussion. We evaluate the prompt designs and present the results in Section D, which determines the prompt as the main prompt for further experiments.We examine the effect of the number of discussion rounds and present the results in Figure 5 . We can observe that Originality increases but Elaboration decreases when increasing the discussion rounds. As as result, we settle for five discussion rounds for further experiments.We evaluate the influence of the number of LLM agents participating in discussions and present the results in Figure 6 . With five-round discussions, involving four LLMs achieves the best overall performance. Therefore, we use four LLMs with five discussion rounds for further experiments.  
llm discussion: 
 We propose LLM Discussion, which enhances LLM creativity by emulating the human process of discussion among LLMs. The proposed framework facilitates multiple rounds of engaging discussions by employing three phases with specialized prompts, e.g., actively discussing or efficiently converging. Moreover, to alleviate the LLM homogeneity issue, i.e., all LLMs think alike, we propose a role-play mechanism, which assigns diverse backgrounds, perspectives, and personalities to LLMs. Through our comprehensive LLM evaluation and human study, our proposed framework outperforms single-LLM methods and existing multi-LLM frameworks in Originality and Elaboration across four benchmarks.Despite the encouraging findings presented in this work, a few aspects can be investigated in the future to deepen the understanding of LLM creativity. First, while the LLM evaluation and human study exhibit correlations, it is often challenging to quantify creativity. Hence, more comprehensive evaluations of creativity, such as interviewing experts in creativity, can provide insightful findings. Second, our proposed framework is designed to enhance LLM creativity by employing a group of LLMs discussing with each other. We believe exploring collective creativity induced by human-LLM interaction is another exciting direction.For single LLM, we conduct the experiment with zero-shot, few-shot (Brown et al., 2020) , zero-shot-CoT (Kojima et al., 2022) , LLM stimuli (Li et al., 2023b) , and "Take a deep breath" (Yang et al., 2023a) . As for multi-LLM, we reproduce the Brainstorm, then Select (Summers-Stay et al., 2023) on AUT, specifically designed for AUT, and LLM Debate (Du et al., 2023) on all the datasets as our baseline.• Few-shot. Provides a few examples to guide the model, enhancing performance with contextual cues.• Zero-shot CoT. Prompts the model to generate intermediate steps before the final answer, improving reasoning.• LLM stimuli. Uses specific emotional stimuli such as "This is really important for my career" to influence its performance.• Take a deep breath. Uses the prompt, "Take a deep breath and work on this problem step-by-step", discovered by Optimization by PROmpting(OPRO), that has shown to outperform human-designed prompts. Table 6 and Table 7 show the settings of our role-play and six thinking hats experiments, including prompts, specialty, and roles. Table 8 shows the evaluation results of the single agent with both settings.Financial success and forwardthinking As a Visionary Millionaire, your mission is to leverage your financial insight and forward-thinking approach to inspire groundbreaking ideas. Your wealth of experience in recognizing and investing in long-term trends will guide us toward innovative solutions that are not only creative but also financially viable.As a Startup Founder, your agility, knack for innovation, and willingness to take risks empower you to challenge the status quo. Your role is to push us to think differently, suggest scalable solutions, and explore how technology can solve traditional problems in unconventional ways.EntrepreneurAs a Social Entrepreneur, you bring a deep commitment to societal change through business. Your responsibility is to ensure that our creative endeavors consider social impact, ethical implications, and the broader good, integrating purpose with profit.Aesthetics, narratives, and emotions As a Creative Professional, your artistic sensibility and mastery of narrative and emotion infuse our projects with beauty and depth. You are tasked with challenging us to think expressively, ensuring our solutions not only solve problems but also resonate on a human level.As the voice of the Customer/User, your role is to anchor our creative discussions in the real-world needs and preferences of those we serve. Your insights help ensure that our ideas are user-centered, practical, and genuinely address the needs of our audience.As an Environmentalist, your mission is to champion ecofriendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth.As a Digital Nomad, your expertise in remote work and the digital lifestyle opens our eyes to the possibilities of the digital economy. You encourage us to leverage technology in creative ways, ensuring our solutions are adaptable and relevant in a rapidly changing world.As an Industry Insider, your deep understanding of specific sectors provides us with insider knowledge and awareness of industry trends. Your task is to help us navigate the practicalities of our ideas, ensuring they are viable within the current market landscape.They can introduce data-driven insights, theoretical frameworks, and evidence-based perspectives to ground creative ideas in solid research.Emerging technologies and future scenarios As a Futurist, you inspire us to think beyond the present, considering emerging technologies and potential future scenarios. Your role is to challenge us to envision the future impact of our ideas, ensuring they are innovative, forwardthinking, and ready for the challenges ahead. Focuses on available data and past information, analyzing trends and gaps in knowledge, striving for an objective viewpoint.Listens to and validates the emotional responses of the group, understanding the values and intuition behind reactions, without judgment or justification.Critically examines all potential flaws and risks, focusing on judgment to avoid pitfalls, ensuring the group is wellprepared for challenges.Explores the positives and the value of decisions, promoting a hopeful and constructive outlook, and highlighting paths to success.Encourages the generation of new ideas and alternative solutions, fostering an environment of creativity and innovation.Oversees and manages the thinking process, ensuring that each hat is utilized effectively and that discussions remain structured and focused. Table 9 shows our prompts for evaluation on 4 different metrics.This ablation study helps us to choose the most effective prompt for our main experiments. Table 10 presents the candidates for our initiation phase prompt generated by GPT-4, and the results are shown in Table 11 . We chose prompt 1 as our official experiment prompt since it has the most stable performance on 4 datasets.Originality You are a helpful assistant and a critical thinker. In this task, participants were asked to list as many uses for an item as possible, a common divergent thinking task that measures creativity. Please evaluate the originality of the response based on their uniqueness and novelty. Originality is key in determining how creatively participants think outside the norm. Rate the overall originality on a scale from 1 to 5, and conclude with the score in the format: ' Upon discovering that Kendall's τ correlation coefficient diverges in the Elaboration metric between LLM evaluation and human evaluation, we conduct an experiment to examine the impact of answer length on the elaboration scores assigned by humans and LLMs. Furthermore, Figure 7 and Figure 8 shows that within same word length intervals, our framework, LLM Discussion, outperforms other baselines in Originality and Elaboration in most of the benchmarks.We do not prioritize Fluency and Flexibility as main metrics for a few reasons:• Possible Inflated Score: Language models can inherently generate a large number of responses, which might artificially inflate flexibility scores. • Focus on Core Creativity Metrics: We prioritized originality and elaboration because they more directly reflect the model's creative potential. Temperature is known to affect the diversity of model outputs in language generation tasks.To investigate its impact on creativity scores, we experiment with increased temperatures ranging from 0.0 to 2.0 (1.0 was used for the main experiments) on AUT. The result is shown in Figure 9 . The results show that increasing temperature leads to higher diversity (a lower self-BLEU score) while not resulting in higher creativity scores. The model outputs nonsensical responses, e.g., unrecognizable symbols, with a temperature ≥ 1.6.While higher temperatures did increase diversity, as indicated by the fall in the self-BLEU score, this did not correlate with higher creativity. In fact, higher temperatures seemed to reduce the originality and elaboration of the responses in some cases. Table 16 shows our ablation study on the effect of role play and the three-phase discussion framework individually. Table 16 : Ablation study on Three-Phase Discussion Framework and Role-Play. LLM Evaluation results of role-play without three-phase discussion, three-phase discussion without role-play, and LLM Discussion (role play and three-phase discussion) on four benchmarks.Our purpose in generating various roles with diverse backgrounds is to encourage agents to discuss creative tasks from different perspectives. This is also known as "role storming," a widely used technique in the ideation stage of design thinking. This technique allows participants to set aside inherent limitations, helping them to come up with unprecedented ideas. For example, millionaires or billionaires are commonly used roles (Panchenko, 2024) that free people from budget constraints.We generated the list of roles in an automated way using the GPT-4 model. The simplified chat log is shown below, where we integrate similar prompts and filter out unrelated results.The full version can be referred to at https://chatgpt.com/share/f35f049d-b668-4089-a4d2-1153e6696221. This is our initial attempt to demonstrate the effectiveness of this framework, but the roles are not the optimal set. We agree that covering a broader range of demographics is crucial, so we will explore this domain in our future work.What are some roles that are typically used in boosting creative thinking, like different personas to diversitize the ideation stage.When looking to diversify the ideation stage, introducing different personas can indeed offer a range of perspectives that might not be considered otherwise. Here are some personas that can be particularly useful in diversifying thought and enhancing creativity:The Visionary Millionaire: This persona brings the perspective of high financial success combined with a forward-thinking approach. They can offer insights into long-term investments, trends, and big-picture thinking. The Startup Founder: Embodying agility, innovation, and risk-taking, this persona can introduce ideas that disrupt traditional markets or suggest scalable, tech-driven solutions.The Social Entrepreneur: Focused on solving societal issues through business, this persona adds a layer of social impact and ethical consideration to the brainstorming process.The Creative Professional: Artists, writers, designers, and musicians can provide unique insights into aesthetic, narrative, and emotional dimensions that might enhance the appeal of a solution.The Customer/User: Representing the end user's voice, this persona helps ensure that ideas are aligned with user needs, preferences, and experiences.The Environmentalist: This persona brings a sustainability angle to the discussion, encouraging solutions that are eco-friendly and promote long-term environmental health.The Digital Nomad: Offering insights into remote work, digital tools, and global trends, this persona can help identify opportunities in the digital economy and lifestyle shifts.The Academic/Researcher: They can introduce data-driven insights, theoretical frameworks, and evidence-based perspectives to ground creative ideas in solid research.The Industry Insider: Someone deeply entrenched in a specific industry can offer insider knowledge, trend awareness, and practical constraints or opportunities within that field.The Futurist: This persona looks at emerging technologies, societal shifts, and potential future scenarios to inspire ideas that are ahead of their time.Incorporating these personas into the ideation process can help teams explore a wider range of ideas, anticipate different needs, and consider various angles, making for a richer and more innovative outcome.Great, now follow this format and generate a .json file for these roles: Remain "type": "openai", "model name": "gpt-3.5-turbo", the same. and the agent name = "GPT Agent 1 -{agent role}"1 [ 2 { 3" type ": " openai "," model_name ": " gpt -3.5 -turbo "," agent_name ": " GPT Agent 1 -White Hat "," agent_role ": " White Hat "," agent_speciality ": " Information Analysis and Facts ", 8 " agent_role_prompt ": " Focuses on available data and past information , analyzing trends and gaps in knowledge , striving for an objective viewpoint ." " type ": " openai "," model_name ": " gpt -3.5 -turbo ", " agent_name ": " GPT Agent 1 -Visionary Millionaire "," agent_role ": " Visionary Millionaire "," agent_speciality ": " Financial Success and Forward -Thinking ", 8 9" agent_role_prompt ": " This persona brings the perspective of high financial success combined with a forwardthinking approach . " type ": " openai "," model_name ": " gpt -3.5 -turbo "," agent_name ": " GPT Agent 10 -Futurist "," agent_role ": " Futurist ", 80 " agent_speciality ": " Emerging Technologies and Future Scenarios "," agent_role_prompt ": " This persona looks at emerging technologies , societal shifts , and potential future scenarios to inspire ideas that are ahead of their time ."82 } 83 ]This JSON array provides a structured way to represent the diverse personas you want to incorporate into your ideation or creative thinking process. Each persona is detailed with a unique agent name, role, specialty, and a prompt that describes their contribution to the brainstorming or innovation process.This section examines the qualitative dimensions of our discussion. Given the extensive nature of the text, we have included the discussion log in our supplementary materials. The "discussion log.zip" contains two html files: llm discussion chatlog.html which documents the discussion log of one of the agents on the Scientific Creativity task within our LLM Discussion framework, and "llm debate chatlog.html" which does the same within the LLM Debate framework.For LLM Discussion, we can see clear discussion in a collaborative tone, as cited, "Building on those ideas, I believe we can combine elements of creativity, systematic testing, and a multidisciplinary approach..." (Final Response, Point 1). The phrase "Building on those ideas" explicitly encourages collaborative engagement, suggesting that each contribution is valued. Moreover, we could see a comprehensive approach here: " By merging elements of sustainability, creativity, advanced technology, and human engagement...", This shows a direct outcome of engaging multiple perspectives.LLM Debate framework showed a more direct and solution-focused response. For entire conversations throughout, agents tend to only update the answer without any discussion at all. We can see in the html file that the agents use an iterative approach, which doesn't usually lead to an original solution for creativity tasks.You are a Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth. Remember to claim your role in the beginning of each conversation. Initiate a discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present a list of as many creative and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can.As an Environmentalist specializing in Sustainability and Environmental Health, I believe it's important to think outside the box and explore creative uses for everyday items like umbrellas. Let's brainstorm some innovative ideas: 1. Shelter for wildlife: Use umbrellas to create makeshift shelters for small animals during inclement weather. 2. Hanging plant holder: Attach umbrellas upside down from the ceiling and use them to hold potted plants. 3. Umbrella art installation: Use colorful umbrellas to create a visually appealing art installation in a public space. 4. Umbrella chandelier: Hang multiple umbrellas from the ceiling to create a unique and quirky chandelier. 5. Umbrella garden border: Line the perimeter of a garden with open umbrellas to create a whimsical border. 6. Umbrella storage organizer: Hang umbrellas on a wall-mounted rack to store keys, hats, and other small items. 7. Umbrella lampshade: Repurpose an old umbrella as a lampshade for a DIY lighting fixture. 8. Umbrella kite: Attach streamers to the spokes of an open umbrella and fly it like a kite on a windy day. 9. Umbrella runway: Set up a row of open umbrellas as a makeshift runway for remote-controlled cars or toy airplanes. 10. Umbrella door wreath: Decorate an umbrella with flowers, ribbons, or other decorative elements to create a unique door wreath. I hope these ideas spark some creativity and inspire us to think differently about how we can use umbrellas in our daily lives! Let's continue to brainstorm and come up with even more creative uses.You are a Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth. Remember to claim your role in the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'As a Creative Professional specializing in Aesthetics, Narratives, and Emotions, let's brainstorm some creative uses for an umbrella: 1. Turn an umbrella into a unique art installation by painting it with vibrant colors and patterns, then suspending it from the ceiling in a gallery or outdoor space. 2. Use an umbrella as a makeshift photography prop by placing it in the foreground of a shot to add a pop of color or create interesting framing. 3. Transform an umbrella into a whimsical outdoor chandelier by attaching fairy lights to the spokes and hanging it above a patio or garden for a magical ambience. 4. Repurpose an umbrella as a stylish beach accessory by attaching pockets to the inside where you can store sunscreen, sunglasses, and other essentials. 5. Create a DIY hanging plant holder by turning an umbrella upside down, removing the fabric, and securing small pots or planters along the spokes for a unique and space-saving garden display. 6. Use an umbrella as a statement piece in a fashion photoshoot, incorporating it into the model's outfit or as a prop to convey a sense of mystery or drama. 7. Turn an umbrella into a mobile kitchen organizer by hanging utensils, spice jars, and other cooking essentials from the spokes for easy access while cooking outdoors. 8. Repurpose an umbrella as a makeshift tent by attaching fabric panels to the spokes and using it as a shelter for outdoor events or picnics. 9. Create a DIY umbrella canopy for a backyard wedding or outdoor gathering by suspending multiple umbrellas above a seating area to provide shade and add a whimsical touch. 10. Use an umbrella as a sculptural element in a room design by hanging it from the ceiling and incorporating it into a larger art installation or decorative scheme."' One agent solution: "'As a Futurist specializing in Emerging Technologies and Future Scenarios, I believe it is crucial for us to think outside the box and explore unconventional uses for everyday items like the umbrella. Let's brainstorm some creative uses for umbrellas: 1. Solar-powered umbrella: Imagine an umbrella that harnesses solar energy to charge your devices while providing shade on a sunny day. 2. Musical umbrella: A high-tech umbrella that plays music based on the rhythm of raindrops hitting the canopy, creating a soothing and unique listening experience. 3. Virtual reality umbrella: An umbrella that incorporates VR technology to transport you to different virtual environments while sheltering you from the elements. 4. Umbrella drone: A drone that can deploy and retrieve an umbrella for you, ensuring you stay dry without having to carry one around. 5. Umbrella garden: Convert your umbrella into a mini garden by hanging plants from the ribs, creating a vibrant and eco-friendly shelter. 6. Umbrella projector: Turn your umbrella into a portable projector that displays images or videos on the canopy, making rainy days a fun and interactive experience. 7. Umbrella communicator: A smart umbrella equipped with communication features like a built-in speaker and microphone for hands-free calls or messaging during a downpour. 8. Umbrella workout: Incorporate resistance bands into the design of the umbrella, turning it into a convenient fitness accessory for on-the-go workouts. 9. Umbrella art installation: Use umbrellas as a canvas for creating large-scale art installations, where each canopy displays a unique design or message when opened together. 10. Umbrella recycling station: Transform old umbrellas into sustainable products or art pieces through a community-driven recycling initiative, promoting environmental awareness and creativity. Let's continue to push the boundaries of innovation and explore the endless possibilities of umbrellas beyond their traditional function."' One agent solution: "'As a Futurist specializing in emerging technologies and future scenarios, I would like to challenge our team to think creatively about the many potential uses for umbrellas beyond their traditional function of providing shelter from the rain. Let's explore some innovative ideas together: 1. Solar-powered umbrellas that can charge your devices on-the-go. Hang open umbrellas in a row as a creative photo backdrop for events, weddings, or parties, adding a splash of color to photographs. 8. Umbrella toy storage: Attach hooks to the inside of an open umbrella to hang and organize children's toys, keeping play areas neat and tidy. 9. Umbrella craft organizer: Use open umbrellas to store and display craft supplies like ribbons, yarn, and thread, turning them into functional and decorative storage solutions. 10. Umbrella fashion accessory: Incorporate mini umbrellas into outfits as unique fashion accessories or statement pieces, adding a fun and quirky touch to any ensemble. I'm excited to see the creative ideas that our team can come up with and how we can continue to think outside the box when it comes to repurposing umbrellas in unconventional and eco-friendly ways! Let's keep pushing the boundaries of innovation and sustainability.You are a Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth. Remember to claim your role in the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'Here are some more creative uses for umbrellas: 1. Umbrella runway fashion show: Use umbrellas as the runway for a fashion show, creating a unique and memorable presentation for the audience. Use the canopy of an umbrella as a screen for shadow puppet shows, entertaining both children and adults with imaginative storytelling. 5. Umbrella costume party: Encourage guests to get creative and dress up in umbrella-inspired costumes for a lively and colorful themed party. 6. Umbrella ice cream stand: Convert an upsidedown umbrella into a makeshift ice cream stand, suspending cones or cups from the spokes to create a playful and whimsical dessert display. 7. Umbrella photo booth: Set up a photo booth with umbrellas of various designs and colors as backdrops, encouraging guests to strike poses and capture fun memories. 8. Umbrella solar cooker: Utilize the reflective surface of an umbrella to create a makeshift solar cooker for outdoor cooking, demonstrating sustainability and innovation in a practical way. 9. Umbrella poetry session: Host a poetry reading under a canopy of open umbrellas, creating a cozy and intimate atmosphere for sharing written works and creative expressions. 10. Umbrella time capsule: Bury a sealed umbrella in the ground as a time capsule, to be unearthed and opened by future generations, revealing messages, mementos, and cultural snapshots from the present day. Let's continue to explore the creative potential of umbrellas and think outside the box for even more inventive uses¡" One agent solution: "'As a Futurist specializing in Emerging Technologies and Future Scenarios, I am excited to build on the innovative ideas shared by my fellow agents and continue exploring creative uses for umbrellas. Let's brainstorm some more out-of-the-box ideas: 1. Weather-responsive umbrellas: Develop smart umbrellas that change color or patterns based on weather conditions to add a fun and functional element to your outdoor attire. 2. Interactive umbrella game: Create an umbrella equipped with sensors that allow you to play interactive games like pong or trivia with friends by moving the canopy strategically. 3. Umbrella messaging system: Design umbrellas with LED displays on the canopy that can be used to transmit messages or images in crowded spaces or events. 4. Emotional umbrella: Develop umbrellas that change shape or emit calming scents based on the user's mood or stress levels, providing a soothing experience during challenging times. 5. Umbrella workout assistant: Incorporate fitness trackers into the handle of the umbrella to track your steps, distance walked, and calories burned while on-thego, promoting wellness and activity. 6. Astronomical umbrella: Create umbrellas with celestial designs and embedded star projectors that illuminate the night sky on the inside of the canopy, turning a rainy evening into a mesmerizing stargazing experience. 7. Educational umbrella: Design umbrellas with educational features like maps, historical facts, or language lessons printed on the fabric, offering a unique way to learn while staying protected from the elements. 8. Umbrella privacy shield: Develop umbrellas with adjustable privacy screens or blinds that can be activated to create a personal cocoon in crowded or open spaces, ensuring moments of solitude and relaxation. 9. Umbrella time capsule: Encourage people to fill designated compartments in the umbrella with mementos, photos, or messages, turning it into a time capsule that captures memories and experiences over time. 10. Soundscaping umbrella: Integrate speakers and soundscaping technology into the handle of the umbrella to create immersive audio experiences that synchronize with the rhythm of rain or ambient sounds in nature. Let's continue to think creatively and envision the limitless possibilities of umbrellas in shaping our future experiences and interactions with the world around us."' One agent solution: "'1. Umbrella Cocktail Bar: Convert umbrellas into unique cocktail bars where mixologists create drinks inside the canopy for a fun and interactive experience. 2. Umbrella Yoga Studio: Use umbrellas as individual yoga pods in a park or outdoor space, providing shelter and privacy for practitioners during their sessions. 3. Umbrella Time Capsule: Bury old umbrellas in a time capsule for future generations to unearth and learn about the history of fashion and technology. 4. Umbrella Karaoke Booth: Transform umbrellas into private karaoke booths where singers can perform their favorite songs in a quirky and memorable setting. 5. Umbrella Beach Tent: Reinvent umbrellas as collapsible beach tents with integrated sand anchors for a stylish and practical sunshade solution. 6. Umbrella Mapping Project: Equip umbrellas with GPS trackers to create an interactive map of their journeys as they are passed from person to person around the world. 7. Umbrella Language Exchange: Host language exchange events where participants hold umbrellas labeled with different words or phrases to facilitate communication and cultural exchange. 8. Umbrella Pet Shelter: Create miniature shelters for pets using modified umbrellas to provide a cozy and portable retreat for furry companions. 9. Umbrella Fashion Show: Organize a fashion show featuring umbrella-inspired designs made from upcycled materials, showcasing the versatility and creativity of this everyday object. 10. Umbrella Light Display: Install LED lights on umbrellas to create a dazzling light show or interactive art installation for special events or festivals."' What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present a list of as many creative and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you canAs an Environmentalist specializing in Sustainability and Environmental Health, I am thrilled to be part of this dynamic brainstorming session with my fellow agents. Let's dive into even more imaginative and unconventional uses for umbrellas: 1. Umbrella community art project: Collaborate with local artists and residents to paint a series of bright and colorful umbrella murals that can be displayed throughout a town or city to promote creativity and unity. 2. Umbrella outdoor cinema: Transform umbrellas into individual outdoor cinema screens where people can gather in a park or public space to watch movies under the stars, creating a unique and cozy movie-watching experience. 3. Umbrella scavenger hunt: Organize a scavenger hunt where participants follow clues hidden under various umbrellas placed around a neighborhood or city, adding an element of surprise and excitement to the traditional game. 4. Umbrella recycle drive: Set up collection points for old or damaged umbrellas and repurpose the materials into new eco-friendly products such as reusable shopping bags, rain ponchos, or even outdoor furniture, promoting sustainability and circular economy practices. 5. Umbrella storytelling circle: Arrange a storytelling circle in a park or outdoor setting where participants gather under a cluster of open umbrellas to share personal anecdotes, fairy tales, or local legends, fostering a sense of community and connection through shared narratives. 6. Umbrella floating art installation: Create a floating art installation using opened umbrellas attached to inflatable platforms on a lake or pond, showcasing innovative and whimsical designs that interact with the water and surrounding environment. 7. Umbrella language exchange picnic: Host a multilingual picnic where participants use umbrellas labeled with different languages as conversation starters to facilitate cultural exchange and language learning in a relaxed and festive setting. 8. Umbrella weather station: Install smart technology in umbrellas equipped with sensors that provide real-time weather data and forecasts when opened, helping users stay informed and prepared for changing weather conditions while promoting awareness of climate patterns. 9. Umbrella wellness retreat: Arrange a wellness retreat in a natural setting where participants engage in mindfulness practices, yoga sessions, and wellness workshops under open umbrellas, offering a serene and rejuvenating experience surrounded by nature. 10. Umbrella sustainability exhibition: Curate an exhibition showcasing innovative and sustainable designs that repurpose umbrellas as eco-friendly solutions for various challenges, such as shelter, mobility, or art, to raise awareness about the importance of environmental conservation and creativity. I am excited to see the creativity and diversity of ideas that our team can come up with as we push the boundaries of innovation and sustainability in reimagining the uses of umbrellas. Let's continue to think boldly and collectively spark inspiration for a greener and more imaginative future!You are a Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth. Remember to claim your role in the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'Here are some more creative uses for umbrellas: 1.  
 
 Appendix:  


Decision: The paper presents a novel LLM Discussion approach to enhance the creativity of large language models through multi-agent discussions and role-playing. Evaluations across four creativity benchmarks demonstrate improvements in originality and elaboration compared to single-LLM and existing multi-LLM frameworks. While some concerns are raised regarding role selection, potential biases, and the need for deeper qualitative analysis, the overall work is considered creative, rigorous, and likely to stimulate further research. I therefore recommend accepting this paper. There is also a related approach here https://arxiv.org/pdf/2310.00280 that the authors may find interesting.

Reviews: review: paper_summary: paper_summary: This paper proposes an approach to enhancing the creativity of large language models (LLMs) by facilitating multi-agent discussions and incorporating role-playing elements. The key ideas of a three-phase discussion framework and role assignment are well-motivated and explained. The authors conduct evaluations across 4 creativity benchmarks, comparing their proposed method against baselines. The results demonstrate improvements in some metrics of TTCT like originality and elaboration, suggesting that the proposed approach can indeed induce more creative and novel responses from LLMs. Overall, this work presents a relatively convincing approach to an important problem in AI, with findings that are likely to stimulate further research in this area.
strength_weakness: strength_weakness: While the paper presents an interesting approach and promising results, there are a few key concerns that should be addressed before it is ready for publication

1) The role-playing component is a novel contribution, but the paper lacks deeper analysis on the specific roles used, how they were chosen/generated, and whether these roles may introduce unintended biases or perspectives that could skew the results.The role-playing aims to induce diverse perspectives, but the paper does not discuss how the chosen roles represent or account for diversity across different demographics, cultures, domains, etc. A more critical examination of this aspect is needed.

2) The task chosen are relatively old and sentence level so not very clear if this approach would be generalizable for other creative tasks such as design, ideation, creative writing, generating metaphors etc

3) The results are primarily quantitative, but a deeper qualitative analysis examining the generated responses, the nature of conversations/discussions, and specific strengths/weaknesses could provide valuable insights.

4) Theoretical grounding: While the motivations are clear, the paper could benefit from stronger theoretical foundations explaining why multi-agent discussion and role-playing are expected to enhance LLM creativity from a cognitive/psychological perspective.
questions: questions: Why AUT ? There is mounting evidence that GPT4 is good at this so to benchmark you should focus on complex creative tasks
https://arxiv.org/abs/2303.12003
https://news.uark.edu/articles/69688/ai-outperforms-humans-in-standardized-tests-of-creative-potential
paper_summary: paper_summary: The presented approach gets much better results on key measures of creativity compared to SOTA.

This paper is a well-written presentation of the novel LLM Discussion approach to generating creative responses on established creativity and scientific innovation benchmark tests. 

The approach is a highly original simulation of "collaborative discussions with diversified peers" that involves (i) a three-phase, initiation-discussion-convergence framework for multi-turn dialogues designed to elicit diverse and creative responses from (ii) LLM-based agents, each of what are assigned distinct roles and are directed to generate answers building on the others' responses.

This framework, (i), is compared against single-LLM and multi-LLM approaches and outperforms these previous methods on several creativity metrics.

The role-playing LLM-based agents, (ii), cleverly leverages the ability of LLMs to imitate very different personalities in order to generate more diversity of ideas.
strength_weakness: strength_weakness: * The authors do not discuss going beyond the role-playing agents as artificial people, and address other ways of coming up with better answers,.

* The paper does not address what is actually happening inside the model, that would also give deeper insights.
questions: questions: * The space of possible prompts and  ways of combining the results of prompts is so broad that it seems unlikely these results are anywhere near the best that could be obtained with present models. Q1: Why, for instance, are the particular roles of "visionary millionaire, startup founder, social entrepreneur, creative professional, customer, environmentalist, digital nomad, industry insider, futurist" (which sounds like a list of people you might encounter at a party in Silicon Valley) chosen instead of, say, homemaker, mathematician, janitor, and professional actor? Or, say, all 16 Meyers-Briggs personality types? Or people from 20 diverse countries or time periods? There's no justification for this and no way of knowing what would be best a priori. At the moment there is so much low-hanging fruit that it is easy to come up with new approaches that get big gains, but it would be helpful to find some automated way of searching the space of possibilities more thoroughly.

* Q2: What might help explain why fluency and flexibility scores are lower for LLM Discussion than LLM Debate, even though these matter less than originality and elaboration as measures of creativity. Perhaps there would be some simple way to alter the prompts to boost these factors as well?

* Q3: how is the prompt in the convergence phase leading to "convergence" as opposed to simply enumerating answers from prior phase, given instruction to  "present a list"?  Did you experiment with an explicit prompt to create a summary or draw a conclusion?
paper_summary: paper_summary: The paper presents a method called LLM Discussion, which is a multi-turn multi-LLM method to attempt to extract more creative responses out of LLMs on common creativity "brainteasers" such as Alternative Uses. The work experiments on four types of tasks, and evaluates different conversational methods to extract creative answers. The LLM Discussion method is found to be more creative both through a robust human evaluation, and using an LLM as a judge, which is found to correlate satisfactorily with human judgments.
strength_weakness: strength_weakness: - Nothing serious, overall the work is creative and rigorous.
- The four tasks studied are known to be somewhat artificial. As the authors point out, the tests to evaluate creativity in humans might not be adequate to evaluate LLMs, since the latter can be infinitely verbose at very low cost. The authors point this out and attenuate the importance of certain TTCW metrics, yet the tasks themselves remain tasks that were created for humans, not LLMs.
- There is a strong correlation between response length and perceived creativity, both in human & LLM evals. The work did not directly control for length of response (which seems like it would have been very easy to do...) and the examples shown in Figure 4 unfortunately show that LLM Discussion outputs are much longer. Are the promising results therefore only reflecting the underlying signal that responses in that condition are significantly longer on average?
questions: questions: Q1: Can you report the average length of response for each task and each approach? This would help explain to the reader whether length explains away the creativity difference or not. Why did you not include in your prompt an expected response length, to normalize for this important effect?

Q2: Besides the tasks you've collected, whose intended audience are humans under a time constraint, do you have a sense of what tasks could be more appropriate to evaluate LLMs' creativity abilities? Do you believe that the "number of turns" reflects a similar effect to human test-takers having "limited time" in TTCT?

Q3: Two of the core contributions in LLM Discussion seem to be: (1) the three phases, (2) the role-playing. Yet, neither of these two elements seems to have been ablated. Do you have experimental evidence showing that such elements indeed lead to improved creativity in the responses?

Q4: It is not 100% clear in the paper which LLM is used in experimentation. The draft clearly states which GPT model is used for auto-eval, but not for generation. Are you using GPT4? Are the different roles all played by the same underlying LLM, or different LLMs? Does that make a difference?

Q5: It is not 100% clear in the paper how the "Convergence Phase" is conducted. It seems logical that only one LLM would perform the final agglomeration based on the conversation so far. How do you pick which "role" gets to do the final response? (e.g., do you get the final response from the environmentalist or the social entrepreneur).

Q6: Temperature plays a strong role on the diverging nature of an answer. What temperature setting did you set the generation models with, and do you expect increased temperature to further improve creativity scores?

Minor notes:
- It seems a bit of a stretch to call tau correlations at ~0.3 "strong correlation"
- This reviewer appreciated the pun in the title of Section 6, kudos :)
paper_summary: paper_summary: The authors present an LLM discussion framework for improving the creativity of LLM outputs by using multiple LLM agents with diversified role-playing instructions in a 3-step pipeline.
strength_weakness: strength_weakness: While the authors have shown that their method can improve the creativity of LLM outputs, it does appear to come at some cost to fluency. In LM generation one can typically increase diversity trivially at the cost of fluency by e.g. increasing the temperature. It's not immediately obvious to me that the proposed method is not just a fancy way of doing the same tradeoff, rather than "pushing the Pareto boundary" in some sense. While the current experimental results are fine, in my opinion this paper would be substantially stronger if the authors showed results on e.g., something like a creative problem solving task where the additional creativity actually results in something like higher success rate or accuracy, rather than solely improving creativity for its own sake.
questions: questions: - Why does your method often perform worse on fluency compared to some baselines? Is there a tradeoff for fluency vs diversity (similar to how you make this tradeoff when you change the temperature of generation)?
- Why is flexibility not a main metric? Isn't coming up with more different answers a pretty reasonable thing to evaluate? Or is the problem just that you don't have a good way to "deduplicate" different answers? Not sure if I'm misunderstanding what is meant by flexibility in your setup.
- Why are all the metrics 0 when you have 3 agents in fig 6?

Comments: title: Thank You
Thanks for your response
comment: I will keep my scores. I don't think I am fully convinced at 3/4 answers
No change to the score of 7 recommending acceptance.

The new table with the AMAP results ought to be included in the paper.  

At a core level what is happening internally is that the diversity of responses from a single prompt is limited.  For the approach taken in this paper, the setup of pretending to be various types of people is one way to add variety into the prompts, yielding extra originality. The setup of taking on different roles throws the generation into somewhat different distributions of training data, yielding different responses. It may be that even more diversity of responses would emerge if there were  increased diversity of the types of people being simulated or otherwise broadening of the diversity of the prompts.  Perhaps translating into different languages and asking the questions would be one way of getting answers from a very different part of the training distribution.
Thanks for the extra analysis on fluency vs diversity. I'll maintain my score of 7 and recommend acceptance.