LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play
Authors: Li-Chun Lu
Shou-Jen Chen
Tsung-Min Pai
Chan-Hung Yu
Hung-yi Lee
Shao-Hua Sun
Venue: COLM (2024)
Keywords: multi-agent discussion, creativity, role-play,  large language models
TL;DR: We introduce a novel LLM Discussion framework that significantly enhances the creativity of large language models via multi-agent discussion and role-playing, surpassing single-agent and existing multi-agent LLM frameworks in creativity tests.

Abstract: Large language models (LLMs) have shown exceptional proficiency in natural language processing but often fall short of generating creative and original responses to open-ended questions. To enhance LLM creativity, our key insight is to emulate the human process of inducing collective creativity through engaging discussions with participants from diverse backgrounds and perspectives. To this end, we propose LLM Discussion, a three-phase discussion framework that facilitates vigorous and diverging idea exchanges and ensures convergence to creative answers. Moreover, we adopt a role-playing technique by assigning distinct roles to LLMs to combat the homogeneity of LLMs. We evaluate the efficacy of the proposed framework with the Alternative Uses Test, Similarities Test, Instances Test, and Scientific Creativity Test through both LLM evaluation and human study. The results show that our proposed framework outperforms single-LLM approaches and existing multi-LLM frameworks across various creativity metrics. The code is available at https://github.com/lawraa/LLM-Discussion.

Published as a conference paper at COLM 2024
LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play
Li-Chun Lu, Shou-Jen Tiunggii Pai, Chan-Hung Yu, Hung-yi Lee, Shao-Hua Sun Department of Electrical Engineering, National Taiwan University
edn. 207, 11 b0,66092 r 2942147, hungyilee, shaohuas Jentu, edu. tw
Abstract
Large language models (LLMs) have shown exceptional proficiency in nat- ural language processing but often fall short of generating creative and original responses open-ended questions. To enhance LLM creativity, our key insight is to emulate the human process of inducing collective creativity through engaging discussions with participants from diverse backgrounds and perspectives. To this end, we propose LLM Discussion, three-phase dis- cussion framework that facilitates vigorous and diverging idea exchanges adopt role-
and ensures convergence to creative answers. Moreover, playing technique by assigning distinct roles to LLMs to combat the homo- geneity of LLMss. We evaluate the efficacy of the proposed framework with the Alternative Uses Test, Similarities Test, Instances Test, and Scientific Creativity Test through both LLM evaluation and human study. The results show that our proposed framework outperforms single- approaches and existing multi-LLM frameworks across various creativity metrics. The code is available at https:/ http:///gttuuu.oomo httptttt..o//wwwaaaa//aaaaaaaa/a www.oo//wwwwaa--iiiiiiiiiaeaeeee LLM-rrassiLMissssiiiinn
1 Introduction
Large language models (LLMs) have emerged as highly efficient tools in addressing daily challenges across various applications, demonstrating exceptional capabilities natural language processing (Lappuin 2024 Yang et al. 2023b). Specifically, LLMs achieve excellent performance in language comprehension tasks, such as sentiment analysis (Zhang et al., 2023a) and question answering (Qin et al.] 2023). Also, LLMs are widely adopted in content generation tasks, ranging from writing articles to composing poetry, by producing coherent and contextually relevant text that closely mimics human- written content (Qin ett al. 2023 Yuan et al. 2022). Nevertheless, recent studies suggest that LLMs demonstrate
limited creativity in answering open-ended questions and often fail to produce original
responses (ippolito et al. 2022 Chakrabarty et al. 2023 Mohammadi 2024).
Psychological studies suggest that engaging in discussions with participants from diverse backgrounds, perspectives, and experiences can significantly enrich individual creativ- ity (Han et al. 2022: Pauluus & Nijstad 2003: McGrath 1984 Sutton & HHrgadon, 1996
Karwowski & Soszynski 2008). Can LLMs likewise strengthen their creative capabilities through collaborative discussions with diversified peers? While Du et al. (2023); ; al. (003b), et al. (2023) explore collaboration among LLMs, most existing works are limited to amproving the pertormance on close-ended tasks, such as Massive Multitask Language Understanding (Hendrycks et al., 2021), Mathematics (Cobbe et al. 2021), ) and Code Gen- eration (Huang et al.) al. 2222)), leaving discussions on creativity largely under-investigated On the other hand, prior works researching LLM creativity are mostly restricted to single LLM agent (6ómez-Rodríguns
& Williams 2023 Stevenson et al. 2022).
Our goal is to boost the creativity in LLMs by designing discussion framework. Sub- sequently, two main challenges arise due to the nature of LLMs. First, most modern
'Equal contribution Correspondence to: Shao-Hua Sun <shachuas@ntt.eduu.ww

---

Published as a conference paper at COLM 2024
commercialized LLMs are not specifically trained reinforced to engage in multi-turn conversations (Ouyang et al. 2022 Ding et al. 2023), which results in unengaging discus- sions among LLMs that quickly converge. Second, the high homogeneity of LLMs renders difficult for discussions to diverge and produce creative outcomes (Ouyang et al. 2022
Padmakumar & He 2024).
To address these issues, we develop a role-play en- hanced LLM discussion framework, dubbed LLM
Topic: Creative Uses of Forks
Discussion, as illustrated in Figure] 1 To facilitate vig- orous and diverging exchanges throughout rounds of discussion and ensure convergence to creative answers, we arrange three-phase discussion and design prompts specialized in the initiation phase, the discussion phase, and the convergence phase.
Initiation Phase
Discussion Phase
Academic Researcher
Let's start by analyzing it from some of features!
LLM A
Then, to tackle the LLM homogeneity, we adopt the role-playing technique, an established practice in De- pro-
Environmentalist
that metal are reusable and durable.
O LLM
sign Thinking [Camachol 2016). Specifically, pose to assign a diverse set of roles with distinct backgrounds and perspectives to LLM agents par- ticipating in a discussion, such as Visionary Million- aire and Fufurist, and roles specialized for various facets of discussion from Six Thinking Hats method- ology (Setyaningtyas Radia 2019).
Visionary Billionaire
OO LLM C
Agreed. How about using it outdoor installation art for higher value?
Convergence Phase
To evaluate our proposed framework, we incorpo- rate four creativity benchmarks, covering broad spectrum of creative thought processes: Alternative Uses Test, Instances Test, and Similarities Test from the Mallach-Kogan Creativity Test (WKCT) (Wallach, M. A., and Kogan, N. 1965), and Scientific Creativity Test (Hu & Adey 2002). We conduct LLM evaluation and human study on four widely used metrics: Original- ihy, Elaboration, Fluency, and Flexibility [Torrance] 1966). The results demonstrate that our proposed framework outperforms baselines in producing creative answers.
Figure 1: Role-Play Enhanced LLM Discussion.
2 Related Work
Multi-LLM Collaboration. Prior works have developed multi-LLM frameworks to improve
the factuality, reasoning, arithmetic skills, and coding abilities of LLMs (Du et al. 2023
Liang et al. 2023 Chan et al. 2024 Wu et al. 2023 Li et al. 2023c Liu et al. 20235 Ning
et al. 2024 Jiang et al. 2023 Sun et al. 2023), mostly evaluated on MMLU (Hendrycks
et al. 2021), GSM8K (Cobbe et al. 2002),, and HumanEval (Chen et al. 2021 In contrast,
our work aims explore enhancing LLM creativity by designing a mu multi- LL discussion framework.
LLM Creativity. Recently, many aspects of LLM creativity (Chakrabarty al. 2023 Ippolito
et al. 2022) have been studied, such as English creative writing (Gömez-Rodriguee &
Williams 2023),, , assessing metaphors (Distefano et 2024), and coming up with alternative
uses of objects (Stevenson et al. 2022). These studies are mainly limited to single-LLM setups; instead, this work aims employ multiple LIMSs to induce collective creativity.
LLM Role-Playing. LLMs have shown significant potential for role-playing (Wang et al.)
2023c
Park et al. 2023 Li et al. 2023a Wei et al. 2023 Shanahan et al. 2023 Salemi et al.
2023).
The ability to impersonate distinct roles has proven to induce human-like behav-
1or (Pak et al. |023 Wang et al. 2023b Li et al. 2023a Shao et al. 2023), interactivity (Wang
ee al.
2023c
Cui et al. (2023), and the ability to tackle complex tasks of LLMs (Cui et al.
2023
Li et al. 2023a Wang et al. 2023c). Our work aims to hamness LLM's ability to role-play
to enhance
L.M creativity.
2

---

Published as a conference paper at COLM 2024
3 Approach: LLM Discussion
Our key insight is to boost creativity in LLMs by designing an effective discussion framework and employing role-play techniques. To this end, we propose LLM Discussion, three-phase discussion framework that incorporates role-playing, allowing multiple LLM agents to build on each other's s thoughts and collectively produce creative outcomes.
Specifically, to combat the inability of LLMs to conduct engaging multi-turn discussions, we propose to arrange three different phases, the initiation, discussion, and convergence phases, as described in Section 3.1 In each phase, a short-term objective is provided to each LLM with specialized prompt, e.g., actively discuss or efficiently converge, facilitating both divergent and convergent thinking. Then, to alleviate the LLM homogeneity issue, i.e, all LLMs think alike, Section 3.2 presents our proposed role-play mechanism, which assigns diverse backgrounds, perspectives, and personalities to LLMs.
3.1 Discussion Framework
We aim to design discussion framework that hosts rounds of dynamic exchanges of ideas, progressively fostering novel solutions and deeper insights. However, most modern commercialized LLMs are not specifically trained reinforced to engage in multi-turn conversations (Ouyang et al. 2022 Ding et al. 2023), resulting in unengaging discussions among LLMs that quickly converge. To facilitate meaningful interactions among LLMs, Du et al. (2023) introduce LLM Debate, which requires multiple LLMs to propose and debate their individual responses with other LLMs over multiple rounds to arrive at joint answer agreed upon by all the LLMs.
Yet, LLM Debate is devised to improve LLM's performance in answering closed-ended questions, such as question answering (Hendrycks et al.) 2021) and mathematical reason- ing (Cobbe al. 20011. In particular, the mechanism employed in LLM Debate requires each LLM to maintain its own answer while observing answers produced by other LLMs, which is effective resolving factual questions but can fall short of promoting divergent thinking building upon others' responses.
To encourage LLMs to discuss with others and inspire others actively to engender collective creativity, we devise three-phase discussion framework that explicitly requires each LLM to build upon others' responses. We arrange an initiation phase, a discussion phase, and convergence phase, which are described in detail below. In each phase, each LLM aims to fulfill an objective, egg,, actively discuss or efficiently converge, as instructed by a specialized prompt. The proposed discussion framework is illustrated in Figure
Initiation Phase. This phase serves as the groundwork for the discussion, where each LLM is introduced to the topic and the objectives of the dialogue. In this phase, aim to explicitly inform the LLMs about the situation and declare the goal. Specifically, we notify each LLM that it is in group discussion setup by stating "you are in a group discussion with other teammates" in the prompt. Moreover, we stress the importance of teamwork spirit with the instruction "Initiate discussion with others to collectively complete the following task." Then, we provide a description of the task that we want the LLMs to solve, egg., "come up with creative uses for forks." "
Discussion Phase. The goal of this phase is to induce multiple rounds of discussions carrying meaningful idea exchanges among LLMs. In each discussion round, each LLM receives responses from all other agents in the previous round, resembling the idea of active listening in human discussions. Additionally, we aim to encourage each L.L. to compose and diverge its answers. For each LLM to balance between generating its own answers and following up on others' thoughts, we experiment with various prompts as discussed in Section 4.5
Convergence Phase. After rounds of diverging and creative discussions, the convergence phase aims to summarize the ideas brought in discussions and converge to collective conclusion. To this end, we inform each LLM by prompting it with the instruction, "This is the last round of the discussion; please finalize and present list of creative answers." Note
3

---

Published as a conference paper at COLM 2024
Initiation Phase
1" Round
LLM Agents
Initiate a discussion with others to collectively complete the following task: (Tesk) leammiates;
are group you can.
diversely creatively
Discussion Phase
2d (N-1)" Rounds
These the solutions problem from other agents:
One agent solution" 1)" + +... + One agent solution" (ssl.. N (Task)
You group discussion with teammates; result, diversely and creatively
Repeal N-2 Rounds
aB you can.
Convergence Phase
N Round
the solutions problem from other agents:
One agent solution" "(soo""""([s ((sol. 1)"" ... + One agent solution" (sol. NY (Task)
You group discussion as you can. This is the last round of the diacussion; please finalize and present list of creative
teammates; result answer diversaly creatively
Final Answers
Figure 2: Discussion Framework. We propose LLM discussion framework that induces collective creativity by bringing divergent and convergent thinking together. The initiation phase informs LLMs of the discussion setup and the task. The discussion phase allows LLMs to build on the ideas of others as as diverge and generate their own answers. The convergence phase summarizes the discussed ideas and draws a collective conclusion.
that the term "finalize" in our convergence prompt serves as an explicit prompt to conclude, while "present list" facilitates parsing.
Full Discussion Procedure. Putting everything together, our proposed discussion procedure with N rounds described as follows.
1st Round: In the first discussion round, we initiate the discussion by prompting LLMs using the prompt described in the initiation phase.
2nd to (N-1)) 11-1thh Rounds: To encourage multiple rounds of discussions, we repeti- tively prompt each LLM and provide it with responses from other LLMs using the prompt presented in the discussion phase.
Nth Round:
The goal of the last discussion round is to summarize the ideas and converge to collective conclusion. Hence, the LLMs are prompted with the convergence phase prompt. Note that we also provide the context by first prompting LLMs with the discussion phase prompt.
five-round discussion qualitative result is shown in Section
3.2 Role-Play
We empirically observe that even with engaging discussions, LLMs often fall short of producing diverging answers given the same prompts due to their homogeneity, which aligns with recent findings reported by Ouyang et al. (002)); P'admakumar & He (2024). To address this issue, we propose employ role-play, technique commonly used in the ideation stage of Design Thinking (Camacho, 2016). This technique aids people in coming up with innovative ideas for various purposes, unlike problem-orientee roles, which focus on solving specific issues within distinct fields.
Role Generation. To generate set of roles with diverse backgrounds, perspectives, and experiences and detailed descriptions of these roles, we propose an automated pipeline that uses GPT-4 (OpenAl et al. 2023) to produce a list of (role, speciality, role prompt), egg,, (Visionary Millionaire, Financial success and forward-thinking As a Visionary Millionaire, your mission is to leverage your financial insight and forward-thinkee approach to inspire groundbreaking ideas ...). The detail process and generation prompts are presented in Section J] We also adopt Six Thinking Hats methodology (Setyaningtyas Radia 2019) to give LLMS different perspectives, e.g., the red hat represents emotions, and the green hat stands for innovation.
4

---

Published as a conference paper at COLM 2024
The entire role set can be found in Section B Our work makes an initial attempt to automati- cally generate diverse and detailed roles for discussion, and systematically exploring other role-generation mechanisms is left for future work.
Role-Play Enhanced Discussion. To incorporate the roles into our discussion framework, we assign dif- ferent roles to different LLMs before the initiation phase. Specifically, we adopt an autoregressive way
LLM agent
LLM agent
LLM egent C
to select the roles for each member in a discussion. Then, the beginning of each discussion round, we
Role Assignment
remind LLMs of their roles and responsibilities and ask them to declare their roles. This ensures that other members of the discussion are aware of each speaker's 's identity, enabling a more structured and coherent exchange of ideas. Note that the roles of LLMs stay unchanged throughout the entire discus- sion. An illustration of role assignment and examples of roles are shown in Figure
OO
O
Visionary Billionare
Academic Reseercher
Environmentalist
Speciality: Finencial Success and Forward-Thinking Role Prompt: Visionary Millionaire, your mission to leverage your financial insight and forward-lhinking approach to inspire groundbreaking ideas. Your wealth experience in recogniingg and investing long-term trends will guide us toward innovative solutions that are not only creative but also financially viable.
4 Experiment
Figure 3: Role Assignment and Ex- ample. At the beginning of a dis- cussion, each LLM assigned role with specialties and a detailed descrip- tion of the role, i.e., role prompt.
4.1 Benchmarks
Creativity encompasses several distinct aspects, in- cluding divergent thinking, connection-making, and practical innovation. To measure LLM creativity, utilize four existing renowned bench- marks for human creativity in our evaluation, each responsible for assessing distinct aspects of creativity. We adopt three benchmarks from the widely-used
Kallach-Koogn Creativity Tests (Wallach, M. A., and Kogan, N. 1965): Alternative Uses Task (AUT), Instances Test (INSTANCES), and Similarities lest (SIMILARITIES). We also include the Scientific Creativity Test (Hu & Adey 2002) (SCIENTIFIC), which measures scientific creativity and scientific process skills.
AUT assesses the ability to develop practical and innovative use cases for objects like a fork or a shoe. INSTANCES requires participants to list things following specific criteria e-g., things that move on wheels or things that are square. SIMILARITIIS evaluates creativity in making connections, challenging participants to brainstorm and link ideas through commonalities, egg,, how a book and a magazine are alike. Lastly, SCIENTIFIC evaluates creativity in scientific contexts and the ability to devise practical and creative solutions to scientific problems. Some sample tasks of these benchmarks are presented in Table 1
Task augmentation. Due to the limited public availability of the tasks from these bench- marks, we generate additional tasks using GPT-4 (OpenAl et al. 2023). Specifically, for each benchmark, we provide GPT-4 with publicly available task samples and ask it to generate 30 additional tasks (Liu et al. 2023a; Ubani et al., 2023 Fang et al. 2023 Peng et al. 2023). Then, we carefully review the generated tasks to ensure their relevance and alignment with the benchmarks. We aggregate the original and generated tasks to form each benchmark for evaluation. We will release the full task sets to facilitate future research studying LLM creativity.
4.2 Baseline
We compare our proposed framework, LLM Discussion, with the following methods.
Single Agent involves querying a single LLM agent to solve the task on its own. We experiment with various prompting techniques, such as few-shot prompting (Brown) et al. 2020), zero-shot-CoT (Kojima et al. 2022), LLM Stimuli (Li al. 2023b), and the fake-a-deep-breati prompt (Yang et al.) 2023a), and the results are shown in Table 5 Since no particular prompt consistently outperforms others, we select the zero-shot prompt, i.e, with only task description, as our main baseline.
5

---

Published as a conference paper at COLM 2024
BENCHMARK
DESCRIPTION
SAMPLE TASK
AUT
This benchmark requires participants to come up with as many creative uses pus- sible for simple object.
What some creative uses for fork?
INSTANCES
This benchmark requires participants to list as many creative answers as possible that a specific criterion.
Name all the round things can think of.
SIMILARITIES
This benchmark asks participants to cre- atively explore similaritics between two
Tell me all the ways which a book and a magazine are alike.
things.
SCIENTIFIC
This benchmark asks participants to gener- ate innovative ideas, envision hypothetical scenarios, propose improvements, and de- sign experiments, all scientifically.
Please think of as many possible improve- ments as you
regular bicycle, mak- ing it more interesting, more useful, and more beautiful.
Table 1: Benchmark Descriptions and Sample Tasks. Alternative Uses Task (AUT), In- stances Test (INSTANCES), and Similarities Test (SIMILARITTES) are designed to assess creative thinking in everyday contexts, while the Scientific Creativity Test (SCIENTIFIC) evaluates scientific creativity and scientific process skills.
Brainstorm, then Select (BTS) is specifically designed for improved originality and utility on AUT proposed by Summers-Star et al. (2023).
It includes a selection phase where potential solutions are judged against specific criteria identify the most original responses. Since is designed for A AUT, we only present the AUT results of this method. Also, BTS is a single-LLM framework, unlike our proposed framework, which encourages iterative enhancements through multi-round discussions and dynamically evolving the brainstorming process rather than filtering static responses.
LLM Debate (Du al. 2023) is multi-LLM framework aiming improve factual- ity and reasoning ability of LLMs by requiring LLMs to debate their responses and reasoning processes over multiple rounds. Specifically, the LLM Debate asks each LLM to examine responses from other L.MSs and verify the correctness and reason- ableness instead of following up on the ideas brought up by others. contrast, our LLM Discussion framework focuses on developing collaborative discussions with LLMs assigned diverse roles, allowing for the emergence of collective creativity.
More details of these methods can be found in Section A
4.3 Evaluation
We use LLMs and humans to quantitatively evaluate the creativity of generated responses.
4.3.1 Metrics
We employ the metrics established by the Torrance Tests of Creative Thinking (TTCT) (Torrance) 1966), recognized for their robustness in measuring creative capabilities and are commonly applied in assessing human creativity (Cramond et al., 2005). The TTCT framework comprises four distinct metrics:
Originality: Considering novelty responses, not familiar and unusual, but relevant.
Elaboration: The amount of details used to extend response.
Fluency: The number of relevant responses.
Flexibility; A variety of categories or shifts in responses.
Originality and Elaboration provide qualitative measures crucial for evaluating the depth and novelty of responses, which reflects the creative potential of LLMs in a 5-point Likert scale. On the other hand, Fluency and Flexibility are measured cunulatively, aggregating the amount of relevant answers categories; yet, unlike humans, LLMSs are inherently

---

Published as a conference paper at COLM 2024
capable of generating large volumes of responses (OpenAl al. 2023), making Fluency and Flexibility less indicative of creativity for LLMs. Therefore, Originality and Elaboration are more informative and can more accurately reflect LLM creativity, while Fluency and Flexibility results are still presented to ensure adherence to the TTCT standards. Section G provides more detailed explanations.
4.3.2 LLM Evaluation
LLMs have demonstrated the ability to resemble human judgment and reasonably evaluate contents, achieving results comparable to human (Chan et al. 2024 Wang et al. 2023a
Adlakha et al., 2023 [ocmi & Federmann, 2023). As result, we adopt to evaluate
the creativity of generated responses. Specifically, we use ChatGPT (Wang et al. 2023a) for the LLM evaluation.
In our LLM evaluation, responses are assessed individually on Originality and Elaboration and collectively for Fluency and Flexibility. That said, the response to "Creative uses of a fork" might be [Eating Pasta, Eating Pancake, Art projects, Drilling]. For Originality and Elaboration, each response receives an individual score, for example, [2, 2, 3, and [2, 2, 2, 1J, respectively. For Fluency and Flexibility, the entire list of responses is collectively evaluated three times, receiving an average score, for instance, 3.33 and 2.33, respectively.
4.3.3 Human Evaluation
We also conduct human evaluation on the responses generated by our method and the baselines. Human evaluation is conducted using the same metrics as those used in LLM evaluation. To ensure consistency across both setups, the same rubric used in the LLM evaluations is also presented to human evaluators, shown in Section responses from 42 distinct annotators are collected online for evaluating the creativity of generated responses. Additionally, we investigate the correlation between human and LLM evaluations in Section 4.4.2
C In total, 1,406
4.4 Results
Our experiments and evaluation both utilize the gpt-3.5-turbo-0255 model.
4.4.1
LLM Evaluation Results
We present the LLM evaluation results across four benchmarks in Table 2 Our proposed framework, LLM Discussion, outperforms the baselines (Single Agent, Brainstrom, then select, and LLM Debate) in Originality on four benchmarks and Elaboration on three benchmarks. In the following, provided in Section K from three perspectives.
discuss the qualitative results observed in the chat logs
Collaborative Dynamics. Our three-phase discussion framework stimulates more col- laborative tone; an example sentence is "Building on those ideas, I believe we can combine..." " This results in more distinctive responses generated in each round, reducing the LLM homo- geneity issue. In contrast, LLM Debate tends to be more discriminative, merely collecting or correcting previous responses without following up on ideas brought up by others.
Role-Specific Responses. Our proposed role-play technique encourages responses specif- ically aligned with the assigned roles, diversifying the creativity from the perspectives of various fields rather than being limited to general purposes. For example, in an AUT scenario an umbrella, the role, Futurist, proposes integrating with VR technology, while the role, Enteivonmentalitt, suggests using it as a shelter for wildlife. In contrast, Single Agent and LLM Debate tend to suggest more general uses, such as re-purposing the umbrella as a tray quirky headpiece for costume.
Conceptual Complexity and Attention to Detail. Our proposed LLM Discussion achieves superior performances in Elaboration by demonstrating expertise in developing more complex concepts and extending them with details. For example, when asked "Name all the things you think of that used in culture" from INSTANCES, Single Agent and LLM
7

---

Published as a conference paper at COLM 2024
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
BENCHMARK METHOD
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Single Agent
3.47
0.38
3.08
0.39
8.99
1.10
8.82
1.49
Brainstorm, then Select
3,84
0.61
3.32
0.65
4.63
1.43
4,60
1.63
AUT
LLM Debute
3.73
0.47
3.78
0.47
10.47
2.96
9.63
2.73
LLM Discussion
4.44
0.30
9.19
2.25
9.68
Single Agent
2.46
0.33
0.29
14.32
4.52
5.82
3.11
INSTANCES
LLM Debate
2.61
0.32
1.90
0.29
26.21
6.73
11.28
8.72
LLM Discussion
3.65
0.34
2.20
0.58
16.88
10.04
11.11
5.26
Single Agent
2.66
0.39
1,99
0,31
7.00
1.76
6.49
1.61
SIMILARITIES
LLM Debate
2.81
0.21
2.61
0.35
9.65
2.34
8.32
2.24
LLM Discussion
3.29
0,30
2.52
0.54
7.27
2.13
8.14
2.04
Single Agent
3.18
0.38
2.77
0.51
6.37
2.35
6.06
2.31
SCIENTIFIC
LLM Debate
3.52
0.38
3.45
0,47
6.91
3.35
6.75
2.28
LLM Discussion
3.95
0.25
3.47
0.55
5.58
2.61
5.91
2.39
Table 2: LLM Evaluation Results. The Originality and Elaboration scores from the LLM Discussion nearly surpass all the baselines with more than one standard deviation across four benchmarks: AUT, INSTANCES, SIMILARITIES, and SCIENTIFIC, The highest Originality and Elaboration scores in each benchmark are highlighted in bold.
Debate generate responses such Art, Music, and Clothing with no further elaboration. Conversely, LLM Discussion generates concrete yet detailed concepts such as Tattoos, Digital art, and Ethical fashion, each accompanied by detailed explanation.
The qualitative results from the AUT and SCIENTIFIC benchmarks presented in Figure 4 show that our LLM Discussion received higher Originality scores by producing more creative responses compared to other frameworks. The impact of temperature on creativity
score is further discussed in Section
Alternative Uses Test (AUT)
Scientific Creativity
Problem
Problem
creative use for "Towel"
How can you test which of two smartphones has a longer battery Explain your testing approach, necessary tools, procedure.
LLM Discussion (ours)
Originality score: (
Towel Storybook Quilt: Collaborate with communities
together lowels into storybook-themet quilt, with each panel telling different tale memory, fostering creativity, storytellngg and community bonding.
LLM Discussion (ours)
Originality score: S
Integrating AR Technology: Develop AR application that overlays real-time battery usage dala onto smartphones, enabling participants visually compare battery life while engag- ing in everyday tasks, creating an immersive and interactive testing experience.
LLM Debate
cozy
long car
Single Agent
LLM Debate
3
Use towel picnic blanket in the park.
Repeat the results are consistent and reliable.
multiple
varying
Transform a towel into a superhero cape for a child's dress-up play.
Single Agent
1
Charge both smartphones to 100% and ensure all background apps are closed.
Figure 4: Qualitative Results. We present responses generated by LLM Discussion and other baselines two benchmarks, AUT and SCIENTIFiC, along with the Originality scores from LLM evaluation. demonstrates that ideas generated through LLM Discussion are more innovative and provide greater detail.
4.4.2 Human Evaluation and Correlation
Human evaluation results presented in Table 3 show that our proposed LLM discussion outperforms the baselines in Originality and Elaboration across three out four benchmarks, aligning with the LLM evaluations.
We investigate the correlation between the LLM evaluation and human evaluation by calculating Kendall's correlation coefficient between LLM and human evaluations, and each human evaluator against the average ratings of the others human evaluators, as shown in Table
8

---

Published as a conference paper at COLM 2024
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
BENCHMARK
METHOD
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Single Agent
2.50
1.24
1.93
0.98
8.13
2.01
4.33
1.95
AUT
L.M Debate
3.80
1.17
2.50
0.96
7.25
2.99
4.38
2.45
LLM Discussion
3.18
0.94
7.88
5.19
Single Agent
1.67
0.85
1.48
0.91
8.55
1.99
4.50
2.36
INSTANCES
LLM Debate
1.25
0.66
1.50
0.71
7:20
1.60
4.90
1.45
LLM Discussion
3.46
1.00
3.18
1.23
7.25
2.57
5.20
2.66
Single Agent
1.97
1.05
2.10
0.90
4.42
0.64
3.33
0.85
SIMILARITIES
LLM Debate
1.90
1.30
2.38
0.48
8.50
0.76
4.50
2.06
LLM Discussion
2.96
1.02
3.54
1.13
5.25
2.09
3.42
2.10
Single Agent
2.20
2.88
1.05
5.04
2.53
3.86
2.13
SCIENTIFIC
LLM Debate
3.00
1.15
4.00
1.00
5,00
1.10
4.40
2.01
LLM Discussion
2.95
1.41
3.70
1.10
3.57
0.73
3,54
1.48
Table 3: Human Evaluation Results. Our proposed framework, LLM Discussion, outper- forms the baselines in Originality and Elaboration on three out of four benchmarks.
KENDALL'ST
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
L.M Human Average
0.5213
0.2753
0.6017
0.5508
Human - Human Average
0.5094
0.4753
0.3692
0.2071
Table 4: Correlation of Evaluation Results. Kendall's T correlation coefficient reveals a strong correlation between LLM evaluations and the average human evaluations in Originality, Fluency, and Flexibility, with moderate correlation observed in Elaboration. Additionally, we present the correlation between individual human evaluations and the average of others, which exhibits high variance.
Note that |r| E (.0,, (0.2,0.3,, (0.1,0.2), and (0,0.),, are considered a strong, moderate, weak, and very weak correlation, respectively (Botsch, 2011 Chiang & yi Lee 2023 Zhang et al. 2023b). The results show a strong correlation between LLM and the average human evaluation, except for Elaboration, where only moderate correlation is observed.
In particular, for the SIMILARITIES, the Elaboration scores from human evaluations exceed those from LLM evaluations. Consider the response to the prompt, "Tell one way in which a donkey and a horse are alike", where the answer provided was: "Interstelaar Companions: Looking towards interstellar exploration, these animals might pioneer new frontiers in space travel, accompanying humans on missions and serving as invaluable companions and assistants in uncharted realms of the cosmos.. "his response was scored point on Elaboration by LLM evaluators but received an average of 3.88 from ten human evaluators. The potential reason is that human evaluators tend to favor lengthier responses, as shown in Section E whereas LLM evaluations focuses on whether the response addresses the question and prioritize the development of ideas over mere verbosity. We also analyze the average length of the answers and their impacts on each benchmarks for Elaboration and Originality in Section F
Interestingly, the correlation coefficients of LLM to the average human scores were higher than those among human evaluators, suggesting that human scores have higher variance. This indicates that LLM evaluations are more correlated with the average human score than individual human evaluations with the average human score.
4.5 Ablation Study
We conduct ablation studies in AUT different prompts, the number of discussion rounds, and the number of LLM agents involved in discussions to determine the settings for the main experiments. Also, verify the effectiveness of role-play and 3-phase discussion framework respectively.
Role-Play and 3-Phase Discussion Framework. We study the effect of role-play and discussion prompts individually. Section[]show! the results three different conditions of multi-LLMM frameworks on each task: role-play without 3-phase discussion prompts,

---

Published as a conference paper at COLM 2024
3-phase discussion prompts without role-play, and both 3-phase discussion and role-play prompts (LLM Discussion).
Solely role-play 3-phase discussion received better scores in Originality and Elaboration compared single agent and LLM Debate. LLM Discussion, which features both discussion framework and role-play, has the best performance among the three, outperforming the other two across four out of eight Originality or Elaboration of each benchmark.
Prompt Design. We design various specialized prompts for our proposed three-phase discussion framework, aiming to explore how the prompts affect creativity in discussion. We evaluate the prompt designs and present the results in Section D which determines the prompt as the main prompt for further experiments.
Rounds of Discussion. We examine the effect of the number of discussion rounds and present the results in Figure 5) We can observe that Originality increases but Elaboration decreases when increasing the discussion rounds. As result, we settle for five discussion rounds for further experiments.
Number of Agents. We evaluate the influence of the number of LLM agents participating With five-round discussions, involving
in discussions and present the results in Figure 6 four LLMs achieves the best overall performance. Therefore, we use four LLMs with five discussion rounds for further experiments.
2.78
Originalty Score
3.11
2,71
3.39
2.38
3.05
2.64
Originaity
3.35
2.28
2.56
3.31
Meen Originaiity
Mean Originaiity
2.91
Mean El Elboration
2.49
3:27
Moan Eibbrration
2.17
Discussion
Agents
Figure 5: Rounds of Discussion. The perfor- mance does not consistently improve with more than rounds.
Figure 6: Number of Agents. A discussion with 4 agents yields the overall best perfor- mance.
LLM Discussion
We propose LLM Discussion, which enhances LLM creativity by emulating the human process of discussion among LLMs. The proposed framework facilitates multiple rounds of engaging discussions by employing three phases with specialized prompts, e.g., actively discussing or efficiently converging. Moreover, to alleviate the LLM homogeneity issue, i.e, all LLMs think alike, we propose a role-play mechanism, which assigns diverse backgrounds, perspectives, and personalities to LLMs. Through our comprehensive LLM evaluation and human study, our proposed framework outperforms single-LLM methods and existing multi-LLM frameworks in Originality and Elaboration across four benchmarks.
Despite the encouraging findings presented in this work, few aspects can be investigated in the future to deepen the understanding of LLM creativity. First, while the LLM evaluation and human study exhibit correlations, it is often challenging to quantify creativity. Hence, more comprehensive evaluations of creativity, such as interviewing experts in creativity, can provide insightful findings. Second, our proposed framework is designed to enhance LLM creativity by employing group of LLMs discussing with each other. We believe exploring collective creativity induced by human-LLM interaction is another exciting direction.
Acknowledgements
Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry Education, Taiwan.
10

---

Published as a conference paper at COLM 2024
References
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy, Evaluating correctness and faithfulness of instruction-folloiing models for question answering. urXiv:2307.16877 2023.
Botsch. Scopes and methods of political science, 2011.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. 2020.
Maria Camacho. David kelley: From design to design thinking at stanford and ideo. She Ji: The Journal of Design, Economics, and Innovation, 2016.
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien- Sheng Wu. Art or artifice? large language models and the false promise of creativity. ar arXiv:2309.14556, 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, and Harrison Edwards et al. Evaluating large language models trained on code. arXivv2107.033: 003374, 2021.
Jia Chi. The evolutionary impact of artificial intelligence on contemporary artistic practices. Communications in Humanities Research, 2024.
Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evaluations? In Annual Meeting of the Association for Computational Linguistics, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXio:2110.14168, 2021.
Bonnie Cramond, Juanita Matthews-Morgan, Deborah Bandalos, and Li Zuo. A report on the 40year follow-up of the torrance tests of creative thinking: Alive and well in the new millennium. Gifted Child Quarterly GIFTED CHILD QUART, 2005.
Christopher Cui, Xiangyu Peng, and Mark O. Riedl. Thespian: Multi-character text role- playing game agents. arXiv:2308.018 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Empirical Methods in Natural Language Processing, 2023.
Paul Distefano, John Patterson, and Roger Beaty. Automatic scoring of metaphor creativity with large language models. Creativity Research Journal, 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. ar axXi::2305.13325, 2023.
Luyang Fang, Gyeong-Gevt Geonn Lee, and Xiaoming Zhai. Using gpt-4 to augment unbalanced data for automatic scoring. arXio:2310.183665 2023.
11

---

Published as a conference paper at COLM 2024
Carlos (6mez-Rodriguee and Paul Williams. A confederacy of models: comprehensive evaluation of LLMs on creative writing. In Findings of the Association for Computational Linguistics: EMNLP, 2023.
Erik E. Guzik, Christian Byrge, and Christian Gilde. The originality of machines: Ai takes the torrance test. Journal of Creativity, 2023.
Eran Hadas and Arnon Hershkovitz. Using large language models to evaluate alternative uses task flexibility score. Thinking Skills and Creativity, 2024.
Ji Han, Dongmyung Park, min Hua, and Peter R. N. Childs. Is group work beneficial for producing creative designs in stem design education? International Journal of Technology and Design Education, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In International Conference on Learning Representations, 2021.
Weiping Hu and Philip Adey. A scientific creativity test for secondary school students. International Journal of Science Education, 2002.
Dong Huang, Qi Bu, Jie M. Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi- agent-based code generation with iterative testing and optimisation. arXit:2312.130110 2023.
Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. Creative writing with an ai-powered writing assistant: Peersecttiess from professional writers. arXiv:2211.05030, 2022.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Annual Meeting of the Association for Computational Linguistics, 2023.
Maciej Karwowski and Marcin Soszynski. How develop creative imagination?: Assump- tions, aims and effectiveness of role play training in creativity (rptc). Thinking Skills and Creativity, 2008
Tom Kocmi and Christian Federmann. Large language models are state-of-the-ar evalu- ators of translation quality. In Annual Conference of the European Association for Machine Translation, 2023.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Process- ing Systems, 2022.
Shalom Lappuin. Assessing the strengths and weaknesses of large language models. Journal of Logic, Language and Information, 2024.
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi Mi, Yaying Fei, Xi- aoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. Chatharuhi: Reviving anime character in reality via large language model. axXiv:2308.09597, 2023a.
Cheng Li, Jindong Wang, Kaijie Zhu, Yixuan Zhang, Wenxin Hou, Jianxun Lian, and Xingxu Xie. Large language models understand and can be enhanced by emotional stimuli. 2023b.
G. Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Neural Information Processing Systems, 2023c.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXio:2305.19918, 2023.
12

---

Published as a conference paper at COLM 2024
Bingbin Liu, Sébastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. Tinygsm: achieving (80% on gsmåk with small language models. arXio:2312.09241, 2023a.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network:
An llm-agent collaboration framework with agent team optimization. arXie:2310.021100 2023b.
Joseph Edward McGrath. Groups: Interaction and performance. 1984.
Behnam Mohammadi. Creativity has left the chat: The price of debiasing language models. arXiv:2406.05587, 2024.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton- of-thought: Large language models can do parallel decoding, In International Conference on Learning Representations, 2024.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, and Janko Altenschmidt et al. Gpt-4 technical report. 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Neural Information Processing Systems, 2022.
Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In International Conference Learning Representations, 2024.
Masha Panchenko. 10 ideation techniques to unleash your creativity, Mar 2024. URL https Seeee--deaaiion---ee wwwpooottteccciiineseeeottttoo
7/www. www.weleken.co www.eeeeeen.ccbbooggggooooo. htt.aeeen.ccolllgggoooos//////ee
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. ACM Symposium on User Interface Software and Technology, 2023.
Paul B Paulus and Bernard A Nijstad. Group creativity: Innovation through collaboration.
Oxford University Press, 2003.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv:2304.03277, 2023.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatGPT general-purpose natural language processing task solver? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization. arXiv:2304.1 11406, 2023.
Funice Setyaningtyas and Elvira Radia. Six thinking hats method for developing critical thinking skills. Journal of Educational Science and Technology, 2019.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, 2023.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-lIm: A trainable agent for role-playing. 2023.
Claire E. Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, and Han J. van der Maas. Putting gpt-3's 3ss creativity to the (alternative uses) test. ar 7xix:2206.08932, 2022.
Douglas Summers-Stay, Clare R. Voss, and Stephanie M. Lukin. Brainstorm, then select: a generattiee language model improves its creativity score. In The AAAI-23 Workshop on Creative AI Across Modalities, 2023.
13

---

Published as a conference paper at COLM 2024
Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries complex reasoning through multi-model collaboration. (#Xi0:2310.00280, 2023.
Robert I Sutton and Andrew Hargadon. Brainstorming groups in context: Effectiveness in product design firm. Administratioe science quarterly, 1996.
Ellis Paul Torrance. Torrance Tests of Creative Thinking. Terns-Techhical Manual. Research Edition. Verbal Tests Forms a and B. Figural Tests Forms a and Personnel Press, 1966.
Solomon Ubani, Suleyman Olcay Polat, and Rodney D. Nielsen. Zeroshotdataaug: Generat- ing and augmenting training data with chatgpt. arXiv:2304. 143344 2023.
Wallach, M. A., and Kogan, N. Modes of thinking in young children: A study of the creativity-intelligenee distinction. American Psychological Association, 1965.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. ChatGPT good NLG evaluator? a preliminary study. In New Frontiers in Summarization Workshop, 2023a.
Xintao Wang, Yunze Xiao, Jen tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, Jiangjie Chen, Cheng Li, and Yanghua Xiao. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. 2023b.
Zekun Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. 2xXi::2300.00766, 2023c.
Jimmy Wei, Kurt Shuster, Arthur Szlam, Jason Weston, Jack Urbanek, and Mojtaba Komeili. Multi-party chat: Conversational agents in group settings with humans and models. arXio:2304.13835, 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. 2023.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXio:2309.03409 2023a.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions Knowledge Discovery from Data, 2023b.
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. Wordcraft: Story writing with large language models. In 27th International Conference on Intelligent User Interfaces, 2022.
Wenxuan Zhang, Yue Deng, Bing-Quan Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment analysis in the era of large language models: A reality check. arXiv:2305.15005, 2023a.
Xia Zhang, Chunxia Jiang, Fang Chen, Hualian Wu, Li Yang, Zhixia Jiang, and Jing Zhou. Icu quasi-speciality nurses' knowledge, attitudes and practices regarding early mobilization: A cross-sectional survey. Nursing Open, 2023b.
14

---

Published as a conference paper at COLM 2024
Appendix
A
Baseline Result
We conduct LLM evaluations using -p-5 model, by calling API with default parameters set to temperature = and top = 1. Table shows all of our baseline scores, including single LLM and multi-LLM debate with different prompts and mechanisms.
For single LLM, we conduct the experiment with zero-shot, few-shot (Brown et al., 2020), zero-shot-CoT (Kojima et al., 2022), LLM stimuli (Li et al.) 2023b), and "lake a deep breath" (Yang et al. 2023a). As for multi-LLM, we reproduce the Brainstorm, then Se- lect (Summers-Stay et al. 2023) on AUT, specifically designed for AUT, and LLM Debate (Du et al. 2023) on all the datasets as our baseline.
Few-shot. Provides a few examples to guide the model, enhancing performance with contextual cues.
Zero-shot CoT. Prompts the model to generate intermediate steps before the final answer, improving reasoning.
LLM stimuli. Uses specific emotional stimuli such as "This is really important for my career" to influence its performance.
Take a deep breath. Uses the prompt, "Take a deep breath and work on this problem step-by-slep" discovered by Optimization by PROmpting(OPRO), that has shown outperform human-designed prompts.
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
BENCHMARK
METHOD
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Zero-shot
3.47
0.38
8.82
Few-shot
3.71
0.14
2.95
0.15
5.10
2.06
12.0
3.04
Zero-shot-CoT
3.52
0.32
3.56
0.25
15.1
3.62
12.37
3.03
LLM stimuli
3.41
0.27
3.34
0.28
11.20
3.17
11.47
3.29
AUT
Take a deep breath
3.58
0.30
14.37
4.45
Brainstorm, then select
3.84
0.61
3.32
0.65
4.63
1.43
4.60
1.63
L.M Debate
3.73
0.47
3.78
0.47
10.47
2.96
9.63
2.73
Discussion (ours)
4.44
0.30
4.22
0.27
9.19
2.25
9.68
1.92
Zero-shot
2.46
0.33
0.29
14.32
4.52
5.82
3.11
Few-shot
2.47
0.27
0.36
12,23
6.22
6.80
3.33
Zero-shot-Cop
2.29
0.23
2.32
0.27
14.87
5.67
8.13
2.68
INSTANCES
L.M stimuli
2.25
0.19
1.81
0.40
13.30
4.46
8.87
3.26
Take a deep breath
2.27
0.39
5.68
7.3
LLM Debate
2.61
0.32
1.90
0.29
26.21
6.73
11.28
8.72
IJM Discussion (ours)
3.65
0.34
2,20
0.58
16.88
10.04
11.11
5.26
Zero-shot
2.66
0.39
1.99
0.31
7.00
1.76
6.49
1.61
Few-shot
2.72
0.22
0.27
7.87
1.56
6.13
Zero-shot-CoT
2.79
0.14
0.28
8.30
1.85
7,90
1.63
SIMILARITIES
stimuli
2.90
0.12
2.43
0.20
8.30
1.10
7.13
2.07
Take a deep breath
2.65
0.19
2,30
0.33
8.60
1.43
7.07
1.16
LLM Debate
2,80
0.38
9.50
2.41
7.99
LLM Discussion (ours)
3.29
0.30
2.52
0.54
7.27
2.13
8.14
2.04
Zero-shot
3.18
0.38
2.77
0.51
6.37
2.35
6.06
2.31
Few-shot
3.28
0.29
2.98
0.42
8.30
2.49
8.17
2.87
Zero-shot-CoT
3.21
0.40
3.33
0.33
7.87
2.55
7.93
2.50
SCIENTIFIC
LLM stimuli
3.31
0.30
3.16
0.39
2.66
7.80
2.91
Take deep breath
3.10
0.53
2,80
0.55
7.80
2.11
6.90
2.71
LLM Debate
3.30
0.51
3.29
0.72
5.85
3.37
5.94
3.13
Discussion (ours)
3.91
0.42
2.80
5.91
2.39
Table 5: Baseline Results, This table shows the results the baselines and our method.
15

---

to leverage technology Published as a conference paper at COLM 2024
B Role-Play Role Sets
Table and Table show the settings of our role-play and six thinking hats experiments, including prompts, specialty, and roles. Table showws the evaluation results of the single agent with both settings.
ROLE
SPECIALITY
PROMPT
Visionary Millionaire
Financial
success forward-
As Visionary Millionaire, your mission to leverage your financial insight and forward- -athiiniiininggggg approach to inspire groundbreaking ideas. Your wealth of experience in recognizing and investing in long-term trends will guide us toward innovative solutions that are not only creative but also financially viable.
and thinking
Startup Founder
Agility, innovation, and risk-taking
As Startup Founder, your agility, knack for innovation, and willingness take risks empower you to challenge the status quo. Your role is to push suggest scalable solutions, and explore how technology can solve traditional problems in unconventional ways.
think differently,,
Social En- trepreneur
Social impact and ethical considera- tion
As Social Entrepreneur, you bring a deep commitment societal change through business. Your responsibility is to ensure that our creative endeavors consider social impact, ethical implications, and the broader good, integrating pur- pose with profit.
Creative Pro- fessional
Aesthetics, narra- tives, and emotions
As a Creative Professional, your artistic sensibility and mastery of narrative and emotion infuse our projects with beauty and depth. You are tasked with challenging to think expressively, ensuring our solutions not only solve problems but also resonate on a human level.
Customer/ User
End user needs and preferences
As the voice of the Customer/ User, your role is to anchor creative discussions in the real-world needs and prefer- ences those we serve. Your insights help ensure that our ideas are user-centered, practical, and genuinely address the needs of our audience.
Environmen- talist
Sustainability and environmental health
As an Environmentalist, your mission is to champion eco- friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth.
Digital no- Remote work and mad
As Digital Nomad, your expertise in remote work and the digital lifestyle opens our eyes to the possibilities of the digital economy. You encourage creative ways, ensuring our solutions are adaptable and relevant rapidly changing world.
digital lifestyle
Industry in- Insider knowledge sider
As an Industry Insider, your deep understanding of specific sectors provides us with insider knowledge and awareness of industry trends. Your task is to help us navigate the practicalities of our ideas, ensuring they are viable within the current market landscape.
and industry trends
Academic/ Researcher
Data-Driven Insights and Theo- retical Frameworks
They can introduce data-driven insights, theoretical frame- works, and evidence-based perspectives to ground creative ideas in solid research.
Futurist
Emerging technolo- gies and future sce- narios
As Futurist, you inspire us think beyond the present, considering emerging technologies and potential future scenarios. Your role is to challenge us to envision the future impact of our ideas, ensuring they innovative, forward- thinking, and ready for the challenges ahead.
Table 6: Role-Play Settings. Role, speciality and prompt generated by GPT-4 for LLM Discussion.
16

---

Published as a conference paper at COLM 2024
ROLE
SPECIALITY
PROMPT
White hat
Ingormation analy- sis and facts
Focuses on available data and past information, analyzing trends and gaps in knowledge, striving for an objective viewpoint.
Red hat
Emotions and feel- ings interpretation
Listens to and validates the emotional responses of the group, understanding the values and intuition behind reac- tions, without judgment justification.
Black Hat
Critical evaluation and caution
Critically examines all potential flaws and risks, focusing on judgment to avoid pitfalls, ensuring the group is well- prepared for challenges.
Yellow hat
optimism and bene- fits
Explores the positives and the value of decisions, promot- ing a hopeful and constructive outlook, and highlighting paths to success.
Green hat
Creativity and inno- vation
Encourages the generation of new ideas and alternative solutions, fostering an environment of creativity and inno- vation.
Blue hat
Overview and pro- management
Oversees and manages the thinking process, ensuring that each hat is utilized effectively and that discussions remain structured and focused.
Table 7: Six Thinking Hats settings. Role, speciality and prompt generated by GPT-4 for LLM Discussion.
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
ROLE
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Visionary Millionaire
3.67
0.43
3.55
0.42
8.73
1.42
9.27
0.74
Social Entrepreneur
3,43
0.33
3,39
0.416
9.5
0.34
9.53
0.85
Creative Professional
3.55
0.43
3.43
0.40
9.83
1.44
9.57
2.33
Environmentalist
3.50
0.40
3.50
0.34
9.50
0.31
9.10
1.25
Academic Researcher
3.54
0.45
3.42
0.48
9.33
0.21
9.97
0.10
Futurist
3.86
0.54
3.69
0.47
9.33
0.89
9.90
0.21
Startup Founder
3.76
0.23
3.89
0.40
8.10
1.92
8.58
1.78
Customer User
3.64
0.25
3.85
0.46
8.07
1.88
8.44
1.88
Digital Nomad
3.55
0.24
3.89
0.24
8.93
1.50
9.11
1.67
Industry Insider
3.58
0.32
3.97
0.11
8.58
1.25
8.70
White Hat
3.61
0.31
3,91
0.34
8.57
1.44
9.01
1.55
Red Hat
3.67
0.31
3.95
0.16
8.27
2.30
8.42
2.05
Black Hat
3.66
0.31
3,96
0,25
6.49
2.20
6.99
2.19
Yellow Hat
3.58
0.29
3,85
0.44
8.86
1.48
9.17
1.23
Green Hat
3.71
0.24
3.98
0.13
8.90
1.28
9.00
1.50
Blue Hat
3.67
0.30
3.88
0.25
8.66
1.57
8.58
1.61
Table 8: Single agent with roles results. specialized roles.
LLM evaluation results of single agent with
C Evaluation Prompts
Table shows our prompts for evaluation on different metrics.
D Ablation Study for Prompts
This ablation study helps us to choose the most effective prompt for our main experi- ments. 10
the candidates for initiation phase prompt generated by GPT-4, and the results are shown in Table [1 We chose prompt as our official experiment prompt since has the most stable performance on 4 datasets.
17

---

Published as a conference paper at COLM 2024
METRICS
PROMPT
Originality
You are a helpful assistant and a critical thinker. In this task, partici- pants were asked to list many uses for an item possible, common divergent thinking task that measures creativity. Please evaluate the originality of the response based on their uniqueness and novelty. Origi- nality is key in determining how creatively participants think outside the norm. Rate the overall originality on a scale from to 5, and conclude with the score in the format: '[[]]]'. Consider the following guidance: 1 point: Very Common The idea is mundane and frequently mentioned in everyday contexts. There's 's a notable lack of novelty, with response being the most typical or expected uses. 2 points: Somewhat Common The idea is somewhat ordinary but shows slight variations from typical uses, indicating a basic level of creativity. 3 points: Moderately Original The idea displays fair amount of creativity and novelty. They are not the usual thoughts but aren't highly rare or unexpected. -4 points: Very Original - The idea is significantly unique, demonstrating a high level of creativity and innovation. They are unexpected and not commonly considered. points: Extremely Original The idea is extraordinarily unique and rare, displaying a high degree of novelty, creativity, and unexpectedness. The idea is seldom thought of in typical contexts. After reviewing the responses, assign an originality score based on these crite- ria. Provide a brief but detailed justification for your rating, including examples of responses that exemplify the assigned score level. It is ex- tremely important to put the score in this format: '[[X]]'
Elaboration
You are a helpful assistant and a critical thinker. Participants were asked to list as many uses for an item as possible. Please evaluate the level of elaboration of the response on scale of 1 to where 1 the least elaborated and 5 is the most elaborated. Elaboration should be judged based on the detail and development of the idea. Conclude with the score in this format: '[[]]I' Consider the following guidance: point: Very Basic - The response is extremely basic with minimal detail or explanation. Idea is presented in very simple or cursory manner. 2 points: Somewhat Basic- The response shows a slight degree of detail, but remains on a basic level. Idea is somewhat developed but lacks depth. 3 points: Moderately Elaborated The response offers moderate level of detail and development. Idea is explained to fair extent, showing some thought and consideration. 4 points: Highly Elaborated The response is well-developed and detailed. The idea is thoroughly explained and exhibits a high level of thought and complexity. 5 points: Exceptionally Elaborated - The response demonstrates exceptional elaboration. Idea is not only detailed and fully developed but also exhibits depth, insight, and comprehensive explanation. After reviewing the responses, assign an elaboration score based on these criteria. Provide brief justification for your rating. It is extremely important to put the score in this format: 'IXXÍI'
Fluency
Your task is to evaluate a list of uses for a specific item provided by par- ticipants, focusing on identifying each unique and practical use listed. It's important to only consider uses that are relevant and feasible. Con- clude your analysis by stating the total number of unique, relevant uses in this specific format: [[X]]. Also, briefly explain how you determined whether a response was relevant and practical.
Flexibility
Your task to assess the range of unique categories or types of uses suggested in responses regarding the uses for a specific item. Your objective is to define and count the distinct categories perspectives evident in the responses, and provide a brief explanation for how you determined these categories. Conclude your analysis by indicating the total number of unique categories or perspectives using the format: [[x]].
Table 9: Prompt for evaluation. Evaluation prompts for four different metrics.
18

---

Published as a conference paper at COLM 2024
PROMPT
CONTENT
Prompt
You would in group discussion with other teammates, as a result, you should answer diverge and creative you
Prompt 2
You're in brainstorming session where each idea leads to the next. Embrace the flow of creativity without limits, encouraging one another build on each suggestion for unexpected connections.
Prompt 3
Pretend your team is at think tank where unconventional ideas are the norm. Challenge each other think from different perspectives, considering the most unusual innovative ideas.
Prompt
Engage in collaborative discussion where each of you contributes a unique insight query, aiming to delve into uncharted territories of thought. Throughout the discussion, focus on expanding the scope and depth of each contribution through constructive feedback, counterpoints, and further questioning. The objective is to achieve a broad spectrum of ideas and solutions, promoting culture of continuous learning and innovation.
Prompt 5
Envision your group as a crew on a mission solve a mystery using only your creativity and wit. How would you piece together clues from each member's s ideas to find the solution? And this would be crucial your member's 's life
Table 10: Initiation Phase Prompts. Initiation phase prompts generated by GPT-4 aim to improve the LLMs' capability of divergent thinking.
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
BENCHMARK
METHOD
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Prompt 3.97
0.36
3.75
0.47
8.43
3.27
7.58
2.3
Prompt
0.32
3.67
0.41
9.52
3.92
8.53
AUT
Prompt
0.37
4.03
0.47
7.50
2.34
7.80
2.24
Prompt
4.04
0.40
3.89
0.33
7.37
2.04
7.18
2.01
Prompt
4.07
0.51
3.84
0.43
8.70
2.75
9.28
2.76
Prompt 3.71
0.53
2.97
0.81
4.52
1.55
4.82
1.48
Prompt 2
3.78
0.23
3.13
0.54
6.17
2.19
6.12
2.28
INSTANCES
Prompt 3
3.59
1.22
2.81
1.22
3.88
2.02
3.63
1.48
Prompt
3.66
0.93
3.17
0.96
4.23
1.76
4.62
1.67
Prompt
0.94
3.05
2.37
4.70
Prompt
3.25
0.51
2.34
0.60
6.08
2.05
6.70
1.82
Prompt
0.21
2.47
3.44
9.38
SIMILARITIES
Prompt
0.34
2.62
0.54
1.62
5.85
1.68
Prompt
3.10
0.47
2.79
0.75
5.28
1.96
5.72
1.60
Prompt 5
2.85
0.24
2.74
0.63
5.02
1.73
6.17
1.89
Prompt
3.56
0.44
2.53
0.87
15.17
11.68
6.50
3.70
Prompt 2
3.21
0.57
2.83
0.84
24.60
16.62
5.97
2.65
SCIENTIFIC
Prompt 3.73
0.45
2.32
0.67
12.83
12.73
6.72
2.12
Prompt 3.25
0.54
2.63
0.70
5.80
3.32
6.32
2,40
Prompt 2.90
0.62
2.25
0.58
4.47
5.75
Table 11: Initiation Phase Prompts Results. LLM evaluation results of GPT-4 generated prompts in Table 10 on four benchmarks.
E Correlation Between Length and Elaboration
Upon discovering that Kendall's correlation coefficient diverges in the Elaboration metric between LLM evaluation and human evaluation, we conduct an experiment to examine the impact of answer length the elaboration scores assigned by humans and LLMss. Table
19

---

Published as a conference paper at COLM 2024
12 indicates that humans are indeed more inclined to award higher elaboration scores to lengthier responses with high correlation.
TYPE
Correlation
Human LLM
0.8300
0.4283
Table 12: Correlation Between Length of Response and Elaboration Scores. Humans tend to give higher elaboration scores to longer responses, showing a strong correlation with response length.
Analysis of Response Length and Its Impact on Scores
Table 13 presents the average response length and its standard deviation for single agent, LLM Debate, and LLM Discussion.
AUT
INSTANCES
SIMILARITIES
SCIENTIFIC
Single Agent
14.59.15.54
1.4810.81
13.26:1..73
23.07
LLM Debate
23.48:17.11
1.95:11.32
21.55-16.85
37.41
LLM Discussion
27.82:4-5.48
8.00:::996
32.03-19.29
47.23-424.12
Table 13: Average Word Count. The average word count of the baselines and our method over four benchmarks.
Additionally, Table 14 illustrates the correlations between the length of responses and scores. The length-score correlations of Originality within each framework on different benchmarks are lower than 0.3, All of the correlations for Elaboration are moderately correlated. As result, this table shows weak correlation between response length and creativity scores.
Single Agent
LLM Debate
LLM Discussion
Originality
Elaboration
Originality
Elaboration
Originality
Elaboration
AUT
0.22
0.47
0.24
0.57
0.07
0.31
INSTANCES
0.25
0.36
0.21
0.42
0.26
0.35
SIMILARITIES
0.20
0.57
0.10
0.46
0.07
0.21
SCIENTIFIC
0.02
0.33
0.04
0.19
0.11
-0.05
Table 14: Correlation Between Word Count and Scores. Pearson correlation coefficient of word count and creativity scores in different frameworks.
Furthermore, Figure 7 and Figure 8 shows that within same word length intervals, our framework, LLM Discussion, outperforms other baselines in Originality and Elaboration in most of the benchmarks.
G Why are Fluency and Flexibility Less Important
We do not prioritize Fluency and Flexibility as main metrics for a few reasons:
Possible Inflated Score: Language models can inherently generate large number of responses, which might artificially inflate flexibility scores.
Focus on Core Creativity Metrics: We prioritzz originality and elaboration because they more directly reflect the model's creative potential.
suggest that fluency and flexibility are high due to LLM's ability to generate many responses quickly. Hadas & Hershkovitz (2024) also suggests that flexibility scores significantly affected by fluency scores.
Guzik et al. 2003) and Chi 2024
20

---

Published as a conference paper at COLM 2024
AUT Word Count Bins v.s. Originality Score
Instances- Word Count Bins v.s. Originality Score
Sigge LUM Decamatoon
Stogle
Debate
Average Originality Score
Average Originality Score
3.5
LUM Dicusatin
3.0
2.5
2.0
1.5
0.5
0:10
11:15
16:20
21:25
38:30
40
45
50+
0.0
11:14
15-18
19.22
23-26
25+
28
31
38
41
46
Word Count Bins
Word Count Bins
Similarities Word Count Bins v.s. Originality Score
Scientific Word Count Bins Originality Score
Average Originality Score
4.0
3.5
LLM Debane
Stngle Apent
EUM Uscusase
Avergge Originality Score
3.5
4.0
MIM Diguncon
Singl
Debute
3.0
3.0
2.5
2.0
2.0
1.5-
1.5
1.0
1.0
0.
0.5
0.0
6.10
11.20
21-30
31-40
4-500
51-60
61:70
70+
0.0
0:10
11-20
21:30
31.40
42-50
51
-0000
61
71-100
101-150
101
Word Count Bins
Word Count Bins
Figure 7: Originality Scores to Response Length Intervals. Our method outperforms the
baselines in the same response length interval for Originality scores.
AUT Word Count Eleboration Score
Instances Word Count Bins Elabaration Score
Singls Agent
Single Agert
LLM Debatte
LLM Debats
LLM Dicoussion
Dioruncion
Average Elaboration Score
4-
Average Elaboratioon Score
2.5
2.0
3
1.5
0.5
0-10
11-15
16:20
21-25
26-30
31-35
38-40
42-45
46-50
50+
0.0
9-10
I1-14
15:18
19:22
23-20
26+
Word Count Dins
Word Count Bims
Similarities Word Count Bins Elaboration Score
Scientific Word Count v.s. Elaboration Score
4.0
Single Apent
Single Agert
Betare
Dubuts
Average Elebbrrttoo Score
3.5
Dacusuiion
Aveeaae Elaboration Score
3.5
L1M Diszuntion
3.0
3.0
2.5
2.5
2.0-
2.0
1.5
1.5
1.0
0.5
0.5
0.0
6-10
11-20
21-30
31-40
$1.60
61:70
70+
0.0
0:10
11.20
21.30
31.40
42-50
60
71-100
101-150
150+
51
61
Word Count Bins
Word Count Bins
Figure& 8: Elaboration Scores to Response Length Intervals. Our method outperforms the
baselines in the same response length interval for Elaboration scores.
21

---

Published as a conference paper at COLM 2024
Table 15 shows that Fluency and Flexibility are easily increased by asking the LLM discussion agent to generate as many answers as possible (AMAP) during the convergence phase.
BENCHMARK
METHOD
ORICINALITY
ELABORATION
FLUENCY
FLEXIBILITY
Mean
Mean
Mean
Mean
Discussion
4.4
4.22
9.19
9.68
AUT
LLM Discussion AMAP
4.40
4.23
9.38
9.58
LLM Discussion
3.65
2.20
16.88
11.11
INSTANCES
LLM Discussion AMAP
3.65
2.22
22.06
11.08
LLM Discussion
3.29
2.52
7:27
8.14
SIMILARITIES
LLM Discussion + AMAP
3.27
2.55
10.89
11.52
Discussion
3.95
3.47
5.58
SCIENTIFIC
LLM Discussion
AMAP
3.89
3.32
7.63
8.08
Table 15: Comparison of LLM Discussion with AMAP Prompt. AMAP corresponds to prompting language models to generate "as many as possible".
H Analysis of Temperature and Its Impact
Temperature is known to affect the diversity of model outputs in language generation tasks. To investigate its impact on creativity scores, we experiment with increased temperatures ranging from 0.0 to 2.0 (1.0 was used for the main experiments) shown in Figure The results show that increasing temperature leads to higher diversity (a lower BLL score) while not resulting in higher creativity scores. The model outputs nonsensical responses, e.g., unrecognizable symbols, with a temperature > 1.6.
AUT. The result is
While higher temperatures did increase diversity, indicated by the fall in the self-BLEU score, this did not correlate with higher creativity. In fact, higher temperatures seemed to reduce the originality and elaboration of the responses in some cases.
Creative Score v.s. Temperature
Self-BLEU Score v.s. Temperature
0.18
8-
0.16
Creative Score
Self-BLEU Score
Mean Fluency
0.14
Mean Flexibility
6-
Mean Originality
Mean Elaboration
0.12
5
0.10
3
0.08
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Temperature
Temperature
Figure 9: Temperature Impact. The left figure shows the creative scores across different temperatures, demonstrating minimal variation as the temperature increases. The right figure illustrates the self-BLEU scores a function of temperature. As temperature increases, self-BLEU scores decrease, indicating an increase in diversity.
I Analysis on Three-Phase Discussion Framework and Role-Play
Table shows our ablation study on the effect of role play and the three-phase discussion framework individually.
22

---

Published as a conference paper at COLM 2024
ORIGINALITY
ELABORATION
FLUENCY
FLEXIBILITY
BENCHMARK
METHOD
Mean
Std.
Mean
Std.
Mean
Std.
Mean
Std.
Role w/o discussion
4.58
0.37
4.46
0.44
8.94
2.03
9.39
1.30
AUT
Discussion w/o role
3.95
0.25
3.89
0.24
9.07
1.51
8.82
1.56
Discussion w/
0.30
4.22
Role w/o discussion
3.33
0.53
0.51
20.99
10.89
5.38
INSTANCES
Discussion w/o role
3.28
0.43
2.21
0.65
21.81
21.44
11.28
7:56
Discussion w/ role
3.65
0.34
2.20
0.58
16.88
10.04
11.11
5,26
Role w/o discussion
2.98
0.40
2.55
0.52
5.30
2.64
5.73
1.91
SIMILARITIES
Discussion w/o role
3.04
0.26
2.83
0.33
5,98
2.25
5.92
1.96
Discussion w/ role
3.29
0.30
2.52
2.61
5.91
2.39
Role w/o discussion
0.51
3.15
7.59
7.73
SCIENTIRI
Discussion w/o role
0.31
3.31
2.29
7.05
1.89
Discussion w/ role
3.95
0.25
3.47
0.55
7.27
2.13
8.14
2.04
Table 16: Ablation study on Three-Phase Discussion Framework and Role-Play. LLM Evaluation results of role-play without three-phase discussion, three-phase discussion without role-play, and LLM Discussion (role play and three-phase discussion) on four benchmarks.
J Role Play Generation
Our purpose in generating various roles with diverse backgrounds is encourage agents to discuss creative tasks from different perspectives. This is also known as "role storming," a widely used technique the ideation stage of design thinking. This technique allows participants to set aside inherent limitations, helping them to come up with unprecedented ideas. For example, millionaires or billionaires are commonly used roles (Panchenko) 2024 that free people from budget constraints.
We generated the list of roles an automated way using the GPT-4 model. The simplified chat log is shown below, where integrate similar prompts and filter out unrelated results. The full version can be referred to at https:/ chatgpt.com com /sans/sa////aaa mhaaref3350049 (tt////5094-dbb0 -01--499---666448------5 4089--449- 1153e6696221 This is initial attempt demonstrate the eftectiveness of this tramework, but the roles are not the optimal set. We agree that covering broader range of demographics crucial, so we will explore this domain in our future work.
Round 1 User Prompt
What are some roles that are typically used in boosting creative thinking, like different personas to diversitize the ideation stage.
Round 1 GPT-4 Response
When looking to diversify the ideation stage, introducing different personas can indeed offer range of perspectives that might not be considered otherwise. Here some personas that can be particularly useful in diversifying thought and enhancing creativity:
The Visionary Millionaire: This persona brings the perspective of high financial success combined with forward-thinking approach. They can offer insights into long-term investments, trends, and big-picture thinking.
The Startup Founder: Embodying agility, innovation, and risk-taking, this persona can introduce ideas that disrupt traditional markets suggest scalable, tech-driven solutions.
The Social Entrepreneur: Focused solving societal issues through business, this persona adds a layer of social impact and ethical consideration to the brainstorming process.
23

---

Published as a conference paper at COLM 2024
The Creative Professional:
Artists, writers, designers, and musicians can provide unique insights into aesthetic, narrative, and emotional dimensions that might enhance the appeal of solution.
The Customer/ User: Representing the end user's voice, this persona helps ensure that ideas are aligned with user needs, preferences, and experiences.
The Environmentalist This persona brings sustainability angle to the discussion, encouraging solutions that are aco-friendly and promote long-term environmental health.
The Digital Nomad: Offering insights into remote work, digital tools, and global trends, this persona can help identify opportunities in the digital economy and lifestyle shifts.
The Academic/ Reseaarherr They can introduce data-driven insights, theoretical frameworks, and evidence-based perspectives to ground creative ideas in solid research.
Someone deeply entrenched in specific industry can offer insider knowledge, trend awareness, and practical constraints or opportunities within that field.
The Industry Insider:
The Futurist: This persona looks at emerging technologies, societal shifts, and poten- tial future scenarios to inspire ideas that are ahead of their time.
Incorporating these personas into the ideation process can help teams explore wider range of ideas, anticipate different needs, and consider various angles, making for a richer and more innovative outcome.
Round 3 User Prompt
Great, now follow this format and generate jsonn file for these roles: Remain "type": "openai", "model name": "gpt-3.5-turbo" the same. and the agent name "GPT Agent (agent role)"
" "type": "openai".
"model_name". 3 "agent_name": "GPT Agent White Hat", "agent_role": "White Hat"
"agent_sppciaality: "Information Analysis and Facts", "agent_role_promp" "Focuses on available data and past information, analyzing trends and gaps in knowledge, striving for an objective viewpoint."
).
10
11 "type": "openai",
"model_named "gpt 5- turbo "agent_name": "GPT Agent Red Hat", "agent_role"; "Red Hat"
14
15
"agent_specialityy" "Emotions and Feelings Interpretation
16
rogent_role__romptt "Listens to and validates the emotional responses of the group, understanding the values intuition behind reactions, without judgment or justification."
]
Round 3 GPT-4 Response
Below is the json format including all the personas you specified:
" "type" "openai". "model_name"" "gpt 3.5 turbo",
24

---

Published as a conference paper at COLM 2024
".agentnaame:: "GPT Agent Visionary Millionaire", "agent_role": "Visionary Millionaire", Dagent_specialien "Financial Success and Forward- Thinking",
gegent_role_ppooee "This persona brings the perspective high financial success combined with a forward- thinking approach. They can offer insights into long- term investments, trends, big-picture thinking.
"type": "openai",
13
imodel_name""
14
"ageet_name"" "GPT Agent Startup Founder", "agent_role": "Startup Founder".
15
16
"agent_specialityy" "Agility, Innovation, and Risk Risk-Taking
17
"agent_role__rromt""
"Embodying agility, innovation, and
risk- this persona introduce ideas that disrupt traditional markets or suggest scalable, tech- driven solutions."
18
19
20
"type": "openai",
21
"model_name", 3.t-33--turbo
22
"agent_name": "GPT Agent Social Entrepreneur",
23
"agent_role": "Social Entrepreneur", "agent_specialityy: "Social Impact and Ethical Consideration".
"agent_role_proom "Focused on solving societal issues through business, this persona adds layer of social impact and ethical consideration to the brainstorming process.
76
}
27
28
"type": "openai",
29
"model_name" 3pt-3..-ttrbb.
"agent_name": "GPT Agent Creative Professional", "agent_role"; "Creative Professional", "agent_specialityy: "Aesthetics, Narratives, and Emotions
33
"agenit_roe__rompt" "Artists, writers, designers, and musicians can provide unique insights into aesthetic. narrative, emotional dimensions that might enhance the appeal of solution."
}.
"type": "openai".
37
"model_name" "gpt -3. turbo. "agent_name"" "GPT Agent Customer/User". "agent_role": "Customer /User"
38
39
10
"agent_speciality"" "End User Needs and Preferences", "agent_role__rompe "Representing the end user voice, this persona helps ensure that ideas are aligned with needs, preferences, and experiences.
41
44
"type": "openai",
15
"model__naee namel__amee -.ttttrbbb" "agent_name" "GPT Agent Enviromennallist",
46
47
"agent_role": "Envirommenaalist. "agent_specialice
18
"Sustainability and Environmental
Health".
"agent_rolee_rrope" "This persona brings sustainability angle to the discussion, encouraging solutions that eco-friendly and promote long term environmental health."
50
51
52
"type": "openai",
25

---

Published as a conference paper at COLM 2024
"mmode__name "got 3.. S-tubb"",
"agent_name": "GPT Agent Digital Nomad", "aaent_role": "Digital Nomad",
85
56
"aggnt_specialitty" "Remote Work and Digital Lifestyle", "agent_role_promg"
57
"Offering insights into remote work, digital tools, and global trends, this persona can help identify opportunities in the digital economy and lifestyle shifts."
}.
"type": "openai", "model_name". "gpt 3..-33.5tubb "agent_name", "GPT Agent Academic/Reseee RReeeeeeeearrr "agent_role": "Academic/ heteemmmc//eeeeceerr "agent_speciality
61
62
63
64
Insights and Theoretical
Frameworks",
65
"agent_role_prompt" "They can introduce data-driven insights, theoretical frameworks, and evidence -based perspectives to ground creative ideas in solid research."
66
67
68
"type": "openai",
69
"model_name"
70
"ageet_namm". "GPT Agent Industry Insider",
71
"agent_role": "Industry Insider".
72
"agent_speciality". "Insider Knowledge and Industry Trends
"ageni_rrol__rrontt" "Someone deeply entrenched in specific industry can offer insider knowledge, trend awareness, and practical constraints or opportunities within that field."
74
75
76
"type": "openai",
77
"model_name": "gpt-3. 5.5---bbbbb" "agent_name": "GPT Agent Futurist", "agent_role": "Futurist",
"agent_specialityy: "Emerging Technologies and Future Scenarios",
81
"agent_role_promtt"" "This persona looks at emerging technologies, societal shifts, and potential future scenarios to inspire ideas that are ahead of their time." "
82
83
This JSON array provides a structured way to represent the diverse personas you want to incorporate into your ideation creative thinking process. Each persona is detailed with unique agent name, role, specialty, and prompt that describes their contribution to the brainstorming innovation process.
K
Chat Log
This section examines the qualitative dimensions of our discussion. Given the extensive nature of the text, we have included the discussion log our supplementary materials. The "discussion log.zip" contains two html files: 11m discussion chatlog. html which documents the discussion log of one of the agents on the Scientific Creativity task within our LLM Discussion framework, and "llm.debate.cif wedeeattecchatog.hhmmll. which does the same within the LLM Debate framework.
For LLM Discussion, we can see clear discussion in collaborative tone, cited, "Building on those ideas, believe we can combine elements of creativity, systematic testing, and a multidisci- plinary approach..." (Final Response, Point 1). The phrase "Building those ideas" explicitly encourages collaborative engagement, suggesting that each contribution valued. More- over, could see comprehensive approach here: By merging elements of sustainability,
26

---

Published as a conference paper at COLM 2024
creativity, advanced technology, and human engagement..." This shows direct outcome of engaging multiple perspectives.
LLM Debate framework showed more direct and polutiionfocused response. For entire conversations throughout, agents tend to only update the answer without any discussion at all. We can see in the html file that the agents usually lead to an original solution for creativity tasks.
iterative approach, which doesn't
K.1
Conversation of One Agent
Round 1 User Prompt
You are Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth. Remember to claim your role in the beginning of each conversation. Initiate a discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, different. Present list of many creative and diverse uses for Umbrella as possible. You are in group discussion with other teammates; as result, answer as diversely and creatively you can.
Round 1 Agent Response
As an Environmentalist specializing Sustainability and Environmental Health, I believe it's important to think outside the box and explore creative uses for ev- eryday items like umbrellas. Let's brainstorm some innovative ideas: Shelter for wildlife: Use umbrellas to create makeshift shelters for small animals during inclement weather. 2. Hanging plant holder: Attach umbrellas upside down from the ceiling and them to hold potted plants. 3. Umbrella art installation: Use colorful umbrellas to create a visually appealing art installation a public space. Umbrella chandelier: Hang multiple umbrellas from the ceiling to create unique and quirky chandelier. 5. Umbrella garden border: Line the perimeter of garden with open umbrellas to create whimsical border. Umbrella storage organizer: Hang umbrellas on a wall-mounted rack to store keys, hats, and other small items. 7. Umbrella lampshade: Repurpose
lampshade for a DIY lighting fixture. 8. Umbrella kite: Attach streamers to the spokes of an open umbrella and fly like a kite on windy day. 9. Umbrella runway: Set up row of open umbrellas as makeshift runway for remote-controlled cars toy airplanes. 10. Umbrella door wreath: Decorate an umbrella with flowers, ribbons, or other decorative elements to create unique door wreath. hope these ideas spark some creativity and inspire us to think differently about how we can use umbrellas in our daily lives! Let's continue to brainstorm and come up with even more creative uses.
old umbrella
Round 2 User Prompt
You are Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth. Remember claim your role the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'As a Creative Professional specializing in Aesthetics, Narratives, and Emotions, let's brainstorm some creative uses for an umbrella: 1. Turn umbrella into unique
27

---

Published as a conference paper at COLM 2024
art installation by painting it with vibrant colors and patterns, then suspending it from the ceiling in a gallery or outdoor space. 2. Use an umbrella as a makeshift photography prop by placing in the foreground of shot to add a pop of color or create interesting framing. 3. Transform an umbrella into whimsical outdoor chandelier by attaching fairy lights to the spokes and hanging it above a patio or garden for magical ambience. 4. Repurpose by attaching pockets to the inside where you can store sunscreen, sunglasses, and other essentials. 5. Create DIY hanging plant holder by turning an umbrella upside down, removing the fabric, and securing small pots or planters along the spokes for unique and space-saving garden display. Use an umbrella as a statement piece in fashion photoshoot, incorporating it into the model's outfit or as prop to convey sense of mystery or drama. 7. Turn an umbrella into mobile kitchen organizer by hanging utensils, spice jars, and other cooking essentials from the spokes for easy access while cooking outdoors. attaching fabric panels to the spokes and using it as shelter for outdoor events or picnics. 9. Create a DIY umbrella canopy for backyard wedding or outdoor gathering by suspending multiple umbrellas above a seating area to provide shade and add a whimsical touch. 10. Use an umbrella as a sculptural element in a room design by hanging it from the ceiling and incorporating it into larger art installation or decorative scheme." "" One agent solution: "'As a Futurist specializing in Emerging Technologies and Future Scenarios, I believe it is crucial for us to think outside the box and explore unconventional uses for everyday items like the umbrella. Let's brainstorm some creative uses for umbrellas: 1. Solar-powered umbrella: Imagine an umbrella that harnesses solar energy to charge your devices while providing shade on a sunny day. 2. Musical umbrella: A high-tech umbrella that plays music based on the rhythm of raindrops hitting the canopy, creating a soothing and unique listening experience. 3. Virtual reality umbrella: An umbrella that incorporates VR technology to transport you to different virtual environments while sheltering you from the elements. 4. Umbrella drone: A drone that can deploy and retrieve an umbrella for you, ensuring you stay dry without having to carry one around. 5. Umbrella garden: Convert your umbrella into a mini garden by hanging plants from the ribs, creating a vibrant and eco-friene friendlenlly shelter. 6. Umbrella projector: Turn your umbrella into a portable projector that displays images or videos on the canopy, making rainy days fun and interactive experience. 7. Umbrella communicator: A smart umbrella equipped with communication features like built-in speaker and microphone for hands-free calls or messaging during downpour. 8. Umbrella workout: Incorporate resistance bands into the design of the umbrella, turning it into a convenient fitness accessory for on-the-go workouts. Umbrella art installation: Use umbrellas canvas for creating large-scale art installations, where each canopy displays a unique design or message when opened together. 10. Umbrella recycling station: Transform old umbrellas into sustainable products or art pieces through a community-driven recycling initiative, promoting environmental awareness and creativity. Let's continue to push the boundaries of innovation and explore the endless possibilities of umbrellas beyond their traditional function." " One agent solution: "" As Futurist specializing in emerging technologies and future scenarios, I would like to challenge our team to think creatively about the many potential uses for umbrellas beyond their traditional function of providing shelter from the rain. Let's explore some innovative ideas together: 1. Solar-powered umbrellas that can charge your devices on-the-go. 2. Umbrella drones that can fly ahead to provide shade on sunny day. 3. Smart umbrellas equipped with GPS and weather sensors to alert you of upcoming storms. 4. Umbrella backpacks that can transform into a tent for camping trips. 5. Umbrella cameras that can capture unique bird's-eye view footage. 6. Edible umbrellas made from biodegradable materials for sustainability. 7. Umbrella art installations that double as interactive public sculptures. 8. Multi-functional umbrella tools with built-in tools like flashlight, bottle opener, and compass. 9. Umbrella gardens for urban farming in small spaces. 10. Virtual reality umbrellas that transport you to different locations with immersive
umbrella
stylish beach accessory
Repurpose an umbrella as a makeshift tent by
28

---

Published as a conference paper at COLM 2024
experiences. Let's keep thinking, outside the box and push the boundaries of what umbrellas can do in the future;" What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of as many creative and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; result, answer as diversely and creatively as you can.
Round 2 Agent Response
As Environmentaliit htapeuennnaiistsepciiiiingg in Sustainability and Environmental Health, am thrilled to be part of this creative brainstorming session with my fellow agents. Let's explore more innovative uses for umbrellas: 1. Umbrella bike rack: Convert umbrellas into a unique bike rack design where bicycles can be securely stored within the canopy. 2. Umbrella bookshelf: Stack open umbrellas horizontally to create whimsical bookshelf display that adds touch of creativity to any room. 3. Umbrella birdhouse: Repurpose old umbrellas into charming birdhouses by adding perches and bird feeders to the canopy. 4. Umbrella wind chimes: Use umbrella spokes as the base for DIY wind chimes, creating soothing sounds with every gentle breeze. Umbrella pet bed: Transform an oversized umbrella into a cozy pet bed by adding plush bedding and creating a comfortable shelter for furry friends. Ümbrella plant stand: Stack open umbrellas of varying sizes to create unique tiered plant stand, perfect for displaying an array of indoor plants. 7. Umbrella photo backdrop: Hang open umbrellas in row creative photo backdrop for events, weddings, or parties, adding a splash of color to photographs. hooks to the inside of an open umbrella to hang and organize children's 's toys, keeping play areas neat and tidy. 9. Umbrella craft organizer: Use open umbrellas to store and display craft supplies like ribbons, yarn, and thread, turning them into functional and decorative storage solutions. 10. Umbrella fashion accessory: Incorporate mini umbrellas into outfits as unique fashion accessories or statement pieces, adding fun and quirky touch to any ensemble. I'm excited to see the creative ideas that our team can come up with and how we can continue to think outside the box when it comes to repurposing umbrellas in unconventional and eco-friendly ways! Let's keep pushing the boundaries of innovation and sustainability.
Umbrella toy storage: Attach
Round 3 User Prompt
You are Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth. Remember to claim your role in the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'Here more creative uses for umbrellas: 1. Umbrella runway fashion show: Use umbrellas as the runway for a fashion show, creating a unique and memorable presentation for the audience. 2. Umbrella meditation station: Set up a tranquil corner with open umbrellas overhead, creating a peaceful and relaxing space for meditation or mindfulness practices. 3. Umbrella obstacle course: Create outdoor obstacle course using open umbrellas as hurdles or tunnels, adding an element of fun and challenge to physical activities. Umbrella runway fashion show: Use umbrellas as the runway for fashion show, creating unique and memorable presentation for the audience. 2. Umbrella meditation station: Set up a tranquil corner with open umbrellas overhead, creating a peaceful and relaxing space for meditation or mindfulness practices. 3. Umbrella obstacle course: Create an outdoor obstacle course using open umbrellas hurdles or tunnels, adding an element
29

---

Published as a conference paper at COLM 2024
of fun and challenge physical activities. 4. Umbrella shadow puppet theater: Use the canopy of an umbrella as a screen for shadow puppet shows, entertaining both children and adults with imaginative storytelling. 5. Umbrella costume party: Encourage guests to get creative and dress up in umbrella-inspired costumes for lively and colorful themed party. 6. Umbrella ice cream stand: Convert an upside- down umbrella into makeshift ice cream stand, suspending cones cups from the spokes to create a playful and whimsical dessert display. 7. Umbrella photo booth: Set up photo booth with umbrellas of various designs and colors rop, encouraging guests to strike poses and capture fun memories. 8. Umbrella solar cooker: Utilize the reflective surface of an unbrella create makeshift solar cooker for outdoor cooking, demonstrating sustainability and innovation in a practical way. Umbrella poetry session: Host poetry reading under canopy of open umbrellas, creating a cozy and intimate atmosphere for sharing written works and creative expressions. 10. Umbrella time capsule: Bury a sealed umbrella in the ground as time capsule, to be unearthed and opened by future generations, revealing messages, mementos, and cultural snapshots from the present day. Let's continue to explore the creative potential of unbrellas and think outside the box for even more inventive uses;" One agent solution: "' "As a Futurist specializing in Emerging Technologies and Future Scenarios, am excited to build on the innovative ideas shared by my fellow agents and continue exploring creative uses for umbrellas. Let's brainstorm some more out-of-the-bou boo-boo ideas: 1. reether-responsive umbrellas: Develop smart umbrellas that change color or patterns based on weather conditions to add a fun and functional element to your outdoor attire. 2. Interactive umbrella game: Create an umbrella equipped with sensors that allow you to play interactive games like pong or trivia with friends by moving the canopy strategically. 3. Umbrella messaging system: Design umbrellas with LED displays on the canopy that can be used to transmit messages or images in crowded spaces or events. 4. Emotional umbrella: Develop umbrellas that change shape or emit calming scents based on the user's 's mood or stress levels, providing a soothing experience during challenging times. 5. Umbrella workout assistant: Incorporate fitness trackers into the handle of the umbrella to track your steps, distance walked, and calories burned while on-the- go, promoting wellness and activity. 6. Astronomical umbrella: Create umbrellas with celestial designs and embedded star projectors that illuminate the night sky the inside of the canopy, turning a rainy evening into mesmerizing stargazing experience. 7. Educational umbrella: Design umbrellas with educational features like maps, historical facts, or language lessons printed on the fabric, offering unique way to learn while staying protected from the elements. 8. Umbrella privacy shield: Develop umbrellas with adjustable privacy screens or blinds that can be activated to create personal cocoon in crowded or open spaces, ensuring moments of solitude and relaxation. 9. Umbrella time capsule: Encourage people to fill designated compartments in the umbrella with mementos, photos, or messages, turning it into a time capsule that captures memories and experiences over time. 10. Soundscaping umbrella: Integrate speakers and soundscaping technology into the handle of the umbrella to create immersive audio experiences that synchronize with the rhythm of rain ambient sounds in nature. Let's continue to think creatively and envision the limitless possibilities of umbrellas in shaping our future experiences and interactions with the world around us." One agent solution: "'1. Umbrella Cocktail Bar: Convert umbrellas into unique cocktail bars where mixologists create drinks inside the canopy for a fun and interactive experience. 2. Umbrella Yoga Studio: Use umbrellas as individual yoga pods in park or outdoor space, providing shelter and privacy for practitioners during their sessions. 3. Umbrella Time Capsule: Bury old umbrellas in time capsule for future generations to unearth and learn about the history of fashion and technology. 4. Umbrella Karaoke Booth: Transform umbrellas into private karaoke booths where singers can perform their favorite songs in a quirky and memorable setting. 5. Umbrella Beach Tent: Reinvent umbrellas as collapsible beach tents with integrated sand anchors for stylish and practical sunshade solution. 6. Umbrella Mapping Project: Equip umbrellas with GPS trackers
30

---

Published as a conference paper at COLM 2024
to create an interactive map of their journeys as they are passed from person to person around the world. 7. Umbrella Language Exchange: Host language exchange events where participants hold umbrellas labeled with different words or phrases to facilitate communication and cultural exchange. 8. Umbrella I'et Shelter: Create miniature shelters for pets using modified umbrellas to provide cozy and portable retreat for furry companions. 9. Umbrella Fashion Show: Organize fashion show featuring mabrelll--nsspire designs made from upcycled materials, showcasing the versatility and creativity of this everyday object. 10. Umbrella Light Display: Install LED lights on umbrellas to create a dazzling light show or interactive art Installation for special events or festivals." What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of as many creative and diverse uses for Umbrella as possible. You are in group discussion with other teammates; result, answer diversely and creatively as you can
Round 3 Agent Response
As an Environmentalist specializing in Sustainability and Environmental Health, I am thrilled to be part of this dynamic brainstorming session with my fellow agents. Let's dive into even more imaginative and unconventional uses for umbrellas: Umbrella community art project: Collaborate with local artists and residents to paint a series of bright and colorful umbrella murals that can be displayed throughout town city to promote creativity and unity. 2. Umbrella outdoor cinema: Trans- form umbrellas into individual outdoor cinema screens where people can gather in park or public space to watch movies under the stars, creating unique and cozy mavie-watching experience. 3. Umbrella scavenger hunt: Organize a scav- enger hunt where participants follow clues hidden under various umbrellas placed around a neighborhood or city, adding an element of surprise and excitement to the traditional game. 4. Umbrella recycle drive: Set up collection points for old or damaged umbrellas and repurpose the materials into new eco-friendly products such reusable shopping bags, rain ponchos, or even outdoor furniture, promot- ing sustainability and circular economy practices. 5. Umbrella storytelling circle: Arrange a storytelling circle in park or outdoor setting where participants gather under a cluster of open umbrellas to share personal anecdotes, fairy tales, or local legends, fostering sense of community and connection through shared narratives. 6. Umbrella floating art installation: Create floating art installation using opened umbrellas attached to inflatable platforms on a lake or pond, showcasing innovative and whimsical designs that interact with the water and surrounding environment. 7. Umbrella language exchange picnic: Host a multilingual picnic where participants use umbrellas labeled with different languages conversation starters to facilitate cultural exchange and language learning in a relaxed and festive setting. 8. Umbrella weather station: Install smart technology in umbrellas equipped with sensors that provide real-time weather data and forecasts when opened, helping users stay in- formed and prepared for changing weather conditions while promoting awareness of climate patterns. 9. Umbrella wellness retreat: Arrange wellness retreat in natural setting where participants engage in mindfulness practices, yoga sessions, and wellness workshops under open umbrellas, offering serene and rejuvenating experience surrounded by nature. 10. Umbrella sustainability exhibition: Curate an exhibition showcasing innovative and sustainable designs that repurpose umbrellas as eco-friendly solutions for various challenges, such as shelter, mobility, or art, to raise awareness about the importance of environmental conservation and creativity. I am excited to see the creativity and diversity of ideas that our team can come up with as we push the boundaries of innovation and sustainability in reimagining the uses of umbrellas. Let's continue to think boldly and collectively spark inspiration for a greener and more imaginative future!
31

---

Published as a conference paper at COLM 2024
Round 4 User Prompt
You are a Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth. Remember to claim your role in the beginning of each conversation. These are the solutions to the problem from other agents: One agent solution: "'Here are some more creative uses for umbrellas: 1. Umbrella poetry installation: Create public art project where open umbrellas display poems or verses written by local poets, adding a poetic touch to urban spaces. 2. Umbrella scavenger hunt: Organize a scavenger hunt where participants follow clues to find hidden umbrellas around the city, each leading to different surprise or activity. 3. Umbrella storytelling circle: Gather a group under a canopy of open umbrellas to share stories, folklore, or personal anecdotes, fostering connections and community through the art of story- telling. 4. Umbrella juggling performance: Showcase a juggling act that incorporates spinning umbrellas in a dynamic and visually captivating performance, blending circus skills with creative flair. 5. Umbrella constellation guide: Design umbrellas with constellation patterns on the canopy and LED lights that illuminate the stars, offering a portable guide to stargazing
cloudy nights. 6. Umbrella street art project: Collaborate with local artists to transform open umbrellas into large-scale street art installations, adding colorful and unexpected twist to urban landscapes. 7. Umbrella sound sculpture: Create an interactive sound sculpture by attaching bells, chimes, musical instruments to the spokes of an open umbrella, allowing visitors to create melodies or harmonies with a touch. 8. Umbrella storytelling parade: Organize parade where participants carry open umbrellas adorned with symbols or images that narrate a collective story or theme, captivating spectators with visual storytelling on the move. 9. Umbrella puppet theater: Set up puppet theater under a large umbrella canopy, performing puppet shows for children and adults alike, combining the magic of puppetry with the shelter of a canopy. 10. Umbrella dance performance: Choreograph dance routine that incorporates umbrellas as props, creating visually striking movements and formations that play with light and shadow, rhythm, and expression. Let's keep exploring the artistic possibilities of umbrellas and how they can be used to inspire creativity, bring people together, and add touch of magic to everyday experiences;" One agent solution: "As Futurist specializing in Emerging Technologies and Future Scenarios, am thrilled to collabo- rate with my creative teammates and further expand upon the innovative uses for umbrellas. Let's continue to push the boundaries of imagination and explore even more unconventional ideas for umbrellas: 1. Virtual umbrella art gallery: Create artwork in unique way. Umbrella weather
digital platform where artists worldwide can showcase umbrella-theme virtual gallery, blending art and technology forecasting system: Develop smart umbrellas equipped with sensors that change color or patterns to reflect the forecasted weather conditions, providing visual representation of upcoming weather changes. 3. Umbrella interactive storytelling experience: Design umbrellas with augmented reality features that allow users to participate in interactive storytelling experiences by scanning different sections of the canopy with a mobile device. 4. Umbrella community art project: Initiate a community-driver initiative where residents contribute to painting large umbrella mural in a public space, fostering creativity, unity, and a sense of pride in the neigh- borhood. 5. Umbrella innovation challenge: Host competition inviting inventors and designers to develop groundbreaking concepts for next-generation umbrellas, encouraging out-of-the-box thinking and technological advancements in umbrella design. 6. Umbrella wellness retreat: Organize outdoor wellness retreats where par- ticipants engage in activities like yoga, meditation, and mindfulness practices under canopy of colorful umbrellas, promoting relaxation and connection with nature. 7. Umbrella music festival stage: Construct a stage made entirely of open umbrellas for music festival, creating a visually captivating and acoustically dynamic environ-
32

---

Published as a conference paper at COLM 2024
ment for performers and attendees. 8. Umbrella sustainability campaign: Launch campaign advocating for eco-friendly umbrella designs that use recyclable materials and biodegradable components, raising awareness about environmental impact and promoting sustainable practices. 9. Umbrella innovation lab: Establish a collabora- tive space where researchers, engineers, and artists work together to experiment with cutting-edge technologies and materials to develop revolutionary concepts for future umbrellas. 10. Umbrella social impact project: Partner with organizations serving vulnerable communities to distribute umbrellas equipped with essential resources such as solar-powered lights, water filtration systems, or emergency communication devices, improving access to critical services during emergencies or challenging situations. Let's continue to think innovatively and inspire each other to envision future where umbrellas transcend their traditional function and become catalysts for creativity, connectivity, and positive change in society." " One agent solution: "1. Umbrella Social Distancing Assistant: Develop smart umbrellas equipped with sensors that emit gentle reminders or alerts to maintain appropriate social distance in crowded areas. 2. Umbrella Art Class: Organize an outdoor art class where partic- ipants umbrellas as canvases, creating unique and weather-resistant artwork. 3. Umbrella Sound Bath Experience: Transform umbrellas into sound instruments with embedded speakers that play soothing soundscapes for relaxation and mindfulness sessions. 4. Umbrella Farmer's ss Market Stall: Convert umbrellas into vibrant market stalls at farmer's 's markets, showcasing local produce and goods in colorful and sustainable way. 5. Umbrella Ice Cream Shop: Establish whimsical ice cream shop where customers can order and enjoy their frozen treats under the shade of creatively decorated umbrellas. 6. Umbrella Poetry Slam: Host a poetry slam event where participants perform spoken word poetry under a canopy of open umbrellas, adding poetic touch to outdoor gatherings. 7. Umbrella Escape Room: Design an outdoor escape room experience where participants solve puzzles and clues hidden under the canopies of various umbrellas to unlock mysteries. 8. Umbrella Silent Disco: Organize a silent disco event where participants dance and listen to music through wireless headphones under canopy of colorful illuminated umbrellas. 9. Umbrella Themed Restaurant: Create a themed restaurant where each table is adorned with unique umbrella decor and diners can enjoy their meals in vibrant and imaginative setting. 10. Umbrella Adventure Quest: Develop an interactive scavenger hunt or geocaching game that leads participants to hidden treasures clues located under different umbrellas throughout a city or park. Let's continue to spark innovation and embrace the unconventional uses of umbrellas to enhance our experiences and create memorable moments in the future;" What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of as many creative and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can
Round 4 Agent Response
As an Environmentalist specializing in Sustainability and Environmental Health, I am thrilled to collaborate with my creative teammates in exploring even more inven- tive and imaginative uses for umbrellas. Let's delve into a realm of innovation and creativity with these unique ideas: Umbrella Nature Sound Sanctuary: Create an outdoor sanctuary where open umbrellas play recordings of soothing natural sounds like forest rustling, bird songs, and ocean waves, providing a serene and immersive experience for visitors seeking relaxation and connection with nature. 2. Umbrella Rco-Fashion Show: Host an eco-conscious fashion show featuring outfits made from recycled umbrella materials, showcasing sustainability and creativity in the world of fashion and design. 3. Umbrella Upcycled Garden Sculptures: Repurpose old umbrellas into whimsical garden sculptures by shaping the fabric into floral or ani-
33

---

Published as a conference paper at COLM 2024
mal shapes and using the umbrella frame as the base, adding art and sustainability to outdoor spaces. 4. Umbrella Virtual Reality Experience: Develop an interactive virtual reality experience where users explore digital worlds and immersive environ- ments while holding an umbrella as controller, merging technology, and everyday objects in a cutting-edge way. 5. Umbrella Botanical Art Workshop: Arrange a work- shop where participants create botanical art pieces by pressing flowers and leaves between the fabric of open umbrellas, turning them into colorful and natural works of art. 6. Umbrella Sustainable Picnic Table: Design a picnic table with an umbrella canopy made from solar panels that generate electricity to power small devices or charge electronics, offering sustainable and functional dining experience in outdoor settings. 7. Umbrella Community Mural Project: Engage local residents in painting mural on large umbrella canopy that represents the diverse stories, cultures, and identities of the community, fostering unity and creativity through collective art-making. 8. Umbrella Biophilic Office Space: Introduce biophilic design elements in office environments by suspending open umbrellas from ceilings and filling them with lush green plants, bringing nature indoors and enhancing the workspace with natural beauty and tranquility. 9. Umbrella Wellness Carnival: Organize wellness carnival where attendees participate in holistic activities like yoga, meditation, and sound therapy under canopy of interactive umbrellas that emit calming scents, sounds, and lighting effects, promoting well-being and relaxation. 10. Umbrella Marine Life Conservation Initiative: Launch marine life conservation campaign by creating awareness through umbrella-themeds educational events and workshops that highlight the impact of plastic pollution on marine ecosystems, inspiring action and advocacy for ocean conservation. Let's continue to embrace innovation and creativity reimagine the uses of umbrellas in ways that promote sustainability, artistic expression, community engagement, and positive societal impact. Together, we can inspire positive change and contribute to a brighter and more eco-friendly future for our planet!
Round 5 User Prompt
You Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is to champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to a healthier earth. Remember to claim your role the beginning of each conversation. These are the solutions to the problem from other agents; One agent solution: HHere are some additional creative uses for umbrellas: 1. Umbrella Art Therapy Session: Host an art therapy session where participants express their emotions and stories through painting on umbrella canopies, creating therapeutic and visually striking experience. 2. Umbrella Networking Event: Organize a networking event where participants carry personalized umbrellas with their skills, interests, or goals written them, sparking conversations and connections in creative way. 3. Umbrella Cul- tural Exchange Fair: Host cultural fair where attendees can visit different "umbrella booths" representing diverse cultures, each with interactive displays, traditional crafts, and educational information. 4. Umbrella Upcycling Workshop: Conduct workshop where participants learn to repurpose old umbrellas into fashion ac- cessories, home decor items, or practical gadgets, promoting sustainability and creativity. 5. Umbrella Dance Party: Host a dance party under a sea of colorful open umbrellas, with each participant spinning and twirling their umbrella to the beat of the music for a visually mesmerizing experience. Umbrella Storybook Reading: Invite children to gather under a large umbrella canopy for storybook reading session, where the colorful umbrellas serve as visual aids and bring the stories to life. 7. Umbrella Sound Healing Circle: Create sound healing circle where partic- ipants sit in a circle under open umbrellas while playing singing bowls, gongs, or other instruments to create soothing and immersive sonic experience. 8. Umbrella
34

---

Published as a conference paper at COLM 2024
Innovation Symposium: Organize a symposium where experts, inventors, and cre- atives present groundbreaking ideas, technologies, and designs related to umbrellas, inspiring conversations and collaboration for future innovations. 9. Umbrella Self- Expression Parade: Coordinate parade where participants decorate their umbrellas with personal messages, artwork, or symbols that reflect their individuality and values, turning the procession into vibrant display of self-expression. 10. Umbrella Mindfulness Retreat: Host a mindfulness retreat in a serene natural setting where participants engage in meditation, yoga, and relaxation activities under umbrellas, promoting mental wellness and harmony with nature. Let's continue to explore the multifaceted ways which umbrellas can be used creatively and symbolically to enhance human experiences, foster connection, and inspire imagination." " One agent solution: "' As Futurist specializing in Emerging Technologies and Future Scenarios, I am excited to build upon the fantastic ideas shared by my fellow agents and explore even more innovative and imaginative uses for umbrellas. Let's continue to think outside the box and envision a future where umbrellas play diverse and creative roles: 1. Umbrella Urban Farming System: Develop umbrellas integrated with hydroponic or aquaponic systems for vertical farming, allowing communities to grow fresh produce in urban settings while providing shade and rain protection. 2. Umbrella Data Visualization Project: Equip umbrellas with LED lights that change colors or patterns based on real-time data such as air quality, pollution levels, or community engagement metrics, turning them into interactive public art installa- tions that raise awareness. 3. Umbrella Sky Writing Showcase: Create large-scale spectacle by synchronizing drones equipped with umbrellas that release colored smoke trails to form intricate patterns or messages in the sky, combining art and technology for stunning visual displays. 4. Umbrella Mobile Library: Transform oversized umbrellas into portable bookshelves on wheels, hosting pop-up libraries in public spaces where people can browse, borrow, and exchange books, fostering love for reading and community engagement. 5. Umbrella Green Energy Solution: Develop innovative umbrellas with built-in solar panels and wind turbines to gener- ate clean energy for charging devices, powering lights, or contributing to sustainable energy grids, promoting eco-conscious practices. 6. Umbrella Virtual Reality Station: Create an immersive VR experience where users can step under high-tech umbrella canopy to be transported to digital worlds or interactive narratives, blurring the lines between physical and virtual reality. 7. Umbrella Marine Conservation Project: Launch a campaign to repurpose old umbrellas into artificial reefs or structures that provide habitats for marine life, promoting biodiversity and environmental conservation in coastal areas. 8. Umbrella Cosplay Convention: Host a cosplay event where participants craft elaborate costumes using umbrellas as a central design element, showcasing futuristic and fantastical characters inspired by the versatility of umbrellas. 9. Umbrella Disaster Relief Kit: Equip emergency response teams with multifunctional umbrellas containing essential supplies such as first aid kits, water purification systems, communication devices, and emergency shelters to assist communities during natural disasters or crises. 10. Umbrella Art Therapy Workshop: Organize art therapy sessions under a canopy of open umbrellas where participants express emotions, explore creativity, and engage in therapeutic activities such as painting, sculpting, or storytelling, fostering mental wellness and self-expression. Let's continue to cultivate our creativity and explore the vast potential for umbrellas to shape the future in imaginative and impactful ways. Together, we can pave the way for a innovative and inspiring future with umbrellas at the forefront of innova- tion." " One agent solution: "'. Umbrella Eco-Art Installation: Create large-scale art installation where open umbrellas are filled with sustainable materials like recycled paper, plastic, or fabric, symbolizing environmental awareness and artistic expres- sion. 2. Umbrella Augmented Reality Tour: Develop an augmented reality app that transforms open umbrellas into interactive guides for virtual tours of historical sites or landmarks, bringing the past to life through a fusion of technology and art. 3. Umbrella Healing Garden: Establish healing garden with open umbrellas serving as sheltered spaces for meditation, reflection, and healing practices, offering
35

---

Published as a conference paper at COLM 2024
a sanctuary for individuals seeking mental and emotional wellness. 4. Umbrella Geocaching Adventure: Organize a geocaching adventure where participants follow clues hidden under umbrellas to discover hidden treasures or messages, combining the thrill of discovery with outdoor exploration. 5. Umbrella Digital Art Gallery: Curate a digital art gallery showcasing uubreell-themee artworks created by digital artists, exploring the intersection of technology, creativity, and visual storytelling. 6. Umbrella Empowerment Workshop: Host empowerment workshops where partic- ipants decorate umbrellas with empowering messages and symbols, encouraging eelf-expression and positive affirmations. 7. Umbrella Sustainability Hackathon: Organize a hackathon where innovators collaborate to develop sustainable solutions using umbrellas, such as upcycling old umbrellas into utility items or creating eco- friendly designs for future use. 8. Umbrella Connection Carnivale: Transform public space into a carnival-themed event with open umbrellas serving as colorful booths for games, performances, and interactive experiences that bring people to- gether in celebration. 9. Umbrella Wellness Expo: Launch wellness expo where attendees can explore holistic practices under open umbrellas dedicated to different wellness modalities like yoga, meditation, sound healing, and aromatherapy. 10. Um- brella Time Capsule Project: Encourage individuals to fill umbrellas with personal mementos, messages, and predictions for the future, sealing them as time capsules to be opened and shared with future generations, preserving memories and aspirations for posterity. Let's embrace the limitless potential of umbrellas versatile tools for creativity, connection, and transformation in shaping a future that celebrates innovation and imagination." " What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of as many creative and diverse uses for Umbrella as possible. You are in group discussion with other teammates; as a result, answer as diversely and creatively as you can. This is the last round of the discussion, please finalize and present list of creative answers. Please list the final response in ... 2. ... 3. and so on.
Round 5 Agent Response
Umbrella Art Therapy Session: Host an art therapy session under open umbrellas where participants express emotions and stories through painting the canopy, cre- ating a therapeutic and visually striking experience. 2. Umbrella Networking Event: Organize networking event where personalized umbrellas spark conversations and connections with skills, interests, or goals written on them, promoting creative interactions. 3. Umbrella Cultural Exchange Fair: Host fair with different "umbrella booths" representing diverse cultures through interactive displays, traditional crafts, and educational information, fostering cultural appreciation. 4. Umbrella Upcycling Workshop: Conduct a workshop teaching participants to repurpose old umbrellas into fashion accessories, home decor items, or gadgets, encouraging sustainability and creativity. 5. Umbrella Dance Party: Throw dance party under colorful open umbrellas, where participants twirl their umbrellas to the music for a visually mes- merizing experience. 6. Umbrella Storybook Reading: Invite children for a storybook reading under a large umbrella canopy, using the umbrellas as visual aids to bring stories to life. 7. Umbrella Sound Healing Circle: Create circle under open umbrel- las for sound healing session with singing bowls or instruments, offering a soothing and immersive sonic experience 8. Umbrella Innovation Symposium: Organize symposium where experts present ideas, technologies, and designs related to umbrel- las, inspiring conversations and collaborations for future innovations. Umbrella Self-Expression Parade: Coordinate parade with participants decorating umbrellas with personal messages or artwork, turning the procession into vibrant display of individual expression. 10. Umbrella Mindfulness Retreat: Host mindfulness retreat in a natural setting where participants engage in relaxation activities under umbrellas, promoting mental wellness and connection with nature.
36

---

Published as a conference paper at COLM 2024
K.2
Settings and Final Responses of Each Agent
Agent 1 Environmentalist Initiation Prompt
You are Environmentalist whose specialty is Sustainability and Environmental Health. As an Environmentalist, your mission is champion eco-friendly solutions that promote sustainability and protect our planet. You guide us to consider the environmental impact of our ideas, pushing for innovations that contribute to healthier earth. Remember to claim your role in the beginning of each conversation. Initiate a discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, different. Present list of many creative and diverse uses for Umbrella as possible. You are in group discussion with other teammates; as result, answer as diversely and creatively you can.
Agent 2 Creative Professional Initiation Prompt
You are a Creative Professional whose specialty is Aesthetics, Narratives, and Emo- tions. As Creative Professional, your artistic sensibility and mastery of narrative and emotion infuse our projects with beauty and depth. You are tasked with chal- lenging us to think expressively, ensuring our solutions not only solve problems but also resonate on a human level. Remember to claim your role in the beginning of each conversation. Initiate a discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is to come with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of many creative and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can
Agent 3 Futurist Initiation Prompt
You are Futurist whose specialty Emerging Technologies and Future Scenarios. As a Futurist, you inspire us to think beyond the present, considering emerging technologies and potential future scenarios. Your role to challenge us to envision the future impact of our ideas, ensuring they are innovative, forward-thinking, and ready for the challenges ahead. Remember to claim your role in the beginning of each conversation. Initiate discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, or different. Present list of many creative and diverse uses for Umbrella possible. You are in group discussion with other teammates; as a result, answer as diversely and creatively as you can
Agent 4 Futurist-2 Initiation Prompt
You are a Futurist whose specialty is Emerging Technologies and Future Scenarios. As a Futurist, you inspire us to think beyond the present, considering emerging technologies and potential future scenarios. Your role is to challenge us to envision the future impact of our ideas, ensuring they are innovative, ready for the challenges ahead. Remember to claim your role in the beginning of each conversation. Initiate discussion with others to collectively complete the following task: What are some creative use for Umbrella? The goal is to come up with creative ideas, which are ideas that strike people as clever, unusual, interesting, uncommon, humorous, innovative, different. Present list of many creative
whhaaddt and
37

---

Published as a conference paper at COLM 2024
and diverse uses for Umbrella as possible. You are in a group discussion with other teammates; as a result, answer as diversely and creatively as you can
Agent 1 Environmentalist- Final Response
Host an art therapy session under open umbrellas where participants express emotions and stories through painting on the canopy, cre- ating a therapeutic and visually striking experience. Umbrella Networking Event: Organize networking event where personalized umbrellas spark conversations and connections with skills, interests, or goals written on them, promoting creative interactions. 3. Umbrella Cultural Exchange Fair: Host fair with different "umbrella booths" representing diverse cultures through interactive displays, traditional crafts, and educational information, fostering cultural appreciation. 4. Umbrella Upcycling Workshop: Conduct a workshop teaching participants to repurpose old umbrellas into fashion accessories, home decor items, or gadgets, encouraging sustainability and creativity. 5. Umbrella Dance Party: Throw a dance party under colorful open umbrellas, where participants twirl their umbrellas to the music for visually mes- merizing experience. 6. Umbrella Storybook Reading: Invite children for a storybook reading under large umbrella canopy, using the umbrellas as visual aids to bring stories to life. 7. Umbrella Sound Healing Circle: Create circle under open umbrel- las for sound healing session with singing bowls or instruments, offering soothing and immersive sonic experience. 8. Umbrella Innovation Symposium: Organize symposium where experts present ideas, technologies, and designs related to umbrel- las, inspiring conversations and collaborations for future innovations. 9. Umbrella Self-Expression Parade: Coordinate a parade with participants decorating umbrellas with personal messages artwork, turning the procession into vibrant display of individual expression. 10. Umbrella Mindfulness Retreat: Host mindfulness retreat in a natural setting where participants engage in relaxation activities under umbrellas, promoting mental wellness and connection with nature.
Umbrella Art Therapy Session:
Agent 2 Creative Professional Final Response
Create serene setting where participants sit under open umbrellas and engage in sound healing sessions with instruments like singing bowls and chimes, promoting relaxation and well-being through immersive auditory experiences. 2. Umbrella Botanical Art Workshop: Host workshops where participants press flowers and leaves between umbrella fabric create botanical art pieces, merging nature and creativity in a unique and beautiful way. 3. Um- brella Virtual Reality Station: Develop immersive virtual reality experiences where users interact with digital worlds under the canopy of high-tech umbrellas, blend- ing cutting-edge technology with everyday objects for captivating simulations. Umbrella Urban Farming System: Implement hydroponic or aquaponic systems within umbrellas to enable vertical farming in urban environments, combining food production with shade provision and sustainability. 5. Umbrella Sky Writing Show- case: Coordinate drone displays equipped with umbrellas that release colored smoke trails to form intricate patterns in the sky, creating captivating visual spectacles that blend artistry and aerial technology. 6. Umbrella Eco-Fashion Show: Organize eco- conscious fashion events featuring outfits made from recycled umbrella materials, exemplifying sustainable design practices and showcasing innovation in fashion. 7. Umbrella Marine Conservation Initiative; Raise awareness about marine life conser- vation by repurposing old umbrellas as artificial reef structures that provide habitats for ocean ecosystems, promoting biodiversity and environmental stewardship. Umbrella Data Visualization Project: Enhance public spaces with LED-equipped umbrellas that convey real-time data like pollution levels or air quality through color changes patterns, turning them into interactive art installations that inform and
1. Umbrella Sound Healing Circle:
38

---

Published as a conference paper at COLM 2024
engage communities. Umbrella Sound Bath Experience: Curate relaxing sound therapy sessions where umbrellas with embedded speakers play calming sound- scapes, offering immersive sonic experiences for stress relief and mental wellness. 10. Umbrella Cosplay Convention: Host cosplay events featuring elaborate costumes designed with umbrellas as a focal point, allowing participants showcase futuristic and imaginative characters inspired by the versatility and creativity of umbrellas. Let's celebrate the multifaceted potential of umbrellas as versatile tools for artis- tic expression, technological integration, environmental advocacy, and community engagement, paving the way for a future where innovation and creativity shine through everyday objects like umbrellas.
Agent 3 Futurist Final Response
Umbrella Nature Sound Sanctuary: Create an outdoor sanctuary where open umbrellas play recordings of soothing natural sounds like forest rustling, bird songs, and ocean waves, providing a serene and immersive experience for visitors seeking relaxation and connection with nature. 2. Umbrella Ecco-Fashion Show: Host eco-conscious fashion show featuring outfits made from recycled umbrella mate- rials, showcasing sustainability and creativity in the world of fashion and design. 3. Umbrella Upcycled Garden Sculptures: Repurpose old umbrellas into whimsi- cal garden sculptures by shaping the fabric into floral or animal shapes and using the umbrella frame as the base, adding art and sustainability to outdoor spaces. 4. Umbrella Virtual Reality Experience: Develop an interactive virtual reality ex- perience where users explore digital worlds and immersive environments while holding an umbrella as a controller, merging technology, and everyday objects cutting-edge way. 5. Umbrella Botanical Art Workshop: Arrange workshop where participants create botanical art pieces by pressing flowers and leaves between the fabric of open umbrellas, turning them into colorful and natural works of art. 6. Umbrella Sustainable Picnic Table: Design picnic table with an umbrella canopy made from solar panels that generate electricity to power small devices or charge electronics, offering sustainable and functional dining experience in outdoor set- tings. 7. Umbrella Community Mural Project: Engage local residents in painting mural on large umbrella canopy that represents the diverse stories, cultures, and identities of the community, fostering unity and creativity through collective art-making. 8. Umbrella Biophilic Office Space: Introduce biophilic design elements in office environments by suspending open umbrellas from ceilings and filling them with lush green plants, bringing nature indoors and enhancing the workspace with natural beauty and tranquility. 9. Umbrella Wellness Carnival: Organize wellness carnival where attendees participate in holistic activities like yoga, meditation, and sound therapy under a canopy of interactive umbrellas that emit calming scents, sounds, and lighting effects, promoting well-being and relaxation. 10. Umbrella Marine Life Conservation Initiative: Launch a marine life conservation campaign by creating awareness through umbrella-themed educational events and workshops that highlight the impact of plastic pollution on marine ecosystems, inspiring action and advocacy for ocean conservation. Let's embrace innovation and creativity to reimagine the uses of umbrellas in ways that promote sustainability, artistic expres- sion, community engagement, and positive societal impact. Together, we can inspire positive change and contribute to brighter and more eco-friendly future for our planet!
Agent 4 Futurist-2 - Final Response
1. Umbrella Bio-Printing Studio: Establish bio-printing studio where umbrellas are used as a canvas for creating intricate cellular structures and biological art pieces, merging art, science, and technology. 2. Umbrella Glow-in-the-Dan Night Market:
39

---

Published as a conference paper at COLM 2024
Organize a night market where vendors set up booths under neon-lit umbrellas that glow in the dark, creating vibrant and futuristic shopping experience. 3. Um- brella Interactive Playground: Design an interactive playground where children play and explore under oversized umbrellas equipped with interactive light and sound features that respond movement and touch. 4. Umbrella Mindfulness Labyrinth: Construct labyrinth maze with open umbrellas as guiding markers, encouraging visitors to walk mindfully, reflect, and navigate a tranquil path for stress relief and introspection. 5. Umbrella Renewable Energy Showcase: Host exhibition featuring volar-poweres umbrellas, wind rindstesistant designs, and innova- tive mergy-harvesting technologies, highlighting sustainable solutions for energy generation and environmental conservation. 6. Umbrella Tech Innovation Lab: Es- tablish collaborative space where tech enthusiasts and designers experiment with embedding sensors, AI, and AR/VR technologies into umbrellas to create smart, interactive, and futuristic accessories. 7. Umbrella Eco-Friendly Transportation Hub: Create a sustainable transportation hub with bike-sharing stations under shade- providing umbrellas equipped with air quality monitors, integrated bike repair tools, and mobile charging stations. 8. Umbrella Sensory Exploration Garden: Develop sensory garden with themed zones under specialized umbrellas that stimulate differ- ent senses like touch, smell, sight, and sound, offering multi-sensory experience for relaxation and sensory development. 9. Umbrella Disaster Response Center: Establish mobile disaster response units equipped with rapid-deplo umbrellas that provide emergency shelter, first aid supplies, communication devices, and support services for communities during crises and emergencies. 10. Umbrella Cultural Exchange Festival: Host a cultural exchange festival where participants showcase diverse traditions and performances under eygionally-themmed umbrellas, celebrating heritage, unity, and global interconnectedned conclude this discussion with vision of a future where umbrellas serve as catalysts for sustainability, innovation, art, community engagement, well-being, and cultural celebration, shaping a world where creativity and imagination thrive.
through art, music, and dance. Let's
40


Reviews:
[{"review_id": "at5sDBtuBt", "reviewer": "Reviewer_NP7f", "summary": "This paper proposes an approach to enhancing the creativity of large language models (LLMs) by facilitating multi-agent discussions and incorporating role-playing elements. The key ideas of a three-phase discussion framework and role assignment are well-motivated and explained. The authors conduct evaluations across 4 creativity benchmarks, comparing their proposed method against baselines. The results demonstrate improvements in some metrics of TTCT like originality and elaboration, suggesting that the proposed approach can indeed induce more creative and novel responses from LLMs. Overall, this work presents a relatively convincing approach to an important problem in AI, with findings that are likely to stimulate further research in this area.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> Role selection & generation\n\nWe made the first attempt to integrate multi-LLM discussion and role-play; hence, we propose an automated pipeline using GPT-4 to generate diverse and detailed role descriptions motivated by creative thinking (role-storming). We will include the dialog with GPT-4 to generate roles in the revision. We agree with the reviewer that systematically exploring diverse roles is a promising direction.\n\n> Qualitative analysis\n\nPlease refer to the \\[Qualitative analysis: what happens inside the model\\] response to Reviewer nFPB.\n\n> Theoretical foundations of multi-agent discussion and role-playing\n\nWe are motivated by [1-4], suggesting that discussing with individuals from diverse backgrounds and perspectives can enhance creativity. Also, [5] shows that role-playing helps participants think from various perspectives, leading to novel solutions. We will revise the paper to discuss these works.\n\n- [1] Han, et al. Is group work beneficial for producing creative designs in STEM design education? International Journal of Technology and Design Education, 2022.\n- [2] Paulus and Nijstad, eds. Group Creativity: Innovation Through Collaboration. Oxford University Press, 2003.\n- [3] McGrath. Groups: Interaction and Performance. Englewood Cliffs, NJ: Prentice-Hall, 1984.\n- [4] Sutton and Hargadon. Brainstorming groups in context: Effectiveness in a product design firm. Administrative Science Quarterly, 1996.\n- [5] Karwowski and Soszynski. How to develop creative imagination?: Assumptions, aims and effectiveness of Role Play Training in Creativity (RPTC). Thinking Skills and Creativity 3.2, 2008.\n\n> Tasks chosen & why AUT? GPT4 is good at it\n- Our experiments use GPT-3.5, which has limited performance on AUT, and our framework can improve it. \n- We additionally evaluated single-agent GPT-4 on AUT, which achieves 3.83 in Originality and 3.74 in Elaboration, outperforming single-agent GPT-3.5 (O: 3.47, E: 3.08) aligning with the papers mentioned by the reviewer, yet underperforming ours with GPT-3.5 (O: 4.44, E: 4.22).\n- While AUT is widely adopted in creativity evaluation, we agree that merely using AUT may not be sufficient; hence, our evaluations considered three additional benchmarks, Instances, Similarities, and Scientific, to cover distinct aspects of creativity.\n\nWe will revise the paper to include the above discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Clever use of multi agent discussion and roleplaying in open domain NLG task\n2) Good evaluation with LLM and humans and also showing correlation between each", "reasons_to_reject": "While the paper presents an interesting approach and promising results, there are a few key concerns that should be addressed before it is ready for publication\n\n1) The role-playing component is a novel contribution, but the paper lacks deeper analysis on the specific roles used, how they were chosen/generated, and whether these roles may introduce unintended biases or perspectives that could skew the results.The role-playing aims to induce diverse perspectives, but the paper does not discuss how the chosen roles represent or account for diversity across different demographics, cultures, domains, etc. A more critical examination of this aspect is needed.\n\n2) The task chosen are relatively old and sentence level so not very clear if this approach would be generalizable for other creative tasks such as design, ideation, creative writing, generating metaphors etc\n\n3) The results are primarily quantitative, but a deeper qualitative analysis examining the generated responses, the nature of conversations/discussions, and specific strengths/weaknesses could provide valuable insights.\n\n4) Theoretical grounding: While the motivations are clear, the paper could benefit from stronger theoretical foundations explaining why multi-agent discussion and role-playing are expected to enhance LLM creativity from a cognitive/psychological perspective.", "questions_to_authors": "Why AUT ? There is mounting evidence that GPT4 is good at this so to benchmark you should focus on complex creative tasks\nhttps://arxiv.org/abs/2303.12003\nhttps://news.uark.edu/articles/69688/ai-outperforms-humans-in-standardized-tests-of-creative-potential", "extra_scores": {}}, {"review_id": "1FGy8VKjIy", "reviewer": "Reviewer_nFPB", "summary": "The presented approach gets much better results on key measures of creativity compared to SOTA.\n\nThis paper is a well-written presentation of the novel LLM Discussion approach to generating creative responses on established creativity and scientific innovation benchmark tests. \n\nThe approach is a highly original simulation of \"collaborative discussions with diversified peers\" that involves (i) a three-phase, initiation-discussion-convergence framework for multi-turn dialogues designed to elicit diverse and creative responses from (ii) LLM-based agents, each of what are assigned distinct roles and are directed to generate answers building on the others' responses.\n\nThis framework, (i), is compared against single-LLM and multi-LLM approaches and outperforms these previous methods on several creativity metrics.\n\nThe role-playing LLM-based agents, (ii), cleverly leverages the ability of LLMs to imitate very different personalities in order to generate more diversity of ideas.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> The specific roles used\n\nWe agree with the reviewer that systematically exploring diverse roles is a promising direction. This work makes the first attempt to integrate multi-LLM discussion and role-play; hence, we focused on proposing an automated pipeline using GPT-4 to generate diverse and detailed roles motivated by creative thinking (role-storming).\n\n> Qualitative analysis: what happens inside the model\n\nWe appreciate the reviewer’s suggestion. We will revise the paper to include more qualitative results and analyses from three aspects:\n\n- Collaborative dynamics: Our framework stimulates a more collaborative tone, e.g., ”building on those ideas, …,” while LLM Debate does not follow up on each other’s ideas.\n- Role-specific responses: LLMs can stay in character, e.g., in an AUT scenario of an umbrella, the Futurist proposes integrating it with VR technology, while the Environmentalist suggests using it as a shelter for wildlife. In contrast, the baselines tend to suggest more general uses.\n- Specific details: LLM Discussion outperforms baselines in Elaboration by coming up with detailed and specific answers. When asked ”Name all the things you can think of that are used in culture” from Instances, ours answers Tattoos, Digital art, and Ethical fashion, while the baselines answer Art, Music, and Clothing.\n\n> Boost Fluency and Flexibility\n\nAs suggested by the reviewer, we additionally experimented with asking the LLM Discussion agent to generate as many answers as possible (AMAP) in the convergence phase, dubbed Ours+AMAP. The results below show that the prompt can increase Fluency and Flexibility scores while maintaining Originality and Elaboration. \n\n|Benchmark|Method|Originality|Elaboration|Fluency|Flexibility|\n|-|-|-|-|-|-|\n|AUT|Ours|4.44|4.22|9.19|9.68|\n||Ours+AMAP|4.40|4.23|9.38|9.58|\n|Instances|Ours|3.65|2.20|16.88|11.11|\n||Ours+AMAP|3.65|2.22|22.06|11.08|\n|Similarities|Ours|3.29|2.52|7.27|8.14|\n||Ours+AMAP|3.27|2.55|10.89|11.52|\n|Scientific|Ours|3.95|3.47|5.58|5.91|\n||Ours+AMAP|3.89|3.32|7.63|8.08|\n\n> Convergence phase\n\nAs the reviewer pointed out, the term \"finalize\" in our convergence prompt serves as an explicit prompt to conclude, while \"present a list\" facilitates parsing. The chat logs show that the number of answers is reduced after convergence since some similar answers are combined and some are filtered out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Ambitious objective to simulate \"collaborative discussions with diversified peers\" with novel design of 3-phase framework for discussions by role-playing LLM agents.\n* Exciting results with strong gains over previous methods of generating creative answers.\n* Clear delineation of where this approach contrasts with state of the art.\n* Solid empirical evaluation methodology with established tests and metrics, engaging both automated and human judgements, as well as documented ablation tests, explaining development of specialized prompts, # discussion rounds and # LLM agents", "reasons_to_reject": "* The authors do not discuss going beyond the role-playing agents as artificial people, and address other ways of coming up with better answers,.\n\n* The paper does not address what is actually happening inside the model, that would also give deeper insights.", "questions_to_authors": "* The space of possible prompts and  ways of combining the results of prompts is so broad that it seems unlikely these results are anywhere near the best that could be obtained with present models. Q1: Why, for instance, are the particular roles of \"visionary millionaire, startup founder, social entrepreneur, creative professional, customer, environmentalist, digital nomad, industry insider, futurist\" (which sounds like a list of people you might encounter at a party in Silicon Valley) chosen instead of, say, homemaker, mathematician, janitor, and professional actor? Or, say, all 16 Meyers-Briggs personality types? Or people from 20 diverse countries or time periods? There's no justification for this and no way of knowing what would be best a priori. At the moment there is so much low-hanging fruit that it is easy to come up with new approaches that get big gains, but it would be helpful to find some automated way of searching the space of possibilities more thoroughly.\n\n* Q2: What might help explain why fluency and flexibility scores are lower for LLM Discussion than LLM Debate, even though these matter less than originality and elaboration as measures of creativity. Perhaps there would be some simple way to alter the prompts to boost these factors as well?\n\n* Q3: how is the prompt in the convergence phase leading to \"convergence\" as opposed to simply enumerating answers from prior phase, given instruction to  \"present a list\"?  Did you experiment with an explicit prompt to create a summary or draw a conclusion?", "extra_scores": {}}, {"review_id": "3gJupmvDxn", "reviewer": "Reviewer_6qWd", "summary": "The paper presents a method called LLM Discussion, which is a multi-turn multi-LLM method to attempt to extract more creative responses out of LLMs on common creativity \"brainteasers\" such as Alternative Uses. The work experiments on four types of tasks, and evaluates different conversational methods to extract creative answers. The LLM Discussion method is found to be more creative both through a robust human evaluation, and using an LLM as a judge, which is found to correlate satisfactorily with human judgments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n> Benchmarks\n\nWhile our evaluation uses existing tasks for human creativity, we agree that developing LLM-customized creativity benchmarks and metrics is a promising direction. We will discuss it in the revision.\n\n> \\# turns vs. limited time\n\nWe do not limit the output length in each turn, so it may not reflect limited time.\n\n> Response length\n\nBelow, we report the average response length and their Pearson correlation coefficient $c$ to Originality/Elaboration: #words/$c_O$/$c_E$.\n||AUT|Instances|Similarities|Scientific|\n|-|-|-|-|-|\n|Single Agent|14.59/0.22/0.47|1.48/0.25/0.36|13.26/0.20/0.57|23.07/0.02/0.33|\n|LLM Debate|23.48/0.24/0.57|1.95/0.21/0.42|21.55/0.10/0.46|37.41/0.04/0.19|\n|LLM Discussion|27.82/0.07/0.31|8.00/0.26/0.35|32.03/0.07/0.21|47.23/0.11/-0.05|\n\nThe results show a weak correlation between response length and creativity scores. Also, we present [histograms](https://shorturl.at/sfrir) analyzing response lengths and scores, showing that our method achieves better scores across varying length intervals. Hence, our improved creativity should not simply be attributed to longer responses. We will revise the paper to include the analysis.\n\n> Ablation study\n\nAs suggested, we evaluated LLM Debate + role-play (w/o discussion), and LLM Discussion w/o role-play. The results below show the efficacy of each component. We will include the ablation study in the revision.\n||Originality|Elaboration|\n|-|-|-|\n|LLM Debate w role-play|3.62 |3.12|\n|LLM Discussion w/o role-play|3.50| 3.02|\n|LLM Discussion|3.83|3.10|\n\n> Which LLM \n\nWe used gpt-3.5-turbo-0125 for all the methods, which will be clarified in the revision.\n\n> Which role performs convergence\n\nEach and every role produces its final responses. All the responses are graded separately and averaged to obtain the score. We will revise the paper to make this clear.\n\n> Temperature vs. creativity\n\nPlease refer to the \\[Temperature vs. creativity\\] response to Reviewer 6CAt.\n\n> Kendall rank correlation coefficient\n\nWe interpret Kendall's tau correlation based on [1-2], stating that a correlation greater than 0.3 is considered strong.\n- [1] Botsch. Chapter 12: Significance and measures of association. Scopes and methods of political science, 2011.\n- [2] Chiang and Lee. Can large language models be an alternative to human evaluations? Empirical Methods in Natural Language Processing, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work is thorough, considers prior both in education and recent CS work seriously, and discusses the implications and limitations in a nuanced and realistic way.\n- The LLM Discussion framework -- although somewhat evident -- is well constructed, particularly with the concept of the three phases of discussion: initiation, discuss & convergence.\n- The human annotation seems thorough, and the release of the collection of judgments might be useful to others studying creativity in similar settings.\n- The paper is generally well-written, and although some details are lacking, these can easily be resolved in a round of revisions.", "reasons_to_reject": "- Nothing serious, overall the work is creative and rigorous.\n- The four tasks studied are known to be somewhat artificial. As the authors point out, the tests to evaluate creativity in humans might not be adequate to evaluate LLMs, since the latter can be infinitely verbose at very low cost. The authors point this out and attenuate the importance of certain TTCW metrics, yet the tasks themselves remain tasks that were created for humans, not LLMs.\n- There is a strong correlation between response length and perceived creativity, both in human & LLM evals. The work did not directly control for length of response (which seems like it would have been very easy to do...) and the examples shown in Figure 4 unfortunately show that LLM Discussion outputs are much longer. Are the promising results therefore only reflecting the underlying signal that responses in that condition are significantly longer on average?", "questions_to_authors": "Q1: Can you report the average length of response for each task and each approach? This would help explain to the reader whether length explains away the creativity difference or not. Why did you not include in your prompt an expected response length, to normalize for this important effect?\n\nQ2: Besides the tasks you've collected, whose intended audience are humans under a time constraint, do you have a sense of what tasks could be more appropriate to evaluate LLMs' creativity abilities? Do you believe that the \"number of turns\" reflects a similar effect to human test-takers having \"limited time\" in TTCT?\n\nQ3: Two of the core contributions in LLM Discussion seem to be: (1) the three phases, (2) the role-playing. Yet, neither of these two elements seems to have been ablated. Do you have experimental evidence showing that such elements indeed lead to improved creativity in the responses?\n\nQ4: It is not 100% clear in the paper which LLM is used in experimentation. The draft clearly states which GPT model is used for auto-eval, but not for generation. Are you using GPT4? Are the different roles all played by the same underlying LLM, or different LLMs? Does that make a difference?\n\nQ5: It is not 100% clear in the paper how the \"Convergence Phase\" is conducted. It seems logical that only one LLM would perform the final agglomeration based on the conversation so far. How do you pick which \"role\" gets to do the final response? (e.g., do you get the final response from the environmentalist or the social entrepreneur).\n\nQ6: Temperature plays a strong role on the diverging nature of an answer. What temperature setting did you set the generation models with, and do you expect increased temperature to further improve creativity scores?\n\nMinor notes:\n- It seems a bit of a stretch to call tau correlations at ~0.3 \"strong correlation\"\n- This reviewer appreciated the pun in the title of Section 6, kudos :)", "extra_scores": {}}, {"review_id": "nhIyEoOwos", "reviewer": "Reviewer_6CAt", "summary": "The authors present an LLM discussion framework for improving the creativity of LLM outputs by using multiple LLM agents with diversified role-playing instructions in a 3-step pipeline.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.\n\n> Temperature vs. creativity\n\nWe additionally experimented with increased temperatures from 0.0 to 2.0 (1.0 was used for the main experiments) on AUT and present the results [here](https://shorturl.at/1ew8I). The results show that increasing temperature leads to higher diversity (a lower self-BLEU score) while not resulting in higher creativity scores. The model outputs nonsensical responses, e.g., unrecognizable symbols, with a temperature ≥ 1.6. This justifies the need to develop frameworks to improve LLM creativity. We will revise the paper to include this insight.\n\n> Fluency vs. diversity\n\nWe would like to clarify that Fluency refers to the number of relevant responses instead of fluency in linguistics. Based on the results presented in the \\[Temperature vs. creativity\\] response, there is no clear tradeoff between Fluency and diversity -- increasing temperature leads to decreased self-BLEU score while Fluency is maintained.\n\n> Creative problem-solving\n\nThis work aims to improve LLM creativity on widely recognized and validated benchmarks, while we agree that evaluating LLMs on creative problem-solving is a promising direction [1], which will be discussed in the revised paper.\n\n- [1] Tian et al. MacGyver: Are Large Language Models Creative Problem Solvers?\n\n> Why is Flexibility not a main metric? \n\nFlexibility measures the number of categories in the responses; for example, when asked to list uses for a fork, eating spaghetti and using it as a hanger would be two different categories while eating spaghetti and eating steak would be in the same category. We did not prioritize Flexibility as a main metric because (1) Unlike humans, LLMs can inherently generate a large number of responses, which inflates Flexibility, and (2) We prioritized Originality and Elaboration, which more directly reflect the model's creativity. The points align with [2-4].\n\n- [2] Guzik et al. The originality of machines: AI takes the Torrance Test. Journal of Creativity\n- [3] Haase et al. Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. Journal of Creativity\n- [4] Hadas et al. Using Large Language Models to Evaluate Alternative Uses Task Flexibility Score. Thinking Skills and Creativity\n\n> Fig 6\n\nThe scores in Fig 6 are normalized to [0, 1]. We will revise it to show the original scores to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of giving different role-playing instructions to different agents seems intuitive and well-motivated, and the method appears empirically effective at increasing creativity. Additionally, although they are evaluating creativity which (I think) should be a fairly difficult thing to nail down evaluation-wise, the experimental setup and analysis seem reasonable and sound, which I think is a solid contribution in its own right (and could perhaps be highlighted more if the authors want to).", "reasons_to_reject": "While the authors have shown that their method can improve the creativity of LLM outputs, it does appear to come at some cost to fluency. In LM generation one can typically increase diversity trivially at the cost of fluency by e.g. increasing the temperature. It's not immediately obvious to me that the proposed method is not just a fancy way of doing the same tradeoff, rather than \"pushing the Pareto boundary\" in some sense. While the current experimental results are fine, in my opinion this paper would be substantially stronger if the authors showed results on e.g., something like a creative problem solving task where the additional creativity actually results in something like higher success rate or accuracy, rather than solely improving creativity for its own sake.", "questions_to_authors": "- Why does your method often perform worse on fluency compared to some baselines? Is there a tradeoff for fluency vs diversity (similar to how you make this tradeoff when you change the temperature of generation)?\n- Why is flexibility not a main metric? Isn't coming up with more different answers a pretty reasonable thing to evaluate? Or is the problem just that you don't have a good way to \"deduplicate\" different answers? Not sure if I'm misunderstanding what is meant by flexibility in your setup.\n- Why are all the metrics 0 when you have 3 agents in fig 6?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel LLM Discussion approach to enhance the creativity of large language models through multi-agent discussions and role-playing. Evaluations across four creativity benchmarks demonstrate improvements in originality and elaboration compared to single-LLM and existing multi-LLM frameworks. While some concerns are raised regarding role selection, potential biases, and the need for deeper qualitative analysis, the overall work is considered creative, rigorous, and likely to stimulate further research. I therefore recommend accepting this paper. There is also a related approach here https://arxiv.org/pdf/2310.00280 that the authors may find interesting.
Decision: Accept