Agents' Room:  Narrative Generation through Multi-step Collaboration

Authors: Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, Mirella Lapata
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: fiction, creative writing, long-form generation, LLMs, agent, collaboration, multi-agent, dataset, evaluation

Abstract:
Writing compelling fiction is a multifaceted process combining elements such as crafting a plot, developing interesting characters, and using evocative language. While large language models (LLMs) show promise for story writing, they currently rely heavily on intricate prompting, which limits their use. We propose Agents' Room, a generation framework inspired by narrative theory, that decomposes narrative writing into subtasks tackled by specialized agents. To illustrate our method, we introduce Tell Me A Story, a high-quality dataset of complex writing prompts and human-written stories, and a novel evaluation framework designed specifically for assessing long narratives. We show that Agents' Room generates stories that are preferred by expert evaluators over those produced by baseline systems by leveraging collaboration and specialization to decompose the complex story writing task into tractable components. We provide extensive analysis with automated and human-based metrics of the generated output.

Decision: accept (poster)
Reviewer scores: 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730575667120

thread_type:
official_review

<thread_object>
creator: reviewer_cty6

<content>
summary:
This paper introduces AGENTS' ROOM, a framework that breaks down complex writing tasks into subtasks handled by specific AI agents working collaboratively. The framework includes planning agents that develop story elements like plot and characters, and writing agents that generate specific narrative sections, all coordinated by a central orchestrator that manages information flow through a shared scratchpad. To demonstrate the framework's effectiveness, authors created TELL ME A STORY, a high-quality dataset of writing prompts and human-written stories developed through writing workshops, which they use to train and evaluate their system. They implement their framework using large language models as agents and explore both zero-shot and fine-tuned approaches, showing that AGENTS' ROOM generates stories preferred by human evaluators over baseline systems. Their evaluation framework assesses multiple dimensions of story quality, including plot, creativity, development, and language use, along with an LLM-based evaluator that correlates significantly with human judgments.

strengths:
There is so much to love about this paper. Great Motivation. Great Design. Great Evaluation. Great Technical Depth. One of the fundamental bottlenecks of autoregressive language models is that they do not look at the bigger picture, hierarchical structure that often drives narratives. If you simply prompt an LLM to write a story it will generate something typically shorter than what you want and often misses out on crucial details because these models aren't fundamentally good at Narrative Planning (https://arxiv.org/abs/2402.01817).  So it makes total sense to break this into agents. It reminded me of Operating systems. Just like how multithreaded systems break down complex tasks into parallel threads, AGENTS' ROOM decomposes the writing task into specialized subtasks (like plot planning, character development, etc.) handled by different agents. The shared memory , orchestrator, State Management etc are all coming from OS and I find the design very neat for a totally different task. I very much appreciate detailed descriptions of agent prompts in Appendix that is very helpful for reproducibility. Its great that authors employ similar practices as that of WordCraft, Dramatron etc to get professional writers come of with fresh prompts and flash fiction stories that are not contained in Pre-training  (yet ? hard to stop from contamination in future).
I like the evaluation done by experts mostly preference evaluation that is very popular in Alignment, Behavioral Science literature. This adds further credibility. Nice ablations. Additional auto eval showing repetition through ngrams. Great to see fresh human written stories do better than models here. Last but not the least nice to include LLM evaluation given current trends for RLAIF (Bai et al) [ Though I would not trust them].

weaknesses:
I don't think these are big weaknesses still

1) Since you are anyway having multi step collaboration and agents why skip Revising ? Flower and Hayes's Cognitive process model of writing includes Planning , Writing and Revising. I am sure expert writers revise when they write. In professional setting a Literary agent takes on that role suggesting both developmental and line level edits. This would add more credibility to the paper and worth discussing this in camera ready

2) Length as a confounder : I believe given that your stories have very different length, how this translates to eval? To control for this (Chakrabarty et al 2024) did iterative expansion till all stories had comparable length. My point is experts could dismiss shorter length AI generated stories as they are both Poor Quality + Brief. Of-course expert writing is infinitely better so experts can recognize that even if 50% shorter

3) Could be interesting to show the diversity of stories in your data. What are the genres ? Are there any recurring motifs or themes.WritingPrompt has a disproportionately high number of Alien stories. We don't want the case to be that.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589838709

thread_type:
official_review

<thread_object>
creator: reviewer_pcjk

<content>
summary:
This paper conceptualize long-form writing as a multi-agent collaboration problem. Leveraging narrative theories and existing multi-agent works in other domains, it decomposes the writing task into sub-tasks tackled by specialized agents (a few for planning and others for writing), which communicate via scratch pad and a central controller called orchestrator.

In addition, the authors contributed TELL ME A STORY, a high-quality dataset of prompts and longform stories collected through multiple rounds of writing workshops with human writers.

strengths:
1) This paper proposes a novel multi-agent multi-step framework for long-form writing, an important task for content generation with LLM
2) It also contributes a dataset containing human generated prompts and stories, which I assume to be high-quality
3) The paper is well-written and easy to follow

weaknesses:
I am most concerned with the soundness of the experiments that currently do not support the claims made. To name the most severe ones:

1) **Effectiveness of multi-agent collaboration**: One of the most fundamental motivating claims is that single-agent works, such as detailed prompts to guide the generation process (Yang et al., 2022; Xie et al., 2023), prompt chaining (Mirowski et al., 2023; Yang et al., 2022), and planning strategies (Yang et al., 2023; Lee et al., 2024) fail to generate high-quality stories. Additionally, this paper which I found interesting [1] also showed that prompting a single LLM to reason about turning points improves overall narrative construction such as reducing plot holes and generating better plot twists.

However, none of these methods are compared against in the experiments. I would expect to see at least 2~3 baselines from the above to verify if indeed the improvement is **because of** multi-agent collaboration. The authors just compared to an end-to-end approach, which is far from enough. 

2) I am not fully convinced that "The LLM evaluator agrees with humans and itself". Based on figure 3, the left and right show very different trends for human and the best method (AR plan+write). Clearly, LLMs prefer its own output over human-written, whereas human evaluators prefer human-written ones. I doubt if the seemly high correlation is a result of taking the mix of extremely bad responses to boost the correlation. If we are already at a time where LLMs can generate moderately good stories, the LLM based evaluators seems highly unreliable. 

 
[1] Are Large Language Models Capable of Generating Human-Level Narratives? EMNLP 2024

questions:
1. Who are the participants of the writing workshop? What are their demographics? What are their backgrounds/expertise in writing and are they allowed to use writing assistant for writing/review/discussion? How motivated are they to produce the best writing pieces?
2. See weakness. I would consider raising my scores if Weakness 1 has been adequately addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667354990

thread_type:
official_review

<thread_object>
creator: reviewer_8uss

<content>
summary:
This paper proposes a multi-agent framework for collaborative creative writing called Agent’s Room. In this framework, planning and writing agents pass along a scratchpad of outputs to collaboratively write a story. Planning agents first plan out a story’s main conflict, character/s, setting, and plot. Then, the system calls writing agents that specialize in a story’s exposition, rising action, climax, falling action, and resolution. These agents are fine-tuned on synthetic data generated by a larger language model, which decomposes human-written stories into relevant parts. Next, the paper evaluates generated stories by Agent’s Room, planning/writing ablated approaches, and an end-to-end baseline in a pairwise manner with human writers and an LLM-based evaluator. The authors find that though human-written stories still outperform LLM-based ones based on human judgements, Agent’s Room performs better than end-to-end story generation based on both human and LLM judgements.

strengths:
This paper is well-written and thoughtful, and it’s evident that effort was put in to involve actual human writers in data collection and evaluation components. Generally I found this work to be pretty solid and empirically shows the utility and promise of currently trendy approaches in AI (e.g. collaborative agents, synthetic data generation, LLMs-as-judge) for creative writing.

weaknesses:
This paper spends a considerable amount of space discussing and advocating for a “general” writing framework (e.g. lines 231-233 as well as the paragraph introducing the orchestrator), but it only provides empirical evidence of its utility for one task: fiction writing. So, it may be helpful to rework the paper’s framing so that its claims actually match with its findings. Discussion around the generalizability of the proposed approach could be left in the conclusion as future work. If the authors want to stick with proposing a general framework, then they need to run similar experiments with additional tasks. 

Figure 1 displays little icons suggesting people, but the orchestrator, planning agents, and writing agents in this work are not people. It may be better to use a different icon. In addition, it may be useful to label agents with the roles you gave them in this current work; though the diagram may be intended to show a generalizable framework, its minimalism makes it somewhat uninformative. 

There is also the dreaded “you could have experimented with more models” weakness that comes with any work that only uses one family of models (in this case, Gemini). I mostly note this because including an open model in the mix could improve the accessibility of this work (e.g. there is a practical difference between paying-per-token versus running open models on your own compute infrastructure). It can also increase the longevity of this work if its findings are explicitly shown to be non-dependent on a specific model, allowing more focus on the framework proposed. Alternatively, if other models are not strong enough to collaborate in the manner proposed by this paper, that in itself may be interesting to note.

Another suggestion is instead of spending space on Figure 2 showing three different prompts, it may be more interesting to show instead one prompt, the beginning of a story generated by an end-to-end system, and the beginning of one generated by your proposed system. That is, this entire paper discusses writing stories but doesn’t really ever show an example of a story or story excerpt. If finding space for this is an issue, I’m also not sure if Algorithm 1 really adds much to the paper and could be removed. 

Finally, it may be useful to do some more reflection on why we would want models to write creatively on their own in the first place, and the (potentially negative) impacts this may have on human writers and the writing industry. This is, why would we want automatic writing systems that go beyond writing assistants? This isn’t to suggest that there are no possibly positive uses of automatic writing systems, but to encourage the authors to consider pros and cons of the basis of their work more thoroughly. Any technology that has the potential to replace rather than support humans surfaces risks which are not really discussed in this paper’s ethical considerations section. The authors do mention that components of their system could be interchanged with human writers to become a human-in-the-loop system, but don’t provide empirical evidence of this.

questions:
How much experience with creative writing do the human writers have prior to participating in writing workshops? 

It’s interesting that human judges prefer the system that involves writing agents alone over the one that has both planning and writing agents, while the LLM evaluator does not. Is there any intuitive/qualitative explanation as to why this may be the case? Did the human judges provide reasons for their judgements?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a method to address the issue of long form narrative generation by breaking the task down into those that can be performed by specialized agents. The reviewers generally agree that the technical depth, motivation, and presentation are of sufficient quality to warrant publication. That said, reviewer 8uSS especially raises some great points in their weaknesses section eg advocating for the the overall framing of the paper to be focused on fiction writing as opposed to a generalized framework that I agree with and hope to see in a camera ready.