情報処理学会 インタラクション 2024
IPSJ Interaction 2024

3B-54
2024/2/28

MyStoryKnight: A Character-drawing Driven
Storytelling System Using LLM Hallucinations
Sechayk Yotam4,a) Penarska Gabriela A.2,†1,b) Randsalu Isa A.1,†1,c)
Arzate Cruz Christian3 Igarashi Takeo4

Abstract: Storytelling is a valuable tradition that plays a crucial role in child development, fostering creativity and a sense of agency. However, many children often consume stories passively, missing out on theopportunity to participate in the creative process. To address this, we propose a storytelling system thatcreates adventure-type stories with multiple branches that users can explore. We generate these interactive
stories using a character drawing as input, with visual features extraction using GPT-4. By leveraging LLM
hallucinations, we generate interactive stories using user feedback as a prompt. Finally, we refine the qualityof the generated story through a complexity analysis algorithm. We believe that the use of a drawing as
input further improves the engagement in the story and characters.

1.

Introduction

Creativity is increasingly recognized as a crucial aspect ofthe modern working environment. In the context of child development, creativity plays a pivotal role in fostering learning and growth [5], which extends to adulthood. As a result, there has been a growing interest in leveraging creative artificial intelligence (AI) and exploring the use of AI
agents to stimulate children’s creativity. Previous researchhas demonstrated the positive effects of incorporating AI
agents with creative abilities in enhancing children’s creativethinking. Additionally, the use of storytelling robots andvirtual characters as interactive tools has gained traction inrecent years [17]. In this paper, we propose MyStoryKnight,
an interactive AI tool for generating stories.
Our proposed system is intended to promote children’s
creativity through storytelling, a popular activity for entertaining and bonding [17]. Both storymaking and storytelling
contribute to the development of verbal and social skills[3], [14], such as broadening the vocanulary and improving
narrative comprehension. Besides, creating fictional worlds
and characters helps children improve their language skills,
both in comprehension and usage [1]. In MyStoryKnight,users, together with an AI, co-create stories with multiple
explorable branches.
1234†1a)b)c)Lund University
Technical University of Denmark
Honda Research Institute JapanThe University of Tokyo
Presently with The University of Tokyo
sechayk-yotam@g.ecc.u-tokyo.ac.jp
gabriela-penarska@g.ecc.u-tokyo.ac.jp
randsalu-isa071@g.ecc.u-tokyo.ac.jp© 2024 Information Processing Society of Japan

Despite all the benefits of practicing storytelling, many
children do not have the opportunity to engage in interactive storytelling. MyStoryKnight addresses this problem bygenerating an adventure-type story through LLM hallucinations. Figure 1 shows an overview of our system. Oursystem uses drawings of characters as input, and generates
an unfolding story that the user can navigate through basedon their choices. Using a complexity analysis algorithm, we
guide the LLM hallucinations to generate a coherent andconsistent story. Resulting in a story that is both engaging
and easy to follow.
Our contributions are:
• Storytelling system that uses the character drawing asa basis for the story.
• LLM hallucinations to generate an adventure-type story
with user navigation.
• Complexity analysis of story generation for coherency
and consistency.

2.

Related Work

2.1 Interactive Storytelling
Interactive storytelling is a form of storytelling in which
the audience is an active participant [19]. Interactive storytelling has been shown to be an effective way of building andstrengthening relationships [15], also playing an importantrole in parent-child bonding [20]. Interactive stories stimulate thinking and imagination [4], and help children makesense of their world through shared experiences [14].
Prior work has explored physical interactivity through theuse of tangible objects [14] to create stories, and physicalgestures to control characters [10] or generate stories [21].
Other studies have explored the use of sketches to influence1297

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

Fig. 1

MyStoryKnight system overview. Drawing is anaylzed with GPT-4. Backstory andaction choices are presented to the user. Story unfolds based on user choice.

character behavior and plot development [4]. However, thesestudies have not explored the use of character drawings as
the driving force behind the story.
2.2 AI Storytelling
AI usage has been experiencing a surge in popularity in recent years, with many applications in the creative arts. Generative AI especially has the potential to contribute to creative processes [18]. One such creative process could be storytelling. Prior work has explored how generative AI can beused to create stories [1], and even collaborate in story generation with humans [16]. Other studies explored how generative AI can be used to expand existing story worlds [2], orgenerate multi-modal storytelling experiences [8]. However,while consistency and coherency are essential in storytelling,
they are often overlooked in AI-generated stories.
2.3 Storytelling with ChildrenChildren oriented storytelling has been explored widely inthe past. Prior work has explored the use of robots as storytelling companions [17], and the use of virtual charactersto promote collaboreation [10]. Other studies have exploredthe use of AI agents to stimulate children’s creativity [5].
Other studies have shown how AI can support parent-childbonding [20], and encourage physical interaction and play[21]. However, utilizing hallucinations to generate stories
has not been explored in the context of user-agency for storytelling with children.

3.

3B-54
2024/2/28

3.1 System Architecture
The system consists of three main components:( 1 ) Drawing Analysis: Extracting visual features fromthe user’s drawing. The visual features are used as aprompt for the LLM to hallucinate a character description and a background story.
( 2 ) Story Generator: Story is generated based on user
selection on a set of actions. LLM hallucinations areused to generate both the story and actions.
( 3 ) Complexity Analysis: Natural language processing(NLP) techniques are used to analyze the story’s coherency and consistency. The results are used to guidethe LLM hallucinations.
To achieve our goals, we utilize GPT-4 [13] for drawing analysis and story generation. Additionally, we use spacy [9] inaddition to GPT-4 for complexity analysis. The system isimplemented with Django, a Python web framework [7], andReact, a JavaScript library for building user interfaces [6].

4.

User Interface

The user interface is shown in Figure 2. The main interaction screen consists of an action selection area (bottom),a restart button (top-right), a microphone indication toggle(bottom-left), and a story output panel (center).

System Overview

We designed MyStoryKnight, an interactive storytelling that uses drawings as input and utilizes LLM hallucinations to generate a branching story. Agency is a critical factor in interactive storytelling [4], as it promotes engagement and immersion [20]. Our system supports user
agency by allowing users to navigate the story through theirchoices. Additionally, we use a complexity analysis algorithm to guide the LLM hallucinations to generate a coherent and consistent story, fitting for children’s comprehension.

Fig. 2Mockup of the user interface.

The system uses a webcam to scan and identify user drawings which are then uploaded and analyzed. The story is© 2024 Information Processing Society of Japan1298

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

3B-54
2024/2/28

then told using speech synthesis with the OpenAI API [12],and story images are generated using DALL-E [11]. The
generated images are optimized to maintain a style similar
to that of the user. Actions are then presented to the user,
and these could be activated by clicking, or alternatively byvoice commands. The user can also choose to restart thestory at any point.

5.

Experiencing MyStoryKnight

In this section, we describe the interaction flow of the system.
Our story begind with a few colord strokes on apiece of paper.
The user draws a character on a piece of paper, and thesystem scans the drawing using a webcam (Fig. 3).

Fig. 4

An example output of the system, the t́ournament hall.́
Generated with OpenAI’s DALL-E [11].

Weight lifting. The user makes their choice and the story
continues. This process repeats until the user chooses to end
the story.

6.

Fig. 3

An example of a user drawing. A drawing of a knight,generated with OpenAI’s DALL-E [11].

The drawing is then uploaded to the system and analyzed.
The system then generates a character description and abackground story based on the drawing. The backgroundstory is then told to the user.
Sir Colorful, the most vibrant knight in all theland. Each color on Sir Colorfuls armor andshield represents a virtue: blue for loyalty, yellow
for generosity, green for hope, and red for courage.
With his trusty sword and shield, Sir Colorful isready to face any challenge.
The user is then presented with a set of actions to choose
from: Go to tournament or Run away. These choices
are then fed into GPT-4 to generate the next part ofthe story, and the next set of actions. The user chooses
Go to tournament and the story continues (Fig. 4).
The tournament hall was filled with knights fromall across the realms, each one more colorful thanthe last. Sir Colorful was eager to make newfriends and show off his skills. But first he had
to comence his training.
This leads to the following actions:© 2024 Information Processing Society of Japan

Jousting andConclusion

In this paper, we presented MyStoryKnight, an interactive storytelling system that uses character drawings as input. We generate an adventure-type story with multiple explorable branches using LLM hallucinations. Additionally,
we use a complexity analysis algorithm to guide the LLM
hallucinations to generate a coherent and consistent story.
We believe that the use of a drawing as input further improves the engagement in the story and characters. In thefuture we plan to hold a user study to evaluate the effectiveness of our system in promoting creativity and engagement.
References
[1]
[2]

[3]

[4]

[5]

[6]
[7]

Burtenshaw, B.: AI Stories: An Interactive Narrative Systemfor Children, CoRR, Vol. abs/2011.04242 (online), available
from ⟨https://arxiv.org/abs/2011.04242⟩ (2020).
Chopra, B., Verma, K., Singhal, S. and Singla, U.: Reality Tales: Facilitating User-Character Interaction with Immersive Storytelling, CHI ’21: CHI Conference on HumanFactors in Computing Systems, Virtual Event / Yokohama
Japan, May 8-13, 2021, Extended Abstracts (Kitamura, Y.,
Quigley, A., Isbister, K. and Igarashi, T., eds.), ACM, pp.
489:1–489:7 (online), DOI: 10.1145/3411763.3451522 (2021).
Currin, F. H., Diederich, K., Pantoja, L. S., Cargo, H., Franzone, N., Geiger-Lee, J. and Hourcade, J. P.: Designing Stories to Inspire Preschoolers’ Creative, Collaborative Roleplay,
Proceedings of the 2023 ACM Conference on Information
Technology for Social Good, GoodIT 2023, Lisbon, Portugal, September 6-8, 2023, ACM, pp. 40–47 (online), DOI:
10.1145/3582515.3609516 (2023).
de Lima, E. S., Gheno, F. J. and Viseu, A.: SketchBased Interaction for Planning-Based Interactive Storytelling, 19th Brazilian Symposium on Computer Gamesand Digital Entertainment, SBGames 2020, Recife, Brazil,
November 7-10, 2020, IEEE, pp. 154–162 (online), DOI:
10.1109/SBGAMES51465.2020.00029 (2020).
Elgarf, M. and Peters, C. E.: CreativeBot: a Creative Storyteller Agent Developed by Leveraging Pre-trained Language Models, IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2022, Kyoto, Japan,October 23-27, 2022, IEEE, pp. 13438–13444 (online), DOI:
10.1109/IROS47612.2022.9981033 (2022).
Facebook: React, https://reactjs.org/ (2021).
Foundation, D. S.: Django, https://www.djangoproject.
com/ (2021).

1299

情報処理学会 インタラクション 2024
IPSJ Interaction 2024

[8]

[9]
[10]

[11]
[12]
[13]
[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

3B-54
2024/2/28

Han, A. and Cai, Z.: Design implications of generative AI
systems for visual storytelling for young learners, Proceedingsof the 22nd Annual ACM Interaction Design and Children
Conference, IDC 2023, Chicago, IL, USA, June 19-23, 2023,
ACM, pp. 470–474 (online), DOI: 10.1145/3585088.3593867(2023).
Honnibal, M. and Montani, I.: spaCy 3.0: IndustrialStrength Natural Language Processing, https://spacy.io/(2020).
Liu, C., Liu, K., Wang, P., Chen, G. and Su, M.: Applying tangible story avatars to enhance children’s collaborative
storytelling, Br. J. Educ. Technol., Vol. 43, No. 1, pp. 39–51(online), DOI: 10.1111/J.1467-8535.2010.01146.X (2012).
OpenAI:DALL-E,https://openai.com/blog/dall-e/(2021).
OpenAI:OpenAIAPI,https://openai.com/blog/
openai-api/ (2021).
OpenAI:
GPT-4
TechnicalReport,CoRR,Vol.
abs/2303.08774(online),
DOI:
10.48550/ARXIV.2303.08774 (2023).
Ryokai, K. and Cassell, J.: StoryMat: a play space forcollaborative storytelling, CHI ’99 Extended Abstracts on
Human Factors in Computing Systems, CHI Extended Abstracts ’99, Pittsburgh, Pennsylvania, USA, May 15-20,
1999 (Atwood, M. E., ed.), ACM, pp. 272–273 (online), DOI:
10.1145/632716.632883 (1999).
Schlauch, M., Sylla, C. and Gil, M.: Investigating Social Emotional Learning at Primary School through Guided
Interactive Storytelling, Extended Abstracts of the Annual
Symposium on Computer-Human Interaction in Play, CHI
PLAY 2022, Bremen, Germany, November 2-5, 2022 (Gerling, K., Iacovides, J., Malaka, R., Bonsignore, B. andFrommel, J., eds.), ACM, pp. 240–245 (online), DOI:
10.1145/3505270.3558313 (2022).
Shakeri, H., Neustaedter, C. and DiPaola, S.: SAGA: Collaborative Storytelling with GPT-3, Companion Publicationof the 2021 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2021, Virtual
Event, USA, October 23-27, 2021 (Birnholtz, J. P., Ciolfi,L., Ding, S., Fussell, S. R., Monroy-Hernández, A., Munson,S., Shklovski, I. and Naaman, M., eds.), ACM, pp. 163–166(online), DOI: 10.1145/3462204.3481771 (2021).
Sun, M., Leite, I., Lehman, J. F. and Li, B.: Collaborative
Storytelling between Robot and Child: A Feasibility Study,
Proceedings of the 2017 Conference on Interaction Designand Children, Stanford, CA, USA, June 27-30, 2017 (Blikstein, P. and Abrahamson, D., eds.), ACM, pp. 205–214 (online), DOI: 10.1145/3078072.3079714 (2017).
Tholander, J. and Jonsson, M.: Design Ideation with AI Sketching, Thinking and Talking with Generative Machine
Learning Models, Proceedings of the 2023 ACM Designing
Interactive Systems Conference, DIS 2023, Pittsburgh, PA,USA, July 10-14, 2023 (Byrne, D., Martelaro, N., Boucher,A., Chatting, D. J., Alaoui, S. F., Fox, S. E., Nicenboim,I. and MacArthur, C., eds.), ACM, pp. 1930–1940 (online),
DOI: 10.1145/3563657.3596014 (2023).
Wang, Z., Romat, H., Chevalier, F., Riche, N. H., MurrayRust, D. and Bach, B.: Interactive Data Comics, IEEE
Trans. Vis. Comput. Graph., Vol. 28, No. 1, pp. 944–954(online), DOI: 10.1109/TVCG.2021.3114849 (2022).
Zhang, Z., Xu, Y., Wang, Y., Yao, B., Ritchie, D., Wu, T.,Yu, M., Wang, D. and Li, T. J.: StoryBuddy: A HumanAI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement, CHI ’22: CHI
Conference on Human Factors in Computing Systems, New
Orleans, LA, USA, 29 April 2022 - 5 May 2022 (Barbosa,S. D. J., Lampe, C., Appert, C., Shamma, D. A., Drucker,
S. M., Williamson, J. R. and Yatani, K., eds.), ACM,
pp. 218:1–218:21 (online), DOI: 10.1145/3491102.3517479(2022).
Zhao, Y. and Bao, X.: Narratron: Collaborative Writing
and Shadow-playing of Children Stories with Large Language Models, Adjunct Proceedings of the 36th Annual ACM
Symposium on User Interface Software and Technology,
UIST 2023, San Francisco, CA, USA, 29 October 20231 November 2023 (Follmer, S., Han, J., Steimle, J. andRiche, N. H., eds.), ACM, pp. 119:1–119:6 (online), DOI:
10.1145/3586182.3625120 (2023).

© 2024 Information Processing Society of Japan1300