Lyric App Framework:
A Web-based Framework for Developing
Interactive Lyric-driven Musical Applications
Jun Kato and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
1. Introduction to lyric apps

3. Discussions

2. Lyric App Framework
TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

11. Introduction to lyric apps
Lyric App Framework: A Web-based Framework for Developing
Interactive Lyric-driven Musical Applications
Jun Kato and Masataka Goto

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

2

What is lyric app?

TextAliveJp/textalive-app-p5js

TextAliveJp/textalive-lyric-tiles

TextAliveJp/textalive-app-dance• Novel interactive lyric-driven visual art proposed in our paper• Lyric and visual effects are synchronized with music playback anddynamically adapt to user input• Visual output is generated on the fly by programmers’ algorithms*Three of eleven example applications that we built and provided on the framework via open-source distribution

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

3

Representative example of lyric app
MiracleUniverse =by Misora Ryo in

Hatsune Miku
“Magical Mirai”
10th Anniversary
Programming Contest
https://developer.text
alive.jp/events/magic
almirai10th

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

4

Representative example of lyric app
MiracleUniverse =by Misora Ryo in

Hatsune Miku
“Magical Mirai”
10th Anniversary
Programming Contest
https://developer.text
alive.jp/events/magicalmirai10th

Three characteristics:

Precise multimedia
synchronization

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Interactive visualexperience

Mass distribution toend-users

Lyric App Framework [CHI 2023]

5Hatsune Miku “Magical Mirai” Programming Contest• Annual programming contests using our framework since 2020
• Held as part of a larger exhibition Hatsune Miku “Magical Mirai” incollaboration with Crypton Future Media, Inc.
• 52 applications (2020 & 2021) were analyzed in our paper
2020

2021

Hatsune Miku “Magical Mirai 2020”
Programming Contest (9/18-11/3)

mmmapper by nolze

歌って頂いた。 by minatty(award winner in 2020)

Lyric Trail by catLee

光粒子とコトバの海 byKaku

9:00-, Apr. 27, 2023 [Room Y07+Y08]

2022 (latest; not in paper)

Hatsune Miku “Magical Mirai 2021”
Programming Contest (7/16-9/30)

配置 de PV by upc1712

キミを探す、夏 by うたろ(award winner in 2021)

Voice Shooter
by sakuramodoki et al.

TouchLyricWorldby huskyB4ll

Hatsune Miku “Magical Mirai 10th Anniv.”
Programming Contest (4/15-7/21)

エモーショナライザー
by しろねぎ

Miracle Universe = by Misora Ryo(award winner in 2022)

MemoryZone
by Team MOROMENMulticoloredLyrics
by mikandaisuki

Lyric App Framework [CHI 2023]

6

This year’s contest: entries open until Ju. 18 noon (JST)!

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

7

Design space exploration: 8 lyric app categories• Paper appendix C has lyric app explanations and notable creativecoding libraries used in all 52 applications• Limitation: no use of experimental web APIs due to contest format(geolocation, accelerometers, Bluetooth, etc.)

Extended reality

Authoring tool

Creative application

Instrument

Game

Augmented music video

Interactive lyric video

Generative lyric video

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

8

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

9Design space exploration: 8 lyric app categories• Paper appendix C has lyric app explanations and notable creativecoding libraries used in all 52 applications• Limitation: no use of experimental web APIs due to contest format(geolocation, accelerometers, Bluetooth, etc.)

Extended reality

Authoring tool

Creative application

Instrument

Game

Augmented music video

Interactive lyric videoGenerative lyric video

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

10

2. Lyric App Framework
Lyric App Framework: A Web-based Framework for Developing
Interactive Lyric-driven Musical Applications
Jun Kato and Masataka Goto

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

11

Challenges and corresponding framework components
Three characteristics:

Precise multimedia
synchronization

Interactive visualexperience

Mass distribution toend-users

TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

Intelligent andflexible webbased workflow
9:00-, Apr. 27, 2023 [Room Y07+Y08]

TextAlive App API
for creative coding
support

Lyric App Customizer
and distribution as
web apps
Lyric App Framework [CHI 2023]

12

Intelligent and flexible web-based workflow
Upload or choose
online music

Check and correct
automatic analysis

Develop, customize,and distribute apps
TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

https://textalive.jp/profile

https://editor.textalive.jp

https://textalive.jp/songs

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Available analysis results: lyric timings,
part of speech, vocal amplitude, valence
and arousal, musical beats, chords,
chorus segments

Lyric App Framework [CHI 2023]

13

Challenges and corresponding framework components
Three characteristics:

Precise multimedia
synchronization

Interactive visualexperience

Mass distribution toend-users

TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

Intelligent andflexible webbased workflow
9:00-, Apr. 27, 2023 [Room Y07+Y08]

TextAlive App APIfor creative codingsupport

Lyric App Customizer
and distribution as
web apps
Lyric App Framework [CHI 2023]

14

TextAlive App API for creative coding support• JavaScript API that can be installed from public package registry• API documentation, explanation videos, and open-source
examples are available at https://developer.textalive.jp• It consists of two parts:
Event-driven API
for state management

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Time-driven API
for time-sensitive interactions

Lyric App Framework [CHI 2023]

15

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬMiku’s Live by tokei39

1.
2.

Designed for a specific music
No pausing nor seeking9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

16

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬ

Miku’s Live by tokei39

1.
2.

Designed for a specific music
No pausing nor seeking

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

17

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬ

Miku’s Live by tokei39

1.
2.

1.
2.

Designed for a specific music
No pausing nor seeking

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Designed for a set of pieces
No seeking but pausing

Lyric App Framework [CHI 2023]

18

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬMiku’s Live by tokei39

1.
2.

1.
2.

Designed for a specific music
No pausing nor seeking

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Designed for a set of pieces
No seeking but pausing

Lyric App Framework [CHI 2023]

19

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬ

Miku’s Live by tokei39

1.
2.

1.
2.

1.
2.

Designed for a specific music
No pausing nor seeking

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Designed for a set of pieces
No seeking but pausing

Designed for any musical piece
Pausing and seeking allowed

Lyric App Framework [CHI 2023]

20

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬ

Miku’s Live by tokei39

1.
2.

1.
2.

1.
2.

Designed for a specific music
No pausing nor seeking

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Designed for a set of pieces
No seeking but pausing

Designed for any musical piece
Pausing and seeking allowed

Lyric App Framework [CHI 2023]

21

Event-driven API for state management (TextAlive App API [1/2])
1. Lyric app can be designed for different sets of musical pieces
2. Lyric app can have different levels of end-user interactions
VR storytelling application:

Rhythm game:

Music player:

キミを探す、夏 by うたろ

Voice Shooter by sakuramodki / すぱりだ / ななしお / HiroyukiIsoe / ya2ha4 / りおんぬ

Miku’s Live by tokei39

1.
2.

1.
2.

1.
2.

Designed for a specific music
No pausing nor seeking

Designed for a set of pieces
No seeking but pausing

Designed for any musical piece
Pausing and seeking allowed

API to control states (request{Play,Pause,Seek}()) and get transitions (addListener({ … }))9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

22

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Event-driven API for musical elements (onBeat, onWord, on…) does
not work well from three perspectives: future planning, considering
multiple musical elements for synthesized visuals, and time precision

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

23

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Event-driven API for musical elements (onBeat, onWord, on…) does
not work well from three perspectives: future planning, considering
multiple musical elements for synthesized visuals, and time precision

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

24

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Event-driven API for musical elements (onBeat, onWord, on…) does
not work well from three perspectives: future planning, considering
multiple musical elements for synthesized visuals, and time precision

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

25

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Event-driven API for musical elements (onBeat, onWord, on…) does
not work well from three perspectives: future planning, considering
multiple musical elements for synthesized visuals, and time precision

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

26

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Event-driven API for musical elements (onBeat, onWord, on…) does
not work well from three perspectives: future planning, considering
multiple musical elements for synthesized visuals, and time precision

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

27

Time-driven API for time-sensitive interactions (TextAlive App API [2/2])
Instead, we propose Time-driven API (findBeat, findWord, find…):
future planning is easy (findBeat(p + 5000)), considering multiple elements isstraightforward (findBeat(p); findWord(p); find...), and no worry for time precision*Refer to the paper for the detailed API usage and improved proposal of time-range-driven API we found in the user study

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

28

Challenges and corresponding framework components
Three characteristics:

Precise multimedia
synchronization

Interactive visualexperience

Mass distribution toend-users

TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

Intelligent andflexible webbased workflow
9:00-, Apr. 27, 2023 [Room Y07+Y08]

TextAlive App API
for creative coding
support

Lyric App Customizerand distribution asweb apps
Lyric App Framework [CHI 2023]

29

Lyric App Customizer and distribution as web apps
• Graphical web interface to customize existing lyric apps
• Lyric apps in <iframe> communicate with Customizer• Customizations can be stored as query string e.g., ?ta_song_url=…

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

30

3. Discussions
Lyric App Framework: A Web-based Framework for Developing
Interactive Lyric-driven Musical Applications
Jun Kato and Masataka Goto

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

31

Lyrics and evolution of media technologies• Vinyl records and compact discs come with lyric sheets• Video-sharing services made lyric videos popular [TextAlive, CHI 2015]• We foresee lyric apps, interactive lyric-driven media, will follow
9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

32

Perspectives on adding interactivity to existing media• Kickstarting of creativity• Programming as communication• Building future of/withincreative culture [SIGCCC, CHI 2023]

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

33

It’s publicly available at https://developer.textalive.jp – try it out!

Lyric App Framework:
A Web-based Framework for Developing
Interactive Lyric-driven Musical Applications
Jun Kato and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
1. Introduction to lyric apps

3. Discussions

2. Lyric App Framework
TextAlive App API
https://developer.textalive.jp$ npm install textalive-app-api
import {Player} from "textalive-app-api";
new Player().createFromSongUrl("…");

9:00-, Apr. 27, 2023 [Room Y07+Y08]

Lyric App Framework [CHI 2023]

34