Lights, Animation, Interaction!
Synchronizing Music
with Computer-controlled Visuals
in Live PerformancesSociety for Animation Studies 34th Annual Conference, 6/12/2023

Jun Kato
National Institute of Advanced Industrial Science and Technology (AIST)
https://junkato.jp (slides: https://bit.ly/sas34kato)

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

11. Introduction
Lights, Animation, Interaction!
Synchronizing Music with Computer-controlled Visuals in Live Performances
Jun Kato (AIST)
Society for Animation Studies 34th Annual Conference, 6/12/2023

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

2

Jun Kato 加藤 淳

junkatohttps://junkato.jpThe University of Tokyo Igarashi Lab ’09 BSc, ’11 MSc, ’14 PhD
Microsoft Research Asia ’12/1-4 Research Intern / Microsoft Research ’12/6-9 Research Intern
Adobe Creative Technologies Lab, Seattle ’13/8-11 Research Intern
National Institute of Advanced Industrial Science and Technology (AIST) ’14/4- Researcher, ’18/10- Senior Researcher
Arch Inc. ’18/7- Technical Advisor (PI at R&D unit Arch Research)

DejaVu[ACM UIST’12]

Picode[ACM CHI’13]

TouchDevelop[ACM PLDI’13]
TextAlive [ACM CHI’15]

f3.js [ACM DIS’17]

Songle Sync [ACM Multimedia’18]

Griffith [SAS’21]

Human-Computer Interaction researcher studying creative activities e.g., programming,
video authoring, storyboarding through building and deploying creativity support tools

In short, I am a “toolsmith” for creators including animators
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

3

SAS 32nd Annual Conference / 20-minute conference paper (slides available online)

Past, Present, and Future of
Storyboarding in JapaneseAnimation
Jun Kato, Ryotaro Mihara, Nao Hirasawa (Arch Inc.)SAS 2021 (Animated Energies), Online, June 15

https://research.archinc.jp/en
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

4

Missing feature in E-Conte: collaborations• The “story corkboard”
part is completely missing• Storyboarding process has
usually been handled
secretively by a director
and not shared with others
until its completion?

More detailed comparison in SAS 2021 slides:
https://research.archinc.jp/en/publications[Images] Chris Pallant and Steven Price, “Storyboarding: A CriticalHistory,” p.50 and p.53 / Seiji Okuda, “Anime no Shigoto wa Omoshiro
Sugiru,” p.38

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

5

SAS 33rd Annual Conference / 20-minute conference paper (slides available online)

Past, Present, and Future of“Toolsmiths”
in Japanese Animation
Jun Kato (Arch Inc.)in collaboration with Yuki Koyama (Graphinica Inc.), Tatsuo Yotsukura(OLM Digital, Inc.), and Koya Imamura (Toei Animation Co., Ltd.)SAS 2022 (Animation Unlocked),Hybrid (Middlesbrough, UK & Online), June 29

https://research.archinc.jp/en
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

6

Three case studies
in the dawn of computer-aided animation

New York Institute of Technology
Computer Graphics Laboratory

Japan Computer Graphics
Laboratory

Toei Douga Co., Ltd.
(Toei Animation Co., Ltd.)

NYIT example image from https://blog.siggraph.org/2020/08/pioneering-pixels-the-nyit-computer-graphics-lab-then-and-now.html
JCGL logo from https://note.com/nahnah/n/ne7e30432c6fe
Toei Animation logo from https://corp.toei-anim.co.jp/ja/company/about_pero.html

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

7

Our research question

https://research.archinc.jp/en

Many studies of animation begin with a question about theobject—what is anime?—but I suggest a different entry point:
Who makes anime?
Ian Condry, “The Soul of Anime,” p.3

We pose yet another entry point—our research began withexploring the design space of building creativity support tools
for anime production studios, which can be summarized as:
how to make anime?

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

8

ACM CHI 2015 (Honorable Mention Award)

TextAlive:
Integrated Design Environmentfor Kinetic Typography

publicly available at https://textalive.jp
Jun Kato, Tomoyasu Nakano, Masataka Goto
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

9TextAlive in use for Hatsune Miku live performances

SNOW MIKU 2018
LIVE!（LamazeP feat.
初音ミク, 愛の詩 &TOKOTOKO feat. 初音ミク, 四角い地球を丸くする）／初音ミク「マジカルミライ2018」ライブ（DIVELA feat. 初音ミク, METEOR）／SNOWMIKU 2019 LIVE!（四角い地球を丸くする）／初音ミク「マジカルミライ 2019」
ライブ（*Luna feat. 鏡音レン, メインキャラクター）© Crypton Future Media, INC. www.piapro.net / © SEGA
Graphics by SEGA / MARZA ANIMATION PLANET INC.
Organized by TOKYO MX / Crypton Future Media, INC.

June 12, 2023

Lights, Animation, Interaction! (SAS 34)©CFM/©TOKYO MX/©SEGA

https://textalive.jp/events

10

SANRIO Virtual Festival 2023
Jan. 13-22, 2023
https://textalive.jp/events/sanrio-v-fes2023

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

11Our research question
We pose yet another entry point—our research began with
exploring the design space of building creativity support tools
for anime production studios, which can be summarized as:
how to make anime?
Jun Kato at 32nd and 33rd SAS Annual Conferences

We have provided a new creativity support tool to author lyric
videos, which has been used in production, such as live music
performances enjoyed by more than 30,000 audiences—how
do the “animating lights” contribute to the performances?
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

122. What is live musicperformance?
Lights, Animation, Interaction!
Synchronizing Music with Computer-controlled Visuals in Live Performances
Jun Kato (AIST)
Society for Animation Studies 34th Annual Conference, 6/12/2023

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

13

新・舞台照明講座 光についての理解と考察
by 岩城 保 (2022)
Title in English: [New Edition] Stage Lighting Lecture – Understanding and discussing lights

Section 8-3. Role of Stage Lighting
8-3. 舞台照明の位置づけ（p.87）• Stage lighting is a part of the performance,
not a work in itself.
舞台における照明はその舞台作品の一部分として位置づけられるのであり、照明単独で作品となるわけではない。

• If the lighting on stage pops out prominently, it is a “lightingshow” and not the real stage lighting.
舞台照明が目立って飛びだしていたら、それは「ライティングショウ」であって、本当の舞台照明ではないのです。（『舞台照明』大庭三郎 1976 p.5）

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

14

新・舞台照明講座 光についての理解と考察
by 岩城 保 (2022)
Title in English: [New Edition] Stage Lighting Lecture – Understanding and discussing lights

Section 8-3. Role of Stage Lighting
8-3. 舞台照明の位置づけ（p.87）• Given that lighting on stage is a part of the performance, todesign lighting, one must understand the intention of theartistic production holistically.
照明が舞台作品の一部分を成すのだとすれば、照明を作るためには、その舞台作品全体の意図を理解していなければならないはずです。

• Stage lighting without a good understanding of the main
body of the play, dance, musical, etc. being performed
results in the stage simply be lit, but not a true (artistic)
picture of light.
ここで最も大事なことは、舞台照明とは演劇・舞踏・ミュージカルなど上演されるその本体をよく理解していなければ、舞台に単なる光は出ても、本当の光の絵にはならないむずかしさがあることです。（『舞台照明』大庭三郎 1976 p.12）

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

15

コンサートライティング入門by 加藤 憲治 (2014)
Title in English: Introduction to Concert Lighting Design[p.7] A stage of atheatrical performancewith photorealistic stage
sets[p.7] A concert stage
designed with beams oflight[p.7] 光のビームでデザインされたコンサートのステージ(Photo in courtesy of 劇団昴)[p.7] 写実的な舞台装置がセットされた演劇の舞台（写真提供＝劇団昴）

June 12, 2023

• Theatrical performance lighting isbased on the script, which
creates the stage scenes inaccordance with the story.

• Concert lighting needs to be
based on the music to be
performed, therefore challenging
the imagination.

• 演劇の舞台照明のデザインの基になるのは台本であり、ストーリーに沿って、場面の情景を舞台の上につくり出していきます。

• コンサートライティングでは、演奏される音楽を基にしながら、そこからイメージされるシーンをつくることが求められます。

Lights, Animation, Interaction! (SAS 34)

16

コンサートライティング入門by 加藤 憲治 (2014)
Title in English: Introduction to Concert Lighting Design[p.7] A stage of a theatricalperformance with photorealistic
stage sets (Photo in courtesy of 劇団昴)[p.7] A concert stage designed withbeams of light[p.7] 光のビームでデザインされたコンサートのステージ[p.7] 写実的な舞台装置がセットされた演劇の舞台（写真提供＝劇団昴）• In theatrical performance lighting,lighting sources such as
spotlights are usually hiddenfrom the audiences.
• 演劇での舞台照明では、基本的に光源となるスポットライトなどの照明器具が、観客から見えないようにセッティングされています。

June 12, 2023

• In concert lighting, the sources
are part of the stage sets; notonly their existence is visible, but
their animations are part of theexpressions.
• コンサートではスポットライトなどの照明器具も、ステージセットの一部と考えられてステージがつくられています。また、光源としての照明器具の灯体はもちろんのこと、明かりの動きや点滅などを直に観客に見せ、これを表現のひとつとして使用しています。

Lights, Animation, Interaction! (SAS 34)

17

Live music, stage, and lighting
as “the animated environment”
コンサートライティング入門by 加藤 憲治 (2014)
Title in English: Introduction to Concert Lighting Design• [p.100] When sound and light are synchronized, thesound is created by the musicians and sound staff,
and the light is created by the lighting staff, but theaudience does not perceive them separately. The
audience perceives the entire synchronized sound andlight as “one event” (with a certain kind of “reality”)•[p.8] The stage where the light source of the luminiaire
becomes part of the design[p.8] 照明器具の光源がデザインの一部となったステージ
June 12, 2023

[p.100] 音と光が同期している時、実際には音はミュージシャンと音響スタッフが、光は照明スタッフがつくりだしているのですが、観客はそのように分けては捉えません。観客は音と光が同期した全体を「一つの出来事」として受け取ります。

新・舞台照明講座 光についての理解と考察
by 岩城 保 (2022)
Title in English: [New Edition] Stage Lighting Lecture – Understanding anddiscussing lights

Lights, Animation, Interaction! (SAS 34)

18

3. Full-color pixels:
from abstract to concreteLights, Animation, Interaction!
Synchronizing Music with Computer-controlled Visuals in Live Performances
Jun Kato (AIST)
Society for Animation Studies 34th Annual Conference, 6/12/2023

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

19

Pixels on the stage
Gobo (lighting)

Vari-Lite(2023, May 13). In Wikipedia.

(2023, May 19). In Wikipedia.

• Conventional stage lighting devices such as spotlights can beregarded as huge and bright pixels, which used to be static,
and their intensity was the only parameter to be controlled• Concrete images can be projected using “Gobo”, and theirmotion became controllable with the Vari-Lite debut in 1981
• Digitally controlling protocol DMX512 became generally
available in 1990s
ステージ＆舞台照明マニュアル by 藤井 直 (2019)
Title in English: Stage & Stage Lighting Manual
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

20

Digitization of lighting design• After the standardization of the DMX protocol, lots of lighting
simulation/controlling/programming software appeared• DMX can handle 512 8-bit channels, but can easily fall in
short when controlling complex lighting devices these days

https://www.avolites.com/avolites-announces-immediate-availability-of-powerful-new-titan-v15-software/
https://www.avolites.com/avolites-announces-the-release-of-new-titan-v16-0-software/

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

21

Lighting sources in the post-digital era© Crypton Future Media, INC. www.piapro.net / © SEGA
Graphics by SEGA / MARZA ANIMATION PLANET INC.
Organized by TOKYO MX / Crypton Future Media, INC.

June 12, 2023

• LED lighting sources
can change their colors
quickly• LED displays andprojectors can playvideos synchronizedwith music• Glowsticks enable
active participation ofthe audiences

Lights, Animation, Interaction! (SAS 34)

22

LED displays:
from abstract lighting to concrete animations
コンサートライティング入門by 加藤 憲治 (2014)
Title in English: Introduction to Concert Lighting Design

METEOR / DIVELA feat. Hatsune Miku

• [p.144] Sometimes more can be conveyed by videos, anddepending on the performed music, the stage features
the video, regarding conventional lighting as support. …
We believe that a total direction method that isconscious of the overall visuals, including lighting, video,
and stage set, will be more required in the future.
•[p.144] 映像による表現の方がより多くのことを伝えられる場合もあり、演奏される曲目によってはステージの表現を映像をメインにし、ライティングはサポートのような表現になることもあります。...これからはライティング、映像、ステージセットと全体的なビジュアルを意識したトータルな演出の方法が、より求められてくると思います。

METEOR / DIVELA feat. Hatsune Miku

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

23

Glowsticks, wristbands, and smartphones:
from the passive to active audience• The audience can wave glowsticks, wear wristbands and holdsmartphones automatically changing colors to music• We even allowed the audience to interact with theanimations played on the smartphones [Songle Sync, 2018]https://magicalmirai.com/10t
h/goods_nic_penlight.html

June 12, 2023

Xylobands.

Wham City Lights.

Songle Sync. https://junkato.jp/songle-sync andhttps://xylobands.com

https://vimeo.com/86363873

https://youtu.be/dCmlEwDTkDY

Lights, Animation, Interaction! (SAS 34)

24

ACM Multimedia 2018 (Oral presentation)

publicly available at https://api.songle.jp/sync
Jun Kato, Masa Ogata, Takahiro Inoue, Masataka Goto
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

25

4. Role of “Animators”
Lights, Animation, Interaction!
Synchronizing Music with Computer-controlled Visuals in Live Performances
Jun Kato (AIST)
Society for Animation Studies 34th Annual Conference, 6/12/2023

June 12, 2023

Lights, Animation, Interaction! (SAS 34)

26

Virtual stages:
Musical performances in post-COVID19 era
MIKU LAND 2023
New Beginning
Happy Coaster / emon (Tes.) feat.
Hatsune Miku
https://textalive.jp/events/mikuland2023June 12, 2023

Lights, Animation, Interaction! (SAS 34)

27

Lights, animation, interaction!
– or the information overload for the audiences?

© Crypton Future Media, INC. www.piapro.net / © SEGA
Graphics by SEGA / MARZA ANIMATION PLANET INC.
Organized by TOKYO MX / Crypton Future Media, INC.

https://textalive.jp/events/mikuland2023• Computers are controlling lights, animation, and interaction for
more immersive and active music listening experiences• The role of animators will be to combine these technical elements
to build the animated environment surrounding the audience
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

28

Lights, Animation, Interaction!
Synchronizing Musicwith Computer-controlled Visuals
in Live Performances
Songle Sync wasused to enable thisinteractiveperformance held at
Magical Mirai 2019.

TextAlive was used
to produce the lyric
video for the MIKU

https://junkato.jp/songle-sync

https://junkato.jp/text
alive

https://youtu.be/dCml
EwDTkDY

https://textalive.jp/eve
nts/mikuland2023

LAND 2023
attraction.

Society for Animation Studies 34th Annual Conference, 6/12/2023

Jun Kato
National Institute of Advanced Industrial Science and Technology (AIST)
https://junkato.jp
June 12, 2023

Lights, Animation, Interaction! (SAS 34)

29