IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

1

PAPER

DanceUnisoner: A Parametric, Visual, and Interactive Simulation
Interface for Choreographic Composition of Group Dance
Shuhei TSUCHIDA†∗a) , Satoru FUKAYAMA†b) , Jun KATO†c) , Hiromu YAKURA††,†d) , Nonmembers,
and Masataka GOTO†e) , Fellow

SUMMARY Composing choreography is challenging because it involves
numerous iterative refinements. According to our video analysis andinterviews, choreographers typically need to imagine dancers’ movements
to revise drafts on paper since testing new movements and formations
with actual dancers takes time. To address this difficulty, we present aninteractive group-dance simulation interface, DanceUnisoner, that assists
choreographers in composing a group dance in a simulated environment.
With DanceUnisoner, choreographers can arrange excerpts from solo-dance
videos of dancers throughout a three-dimensional space. They can adjust
various parameters related to the dancers in real time, such as each dancer’s
position and size and each movement’s timing. To evaluate the effectivenessof the system’s parametric, visual, and interactive interface, we asked sevenchoreographers to use it and compose group dances. Our observations,interviews, and quantitative analysis revealed their successful usage initerative refinements and visual checking of choreography, providing insights
to facilitate further computational creativity support for choreographers.
key words: Choreography, Choreographic composition, GUI-based interaction, Group dance

1.

Introduction

Street dance is a popular dance style, and dancers often
compete with each other and give unique artistic performances
in official competitions. The competitions are held as onsite
events with audiences, often recorded and broadcasted on TV
shows and online. Break dancing is a prominent example ofstreet dance. It was adopted as an official sport in the 2018
Youth Olympic Games in Buenos Aires [1] and was selected
as a candidate for inclusion in the 2024 Olympics in Paris [2].
Competitions can employ a variety of rules depending on thenumber of dancers. For instance, the International Dance
Organization categorizes competitions into “solo,” “duo,”“crew” involving 3 to 7 dancers, “formation” involving 8 to 24
dancers, and “production” involving 25 dancers or more [3].
To win, choreographers must compose an appealing
dance performance well before the competition. However,
Manuscript received March 3, 2023.
† Shuhei Tsuchida, Satoru Fukayama, Jun Kato, and Masataka

Goto are with National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba-shi, 305-8568 Japan.
†† Hiromu Yakura is with University of Tsukuba, Tsukuba-shi,
305-8574 Japan.
∗ Presently, Shuhei Tsuchida is with Kobe University, Kobe-shi,
657-8501 Japan.
a) E-mail: t.sway.tmpp@gmail.comb) E-mail: s.fukayama@aist.go.jpc) E-mail: jun.kato@aist.go.jpd) E-mail: hiromu.yakura@aist.go.jpe) E-mail: m.goto@aist.go.jp
DOI: 10.1587/transinf.E106.D.1

the composition process becomes more challenging when theperformance involves more dancers. For choreographers, itwould be ideal if dancers could instantly learn a choreography they came up with and perform it without any mistakes.
Then, the choreographers could focus on making iterativerefinements to the choreography without significant interruptions. In reality, dancers cannot always gather at one
place at one time; they need to spend a certain amount oftime learning a choreography and the intention behind it, andthey often have difficulties performing it. As a result, thechoreographer usually needs to rely on a set of conventional
analog devices—sheets of pen and paper—to make iterativerefinements without the actual dancers.
The goal of our work is to support the creativity ofchoreographers of group dances with the computer, enabling
choreographies to be composed and iteratively refined without
dancers present. We analyzed existing group-dance videos
and interviews with expert choreographers to uncover designprinciples that effectively support creativity and determinedthat there needs to be a group-dance simulation interface
that is parametric, visual, and interactive (Section 2). Thesefindings led us to develop DanceUnisoner, a tool that allows
the choreographer to choose, edit, arrange, and combine
videos of a single dancer with a dedicated user interface andsimulate a group dance performance (Figure 1; Section 4).
In creativity, the cycle of imagination and its implementation
is important [4–7]. Our system supports the imagination in
the creative process and makes the process of the cycle fromimagination to implementation more efficient.
The main contributions of this work are threefold:(1) Through interviews with choreographers, we clarify
that three design principles—parametric, visual, andinteractive—could be used to support their creativity.
(2) On the basis of (1), we propose and implement a novel
interface for easily adjusting the various properties ofdancers, such as the position and size of every dancer
and the pattern and timing of every movement.
(3) Based on our user study, we provide insights on theprocess of composing a choreography with the supportof computers.
2.

Design Process

We aim at providing efficient computational creativity support
for choreographers. Toward this goal, we need to understandCopyright © 2023 The Institute of Electronics, Information and Communication Engineers

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

2

Fig. 1 DanceUnisoner allows choreographers to simulate group dances by placing and duplicating
excerpts from recorded dance videos in 3D space (a). We designed it according to three principles
informed through our video analysis and interviews (b). We observed that all seven participating
choreographers could effectively utilize it to compose group dances in their respective styles (c).

the current workflow for composing choreographies. We firstinvestigated the typical patterns appearing in choreographies
for group dancing by reviewing dance videos available online.
We also conducted interviews with choreographers in regard
to their workflow. In this section, we describe the findings anddiscuss the design principles we figured out for establishingcomputational support.
2.1

Current Workflow of Choreographic Composition

To investigate common choreographic patterns, we analyzed
a set of 94 highly-ranked videos available on YouTube andinterviewed seven choreographers about their own processes
for choreographing group dances.
2.1.1

Video Analysis

To collect group-dance videos, we searched the phrase “group
street dance” on YouTube and chose the top 137 videos. Of
these videos, we removed 36 that could not be regarded as
dancing, such as those that were interviews or audio only, andchecked a total of 101 videos. These 101 videos consistedof dance performances on a stage for competitions and forentertaining people (78 videos), video works for promoting
dance groups or choreographies (12 videos), improvised
street-dance performances (5 videos), flash mobs (3 videos),practice scenes in a studio (2 videos), and a number of people
having fun dancing outdoors in their own way (1 video). We
then excluded the five improvised videos and one video of
people dancing outdoors from the analysis because thesevideos did not fit the purpose of this paper of supportingthe creation of group dance choreography. We finally set 95
videos for analysis.
Although we had difficulty counting the number ofdancers for 13 of the videos, other dance performancesconsisted of 3 to 22 dancers. Moreover, we observed several
common characteristics of group dance among those videos
including various street-dance genres:
Choreography synchronization. In all of the dance
videos, we confirmed that there were shots in which multiple
dancers simultaneously performed to the same choreography.

Front-focused performance. In 89 out of the 95 videos,dancers performed assuming that the audience could watch
them from the front. The exceptions include large-scale
flash mobs, dance performances on the street, and solo dance
performances surrounded by an audience.
Crossing. In 75 out of the 95 videos, there werevarious formation-transition scenes, such as dancers moving
to another formation while crossing each other.
Flipping of choreography. In 66 out of the 95 videos,
we observed multiple dancers performing flipped choreography, such as symmetric choreography centered on the middleof the stage.
Shifting timing of movements. In 41 out of the 95
videos, there were moments where each dancer performed
the same movements but their start time was different. We
could see that the choreography seemed to be propagating to
the next dancer. We also observed that the number of dancers
dancing tended to increase.
In summary, the video analysis revealed representativetechniques used in choreographic compositions. However,while this analysis suggested techniques that should be covered in the computational support, the actual process used bychoreographers to compose group dances was still unclear.
To further understand the workflow of choreographers, we
conducted interviews. We interviewed both professional
and amateur choreographers who had experience with choreographing group dances to find out how they do so.
2.1.2

Interview with Choreographers

We interviewed a dance instructor with more than 9 yearsof street-dance experience and experience with group-dancecomposition, and 6 amateur choreographers (each with 3, 4,
7, 9, 9, and 16 years of dance experience) who belonged to auniversity dance club and had experience with group-dancecomposition. We asked 7 choreographers in total about theirworkflow from choreographic composition to group-dance
performance. The participants volunteered to participate
in the interview. We conducted the interviews in Japanese,and the translation was conducted by the first author. Fromthe comments obtained, we found that the choreographers

TSUCHIDA et al.: DANCEUNISONER

3

Fig. 2 (Top) Current workflow for composing choreography. It involves
iterative refinement process that includes asking dancers to gather and learn
modified version for confirmation, and thus, often takes long to get to final
performance. (Bottom) Workflow with the proposed system. Simulator cansimulate group dance and allows choreographers to create, record, confirm,
and revise group dances faster.

Modification. The choreographers then correct any
problems that were noticed. For example, if dancers do not
move smoothly, or they collide or overlap with each other,
the choreographers adjust the choreography. Moreover, tocompose an appealing dance performance well, the choreographers adjust the balance of various properties such as
the position, direction, and size of the dancers, as well as
the pattern and timing of every movement in a group dance.
After correcting the problems, the choreographers return to
the Confirmation step and repeat both steps until they aresatisfied.
One of our unique findings was that, as mentioned in the
Ideation and Recording step, the choreographers compose thepositions and movements of the dancers by visualizing choreographic information on paper. If dancers are waiting near
the choreographers throughout all steps, the choreographers
can confirm the dance they composed, correct it, and thenquickly move on to the Confirmation step again. However,choreographers cannot easily arrange for such a situation.
From one interviewee, “We cannot confirm the choreography
until the next meeting,” and in some cases, “We have to wait
until a week later.” That is, we found that the choreographers
took too much time going back to the Confirmation step fromthe correction phase because they had to wait for the dancers
to gather again.
2.1.3

Fig. 3 Notes for visualizing choreography that we collected. They wererecorded in various formats: paper, blackboard, and simple drawing software.
Overwritten cross marks (rejection marks in Japanese culture) indicate trial
and error in composing group dance.

proceeded with the workflow shown at the top of Figure 2.
Each part of the workflow is explained below.
Ideation and Recording. Choreographers decide upona musical piece and compose the dance performance. Whencomposing a dance, each choreographer uses various methods,
such as imagining the dance while listening to the musicalpiece, trying to dance to the piece, searching for dance videos
that use the same piece and referring to them, deciding upon
movements while talking with dancers, and so on. In common
with all choreographers, they visualize information such as
the movement of dancers on paper or simple drawing software(e.g., Keynote), as shown in Figure 3, and organize and record
them.
Confirmation. Since just imagining on paper is notenough, the choreographer confirms the created choreography
by watching dancers perform the piece. To do so, not only
do the choreographers have to watch the performance, butthey also film the dance to review it with the dancers or haveothers in to observe.

Questionnaire of dancers

We further conducted a questionnaire survey with dancers
regarding how they perceive such workflows, especially about
how they are instructed in the process of composing of thegroup dance. We asked 28 dancers with more than 3 years ofstreet-dance experience who belonged to a university danceclub and had experience with a group dance performance.
The participants volunteered to answer the questionnaire. As
the result, 16 out of 28 participates answered that they had
experiences they received paper-based explanations about thecomposition of the choreography, and 8 out of 28 participates
answered that they had received video-based explanations
about it. Also, one dancer mentioned that it was easier tounderstand the choreography of a group dance if they received
a video “where I can see the movements from both front andback, shot in a place with mirrors.” In addition, one dancer
commented, ‘When the choreography required each dancer
to move at different timing for each beat, it was difficult todance because the timing was different from the other dancers
and there was no sample.” We found that when dancers learngroup dance choreography, it is important to visualize theoverall image of the choreography clearly.
2.2

Design Principles

In the previous sections, the analysis of existing dance videos
showed four common characteristics found in choreography
for group dances. The interviews showed that the choreographers used pen and paper to visualize, organize, andrecord their choreography and that the choreography ideation,

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

4

recording, confirmation, and modification cycles took too
much time. In addition, we revealed that such visualization
would often be too poor for dancers to learn the choreography. These findings conversely suggest that we can leveragecomputers to free choreographers from the workload of suchtedious and time-consuming tasks (see the bottom of Figure 2)
as well as helping dancers learn choreography. For this aim,on the basis of the knowledge obtained in the previous section
and the knowledge of the first author, who has more than 10
years of street dance experience (mainly breakdancing), we
created the following three design principles.
Parametric. To express crossing, choreography flipping,
and the timing of shifting, choreographers can adjust thebalance of various dancer properties such as the position,
direction, and size of every dancer and the pattern and timingof every movement in a group dance in sync with the beat.
Visual. Choreographers can intuitively understand awhole front-focused group dance. This is also helpful fordancers who learn the composed choreography afterward.
Interactive. Choreographers can quickly and easily
apply iterative cycles of ideation, recording, confirmation,and modification through an interface.
A system that meets the above principles will allow
choreographers to refine their iterative process more efficiently, which will lead to more time spent on creative tasks
such as choreography and composition of group dance.
3.

Related Work

Our preliminary interviews revealed three principles required
for supporting choreographers with computers. However,we acknowledge that some studies have proposed systems
for supporting choreographic composition. To situate ourwork, we review related work, especially highlighting thatthe requirements have not been well supported yet.
3.1

Choreographic Composition Support Tools

Many support tools for choreographic compositions havealready been proposed, and Alaoui et al. [8] compiled thesepieces of research. COMPOSE [9], Animate Tokens [10],Life Forms [11], and CorX [12] were presented in the firsthalf of the 1990s and then expanded into various research
directions, such as one work using a motion capture system [13] and one using a stylus pen and tablet [14]. Carlson
et al. [15, 16] proposed tools that support choreography analysis by visualizing Laban efforts [17]. These systems employ
visualization with a stick man or simple polygon human. It is
desirable to visualize a dancer with an appearance similar tothat of an actual dancer, where choreographers can grasp thesubtle differences of the choreography.
Compared with single-dance choreographic composition, there is relatively little research focusing on group-dancechoreographic composition. Schulz et al. [18] proposed authoring environments for dance performances that use movement information obtained from a motion capture system.
Choreographers can compose choreography by dragging anddropping movements selected from prepared choreography
lists onto a timeline. Moreover, they can easily compose agroup dance by setting the dancer positions displayed in 2Dwith a mouse. Furthermore, the system displays the resultswith character animation, which allows choreographers to
confirm the choreography in 3D. A similar system using a
motion capture system, DanceDesigner [19], was released.
While not including dancers displayed with character animation, this system allows choreographers to transform the
movements of dancers into textual instructions for dancers.
They can also check dance videos shot from the front anda path plan including the dancer’s positions. The above
systems are suitable for the purpose of preserving and editing
detailed choreographic compositions and visualizing group
dance choreography. However, in terms of efficiency for theiterative cycles of ideation, recording, confirmation, and modification, a motion capture system is not suited for repeatedtrial and error done for the many choreographic-composition
steps in a limited amount of time. Without a motion capturesystem, though it has been possible to obtain movement information from general-purpose devices such as smartphones
equipped with a depth sensor recently, these types of devices
have difficulty accurately capturing movement information
since many errors can be included depending on the dancer’s
posture.
Some systems focus on methods for moving and changing the formations and positions of dancers on a simulator.
Systems that operate with Kinect [20] and gamepads [21]have been proposed. While looking at a group of dancers
visualized on screen, users of these systems can control theformation with their own body gestures or check the positionsof the dancers in 3D by moving the virtual camera with the
gamepad. Given that, we propose a GUI-based interface
with a trackpad that can not only control multiple dancers
at once as in the above system, but also graphically displays
the positions of dancers by duplicating boards (billboards)
on which a solo-dance video is pasted and place.
3.2

Video Editing and Broadcasting

We are inspired by projects designed to support video editing
and broadcasting. Video editing methods [22] that display
dances in 3D by using depth data have been proposed. Kurihara et al. [23] proposed a video editing system that can
express group dance in 3D by showing colored 3D point
clouds generated by combining depth information obtained
from Kinect and RGB color information. Ohta et al. [24–26]
proposed a real-time 3D video transmission technology witha simplified 3D model consisting of a single plane on which
2D athletes extracted from multiple cameras are pasted. We
presume that such techniques would also be useful for choreographic situations that choreographers imagine the dance.
That is, they can visually understand the movements andpositions of dancers in 3D by using multiple flat planes on
which 2D athletes extracted from an RGB camera are pasted.
Some video editing tools that use the synchronizationof audio and image tracks of videos have also been proposed.

TSUCHIDA et al.: DANCEUNISONER

5

Truong et al. [27] presented QuickCut, which can help inquickly creating narration videos. Leake et al. [28] proposeda system for efficiently editing video of dialogue-driven
scenes. Tsuchida et al. [29] developed an automatic systemfor editing multiple dance videos. Some content-based
editing techniques that use content synchronization have also
been proposed. Rubin et al. [30] proposed a transcript-based
speech editing tool and also presented UnderScore [31], which
automatically refines, aligns, and adjusts speech and music.
PodCastle [32] provides an interface that enables users toeasily correct speech recognition errors. In this paper, ourproposed system will utilize the synchronization between theaudio and images (dance video shots) of videos to enable
editing in beat units, which have a close relationship withdance movements.
3.3

Choreographic Communication

The creation of choreography also has a social aspect between
the choreographers and dancers communicated via choreography. Some studies have examined the ways that choreographers and dancers interact with technology. Studies [33, 34]
done through art activities and studies [35,36] aimed at design
done collaboratively through technologies between people
involved in various genres have been conducted. Additionally,
Hsueh et al. [37] examined how interactive visual systems
can support choreographic composition. Felice et al. [38]stated that tools for supporting choreographers in the creative
process should enable them to visualize and manipulate theirideas and share them with dancers and collaborators. We
anticipate that our work can complement the previous findings through its observation of how choreographers compose
group dance using the proposed system.
4.

DanceUnisoner

4.1

Prototype Implementation

We built a prototype system according to the design principles(see Section 2.2). The interface is shown in Figure 4. We
introduce the functions of the prototype system below.
4.1.1

Input data

The prototype system uses three kinds of input data: a musicalpiece used for a group-dance performance, a dance video inwhich a single choreographer or a single dancer dances along
the musical piece, and an image of the background used infilming the dance video of a single dancer. It does not requirea special device such as a motion capture system or depth
sensor. Choreographers simply film themselves dancing to apiece by using a simple RGB camera like the one attached
to a smartphone. The background behind a dancer can beremoved by subtracting an image of the background fromthe video. The image of the background can be prepared by
cutting out an image from a dance video in which dancers do
not appear. All input data was designed to be prepared easily.

4.1.2

Pre-processing

Using the audio of the input musical pieces, the prototype
system automatically trims segments in which the musicis played in the dance video and choreography appears.
Specifically, we use landmark-based audio fingerprinting [39]to automatically detect a musical piece’s starting position
in a dance video. We then cut it out as a video of one
dancer within the period of time length specified by thechoreographer from that start position. After that, the humanregion corresponding to the dancer is cut out in each frameof that dance video by subtracting the background image.
4.1.3

Group-dance display method

To display a group dance by combining videos in which asingle dancer dances, the prototype system can be used to
duplicate boards (billboards) on which a solo-dance video
is pasted and place (i.e., copies and pastes) the duplicatedboards on the floor throughout 3D space. Through loop
playback of all videos in sync with the music, the prototype
system enables multiple dancers to be visualized as if they are
dancing at the same time. An example is shown in Figure 5.
This method is easier to implement than displaying dancers
such as computer animated 3D characters and is suited tochoreographic composition since choreographers can easily
compose a group dance while imagining the visuals of an
actual group dance such as the swaying of and texture ofclothes.
4.1.4

Adjustment GUI

Focusing on the parametric aspect of choreography composition (i.e., the number, position, appearance, and movementof dancers), which is important in composing group dances,the prototype system provides a simple GUI such as a slider
and 2D pad (see Figure 4) that can be used to adjust those
properties. The choreographer can change the number andpositions of the dancers with simple GUI-based operations.
Moreover, the dancers can be flipped by horizontally, andthe timing of movements can be shifted in units of video
frames. Additionally, in group dances, the choreographer
often creates choreography to which dancers perform thesame movement shifted one beat (as suggested in Section 2.1);
therefore, the prototype system makes it possible to shift thetiming of dancers’ movements in beat units according to thepositions of the dancers (shifting by 1, 2, or 4 beats can beselected).
4.2

Interview with Expert

To arrange the functions necessary for composing group
dances, we asked an expert choreographer with 18 years of
hip-hop dance experience to use the prototype system and toallow us to interview her about the functions that needed to
be improved or added.

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

6

Fig. 4 Interface window. Choreographer can adjust in real time elements (dancer position, size, andtiming of movements) that should be considered when composing group dance while overviewing wholechoreography.

even if all dancers have not gathered, so I feel like the choreographing speed will become significantly faster [...] I feellike I can make a choreography in half the time.” However, the choreographer pointed out that one function needs
improvement and one function is missing.
4.2.1

Fig. 5

An example of displaying dancers placed in 3D space.

The interview took 3 hours. For the first hour, thechoreographer received an explanation on how to use theprototype system, and she then imagined and composed agroup dance based on a musical piece that had 32 beats (4beats × 8 measures in four-four time), where the term “beat”
denotes a quarter note. After the choreographer organized
the choreography, we shot her dancing to that choreography.
We imported the video into the prototype system, and thechoreographer then visualized the group dance in 3D spaceby using the system. For the remaining two hours, we interviewed the choreographer regarding which functions shouldbe improved and what functions are required while asking
how a group dance is usually composed. The choreographer was compensated Y=15,000 (JPY) for taking part in theinterview.
Through the interview, the choreographer expressed herpleasure with each of the functions built in accordance witheach of the three design principles. In particular, for thedesign principle interactive, from the viewpoint of composing a group dance efficiently and quickly, the choreographer
positively commented “I can see the whole choreography

Fine-grained timing control

The choreographer requested that we allow for finer settings
regarding the dancer properties along the timeline. The
prototype changes the properties for an entire section of an
input dance video. For example, if the choreographer shifts
one beat against the timing of the movements of a dancer,that dancer always dances shifted one beat while the video isplaying. However, it is rare for the dancer properties to notchange during an entire section of choreography. Although
the choreographer agreed with the current functions, suchas the simple GUI based on the design principle parametric,
she commented that “If I could set the choreography finely,
I wouldn’t feel bothered re-filming the video or changing
the properties,” and we determined that the choreographer
needed additional functions to be able to set properties witha finer granularity along the timeline.
4.2.2

Path planning

The prototype interface cannot be used to plan the movement
paths of dancers. Since the interface places a dance video
in 3D space, choreographers are allowed to experiment withboth left-and-right and back-and-forth movements. However,these movements cannot be easily edited on the prototype
system. From comments such as “When dancers dance forthe first time, they face problems such as colliding into eachother and movements not reaching the designated position,”
the system must avoid such problems by making it possible
to confirm and edit choreography including its movements inadvance.

TSUCHIDA et al.: DANCEUNISONER

7

We refined the prototype system into DanceUnisoner†on the basis of the comments from the interview. Threemodes, Setup, Dancer Parameters, and Position 2D Map,
were added as a result of the improvements besides Formationmode, which allows users to change the number of dancers
and their position as the prototype system does.
4.3

User Workflow

We introduce the user workflow of DanceUnisoner. The
choreographer first films themselves dancing to their choreography while imagining the choreography as a group dance.
We pre-process the captured video into a form that can beused with DanceUnisoner in the same way as the prototype
system.
When a choreographer launches DanceUnisoner, Setup
mode is displayed. In this mode, to set the position of
the dancers on the z-axis in 3D space, the choreographer
can manually add depth information to the videos shot by
adjusting the depth slider (see Figure 6). This makes it
possible to display and move dance videos in 3D space. The
choreographer adjusts the depth information by using the
GUI so that the movements displayed look as much like theactual movements of a group dance as possible.
The choreographer then explores the initial formationof dancers. Formation mode has the same functions as the
prototype system, and the choreographer can easily adjust the
number of people and design a formation. Position 2D Map
mode shows the layout of the dancers’ positions as seen fromabove (i.e., overhead view). The choreographer can freely
move and set the positions of the dancers with a mouse.
After setting the initial formation of dancers, the choreographer switches from Position 2D Map mode to Dancer
Parameters mode. In this mode, the choreographer can edit
dancers’ properties such as the kinds of choreography (preshot dance videos with different choreography), flipping, andshifts in beats. The choreographer can compose a group
dance while constantly switching between Position 2D Map
mode and Dancer Parameters mode.
We give details on the functions of DanceUnisoner in
the next section.
4.4

Implementation

We modified DanceUnisoner based on the improvement(Section 4.2.1) and missing point (Section 4.2.2) that thechoreographer pointed out in the previous section. We
implemented DanceUnisoner using openFrameworks 0.9.8
and conducted the operation check on macOS Sierra.
4.4.1

Setup

In Setup mode, the choreographer can manually set depth
information, which is difficult to estimate with only RGB
video. On the interface window shown in Figure 6, dancers† https://youtu.be/sMYKIA6DSJQ

are displayed translucently for overviewing the movementsof the whole video at every four beats of the musical piece.
Moreover, the top right part of the screen shows an overhead
view of the dancers’ positions, and that view displays thedancers’ movement trajectories for eight beats in 3D space.
When the music starts playing, the main dancer begins
to dance. The choreographer can set the depth informationof the dancers by moving the depth slider on the upper rightside of the window (see Figure 6) up and down according to
the main dancers’ movements. The choreographer can also
increase the level of detail of the movements by clicking theline chart that is displayed on the timeline of the interface.
Furthermore, in the original input video, dancers become
smaller if they move to the back away from the camera, andtheir left and right movements become shorter. Similarly,
when moving toward the camera, they become bigger, andthe left and right movements become longer. Therefore, thechoreographer can adjust these differences by controlling theslider on the interface.
4.4.2

Position 2D map

The Position 2D Map mode allows the choreographer to assign
and confirm dancer positions from an overhead view. The
choreographer can select groups of dancers by clicking anddragging a rectangular selection box comprised of triangles
over dancers. They can then change the dancers’ positions
by dragging the rectangle (see Figure 7). This mode also
allows the choreographer to rearrange selected dancers in aformation such as a circle or a triangle that fits the size ofthe designated rectangular area with a single button. The
choreographer can enlarge and reduce the formations by using
the 2D pad. They can also apply beat shifts and flips (if thedancer is facing to the right, we can flip them so that they
face to the left) to the selected dancers like with the prototype
system functions.
4.4.3

Path PlanningThis mode also allows the choreographer to register the destination the choreographer lets the dancers move to. The
choreographer can use the registered formation in Dancer
Parameters mode, which will be explained in the next paragraph, and let dancers move to the designated destinations.
This function enables the dance video to visually imitate themovements dancers make when walking in 3D space, solving
the Path Planning problem mentioned in Section 4.2.2.
4.4.4

Dancer parameters mode

To set properties with a finer granularity along the timeline
than the low granularity mentioned in Section 4.2.1, thismode allows the choreographer to adjust the properties perbeat. First, the choreographer selects the dancer that thechoreographer wants to adjust with the slider labeled Dancer
ID at the top left of the interface (see Figure 8). By dragging
a mouse pointer across the rectangular frames at the top of

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

8

Fig. 6 Setup mode. Choreographer can manually set depth information of dancer’s movements, whichallows them to edit movements of dancer in two dimensions. Prototype system did not support editing of
this information.

Fig. 7

Position 2D map mode. The choreographer can arrange the dancers’ position.

the interface window, the choreographer can designate thebeats that the choreographer wants to adjust.
5.

User Study

The goal of our work is to support choreographers’ creativity by providing an efficient choreographic process withcomputers. To investigate how DanceUnisoner can contribute to choreographers and whether the system makes
the choreographic process more efficient, we asked sevenchoreographers to choreograph group dances by using the
proposed system. We recruited 7 participants (2 males and 5
females) who had between 4.5 to 18 years of dance experience(average = 10 years). Their experience covered a variety ofdance genres such as locking, hip-hop, popping, breaking,house, jazz, waack, and ballet dance. One of them was thesame choreographer who participated in the above interview.
We conducted the user study in a meeting room withan area of about 6 m × 9 m. Participants could move freely

in the room, which had chairs and tables. Each participant
used DanceUnisoner, and it ran in openFrameworks 0.9.8 on
a MacBook Pro (Retina, 15-inch, Mid 2015). The study was
conducted for each choreographer and lasted for five days.
The participants were compensated Y=15,000 (JPY) for taking
part in. We conducted the interviews in Japanese, and thetranslation was conducted by the first author.
5.1

Task

Each participant simulated two group dances of seven dancers
in DanceUnisoner. They choreographed to the same musicalpiece that had a duration of 32 beats (4 beats × 8 measures infour-four time), at a tempo of 90 beats per minute. To havethe participants use the distinct functions of DanceUnisoner,
we asked them to compose a group dance that included thefollowing four elements.
• Back and forth movements.
• Left and right movements.

TSUCHIDA et al.: DANCEUNISONER

9

Fig. 8 Dancer parameters mode. Choreographer adjusts dancers’ properties (stop, flip, etc.) for each beat. “Stop” is written with S, and “flip” isrepresented by reversed numbers (numbers mean difference in beats). Whenchoreographer inputs video, choreography number is assigned to every beatof video. Choreographer then shifts timing of movements on basis of thisnumber.

• Flipping of choreography horizontally.
• Shifting of the timing of movements.
5.2

Fig. 9 Pick-up screen shots of the output videos that the participants
created.

their heads in DanceUnisoner by using the videos shot during
the second activity. We stayed near participants so that we
could answer questions whenever they did not understandhow to use the interface. For this activity, pen and paper werealso available.

Procedure
5.2.5

The participants performed the following five activities. Completing all activities took about 3.5 hours. Participants took ashort break between activities.
5.2.1

Explanation (15 min)In order for the participants to understand what functions areincluded in DanceUnisoner, we gave a verbal explanationof about 15 minutes to them while actually operating theinterface.
5.2.2

Composition and filming (30-50 min)

The participants first imagined the choreography of a group
dance as per usual. While composing the dance, we provided
them with pen and paper, which they were free to use. At theend, we filmed them dancing as an element for constructing
the choreography.
5.2.3

Training (30 min)So as to familiarize themselves with controlling the interface,
the participants practiced simulating the group dance inDanceUnisoner with pre-prepared videos while imagining
the choreography in their heads. We answered any questions
on how to use DanceUnisoner during the practice. During
this activity, we pre-processed the recorded videos and madeDanceUnisoner ready for use.
5.2.4

Simulating choreography (60–90 min)

The participants simulated the choreography imagined inInterview (30 min)

We asked participants what the benefits are of using DanceUnisoner, what functions are convenient, and what functions
are essential.
After all participants completed the above activities, weshared the output videos created by each participant among
the participants. To predict how using DanceUnisoner may
affect communication between choreographers and dancers(via a choreography), we collected comments regarding thequestion: “When you watched the videos created by otherparticipants from a dancer’s point of view, were there any
points of concern or improvement?”.
Here, we recorded the interface windows while theparticipants used DanceUnisoner and logged the participants’interaction with it. We also recorded the voices of theparticipants in the interviews and gathered the pieces of
paper used in composing and simulating group dances. We
anonymized the data, and we refer to participants as P1-P7.
5.3

Results

We confirmed that all participants successfully simulated
their imagined choreography. Some of the output of theparticipants is shown in Figure 9. Moreover, we received
various positive comments from all participants: “Easyto imagine because I can see the video with my real eyes,”“Convenient,” “I understand the impressiveness of group dance,”“It is good to be able to confirm the music, composition, andchoreography at the same time,” “I can notice the subtle
difference of movements,” “most functions are essential,” “I
can see what I visualize in my head with the video.”IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

10

5.3.1

User feedback

Table 1 The operation logs of DanceUnisoner. The number of operations
in which each function was used is displayed.
id

Looking at the feedback in terms of the efficiency of choreographic composition, we found that three participants (P2, P6,
and P7) kept on changing the choreography while adjusting
the group dance simulation. P6 commented, “When I feltthat this part includes few [formation] movements, I added aplus alpha [forth and back movements] to the choreographic
composition.” As this participant simulated a group dancewith the system, the participant noticed and supplemented
the parts that lacked formation movements. We found that
DanceUnisoner made it possible to confirm choreography ina way that could not be done without dancers and allowed theparticipants to modify their compositions. P7 said, “As I use
the system [DanceUnisoner], I can notice things that I neverthought of in my head and put them on paper.” The simple
GUI of DanceUnisoner allowed them to easily change theproperties of dancers, so they could check various patterns
instantly. This feature may make it easier to notice newpatterns in choreography. P2 commented, “I got inspirationfrom seeing the actual movements of dancers [on DanceUnisoner].” The gap between imagination and simulation
possibly affected the participants’ creativity. DanceUnisoner
could not only accelerate the confirmation and modification
cycles of choreography as we expected but could also affecttheir creativity.
Four participants (P1, P3, P4, and P5) focused on using
the system to modify a group dance they were imagining intheir head. They mainly used the interface to confirm theirchoreography and did not change their overall compositionstyle. P1 said “I will first make the choreography basedon music and then use this [DanceUnisoner],” and P4 alsosaid, “I do not think I will change how I compose choreography [...] need a mirror and earphones [...] I may use it[DanceUnisoner] as a tool for confirming and sharing choreography.” What they had in common is that they modified
their choreography while visually checking the choreography
in DanceUnisoner after creating a group-dance simulation.
5.3.2

Quantitative analysis

To examine the effectiveness of DanceUnisoner, we analyzed
the operation logs from the user study, as shown in Table 1.
EX year indicates the number of years of dance experience.
Since one choreography log of P1 was not taken, a dash isdisplayed in the respective boxes.
The logs revealed that the formation function was mostfrequently used. The range of group dances that could be
expressed is narrowed when using only the original movements in the input video, so the choreographers struggled tocompose a variety of group dances. In addition to the movements of dancers, the path planning function mentioned in
Section 4.4.2 was essential as we learned from the interview
with the professional (in Section 4.2).
We found that the flip function was the second mostfrequently used function. Four participants (P3, P4, P6,

EX
year

P1

7.5

P2

8

P3

17

P4

18

P5

4.5

P6

6

P7

7.5

time
(min)
31
15
10
11
32
55
11
15
14
20
36
31
60
22

Fig. 10

film

flip

stop

1
2
2
4
1
1
1
1
1
1
3
2
2
1

0
–
24
8
24
0
0
48
56
24
4
28
24
26

9
–
0
0
0
14
8
0
0
18
0
0
13
0

beat
shift
58
–
0
0
0
86
4
0
0
13
0
0
8
0

formation
28
–
0
28
56
66
0
28
84
14
21
30
80
56

Papers written by P5 and P6.

and P7) commented that this function was most convenient.
When composing group dance, choreographers have many
opportunities to flip choreography as can be seen from thevideo analysis in Section 2.1.1. We can say that this function
is essential when designing a system similar to the current
one.
Moreover, when using the beat shift and stop functions,the time spent on composing group dance was relatively
longer than when not using it. The functions might haveconfused the participants since the parts of the choreography
were skipped or the positions were shifted from the original
position when the beat was shifted. For beat shift and stopfunctions, a clearer interface would be required.
To summarize, we found that all choreographers wereable to simulate their choreography by using DanceUnisoner
and appreciated its dedicated support for choreographic composition. Moreover, we found that three choreographers not
only accelerated the making of iterative refinements to theirchoreography but also had their creativity stimulated by thesystem.
6.

Discussion

To facilitate further computational support for choreographers, we describe insights gained through feedback from theparticipants of the user study.
6.1

Using DanceUnisoner Together with Pen and Paper

Participants were free to use pen and paper while using
DanceUnisoner. All choreographers except for P2 wrote floor

TSUCHIDA et al.: DANCEUNISONER

11plans (diagrams that represent spatial trajectories, as seenin Figure 10) by using symbols such as circles and arrows.
In addition, as shown in Figure 10, P5 and P6 wrote words
recalling the functions of the system, such as inversion, left,right, top, and bottom. They seemed to take advantage of thefunctions of DanceUnisoner by writing the shifts of beats andexpressing differences in the choreography with numbers andcolors. Three participants (P3, P5, and P6) wrote charactersand short sentences. P5 commented, “I think it will be [more]convenient if I can add words [...] if I can write points [onDanceUnisoner],” and asked for functions for adding words.
6.2

Difference in Number of Shots

We observed three major patterns in how the participants
used the input video.
(1) The participants simulated a group dance with only one
video of a single dancer dancing.
(2) To change how the dancers looked, the participants
danced to the same choreography multiple times.
(3) The participants shot multiple different choreographies
to obtain necessary shots.
For item (1), five participants (P1, P3, P4, P5, and P7)
composed a group dance with one video. P7 regarded thedancer as a controllable object that can represent choreography and said that she wanted to rotate the dancer with theinterface instead of taking multiple video shots.
Regarding item (2), since DanceUnisoner can only capture video shot from the front, P2 danced to the same choreography twice to simulate the choreography as shown fromthe back. P2 and P5 commented, “dance video shot frombehind” is necessary to make it “easy to learn choreography.”
As for P4, she requested that “inversion is not only left andright but also back and forth,”, and we confirmed that there
exists choreography that is difficult to express by using only
video shot from the front.
For item (3), three participants (P1, P6, and P7) increased
the number of shots of different choreographies to adjust
elements that cannot be changed by DanceUnisoner, such asheight differences between dancers.
To summarize, by regarding the dancer as a controllable
object that can represent choreography, the interface could
be expanded to offer more ways of expressing a composition.
Moreover, we could design the interface to be able to automatically generate a dancer’s back from video shot from thefront and to allow the choreographer to modify the dancer’s
posture directly in the video.
6.3

Large-scaled Group Dance Choreography

After creating the choreography, all participants mentionedthat DanceUnisoner would be particularly beneficial to thechoreographer who composes a group dance with a large
number of dancers. The participants commented about the
number of people, 10 to 40 dancers, and P4, P5, and P6
commented about 20 dancers.

Five participants (P1, P2, P3, P4, and P7) mentionedthat they could visualize large-scale choreography, evenwith a large number of dancers, for which there are “fewopportunities for all members to gather”. In particular,
P4 commented that even if there were a lack of dancers,
the choreographer could compose a group dance with theregular number of dancers by duplicating the dancers with
DanceUnisoner.
Four participants (P2, P3, P4, and P5) said that DanceUnisoner would help to “communicate” the “image” of acomposition to dancers. The larger the number of dancers, theharder it becomes for the choreographer to imagine how thedance group looks. The participants pointed out that it would
be useful to practice choreography with an output video thatcan be regarded as the goal. Also, from the viewpoint ofteaching, P2 and P4 commented that the interface seemeduseful in “understanding” the choreography of group dances
comprised of a large number of dancers.
6.4

Sharing Output Videos Among Choreographers

P5 noted that choreographers could share DanceUnisoner
among themselves to avoid problems when sharing choreography. Also, P7 mentioned that she wanted to registeran excellent choreographic composition into DanceUnisoner
and analyze the formation movements and the choreographyon the interface rather than watching the video.
DanceUnisoner has the potential to provide opportunities
to receive advice and modification from other choreographersthrough the sharing of choreographies. As future work, it isbeneficial to extend DanceUnisoner to allow choreographers
to share, analyze, and polish their compositions, resulting in
more varied choreographies. For example, DanceUnisoner
could store shared choreographies in a library so that choreographers can reuse them for a different musical piece. Thisconfirms our design principles that were partially informedby the survey in Section 2.1.3.
7.

Limitations and Future Work

Despite the affirmative responses from the participants, there
are some limitations with DanceUnisoner. First, DanceUnisoner demands dance skills of the choreographer since ituses video of them dancing. When the choreographer does
not have enough skills, they would need to ask another dancer
to dance for the video. Even if the choreographer could
dance, they might not be able to imitate other dancers’ styles
and also their appearance. To address this issue, it would
be helpful to develop a function that can support a model ofdancer’s individual characteristics and apply it to dances.
Second, DanceUnisoner has a limitation with the 2D billboard on which a solo-dance video is pasted. As discussed in
the video analysis in Section 2.1.1, most dance performances
are assumed to be seen from the front, but in some cases, theaudience surrounds the dancers. Even if the choreographer
using DanceUnisoner would want to check a performancefrom the side, they would barely be able to see the dancers

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 202312

because of the 2D billboard. To address this issue, a functionof representing the dancers’ movements and appearances in
3D space would be necessary. In addition, DanceUnisoner
has difficulty representing dance performances in which somephysical objects are used by dancers. To display dancers
throwing a physical object like a ball, for example, a functionof supporting such a 3D model would be necessary.
Third, although our user study illustrated the usefulness
and creative aspect of DanceUnisoner, its generalization toother dance genres is left as future work since this paper targets
the choreography of group dances in street dance genres.
Further investigation on observing how choreographers inother genres (e.g., K-pop, salsa, and cheerleading) utilize
DanceUnisoner would indicate areas of improvement. We
know that DanceUnisoner cannot support every genre sincesome genres involve movements improvised by dancers duringperformances. Even a musical piece could be improvised (i.e.,
not pre-recorded). Such improvisational nature is out of thescore of this research and is not supported by DanceUnisoner.
8.

Conclusion

We have described DanceUnisoner that enables a single choreographer to easily compose, by leveraging videos in whicha single dancer dances, group dance videos that look as if
multiple dancers are dancing. We first investigated the current group-dance choreographic composition, figured out theexisting group-dance composition issues, and determined design principles. Based on the design principles, we developed
the prototype system and conducted a professional interview
where an expert choreographer used the proposed interface
on the system. On the basis of comments obtained from theinterview, we improved the interface and built DanceUnisoner.
To investigate how choreographers use DanceUnisoner, we
conducted a user study with seven professional choreographers. We confirmed that all of them could simulate a group
dance by themselves; in particular, three of them utilized theiterative refinement process, which cannot be done efficiently
without DanceUnisoner.
Acknowledgments
This work was supported in part by JST ACCEL Grant
Number JPMJAC1602, Japan.
References[1] International Olympic Committee, “Head over heels about dance
sport.” https://www.olympic.org/news/head-over-heelsabout-dance-sport, 2020. Accessed on Mar. 1, 2023.
[2] The Washington Post, “Break dancing among new events proposed for2024 olympics.” https://www.washingtonpost.com/sports/
2019/02/22/break-dancing-among-new-events-proposedolympics, 2020. Accessed on Mar. 1, 2023.
[3] International Dance Organization, “IDO dance sport rules & regulations.” https://www.ido- dance.com/ceis/ido/rules/
competitionRules/2022.09-IDO-Rule-Book.pdf, 2022. Accessed on Mar. 1, 2023.
[4] R.A. Finke, T.B. Ward, and S.M. Smith, “Creative cognition: Theory,

research, and applications,” MIT press Cambridge, 1992.
[5] M.A. Runco and I. Chand, “Cognition and creativity,” Journal of
Educational psychology review, vol.7, no.3, pp.243–267, 1995.
[6] M. Botella, V. Glaveanu, F. Zenasni, M. Storme, N. Myszkowski,
M. Wolff, and T. Lubart, “How artists create: Creative process andmultivariate factors,” Journal of Learning and Individual Differences,vol.26, pp.161–170, 2013.
[7] M.A. Mace and T. Ward, “Modeling the creative process: A groundedtheory analysis of creativity in the domain of art making,” Creativity
research journal, vol.14, no.2, pp.179–192, 2002.
[8] S.F. Alaoui, K. Carlson, and T. Schiphorst, “Choreography as mediatedthrough compositional tools for movement: Constructing a historicalperspective,” Proceedings of the ACM International Workshop on
Movement and Computing (MOCO ’14), pp.1:1–1:6, 2014.
[9] T. Schiphorst, T. Calvert, C. Lee, C. Welman, and S. Gaudet, “Tools
for interaction with the creative process of composition,” Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems(CHI ’90), pp.167–174, 1990.
[10] J.H. Bradford and C.L. Paulette, “Animate tokens: Their design andapplication to dance,” Journal of IEEE Transactions on Learning
Technologies, vol.24, no.5, pp.557–562, 1991.
[11] T.W. Calvert, A. Bruderlin, S. Mah, T. Schiphorst, and C. Welman,“The evolution of an interface for choreographers,” Proceedings of the
INTERCHI ’93 Conference on Human Factors in Computing Systems(INTERCHI ’93), pp.115–122, 1993.
[12] J.H. Bradford and P. Côté-Laurence, “An application of artificial
intelligence to the choreography of dance,” Journal of Computers andthe Humanities, vol.29, no.4, pp.233–240, 1995.
[13] J.C.P. Chan, H. Leung, J.K.T. Tang, and T. Komura, “A virtual reality
dance training system using motion capture technology,” Journal ofIEEE Transactions on Learning Technologies, vol.4, no.2, pp.187–
195, 2011.
[14] M.C. Felice, S.F. Alaoui, and W.E. Mackay, “Knotation: Exploringand documenting choreographic processes,” Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems (CHI ’18),
pp.448:1–448:12, 2018.
[15] K. Carlson, T. Schiphorst, and C. Shaw, “Actionplot: A visualization tool for contemporary dance analysis,” Proceedings of theInternational Symposium on Computational Aesthetics in Graphics,
Visualization, and Imaging (CAe ’11), p.113–120, 2011.
[16] K. Carlson, T. Schiphorst, and P. Pasquier, “Scuddle: Generating
movement catalysts for computer-aided choreography,” Proceedingsof the Second International Conference on Computational Creativity,
pp.123–128, 01 2011.
[17] I. Bartenieff and D. Lewis, Body movement: Coping with the environment, Routledge, 2013.
[18] A. Schulz, W. Matusik, and L. Velho, “Choreographics: An authoring
tool for dance shows,” Journal of Graphics Tools, vol.17, no.4, pp.159–
176, 2013.
[19] Dance Designer, “Choreopro.” https://www.capterra.com/p/
128303/Dance-Designer/, 2010. Accessed on Mar. 1, 2023.
[20] A. Soga and I. Yoshida, “Interactive control of dance groups using
kinect,” Proceedings of 2014 International Conference on Computer
Graphics Theory and Applications (GRAPP ’15), 2015.
[21] A. Soga and I. Yoshida, “Simulation system for dance groups using a
gamepad,” Proceedings of the International Conference on Computer
Graphics Theory and Applications and International Conference on
Information Visualization Theory and Applications (GRAPP/IVAPP’12), pp.365–368, 2012.
[22] T. Hirai, S. Nakamura, T. Yumura, and S. Morishima, “Vrmixer:
Mixing video and real world with video segmentation,” Proceedingsof the 11th Conference on Advances in Computer Entertainment
Technology (ACE ’14), pp.30:1–30:7, 2014.
[23] T. Kurihara, M. Okabe, and R. Onai, “Ddmixer2.5d: Drag and drop
to mix 2.5d video objects,” Proceedings of the Adjunct Publicationof the 26th Annual ACM Symposium on User Interface Software andTechnology (UIST ’13 Adjunct), pp.69–70, 2013.

TSUCHIDA et al.: DANCEUNISONER

13[24] Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama, “Live
3d video in soccer stadium,” International Journal of Computer Vision,
vol.75, no.1, pp.173–187, 2007.
[25] T. Koyama, I. Kitahara, and Y. Ohta, “Live mixed-reality 3d video insoccer stadium,” Proceedings of the 2Nd IEEE/ACM International
Symposium on Mixed and Augmented Reality (ISMAR ’03), p.178,
IEEE Computer Society, 2003.
[26] I. Kitahara and Y. Ohta, “Scalable 3d representation for 3d video
display in a large-scale space,” Proceedings of the IEEE Virtual
Reality (VR ’03), pp.45–52, 2003.
[27] A. Truong, F. Berthouzoz, W. Li, and M. Agrawala, “Quickcut:
An interactive tool for editing narrated video,” Proceedings of the
29th Annual Symposium on User Interface Software and Technology(UIST ’16), pp.497–507, 2016.
[28] M. Leake, A. Davis, A. Truong, and M. Agrawala, “Computational
video editing for dialogue-driven scenes,” Journal of ACM Transactions on Graphics, vol.36, no.4, 2017.
[29] S. Tsuchida, S. Fukayama, and M. Goto, “Automatic system for editing
dance videos recorded using multiple cameras,” Proceedings of 14th
International Conference on Advances in Computer Entertainment
Technology (ACE ’17), pp.671–688, 2017.
[30] S. Rubin, F. Berthouzoz, G.J. Mysore, W. Li, and M. Agrawala,“Content-based tools for editing audio stories,” Proceedings of the 26th
annual ACM symposium on User interface software and technology(UIST ’13), p.113–122, 2013.
[31] S. Rubin, F. Berthouzoz, G. Mysore, W. Li, and M. Agrawala,“Underscore: Musical underlays for audio stories,” Proceedings ofthe 25th Annual ACM Symposium on User Interface Software andTechnology (UIST ’12), p.359–366, 2012.
[32] M. Goto, J. Ogata, and K. Eto, “Podcastle: a web 2.0 approachto speech recognition research,” Proceedings of the 8th Annual
Conference of the International Speech Communication Association(INTERSPEECH ’07), pp.2397–2400, 2007.
[33] S. Fdili Alaoui, “Making an interactive dance piece: Tensions in
integrating technology in art,” Proceedings of the 2019 on Designing
Interactive Systems Conference (DIS ’19), pp.1195–1208, 2019.
[34] YCAM InterLab, “Ramdancetoolkit.” https://special.ycam.
jp/ram/en/tools/dance_tool_kit, 2013. Retrieved Mar. 1,
2023.
[35] K.E. Raheb, G. Tsampounaris, A. Katifori, and Y. Ioannidis, “Choreomorphy: A whole-body interaction experience for dance improvisationand visual experimentation,” Proceedings of the 2018 International
Conference on Advanced Visual Interfaces (AVI ’18), pp.27:1–27:9,
2018.
[36] V. Singh, C. Latulipe, E. Carroll, and D. Lottridge, “The choreographer’s notebook: A video annotation system for dancers andchoreographers,” Proceedings of the 8th ACM Conference on Creativity and Cognition (C&C ’11), pp.197–206, 2011.
[37] S. Hsueh, S.F. Alaoui, and W.E. Mackay, “Understanding kinaesthetic
creativity in dance,” Proceedings of the 2019 CHI Conference on
Human Factors in Computing Systems (CHI ’19), pp.511:1–511:12,
2019.
[38] M. Ciolfi Felice, S.F. Alaoui, and W.E. Mackay, “How do choreographers craft dance?: Designing for a choreographer-technology
partnership,” Proceedings of the 3rd International Symposium on
Movement and Computing (MOCO ’16), pp.20:1–20:8, 2016.
[39] A. Wang, “An industrial strength audio search algorithm,” Proceedingsof the 4th International Conference on Music Information Retrieval(ISMIR ’03), 2003.

Shuhei Tsuchidareceived his Ph.D. from
Kobe University in 2017. He currently works
as a Project Lecturer at Kobe University. He
worked as an AIST Postdoctoral Researcher at the
National Institute of Advanced Industrial Scienceand Technology (AIST) from 2017 to 2019. His
research area is human-computer interaction. He
has been focusing on enhancing and supporting
dance experiences by using various information
processing techniques.

Satoru Fukayama
received his Ph.D. degree in information science and technology in2013 from the University of Tokyo. He is currently a senior researcher at the National Instituteof Advanced Industrial Science and Technology(AIST), Japan. His interests are in signal processing and music information retrieval, especiallymusic generation with probabilistic models. He
has received awards, including IPSJ Yamashita
SIG Research Award, several Best Presentation
Awards, and Specially Selected Paper Award
from the Information Processing Society of Japan.

Jun Kato
received his Ph.D. degree from The
University of Tokyo in 2014. Prior to graduation,
he worked for Microsoft and Adobe Research.
He is currently a Senior Researcher at the National Institute of Advanced Industrial Scienceand Technology (AIST) and also serves as theTechnical Advisor at Arch Inc., Japan. He has focused on Human-Computer Interaction research,
designing tools for programming and authoring
multimedia content, and regularly gained academic recognition such as ACM CHI 2013/2015
Best Paper Honorable Mention and 2021 IPSJ/ACM Award for Early Career
Contributions to Global Research.

Hiromu Yakurareceived the B.E. and M.E.
degrees in 2019 and 2021, respectively, from the
University of Tsukuba, Tsukuba, Japan, where
he is currently working toward the Ph.D. degree.
His research interests include intersection of machine learning and human-computer interaction,
which involves various application areas, such as
creativity support, human resource development,
and virtual reality. He was the recipient of Google
Ph.D. Fellowship and Microsoft Research Asia
Ph.D. Fellowship.

Masataka Goto
received the Doctor of Engineering degree from Waseda University in 1998.
He is currently a Prime Senior Researcher at the
National Institute of Advanced Industrial Scienceand Technology (AIST), Japan. Over the past 30
years he has published more than 300 papers inrefereed journals and international conferences
and has received 58 awards, including several
best paper awards, best presentation awards, theTenth Japan Academy Medal, and the Tenth JSPS

IEICE TRANS. INF. & SYST., VOL.E106–D, NO.12 DECEMBER 2023

14

PRIZE. He has served as a committee memberof over 120 scientific societies and conferences, including the General Chairof ISMIR 2009 and 2014. As the research director, he began OngaACCEL
project in 2016 and RecMus project in 2021, which are five-year JST-funded
research projects (ACCEL and CREST) related to music technologies.