IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

386

PAPER

DanceUnisoner: A Parametric, Visual, and Interactive Simulation
Interface for Choreographic Composition of Group Dance
Shuhei TSUCHIDA†∗a) , Satoru FUKAYAMA†b) , Jun KATO†c) , Hiromu YAKURA†,††d) , Nonmembers,
and Masataka GOTO†e) , Fellow

SUMMARY Composing choreography is challenging because it involves numerous iterative refinements. According to our video analysis
and interviews, choreographers typically need to imagine dancers’ movements to revise drafts on paper since testing new movements and formations
with actual dancers takes time. To address this diﬃculty, we present aninteractive group-dance simulation interface, DanceUnisoner, that assists
choreographers in composing a group dance in a simulated environment.
With DanceUnisoner, choreographers can arrange excerpts from solo-dance
videos of dancers throughout a three-dimensional space. They can adjust
various parameters related to the dancers in real time, such as each dancer’s
position and size and each movement’s timing. To evaluate the eﬀectivenessof the system’s parametric, visual, and interactive interface, we asked sevenchoreographers to use it and compose group dances. Our observations,interviews, and quantitative analysis revealed their successful usage in iterative refinements and visual checking of choreography, providing insights
to facilitate further computational creativity support for choreographers.
key words: choreography, choreographic composition, GUI-based interaction, group dance

1. Introduction
Street dance is a popular dance style, and dancers often compete with each other and give unique artistic performancesin oﬃcial competitions. The competitions are held as onsite
events with audiences, often recorded and broadcasted on
TV shows and online. Break dancing is a prominent example of street dance. It was adopted as an oﬃcial sport in the2018 Youth Olympic Games in Buenos Aires [1] and wasselected as a candidate for inclusion in the 2024 Olympics
in Paris [2]. Competitions can employ a variety of rules
depending on the number of dancers. For instance, the International Dance Organization categorizes competitions into
“solo,” “duo,” “crew” involving 3 to 7 dancers, “formation”
involving 8 to 24 dancers, and “production” involving 25
dancers or more [3].
To win, choreographers must compose an appealing
Manuscript received April 1, 2023.
Manuscript revised September 4, 2023.
Manuscript publicized November 27, 2023.
† The authors are with National Institute of Advanced Industrial
Science and Technology (AIST), Tsukuba-shi, 305–8568 Japan.
†† The author is with University of Tsukuba, Tsukuba-shi, 305–
8574 Japan.
∗ Presently, with Kobe University, Kobe-shi, 657–8501 Japan.
a) E-mail: t.sway.tmpp@gmail.comb) E-mail: s.fukayama@aist.go.jpc) E-mail: jun.kato@aist.go.jpd) E-mail: hiromu.yakura@aist.go.jpe) E-mail: m.goto@aist.go.jp
DOI: 10.1587/transinf.2023EDP7063

dance performance well before the competition. However,the composition process becomes more challenging when
the performance involves more dancers. For choreographers, it would be ideal if dancers could instantly learn achoreography they came up with and perform it without any
mistakes. Then, the choreographers could focus on making
iterative refinements to the choreography without significant
interruptions. In reality, dancers cannot always gather at
one place at one time; they need to spend a certain amountof time learning a choreography and the intention behind it,
and they often have diﬃculties performing it. As a result, thechoreographer usually needs to rely on a set of conventional
analog devices—sheets of pen and paper—to make iterativerefinements without the actual dancers.
The goal of our work is to support the creativity ofchoreographers of group dances with the computer, enabling choreographies to be composed and iteratively refinedwithout dancers present. We analyzed existing group-dance
videos and interviews with expert choreographers to uncover
design principles that eﬀectively support creativity and determined that there needs to be a group-dance simulationinterface that is parametric, visual, and interactive (Sect. 2).
These findings led us to develop DanceUnisoner, a tool thatallows the choreographer to choose, edit, arrange, and combine videos of a single dancer with a dedicated user interface
and simulate a group dance performance (Fig. 1; Sect. 4). In
creativity, the cycle of imagination and its implementation isimportant [4]–[7]. Our system supports the imagination in
the creative process and makes the process of the cycle fromimagination to implementation more eﬃcient. This work targets the choreography of group dances in street dance genres(e.g., locking, hip-hop, popping, breaking, house, jazz, andwaack).
The main contributions of this work are threefold:(1) Through interviews with choreographers, we clarify
that three design principles—parametric, visual, andinteractive—could be used to support their creativity.
(2) On the basis of (1), we propose and implement a novel
interface for easily adjusting the various properties ofdancers, such as the position and size of every dancer
and the pattern and timing of every movement.
(3) Based on our user study, we provide insights on theprocess of composing a choreography with the supportof computers.

Copyright © 2024 The Institute of Electronics, Information and Communication Engineers

TSUCHIDA et al.: DANCEUNISONER

387

Fig. 1 Application overvierw. DanceUnisoner allows choreographers to simulate group dances byplacing and duplicating excerpts from recorded dance videos in 3D space (a). We designed it according
to three principles informed through our video analysis and interviews (b). We observed that all
participating choreographers could eﬀectively utilize it to compose group dances in their respective
styles (c).

2. Design Process
We aim at providing eﬃcient computational creativity support for choreographers. Toward this goal, we need to understand the current workflow for composing choreographies.
We first investigated the typical patterns appearing in choreographies for group dancing by reviewing dance videos available online. We also conducted interviews with choreographers in regard to their workflow. In this section, we describe
the findings and discuss the design principles we figured out
for establishing computational support.
2.1 Current Workflow of Choreographic Composition
To investigate common choreographic patterns, we analyzed
a set of 95 highly-ranked videos available on YouTube andinterviewed seven choreographers about their own processes
for choreographing group dances.
2.1.1 Video Analysis
To collect group-dance videos, we searched the phrase“group street dance” on YouTube and chose the top 137
videos. Of these videos, we removed 36 that could not
be regarded as dancing, such as those that were interviews
or audio only, and checked a total of 101 videos. These
101 videos consisted of dance performances on a stage forcompetitions and for entertaining people (78 videos), video
works for promoting dance groups or choreographies (12videos), improvised street-dance performances (5 videos),
flash mobs (3 videos), practice scenes in a studio (2 videos),and a number of people having fun dancing outdoors in theirown way (1 video). We then excluded the five improvised
videos and one video of people dancing outdoors from theanalysis because these videos did not fit the purpose of thispaper of supporting the creation of group dance choreography. We finally set 95 videos for analysis.
Although we had diﬃculty counting the number ofdancers for 13 of the videos, other dance performances consisted of 3 to 22 dancers. Moreover, we observed several

common characteristics of group dance among those videos
including various street-dance genres:
Choreography synchronization. In all of the dance
videos, we confirmed that there were shots in which multiple
dancers simultaneously performed to the same choreography.
Front-focused performance. In 89 out of the 95
videos, dancers performed assuming that the audience could
watch them from the front. The exceptions include largescale flash mobs, dance performances on the street, and solo
dance performances surrounded by an audience.
Crossing. In 75 out of the 95 videos, there were various formation-transition scenes, such as dancers moving toanother formation while crossing each other.
Flipping of choreography. In 66 out of the 95 videos,
we observed multiple dancers performing flipped choreography, such as symmetric choreography centered on the middleof the stage.
Shifting timing of movements. In 41 out of the 95
videos, there were moments where each dancer performed
the same movements but their start time was diﬀerent. We
could see that the choreography seemed to be propagating
to the next dancer. We also observed that the number ofdancers dancing tended to increase.
In summary, the video analysis revealed representativetechniques used in choreographic compositions. However,while this analysis suggested techniques that should be covered in the computational support, the actual process usedby choreographers to compose group dances was still unclear. To further understand the workflow of choreographers,
we conducted interviews. We interviewed both professional
and amateur choreographers who had experience with choreographing group dances to find out how they do so.
2.1.2

Interview with Choreographers

We interviewed a dance instructor with more than 9 yearsof street-dance experience and experience with group-dancecomposition, and 6 amateur choreographers (each with 3, 4,
7, 9, 9, and 16 years of dance experience) who belonged to auniversity dance club and had experience with group-dance

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

388

Fig. 2 (Top) Current workflow for composing choreography. It involves
iterative refinement process that includes asking dancers to gather and learn
modified version for confirmation, and thus, often takes long to get topractice. (Bottom) Workflow with the proposed system. Simulator cansimulate group dance and allows choreographers to create, record, confirm,
and revise group dances faster.

Fig. 3 Notes for visualizing choreography that we collected. They wererecorded in various formats: paper, blackboard, and simple drawing software. Overwritten cross marks (rejection marks in Japanese culture) indicate trial and error in composing group dance.

Confirmation. Since just imagining on paper is notenough, the choreographer confirms the created choreography by watching dancers perform the piece. To do so, thechoreographers use the notes to explain to dancers how theyshould move. This role of the notes as instructional materials is why they contain identification information, such as
diﬀerently colored circles or names next to circles depicting
individual dancers. The choreographers do not only watch
the performance by the dancers, but also film the dance toreview it with the dancers or have others in to observe.
Modification. The choreographers then correct any
problems that were noticed. For example, if dancers do not
move smoothly, or they collide or overlap with each other,
the choreographers adjust the choreography. Moreover, tocompose an appealing dance performance well, the choreographers adjust the balance of various properties such as
the position, direction, and size of the dancers, as well as
the pattern and timing of every movement in a group dance.
After correcting the problems, the choreographers return to
the Confirmation step and repeat both steps until they aresatisfied.
One of our unique findings was that, as mentioned in the
Ideation and Recording step, the choreographers compose
the positions and movements of the dancers by visualizing
choreographic information on paper. If dancers are waitingnear the choreographers throughout all steps, the choreographers can confirm the dance they composed, correct it,
and then quickly move on to the Confirmation step again.
However, choreographers cannot easily arrange for such asituation. From one interviewee, “We cannot confirm thechoreography until the next meeting,” and in some cases,“We have to wait until a week later.” That is, we found thatthe choreographers took too much time going back to the
Confirmation step from the correction phase because they
had to wait for the dancers to gather again.
2.1.3

composition. We asked 7 choreographers in total about theirworkflow from choreographic composition to group-dance
performance. The participants volunteered to participate in
the interview. We conducted the interviews in Japanese,and the translation was conducted by the first author. Fromthe comments obtained, we found that the choreographers
proceeded with the workflow shown at the top of Fig. 2.
Each part of the workflow is explained below.
Ideation and Recording. Choreographers decide upona musical piece and compose the dance performance. Whencomposing a dance, each choreographer uses various methods, such as imagining the dance while listening to the musical piece, trying to dance to the piece, searching for dance
videos that use the same piece and referring to them, deciding upon movements while talking with dancers, and so on.
In common with all choreographers, they visualize information such as the movement of dancers on paper or simple
drawing software (e.g., Keynote), as shown in Fig. 3, andorganize and record them.

Questionnaire of Dancers

We further conducted a questionnaire survey with dancers regarding how they perceive such workflows, especially about
how they are instructed in the process of composing of thegroup dance. We asked 28 dancers with more than 3 years ofstreet-dance experience who belonged to a university danceclub and had experience with a group dance performance.
The participants volunteered to answer the questionnaire. As
the result, 16 out of 28 participates answered that they had
experiences they received paper-based explanations about
the composition of the choreography, and 8 out of 28 participates answered that they had received video-based explanations about it. Also, one dancer mentioned that it was easierto understand the choreography of a group dance if they received a video “where I can see the movements from both
front and back, shot in a place with mirrors.” In addition,
one dancer commented, ‘When the choreography required
each dancer to move at diﬀerent timing for each beat, it wasdiﬃcult to dance because the timing was diﬀerent from theother dancers and there was no sample.” We found that when

TSUCHIDA et al.: DANCEUNISONER

389

dancers learn group dance choreography, it is important tovisualize the overall image of the choreography clearly.
2.2 Design Principles
In the previous sections, the analysis of existing dance videos
showed five common characteristics found in choreography for group dances. The interviews showed that thechoreographers used pen and paper to visualize, organize,
and record their choreography and that the choreography
ideation, recording, confirmation, and modification cycles
took too much time. In addition, we revealed that suchvisualization would often be too poor for dancers to learn
the choreography. These findings conversely suggest that
we can leverage computers to free choreographers from theworkload of such tedious and time-consuming tasks (see thebottom of Fig. 2) as well as helping dancers learn choreography. For this aim, on the basis of the knowledge obtainedin the previous section and the knowledge of the first author, who has more than 10 years of street dance experience(mainly breakdancing), we created the following three designprinciples.
Parametric. To express crossing, choreography flipping, and the timing of shifting, choreographers can adjust
the balance of various dancer properties such as the position,
direction, and size of every dancer and the pattern and timingof every movement in a group dance in sync with the beat.
Visual. Choreographers can intuitively understand awhole front-focused group dance. This is also helpful fordancers who learn the composed choreography afterward.
Interactive. Choreographers can quickly and easily
apply iterative cycles of ideation, recording, confirmation,and modification through an interface.
The primary goal of our system is to reduce the time
conventionally allocated to trial and error during the choreography creation process. The system that meets the above
principles will allow choreographers to refine their iterative process more eﬃciently, which will lead to more timespent on creative tasks such as choreography and composition of group dance. Moreover, our proposed interface gives
choreographers the flexibility to decide if they wish to sharethe choreography visualized by our system. By removing
the need to share choreography with dancers immediately,
choreographers can expedite the choreographic process, enabling a more eﬃcient approach to trial and error in group
dance composition.
3. Related Work
Our preliminary interviews revealed three principles required for supporting choreographers with computers. However, we acknowledge that some studies have proposed systems for supporting choreographic composition. To situate
our work, we review related work, especially highlightingthat the requirements have not been well supported yet.

3.1

Choreographic Composition Support Tools

Many support tools for choreographic compositions have already been proposed, and Alaoui et al. [8] compiled thesepieces of research. COMPOSE [9], Animate Tokens [10],Life Forms [11], and CorX [12] were presented in the firsthalf of the 1990s and then expanded into various research directions, such as works using a motion capture system [13],[14] and one using a stylus pen and tablet [15]. Carlson etal. [16], [17] proposed tools that support choreography analysis by visualizing Laban eﬀorts [18]. These systems employ
visualization with a stick man or simple polygon human. It
is desirable to visualize a dancer with an appearance similar
to that of an actual dancer, where choreographers can grasp
the subtle diﬀerences of the choreography.
Compared with single-dance choreographic composition, there is relatively little research focusing on groupdance choreographic composition. Schulz et al. [19] proposed authoring environments for dance performances thatuse movement information obtained from a motion capture system. Choreographers can compose choreography by
dragging and dropping movements selected from preparedchoreography lists onto a timeline. Moreover, they can easily compose a group dance by setting the dancer positions
displayed in 2D with a mouse. Furthermore, the system
displays the results with character animation, which allows
choreographers to confirm the choreography in 3D. A similarsystem using a motion capture system, DanceDesigner [20],was released. While not including dancers displayed withcharacter animation, this system allows choreographers totransform the movements of dancers into textual instructions
for dancers. They can also check dance videos shot from thefront and a path plan including the dancer’s positions. The
above systems are suitable for the purpose of preserving andediting detailed choreographic compositions and visualizing
group dance choreography. However, in terms of eﬃciency
for the iterative cycles of ideation, recording, confirmation,and modification, a motion capture system is not suited for
repeated trial and error done for the many choreographiccomposition steps in a limited amount of time. Without amotion capture system, though it has been possible to obtain movement information from general-purpose devices
such as smartphones equipped with a depth sensor recently,
these types of devices have diﬃculty accurately capturing
movement information since many errors can be included
depending on the dancer’s posture.
Some systems focus on methods for moving and changing the formations and positions of dancers on a simulator.
Systems that operate with Kinect [21] and gamepads [22]have been proposed. While looking at a group of dancers
visualized on screen, users of these systems can control theformation with their own body gestures or check the positions of the dancers in 3D by moving the virtual camera withthe gamepad. Given that, we propose a GUI-based interface
with a trackpad that can not only control multiple dancers
at once as in the above system, but also graphically displays

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

390

Fig. 4 Interface window. Choreographer can adjust in real time elements (dancer position, size, andtiming of movements) that should be considered when composing group dance while overviewing wholechoreography.

the positions of dancers by duplicating boards (billboards)
on which a solo-dance video is pasted and place.

relationship with dance movements.
3.3

Choreographic Communication

3.2 Video Editing and Broadcasting
We are inspired by projects designed to support video editing
and broadcasting. Video editing methods [23] that display
dances in 3D by using depth data have been proposed. Kurihara et al. [24] proposed a video editing system that can
express group dance in 3D by showing colored 3D point
clouds generated by combining depth information obtained
from Kinect and RGB color information. Ohta et al. [25]–[27] proposed a real-time 3D video transmission technology
with a simplified 3D model consisting of a single planeon which 2D athletes extracted from multiple cameras are
pasted. We presume that such techniques would also be useful for choreographic situations that choreographers imagine
the dance. That is, they can visually understand the movements and positions of dancers in 3D by using multiple flat
planes on which 2D athletes extracted from an RGB camera
are pasted.
Some video editing tools that use the synchronizationof audio and image tracks of videos have also been proposed. Truong et al. [28] presented QuickCut, which canhelp in quickly creating narration videos. Leake et al. [29]proposed a system for eﬃciently editing video of dialoguedriven scenes. Tsuchida et al. [30] developed an automatic
system for editing multiple dance videos. Some contentbased editing techniques that use content synchronization
have also been proposed. Rubin et al. [31] proposed atranscript-based speech editing tool and also presented UnderScore [32], which automatically refines, aligns, and adjusts speech and music. PodCastle [33] provides an interface
that enables users to easily correct speech recognition errors.
In this paper, our proposed system will utilize the synchronization between the audio and images (dance video shots)of videos to enable editing in beat units, which have a close

The creation of choreography also has a social aspect between the choreographers and dancers communicated via
choreography. Some studies have examined the ways thatchoreographers and dancers interact with technology. Studies [34], [35] done through art activities and studies [36],[37] aimed at design done collaboratively through technologies between people involved in various genres have been
conducted. Additionally, Hsueh et al. [38] examined howinteractive visual systems can support choreographic composition. Felice et al. [39] stated that tools for supporting
choreographers in the creative process should enable themto visualize and manipulate their ideas and share them withdancers and collaborators. We anticipate that our work cancomplement the previous findings through its observationof how choreographers compose group dance using the proposed system.
4. DanceUnisoner
4.1

Prototype Implementation

We built a prototype system according to the design principles (Sect. 2.2). The interface is shown in Fig. 4. We
introduce the functions of the prototype system below.
4.1.1

Input Data

The prototype system uses three kinds of input data: a musical piece used for a group-dance performance, a dance videoin which a single choreographer or a single dancer dancesalong the musical piece, and an image of the backgroundused in filming the dance video of a single dancer. It does

TSUCHIDA et al.: DANCEUNISONER

391

picting individual dancers, we did not emphasize the identification but aimed to allow the user to concentrate on theholistic view of the group dance. As discussed in Sect. 2.1.2,
the identification labels were important primarily for communicating with the actual dancers, and our prototype did
not need them as we designed the tool to be used solely by
the choreographers.
4.1.4

Fig. 5

An example of displaying dancers placed in 3D space.

not require a special device such as a motion capture system or depth sensor. Choreographers simply film themselves
dancing to a piece by using a simple RGB camera like theone attached to a smartphone. The background behind adancer can be removed by subtracting an image of the background from the video. The image of the background canbe prepared by cutting out an image from a dance video inwhich dancers do not appear. All input data was designed to
be prepared easily.
4.1.2 Pre-Processing
Using the audio of the input musical pieces, the prototype
system automatically trims segments in which the music is
played in the dance video and choreography appears. Specifically, we use landmark-based audio fingerprinting [40] to
automatically detect a musical piece’s starting position in adance video. We then cut it out as a video of one dancer
within the period of time length specified by the choreographer from that start position. After that, the human region
corresponding to the dancer is cut out in each frame of thatdance video by subtracting the background image.
4.1.3 Group-Dance Display Mpethod
To display a group dance by combining videos in which asingle dancer dances, the prototype system can be used to
duplicate boards (billboards) on which a solo-dance video
is pasted and place (i.e., copies and pastes) the duplicatedboards on the floor throughout 3D space. Through loop
playback of all videos in sync with the music, the prototype
system enables multiple dancers to be visualized as if they
are dancing at the same time. An example is shown in Fig. 5.
This method is easier to implement than displaying dancers
such as computer animated 3D characters and is suited tochoreographic composition since choreographers can easily
compose a group dance while imagining the visuals of an
actual group dance such as the swaying of and texture ofclothes.
While the choreographer notes (Fig. 3) contained identification information, such as diﬀerently colored circles de-

Adjustment GUI

Focusing on the parametric aspect of choreography composition (i.e., the number, position, appearance, and movement ofdancers), which is important in composing group dances, theprototype system provides a simple GUI such as a slider and2D pad (Fig. 4) that can be used to adjust those properties.
The choreographer can change the number and positions of
the dancers with simple GUI-based operations. Moreover,the dancers can be flipped by horizontally, and the timingof movements can be shifted in units of video frames. Additionally, in group dances, the choreographer often createschoreography to which dancers perform the same movement
shifted one beat (as suggested in Sect. 2.1); therefore, theprototype system makes it possible to shift the timing ofdancers’ movements in beat units according to the positionsof the dancers (shifting by 1, 2, or 4 beats can be selected).
Please note that we do not provide a function to directly
edit how dancers cross each other, while we found suchcrossings to be common in the video analysis in Sect. 2.1.1.
We made this design decision because it can be achieved byutilizing appropriate video excerpts. For instance, the user
can choose a video excerpt in which a dancer moves fromleft to right, put two instances of the excerpt in 3D space,and flip one of them to get a crossing scene.
4.2

Interview with Expert

To arrange the functions necessary for composing group
dances, we asked an expert choreographer with 18 years of
hip-hop dance experience to use the prototype system and toallow us to interview her about the functions that needed to
be improved or added.
The interview took 3 hours. For the first hour, thechoreographer received an explanation on how to use theprototype system, and she then imagined and composed agroup dance based on a musical piece that had 32 beats (4beats × 8 measures in four-four time), where the term “beat”
denotes a quarter note. After the choreographer organized
the choreography, we shot her dancing to that choreography.
We imported the video into the prototype system, and thechoreographer then visualized the group dance in 3D spaceby using the system. For the remaining two hours, we interviewed the choreographer regarding which functions shouldbe improved and what functions are required while asking
how a group dance is usually composed. The choreographer was compensated Y=15,000 (JPY) for taking part in theinterview.
Through the interview, the choreographer expressed her

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

392

Fig. 6 Setup mode. Choreographer can manually set depth information of dancer’s movements, whichallows them to edit movements of dancer in two dimensions. Prototype system did not support editingof this information.

pleasure with each of the functions built in accordance witheach of the three design principles. In particular, for thedesign principle interactive, from the viewpoint of composing a group dance eﬃciently and quickly, the choreographer
positively commented “I can see the whole choreographyeven if all dancers have not gathered, so I feel like the choreographing speed will become significantly faster [. . . ] I feellike I can make a choreography in half the time.” However, the choreographer pointed out that one function needs
improvement and one function is missing.

face problems such as colliding into each other and movements not reaching the designated position,” the system must
avoid such problems by making it possible to confirm andedit choreography including its movements in advance.
We refined the prototype system into DanceUnisoner†on the basis of the comments from the interview. Threemodes, Setup, Dancer Parameters, and Position 2D Map,
were added as a result of the improvements besides Formation mode, which allows users to change the number ofdancers and their position as the prototype system does.

4.2.1 Fine-Grained Timing Control

4.3

The choreographer requested that we allow for finer settings
regarding the dancer properties along the timeline. The
prototype changes the properties for an entire section of an
input dance video. For example, if the choreographer shifts
one beat against the timing of the movements of a dancer,that dancer always dances shifted one beat while the video isplaying. However, it is rare for the dancer properties to notchange during an entire section of choreography. Although
the choreographer agreed with the current functions, such as
the simple GUI based on the design principle parametric,
she commented that “If I could set the choreography finely,
I wouldn’t feel bothered re-filming the video or changing
the properties,” and we determined that the choreographer
needed additional functions to be able to set properties witha finer granularity along the timeline.

We introduce the user workflow of DanceUnisoner. The
choreographer first films themselves dancing to their choreography while imagining the choreography as a group dance.
We pre-process the captured video into a form that can beused with DanceUnisoner in the same way as the prototype
system.
When a choreographer launches DanceUnisoner, Setup
mode is displayed. In this mode, to set the position of thedancers on the z-axis in 3D space, the choreographer canmanually add depth information to the videos shot by adjusting the depth slider (Fig. 6). This makes it possible todisplay and move dance videos in 3D space. The choreographer adjusts the depth information by using the GUI sothat the movements displayed look as much like the actual
movements of a group dance as possible.
The choreographer then explores the initial formationof dancers. Formation mode has the same functions as the
prototype system, and the choreographer can easily adjust the
number of people and design a formation. Position 2D Map
mode shows the layout of the dancers’ positions as seen fromabove (i.e., overhead view). The choreographer can freely
move and set the positions of the dancers with a mouse.
After setting the initial formation of dancers, the chore-

4.2.2 Path Planning
The prototype interface cannot be used to plan the movement
paths of dancers including complex crossing scenes. Since
the interface places a dance video in 3D space, choreographers are allowed to experiment with both left-and-rightand back-and-forth movements. However, these movements
cannot be easily edited on the prototype system. From comments such as “When dancers dance for the first time, theyUser Workflow† https://youtu.be/sMYKIA6DSJQTSUCHIDA et al.: DANCEUNISONER

393

Fig. 7

Position 2D map mode. The choreographer can arrange the dancers’ position.

Fig. 8 In steps (a) and (b), dancers are grouped by encircling them within a rectangular selection.
In step (c), a simple click action transforms the group formation into a triangular arrangement. Step(d) demonstrates how a click action introduces choreography with a staggered timing, causing eachsubsequent dance movement to be oﬀset by a single beat, starting from the front.

ographer switches from Position 2D Map mode to Dancer
Parameters mode. In this mode, the choreographer can edit
dancers’ properties such as the kinds of choreography (preshot dance videos with diﬀerent choreography), flipping,and shifts in beats. The choreographer can compose a group
dance while constantly switching between Position 2D Map
mode and Dancer Parameters mode.
We give details on the functions of DanceUnisoner in
the next section.
4.4 Implementation
We modified DanceUnisoner based on the improvement(Sect. 4.2.1) and missing point (Sect. 4.2.2) that the choreographer pointed out in the previous section. We implemented
DanceUnisoner using openFrameworks 0.9.8 and conducted
the operation check on macOS Sierra.
4.4.1 Setup Mode
In Setup mode, the choreographer can manually set depth
information, which is diﬃcult to estimate with only RGB
video. On the interface window shown in Fig. 6, dancers
are displayed translucently for overviewing the movementsof the whole video at every four beats of the musical piece.
Moreover, the top right part of the screen shows an overhead
view of the dancers’ positions, and that view displays thedancers’ movement trajectories for eight beats in 3D space.

When the music starts playing, the main dancer (nontransparent dancer) begins to dance. The choreographer
can set the depth information of the dancers by moving thedepth slider on the upper right side of the window (Fig. 6)
up and down according to the main dancers’ movements.
The choreographer can also increase the level of detail of the
movements by clicking the line chart that is displayed on thetimeline of the interface.
Furthermore, in the original input video, dancers become smaller if they move to the back away from the camera,
and their left and right movements become shorter. Similarly, when moving toward the camera, they become bigger,
and the left and right movements become longer. Therefore,
the choreographer can adjust these diﬀerences by controllingthe slider on the interface.
4.4.2

Position 2D Map Mode

The Position 2D Map mode allows the choreographer to assign and confirm dancer positions from an overhead view.
The choreographer can select groups of dancers by clicking and dragging a rectangular selection box comprised oftriangles over dancers. They can then change the dancers’
positions by dragging the rectangle (Fig. 7 and Fig. 8 (a) (b)).
This mode also allows the choreographer to rearrange selected dancers in a formation such as a circle or a triangle
that fits the size of the designated rectangular area with asingle button (Fig. 8 (c)). The choreographer can enlarge

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

394

Fig. 9 In step (1), choreographers design the desired formation following a movement sequence andcapture this configuration with a single click. As demonstrated from steps (2) to (5), dancers can beseamlessly transitioned to align with the formation established in step (1).

and reduce the formations by using the 2D pad. This mode
incorporates features similar to those in standard graphic editors and presentation software, such as drag-and-drop functionality and click-based selection. Additionally, alignment
adjustments are integrated. Such an intuitive design aims
to ensure that choreographers can swiftly become proficientwith the system’s operations. The choreographer can also
apply beat shifts (Fig. 8 (d)) and flips (if the dancer is facing to the right, we can flip them so that they face to theleft) to the selected dancers like with the prototype system
functions.
4.4.3 Path Planning Function
The Path Planning function was implemented to address theissue reported in Sect. 4.2.2, allowing the choreographer toplan the movement paths of dancers with a dedicated userinterface (Fig. 9). The choreographer can use the registeredformation in Dancer Parameters mode (Sect. 4.4.4) and specify the destination of each dancer. Then, the dancers moveto the specified destinations at a constant speed, allowing thechoreographer to design the formation transition. This way,
the choreographers can iteratively design, check, and create
complex formation changes including crossing scenes that
could not be achieved in the prototype.
4.4.4 Dancer Parameters Mode
To set properties with a finer granularity along the timeline
than the low granularity mentioned in Sect. 4.2.1, this mode
allows the choreographer to adjust the properties per beat.
First, the choreographer selects the dancer that the choreographer wants to adjust with the slider labeled Dancer ID at
the top left of the interface (Fig. 10). By dragging a mousepointer across the rectangular frames at the top of the interface window, the choreographer can designate the beats thatthe choreographer wants to adjust.
By utilizing the above functions and going through theuser workflow described in Sect. 4.3, choreographers caneasily compose the group dance. As an example, we show
here that it becomes feasible to easily design choreography
where multiple dancers form a triangular formation, witheach dancer’s movement being oﬀset by one beat from thefront. First, the choreographer can record the dance movement of themselves, input its video into the system, set depth
information manually, and then easily expand the number ofFig. 10 Dancer parameters mode. Choreographer adjusts dancers’ properties (stop, flip, etc.) for each beat. “Stop” is written with S, and “flip” isrepresented by reversed numbers (numbers mean diﬀerence in beats). Whenchoreographer inputs video, choreography number is assigned to every beatof video. Choreographer then shifts timing of movements on basis of thisnumber.

dancers using a slider in the GUI. After grouping the dancers
via drag-and-drop, the formation can be altered to a triangle
with a single click (Fig. 8 (a) (b) (c)). The size of the formation can be adjusted using the 2D pad. Furthermore, after
re-grouping the left half of the dancers again via drag-anddrop, a choreography where the movements are sequentiallyoﬀset by one beat from the front can be set with a single click(Fig. 8 (d)). This operation should then be repeated after regrouping the right half of the dancers to simulate the target
group dance. While simulating these group dances in existing choreography assistance systems is time-consuming,
they can be easily achieved through the procedure describedabove.
Moreover, choreographies with transition formations,such as moving from a triangle to an inverted triangle, canbe realized by using the path-planning function (Fig. 9).
5. User Study
The goal of our work is to support choreographers’ creativity
by providing an eﬃcient choreographic process with computers. To investigate how DanceUnisoner can contribute tochoreographers and whether the system makes the choreographic process more eﬃcient, we asked seven choreographers to choreograph group dances by using the proposedsystem. We recruited 7 participants (2 males and 5 females)

TSUCHIDA et al.: DANCEUNISONER

395

who had between 4.5 to 18 years of dance experience (average = 10 years). Their experience covered a variety of dance
genres such as locking, hip-hop, popping, breaking, house,waack, and jazz, and ballet dance. One of them was the samechoreographer who participated in the above interview.
We conducted the user study in a meeting room withan area of about 6 m × 9 m. Participants could move freely
in the room, which had chairs and tables. Each participant
used DanceUnisoner, and it ran in openFrameworks 0.9.8
on a MacBook Pro (Retina, 15-inch, Mid 2015). The study
was conducted for each choreographer and lasted for fivedays. The participants were compensated Y=15,000 (JPY) for
taking part in. We conducted the interviews in Japanese, andthe translation was conducted by the first author.

5.2.4

Simulating Choreography (60–90 min)

The participants simulated the choreography imagined intheir heads in DanceUnisoner by using the videos shot during the second activity. We stayed near participants so that
we could answer questions whenever they did not understandhow to use the interface. During the simulating choreography phase, participants were provided with traditional tools
like pen and paper, which they could opt to utilize. It is worthhighlighting that DanceUnisoner is designed to complement,
rather than replace, such conventional tools. Specifically,
pen and paper can be beneficial for tasks like jotting down
quick notes or annotating specific dance movement timings
and patterns corresponding to musical beat cues.

5.1 Task
5.2.5
Each participant simulated two group dances of seven
dancers in DanceUnisoner. They choreographed to the same
musical piece that had a duration of 32 beats (4 beats ×
8 measures in four-four time), at a tempo of 90 beats per
minute. To have the participants use the distinct functionsof DanceUnisoner, we asked them to compose two diﬀerent
choreographies of group dances that covered the following
four elements.
• Back and forth movements.
• Left and right movements.
• Flipping of choreography horizontally.
• Shifting of the timing of movements.
5.2 Procedure
The participants performed the following five activities.
Completing all activities took about 3.5 hours. Participants
took a short break between activities.
5.2.1 Explanation (15 min)In order for the participants to understand what functions areincluded in DanceUnisoner, we gave a verbal explanationof about 15 minutes to them while actually operating theinterface.

Interview (30 min)

We asked participants what the benefits are of using DanceUnisoner, what functions are convenient, and what functions
are essential.
Here, we recorded the interface windows while the participants used DanceUnisoner and logged the participants’interaction with it. We also recorded the voices of the participants in the interviews and gathered the pieces of paper usedin composing and simulating group dances. We anonymized
the data, and we refer to participants as P1-P7.
6. Results and Discussion
We confirmed that all participants successfully simulated
their imagined choreography. Some of the output of the participants is shown in Fig. 11. Moreover, we received various
positive comments from all participants: “Easy to imagine
because I can see the video with my real eyes,” “Convenient,” “I understand the impressiveness of group dance,”“It is good to be able to confirm the music, composition, andchoreography at the same time,” “I can notice the subtle
diﬀerence of movements,” “Most functions are essential,”“I can see what I visualize in my head with the video.”
6.1

Eﬃcient Choreographic Process

5.2.2 Composition and Filming (30–50 min)

6.1.1

The participants first imagined the choreography of a group
dance as per usual. While composing the dance, we provided
them with pen and paper, which they were free to use. At theend, we filmed them dancing as an element for constructing
the choreography.

Looking at the feedback in terms of the eﬃciency of choreographic composition, we found that three participants (P2,
P6, and P7) kept on changing the choreography while adjusting the group dance simulation. P6 commented, “When I feltthat this part includes few [formation] movements, I added aplus alpha [forth and back movements] to the choreographic
composition.” As this participant simulated a group dancewith the system, the participant noticed and supplemented
the parts that lacked formation movements. We found that
DanceUnisoner made it possible to confirm choreography ina way that could not be done without dancers and allowed theparticipants to modify their compositions. In other words,

5.2.3 Training (30 min)So as to familiarize themselves with controlling the interface, the participants practiced simulating the group dancein DanceUnisoner with pre-prepared videos while imagining
the choreography in their heads.

Iterative Refinements

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

396

Fig. 11

Pick-up screen shots of the output videos that the participants created.

choreographers could quickly and easily apply iterative cycles of ideation, recording, confirmation, and modification
through an interface (from interactive design perspective).
As described in Sect. 2.1.2 under “We cannot confirm thechoreography until the next meeting,” and “We have to wait
until a week later,” we can infer that creating choreographieswith actual dancers often spans several days. Consequently,
once it is confirmed that choreographies can be visualized,
eﬃciency is inherently assured.
6.1.2 Instant Visualization
Some comments were observed regarding the visualizationof choreography. Four participants (P1, P3, P4, and P5) focused on using the system to modify a group dance they wereimagining in their head. They mainly used the interface toconfirm their choreography and did not change their overallcomposition style. P1 said “I will first make the choreography based on music and then use this [DanceUnisoner],”and P4 also said, “I do not think I will change how I compose
choreography [. . . ] need a mirror and earphones [. . . ] I
may use it [DanceUnisoner] as a tool for confirming andsharing choreography.” What they had in common is thatthey modified their choreography while visually checking
the choreography in DanceUnisoner after creating a groupdance simulation.
The distinctive feature of our proposed system is its useof real dancers extracted from video, rather than skeletons,
simple polygons or CG characters. Looking at the user interview comments on this point, we obtained high evaluationfrom P1, P2, P5, and P7. For instance, some comments
included: “Although billboards are 2D, it is not that bad be-cause I usually see it from the front in the practice of dancers.
It is very easy to imagine how they really are,” “It is easy
to understand how powerful they are. I usually cannot know
it when I choreograph alone,” and “It is possible to noticesmall deviations since I can immediately see the dancers’
movements on the actual images.” Such comments suggest that real dancer footage enhances ease of imagination,
comprehension of intensity, and the ability to notice the differences in choreography. These benefits are not attainablewith existing systems and methods.
Four participants (P2, P3, P4, and P5) said that DanceUnisoner would help to “communicate” the “image” of acomposition to dancers. The larger the number of dancers,
the harder it becomes for the choreographer to imagine how
the dance group looks. The participants pointed out that
it would be useful to practice choreography with an output video that can be regarded as the goal. Also, from theviewpoint of teaching, P2 and P4 commented that the interface seemed useful in “understanding” the choreographyof group dances comprised of a large number of dancers.
Also, P5 noted that choreographers could share DanceUnisoner among themselves to avoid problems when sharing
choreography.
As mentioned above, the proposed system is not only effective for choreographers to intuitively understand a whole
front-focused group dance but also valuable for dancers wholearn the choreography afterward (from visual design perspective).
6.2

Quantitative Analysis

To examine the frequency of parameter changes in DanceU-

TSUCHIDA et al.: DANCEUNISONER

397
Table 1 The operation logs of DanceUnisoner. The number of operations
in which each function was used is displayed.
id

EX
year

P1

7.5

P2

8

P3

17

P4

18

P5

4.5

P6

6

P7

7.5

time
(min)
31
15
10
11
32
55
11
15
14
20
36
31
60
22

film

flip

stop

1
2
2
4
1
1
1
1
1
1
3
2
2
1

0
–
24
8
24
0
0
48
56
24
4
28
24
26

9
–
0
0
0
14
8
0
0
18
0
0
13
0

beat
shift
58
–
0
0
0
86
4
0
0
13
0
0
8
0

cycles of choreography as we expected but could also aﬀect
their creativity.

formation
28
–
0
28
56
66
0
28
84
14
21
30
80
56

nisoner, we analyzed the operation logs from the user study,
as shown in Table 1. EX year indicates the number of years ofdance experience. Since one choreography log of P1 was nottaken, a dash is displayed in the respective boxes. The formation refers to the command name that executes the formation
movement registered with the path planning function. We
found that choreographers can adjust various characteristics
of dancers, such as individual positions, directions, sizes,
and the patterns and timings of all group dance movements,
in sync with the beat, to express intersections, inversions ofchoreography, and timing shifts by utilizing the proposedsystem (from parametric design perspective).
Also, the logs revealed that the formation function to
execute the formation movement registered with the path
planning was most frequently used. The range of group
dances that could be expressed is narrowed when using only
the original movements in the input video, so the choreographers struggled to compose a variety of group dances.
In addition to the movements of dancers, the path planning
function mentioned in Sect. 4.4.2 was essential as we learned
from the interview with the professional (in Sect. 4.2). Notethat some users opted not to utilize the beat shift function.
Instead, they manually adjusted the dance movements toachieve a similar eﬀect to the beat shift function.
6.3 Enhancing Creativity
DanceUnisoner is expected to not only make more eﬃcient
process of choreography creation but also support the creativity of choreographers. P7 said, “As I use the system[DanceUnisoner], I can notice things that I never thoughtof in my head and put them on paper.” The simple GUI ofDanceUnisoner allowed them to easily change the propertiesof dancers, so they could check various patterns instantly.
This feature may make it easier to notice new patterns inchoreography. Moreover, P2 commented, “I got inspirationfrom seeing the actual movements of dancers [on DanceUnisoner].” The gap between imagination and simulation
possibly aﬀected the participants’ creativity. DanceUnisoner
could not only accelerate the confirmation and modification

7. Limitations and Future Work
Despite the aﬃrmative responses from the participants, there
are some limitations with DanceUnisoner. First, DanceUnisoner demands dance skills of the choreographer since
it uses video of them dancing. When the choreographer
does not have enough skills, they would need to ask anotherdancer to dance for the video. Even if the choreographer
could dance, they might not be able to imitate other dancers’
styles and also their appearance. To address this issue, itwould be helpful to develop a function that can support amodel of dancer’s individual characteristics and apply it todances.
Second, DanceUnisoner has a limitation with the 2D
billboard on which a solo-dance video is pasted. As discussed in the video analysis in Sect. 2.1.1, most dance performances are assumed to be seen from the front, but insome cases, the audience surrounds the dancers. Even if thechoreographer using DanceUnisoner would want to checka performance from the side, they would barely be able tosee the dancers because of the 2D billboard. To address
this issue, a function of representing the dancers’ movements and appearances in 3D space would be necessary. In
addition, DanceUnisoner has diﬃculty representing dance
performances in which some physical objects are used bydancers. To display dancers throwing a physical object like aball, for example, a function of supporting such a 3D model
would be necessary.
Third, although our user study illustrated the usefulness
and creative aspect of DanceUnisoner, its generalization toother dance genres is left as future work since this paper targets the choreography of group dances in street dance genres(e.g., locking, hip-hop, popping, breaking, house, jazz, andwaack). Further investigation on observing how choreographers in other genres (e.g., K-pop, salsa, and cheerleading)utilize DanceUnisoner would indicate areas of improvement.
We know that DanceUnisoner cannot support every genre
since some genres involve movements improvised by dancers
during performances. Even a musical piece could be improvised (i.e., not pre-recorded). Such improvisational nature
is out of the score of this research and is not supported byDanceUnisoner.
8. Conclusion
We have described DanceUnisoner that enables a single
choreographer to easily compose, by leveraging videos inwhich a single dancer dances, group dance videos that look
as if multiple dancers are dancing. We first investigatedthe current group-dance choreographic composition, figuredout the existing group-dance composition issues, and determined design principles. Based on the design principles,
we developed the prototype system and conducted a professional interview where an expert choreographer used the

IEICE TRANS. INF. & SYST., VOL.E107–D, NO.3 MARCH 2024

398

proposed interface on the system. On the basis of comments
obtained from the interview, we improved the interface andbuilt DanceUnisoner. To investigate how choreographers
use DanceUnisoner, we conducted a user study with sevenprofessional choreographers. We confirmed that all of them
could simulate a group dance by themselves; in particular,
three of them utilized the iterative refinement process, whichcannot be done eﬃciently without DanceUnisoner.
Acknowledgments
This work was supported in part by JST ACCEL Grant Number JPMJAC1602, Japan.
References[1] International Olympic Committee, “Head over heels about dance
sport.” https://www.olympic.org/news/head-over-heels-aboutdance-sport, 2020. Accessed on March 1, 2023.
[2] The Washington Post, “Break dancing among new events proposedfor 2024 olympics.” https://www.washingtonpost.com/sports/2019/02/22/break-dancing-among-new-events-proposed-olympics, 2020.
Accessed on March 1, 2023.
[3] International Dance Organization, “IDO dance sport rules & regulations.” https://www.ido-dance.com/ceis/ido/rules/competitionRules/
2022.09-IDO-Rule-Book.pdf, 2022. Accessed on March 1, 2023.
[4] R.A. Finke, T.B. Ward, and S.M. Smith, “Creative cognition: Theory,
research, and applications,” MIT press Cambridge, 1992.
[5] M.A. Runco and I. Chand, “Cognition and creativity,” Journal of
Educational psychology review, vol.7, no.3, pp.243–267, 1995.
[6] M. Botella, V. Glaveanu, F. Zenasni, M. Storme, N. Myszkowski,
M. Wolﬀ, and T. Lubart, “How artists create: Creative process andmultivariate factors,” Journal of Learning and Individual Diﬀerences,vol.26, pp.161–170, 2013.
[7] M.-A. Mace and T. Ward, “Modeling the creative process: A
grounded theory analysis of creativity in the domain of art making,”
Creativity research journal, vol.14, no.2, pp.179–192, 2002.
[8] S.F. Alaoui, K. Carlson, and T. Schiphorst, “Choreography as mediated through compositional tools for movement: Constructing ahistorical perspective,” Proc. ACM International Workshop on Movement and Computing (MOCO ’14), pp.1:1–1:6, 2014.
[9] T. Schiphorst, T. Calvert, C. Lee, C. Welman, and S. Gaudet, “Tools
for interaction with the creative process of composition,” Proc.
SIGCHI Conference on Human Factors in Computing Systems (CHI’90), pp.167–174, 1990.
[10] J.H. Bradford and P. Côté-Laurence, “Animate tokens: Their design
and application to dance,” Journal of IEEE Transactions on Learning
Technologies, vol.24, no.5, pp.557–562, 1991.
[11] T.W. Calvert, A. Bruderlin, S. Mah, T. Schiphorst, and C. Welman,“The evolution of an interface for choreographers,” Proc. INTERCHI’93 Conference on Human Factors in Computing Systems (INTERCHI ’93), pp.115–122, 1993.
[12] J.H. Bradford and P. Côté-Laurence, “An application of artificial
intelligence to the choreography of dance,” Journal of Computers
and the Humanities, vol.29, no.4, pp.233–240, 1995.
[13] J.C.P. Chan, H. Leung, J.K.T. Tang, and T. Komura, “A virtual reality
dance training system using motion capture technology,” Journal ofIEEE Transactions on Learning Technologies, vol.4, no.2, pp.187–
195, 2011.
[14] Y. Yoshida and T. Yonezawa, “Choreographic design visualizationof enormous dancers for authoring and browsing dance motion andformation,” Proc. International Association of Societies of Design
Research (IASDR ’13), 2013.
[15] M.C. Felice, S.F. Alaoui, and W.E. Mackay, “Knotation: Exploring and documenting choreographic processes,” Proc. 2018 CHI

Conference on Human Factors in Computing Systems (CHI ’18),
pp.448:1–448:12, 2018.
[16] K. Carlson, T. Schiphorst, and C. Shaw, “Actionplot: A visualization
tool for contemporary dance analysis,” Proc. International Symposium on Computational Aesthetics in Graphics, Visualization, andImaging (CAe ’11), pp.113–120, 2011.
[17] K. Carlson, T. Schiphorst, and P. Pasquier, “Scuddle: Generating
movement catalysts for computer-aided choreography,” Proc. Second
International Conference on Computational Creativity, pp.123–128,
2011.
[18] I. Bartenieﬀ and D. Lewis, Body movement: Coping with the environment, Routledge, 2013.
[19] A. Schulz, W. Matusik, and L. Velho, “Choreographics: An authoring tool for dance shows,” Journal of Graphics Tools, vol.17, no.4,
pp.159–176, 2013.
[20] Dance Designer, “Choreopro.” https://www.capterra.com/p/128303/
Dance-Designer/, 2010. Accessed on March 1, 2023.
[21] A. Soga and I. Yoshida, “Interactive control of dance groups using
kinect,” Proceedings of 2014 International Conference on Computer
Graphics Theory and Applications (GRAPP ’15), 2015.
[22] A. Soga and I. Yoshida, “Simulation system for dance groups usinga gamepad,” Proc. International Conference on Computer Graphics
Theory and Applications and International Conference on Information Visualization Theory and Applications (GRAPP/IVAPP ’12),
pp.365–368, 2012.
[23] T. Hirai, S. Nakamura, T. Yumura, and S. Morishima, “Vrmixer:
Mixing video and real world with video segmentation,” Proc. 11th
Conference on Advances in Computer Entertainment Technology(ACE ’14), pp.30:1–30:7, 2014.
[24] T. Kurihara, M. Okabe, and R. Onai, “Ddmixer2.5d: Drag and drop
to mix 2.5d video objects,” Proc. Adjunct Publication of the 26th Annual ACM Symposium on User Interface Software and Technology(UIST ’13 Adjunct), pp.69–70, 2013.
[25] Y. Ohta, I. Kitahara, Y. Kameda, H. Ishikawa, and T. Koyama, “Live
3d video in soccer stadium,” International Journal of Computer Vision, vol.75, no.1, pp.173–187, 2007.
[26] T. Koyama, I. Kitahara, and Y. Ohta, “Live mixed-reality 3d video insoccer stadium,” Proc. 2Nd IEEE/ACM International Symposium on
Mixed and Augmented Reality (ISMAR ’03), p.178, IEEE Computer
Society, 2003.
[27] I. Kitahara and Y. Ohta, “Scalable 3d representation for 3d video
display in a large-scale space,” Proc. IEEE Virtual Reality (VR ’03),
pp.45–52, 2003.
[28] A. Truong, F. Berthouzoz, W. Li, and M. Agrawala, “Quickcut:
An interactive tool for editing narrated video,” Proc. 29th Annual
Symposium on User Interface Software and Technology (UIST ’16),
pp.497–507, 2016.
[29] M. Leake, A. Davis, A. Truong, and M. Agrawala, “Computational
video editing for dialogue-driven scenes,” Journal of ACM Transactions on Graphics, vol.36, no.4, pp.1–14, 2017.
[30] S. Tsuchida, S. Fukayama, and M. Goto, “Automatic system for editing dance videos recorded using multiple cameras,” Proceedings of
14th International Conference on Advances in Computer Entertainment Technology (ACE ’17), vol.10714, pp.671–688, 2017.
[31] S. Rubin, F. Berthouzoz, G.J. Mysore, W. Li, and M. Agrawala,“Content-based tools for editing audio stories,” Proc. 26th annual
ACM symposium on User interface software and technology (UIST’13), pp.113–122, 2013.
[32] S. Rubin, F. Berthouzoz, G. Mysore, W. Li, and M. Agrawala, “Underscore: Musical underlays for audio stories,” Proc. 25th Annual
ACM Symposium on User Interface Software and Technology (UIST’12), pp.359–366, 2012.
[33] M. Goto, J. Ogata, and K. Eto, “Podcastle: a web 2.0 approach tospeech recognition research,” Proc. 8th Annual Conference of theInternational Speech Communication Association (INTERSPEECH’07), pp.2397–2400, 2007.
[34] S. Fdili Alaoui, “Making an interactive dance piece: Tensions inTSUCHIDA et al.: DANCEUNISONER

399

integrating technology in art,” Proc. 2019 on Designing Interactive
Systems Conference (DIS ’19), pp.1195–1208, 2019.
[35] YCAM InterLab, “Ramdancetoolkit.” https://special.ycam.jp/ram/
en/tools/dance_tool_kit, 2013. Retrieved March 1, 2023.
[36] K.E. Raheb, G. Tsampounaris, A. Katifori, and Y. Ioannidis, “Choreomorphy: A whole-body interaction experience for dance improvisation and visual experimentation,” Proc. 2018 International Conference on Advanced Visual Interfaces (AVI ’18), pp.27:1–27:9, 2018.
[37] V. Singh, C. Latulipe, E. Carroll, and D. Lottridge, “The choreographer’s notebook: A video annotation system for dancers and choreographers,” Proc. 8th ACM Conference on Creativity and Cognition(C&C ’11), pp.197–206, 2011.
[38] S. Hsueh, S.F. Alaoui, and W.E. Mackay, “Understanding kinaesthetic creativity in dance,” Proc. 2019 CHI Conference on Human
Factors in Computing Systems (CHI ’19), pp.511:1–511:12, 2019.
[39] M. Ciolfi Felice, S.F. Alaoui, and W.E. Mackay, “How do choreographers craft dance?: Designing for a choreographer-technology
partnership,” Proc. 3rd International Symposium on Movement andComputing (MOCO ’16), pp.20:1–20:8, 2016.
[40] A. Wang, “An industrial strength audio search algorithm,” Proc. 4thInternational Conference on Music Information Retrieval (ISMIR’03), 2003.

Shuhei Tsuchidareceived his Ph.D. from
Kobe University in 2017. He currently works
as a Project Lecturer at Kobe University. He
worked as an AIST Postdoctoral Researcher at
the National Institute of Advanced Industrial Science and Technology (AIST) from 2017 to 2019.
His research area is human-computer interaction. He has been focusing on enhancing andsupporting dance experiences by using various
information processing techniques.

Satoru Fukayama
received his Ph.D. degree in information science and technology in2013 from the University of Tokyo. He is currently a senior researcher at the National Instituteof Advanced Industrial Science and Technology(AIST), Japan. His interests are in signal processing and music information retrieval, especially music generation with probabilistic models. He has received awards, including IPSJ
Yamashita SIG Research Award, several Best
Presentation Awards, and Specially Selected Paper Award from the Information Processing Society of Japan.

Jun Kato
received his Ph.D. degree fromThe University of Tokyo in 2014. Prior to graduation, he worked for Microsoft and Adobe Research. He is currently a Senior Researcher at
the National Institute of Advanced Industrial Science and Technology (AIST) and also serves
as the Technical Advisor at Arch Inc., Japan.
He has focused on Human-Computer Interaction research, designing tools for programming
and authoring multimedia content, and regularly
gained academic recognition such as ACM CHI
2013/2015 Best Paper Honorable Mention and 2021 IPSJ/ACM Award for
Early Career Contributions to Global Research.

Hiromu Yakurareceived the B.E. and M.E.
degrees in 2019 and 2021, respectively, from the
University of Tsukuba, Tsukuba, Japan, where
he is currently working toward the Ph.D. degree.
His research interests include intersection of machine learning and human-computer interaction,
which involves various application areas, suchas creativity support, human resource development, and virtual reality. He was the recipientof Google Ph.D. Fellowship and Microsoft Research Asia Ph.D. Fellowship.

Masataka Goto
received the Doctor ofEngineering degree from Waseda University in
1998. He is currently a Prime Senior Researcher
at the National Institute of Advanced Industrial
Science and Technology (AIST), Japan. Overthe past 31 years he has published more than
300 papers in refereed journals and international
conferences and has received 65 awards, including several best paper awards, best presentation
awards, the Tenth Japan Academy Medal, andthe Tenth JSPS PRIZE. He has served as a committee member of over 120 scientific societies and conferences, including
the General Chair of ISMIR 2009 and 2014. As the research director, hebegan OngaACCEL project in 2016 and RecMus project in 2021, which
are five-year JST-funded research projects (ACCEL and CREST) related tomusic technologies.