# Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with SPARK

Nathan Salin, Florent Berthaut, Ronan Gaugne, Valérie Gouranton

► To cite this version:

Nathan Salin, Florent Berthaut, Ronan Gaugne, Valérie Gouranton. Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with SPARK. VRST 2025 - Virtual Reality Software and Technology, Nov 2025, Montréal, Canada. pp.1-10, 10.1145/3756884.3765998 . hal-05273140

HAL Id: hal-05273140

https://hal.science/hal-05273140v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with S.P.A.R.K

Nathan Salin

nathan.salin@irisa.fr

Univ Rennes, INSA Rennes, Inria, CNRS, IRISA

Rennes, France

Florent Berthaut

florent.berthaut@univ-lille.fr

CRISTAL, Univ. Lille

Lille, France

Ronan Gaugne

ronan.gaugne@irisa.fr

Univ Rennes, Inria, CNRS, IRISA

Rennes, France

Valérie Gouranton

valerie.gouranton@irisa.fr

Univ Rennes, INSA Rennes, Inria, CNRS, IRISA

Rennes, France

<figure><img src="image_2.png" /><figcaption>**Figure 1:** User mapping a sound with S.P.A.R.K: the wrist-attached UI appears, a virtual box is selected, and an OSC message is configured with desired values, which will be sent to the Pure Data patch when the avatar passes through the box.</figcaption></figure>

## Abstract

Recent artistic explorations in VR environments have explored how users engage with virtual spaces, sounds, and bodies whether, as artists, coders, or spectators. While many applications focus primarily on musical interaction or motion capture, few allow users to actively compose spatial and temporal relationships within a multi-modal collaborative creation context merging altogether different artistic modalities. In this project, we present a Virtual Reality application that enables novel forms of collaboration between dancers and musicians. The application allows dancers to record full-body performances, which musicians can then use as the basis for sonic composition by spatially mapping sound triggers onto the dancer's recorded movement. Rather than relying on live capture or real-time gesture tracking, our approach treats movement as a timeline for interaction blending choreography with sound design in an asynchronous workflow. We employ an iterative design process to ensure usability among experts. This paper details the first implementation and study involving 10 participants recruited from professional and amateur artists with electronic music backgrounds, highlighting positive reception of the application's creative potential and usability.

## CCS Concepts

* Human-centered computing → Interaction design; Human computer interaction (HCI); Interaction design: Human computer interaction (HCI); User studies; Usability testing; Virtual reality; Auditory feedback; *Applied computing → Sound and music computing*.

## Keywords

Virtual Reality (VR), Multimodal Interaction, Creative Collaboration, Asynchronous Interaction, Spatial Sound Composition, User-Centered Design

## ACM Reference Format:

Nathan Salin, Florent Berthaut, Ronan Gaugne, and Valérie Gouranton. 2025. Toward Multimodal Asynchronous Collaboration in VR Artistic Creation with S.P.A.R.K. In *31st ACM Symposium on Virtual Reality Software and Technology (VRST '25), November 12–14, 2025, Montreal, QC, Canada*. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3756884.3765998

## 1 Introduction

The rise of immersive technologies has opened new creative frontiers for artists working at the intersection of sound, movement, and digital media [8, 14, 24]. Virtual Reality (VR) environments, in particular, offer embodied modes of interaction and spatial composition that challenge traditional models of performance and artistic collaboration. While many VR-based systems have focused on either real-time gesture tracking or immersive sound interaction,
few enable users to compose and control the spatial and temporal relationships between movement and sound in ways that support asynchronous, cross-disciplinary creation.

In the field of media arts and performance, collaboration between dancers and musicians often requires shared timing, physical co-presence, and complex mappings between motion and audio outputs. Recent developments in networked VR systems, such as musical metaverse platforms, have demonstrated how multiple users can co-create within virtual spaces. However, these systems typically emphasize a single modality, such as shared music creation, which limits the richness of multimodal interaction. By contrast, live performance systems involving movement-to-sound mappings, like motion capture and sensor-based setups, rely on colocated and synchronous interaction, or advanced technical knowledge. These constraints may limit their accessibility and flexibility.

This paper presents S.P.A.R.K (Spatial Performance Avatar Response Kit), a Virtual Reality (VR) prototype designed to explore a different paradigm. Rather than mapping sound to live movement, the system treats prerecorded dance performances as interactive timelines. Users, primarily musicians, can spatially place boxes within a 3D environment that the recorded avatar will traverse, and which will send movement features to music production software through OpenSoundControl<fnref n="1" /> (OSC) messages. These interactions allow the creation of dynamic, multimodal compositions blending choreography and sound in a temporally decoupled process.

As a first step of an iterative design process to ensure the creative relevance and usability of S.P.A.R.K, we conducted a user study with both professional and amateur artists experienced in computer-assisted music composition. We aimed to investigate whether the VR interface would be perceived as usable and accessible across varying levels of expertise, and whether its immersive and replayable environment would facilitate a sense of creative flow, a key component in supporting artistic engagement. We also sought to understand to what extent participants would consider the system supportive of their creative process, particularly in relation to cross-modal expression between movement and sound.

These hypotheses were tested using a combination of standardized questionnaires (System Usability Scale [3] and Creativity Support Index [5]), custom surveys, and semi-structured interviews. The results helped identify areas of the prototype in need of improvement, while also highlighting the system's strengths.

# 2 Related work

Creative expression through immersive technologies spans multiple domains, from individual VR experiences to collaborative networked performances.

## 2.1 Single-User VR Creative Systems

A number of VR-based systems emphasize the creative potential of immersive environments, yet many are designed to only support one user. For instance, Flow with the Beat! by Schlagowski et al. [20] explores how virtual settings influence individual musical creativity in VR. While users expressed strong preferences for customizable environments, the system remains limited to a single user composing or playing in isolation. Similarly, Keep the

VRhythm Going by Ppali et al. [18] demonstrates that virtual reality can support musicians across rehearsal, improvisation, and composition. It even suggests benefits in reducing performance anxiety. However, despite its versatility, the system operates with only one user at a time and lacks any collaborative dimension, something central to many artistic practices.

Within these environments, dedicated 3D controls have been designed, in particular for musical expression [21]. Among potential 3D interaction techniques that can be used [1], 3D widgets that serve as virtual controls offer a flexible approach to mapping interactions in the Virtual Environment (VE) to sound parameters. Wang and Martin propose a widget composed of boxes that can be used to trigger notes [25]. Camci et al. [4] describes Inviso, which provides widgets for creating and manipulating sound trajectories in a VE. Palumbo et al. [17] explored 3D widgets that recreate a modular synthesizer patching in VR. To our knowledge, however, none of these can be used for mapping 3D captured dance movements to sound parameters.

## 2.2 Movement-Based Systems

Other systems try to explore the expressive potential of the dancer's movement through diverse means. Schlagowski et al. [26] explores the use of mixed reality mirrors to support improvisational dance by providing real-time visual feedback. While the system encourages movement exploration through visual augmentation, it is designed for solo use and focuses exclusively on visual modalities, without integrating sound or collaborative interaction directly.

As motion tracking becomes more accessible, it has become affordable for many artists. However, these technologies are not without limitations. Solberg and Jensenius [22] compared different motion capture systems for studying group dance behavior. Optical systems provide high spatial precision and accurate 3D positioning in controlled environments, but are limited by issues such as occlusion and interference from reflective surfaces. In contrast, inertial sensor systems operate independently of lighting and spatial conditions, allowing long-duration recordings with multiple participants in natural settings. However, these technologies suffer from position drift and require frequent recalibration. Additionally, relative positioning often demands more computational processing before yielding usable data compared to the direct positioning offered by optical tracking. Inertial motion tracking is used by Khutorna et al. [11], where dancers' movements are recorded and projected as reactive shadows behind them, enhancing the dancer's performance. However, the system requires frequent calibration and complex data interpretation, making it difficult to integrate into creative workflows, particularly in collaborative contexts.

Jaimovich in Emovere [10] goes a step further by replacing motion tracking with biosignals such as EMG and ECG to modulate sound in real-time. While this approach offers a novel expressive layer, it presents technical challenges: the signals require extensive processing to yield meaningful output, and tend to be unstable.

In this paper, we separate the capture of the movement and its mapping, ensuring high quality positional data that can be mapped to sound parameters.

<footnote n="1">¹https://ccrma.stanford.edu/groups/osc/index.html</footnote>
## 2.3 Multimodal Collaboration and Networked Creativity

Some projects attempt to enable collaboration in art through shared virtual spaces. LeMo by Men et al. [15], Wish You Were Here by Schlagowski et al. [19], and Musical Metaverse Playgrounds by Boem et al. [2] enable multiple users to make music together online or through LAN setups. However, to manage latency and ensure stability, these systems use simplified interaction models and imposed fixed roles, which greatly limit artistic freedom. Indeed, latency remains one of the major obstacles to real-time remote artistic collaboration. Tsioutas et al. [23] challenged traditional Network Music Performance (NMP) latency thresholds by demonstrating that musicians could tolerate one-way delays up to 40 ms, beyond the commonly accepted 20–30 ms. However, such thresholds are difficult to maintain consistently in open networks. These technical constraints are a key reason why our own system embraces asynchronous interaction.

A few systems do attempt to support real-time multimodal collaboration. CO/DA by Françoise et al. [7] stands out by allowing a programmer to live-code sound responses to dancers' movements during their performance. This project brings together music making and the dance, allowing both mediums to merge in a single collaboration. Therefore, CO/DA [7] comes closest to the approach explored here. However, it relies on a custom coding environment that does not align with the tools most artists already use, and it demands a high level of technical fluency that restricts its accessibility for a broader community of creators.

Another example is Modosc by Dahl et al. [6]. Modosc is a Max-based toolkit that extracts a comprehensive set of real-time movement descriptors, such as joint velocities, accelerations, and intermarker distances, from optical marker-based motion capture data and broadcasts them via OSC. While Modosc greatly simplifies the integration of precise motion capture data into interactive applications, its dependence on professional, marker-based hardware limits its accessibility, portability, and data stability. Furthermore, new users face a substantial learning curve: they must master both the Modosc Max patch and the interpretation of its descriptor outputs before they can begin creative work.

The systems reviewed highlight several persistent gaps. VR creative tools largely remain single-user, overlooking the collaborative nature of many artistic practices. While movement-based systems and 3D interaction techniques show promise, none offer accessible, expressive mappings between full-body dance and sound. This is largely due to motion capture technologies, which, despite becoming more accessible, still present challenges related to calibration, data stability, and integration into creative workflows. Finally, collaborative platforms often simplify interactions or impose fixed roles to manage latency, limiting artistic freedom.

## 3 Concept and Prototype

To address these shortcomings, we decided to introduce a compositional rather than performative paradigm that supports asynchronous, multimodal collaboration without the issues of real-time capture and processing. Therefore, we shift from live capture to prerecorded, avatar-driven choreography within a VR environment.

Artistically, this enables dancers and musicians to contribute separately yet coherently in a shared environment. Dancers supply expressive prerecorded choreography while musicians configure sound by arranging spatial triggers within a virtual stage. In this way, we hope to lower the technical barrier and enhance the portability of such practice.

### 3.1 Global S.P.A.R.K Overview

<figure><img src="image_3.png" /><figcaption>**Figure 2:** The system combines prerecorded avatar choreography with spatially placed interactive boxes in VR. Triggered intersections send real-time OSC messages containing motion data to external music software. This supports asynchronous collaboration between dancers and musicians while preserving familiar digital audio workflows.</figcaption></figure>

Before implementing our solution, several core challenges had to be addressed. First, we needed to find a way to transform prerecorded movement into a meaningful structure for real-time sonic interaction. Second, we sought to enable asynchronous collaboration between dancers and musicians, while maintaining expressive depth and temporal precision. Finally, the system had to remain accessible to artists with diverse technical backgrounds and integrate smoothly with their existing creative tools.

To apply those concepts, we created a VR application using Unity<fnref n="2" /> that combines a dancing avatar with spatially placed boxes, allowing the user, typically a sound artist or composer, to create sound as illustrated in Figure 2. The avatar plays animations from motion capture recordings of real dancers, performing a predefined choreography that can be played and paused at the artist's will. The artist exploits the dancer's movement through a set of interactive boxes in 3D space. These boxes are configured to send data to external software, using parameters linked to the avatar's recorded motion. This setup allows users to map sound events to the dancer's movements, effectively composing music in response to spatial and temporal patterns.

### 3.2 S.P.A.R.K Boxes

S.P.A.R.K boxes are configured to send data each time the avatar's body intersects with them during playback. These boxes do more than send static "on/off" signals. They transmit dynamic values linked to intersection data, such as the exact 3D position, limb

<footnote n="2">²Unity: https://unity.com/</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 3:** S.P.A.R.K allows users to place boxes in virtual space that generate motion data (e.g., position, speed, ...), which are sent as discrete or continuous signals to audio platforms like Pure Data, SuperCollider, and Max/MSP.</figcaption></figure>

orientation, and instantaneous velocity of the avatar segment at the moment of contact, as shown in Figure 3. This setup enables the artist to sculpt sonic responses in time and space, turning the avatar's data into music-making events. By spatially mapping sound triggers to the avatar's trajectory, the artist creates a sonic composition in three dimensions and over time, treating the dancer's movement as a reliable timeline. Boxes can be fixed in space or attached to the avatar's body, in which case they continuously transmit motion-related data associated with specific body parts, such as real-time position, rotation, or velocity. This feature enables artists to access detailed, animation-specific parameters, expanding the range of expressive control available for composition or interaction.

This approach, using boxes as an interaction medium, is similar to techniques found in both 3D non-musical interaction, such as the slicing-volume technique [16], and 2D musical interfaces, like the synthesis widget in Golan Levin's YellowTail system [13].

To ensure that the data generated by the boxes could be integrated with standard creative environments, most importantly, OSC is widely supported by standard tools in digital and media arts including, Pure Data, SuperCollider and Max/MSP, which makes it a natural choice for integration into artists' existing workflows. Its popularity in interactive and audiovisual art contexts also ensures accessibility and familiarity among users. Moreover, OSC offers a high degree of flexibility as it is platform-independent and supports a wide range of data types. This makes it well-suited to the evolving needs of our prototype, since it can be easily extended to support new modalities or connected systems without reworking the communication layer.

Artistically, this design supports a two-layer authorship. Dancers supply expressive choreography, and musicians act as spatial composers by placing trigger boxes and defining how each box's data maps to sound. Because the avatar's movement is prerecorded, each artist can work independently. Dancers can refine their sequences, while musicians adjust mappings, and both contributions can be combined into a coherent performance. This directly addresses the single-user, single-modality constraints identified in prior systems, without requiring live capture or complex real-time coding frameworks like CO/DA [7]. Since most audio synthesis platforms

already support OSC messages, the system also preserves familiar workflows.

Technically, relying on prerecorded motion and OSC messages allows musicians to integrate the system into their established music production workflow, reducing the learning curve. Dancers can focus on performance, confident that control data will remain consistent. Furthermore, stored choreography functions as a reusable asset across sessions, facilitating rapid iteration and collaboration.

### 3.3 Interaction Widgets

To enable users to manipulate both the S.P.A.R.K boxes and the avatar, we first needed to address the challenge of providing intuitive yet precise control within a fully immersive VR environment. This involved balancing both 3D and 2D interactions through the controller with the need for exact parameter adjustments, especially for complex elements like OSC message configurations.

The prototype includes a set of widgets to control each box's position, rotation, and scale in 3D space (Figure 4). These transformations can be adjusted via sliders or numerical input through a VR virtual keyboard, allowing both coarse placement and precise fine-tuning. Beyond standard UI controls, boxes can be directly grabbed for movement and rotation. Scaling is achieved with a two-handed gesture, where moving the hands apart enlarges the box and bringing them together reduces it, reflecting physical interactions.

<figure><img src="image_5.png" /><figcaption>**Figure 4:** An example of the position widget on a selected soundBox.</figcaption></figure>

A dedicated interface lets users configure OSC messages and create boxes as shown in the center of Figure 1. Users can define the IP address, port, OSC address, data type, and value. Multiple messages per box can be added, deleted, and navigated efficiently. To simplify the configuration of multiple boxes, a global tool allows updating these parameters across all boxes, simplifying large-scale setups.

Tools are also available for avatar management. Users can import FBX files containing recorded choreography via a standard file browser. A timeline interface, similar to those in video editing software, lets artists play, pause, and navigate the animation, providing precise control over playback.

The prototype serves as both a creative tool and research platform, designed to evolve iteratively through artist feedback from workshops and interviews. These insights will guide future development toward an open, adaptable system for VR-based collaborative creation.
# 4 Iterative Design Process and Evaluation

In this section, we present an overview of the design process and the initial evaluation phase of our prototype.

## 4.1 Iterative Design Process

The project employs an iterative design methodology to ensure that the system remains aligned with artists' needs and established workflows. By alternating between prototyping, evaluation, and refinement, usability issues and artistic insights are identified early, allowing progressive shaping of the tool based on direct feedback. Iterative development offers three main advantages: it accelerates learning by generating data at each stage; ensures the system remains responsive to artists' requirements, thereby increasing engagement and supporting richer creative exploration; and reduces technical risk by revealing integration challenges early. Nevertheless, this approach involves trade-offs: multiple iterations require additional time and coordination, and without careful scope management, feature proliferation can occur, potentially detracting from core objectives.

## 4.2 User study

In the evaluation phases of our design process, we aim to verify the following hypotheses:

* H1: The VR interface will demonstrate good usability. We expect participants to report good usability scores, indicating that the system is intuitive and accessible. This hypothesis is assessed using the standardized System Usability Scale (SUS) [3], complemented by a custom questionnaire focusing on usability aspects specific to our prototype.

* H2: Participants will perceive that the system supports of their creative process and elevates flow states. We anticipate that the prototype will be perceived not only as functional, but also as a facilitator of artistic expression and will promote flow states. This hypothesis will be assessed through responses to the Creativity Support Index (CSI)[5] which includes subscales such as immersion, exploration, expressiveness, and results worth effort, and a custom questionnaire addressing creative practices enabled by the system.

To evaluate the system and validate our first prototype in a controlled setting, we conducted a user study with participants from music and digital media backgrounds. This first user study focuses on evaluating whether the interface and prototype were easy to use for musicians and on identifying any issues that could indirectly affect collaboration between dancers and musicians. Therefore, this user study is not intended to examine or evaluate the collaborative interactions between musicians and dancers. Subsequent studies will examine dancers' experiences and creative workflows with the system. Each session of the user study began with a demographics questionnaire, collecting data on participants' gender, age, and musical experience. This was followed by an introduction to the system's objectives and core functionalities, along with a brief demonstration of key features, including avatar playback controls, trigger-box placement, and OSC message mapping.

Before performing a series of guided tasks, participants completed the Virtual Reality Sickness Questionnaire (VRSQ) [12] to establish a baseline and ensure the prototype did not induce VR discomfort.

The guided task was split into two parts. In the first, participants spawned and placed two boxes near the avatar's hands to practice spawning, positioning, and using the timeline. In the second, they filled the boxes with required data (e.g., IP address, port, OSC address, message type) to become familiar with the Configuration UI. Sessions typically lasted between one hour and an hour and a half, with participants spending 5-10 minutes in VR on each guided task. These steps ensured a consistent foundation of interaction across all sessions.

Following the guided phase, participants were offered a short break. This pause also provided an opportunity to gather initial impressions and informal feedback on the prototype. After the break, participants entered a free exploration phase, where they were encouraged to engage creatively and independently with the system. This phase was time-limited to 20 minutes, during which participants were not given specific instructions, could speak aloud, and were free to ask questions if needed. To support artistic experimentation within the limited timeframe, we provided a preconfigured Pure Data patch along with a set of prepared trigger-boxes offering ready-to-use OSC-triggered audio effects. In all tasks, a choreography was imposed on the user, taken from the Mixamo website<fnref n="3" />. This dance animation was not chosen at random; the movements are similar to basic Hip Hop steps commonly taught to beginners.

At the end of the session, participants completed three questionnaires. Immediately after removing the headset, they filled out a second VRSQ [12] to detect any changes in VR discomfort. Next, the System Usability Scale (SUS) [3] was administered to obtain a benchmark usability score ranging from 1 to 5. Then, participants completed the Creativity Support Index (CSI) [5], allowing us to assess whether the prototype had a supportive or limiting effect on their creative experience. It consists of six factors, Exploration, Expressiveness, Immersion, Enjoyment, Results Worth Effort and Collaboration, each is evaluated through two Likert-scale items with scores between 0 and 10 and a set of 15 pairwise comparisons where users indicate which factors they value most. The CSI yields a weighted score (0-100) reflecting both perceived support and the importance of each factor. In accordance with the recommendations provided by Cherry and Latulipe [5], we set the Collaboration factor score to zero by default, as our prototype does not support collaborative features. However, the option to assign weight to the Collaboration factor during pairwise comparisons was retained, as suggested by the authors, to potentially reveal users' perceived need for collaboration within the context of our prototype.

Finally, a structured questionnaire was also administered as a post-experiment interview to gather in-depth qualitative feedback from participants. It focused on four key themes: Understanding and Handling, Relationship with the Danced Movement, Sound Creation and Control, and Suggestions and Projections. To analyze the data collected from this custom questionnaire, we proceeded by grouping participants' comments thematically according to each question. Each remark was then linked to its corresponding item in the SUS [3] or CSI [5] questionnaires to establish connections between qualitative feedback and quantitative measures. Regarding feedback on the prototype, all user suggestions were noted,

<footnote n="3">³https://www.mixamo.com</footnote>
and particular attention was given to those recommendations men-
tioned by multiple participants. This allowed us to identify the
most frequently requested improvements to inform subsequent
development iterations.

5 Results

This section presents the findings of the study, beginning with a
summary of participant demographics, followed by quantitative
results derived from the collected data.

**5.1 User Study Demographic**

The user study was conducted with ten participants, eight of whom self-identified as male and two as female. The limited number of participants was due to the specific background required, as the study targeted individuals with experience in music and digital sound creation to ensure the prototype's relevance, which significantly reduced the potential participant pool. All participants had musical backgrounds and were recruited through calls for participation distributed via art universities, research laboratories, and direct outreach to local artists. Participants' ages ranged from 22 to 44 years (M = 30.9, SD = 7.9). According to the pre-study questionnaire, participants' general musical experience ranged from less than one year to more than ten years. Six participants were professional artists, while four were amateur musicians. Regarding electronic music experience specifically, seven participants had between one and ten years of experience, while three had no prior experience with computer-generated music. Concerning virtual reality familiarity, three participants had no prior VR experience, five used VR regularly, and two used VR in their daily activities.

5.2 Questionnaires

<figure><img src="image_6.png" /><figcaption>Figure 5: SUS and CSI score analysis.</figcaption></figure>

As can be seen in Figure 5, the global SUS score has a median
score 57.5 for an average of 52 with a standard deviation of 12.5.
This places the prototype in the "Acceptable" usability category.
Participant scores ranged from a minimum of 27 to a maximum
of 65. Question 5, regarding application functionality, received the
most favorable responses (M = 3.80, SD = 0.6), while question 9,
concerning user confidence, was also generally answered positively,
though to a lesser extent (M = 3, SD = 0.9) when taking into
consideration the user study conditions. On the contrary, question
1 (M = 2.80, SD = 1.2), about frequency of potential usage and
question 10 (M = 3.30, SD = 1.5) regarding learning difficulty

were rated poorly by most participants, with only 3 participants
providing favorable responses to these areas.

<figure><img src="image_7.png" /><figcaption>Figure 6: SUS question score analysis</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 7: CSI question score analysis.</figcaption></figure>

The CSI questionnaire [5] results revealed mixed patterns across different dimensions of creativity support, as illustrated in Figures 5 and 7. Overall CSI scores averaged 52.7 (SD = 15.5, Min = 33, Max = 79.3) with the median value at 50.5. These findings suggest that the prototype neither significantly impairs nor substantially enhances creativity support. The analysis revealed distinct patterns across questionnaire items. Questions addressing usage frequency (SUS_Q1: M = 4.0, SD = 1.35), possibilities for creative expression (Q6: M = 5.2, SD = 1.93), and user immersion within tasks (SUS_Q8: M = 5.0, SD = 2.26) received consistently lower ratings from participants. In contrast, questions about creativity enhancement (SUS_Q5: M = 6.1, SD = 2.51) and task focus (SUS_Q7: M = 6.1, SD = 2.26) showed the greatest response variability, with answers spanning the full range of possible ratings. The most positively rated aspects were user happiness while using the tool (SUS_Q2: M = 8.0, SD = 1.12) and the tool's ability to enable action tracking (SUS_Q4: M = 6.7, SD = 1.1).

To assess potential cybersickness effects, we administered the Virtual Reality Symptom Questionnaire (VRSQ) before and after prototype use. The mean difference between pre- and post-VR scores was -0.3 (SD = 6.5), indicating a slight overall decrease in reported symptoms following VR exposure. The high standard deviation can be attributed to the large number of participants who reported fewer symptoms after using the prototype than before. This suggests that, within the limited duration of the experimental session, the prototype did not induce significant cybersickness.
Note that one participant (P3) did report a marked increase in discomfort, with symptoms characteristic of VR sickness, mainly nausea. However, this case was isolated and occurred even though P3 belonged to the subgroup with no prior VR experience. No other participant, including other VR novices, reported similar effects.

Overall, these results suggest that the prototype is well-tolerated in short sessions and does not present significant risks of cyber-sickness for most users.

# 6 Discussion

The following section analyzes the results of the user study in light of our research hypotheses. We combine quantitative findings from the SUS and CSI questionnaires with qualitative insights gathered through participant interviews to better understand the system's usability, creative support, and perceived strengths and limitations.

## 6.1 Specifics of S.P.A.R.K's Asynchronous workflow

To contextualize the contributions of S.P.A.R.K, it is useful to compare it with representative real-time movement-to-sound systems such as CO/DA [7] and Modosc [6]. Unlike these synchronous systems, which depend on live motion capture and low-latency data streaming, S.P.A.R.K introduces an asynchronous workflow. This allows dancers and musicians to contribute independently while still producing a coherent final performance.

From a technical perspective, the asynchronous design eliminates the need for continuous live tracking, frequent calibration, and high-performance network infrastructure to reduce potential latency. This simplifies implementation and increases stability, allowing artists to focus on creative work rather than system maintenance. In contrast, CO/DA requires live coding to map dancer movements to sound, presenting a steep technical barrier, while Modosc demands expertise in both motion capture hardware and descriptor interpretation, which limits accessibility and portability.

Artistically, S.P.A.R.K provides greater temporal and conceptual freedom. Musicians can experiment with mappings, rearrange spatial triggers, and iteratively refine sound design without the pressure of real-time performance constraints. Dancers can perfect their choreography separately, knowing that their movements will remain stable and reusable across sessions. While S.P.A.R.K does not replace synchronous systems for live performance, it complements them by emphasizing iterative composition, accessibility, and distributed collaboration.

## 6.2 Hypotheses

The SUS score of 52.0 places the system in the "Acceptable" category, which falls short of demonstrating "good" usability according to overall benchmarks. The wide range of scores (27-65) and substantial standard deviation (12.5) indicate inconsistent user experiences. Nevertheless, detailed analysis of individual SUS questions reveals encouraging usability patterns. Questions addressing application complexity (SUS_Q2 : M = 2.2, SD = 0.9), system inconsistencies (SUS_Q6 : M = 2.4, SD = 0.69), and cumbersome-ness (SUS_Q8 : M = 2.5, SD = 0.9) all received favorable ratings, indicating low perceived complexity and good system consistency. These findings are corroborated by complementary CSI results

showing high user enjoyment (CSI_Q2 : M = 8.0, SD = 1.15), moderate ease of exploring different ideas (CSI_Q3 : M = 5.9, SD = 1.5), and good support for tracking ideas (CSI_Q4 : M = 6.7, SD = 1.16). Collectively, these results suggest that while the VR interface may not achieve excellent usability, its core interaction design demonstrates good usability characteristics, supporting partial acceptance of H1.

The results present a mixed picture regarding creative support and flow states. While participants reported high happiness using the tool (CSI_Q2 : M = 8.0, SD = 1.15), suggesting a positive user experience, the possibilities for creative expression received moderate ratings (CSI_Q6 : M = 5.2, SD = 1.93; CSI_Q5 : M = 6.1, SD = 2.51). The moderate overall CSI score (52.7) and particularly the low ratings for creative expressiveness indicate that most participants did not perceive the system as strongly supportive of their creative process. However, two participants (P5 and P9) diverged from this trend, with notably higher CSI scores of 79.33 and 74.33, likely reflecting prior familiarity with the style of dance and sound used in the prototype. This could suggest that the lower scores from other participants may be due to the constrained artistic context of the experiment, which did not align with their creative background. Regarding flow states, immersion within tasks received low ratings (CSI_Q8 : M = 5.0, SD = 2.26), contradicting the hypothesis of elevated flow. Task focus showed mixed results (CSI_Q7 : M = 6.1, SD = 2.26), with high variability indicating inconsistent flow experiences across participants. Overall, the moderate CSI score (52.7) suggests that the system neither strongly enhances nor impairs flow states. Taken together, these results indicate partial support for the hypothesis: the system can support creativity and flow under certain conditions, but effects seem widely inconsistent across participants, supporting partial acceptance of H2.

## 6.3 Result Analysis

### 6.3.1 Intuitive Interaction Design and High Engagement.
Most participants understood the role and function of the S.P.A.R.K boxes within the virtual environment with relative ease as understood from the interview. While a few needed more time to become familiar with the system's logic, all participants ultimately grasped the connection between box placement and sound triggering. The manipulation of boxes through grabbing, moving, and scaling was reported as intuitive by nearly all participants.

Only one participant expressed dissatisfaction with the current placement mechanism, stating a preference for a raycast-based interaction (i.e., pointing to place boxes at a distance). Aside from this comment, feedback regarding interaction and control was overwhelmingly positive. The only other common point of confusion stemmed from the preconfigured OSC addresses in the provided Pure Data patch, which some participants found difficult to memorize or to understand at first.

These observations align with results from the questionnaires, which included several related items that consistently received high ratings. In the CSI, participants reported a high level of enjoyment (CSI_Q2 : M = 8.0, SD = 1.15), moderate ease in exploring different ideas (CSI_Q3 : M = 5.9, SD = 1.50), and good support for
tracking evolving ideas (CSI_Q4 : M = 6.7, SD = 1.16). These re-
sponses suggest that the prototype was not only engaging but also
supported users in structuring and navigating their creative process.
Comparable results were observed in the SUS scores, particularly in
items related to system integration (SUS_Q5 : M = 3.8, SD = 1.34)
and perceived ease of use (SUS_Q8 : M = 2.5, SD = 0.94), indicat-
ing that participants found the prototype broadly usable, even if
some aspects were challenging under experimental constraints.

Overall, the prototype’s interaction design was well received. Participants found the core mechanics intuitive and engaging, with both questionnaire scores and interview feedback confirming that the system effectively supported user navigation and creative structuring. Minor usability issues were noted but did not significantly hinder the overall positive experience.

6.3.2 *Creative Expressiveness Is Highly Dependent on Content Fit.*
The CSI item “The system or tool allowed me to be very expressive” received highly polarized responses (CSI_Q8 : M = 5.20, SD = 1.93). Interviews pointed to two main sources of frustration: the fixed choreography and the limited sound palette in the provided Pure Data patch. Some participants found the dance sequence too fast or uninspiring for creative use, while others felt constrained by the lack of sound variety.

These limitations seemed to come more from the study con-
tent than from the prototype itself. This was particularly evident in
participants P5 and P8, both of whom had familiarity with the chore-
ographic style and sound aesthetics presented. These participants
reported significantly higher CSI scores (P5: 79.33, P8: 74.33) and
felt enabled to explore creatively. This highlights the importance
of aligning system content with users’ artistic backgrounds.

Despite the fixed elements, participants still employed diverse
interaction strategies. Some placed boxes on visually striking mo-
ments, others started with sound experiments and adjusted place-
ments accordingly. A few used larger boxes to target general ges-
tures rather than precise timings. Two participants even used the
boxes performatively, keeping them in hand and moving them
around the avatar to trigger sounds in real-time.

These results suggest that the prototype seems to support cre-
ative engagement, but fixed content and the short session duration
limited expressiveness for some users. Future iterations should of-
fer more flexible movement and sound options to better support
diverse artistic practices and reduce constraints on user creativity.

6.3.3 Steep Learning Curve Limits Short-Term Adoption. The Sys-
tem Usability Scale (SUS) scores obtained were in the average range
(M = 52, SD = 12.51), while the Creativity Support Index (CSI)
followed a similar trend (M = 52.7, SD = 15.51). This outcome,
though initially disappointing, can be partially explained by the
limited applicability of certain questionnaire items to the nature
of the prototype and the nature of our User study. For instance,
the SUS item "I think that I would like to use this application fre-
quently" received low scores (SUS_Q1 : M = 2.8, SD = 1.12)
from most participants. Several noted that while they did not en-
vision using the system as part of their daily creative workflow,
they considered it highly relevant for specific artistic projects. This
distinction is consistent with the intended scope of the prototype
as a specialized, project-based tool rather than a general purpose

application. Other negatively rated items from the SUS question-
naire, such as “I think that I would need assistance to be able to
use this application properly” (SUS_Q4 : M = 3.6, SD = 1.34), “I
would imagine that most people would learn to use this application
very quickly” (SUS_Q7 : M = 2.4, SD = 1.26), and “I needed to
learn a lot of things before I could get going with this application”
(SUS_Q10 : M = 3.3, SD = 1.49), significantly impacted the overall
scores of the SUS questionnaire. This suggests a relatively steep
learning curve, an expected result for a tool that, like many expert
systems, cannot be fully mastered in a short session. Only one
participant (P5), who had prior experience with similar systems,
reported minimal difficulty in adapting to the interface.

A similar pattern was observed with the CSI item “I would be
happy to use this system or tool on a regular basis” (CSI_Q3 : M
= 4.4, SD = 1.34), where responses mirrored the same reservations
about frequency of use rather than value or relevance.

These findings suggest that a longer-term evaluation might yield
more nuanced insights, particularly regarding learnability. Partici-
pants might have rated the system's usability and creativity support
more favorably had they been given time to develop proficiency
and integrate the tool into their existing workflows.

6.3.4 Limited Real-Time Feedback Hinders User Confidence. One recurring issue across both user groups concerned the lack of feedback during interaction. In the current implementation, there is no explicit visual indication when a message is sent, other than the resulting effect on sound. The only visual feedback available is the numeric data stream displayed in the UI panel of the selected box. It was reported to be too limited and cumbersome, as it required users to constantly select and inspect individual boxes. A feature currently in development, which would display real-time activation and values directly within each box in the 3D environment, was seen as a promising improvement that would address this concern.

Feedback regarding sound design and control through OSC re-
vealed significant differences between amateur and professional
participants. Those with less experience in audio software or OSC
configuration generally found the interface understandable but
the lack of feedback made it difficult to grasp the impact of their
adjustments. As a result, these users sometimes felt frustrated or
uncertain, particularly when it came to interpreting the impact of
the values they were inputting. Interestingly, these participants did
not request more functionality, but rather clearer, more immediate
feedback from the existing features.

This observation aligns with the results from the SUS question-
naire, where the item "I felt very confident using this application"
received a moderate score (SUS_Q9 : M = 3.0, SD = 0.94).

Overall, the lack of clear real-time feedback emerged as a key
factor limiting user confidence, particularly for less experienced
users, highlighting the need for more clear and immediate visual
cues to support the different type of interaction.

6.3.5 *Strong Interest in Collaboration and Environment Customization.* The final interview question addressed participants’ views on the collaborative potential of the prototype. Unsurprisingly, all participants recognized the system’s relevance in a collaborative context, particularly in pairing a dancer with a musician. Several participants also expressed a desire for an extended version of the system that would allow multiple users to interact simultaneously
within the virtual environment, configuring and manipulating the
boxes together in real-time.

However, not all responses were unreservedly optimistic. One
participant, while acknowledging the appeal of co-creation, ex-
pressed some hesitation regarding its feasibility. They pointed out
that shared sound design can often be a source of creative tension,
particularly when artistic intentions or expressive priorities differ
between collaborators. In their view, while real-time collaboration
is conceptually compelling, it also introduces the risk of interper-
sonal, aesthetic conflict or challenges that could be exacerbated
by a tool that gives users direct and overlapping influence over
sonic outcomes. According to this participant, collaborative use is
possible, but would likely require careful design and mediation to
function effectively in practice.

This emphasis on collaboration from users also helps explain
some of the discrepancies observed in the CSI scores. As recom-
mended by Cherry and Latulipe [5], the items specifically related
to collaboration were scored as zero in cases where the task was
performed individually. However, participants were still asked to
express their preferences in the paired-statement format, including
those involving collaboration. In several cases, participants consis-
tently favored collaborative features in their responses, despite the
fact that these dimensions were excluded from the final score calcu-
lation. This discrepancy resulted in responses that, while strongly
positive in terms of intent and expectation, were numerically pe-
nalized for the score due to the methodological constraints.

Another frequent remark concerned the lack of environmental
variety within the prototype. Currently, the system offers only a
single virtual environment, and several users expressed a desire for
multiple scene options that could better support their creative en-
gagement. This feedback is consistent with findings from previous
research, notably by Ppali et al. [18] and Schlagowski et al. [20],
which highlight the significant influence of virtual environments on
users' creative states and flow experiences in VR systems. Another
suggested improvement is the ability to customize the avatar's ap-
pearance, as prior research by Hu et al. [9] has shown that visual
identity can influence user behavior in virtual environments.

These results confirm that collaboration and environment cus-
tomization are key expectations for artists. While participants val-
ued the system's collaborative potential, they also pointed to possi-
ble creative conflicts, highlighting the need for thoughtful design.
The lack of environmental variety was seen as limiting, reinforcing
prior findings by other researchers [18, 20] that virtual context
influences creative engagement.

6.3.6 *Other recurring suggestions.* The interviews also served as a
crucial source of feedback on missing or misunderstood features,
directly supporting the iterative design process. One of the most
frequently requested features was a memory system for OSC ad-
dresses. Participants noted that having to manually re-enter OSC
strings was both tedious and repetitive, particularly in cases where
only a few addresses were used across the entire session. Sugges-
tions included storing the most recently used addresses or allowing
users to favorite commonly used parameters for quick access.

Another recurring suggestion was the inclusion of a tool for
manually sending OSC messages without the need to interact with
a physical box. This feature would be seen as especially useful

for testing parameter changes or resetting values during the music
making process, and would reduce friction when switching between
design and testing phases without removing the headset.

The timeline interface also drew several suggestions for improve-
ment. Participants found the current slider imprecise, making it
difficult to select specific moments in the choreography. They pro-
posed the ability to enter a timestamp numerically, or an improved
control mechanism for more responsive adjustment using VR con-
trollers. Some participants requested control over playback speed,
arguing that the ability to slow down or speed up the animation
would make it easier to analyze complex movement and position
interaction boxes more effectively.

Regarding the S.P.A.R.K. boxes themselves, many participants
asked for a duplication feature that would allow them to copy
boxes in place while preserving their configuration. Although it
is currently possible to copy OSC messages between boxes, this
feature was underused, likely due to its unintuitive implementation.
A simpler and more direct method for duplicating entire boxes
should improve workflow efficiency. On the same topic, participants
requested new types of boxes, such as mobile boxes that move along
predefined paths for more dynamic interactions, or time-sensitive
boxes that activate only during specific timeline windows.

Finally, some users, particularly professionals, requested ad-
vanced OSC controls. This included options to scale data, constrain
outputs within specific ranges, or remap values to fit musical inten-
tions. These were not seen as flaws in the core interaction model,
but as refinements that would improve the system's usability in
more demanding, professional contexts.

7 Conclusion

This paper presented S.P.A.R.K, a VR-based prototype designed to
support asynchronous, multimodal collaboration between dancers
and musicians. The system reimagines interaction by linking pre-
recorded motion to sound triggers in space, enabling an original
form of composition where movement drives sonic outcomes asyn-
chronously. Our user study with artists of varying backgrounds
revealed both the expressive potential and current limitations of
the first iteration of the prototype. While usability and creative
support scores were moderate, qualitative feedback emphasized
strong interest in the system's collaborative and artistic capabilities.
These insights have directly informed the next design iteration,
with improvements focused on workflow clarity, expanded sound
control, and enhanced customization. As future work, we plan to
integrate the feedback received into the next version of the pro-
totype and conduct a follow-up workshop in a more artistically
appropriate context, allowing for longer-term engagement and re-
ducing the learning curve and creative constraints observed in this
initial evaluation.

Acknowledgments

This work was partially funded by INSA Rennes and Equipex+
Continuum ANR-21-ESRE-0030. This work was supported by a
French government grant managed by the Agence Nationale de la
Recherche as part of the France 2030 program, reference ANR-22-
EXEN-0002 (PEPR eNSEMBLE / CATS)
References

[1] Florent Berthaut. 2020. 3D interaction techniques for musical expression. *Journal of New Music Research* 49, 1 (2020), 60–72.

[2] Alberto Boem and Luca Turchet. 2023. Musical Metaverse Playgrounds: exploring the design of shared virtual sonic experiences on web browsers. In *2023 4th International Symposium on the Internet of Sounds*. IEEE, 1–9.

[3] John Brooke. 1996. SUS – a quick and dirty usability scale. 189–194.

[4] Anil Çamci, Kristine Lee, Cody J Roberts, and Angus G Forbes. 2017. INVISO: a cross-platform user interface for creating virtual sonic environments. In *Proceedings of the ACM Symposium on User Interface Software and Technology*. ACM, 507–518.

[5] Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. *ACM Transactions on Computer-Human Interaction (TOCHI)* 21, 4 (2014), 1–25.

[6] Luke Dahl and Federico Visi. 2018. Modosc: a library of real-time movement descriptors for marker-based motion capture. In *Proceedings of the 5th International Conference on Movement and Computing*. 1–4.

[7] Jules Françoise, Sarah Fdili Alaoui, and Yves Candau. 2022. CO/DA: Live-Coding Movement-Sound Interactions for Dance Improvisation. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*. 1–13.

[8] Max Graf and Mathieu Barthet. 2022. Mixed reality musical interface: Exploring ergonomics and adaptive hand pose recognition for gestural control. In *NIME 2022. PubPub*.

[9] Yong-Hao Hu, Yuji Hatada, and Takuji Narumi. 2023. Beyond Mirrors: Exploring Behavioral Changes through Comparative Avatar Design in VR Taiko Drumming. In *Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology (Christchurch, New Zealand) (VRST '23)*. Association for Computing Machinery, New York, NY, USA, Article 31, 11 pages. doi:10.1145/3611659.3615690

[10] Javier Jaimovich. 2016. Emovere: Designing sound interactions for biosignals and dancers. In *Proceedings of the international conference on new interfaces for musical expression*. 316–320.

[11] Sofia Khutorna, Derek Jacoby, Anthony Estey, and Yvonne Coady. 2024. Motion Capture Technology for Enhancing Live Dance Performances. In *2024 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM)*. IEEE, 1–6.

[12] Hyun K Kim, Jaehyun Park, Yeongcheol Choi, and Mungyeong Choe. 2018. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. *Applied ergonomics* 69 (2018), 66–73.

[13] Golan Levin. 2000. *Paintery interfaces for audiovisual performance*. Ph. D. Dissertation. Massachusetts Institute of Technology.

[14] Julien Lomet, Ronan Gaugne, and Valérie Gouranton. 2022. Could you relax in an artistic co-creative virtual reality experience?. In *ICAT-EGVE, joint international conference of the 32nd International Conference on Artificial Reality and Telexistence & the 27th Eurographics Symposium on Virtual Environments*. 1–9.

[15] Liang Men and Nick Bryan-Kinns. 2019. LeMo: exploring virtual space for collaborative creativity. In *Proceedings of the 2019 Conference on Creativity and Cognition*. 71–82.

[16] Roberto A. Montano-Murillo, Cuong Nguyen, Rubaiat Habib Kazi, Sriram Subramanian, Stephen DiVerdi, and Diego Martinez-Plasencia. 2020. Slicing-Volume: Hybrid 3D/2D Multi-target Selection Technique for Dense Virtual Environments. In *2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*. 53–62. doi:10.1109/VR46266.2020.00023

[17] Michael Palumbo, Alexander Zonta, and Graham Wakefield. 2020. Modular reality: Analogues of patching in immersive space. *Journal of New Music Research* 49, 1 (2020), 8–23.

[18] Sophia Ppali, Vali Lalioti, Boyd Branch, Chee Siang Ang, Andrew J Thomas, Bea S Wohl, and Alexandra Covaci. 2022. Keep the VRhythm going: A musician-centred study investigating how Virtual Reality can support creative musical practice. In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*. 1–19.

[19] Ruben Schlagowski, Dariia Nazarenko, Yekta Can, Kunal Gupta, Silvan Mertes, Mark Billinghurst, and Elisabeth André. 2023. Wish you were here: Mental and physiological effects of remote music collaboration in mixed reality. In *Proceedings of the 2023 CHI conference on human factors in computing systems*. 1–16.

[20] Ruben Schlagowski, Fabian Wildgrube, Silvan Mertes, Ceenu George, and Elisabeth André. 2022. Flow with the beat! Human-centered design of virtual environments for musical creativity support in VR. In *Proceedings of the 14th Conference on Creativity and Cognition*. 428–442.

[21] Stefania Serafin, Cumhur Erkut, Juraj Kojs, Niels C Nilsson, and Rolf Nordahl. 2016. Virtual reality musical instruments: State of the art, design principles, and future directions. *Computer Music Journal* 40, 3 (2016), 22–40.

[22] Ragnhild Torvanger Solberg and Alexander Refsum Jensenius. 2016. Optical or inertial? Evaluation of two motion capture systems for studies of dancing to electronic dance music. (2016).

[23] Konstantinos Tsioutas, George Xylomenos, and Ioannis Doumanis. 2021. An empirical evaluation of QOME for NMP. In *2021 11th IFIP International Conference on New Technologies, Mobility and Security (NTMS)*. IEEE, 1–5.

[24] Adinda van't Klooster and Nick Collins. 2024. Voice Responsive Virtual Reality. In *Proceedings of the International Conference on New Interfaces for Musical Expression*. 346–350.

[25] Yichen Wang and Charles Martin. 2022. Cubing sound: Designing a NIME for head-mounted augmented reality. In *NIME 2022. PubPub*.

[26] Qiushi Zhou, Louise Grebel, Andrew Irlitti, Julie Ann Minaai, Jorge Goncalves, and Eduardo Velloso. 2023. Here and now: Creating improvisational dance movements with a mixed reality mirror. In *Proceedings of the 2023 CHI conference on human factors in computing systems*. 1–16.