# MagneTip: Reintroducing a Physical Interaction Loop for 3D Musical Drawing in Extended Reality

Esther Gruy
CRISTAL, CNRS,
University of Lille, France
esther.gruy@univ-lille.fr

Florent Berthaut
CRISTAL, CNRS,
University of Lille, France
florent.berthaut@univ-lille.fr

## ABSTRACT

Extended Reality interfaces open numerous opportunities for musical expression. One of them is 3D musical drawing, i.e., the ability to draw sonic and visual paths in virtual and physical spaces. However, in existing instruments (controllers, hand tracking) most of the interaction happens in the digital/virtual space, which reduces primary audio and haptic feedback and might limit the expressiveness one has with physical drawing and acoustic instruments.

In this paper, we propose a novel design approach which moves part of the 3D interaction back to the physical space for more intimate controls and more direct feedback, creating a physical interaction loop connected with the virtual interaction loop. We then present MagneTip, a first implementation of this approach for 3D musical drawing, which enables one or two handed interaction and combines co-localised and spatialised feedback.

## Author Keywords

3D musical drawing, extended reality, audio-haptic feedback

## CCS Concepts

*Applied computing → Sound and music computing; Performing arts; *Human-centered computing → Mixed / augmented reality;

## 1. INTRODUCTION

Musical expression and (live) drawing have often been associated, from drawing-based composition tools (e.g., Iannis Xenakis' Upic) to instruments that generate both visual and sonic traces. What they have in common is the use of movements and positions to create audiovisual spaces.

The generalisation of Extended Reality (XR) display technologies now facilitates the design of instruments based on 3D musical drawing. We believe that these constitute an important research subject, because: 1) they associate expressiveness in both visual and auditory modalities; 2) they allow for a variety of mappings between gestures, visuals

<figure><img src="image_1.png" /><figcaption>Figure 1: A) User equipped with a mixed-reality HMD and using MagneTip B) 3D drawing as seen through the HMD</figcaption></figure>

and sounds; 3) they may benefit from the expertise of both musicians and visual artists.

However, the devices available for 3D drawing are far from providing the expressiveness of physical drawing, i.e., intimate brush to paper interaction and the corresponding audio and haptic feedback. In particular, all controls and feedback go through a digital loop that induces latency and reduces complexity through gestures tracking, audio/haptic/visual processing and feedback rendering.

In this paper, we propose to achieve more intimate interactions in 3D musical drawing by: 1) introducing a novel approach that moves part of 3D musical interactions out of the digital space and back to the physical space, and 2) designing a novel 3D drawing device that relies on this approach.

### 1.1 Related work

#### 1.1.1 Audio-Haptic feedback in 3D interaction

While 3D interactions in virtual environments rely mostly on visual feedback and gesture tracking, multisensory feedback can help with the perception of a given space and its content [15] [17], and reduces the uncertainty of an event [19].

In the context of musical creation (digital instruments), audio and haptic feedback are combined as an attempt to replicate the intimacy provided by an acoustic instrument. Leonard et al. [13] propose a platform for multisensory interaction in virtual reality using mass-interaction physical modelling. Çamcı and Granzow tackle musical expression with Hyperreal Instruments [10] by constructing an instrument providing audio, visual and haptic feedback. Arslan et al. [5], in Vibrating Shapes, combine 3D interaction in Spatial Augmented Reality (SAR) and actuated instruments to

<figure><img src="image_2.png" /></figure>

Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s).

NIME'24, 4–6 September, Utrecht, The Netherlands.
generate sound in the physical space.

In our approach, audio and haptic feedback come from
the virtual interaction loop, but also as part of a physical
interaction loop that communicates with the virtual com-
ponents and provides active feedback which the user can
appropriate, so that the result of the musical interaction is
both virtual and physical.

1.1.2 3D drawing devices and techniques

3D drawing implies more constraints than 2D drawing, as it
has more degrees of freedom. It, however, allows for inter-
esting or unusual interaction techniques, which would not
be possible inside a 2D space.

Dudley et al. [11] use a HoloLens to create an applica-
tion for lines, splines and freehand drawing. With HoloARt,
Amores et al. [4] also use a HoloLens to paint, spray or
splatter virtual paint onto the environment. "Drawing on
air" [12] mixes 3D drawing in virtual reality with haptic
feedback, using a Phantom force feedback stylus. A num-
ber of commercial 3D drawing applications have been im-
plemented in recent years, such as Gravity Sketch [2], A-
Painter [1] or TiltBrush [3].

It is important to note that 3D musical drawing is a more
specific case. The main focus is split between the visual
and the audio feedback, meaning that less complexity can
be expected on the drawing side of the application.

1.1.3 2D/3D musical drawing

Drawing has been used to create music or sounds on several
occurrences. In Levin's work on audiovisual interfaces [14],
both *Yellowtail* and *Loom* use drawings as a way to generate
sound, by either animating or looping them. With Differ-
ent Strokes [20], particles spawn and follow the drawing
according to the user's gesture to produce sound. In Illusio
[6], the drawings serve as a control interface for live-looping
performances.

Fewer projects have explored 3D musical drawing. Mag-
ical Brush [18] uses a mobile device to draw in augmented
reality, with audio and visual feedback mapped to its mo-
tion characteristics. In the Reflets system, Berthaut et al.
[9] use a SAR display to create 3D sound paths that are
revealed by both musicians and spectators. Nakagawa et al.
[16] present a 3D audiovisual environment where users
can draw bare-handed, with sounds mapped to the height
of a stroke.

Although there are projects which explore musicality, hap-
tic feedback and drawing, none actually combine all of them.
MagneTip aims to create a 3D musical drawing environ-
ment for both musicians and visual artists, that is focused
on achieving more intimate interactions with the system.

2. PHYSICAL AND VIRTUAL
3D INTERACTION LOOPS

Expertise in physical drawing and acoustic music playing
relies on an interaction loop with fast, precise and rich audi-
tory and haptic feedback. Transferring these activities to a
virtual space, for instance in the case of 3D musical drawing,
implies the creation of an interaction loop which involves
gesture tracking, 3D interaction techniques, visual to sound
and visual to haptic mappings, and audio/visual/haptic ren-
dering. This virtual interaction loop introduces indirections
[7] and a potential loss in temporal and spatial accuracy of
both control and feedback. This loss comes from the choice
of gestures tracking and feedback technologies, and of soft-

<figure><img src="image_3.png" /><figcaption>Figure 2: General Approach: Gestures are separated be-
tween physical and virtual interaction loops which exchange
data. Feedback from the two loops are combined for the user
to perceive.</figcaption></figure>

ware mapping and rendering techniques. In addition, most
feedback on the resulting visual and sonic drawing remains
secondary, meaning that it is produced after virtual pro-
cessing of gestures and mappings. One solution to bring
virtual interaction closer to the physical one is to improve
hardware and software components of the virtual interac-
tion loop, e.g., increase sampling and display rates and res-
olutions.

Instead, we propose to take elements of the interaction
loop out of the digital (virtual) space and place them back
into the physical space, so that part of the 3D musical in-
teraction happens without sampling and synthesis steps,
and benefits from the richness of interaction in the phys-
ical world.

As shown in Figure 2, our approach relies on a separation
of musical gestures, on physical and virtual interaction loops
that exchange data, and on the combination of feedback.

## 2.1 Gestures Separation

Gestures can be used in both the virtual and physical in-
teraction loops. In the virtual one, they will be sensed,
through motion tracking and the various sensors present on
controllers, sampled, and used as part of interaction tech-
niques to select, manipulate and navigate 3D content [8]. In
the physical one, they will directly affect the behaviour of
physical components (magnetic, mechanical, optical) which
in turn will generate audio, haptic and/or visual feedback.

## 2.2 Physical Interaction

Components of the physical interaction loop enable the di-
rect production of sounds, visuals and haptic feedback from
artists' gestures, co-localised with these gestures, without
the need for digital conversion, i.e., without sampling, pro-
cessing and displaying steps. Audio and haptic interactions
can be obtained, for example, through the manipulation of
mechanical devices, e.g., vibration speakers or actuators,
or electromagnetic devices, e.g., coil-magnet pairs. Visual
interactions can be performed using projectors, mirrors,
lenses, and so on. The result of the physical interaction
loop is intimately connected with the performed gestures,
co-localised and temporally synchronous.
## 2.3 Virtual Interaction

Components of the virtual interaction loop are comprised of 1) 3D interaction techniques that allow users to manipulate a virtual scene, 2) audio, visual and haptic synthesis processes 3) mappings between the manipulated scene and synthesis parameters. In the case of 3D musical drawing, the virtual interaction consists in moving a 3D brush inside a virtual space and generating a 3D geometry in addition to a soundscape.

## 2.4 Physical-Virtual Exchanges

Between the two loops, visual, audio and haptic content can be exchanged through analysis and synthesis.

Exchange of sound from virtual to physical can be done via coils, actuators, vibration speakers, so that the sound produced through 3D interaction techniques can be modulated. In the other direction, microphones/piezoceramic transducers can capture the sound to be transformed digitally (amplified, processed...) or to serve as input for the 3D interaction techniques. In our 3D drawing context, perceptual sonic features (loudness, brightness, noisyness...) could drive the appearance of the visual brush. Exchange of haptic data from virtual to physical can be done using actuators, vibrotactile, force-feedback devices and from physical to virtual using passive magnets, tangible objects that will influence the gestures performed in the virtual space (e.g., as done in encounter-type haptics). Finally, exchanges of visual data is enabled in one direction by projecting visual content with projectors, such as with SAR displays [5], and in the other direction using cameras or photoresistors.

Exchanges also provide the opportunity of many types of multimodal physical-virtual feedback loops, where capture of physical audio interaction can drive visual interaction resulting in virtual haptic feedback, which can in turn generate vibrations picked up as a sound signal...

## 2.5 Feedback Combination

Finally, the feedback, i.e., result of interaction, of both loops, is merged and provided to the user. Part of the feedback will be co-localised with the device, e.g., sound and haptic display from the physical loop or visual feedback from a head-mounted display (HMD) or SAR. Another part can be distant, e.g., audio display from an array of speakers or visual content placed away from the user. In the case of 3D musical drawing, visual feedback from the virtual loop will consist in drawings placed around the user in the virtual space. Audio feedback can be both localised at the current brush position, but it can also come from non-spatialised ambient sound generated from the combination of previously drawn paths.

# 3. DESIGN OF MAGNETIP

In this section, we describe how we implemented our approach in the form of MagneTip<fnref n="1" />.

For the physical interaction loop, MagneTip relies on the principle of a loudspeaker. A membrane with a flat coil (created using a LPKF ProtoLaser H4 device and an 8.5x8.5 cm copper panel on fiberglass) is attached to a wooden sound box (figure 4). The audio signal generated from the virtual interaction loop (using a PureData patch), which depends on 3D drawing brush movements, is transmitted to the coil through a soundcard and amplifier, hence generating an oscillating magnetic field (figure 3). When a magnet

<figure><img src="image_4.png" /><figcaption>Figure 3: Base components and signal/data flow in MagneTip</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Sketch of MagneTip's device</figcaption></figure>

is brought closer to the coil, the magnetic attraction makes both the membrane and the magnet vibrate. This produces an audio-haptic feedback directly from the physical space, co-localised with the user's gesture. Movements of the magnet above the coil affect both the amplitude of the feedback but also the sound timbre. <fnref n="2" /> The virtual interaction loop receives sound captured from the physical one to influence brush properties. It also receives the overall position of the coil (with 4 Optitrack cameras) which is used in the virtual interaction loop to control 3D drawing (inside Godot 4.2), in turn influencing the sound generated which is sent back to the physical loop. Virtual sound feedback relies on granular synthesis with controls in grain window position and pitch, which passes through a feedback delay and a volume envelope modulator with controllable tempo and sustain. The volume envelope of this synthesis is also applied to a low frequency signal added to the output in order to increase the haptic vibrations felt from the physical loop. Fine grained gestural control of the magnet position constitute the physical interaction, while larger gestures are used to drive virtual interaction. Audio feedback can be output from both interaction loops, while the visual feedback only comes out of the virtual loop, using a Varjo XR-3 mixed-reality HMD, and haptic feedback only from the physical loop.

## 4. CONCLUSION

<footnote n="2">²This work was in part inspired by Belgian sound artist Els Viaene</footnote>

<footnote n="1">¹A video of interactions with MagneTip can be found here</footnote>
<figure><img src="image_6.png" /><figcaption>**Figure 5:** Interaction configurations with MagneTip: A) Unimanual version with coil attached to forearm and stylus B) Unimanual version with finger glove C) Bimanual version with stylus D) Bimanual version with finger glove</figcaption></figure>

In this paper, we proposed a novel approach for 3D musical interaction that reintroduces a physical interaction loop in addition to the digital/virtual one. We implemented this approach for 3D musical drawing with the MagneTip system.

Future work will investigate long term appropriation of this device by musicians and visual artists and explore the various mappings for 3D musical drawing that can be created in the physical and virtual interaction loops.

## 5. ETHICAL STANDARDS

This research was conducted using public University funding for the equipment, with the aim of reducing the technological footprint of 3D musical drawing.

## 6. ACKNOWLEDGMENTS

The authors would like to thank the PIRVI platform (Samuel Degrande) for their help with setting up the Varjo headset and Optitrack tracking, and the PCMP platform, Pierre Laly and Rédha Kassi for their help building the MagneTip device.

## 7. REFERENCES

[1] A-painter. https://blog.mozvr.com/a-painter/.

[2] Gravity sketch. https://www gravitysketch.com/.

[3] Tilt brush. https://tiltbrush.com/.

[4] J. Amores and J. Lanier. Holoart: painting with holograms in mixed reality. In *Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems*, pages 421–424, 2017.

[5] C. Arslan, F. Berthaut, A. Beuchey, P. Cambourian, and A. Paté. Vibrating shapes : Design and evolution of a spatial augmented reality interface for actuated instruments. In *New Interfaces for Musical Expression*, Auckland, New Zealand, June 2022.

[6] J. Barbosa, F. Calegario, V. Teichrieb, G. Ramalho, and G. Cabral. Illusio: A drawing-based digital music instrument. In *Proceedings of the International Conference on New Interfaces for Musical Expression*, pages 499–502, 2013.

[7] M. Beaudouin-Lafon. Instrumental interaction: an interaction model for designing post-wimp user interfaces. In *Proceedings of the SIGCHI conference on Human factors in computing systems*, pages 446–453, 2000.

[8] F. Berthaut. 3D interaction techniques for musical expression. *Journal of New Music Research*, 2020.

[9] F. Berthaut, D. Martinez Plasencia, M. Hachet, and S. Subramanian. Reflets: Combining and Revealing Spaces for Musical Performances. In *New Interfaces for Musical Expression (NIME)*, Baton Rouge, United States, May 2015.

[10] A. Çamcı and J. Granzow. Hyperreal instruments: Bridging vr and digital fabrication to facilitate new forms of musical expression. *Leonardo Music Journal*, 29:14–18, 2019.

[11] J. J. Dudley, H. Schuff, and P. O. Kristensson. Bare-handed 3d drawing in augmented reality. In *Proceedings of the 2018 Designing Interactive Systems Conference*, pages 241–252, 2018.

[12] D. Keefe, R. Zeleznik, and D. Laidlaw. Drawing on air: Input techniques for controlled 3d line illustration. *IEEE transactions on visualization and computer graphics*, 13(5):1067–1081, 2007.

[13] J. Leonard, R. Cadoz, N. Castagné, J.-L. Florens, and A. Luciani. A virtual reality platform for musical creation: Genesis-rt. In *Sound, Music, and Motion: 10th International Symposium, CMMR 2013, Marseille, France, October 15-18, 2013. Revised Selected Papers 10*, pages 346–371. Springer, 2014.

[14] G. Levin. *Painterly interfaces for audiovisual performance*. PhD thesis, Massachusetts Institute of Technology, 2000.

[15] A. Marquardt, E. Kruijff, C. Trepkowski, J. Maiero, A. Schwandt, A. Hinkenjann, W. Stuerzlinger, and J. Schöning. Audio-tactile proximity feedback for enhancing 3d manipulation. In *Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology*, pages 1–10, 2018.

[16] R. Nakagawa, R. Komatsubara, T. Ota, and T. Mitsumori. Drawing sound in mr space: A multi-user audiovisual experience in mixed reality space. In *Proceedings of the Virtual Reality International Conference-Laval Virtual*, pages 1–4, 2018.

[17] R. Pugliese, A. Politis, and T. Takala. Spatial rendering of audio-tactile feedback for exploration and object interaction in virtual environments. In *SMC 9th Sound and Music Computing Conference (SMC 2012)*; Copenhagen, Denmark; July 11-14, 2012, 2012.

[18] R. Valer, R. Schramm, and L. Nedel. Musical brush: Exploring creativity in an ar-based tool combining music and drawing generation. In *2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)*, pages 634–635. IEEE, 2020.

[19] N. Van Atteveldt, M. M. Murray, G. Thut, and C. E. Schroeder. Multisensory integration: flexible use of general operations. *Neuron*, 81(6):1240–1253, 2014.

[20] M. Zadel and G. Scavone. Different strokes: a prototype software system for laptop performance and improvisation. In *Proceedings of the 2006 conference on New interfaces for musical expression*, pages 168–171, 2006.