# HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument
## 4 Implementation of the Interaction Subsystem  
In this section, the implementation of the Interaction Subsystem is described in more detail. As already mentioned in Section 3.3, two ToF cameras are used to capture the  
Figure 8: Interaction Subsystem. Figure 7: TMI replica with 2 ToF cameras. user's gestures. Camera 1 records and monitors the distance of the instrumentalist's left hand touching the keyboard (we consider the performer to be right-handed), while camera 2 records the tempo given by the instrumentalist with his/her right hand (Figure 9). These ToF cameras are IFM3D O3X101, which provide point clouds, depth maps, and grayscale images. The resolution of the camera is 224 \(\times\) 172 pixels with corresponding viewing angles of 60\({}^{\circ}\)\(\times\) 45\({}^{\circ}\). The IFM3D cameras capture the user's finger gestures. A Region of Interest (ROI) in the view field of each camera is specified (Figure 10) and the captured data corresponding to the ROI are processed by the Raspberry Pi 4 microcontroller. For finger gesture recognition, an algorithm was developed in Python, utilizing the OpenCV library. The algorithm runs on the microcontroller in order to extract features such as position coordinates (including depth), angles, velocity, etc., which are the input data to the system. In addition, the user interface was implemented in the Max/MSP programming environment, which is one of the most popular software environments for creating interactive and real-time DMIs [5,13,15] that can be used to recognize a wide range of hand gestures. The user interface was then embedded into Unity, through the use of mira.frame ([[https://docs.cycling74.com/maX7/refpages/](https://docs.cycling74.com/maX7/refpages/)]([https://docs.cycling74.com/maX7/refpages/](https://docs.cycling74.com/maX7/refpages/)) mira.frame, accessed on 7 May 2023), where the Interaction System runs (Figure 11). Figure 10: **(a)** ROI image from ToF camera in 2D, **(b)** ROI image from ToF camera in 3D (capturing the depth). Figure 9: ToF Camera Placement on the Music Instrument’s Replica. For Case 1 (performing single musical notes and receiving auditory feedback in real time), initially, all single notes are recorded while the expert music teacher performs with the real traditional musical instrument and stored in the HapticSOUND Database. Next, the user performs fingerings on the TMI replica, where his/her gestures are captured. These gestural data, which correspond to predefined values of variables A, B, and C (Figure 9), are mapped to single notes each time the user's hands take a specific position. To achieve this, the mapping that has been implemented is a well-known technique called direct (explicit) mapping [21] and one-to-one, where an input parameter is mapped to a composition parameter. As a result, single notes are produced (auditory feedback) according to the user's fingerings. For example, when the distance of the left hand is A1 with a specific finger arrangement B1 (output by camera 1) and the rhythm given by the right hand is C1 (output by camera 2), the sound of note D1, which was recorded from the corresponding position and is in the database, is triggered. Summarizing, in this case, the intuitive and straightforward nature of direct and one-to-one mapping simplifies the learning process for users using the DMI because it allows them to quickly establish a connection between their gestures and the sound they want to produce, facilitating a more natural and expressive interaction with the instrument. For Case 2 (performing a music excerpt and receiving visual and auditory feedback in real time), both the gestural data and the corresponding audio excerpt are captured while the expert music teacher performs with the real traditional musical instrument. These data feed the machine learning algorithm (training phase) and, as a result, gestural data are mapped to audio excerpts through the technique of implicit mapping [22].