# HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument
## 2 Related Work
The results showed high accuracy in recognizing the finger gestures, highlighting the effectiveness and potential of machine learning for enhancing virtual musical instruments and providing new opportunities for musical expression. [PERSON] (2016) explored the development of a virtual keyboard instrument and hand tracking in a virtual reality (VR) environment. The aim was to investigate the feasibility of using VR technology to create a realistic and interactive keyboard instrument that allowed users to play music with their hands in a virtual space. The evaluation of the study showed that the virtual keyboard instrument and hand-tracking system were effective and intuitive for users to use and that the system had the potential to be used in music education and performance. Moreover, [PERSON] (2017) introduced Entangled, which was an interactive instrument that enabled multiple users to create music in a virtual 3D space using their smartphones for gesture control. The instrument used a combination of audio, visual, and haptic feedback to create a unique, immersive experience for the users. In Entangled, users could apply gravitational force to a swarm of particles, where their gestures and movements affected the sound and visuals in the virtual space. The instrument was designed to be accessible and easy to use, with no prior musical experience necessary. The study by [PERSON] and [PERSON] (2018) introduced a new interactive musical interface called MuTable that can turn any surface into a musical instrument, allowing users to create and perform music using gestures and movements. MuTable consisted of a camera, a projector, and a computer running specialized software. The camera captured the user's hand movements and gestures, which were then processed by the software to generate musical notes and sounds, while the projector projected visual feedback onto the surface, indicating where the user's hands should go to play different notes and sounds. They evaluated the system in terms of performance and usability and they found that the system was easy to use and allowed users to create and perform music without prior musical training. Finally, Ref. [19] presented the development and evaluation of a unique interactive musical instrument called Sound Forest/Ljudskogen, which utilized a large-scale setup of strings to create an immersive and engaging musical experience for users. The researchers described the design process, including the placement and tension of the strings, the sensing technology, and the mapping between the user's gestures and the produced sound. The findings of the evaluation demonstrated the positive reception and impact of Sound Forest/Ljudskogen placed at a museum. Participants expressed enjoyment and excitement in playing the instrument, highlighting its immersive and intuitive nature. In conclusion, in the research area of DMIs, special emphasis is placed on interaction and specifically on gestures. Hand gesture analysis and control of audio processing is an active area of research in the fields of computer science and music technology. The use of hand gestures to control audio processing has the potential to provide a more natural and expressive form of interaction between the user and the computer. Therefore, researchers have explored various machine learning models, datasets, and applications to enhance gesture recognition and sound production in DMIs. The current paper proposes the Interaction Subsystem, which implements well-known techniques for capturing, analyzing, and recognizing gestures as well as producing sounds, according to the literature review. In addition, to the best of our knowledge, it contributes to the field of museums, since it presents the first investigation of a holistic approach, called HapticSOUND, which includes three subsystems, the Interaction, the Information, and the Entertainment Subsystems, for visitors to learn about traditional musical instruments, play games, and interact with a DMI.