# Interactions in Augmented and Mixed Reality: An Overview
## 3 Research Results
### Audio-Based Modality
#### 3.2.2 Speech-Based  
Besides engineering, sound-based interactions can produce benefits in education by exploiting _speech recognition_. [PERSON] et al. [123] developed an AR application to enhance vocabulary learning in early education. The authors integrated visual scripts for orthography and audio for phonology. As a result, they concluded that such applications provide high levels of user satisfaction and can significantly affect pronunciation as students repeat words and phrases until they get correct feedback. In [PERSON] et al.'s [124] work, the authors modified the VOMAR application [125]. They added the Ariadne [126] spoken dialogue system using Microsoft's Speech API [127] as a speech recognition engine to allow people to rapidly put together interior designs by arranging virtual furniture in  
Figure 3: Displaying all categories of the audio-based modality.  
empty rooms. _Spech recognition_ is commonly used for computers to understand auditory commands and execute tasks. Among others, the authors used in their application the \"select\", \"move\", and \"place\" commands. Having an answer to the question \"what are they saying\", the next step is to answer the question \"who is speaking\". _Speaker recognition_ is the method of interaction where machines can produce several pieces of information about the person who is speaking or even identify the person when the information provided can support that. In [PERSON] et al.'s [128] exceptional review, several research papers regarding the _speaker recognition_ method were included. They further classified this method in identification, verification, detection, segmentation, clustering, and diarization and proposed issues regarding variability, insufficient data, background noise, and adversarial attacks. [PERSON] et al. [129] reviewed technological developments in augmented reality worlds, emphasizing speech and gesture interfaces. They stressed that speaker verification could be used for authentication purposes before starting a dialogue, for example, regarding a bank transfer in a virtual world.  
[PERSON] et al. [130] designed and built an emotion recognition system in smart glasses to help improve human-to-human communication utilizing _speech emotion analysis_. The target group for such applications that could benefit the most includes doctors and autistic adults. The authors collected speech sentiments and speech intonation data using Microsoft's Azure API and their intonation model to analyze the collected data. The system communicates the detected emotions to the users via audio, visual, or haptic feedback in real time. _Text to speech synthesis_ is an interaction method that synthesizes auditory outputs based on texts. [PERSON] et al. [131] combined _speech recognition_, _speech to text synthesis_, and _text to speech synthesis_ methods to support deaf people. The system they presented captures the narrator's speech, converting it into texts, and displays it to the user's AR display. Additionally, the system converts typed text into speech to talk back to the narrator if the user wants to respond.