# Dialogue Enhanced Extended Reality:
## 5 System Implementation  
In this section, the sequence of workflow steps that are involved in the systems implemented for our particular use cases is described in more detail:  
1. The DCL is responsible for gathering the operators' input to communicate with the system. For the proposed use cases, the first one uses only voice input while the second one combines additional devices such as optical readers too. For the voice interaction, microphones with noise cancellation capture the audio signals. These signals are streamed using media streaming libraries such as FFmpeg into an energy-based automata that discards any silent or noisy audio segment. The remaining audio segment is sent to the Input Communication API. 2. The IL has to encode the raw and unstructured data into formats that can be interpreted by machines and ontologies. In the presented use cases, the audio segments are transcribed to text by the Speech-to-Text module. The Language Understanding module encodes transcriptions into a semantic structure that can be easily interpreted by the IML. As it is commonly done for Spoken Dialogue Systems, we use a scheme based on act-slot-value sets for representing  
Figure 5: Operator performing a cabling task. symbolic semantic meaning of the operator input. The semantic representation of the user action is dispatched to the IML for processing. For the optical reader input, the detected wires' label is transferred to the IML to contextualise the interaction. 3. The DKL works both as the persistence layer and as semantic ontology for each system. The persistence layer stores the interaction context of each operator to be kept turn-by-turn, as well as the specific domain knowledge encoded in an ontology. In both use cases, this ontology describes the know-how of the steps to perform in the process and their sequence, the physical devices that interact during the task and their properties (e.g., screwdriver size, or wire's location). In addition, it encodes the physical devices tied to each step of the interaction to handle ambiguous questions, for example, if the operator asks \"Which size?\" in the _Open fingers_ step of the first use case, the ontology relations are used by the system's IML to understand that the operator refers to the size of the tool required to this specific step, the precision screwdriver, and not to the hex key used in the _Decouple gripper_ step. 4. The IML plans and selects the next action to be performed by the system using both contextual information and the semantic concepts received from the IL. First, it retrieves the task information and the planning rules to complete the task from the Domain Knowledge Layer, and then it defines a strategy to reach the user's objectives. In other words, it consists of a set of expert rules \(\mathcal{S}=\{(x,\,c_{t})\longrightarrow(y,c_{t+1})\mid x\in\mathcal{R},c_{t}, c_{t+1}\in\mathcal{C},y\in\mathcal{A}\}\) that evaluate events \(x\) and the interaction context \(c_{t}\) into system actions \(y\) and the also updates the context \(c_{t+1}\) based on the Attributed Probabilistic Finite State Bi-Automata schema as in [33; 43; 46]. This context, which the IML reads from and writes to the DKL, is maintained and updated through the interaction. In our particular tasks, the input \(x\) corresponds to the user semantic representation and the context \(c\), which takes into account the interaction state (e.g., current step), the history of shown AR animations or the selected wire. 5. The RGL translates the IML's output actions into understandable interactions for the users, for example, answering the information in natural language or augmenting the user's surroundings with visuals. For our particular use case, the system uses the Text-to-Speech module to generate synthesised audio when speech modality is required, using the module described in AhoTTS [47]. Additionally, this layer computes suitable visual properties for the action, which include animation selection, as well as feedback duration. 6.