dataesr/scientific-paragraphs-categorization huggingface dataset:
Moving towards the development of aircraft maintenance metaverse, in this work, we built aircraft maintenance education using mixed reality content in Microsoft HoloLens 2 smart glasses enhanced with a proposed speech interaction module that helps to orient inside the virtual Boeing 737 maintenance world (see Figure  1 ). In Figure  1 , the trainee wears a HoloLens 2 and performs the education process of landing gear removal, whereas another student using Spectator View (a special application that allows the visualization of virtual content of reference HoloLens)  [27]  on a tablet can interact in the virtual space with MR content of his fellow engineer. As can be seen, the physical world combined with virtual 3D assets of aircraft have enormous potential for educational purposes when shared among trainees.
Speech interaction creates a tremendous possibility to imitate an expert in the field of aircraft maintenance that guides trainees during the education process and facilitates tutoring by providing required materials such as technical manuals, instructions, and reference videos that support maintenance education. We addressed the above-mentioned limitations of voice interaction in smart glasses and created a custom voice interaction module, which is flexible enough to recognize the wide variation of speech commands for operation and supports intermixed requests in English and Korean languages, recognizing the corresponding language of command (see Figure  2 ). First, we collected a dataset of user commands in English and Korean languages. Next, we created a deep learning-based model that predicts the requested command and the language in which the command was spoken.
The network has customized architecture that uses CNN for audio features learning, and two output classification parts for commands and language identification, respectively. The module records speech commands by the user and extracts MFCC features for inference. Based on the speech interaction module result (see Figure  2 ), the MR application performs a particular action, and voice feedback is given back to a user in the predicted language (i. e. , if a person makes a speech request in Korean, the response from the application will be in Korean, accordingly). Finally, we embedded the model into the HoloLens smart glasses and performed an evaluation process. In addition, combining cutting-edge technologies such as mixed reality and artificial intelligence in the era of Industry 4. 0, we built a speech interaction module powered by a deep learning model that enables trainees "to talk" with mixed reality content and control the workflow of the maintenance process.
The network has customized architecture that uses CNN for audio features learning, and two output classification parts for commands and language identification, respectively. The module records speech commands by the user and extracts MFCC features for inference. Based on the speech interaction module result (see Figure  2 ), the MR application performs a particular action, and voice feedback is given back to a user in the predicted language (i. e. , if a person makes a speech request in Korean, the response from the application will be in Korean, accordingly). Finally, we embedded the model into the HoloLens smart glasses and performed an evaluation process. To evaluate the performance of the speech interaction module, we created several sets of test data, including intermixed commands, and separate English and Korean test sets.
In the following sections, we described the importance and breakthrough of mixed reality in the field of maintenance and education. Next, we discussed deep learning techniques for developing a custom speech interaction module. In Section 3, we presented the proposed mixed reality environment for aircraft maintenance education and training. Section 4 and Section 5 presented step-by-step explanations on the dataset collection, audio features extraction, and the network architecture of the proposed module. Lastly, we demonstrated evaluation techniques and experimental results to make a conclusion.
In the following sections, we described the importance and breakthrough of mixed reality in the field of maintenance and education. Next, we discussed deep learning techniques for developing a custom speech interaction module. In Section 3, we presented the proposed mixed reality environment for aircraft maintenance education and training. Sections 4 and 5 presented step-by-step explanations on the dataset collection, audio features extraction, and the network architecture of the proposed module. Lastly, we demonstrated evaluation techniques and experimental results to make a conclusion.
First, system-built voice interaction is based on static pre-defined commands. When certain voice commands are set to invoke a particular action, a person wearing the smart glasses must speak out the command in exactly the way it was programmed. For example, if "next instruction" is defined in the system, then "alright, show me the next instruction" is not going to work. Users must either remember all the commands exactly how they are set, which is a contradiction to the user-friendliness paradigm, or developers need to handle this with multiple variations of the same voice command in the system, which is inconvenient to maintain when there are many control functions. Second, built-in speech recognition is not suitable for a domain-specific language. Taking the example of maintenance in the aviation industry, there exist special manuals such as the IPC (Illustrated Parts Catalogue), and when the dictation speech input option is invoked in the HoloLens 2 smart glasses, the speech command "show me IPC view" is recognized as "show me my PC view".
Therefore, sound and speech classification works are further analyzed. The recent works in speech and sound classification mostly dealt with simple singleword speech commands (A Dataset for Limited-Vocabulary Speech Recognition  [21] ) or urban sounds (A Dataset and Taxonomy for Urban Sound Research  [32] ). The fundament of their networks' architecture is the convolutional neural network (CNN); however, the dimensions of the convolutional block and type of audio feature are divided into two approaches. First,  [13, 14, 16, 20, 23]  used raw audio directly, without extracting features. Since audio is a one-dimensional signal, their CNN architecture is also based on one-dimensional filters. Second,  [13, 15, 17, 18, 24, 25, 31]  applied various audio features such as mel frequency cepstral coefficients (MFCCs)  [26]  or Spectrograms with 2D convolutions.
Since these features have a two-dimensional representation, CNN filters have the corresponding shape. Even though both approaches perform nearly the same, the analyses in  [13, 26, 33]  show evidence that MFCCs are great at the approximation to the human voice, which helps to identify linguistic content and discard background noise or certain emotions present in the audio. Therefore, in our work, we decided to follow the second approach-using 2D CNN-based architecture with MFCC as audio features. The referenced works in speech command recognition demonstrated the ability to easily handle simple single-word classification; however, the following limitations are present: 1. They do not deal with complex commands classification, where multiple variations of request belong to one class; 2. The datasets they experiment with and trained networks are monolingual.
Dataset for Speech Interaction Module Dataset creation and collection for our task was performed in three stages (see Figure  5 ). First, we analyzed the semantic structure of user commands in English and Korean and created a list of potential user requests. Second, based on the list, we collected audio recordings of these speech commands using Text-to-speech (TTS) application programming interface (API) services and asked several people to speak them out. Third, in order to bridge the gap between API and human recordings, we applied various audio augmentation techniques to the TTS-generated data. teraction module that uses a deep learning model that maps spoken commands to the defined command types and at the same time identifies the language of request to proceed with corresponding feedback.
Dataset for Speech Interaction Module Dataset creation and collection for our task was performed in three stages (see Figure  5 ). First, we analyzed the semantic structure of user commands in English and Korean and created a list of potential user requests. Second, based on the list, we collected audio recordings of these speech commands using Text-to-speech (TTS) application programming interface (API) services and asked several people to speak them out. Third, in order to bridge the gap between API and human recordings, we applied various audio augmentation techniques to the TTS-generated data.
Speech Commands To develop a robust and flexible voice interaction module, it is essential to consider the nature of user requests. Therefore, we analyzed variations in speech commands and semantics for each command type (see Section 3. 2). Different users use various combinations to address the same functionality. For example, "next instruction", "display the next instruction", "can you show me the next task" map to the same command-"next instruction". Moreover, end-users of the application use English and Korean languages to have a voice interaction with the MR application. Taking into consideration all these factors and goals, we made a list of potential user speech requests based on command type and language. Figure  6  summarizes the distribution of created speech commands. Dataset creation and collection for our task was performed in three stages (see  Figu 5).
Audio Data The audio dataset collection is based on created speech commands. We used two methods to record speech audio: using existing text-to-speech (TTS) API services to create audio recordings, and employing people to record spoken commands. For most of the audio collection, we applied API services since this is the cheapest approach. The ones we used were Google Cloud text-to-Speech  [36] , Microsoft Azure text-to-speech  [37] , Naver speech synthesis  [38] , and Kakao speech API  [39]. In total, we applied 83 voices-60 English and 23 Korean. Next, we involved people to record spoken commands: 15 volunteers for English language commands and seven participants for Korean. Overall, for English commands, we have 60 API, and 15 real people voices, which makes 40,575 (75 × 541) recordings, and for Korean ones, 23 API, and seven real voices, which is 18,690 (30 × 623).
Therefore, overall, we collected 59,265 audio recordings as a dataset for all types of commands. Those recordings have a duration of about one to four seconds, which makes the dataset about 24. 7 hours long. The voices of API and volunteers have different accents and originated from countries including the USA, Australia, Canada, India, Ireland, United Kingdom, Korea, Uzbekistan. Thus, the dataset has a wide variety of spoken examples for the same phrases, which makes the speech model adapted to variations. Figure  7  summarizes the amount and distribution of audio data. approach. The ones we used were Google Cloud text-to-Speech  [36] , Microsoft Azure text-to-speech  [37] , Naver speech synthesis  [38] , and Kakao speech API  [39]. In total, we applied 83 voices-60 English and 23 Korean.
Next, we involved people to record spoken commands: 15 volunteers for English language commands and seven participants for Korean. Overall, for English commands, we have 60 API, and 15 real people voices, which makes 40,575 (75 × 541) recordings, and for Korean ones, 23 API, and seven real voices, which is 18,690 (30 × 623). Therefore, overall, we collected 59,265 audio recordings as a dataset for all types of commands. Those recordings have a duration of about one to four seconds, which makes the dataset about 24. 7 hours long. The voices of API and volunteers have different accents and originated from countries including the USA, Australia, Canada, India, Ireland, United Kingdom, Korea, Uzbekistan. Thus, the dataset has a wide variety of spoken examples for the same phrases, which makes the speech model adapted to variations.
e. , white noise, street noise, crowd noise, etc. ) to the API-generated data to make them more realistic. As in  [13] , we added noise with random volume to have more variations. Second, text-to-speech API data has a static playback starting point. In recordings made by humans, the actual point of time when a person begins to speak varies from 0. 5 to 1. 5 s. This makes people's audio recordings diverse. In contrast, in the case of API data, As can be seen from the charts in Figure  7 , API audio data make about 78. 1% of the dataset, where the rest 21. 9% are spoken commands of employed people. The amount of Korean data is two times less than English, but the proportion of API and real data remains similar. However, having speech API-synthesized audio recordings has its own issues.
In the speech command retrieval, the module first accepts a request for a command. Following approaches of voice assistants such as Apple Siri  [41]  ("Hey, Siri") or Google  [42]  ("Hey, Google"), we used our own trigger "Hey, AK" (AK is short for Augmented Knowledge Corp. , the company that supported our work and provided Microsoft HoloLens 2 smart glasses and virtual 3D assets of the Boeing 737). Thus, whenever a user wants to give a command, "Hey, AK" should be spoken, and it is then detected by the keyword recognizer, which gives the notification sound that denotes recording, and next triggers the audio recording using a microphone with a duration of n seconds. In our case, four seconds was sufficient for an average request. The speech command retrieval produces an audio speech command that is processed in the next stage.
API recording to have a diverse beginning time of playback. This approach made API audio recordings sound even more like real ones. The above-mentioned audio augmentation techniques made the real and API distributions closer, and recordings of both types sound similar now. Hence, by finishing the dataset collection stages described in Figure  5 , as a result, we have a speech commands dataset in the form of audio files, each having a sampling rate of 16 kHz and a bit rate of 256 kbit/s. Next, we created and trained our speech model.
Based on the command classification result, the MR application performs a particular action, and voice feedback given in the language predicted by the language classifier (i. e. , if a user makes a speech request in Korean, the response from the application will be in Korean, accordingly). In the case of when a request mixes a few Korean words and technical English terms or any Korean word is pronounced, the model will classify the language as Korean. To train the model on our custom dataset, which consists of complex speech requests in two languages that sound completely different, we extracted audio features as data representation and presented training methods that played a significant role in network convergence. Let us start with audio data representation.
Model Training Training deep learning models requires a proper pick of methods based on the dataset and overall task. In our work, we use data in English and Korean languages with diverse spoken commands. In addition, we want the model to work in a real-world environment, which has various background noises and new human voices with their unique characteristics. Thus, the most essential goal we have is to avoid overfitting. To handle this task, we applied weight decay, which applies regularization for model weights  [45]. This approach improves the generalization of the model and makes it perform well on data that is not present in the dataset. Next, we found that using the cosine annealing scheduler  [46]  for setting a proper learning rate helped a lot for the model to converge  [14, 47].
5), and the Softmax activation function, which gives the final output vector of shape 1 × 8, where 8 is a representation for each command type. Third, the language classification layers, that take the output from the CNN base to predict the language in the input command was spoken. It is similar to the commands classification part; however, it has a lower number of dense layers. The output vector of this part is the predicted language with the shape of 1x2, where 2 is the score for each language (English and Korean). All three parts combined make the architecture of the proposed model, which consists of only 181,066 trainable and 384 non-trainable parameters, respectively. However, building the architecture is only one part of the work; the next thing is to find proper training techniques to make it work with complex characteristics of the dataset.
The general working principle of the cosine annealing scheduler is setting the learning rate starting with a large value that is relatively rapidly decreased to a minimum value before being increased rapidly again  [46]. With each restart, the scheduler tries to start with the weights that contribute to the training most; therefore, this technique is referred to as a "warm restart". Talking about parameters such as loss and optimizer, for each command and language classification part, we applied categorical cross entropy loss, having loss weights to be equally distributed. Moreover, Adam optimizer was applied, having the following hyperparameters: Beta 1-0. 9, Beta 2-0. 999, Epsilon-1e-07. Overall, the mentioned techniques helped the model to converge and learn diverse features of the dataset.
Hardware and Software Environment To set up a deep learning development environment for our work, we used a PC with the following specifications and software: The created model in the h5 extension was exported to ONNX format and imported to the Unity framework  [49]  with Barracuda 1. 12. 1 (cross-platform neural network inference library for Unity  [50] ), where the mixed reality application is built. After exporting to the corresponding format, the model has a 716KB disk memory allocation size. For the development of the mixed reality application that later deployed to HoloLens 2, we used a PC with the following specifications and software: Next, the application with the speech interaction module deployed to Microsoft HoloLens 2 (untethered holographic computer or new generation smart glasses  [10] ) to check the inference time of the model, which was about 40 to 60 milliseconds to process a speech request.
Test Data To perform the evaluation process of the proposed speech interaction model, first, we created the test sets. We randomly picked 2000 audio samples from our dataset. The network was not trained on these audios; however, it might see the other samples of the same voice. We denoted this set as test data. Next, since we have two languages, we decided to assess performance for each one with real people's data, which the network has not seen. Therefore, we applied two sets of spoken commands by humans for English and Korean data. One set consists of all spoken commands by a person. Thus, for English data, we used 541 test audio samples, and 623 for Korean. We denoted these sets as test English speaker and test Korean speaker. Next, we discuss the evaluation metrics that were be used for testing.
Evaluation Metrics In this work, we used accuracy, precision, recall, and F1-Score classification metrics to evaluate the speech recognition model. First, let us define notations: Based on these notations, we explained the evaluation metrics below: • Accuracy: the ratio of accurately classified predictions to the total number of predictions (see Formula (1)). Accuracy = (TP + TN)/(TP + TN + FP + FN) Accuracy is a technique to evaluate the classification model and works best in the case of a balanced dataset  [52]. Since our test data followed the overall distribution of speech commands (see Figure  6 ), which is nearly balanced, it is appropriate to use accuracy.
• F1-score: This is a combination of precision and recall scores, making a weighted average between them. F1-score is described in Formula (4): F1-score = 2 × (Precision × Recall)/(Precision + Recall) (4) F1-score is useful when having an unbalanced test dataset. A high F1 score means that you have low false positives and low false negatives, thus, correctly identifying true classes.
Results Based on the performance of the proposed model experimented on test data, we evaluated our network using classification metrics such as accuracy, precision, recall, and F1-score. Since we have two classification outputs (command and language), we evaluated each separately. Table  1  summarizes the assessment for the commands classification part. As can be seen, performance on test data is near perfect with a 99% success rate in each metric. However, in the case of English and Korean test voices, results are about five percent lower but still great with 94. 5% on average for each metric. Results with test data show better performance because the network might have seen some voice characteristics of test samples (other data of the same voice); however, still, it was not trained on this data.
Conclusions and Future Works To conclude, in this work, we performed a forward step towards the aircraft maintenance metaverse, creating a mixed reality digital place for Boeing 737 maintenance education enhanced with a speech interaction module in the Microsoft HoloLens 2 smart glasses. Using a virtual 3D twin of the aircraft and supportive aviation-specific materials, the proposed MR application provides guidance and supervision during the maintenance process, delivering trainees step-by-step instructions, mixed reality visualizations, reference videos, and aircraft operation manuals to complete MRO tasks. Moreover, the developed speech interaction module, which is based on the deep learning model integrated into the HoloLens 2, enables users to intuitively deal with virtual objects in the aircraft metaverse and seamlessly navigate the maintenance process using speech commands without being distracted from the workflow by pressing buttons or using hand gestures, which is essential when hands are busy with equipment.
To develop the speech interaction model, firstly, based on the defined requirements of the application, we collected a bilingual speech commands dataset in English and Korean languages with domain-specific vocabulary, which included a variety of requests to invoke functions. Secondly, for a core of the speech interaction module, we built and trained a unique deep learning-based CNN network with two outputs that recognizes a referenced command and its language from extracted audio features of the user's speech request. Having a small number of training parameters and low memory size, the model is lightweight and inferences in real-time, which perfectly suits standalone devices such as smart glasses. It was successfully integrated into the Microsoft HoloLens 2, replacing built-in voice interaction.
, 2015). We analyze here this dataset in further detail to determine three approaches related to studying ants. (1) The complementarity and the pertinence of the methods for studying the entire ant assemblage, including the understorey species. (2) The validity of the null hypothesis that all three strata show similar horizontal species turnover based on the best method for studying the ant assemblages in multiple strata. We predicted the following. (a) The turnover will be higher in the arboreal strata (canopy and understorey), representing islands of habitats  (Adams et al. 2017) , than at ground level, a continuous habitat  (Theunis et al. , 2005). (b) The local site conditions, whether abiotic or biotic, likely influence local ant species composition (i. e. , the non-random distribution of species diversity within-sites will be lower than between sites, or distance-decay).