Almawave-SLU: A New Dataset for SLU in Italian
Authors: Valentina Bellomaria, Giuseppe Castellucci, Andrea Favalli, Raniero Romagnoli (Almawave, 2019)
Topic: Chatbots and Dialogue Systems

Abstract
The widespread use of conversational and question answering systems made it necessary to improve the performances of speaker intent detection and understanding of related semantic slots, i.e., Spoken Language Understanding (SLU). Often, these tasks are approached with supervised learning methods, which need considerable labeled datasets. This paper presents the first Italian dataset for SLU in a voice assistant scenario. It is the product of a semi-automatic procedure and is used as a benchmark for various open-source and commercial systems.

Introduction
Conversational interfaces, such as Google’s Home or Amazon’s Alexa, are becoming pervasive in daily life. As an important part of any conversation, language understanding aims at extracting the meaning a partner is trying to convey. Spoken Language Understanding (SLU) plays a fundamental role in such a scenario. Generally speaking, in SLU, a spoken utterance is first transcribed, and then semantic information is extracted. Language understanding, i.e., extracting a semantic 'frame' from a transcribed user utterance, typically involves: i) Intent Detection (ID) and ii) Slot Filling (SF) (Tur et al., 2010). The former classifies a user utterance into an intent, i.e., the purpose of the user, while the latter identifies the 'arguments' of such intent. As an example, let us consider Figure 1, where the user asks for playing a song (Intent=PlayMusic) (with or without you, Slot=song) of an artist (U2, Slot=artist). Usually, supervised learning methods are adopted for SLU. Their efficacy strongly depends on the availability of labeled data. There are various approaches to the production of labeled data, depending on the intricacy of the problem, the characteristics of the data, and the available resources (e.g., annotators, time, and budget). When the reuse of existing public data is not feasible, manual labeling should be accomplished, eventually by automating part of the labeling process. In this work, we present the first public dataset for the Italian language for SLU. It is generated by a semi-automatic procedure from an existing English dataset annotated with intents and slots. We have translated the sentences into Italian and reported the annotations based on a token span algorithm. Then, the translation, spans, and consistency of the entities in Italian have been manually validated. Finally, the dataset is used as a benchmark for NLU systems. In particular, we will compare a recent state-of-the-art (SOTA) approach (Castellucci et al., 2019) with Rasa (ras, 2019), taken from the open-source world, IBM Watson Assistant (wat, 2019), Google DialogFlow (dia, 2019), and, finally, Microsoft LUIS (msl, 2019), some commercial solutions in use. Following, in Section 2, related works will be discussed; in Section 3, the dataset generation process will be explained; in Section 4, we will present the experiments. Finally, in Section 5, we will draw the conclusions."

Conclusion
The contributions of this work are two-fold: first, we presented and released the first Italian SLU dataset (Almawave-SLU) in the context of voice assistants. It is composed of 7,142 sentences annotated with respect to intents and slots, almost equally distributed across 7 different intents. The effort spent on the construction of this new resource, according to the semi-automatic procedure described, is about 24 FTE (Full-Time Equivalent), with an average production of about 300 examples per day. We consider this effort to be lower than typical efforts required to create linguistic resources from scratch. Second, we compared some of the most popular NLU services with this data. The results show that they all have similar features and performances. However, when compared to another specific architecture for SLU, namely Bert-Joint, they perform worse. This was expected and demonstrates that Almawave-SLU can be a valuable dataset for training and testing SLU systems in the Italian language. In the future, we hope to continuously improve and extend the dataset.