# ATOSE: Audio Tagging with One-Sided Joint Embedding
## 1 Introduction  
Audio auto-tagging automatically assigns metadata, or tags, to audio clips. Basically, these tags denote attributes, such as the sound type, the music genre, or specific instruments, that are designed to provide a substantial summary of the audio content. This process can help in organizing extensive audio file databases and formulating content-driven recommendations.  
In the early stages of audio auto-tagging, traditional machine learning techniques, such as decision trees [1], the K-nearest neighbor algorithm [2], and support vector machines [3], were commonly employed after extracting a wide range of simple features from the audio [4; 5; 6]. Later on, using preprocessing techniques like spectrograms and mel-spectrograms, raw audio data could be transformed into image data and then fed into machine learning models. With the advance of artificial intelligence techniques, most recent work on audio auto-tagging can directly deal with the raw data without preprocessing by leveraging image-based deep learning models, such as convolutional neural networks (CNNs), to extract acoustic features and correlate them with the respective tags.  
However, these recent deep learning models still lack utilization of the _semantic_ information linking tags. For instance, the association between \"classical music\" and \"cello\" is intuitively stronger than that between \"techno\" and \"quiet\". To address this issue, we introduce a joint embedding model that utilizes the semantic information from tags in order to enhance the performance of prior auto-tagging models. Our approach involves two distinct modules: one for extracting features from audio domain data, and another for tag domain data. Then, they are subsequently embedded into a shared (tag) feature space. Given the generalizability of our method, we could incorporate the audio feature extraction components from previously established audio auto-tagging models.  
To summarize, the contributions of our paper are as follows:* We extract features from diverse audio data through asymmetric multi-modal embedding and project these features onto a relatively simple tag domain. Therefore, we are able to maintain various feature vectors of the audio data continuously and also encapsulate the correlation between the tag data and the audio data within the features. In contrast to traditional methods that directly map different modalities of data into one common embedding space, we are able to sufficiently maintain the distinct characteristics of each data modality through our one-sided joint embedding technique;
* Our experiments showed that the proposed model outperforms the previous methods with datasets with higher tag correlations. We drew this result from comparative experiments using two datasets with distinct characteristics: one with a high degree of tag correlation and another with quite independent tags. The experiments showed that relevant internal tag information can improve the final results, whereas independent tags degrade the performance;
* Our method is a general framework so it can be applied to existing auto-tagging models to utilize the tag semantic information without additional training data.  
The rest of this paper is structured as follows: in Section 2, we describe the background and related research work. Section 3 describes the components, loss functions, and training algorithms for the proposed joint embedding model. Section 4 presents the experimental environments and the results. Finally, in Section 5, we conclude this paper and discuss the future research directions.