# ATOSE: Audio Tagging with One-Sided Joint Embedding
## 3 Architecture: Audio Auto-Tagging with One-Sided Joint Embedding
### Model Architecture  
First, we describe our tag autoencoder, which extracts tag features. Autoencoders are well-known models that train input data to be the same as output data and are composed of an encoder and a decoder. The encoder receives data and transforms them into a latent vector form that has the characteristics of the data, and this latent vector is then passed to the decoder to approximately restore the original input data. The latent space of this vector is considered as the inherent feature space of the tags. Our **tag autoencoder (TAE)** uses  
Figure 1: ATOSE Architecture. an overcomplete autoencoder to effectively extract additional meaning information that includes the relationships between the tags that are embedded. An overcomplete autoencoder is a type of autoencoder that uses a vector of larger dimensions than the dimensions of the input to be learned, even though most autoencoders generally have smaller latent dimensions than the input dimensions. The reason we chose larger latent dimensions is that it is advantageous if the tag information and the information for the relations between tags are stored independently in separate dimensions. In general, it is difficult to train overcomplete autoencoders since the model may tend to learn identity mapping given the data. To overcome the challenge, we train the autoencoder in conjunction with an audio feature extractor and a projection module, subject to constraints, aiming to ensure that the resultant latent feature vector from the autoencoder aligns closely with the projected tag feature vector derived from the audio data. This can be considered as regularization that makes the training possible while avoiding identical mapping. Additionally, during training, we randomly select positions from a single audio file to generate multiple fixed-length samples. This data augmentation technique effectively trains the autoencoder to learn from different samples with the same tag label. It helps improve the model's generalization and performance. Note that we set the latent vector dimensions of the overcomplete autoencoders as approximately two times larger than the nearest power of two of the input tag dimensions. Also, we apply layer normalization [20] to each layer to avoid extracting unnecessary features. Now, we describe how to extract the features from audio and project them to the tag feature space. The **feature extractor (FE)** is a module responsible for extracting the acoustic features of an audio domain. As the proposed joint embedding method was designed to be applicable to existing models, we utilized previous models when implementing the FE. Specifically, we employed SampleCNN [9; 12; 13], HarmonicCNN [14], and SincNet [15] to predict audio tags by extracting features from the waveforms of the audio. Additional classifier layers were then used to classify the resulting tags. In order to ensure consistency across all models, we standardized the dimensionality of the acoustic features, transforming them into vectors of 1024 dimensions. The **projector (PR)** module is responsible for mapping diverse audio features provided by the FE to a relatively simpler tag latent domain. Unlike traditional joint embedding methods that map significantly different characteristic domains into a common domain, we employ a one-sided joint embedding approach. The approach enables us to map the complex audio domain to the simple tag domain while preserving the distinct characteristics of both domains and capturing additional relational information. The PR consists of two dense layers and a unit normalization layer. The input dimensions are the audio feature dimensions, and the output dimensions are set to match the latent vector dimensions of the TAE. During optimization, the resulting vector of the projection model is adjusted to closely resemble the latent vector of the TAE. A more detailed explanation of this matter is provided in the following section.