# ATOSE: Audio Tagging with One-Sided Joint Embedding
## 2 Related Work  
Previously, audio auto-tagging models employed preprocessing techniques, such as spectrograms and mel-spectrograms [7], to process raw audio data before using machine learning or deep learning techniques for tagging. However, recent auto-tagging deep learning models directly handle raw data without preprocessing.  
[PERSON] and [PERSON] [8] initially used raw waveforms as input for CNN models directly and achieved comparable results to the previous models using mel-spectrograms. [PERSON] et al. [9] proposed a CNN-based deep learning model called SampleCNN for audio auto-tagging. They sampled a raw audio frame with small size filters (e.g., two or three) through multiple CNN layers to obtain its frame-level representation. Later, they extended their results with squeeze and excitation [10] techniques and residual connection [11; 12; 13].  
[PERSON] et al. [14] proposed HarmonicCNN using an audio representation module with a set of harmonic filters as a feature extraction module. Note that a harmonic is an integer multiple of the fundamental frequency. By combining harmonic knowledge and data-based learning methods, they built an audio representation module to reduce the dimensions of data waveforms and achieve fast convergence.  
[PERSON] and [PERSON] [15] attempted a novel approach by using frequency filters in the frequency domain instead of the small-sized CNN filters traditionally employed for audio feature extraction. They proposed SincNet, which can learn using bandpass filters from raw audio data waveforms instead of using fixed frequency filters. Note that the sinc function is a function represented by the ratio of the sine function and its variables and has a rectangular shape in the frequency domain. The bandpass filter implemented as a sinc function learns only the cut-off frequency that can pass through the band, regardless of the length. As a result, the authors were able to dramatically decrease the number of parameters that need to be learned.  
All of the above research work mainly used techniques to extract features from audio and predict tags but did not utilize the inherent relationships between the tags. Joint embedding is an advanced technique that can consolidate data from multiple domains or modalities into a coherent representation and is useful in tasks such as cross-modal retrieval and multi-modal learning.  
[PERSON] [16] proposed a framework for learning joint embeddings in tags and acoustic space, which was applied to a cross-modal audio search engine. Instead of using separate embedding networks to extract features from audio and tag data, they trained a common embedding network, known as a Siamese network, with both audio and tag data. Using this network, they improved both text and audio queries for search and retrieval.  
[PERSON] [17; 18] demonstrated a joint embedding scheme using two distinct autoencoders to extract features for sound data and tag data, respectively. In order to ensure that the latent vectors from each autoencoder were as similar as possible, they added a constraint to maximize the similarity between the latent vectors as a regularization term in their loss function.  
[PERSON] et al. [19] proposed the Video-Audio-Text Transformer (VATT) model, which learns multi-modal representations using large-scale unlabeled video data. VATT linearly projects each modality (video, audio, text) into a feature vector and then feeds it into a shared transformer encoder. The authors achieved high performance in video action recognition, acoustic event detection, and image classification as downstream tasks after pretraining the model via self-supervised learning with large-scale video data.  
However, these models may encounter an asymmetry issue when mapping the diverse range of audio data and the relatively limited range of tag data into a common embedding space. To address this problem, we propose an audio auto-tagging model with joint embedding that can learn not only acoustic features from the audio domain but also semantic information from the tag domain.