# Authenticity at Risk: Key Factors in the Generation and Detection of Audio Deepfakes +
## 4 Methodology
### Automatic Determination of the Acoustic Context  
To acoustically characterize the places where audio notes were recorded, we performed Acoustic Scene Classification (ASC). We used the _PaSST_ (_Patchout faSt Spectrogram Transformer_) [44], a transformer-based model optimized for audio spectrogram processing, which addresses the limitations of traditional transformers in audio applications due to the quadratic growth in memory and computational requirements as the length of the input sequence increases. For this purpose, the authors introduced _Patchout_, a technique that significantly reduces the input sequence during training by randomly removing parts of it, which decreases both memory consumption and computational complexity. In addition, it introduces variants such as _PaSST-U_, where patches are removed in an unstructured manner, and _PaSST-S_, where entire rows or columns of patches are removed, based on approaches such as _SpecAugment_[45].  
The PaSST architecture is depicted in Figure 4 and is extracted from [44], where the input spectrogram is first divided into small patches, to which positional encodings representing time and frequency are added. Then, a patch-out masking strategy is applied, and two special tokens are summed up to the resulting sequence: one for classification (C) and one for distillation (D). Subsequently, the sequence passes through several layers of self-attention and feed-forward networks (FFN), where the relationships between patches are processed. Finally, the processed tokens are combined and prediction is performed with a multi-layer perceptron (MLP).  
PaSST has several applications in audio classification and labeling tasks, among which is the classification of acoustic scenes. In audio tagging in Audioset [46], it outperforms the AST transformer [47] and Convolutional Neural Networks (CNNs), improving the training speed and memory usage due to the patch-out process. In addition, it can be used in polyphonic musical instrument recognition.  
We had previously trained the model for this task on a subset of _WELIVE_ for which we had previous information about the location of the user. Note that there is no overlapping of these training audios with the audio notes mentioned before. The model was fine-tunedusing five-fold stratified cross-validation, resulting in five models trained on different data partitions [36].  
Then, the last layer of the _PaSST Transformer_ was replaced by one with eight outputs and it was fine-tuned during 50 epochs using the stochastic gradient descend optimizer with a Nesterov momentum of 0.9 and a learning rate of 0.0001. Early-stopping was used on the F1-score macro, and the model performance reached the criteria at 30 epochs.  
For each real-life audio note file, inference was performed using all five models. Predictions were considered valid if at least three of the models agreed on the classification. On the other hand, only seven audios were close to the decision boundary, i.e., for which at least three models did not agree with their prediction. These were listened to and re-labeled. The uncertainty was found to be due to audio files belonging to a category not accounted for in the labeling options, _Street_. Finally, audio notes were classified into nine different acoustic environments: _Home, Work, Transport, Medical Center, Street, Bar, Restaurant, School_ and _Cafeteria_. More details about the training of this model are provided in [36].