Title: LEARNING AUDIO REPRESENTATIONS WITH SELF-SUPERVISION

Abstract: We explore self-supervision as a way to learn general purpose audio representations. Specifically, we propose self-supervised tasks that exploit the temporal context in the spectrogram domain. The TemporalGap task estimates the distance between two short audio segments extracted at random from the same audio clip. The Audio2Vec task is inspired by Word2Vec, a popular technique used to learn word embeddings, and aim at reconstructing a spectrogram slice from past and future slices or, alternatively, at reconstructing the context of surrounding slices from the current slice. We evaluate the quality of the embeddings produced by the self-supervised learning models, measuring the accuracy of linear classifiers, which receive the embeddings as input and aim at addressing a variety of downstream audio tasks. Our results show that the learned representations partially bridge the performance gap with fully supervised models of similar size, and for some tasks even approach their performance.

1. INTRODUCTION:

Thanks to advances in supervised audio learning, it is now possible to train models that are able to successfully perform different tasks, including audio annotation (Hershey et al., 2017) , music recognition (Arcas et al., 2017) , automatic speech recognition (Chan et al., 2016) , speaker identification (Matejka et al., 2016) , etc. Despite the indisputable success, this approach suffers from three main shortcomings. First, it requires collecting large annotated datasets specific to each task to be solved. Second, separate models are typically trained for each task, making it difficult to reuse computational resources when multiple such models are deployed on a mobile device. Third, inference is performed on device, but model training is still done on the server side using datasets representing surrogate distributions, which might potentially differ from the true data distribution.

Unsupervised learning attempts to overcome these limitations, by making it possible to learn from widely available unlabelled datasets and by learning general purpose representations that can be reused for different downstream tasks. In addition, unsupervised learning lends itself to be deployed on device, where no explicit labeling of the data is available. Therefore, by leveraging the recent advances in federated learning (Bonawitz et al., 2019) , it might be possible to distribute the training process across numerous devices, thus training models directly on the true data distribution, while fully preserving users' privacy.

In the area of unsupervised learning, self-supervised learning has emerged as an attractive approach. In a nutshell, an auxiliary task is formulated based on the available unlabelled data and a fully supervised model is trained to solve such a task. The key idea is that, by solving the auxiliary task, the model is also learning some general purpose representations in a lower dimensional embedding space. Therefore, the embedding encoder, e.g., the portion of the model architecture mapping the input data to the embedding space, can be reused as a feature extractor for different downstream tasks.

One of the earliest successes of self-supervised learning was obtained in the context of language models, where Word2Vec is used to map one-hot-encoded words to word embeddings (Mikolov et al., 2013) . Word2Vec can be formulated in two variants: i) continuous bag-of-words (CBoW), or ii) skip-gram. In the former, the model predicts the current word based on the context of surrounding words. In the latter, the model predicts surrounding words given the current word. Recently, a similar approach has been proposed to map speech to fixed-dimensional embeddings (Chung & Glass, 2018;  Chung et al., 2018b) . The Speech2Vec architecture consists of a RNN encoder-decoder which can handle variable-length inputs and outputs.

In this paper we explore self-supervised learning of audio representations. We posit that contextual temporal information can be exploited in the case of general audio signals without resorting to any form of explicit supervision. We argue that solving properly designed tasks that involve the temporal context requires extracting some sort of high level semantic information from the underlying raw data, thus leading to reusable embeddings. In this respect, this paper makes the following main contributions:

• We propose Audio2Vec, a self-supervised learning task that is inspired by Word2Vec, but applied to audio spectrograms. In the CBoW formulation (Figure 1a ) the auxiliary task consists of reconstructing a temporal slice of pre-determined duration from a number of past and future slices. In the skip-gram formulation (Figure 1b ) the roles of the target and surrounding slices are reversed.

• We propose TemporalGap, a self-supervised learning task that consists of estimating the distance in time between any two pairs of audio segments extracted at random from a longer audio clip (Figure 1c ).

• We quantitatively evaluate the quality of the embeddings produced by the feature encoders obtained by solving the aforementioned self-supervised tasks. To this end we consider a wide variety of downstream tasks, including speech, music detection, speaker identification and language identification, among others. Our results show that all self-supervised models are able to partially bridge the accuracy gap with fully supervised models. This is true despite the fact that labels are only used to train simple linear classifiers that receive the embeddings as input.

While self-supervised learning has been widely explored in the context of visual and multi-modal representations, its potential to learn audio representations has been largely left unexplored, with the exception of just a few works (further discussed in Section 2). Therefore, our paper contributes to filling this gap, proposing in Section 3 two self-supervised tasks specifically tailored to the temporal nature of audio data. The proposed methods are compared to unsupervised baselines and fully supervised benchmarks in Section 4. Conclusions and future work are given in Section 5.

2. RELATED WORK:

The work presented in this paper is related to several different areas that have received attention in the recent literature. In particular, learning representations has been explored for different modalities.

Learning audio representations: Unsupervised feature learning can lead to more compact and descriptive representations than traditional handcrafted features, e.g., MFCCs. For example, (Lee et al., 2009) adopt convolutional deep belief networks to learn audio representations, applicable to both speech and music related tasks. More recently, different autoencoder architectures have been explored, e.g., denoising (Xu et al., 2017) , convolutional LSTM autoencoders (Meyer et al., 2017) and sequence-to-sequence autoencoders (Chung et al., 2016) . A self-supervised version of the triplet loss is proposed in (Jansen et al., 2018) . In the absence of labels, the authors create anchor-positive pairs by adding noise, shifting in time and/or frequency, and sampling temporal neighbors. When tested on AudioSet (Gemmeke et al., 2017) , the self-supervised embeddings partially bridge the gap between a simple log spectrogram baseline and a fully supervised classifier.

Learning visual representations: Several auxiliary tasks have been explored to learn image representations, e.g., predicting the relative position of a pair of patches extracted from the same image (Doersch et al., 2015) , re-ordering image patches and solving jigsaw puzzles (Noroozi & Favaro, 2016) , or asking the model to discriminate between a patch and transformed version of it (Dosovitskiy et al., 2016) . In some cases solving seemingly simple tasks can lead to very powerful representations such as, for example, detecting image rotations (Gidaris et al., 2018) . In other cases, representations can be learned as a by-product of solving useful tasks, e.g., in the case of image colorization (Zhang et al., 2016) and image inpainting (Pathak et al., 2016) . The latter is to some extent similar to our work, since the CBoW version of Audio2Vec can be seen as a form of inpainting in the spectrogram domain. The representations learned by different self-supervised learning tasks can also be combined to obtain a single representation, as presented in (Doersch & Zisserman, 2017) . In the case of video, it is possible to exploit the temporal dimension to learn visual representations by asking a model to learn whether frames are in the correct temporal order (Misra et al., 2016; Fernando et al., 2017) , to infer motion by observing a static image (Pathak et al., 2017) , or detect whether a video is playing forwards or backwards (Wei et al., 2018) .

Learning multimodal representations: Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text (Chung et al., 2018a) , images (Owens et al., 2018) and videos (Owens & Efros, 2018; Gao et al., 2018; Arandjelović & Zisserman, 2018; Korbar et al., 2018; Cramer et al., 2019) .

Contextual predictions: After the seminal work on Word2Vec, contextual prediction has been successfully explored as a means for learning representations in other modalities, e.g., in the case of image inpainting (Pathak et al., 2016) , symbolic music prediction (Bretan et al., 2017) , and speech (Chung & Glass, 2018) . Recently (van den Oord, Yazhe Li, 2019) proposed to use contrastive predictive coding, i.e., predicting future samples directly in the embedding space, reporting promising results also in the case of audio-based tasks. Our work is mostly related to this strand of research, in that we evaluate contextual prediction for general audio-based tasks, but we put particular emphasis on learning models that can be deployed on device.

3. METHODS:

Let x = {x 1 , x 2 , . . . , x n } denote an audio clip of n samples in the time domain and X ∈ R T ×F the corresponding mel spectrogram, which consists of T temporal frames and F frequency bins. Note that we compute the logarithm of the modulus of the spectrogram to compress the dynamic range of the amplitudes. Let X i denote a N × F slice of the spectrogram X, starting at frame i with N < T temporal frames and z i = Enc(X i ) a d-dimensional embedding computed by processing the input spectrogram X i with an encoder Enc(), whose architecture is detailed in Section 4. Using this notation, in the following we describe the proposed self-supervised learning models.

Audio2Vec (CBoW)::

The first self-supervised learning task that we propose, Audio2Vec, comes in two variants. In the CBoW variant, we first select a target slice at random, together with a set of surrounding slices used for prediction. Each of the predictor slices is processed by the same encoder, which maps its input into a fixed-dimensional embedding. These embeddings are then concatenated and fed into a decoder, mimicking the same architecture as the encoder, which computes a reconstruction of the target slice. More specifically, let X (0) = X i be a slice selected at random from X. Then, a set of past (X (-P ) , . . . , X (-1) ) and future slices (X (1) , . . . , X (P ) ) are extracted from the same audio clip. The temporal location of the slice X (p) is equal to X i+p(N +G) , i.e., we consider nonoverlapping slices of size N , with an extra gap of G temporal frames between any two consecutive slices. The gap is introduced to avoid that the self-supervised model exploits the leakage between adjacent STFT temporal frames as a shortcut to solve the task. Each slice is processed by the same encoder to obtain z (p) = Enc(X (p) ). Then, a vector z (0) = [z (-P ) , . . . , z (-1) , z (1) , . . . , z (P ) ] is obtained by concatenating the embeddings of each of the predictor slices and fed into a convolutional decoder to obtain a reconstruction X(0) = Dec(z (0) ). Note that the architecture of the decoder is obtained by reversing the order of the layers in the encoder and replacing max-pooling with nearest-neighbor upsampling. The overall encoder-decoder architecture is trained end-to-end by minimizing the mean-square error loss function X (0) -X(0) .

Audio2Vec (skip-gram)::

The skip-gram variant of Audio2Vec uses a similar architecture. In this case we compute the embeddings of the middle slice z (0) = Enc(X (0) ), and then let the decoder reconstruct the surrounding slices, i.e., [ X(-P ) , . . . , X(-1) , X(1) , . . . , X(P ) ] = Dec(z (0) ). The decoder is identical to the one used by the CBoW variant, except for one important difference: the last convolutional layer has 2P output channels, one for each of the slices to be reconstructed. The loss function minimizes the average mean-square error computed across the 2P reconstructed slices.

Temporal gap: For the TemporalGap task, we ask the model to estimate the absolute value of the distance in time between two slices sampled at random from the same audio clip. More specifically, we sample the ground truth temporal gap from a uniform distribution, i.e., ∆ ∼ U(0, N max -N ), where N and N max are the lengths (in time frames) of the slices and the original sample, respectively, and define the normalized temporal gap as δ = ∆/(N max -N ) ∈ [0, 1]. Then, we extract two slices X i and X j such that ∆ = |i -j|. Note that we do not impose a temporal order between the two slices. We concatenate the embedding representations in a single 2d-dimensional vector z = [Enc(X i ), Enc(X j )] and we feed this vector into a fully connected feed forward network with a single hidden layer of size 64 that produces the scalar output δ. We train the model end-to-end so as to minimize a cross-entropy loss L CE (δ, δ) between the ground-truth and the predicted gap. In our experiments, we found that this loss is to be preferred to the mean-square error δ -δ , presumably because it gives more weight to errors when the ground truth gap δ is small.

4. EXPERIMENTS:

To evaluate the quality of the embeddings produced by different self-supervised learning methods, it is interesting to observe how the learned representations transfer to several, potentially heterogeneous, downstream tasks. Therefore, for each downstream task, we train a simple fully supervised head.

The head consists of a linear classifier that receives the embeddings as input, and produces as output one of the task classes. This is compared with a fully supervised model of similar size, in which the parameters of both the encoder and the head can be trained having access to labelled data. In the following we describe the datasets used in our evaluation campaign and the baselines to which we compare our results.

Datasets: We use AudioSet (Gemmeke et al., 2017) to train all the self-supervised learning tasks, unless stated otherwise. AudioSet contains excerpts of 10 seconds from the soundtracks of YouTube videos. Although the dataset is annotated with labels of more than 500 classes, we discard them in our study. Note that each AudioSet sample can be potentially reused multiple times during training, each time extracting a different target slice (together with surrounding slices) uniformly at random.

We use six publicly available datasets to evaluate a variety of downstream tasks, covering both speech and non-speech related tasks. We use the Speech Commands dataset (Warden, 2018) to evaluate keyword spotting on 35 distinct keywords. LibriSpeech (Panayotov et al., 2015) contains audio books read by 251 different speakers. We use the 100 hours training set to evaluate a speaker identification task. The Spoken Language Identification dataset (Oponowicz, 2018) contains samples that belong to three different languages: English, Spanish and German, while the MUSAN dataset (Snyder et al., 2015) distinguishes across three classes, namely music, speech and noise. Finally, we use two datasets released in the context of the recent DCASE2018 Challenge, Bird Audio Detection (Stowell et al., 2018) and TUT Urban Acoustic Scenes 2018 (Mesaros et al., 2018) , which contains labeled audio samples from 10 different urban environments. To the best of the authors' knowledge, this is the first paper that comprehensively evaluates the quality of self-supervised learning for audio on such a wide variety of tasks. Note that we deliberately avoid exploiting during training any of the datasets used for evaluating the downstream tasks.

Since each dataset is characterized by samples having different durations, during training we preprocess the downstream datasets extracting equal-length slices uniformly at random from the original sample and assign the corresponding label to all of the extracted slices. We consider input samples having the duration of T = 975 ms, so as to match the size of the temporal slices used when training the self-supervised tasks. During evaluation, we apply a sliding window of size T and a hop size of T /2, so as to one or more predictions for each input samples, depending on its length. In order to aggregate such predictions and produce a single output for each sample, we apply a simple naive-Bayes classifier. Note that the choice of the tasks used for the evaluation is consistent with the selected temporal granularity. As such, we do not consider automatic speech recognition tasks, which generally require processing much shorter temporal slices.

Encoder architecture: In our work we consistently use the same audio frontend, which processes input sequences sampled at 16 kHz, with a window size of 25 ms and a hop size equal to 10 ms to compute the short-time Fourier transform (STFT), and then computes F = 64 mel-spaced frequency bins in the range 60-7800 Hz. For the encoder Enc(), we use a convolutional neural network, whose architecture is described in Table 1 . We also repeated the experiments using a larger architecture, which is described together with the corresponding results at the end of this section. Each convolutional layer consists of a series of two convolutions, one along the time axis (with size 3 × 1 × C in × C out ) and one along the frequency axis (with size 1 × 3 × C in × C out ), in parallel to a pointwise 1 × 1 convolution as a residual connection. All activation functions are ReLUs and batch normalization is used in all convolutional layers. Each layer is followed by max-pooling, to reduce the time-frequency dimensions by a factor of two at each layer. Finally, a global max-pooling layer produces a d-dimensional vector, which is further processed by a fully-connected layer to get the embeddings. By default, we set N = 96 (corresponding to 975 ms) and d = 128, thus reducing the dimensionality of the raw audio samples by a factor of about 122.

Models::

We compare different self-supervised models trained on AudioSet: Audio2Vec, in its two variants, CBoW and skip-gram, and TemporalGap. For Audio2Vec we use P = 2 slices on each side of the target, and a gap of G = 2 temporal frames between consecutive slices. We also include the TripletLoss methods proposed in (Jansen et al., 2018) in our evaluations. More specifically, positive/negative pairs are obtained by extracting a slice from, respectively, the same or a different original sample. In addition, we also train an AutoEncoder sharing the same encoder and decoder architectures as Audio2Vec. We tested different variants, including denoising and variational autoencoders, but we did not observe significant differences with respect to the default autoencoder. When evaluating the accuracy in downstream tasks, we extract the portion of the model corresponding to the encoder and use it to map input log-mel spectrograms to 128-dimensional embeddings.

We compare our results to two different fully supervised baselines based on a simple linear classifier model: i) the Spectrogram model receives directly the (flattened) spectrogram features as input; ii) the Untrained model computes the embeddings with the very same encoder architecture described in Section 3, but using randomly initialized weights. Supervised 0.75 ± .01 0.90 ± .03 1.00 ± .00 0.99 ± .00 0.97 ± .01 0.79 ± .01

(+100%) (+100%) (+100%) (+100%) (+100%) (+100%)

Since each task is charaterized by a different number of target classes and intrinsic difficulty, we compare the accuracy to the level attained by task-specific fully supervised models (Supervised), each using the same encoder, but trained end-to-end on each of the labeled downstream datasets. In addition, we also trained a MultiHead model, where a single shared encoder is composed with a different fully connected layer for each downstream task. This provides an upper bound for the best performance we could expect, as it uses the same architecture as when using the self-supervised embeddings, but leverages the in-domain labeled data for end-to-end training.

All models are trained with stochastic gradient descent and Adam optimizer with default hyperparameters. The learning rate was set to 10 -3 for Audio2Vec, AutoEncoder, and all the supervised models, while it was set to 10 -4 for TemporalGap and TripletLoss . We use a mini-batch size equal to 256 and we stop training after approximately 2 days (on five Tesla V100 GPUs), thus iterating between 1.3 and 1.8 million mini-batches. In most cases, the accuracy of downstream tasks saturated after iterating over 500k mini-batches.

Main results: In our results we report the prediction accuracy on the eval set of each of the six datasets, averaging the values observed by repeating the training of the head five times. During training we monitor both the loss of the self-supervised task as well as the accuracy on each of the downstream tasks. As an illustration, Figure 2a shows that the accuracy of the MUSAN downstream task increases as the reconstruction loss of Audio2Vec (skip-gram) decreases, and both tend to saturate after approximately 300k iterations. For the same dataset, Figure 2b shows that all self-supervised methods attain a level of accuracy that is in-between the baselines and the fully supervised benchmarks, with Audio2Vec (skip-gram) outperforming the other models on this task. We repeated the evaluation on all downstream tasks and show the results in Table 2 . We report the level of accuracy, with 95% confidence intervals capturing the uncertainty due to the finite size of the evaluation datasets. In brackets we also report the accuracy normalized between 0% (Spectrogram) and 100% (Supervised). We observe that the proposed self-supervised learning models are able to recover between 11% and 98% of the accuracy of the Supervised model. Generally, Audio2Vec (skip-gram) and TripletLoss seem to outperform other selfsupervised models. The best results are obtained on MUSAN and LibriSpeech, presumably because these tasks require to capture relatively stationary spectral characteristics of the inputs. Conversely, all self-supervised models achieve relatively poor performance on the Speech Commands dataset. This might be explained by the fact that for this dataset it is particularly important to recognize the non-stationary variation of the spectral features along the temporal dimension, which does not seem to be captured by the embeddings generated by the self-supervised models. Note that the different self-supervised models might be capturing different characteristics of the underlying audio data. Therefore, there might be the possibility of merging the different representations, as recently proposed in (Pascual et al., 2019) for the case of speech embeddings.

Impact of training dataset: All the results reported so far use the AudioSet dataset to train the self-supervised models. AudioSet contains a wide variety of audio clips, including music, speech, ambient noise, acoustic events, etc. In order to evaluate the impact of the choice of the dataset, we repeated self-supervised training using LibriSpeech (discarding the speaker labels). We chose LibriSpeech because the original samples are sufficiently long to support our self-learning tasks and because it contains audio of different content than AudioSet (i.e., speech only). Table 3 reports how the evaluation results shown in Table 2 change when training the self-supervised models on LibriSpeech instead of AudioSet. In most cases, we observe a decrease in the level of accuracy on downstream tasks, especially for TemporalGap and TripletLoss, suggesting that a richer content variety in the training set is preferable when learning general-purpose audio representations.

Encoder fine-tuning: So far we considered the case in which the encoder is shared completely across different tasks, and only the last layer is allowed to learn task-specific parameters. It is interesting to observe what happens when we relax this assumption, allowing to retrain one (or more) of the deepest layers of the encoder. Figure 3 shows the trade-off between the level of accuracy and the number of task specific parameters for two datasets, Speech Commands and TUT Urban Acoustic Scenes 2018, for which Audio2Vec (skip-gram) was able to only partially bridge the accuracy gap with respect to the Supervised model. The left-most (blue) point corresponds to the accuracy already reported in Table 2 . Note that in this case the number of task-specific parameters is equal to 128 × C, where C is the number of classes (equal to, respectively, 35 and 10 for these datasets). The second (orange) point from the left corresponds to retraining the fully-connected layer, while the remaining points correspond to retraining the until the fifth and fourth convolutional layers included. Generally, retraining the last two layers is needed to recover most of the accuracy of the fully supervised model. Note that, although the last two layers account for approximately 80% of the parameters, they only contribute to 20% of the FLOPs, and this could be particularly useful when deploying on mobile devices.

Impact of encoder architecture size: We repeated our evaluation by increasing the size of the encoder architecture described in Table 1 . Namely, we increased the number of channels in each convolutional layer by a factor of 4, and we increased the number of outputs in the last fully connected layer to obtain 256-dimensional embeddings. Table 4 shows that the accuracy on downstream tasks increases, and Audio2Vec (skip-gram) achieves the highest accuracy on almost all datasets.

Beyond linear separability: All the results presented so far are obtained by training a linear classifier on top of the encoder that produces the embeddings. We repeated our experiments training a more complex head, which includes an additional fully connected layer (followed by a ReLU), to see if downstream classes can be better disentangled using a non-linear model. The results are presented in Table 5 . Generally, a higher level of accuracy is achieved in this case (with one exception, which could be explained with overfitting to the relatively small dataset). Moreover, it is interesting to observe that the AutoEncoder model benefits more than the other self-supervised models.

5. CONCLUSION:

In this paper we present self-supervised learning methods that exploit the temporal context in audio clips. Our results show that both Audio2Vec and TemporalGap are able to produce representations that can be re-used for different downstream tasks, without having access to labelled datasets during training. In our future work we will investigate training self-supervised models directly on device in a distributed fashion, by taking advantage of federated learning. Another interesting direction is merging representations learned by different self-supervised models, as recently proposed in (Pascual et al., 2019) for the case of speech embeddings.

:



Figures and Tables:

Figure 1: Overview of the proposed self-supervised learning tasks.

Figure 2: (a): Training loss for the Audio2Vec (skip-gram) model and corresponding accuracy on the MUSAN downstream dataset. (b) Accuracy on MUSAN datasets for all models under evaluation.

Figure 3: Accuracy obtained when retraining the last layers of the Audio2Vec (skip-gram) encoder.

Encoder architecture. Size of activations, number of parameters and FLOPs.

Table content: <table><tr><td/><td/><td/><td colspan="2">Output Size</td><td colspan="2">Num. params FLOPs</td></tr><tr><td/><td/><td>Input layer</td><td colspan="2">96 × 64 × 1</td><td>-</td><td>-</td></tr><tr><td/><td/><td colspan="3">Conv. layer 1 48 × 32 × 8</td><td>0.2k</td><td>2.9M</td></tr><tr><td/><td/><td colspan="3">Conv. layer 2 24 × 16 × 16</td><td>1k</td><td>4M</td></tr><tr><td/><td/><td colspan="3">Conv. layer 3 12 × 8 × 32</td><td>5k</td><td>4M</td></tr><tr><td/><td/><td>Conv. layer 4</td><td/><td>6 × 4 × 64</td><td>20k</td><td>3.9M</td></tr><tr><td/><td/><td colspan="3">Conv. layer 5 3 × 2 × 128</td><td>82k</td><td>3.9M</td></tr><tr><td/><td/><td>FC layer</td><td colspan="2">1 × 1 × 128</td><td>16k</td><td>33k</td></tr><tr><td/><td/><td>Total</td><td/><td>-</td><td>125k</td><td>18.7M</td></tr><tr><td>Accuracy</td><td>0.75 0.80 0.85 0.90 0.95 1.00</td><td colspan="2">0.2 0.4 0.6 0.8 1.0 Loss</td><td colspan="2">TripletLoss AutoEncoder Audio2Vec (CBoW) Audio2Vec (skip-gram) TemporalGap</td><td>MUSAN</td><td>Spectrogram Untrained MultiHead Supervised</td></tr><tr><td/><td>0k 0.70</td><td>100k 200k 300k 400k 500k Number of batches 0.0</td><td/><td/><td colspan="2">0.70 0.75 0.80 0.85 0.90 0.95 1.00 Accuracy</td></tr><tr><td/><td/><td>(a)</td><td/><td/><td/><td>(b)</td></tr></table>

Accuracy on downstream tasks (and fraction of accuracy recovered wrt. baselines). Downstream tasks: SPC: (Speech Commands), LSP: (LibriSpeech), TUT: TUT Urban Acoustic Scenes 2018, MUS: MUSAN, BSD: Bird Audio Detection, LID: Spoken Language Identification. In bold the highest accuracy attained by self-supervised models for each task.TripletLoss 0.18 ± .01 0.62 ± .04 1.00 ± .00 0.97 ± .00 0.73 ± .03 0.73 ± .01 MultiHead 0.72 ± .01 0.82 ± .03 1.00 ± .00 0.98 ± .00 0.94 ± 0.78 ± .01

Table content: <table><tr><td>model</td><td>SPC</td><td>LID</td><td>LSP</td><td>MUS</td><td>TUT</td><td>BSD</td></tr><tr><td colspan="7">Spectrogram 0.16 ± .01 0.28 ± .04 0.97 ± .01 0.74 ± .01 0.36 ± .03 0.65 ± .02</td></tr><tr><td/><td>(+0%)</td><td>(+0%)</td><td>(+0%)</td><td>(+0%)</td><td>(+0%)</td><td>(+0%)</td></tr><tr><td colspan="7">Untrained 0.16 ± .01 0.48 ± .04 0.54 ± .02 0.93 ± .00 0.57 ± .03 0.70 ± .02</td></tr><tr><td/><td>(-1%)</td><td>(+33%)</td><td>(-1338%)</td><td>(+77%)</td><td>(+35%)</td><td>(+31%)</td></tr><tr><td colspan="7">AutoEncoder 0.28 ± .01 0.64 ± .04 0.99 ± .00 0.94 ± .00 0.59 ± .03 0.69 ± .02</td></tr><tr><td/><td>(+21%)</td><td>(+56%)</td><td>(+55%)</td><td>(+81%)</td><td>(+38%)</td><td>(+27%)</td></tr><tr><td colspan="7">A2V(CBoW) 0.30 ± .01 0.57 ± .04 0.99 ± .00 0.98 ± .00 0.66 ± .03 0.71 ± .01</td></tr><tr><td/><td>(+23%)</td><td>(+47%)</td><td>(+82%)</td><td>(+97%)</td><td>(+50%)</td><td>(+40%)</td></tr><tr><td colspan="7">A2V(SG) 0.28 ± .01 0.55 ± .04 1.00 ± .00 0.98 ± .00 0.67 ± .03 0.69 ± .02</td></tr><tr><td/><td>(+21%)</td><td>(+44%)</td><td>(+85%)</td><td>(+98%)</td><td>(+52%)</td><td>(+28%)</td></tr><tr><td colspan="7">TemporalGap 0.23 ± .01 0.45 ± .04 0.97 ± .01 0.97 ± .00 0.63 ± .03 0.71 ± .01</td></tr><tr><td/><td>(+12%)</td><td>(+27%)</td><td>(+11%)</td><td>(+92%)</td><td>(+44%)</td><td>(+44%)</td></tr><tr><td/><td>(+3%)</td><td>(+55%)</td><td>(+96%)</td><td>(+95%)</td><td>(+61%)</td><td>(+55%)</td></tr><tr><td/><td>(+95%)</td><td>(+88%)</td><td>(+99%)</td><td>(+95%)</td><td>(+96%)</td><td>(+90%)</td></tr></table>

Accuracy obtained when training self-supervised models on LibriSpeech (and the relative difference with respect to training on AudioSet). Red indicates a decrease in the level of accuracy.

Table content: <table><tr><td/><td>model</td><td>SPC</td><td>LID</td><td/><td>LSP</td><td>MUS</td><td>TUT</td><td>BSD</td></tr><tr><td/><td>AutoEncoder</td><td>0.27</td><td>0.65</td><td/><td>0.96</td><td>0.87</td><td>0.56</td><td>0.67</td></tr><tr><td/><td/><td>(-3%)</td><td>(+1%)</td><td colspan="2">(-3%)</td><td>(-7%)</td><td>(-5%)</td><td>(-2%)</td></tr><tr><td/><td>A2V(CBoW)</td><td>0.26</td><td>0.55</td><td/><td>0.99</td><td>0.96</td><td>0.65</td><td>0.70</td></tr><tr><td/><td/><td>(-13%)</td><td>(-3%)</td><td colspan="2">(+0%)</td><td>(-2%)</td><td>(-1%)</td><td>(-1%)</td></tr><tr><td/><td>A2V(SG)</td><td>0.23</td><td>0.65</td><td/><td>0.99</td><td>0.97</td><td>0.66</td><td>0.71</td></tr><tr><td/><td/><td>(-17%)</td><td>(+18%)</td><td colspan="2">(-1%)</td><td>(-1%)</td><td>(-1%)</td><td>(+2%)</td></tr><tr><td/><td>TemporalGap</td><td>0.18</td><td>0.55</td><td/><td>0.93</td><td>0.94</td><td>0.59</td><td>0.64</td></tr><tr><td/><td/><td>(-21%)</td><td>(+22%)</td><td colspan="2">(-4%)</td><td>(-3%)</td><td>(-6%)</td><td>(-9%)</td></tr><tr><td/><td>TripletLoss</td><td>0.10</td><td>0.34</td><td/><td>1.00</td><td>0.93</td><td>0.56</td><td>0.65</td></tr><tr><td/><td/><td>(-44%)</td><td>(-45%)</td><td colspan="2">(+0%)</td><td>(-4%)</td><td>(-23%)</td><td>(-10%)</td></tr><tr><td>Accuracy</td><td>0.2 0.4 0.6 0.8 1.0</td><td/><td>simple_head retrain_last_1 retrain_last_2 retrain_last_3 Spectrogram Untrained MultiHead Supervised</td><td>Accuracy</td><td>0.2 0.4 0.6 0.8 1.0</td><td/><td>simple_head retrain_last_1 retrain_last_2 retrain_last_3 Spectrogram Untrained MultiHead Supervised</td></tr><tr><td/><td colspan="2">0k 20k 40k 60k 80k 100k 120k 140k 160k Number of task-specific params 0.0</td><td/><td/><td colspan="3">Number of task-specific params 0k 20k 40k 60k 80k 100k 120k 140k 160k 0.0</td></tr><tr><td/><td colspan="2">(a) Speech Commands</td><td/><td/><td colspan="3">(b) TUT Urban Acoustic Scenes 2018</td></tr></table>

Accuracy obtained when using a larger encoder architecture (relative change wrt. Table2).

Table content: <table><tr><td>model</td><td>SPC</td><td>LID</td><td>LSP</td><td>MUS</td><td>TUT</td><td>BSD</td></tr><tr><td>AutoEncoder</td><td>0.35</td><td>0.62</td><td>1.00</td><td>0.96</td><td>0.65</td><td>0.70</td></tr><tr><td/><td>(+24%)</td><td>(-3%)</td><td>(+1%)</td><td>(+2%)</td><td>(+10%)</td><td>(+1%)</td></tr><tr><td>A2V(SG)</td><td>0.46</td><td>0.81</td><td>1.00</td><td>0.99</td><td>0.78</td><td>0.76</td></tr><tr><td/><td>(+64%)</td><td>(+47%)</td><td>(+0%)</td><td>(+1%)</td><td>(+16%)</td><td>(+10%)</td></tr><tr><td>TemporalGap</td><td>0.37</td><td>0.77</td><td>1.00</td><td>0.98</td><td>0.73</td><td>0.74</td></tr><tr><td/><td>(+60%)</td><td>(+71%)</td><td>(+3%)</td><td>(+1%)</td><td>(+15%)</td><td>(+4%)</td></tr><tr><td>TripletLoss</td><td>0.30</td><td>0.73</td><td>1.00</td><td>0.99</td><td>0.81</td><td>0.76</td></tr><tr><td/><td>(+66%)</td><td>(+17%)</td><td>(+0%)</td><td>(+2%)</td><td>(+10%)</td><td>(+4%)</td></tr></table>

Accuracy using a head with two fully connected layers (relative change wrt. Table2).

Table content: <table><tr><td>model</td><td>SPC</td><td>LID</td><td>LSP</td><td>MUS</td><td>TUT</td><td>BSD</td></tr><tr><td>AutoEncoder</td><td>0.46</td><td>0.85</td><td>1.00</td><td>0.98</td><td>0.83</td><td>0.76</td></tr><tr><td/><td>(+58%)</td><td>(+32%)</td><td>(+2%)</td><td>(+4%)</td><td>(+45%)</td><td>(+11%)</td></tr><tr><td>A2V(SG)</td><td>0.33</td><td>0.60</td><td>1.00</td><td>0.99</td><td>0.89</td><td>0.76</td></tr><tr><td/><td>(+13%)</td><td>(+9%)</td><td>(+1%)</td><td>(+1%)</td><td>(+30%)</td><td>(+7%)</td></tr><tr><td>TemporalGap</td><td>0.28</td><td>0.60</td><td>0.99</td><td>0.99</td><td>0.83</td><td>0.76</td></tr><tr><td/><td>(+16%)</td><td>(+1%)</td><td>(+2%)</td><td>(+2%)</td><td>(+31%)</td><td>(+7%)</td></tr><tr><td>TripletLoss</td><td>0.22</td><td>0.37</td><td>1.00</td><td>0.99</td><td>0.92</td><td>0.76</td></tr><tr><td/><td>(+9%)</td><td>(-44%)</td><td>(+0%)</td><td>(+2%)</td><td>(+27%)</td><td>(+5%)</td></tr></table>

Peer review (iclr2020):
This paper presents two self-supervised algorithms for representation learning from audio, with artificially created supervised tasks. In particular, audio2vec borrows the ideas of CBoW and SkipGram from word2vec, and TemporalGap predicts the temporal distance between two segments. There is an ongoing theme of self-supervised learning in the machine learning community, with the goal of learning general features from large amount of unsupervised data with auxiliary tasks that may correlate with downstream tasks, so as to reduce the sample complexity for downstream supervised learning. The topic of this paper is quite relevant to the community. The intuitions behind the algorithms do seem plausible to me; after all, word2vec was very successful for NLP. My concerns of this submission are on the execution of the ideas, and the empirical analysis. 1. While the intuition behind representation learning for audio is to capture the temporal structure/contextual information, the proposed algorithms are not demonstrated on downstream tasks that require such information the most, e.g., ASR. The authors argued that *Note that the choice of the tasks used for the evaluation is consistent with the selected temporal granularity. As such, we do not consider automatic speech recognition tasks, which generally require process much shorter temporal slices.* But this argument is not convincing, as the large receptive fields of the architecture (T=975ms) is not an issue, you just need to reduce the hop size (to sth like 10ms) to get per-frame features for ASR. The potential impact of the work could be higher if demonstrated on more significant audio tasks. 2. For the downstream supervised tasks presented in the paper, the baselines seem a bit weak. There was not much details on the *naive-Bayes* classifiers for aggregating segment (with duration 975ms) predictions to sample-level prediction, and how this choice is justified. I could imagine using a LSTM-based aggregator. Furthermore, if only an utterance-level feature is needed for the downstream tasks, one could even use the features learned by the method of Jansen et al, ICASSP 2018 (in fact their features are learned from AUDIOSET as in this submission). I am not sure if all tasks/datasets presented here are standard benchmarks; the authors shall try to provide the state-of-the-art performance for them. Ideally, the results shall be close to state-of-the-art with reasonable model architectures. Additionally, some of the tasks seems not very challenging: baseline linear classifier can achieve almost perfect performance on LSP? Also, for encoder fine-tuning, it appears fine-tuning the top layers could significantly improve the performance, without increasing model complexity. Then why do not we always fine-tune all layers for downstream tasks, instead of using just linear or shallow networks on top of the features (freezing the parameters of the feature extraction networks)? In fact, in a pre-training + fine-tuning scheme, we would hope representation learning improves the generalization performance of downstream tasks. The experimental results of *retrain_last_3* does not show such improvement yet. 3. Some of the experimental results contradicts with my intuition and perhaps more analysis is required to explain why. For example, while I can see that Librispeech as the feature learning dataset is inferior for non-speech audio event based tasks, it is not clear why it does not help on more human-speech related tasks (SPC and LSP, and in fact representation learning on Librispeech does not help LSP?) 4. The authors argued in a few places the choices are suited for *on device* federated learning. I think it should be made clearer at the beginning of the paper the problem setup, and how it constrain learning (smaller model size, low computational budget, or limited communication bandwidth etc al). In any case, the justifications seemed somewhat contrived to me. If the goal is to learn generally useful features suited for multiple downstream tasks, I do not expect this can be achieved with a very small model. Besides, there are other ways of making models small, e.g., knowledge distillation or model quatization etc. 5. Impact of encoder architecture size: when you increase the model complexity, should not the *supervised* baseline also improve? Are the results in Table 4 comparing against the stronger *supervised* baseline or the one in Table 2?