Pretext Tasks Selection for Multitask Self-Supervised Speech Representation Learning
Authors: Salah Zaiem
Titouan Parcollet
Slim Essid
Abdelwahab HEBA
Venue: ICLR 2022 Submitted (2022)
Keywords: Self-Supervised Learning
Speech Processing
Representation Learning

Abstract: Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experiments conducted on automatic speech recognition, speaker and emotion recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.


Under review as a conference paper at ICLR 2022
PRETEXT TASKS SELECTION FOR MULTITASK SELF- SUPERVISED SPEECH REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features the downstream task. In audio/speech signal processing, wide range of features where engineered through decades of research efforts. As it turns out, learning predict such features (a.k.a pseudo-labels) has proven to be a particularly rel- evant pretext task, Ieading to useful self-supervised representations which prove be effective for downstream tasks. However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces method to select group of pretext tasks among set of candidates. The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experi- ments conducted on automatic speech recognition, speaker and emotion recogni- tion validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combina- tion of relevant pseudo-labels for self-supervised representation learning.
INTRODUCTION
Self-supervised learning (SSL) methods usually rely on a supervision obtained from the data itself through solving specific pretext tasks leveraging the underlying structure of the considered data (Doersch et al. | AAaanjeelovic & Zisserman 2018). This technique is used in various domains including image processing (Misra & Maaten| 2020 Jing & Tian 2020 Grill et al. (020), natural language understanding (Chen et 2020bb Du et al. 2020 Lan el al.) 2019) or speech and audio processing (Baevski et al. 2020b Liu et al. 2020 Jiang al. 2020). It offers numerous advantages, such as the independence from labeled data, stronger performance on downstream tasks, more robust models and easier transfer low-resource setups (e.g., low-resource languages) (Baevski al., 20206 Jing & Tian 2020).
The numerous existing SSL approaches characterized by the nature of the pretext tasks they solve. For instance, common techniques include predictive coding (Baevski et al. 2020b Liu et al.
2020
Song al., 2020; Zhang al. 2020 Hsu et al. [2021), pseudo-label learning (Pascual al.
2019 Ravanelli et al. (220)), auto-encoding (Renshaw et al. 2015) Algayres et al. 2020). triplet-
loss learning (Shor et 2020 Peplinski et al. 2020) generative modelling (Khurana et 2020)
J02ng or contrastive learning (Saced et al. 2020 Jiang et al.] 2020). More precisely, these pretext tasks may be defined through the choice of pretext labels, hereafter referred to as pseudo-labels. The auto- matic extraction of pseudo-labels for SSL (i.e. from the data itself) is common in many application domains, such as computer vision NNrroozi Favaro 2017 Gidaris et al. 2018 music processing (Hung al.) 2019 Wu et al. 2 and speech processing (Pascual et al. 2019 Shukla al. 2020), and is commonly referred to as multitask self supervised learning. In the specific context of speech processing, the process of designing pseudo-labels may benefit from decades of research in signal processing. For instance, potential candidates are pitch estimators, energy-based features, voicing state and many more.

---

Under review as a conference paper at ICLR 2022
As demonstrated by Pascual et (2019), multitask speech representation learning is a powerful tool to build representations that are beneficial for a wide range of distinct downstream tasks, by com- bining different pseudo-labels which "intuitively" correspond to these tasks. Unfortunately, there is no clear understanding of how these pseudo-labels may interact when optimised together, and there- fore, common practice of how select groups of pseudo-labels to obtain better performance on a known downstream task. As a matter of fact, this design process has been essentially driven by empirical validation and there therefore no evidence that the obtained model is even the best one. This empirical approach can rapidly become intractable with modern SSL architectures which may contain hundreds of millions or billions of parameters trained on thousands of hours of speech, not to mention the carbon footprint of such pseudo-labes searches. For instance, the self-supervised training of single state-of-the-art large wav2vec 2.0 model (Baevski et 2020b) on 2k hours of speech requires 128 GPUs for 5.2 days.
This work aims at providing a clear, efficient and theoretically motivated procedure for pseudo- label group selection and weighting based on conditional independence (CI). The method presented allows one to design ahead of training the most adapted multitask self-supervised speech representa- tion learning model which perfectly suits the considered downstream tasks. Such approach may also enable researchers to save a substantial amount of time and computation usually devoted to pseudo-label search. Hence, the contributions of this work are fourfold:
1. Introduce a theoretically motivated and computationally efficient method for the selection of pseudo-label groups among set of candidates and with respect to the considered down- stream tasks (Sections 3 and
2. Validate empirically the proposed approach with a first model SSL model relying on dif- ferent sets of pseudo-labels corresponding to the ones obtained for three considered speech tasks. (Sections 5).
3. Extend our method to the SOTA wav2vec 2.0 enhance its performance (Section 6
4. Release the code base developed with SpeechBrain (Ravanelli et al. 2021 for replication and to encourage further investigations!
The conducted experiments demonstrate that the proposed method allows for a more intelligent, i.e. better informed, pseudo-label group selection for multitask SSL settings. Indeed, we find that the models built with the proposed method obtain a word error rate and an equal error rate, respectively, 31.6% and 27.4% lower than the baseline, without the need for any empirical search.
2 RELATED WORKS AND MOTIVATIONS
SSL recently became a key component to achieve good performance on downstream tasks especially with low-resource setups, either in speech (Baevski et al., 2020b) Conneau al.) 2020), natural language processing (Lan et al. 2019 Chen et al. 2020b) or computer vision (Gidaris et al. 2019 Misra & Maaten 2020; Jing Tian 2020p. Due to very nature, SSL relies on large amounts of unlabeled data used to train large deep neural networks over long periods of time. It it thus crucial to understand properly what makes a good SSL model to lower the amount of computation and time needed to obtain the best downstream performance.
SSL for Speech. Self-supervised learning for speech has recently enabled researchers to reach state- of-the-art results on various speech processing tasks (Fan et al. (02T). The most successful models rely on predictive and contrastive objectives (Baevski et al. 2020b Chung et all 2019 Hsu et al. 202 Shor et a 202I) performing well across the different tasks even in low-resouree settings. This led to the design of different benchmarks evaluating the self-supervised representations in dif- ferent languages (Yang et al.) 2021 Evain et al.] 2021). However, in contrast to this proposition, these works have not tried to motivate beforehand the different choices made in the self-supervision
pipeline.
Understanding SSL. A few works have tried to shed some theoretical light on the mainly empiri- cal field of self-supervised learning. Following the different paradigms in SSL, various tracks have
https://github.com (ttpuummmitte222//1eae
ssian
2

---

Under review as a conference paper at ICLR 2022
1. Pastact Treis Groug Seletian and Wiightin
Silettin of Protent
Incependene: Extimatic
Tosis
Conneponding
Weights
Dommermam labdl
Sc5-Brpervied
Sell Supervised
Fistue
Rusustrrcton
Pattninin Datastt
Turget
Eacnder
Protest Tadls Group Prodiction
Protext Thoks
Domnsteoun Triining
Downdtrcom Dutezee
Model
Figure 1: Illustration of the training pipeline. The three steps are depicted: Selecting the group of pseudo-labels and their corresponding weights; 2. SSL training with the selected pretext task; 3. Training on the downstream task with the pretrained SSL model.
been followed to understand what makes for good self-supervised representation, exploring differ- ent approaches (Lee al. 2020 Arora et al., 2019 Wei et al.] 2020). On the one hand, contrastive learning (Oord et al. 2018; Chen et al. 2020a) has been advocated both theoretically and empiri- cally to achieve balance the mutual information between alternative representations of the data, keeping just enough shared information to keep the class-related content (Tschannen et al.) 2020)] Tian et al. 2020; Bachman al. 2019). In recent work from Li al. (2021), independence testing has been used to produce better transformations in contrastive learning settings for image represen- tations. Predictive learning, on the other hand, requires the model to predict masked elements in sequential data. This technique is powerful downstream tasks that can be reduced to mask- ing problem, suggested by research on language modeling (Saunshi et al., 2020). However, all these works have been focusing on computer vision or text-related applications, and none of them addressed the multi-tasked self supervision problem.
Multi-task self-supervised learning. While the literature on multi-tasking in self-supervised learn- ing remains scarce, it has been shown in classic supervised learning settings, that through estimates of similarity between tasks thorough empirical testing, several tasks take advantage of being solved with common encoder (Zamir et al.] 2018 Dwivedi Roig 2019 [hafey et al., 2019 Chen al. 2015). More specifically, combining pretext tasks with SSL has been mainly explored in computer vision and speech (Pascual et al.) 2019] Ravanelli et al.] 2020)_ Pretext tasks such as Jigsaw (Doersch et al., (2016). colourisation and rotation (Gidaris al. al.[20188 have been combined successfully to improve downstream performance (Kim et al. 2018 Shin 'aa Yamaguchi et al.) The closest works to our line of research are from Lee et al. (020)) and Doersch al. (2017). The former shows that a theoretical link can be established between conditional independence and an improvement of the performance on the downstream task, while the latter proposes to select layers from a multitask self-supervised encoder according to the pretext task to be solved. However, in both cases, the studies do not offer practical and theoretical solutions to select groups of pseudo-labels to build adapted SSL model that will perform well on the considered downstream tasks.
Group feature selection. Finally, feature selection, and especially group feature selection is another close and inspiring field given the problem we consider. The relationship and interactions between features have been largely investigated in the supervised learning literature (Guyon & Elisseeff] 2003). This led to multiple solutions to the feature group selection problem, including LASSO- based techniques (Yuan & Lin 2006), or multiple kernel formulations (Sonnenburg et al., 2006 Raakotomamnnyy et al. 2007). However, these works do not involve any self-supervissio,, and links

---

Under review as a conference paper at ICLR 2022
between feature selection and self- suserviueriiion design and pretext-task selection yet to be proved. We will further consider these lines of works for concurrent baselines.
With this work, we aim at shortening the process of designing SSL models while giving insights on the pseudo-label importance and the underlying mechanisms between pretext and downstream tasks at the same time. We decided to experiment with speech due to the lack of literature on this domain for multitask SSL, and for the various pseudo-labels available, which are based on decades of signal processing research. The whole pipeline starting from the acoustic feature extraction to the downstream task scoring follows three major steps summarized in Figure []] First, for every downstream task, our method produces a pretext task selection and weighting. Then, SSL model is trained, before being used as a feature extractor front-end to one or many downstream tasks.
3 CONDITIONAL INDEPENDENCE FOR UTILITY ESTIMATION
As a first step, we require a function that estimates the utility of learning to solve a pretext-task to improve the performance on the downstream task. We use an estimation of the conditional inde- pendence between the pseudo-label values and the downstream data points given the downstream labels. Hereafter, we explain the thcoretical motivations and describe the computation steps.
3.1 PROBLEM DEFINITION AND INTUITION
Let X, Y and Z be, respectively, the downstream data points, their downstream labels and their pseudo-labels. Let also C' be the set of possible downstream classes. As an example, if one considers speaker recognition as downstream task, X would be the speech samples, Y the speaker IDs, C the set of unique speaker IDs, and Z a computed signal feature, such as the fundamental frequency.
As stated in Section Lee et al. (2020) linked the utility of a aseudo-label (Z) to the conditional independence (CI) between and X given Y. The approach prescribes that, given the labels Y, one may seek quantify how much it is possible predict the pseudo-labels Z without knowing much about X. The authors bounded, under certain assumptions, the downstream classifier error with a function of the downstream training set size, and a measure of the CI. More precisely, the main theorem shows that the bounding function decreases linearly with the downstream-tass dataset size (M) and quadratically with the CI, thus making a potential estimator for pseudo-label utility.
The proposed function depends on the final downstream task to be solved. This is motivated by two main reasons. First, can be seen through the large literature on feature selection for various speech or computer vision tasks (Liu et al.) [2020) Serizel et al.) 2017 Schuller et al., 2007; Wang et al.) 2019), that different tasks require the description of different aspects of the data. This suggests that different downstream tasks may perform better after different pre-train A second argument is the difficulty to evaluate representations quality intrinsically, i.e. independently from the choice of a particular downstream task. A few metries and tests (Schatz et al. 2013 Carlin al. 2011 Lakhotia ct al. 2021 have been proposed for speech, but the correlation between these and downstream-task performance has not been clearly identified (Algayres al. 2020 Gump al. 2020).
The principal issue with CI is the difficulty of computing good estimates of how much two variables independent given third one realistic data (Shah & Peters 2018). In a previous work (not cited to respect the double blind reviewing). we proposed a simple way to get an estimation of the conditional independence. This method has proven effective for individual pretext task selection, as the utility estimator correlated highly with the final downstream performances. In our case, the considered pseudo-labels are not independent of the speech samples, as they are signal features. The approach resorts to performing classic independence testing on data sliced by the downstream classes, to check whether this dependence remains given the downstream labels information.
3.2 CONDITIONAL INDEPENDENCE ESTIMATOR COMPUTATION
This section details the computation of the conditional independence estimate that is used mea- sure of pseudo-label utility. Let X = {a, hig (o......) with M being the cardinal of X and x; data samples (e.g. Mel-band spectrogram for audio and speech). Every sample I; has corresponding downstream label y, and an automatically generated pseudo-label 2,. We assume that y; is discrete reducing the task to a classification problem such with speaker IID for speaker recognition. We
4

---

Under review as a conference paper at ICLR 2022
also assume that for every pretext-task Z, single 2; value corresponds each 24. In case, 2, values are the mean of the frame-wise wise pseudo-label values.
For independence testing, Criterion (HSIC) (Gretton et al.) 2007) for two reasons. First, HSIC has already proven successful for textual data in testing statistical dependence between translated sentences (Gretton al. 2007). Second, kernel-based techniques facilitate the handling of multivariate and varying-lengte data such as speech, the estimation then boils down to the computation of similarity measure between the considered variables.
decided to rely on the kemel-bbased Hilbert Schmidt Independence
Computation steps. The estimation of the CI of pseudo-label Z for downstream task (X, Y) consists of three steps. We start by splitting the data samples X according to the downstream (discrete) classes. Then, we compute for every downstream class E C. the kernel matrices Ke and Le representing the similarity measures for the data samples, and the pseudo-labels, respectively. Finally, we perform the independence test for every split group using K2 and Le and aggregate the estimates with a weighted mean taking into account the number of samples per downstream class. Thus, for two speech samples z; and Ty, holding two pseudo-label values 2j and Zj. the coefficients of the similarity matrices Kg and Le are computed as follows:
Kij K (mi mry = cos (()())) GD(z))), La= RBF zz);
with GD(.) the Gaussian Downsampling function (more details in the appendix A.7) cos(.,.) the cosine similarity, and RBF((..) the Radial Basis Function kernel, defined as:
cos(æ, x²) trace(x"
2111
RBF(2,2)) = exp( 20
||<|||||>|||
where a is the width of the RBF kernel and trace(.) the sum of elements the main diagonal. Note that we compute the matrices K₂ and Te, for each group of samples sharing the same downstream class e E C. Hence, Ke and Le correspond to the definitions above, but restricted to the points with as downstream label. For each downstream class c, and as in Gretton al. (2007). the HSIC value is given by:
1
HSIC.(X, 2) - (K.uee(,HHLLell
ni
with H. = In. - In. no. being the number of points with downstream label and 1n, n. vector of ones of size nhe x 1.
The HSIC value is non-negative and corresponds to the Hilbert norm of their cross-covariance ma- trix. It is used to characterize the independence of the two considered quantities. Intuitively, the HSIC value is high if samples similar in Kg are similar in L.. Therefore, the lower this value is, the more independent the two arguments of HSIC are and the better the pseudo-label should be for self-supervision before fine-tuning on the downstream class. The final value for given pseudo-label and downstream task is expressed as:
1 E (X, X n..
HSIC(X, M
(4)
GEC
Computational efficiency. Efficiency one of the key motivations of this work, and the gain in time observed with approach is considerable. The K and L. matrices used for the CI estimate are only computed for the downstream datasets. But since these datasets may get very large, we can sample among the downstream classes keep the computations tractable as shown in appendix A.3
4 PRETEXT TASK GROUP SELECTION AND WEIGHTING
While we now are able to predict the utility of every considered pretext task independently, the next step remains to define a clever way to combine them optimally within the same pre-training process. Hence, we introduce method to select a group of pseudo-labels and weight their respective losses to increase or decrease their importance in the self-supervised representation. Such an optimisation of the latent representation is expected to provide better downstream performance. Our method min- imises the conditional dependence between the resulting group pretext task, entailing the prediction of selected pseudo-label group and the downstream samples given the downstream labels.
5

---

Under review as a conference paper at ICLR 2022
Given set of k possible pseudo-labeli (0,))) seek estimate set of parameters (As) weighting the loss of every pseudo-label during the pre-trainn training phase. Hence, we define grouping pseudo-label Za as orthogonal concatenation of (Zi)ie(o.x) weighted with (As)ke(o.A) Z^ (1, ..., Ax))).
The custom conditional HSIC computation pipeline described above is fully differentiable with re- spect to (Ae)seju,K) as proved in appendix A.I In the HSIC computation, the data similarity matri- ces (ee) are independent of 7 and therefore of X. Only the pseudo-label similarity matrices {TeJCEC are changed. For every downstream class c, Le is defined as:
k
[Lcks = RBF((BFF(() (Zx),) = exp(
>nllzn, - Zh. 12),
(5)
20
h=1
with Zh,i denotes the mean value of the f-th pseudo-label for the i-th data point the dataset.
4.1 CONSTRAINTS ON THE WEIGHTS
The conditional-indes -indeendeeeeeeeeeeeeeeeeeeeeeeee based utility estimator must be optimized with respect to the weight- ing parameters (A.) and three constraints.
First, the parameters (At)sefo.a) must be positíve, as they are used as weights for the correspond- it could imply that the self-
ing losses. negative weighting loss would lack interpretability supervised model should "unlcarn" the corresponding pretext task. This may be the case for adver- sarial learning methods, but we are not considering this case in the present work.
Second, the value of the weights must be high enough. Indeed, the presented method for estimating the conditional independence assumes that the considered pseudo-label It is fortunately true for speech features, as Z is a function of X, but not always valid. For instance, with (A:) = 0, the utility estimator would would break the assumption of non independence between Z and X. Furthermore, the HSIC value decreases with positive decreasing values of (Ao)sgjo.e and we thus need to ensure that the sum of the weights is significantly higher than zero, or it would mean that we are not really doing multi-task learning as the losses of the pseudo-labels would be barely considered.
not independent of X.
null and thus the lowest (i.e. the best), but
Finally, for a fair comparison between the weighting choices during the optimization, the sum of the weights should remain constant. In the following, the sum of the (Ar)iejo.e) is fixed to one and the problem is summarized as follows:
(6) To minimize the estimator quantity while respecting the constraints, the weights used in the com- putation of the CI value are the softmax output of free learnable parameters (Wa)sejo.u) Softmax enforces the conditions while being differentiable and the problem becomes:
min XER
HSTC X,Y), s.t. ZA =
ZR), X O, ViE (iE Σ
min HSIC X,Y), Y), s.t. = Sofftmr(WW), Zx = (A, Ak..) WERS
4.2 WEIGHTS SPARSITY
Another trait that is desirable for the weighting vector is sparsity. If a few pseudo-labels are not needed for the given downstream task, they would rather discarded than given low weight. First, this would save computation time including the extraction of the pseudo-labels, and their extraction and prediction during the self-supervised training process. Second, would help with transparency to understand what features are included or not in the latent space. This sparsity property is also related to feature selection such as with LASSO (Yuan Lin 2006). To ensure the sparsity of the output weighting vector, while maintaining the desired property of differentiability, choose the sparsemax function (Martins Astudillo 2016) to replace the softmax in eq.
EXPERIMENTAL STUDY
This section details the experiments validating the introduced estimator. It describes the selection and weighting processes (S. the SSL models (.22), the downstream tasks (Section 5.3
and the obtained results (Section 5.4

---

Under review as a conference paper at ICLR 2022
5.1 GROUP SELECTION AND WEIGHTING
Individual pseudo-labels of interest are obtained with the OpenSmile library (Eyben et al. 2010). We decided focus on markers mostly related prosody and spectral descriptors as the signal pro- cessing literature commonly associates them to the three considered downstream tasks (i.e. speech, speaker and emotion recognition). Selected pseudo-labels include: Loudness, FO, Voicing, Ra- tio (Sundberg & Nordenberg 2006), Zero Crossing Rate, LI Norm of Rasta Spectrum (Hermansky al. 1992). log of Harmonicity to Noise Ratio (Murphy Akande (005). Then, and according to Figure I (step f), we group these pseudo-labels based on their weights, i.e. by optimising equation to obtain the different À values associated to each one of them.
Comparative baselines follow common feature group selection strategies or natural intuitions. The first one simply bundles all the pseudo-labels together without any weighting (i.e. = 1 for all proposed for PASE (Pascual al. 2019). As SSL group pretext-task selection is yet to be fully explored, the two other baselines come from the feature selection literature as represents the closest field with well established techniques. The Cl-based pseudo-label selection is thus compared to Recursive Feature Elimination (RFE, Guyon et al. )2002) and Maximum Rel- evance Minimum Redundancy (MRMR, Peng et al. (2005)). We also add experiment with the remaining pretext tasks after the MRMR selection, this is to show the effect of learning supposedly unrelated workers, we will call it "Remaining". More details about these baseline algorithms are given in Appendix A.9, while Appendix [A.I] shows the workers selected and their corresponding weights for every experiment. Noise (HNR) seems to be the most important information learn to predict for speaker recognition while fundamental frequency is privileged for ASR.
pseudo-labels)
5.2 SELF-SUPERVISED TRAINING
In the second step of Figure T the SSL model learns to predict the selected pseudo-labels. For every one of those, the loss is multiplied by the corresponding assigned weight. Based previous work conclusions (Ravanelli et al. 2020 Diang et al., 2020) and apart from the considered pretext task, the network learns to reconstruct the input Mel spectrograms, and to compute 40- 40-dimensional Mel- Frequency Cepstral Coefficients (MFCC) feature vectors. These targets are usually kept to avoid information loss harming heavily downstream performance and are used in all experiments. For a given weighting vector (As ()s))E(o.M) the self-supervised loss is defined as:
Σ X,)) i=1
k
LSst = MSE MmE +
(8)
with MSE the classic mean squared error computed for Mel spectra (MSE and MFCC (MSEmfac), and (2) the llos of the pretext task related to pscudo-label. Z.
Prior to extending our method to state-of-the-an art architectures such as wav2vec 2.0 that are par- ticularly costly to train, we propose to first employ a PASE-like model to empirically validate the approach. Hence, the encoder is composed of three distinct parts: VGG-like feature extractor (Si- monnan & Zisserman 2015), bidirectional LSTM, and a two-layered dense neural network. All the details of the architecture are given in the appendix A.5 Then, and inspired by Ravanelli et al. 2020), the encoder is followed by simple one-layered predictors voluntarily limited in capacity.
SSL dataset. The SSL model optimised on the training set of the English Common Voice dataset (version 5. 700 hours of training, Ardila et al. (2020)) Common Voice is a collection of speech utterances from worldwide users recording themselves from their own devices. Hence, the closeness to natural settings makes it a suitable choice for self-supervised learning. 700 hours of speech relatively small amount compared to what is generally used for state-of-the-ar SSL models. However, we believe it is sound choice as this is generally greater than what is typically available in SSL use-cases like low-resource languages. We decided to not use the LibriSpeech dataset for pre-training as it is part of our downstream evaluation protocol hence alleviating strong bias.
5.3 DOWNSTREAM TASKS
Our proposed pseudo-label selection strategy is compared with the two baselines on three differ- ent downstream tasks leading to different groups of pseudo-labels: automatic speech recognition (ASR, with LibriSpeech 100 hours) speaker recognition (SR, with VoxCeleb 1), and emotion recog- nition (ER with IEMOCAP). Datasets and downstream architectures are inspired from the SUPERB
7

---

Under review as a conference paper at ICLR 2022
Table 1: Results observed with the proposed selection strategies the two considered downstream tasks. Word Error Rate (WER) Equal Error Rate (EER), and Accuracy (Acc) are expressed in percentage and used for LibriSpeech 100 hours, VoxCelebl and IEMOCAP respectively (i.e. lower is better). ASR results are given with and without Language Modeling (LM). All SSL models contain 16.3M neural parameters.
Models
LibriSpeech (WER &
VoxCeleb1 (EER %
JEMOCAP (Ace
I.M
I.M
PASE+ Ravanelli et al 2020
25.11
16.62
11.61
57.86
vq wav2vee Baevski
202
12.80
58.24
Selections
All
21.98 + 0.36
11.70 + 0.27
11,90+ 0.32
56.4 + 1.3
MRMR
18.94 0.34
10.36 0.26
10.56 - 0.31
59.6 1.229
Remaining
0.34
11.65 +: 0.27
11.67 0.32
1.29
RFE
20.02 + 0.34
11.42 + 0.27
11.91
+ 0.33
55.8 = 1.3
Softmax
13.17::: 0.28
8.00 0.23
9.24 0.29
60.6 1.27
Sparsemax
17.18 0.32
10.41 + 0.26
8.63 0.27
60.8 1.28
benchmark (Yang al. 2021) for self-supervised learning representations and carefully described in Appendix A.5.3 Prior to downstream training, the SSL model are frozen to be used as feature extractor with the new pipeline that is task-dependent. We do not use any data augmentation for pristine comparison of the learned models.
5.4
RESULTS
Baselines detailed in Section 4 are respectively referred "AIl", "RFE" and "MRMR". All the details about the selection and weights are available in Appendix [A.1] First, it is clear from the results reported in Table that, for the considered downstream tasks, the two introduced strategies (Sparsemax and Sofimax) perform better than the group selection baselines with a gain of 3.28 of EER for Sparsemax against the RFE approach on VoxCeleb, and 8.81 of WER with Softma com- pared to the All baseline. Interestingly, simply bundling all the pseudo-labed together may lead to poor performance as observed on LibriSpeech with very high 21 21.98% of WER obtained. Hence, intuitively building sets of labels could be harmful for the final representation. This motivates the need for better pseudo-lahel selection strategy such as the one introduced in this work, as the WER dropped to 13.17%. As comparison, the exact same architecture trained with Mel spectra only (i.e. no SSL) obtains a WER 17.3% without L.M. Hence, our method even further decreases the WER while being only pretrained with reasonable amount of data (i.e. only 700 hours compared to few thousands for common SSL techniques (Baevski et al.) (020b)). As expected, introducing the joint decoding with a language model strongly decreases the WER but also introduces bias in our comparison as probabilitin bobbabiiities are smoothed with a third-party neural model. Nevertheless, and even in this scenario, our weighting strategy outperforms all the baselines. In the context of speaker recog- nition, Sparsemax beats Softmax with an EER 0.61 lower. For IEMOCAP, Sofumax and Sparsemax weighting still perform the best among all methods. To investigate how strongly improvements are correlated to the task, we took the best learned model for LibriSpeech (i.e. softmax weighting) and fine-tuned it on VoxCeleb1 and IEMOCAP. It reaches an EER of 10.55% and an accuracy of 59.9% respectively. While it performs better than the baselines, the difference between these results and the best performing selections shows that the weightings are indeed task-related.
EXTENDING WAV2VEC 2.0 TO MULTITASK SSL
To the best of our knowledge, multi-task speech representation learning has not been scaled to point where could represent a state-of-the-ar alternative. Contrastive predictive coding (Oord et al., 2018) based techniques like wav2vec 2.0 (Baevski et al., 2020b), on the other hand, currently trust most of the leader- boards for speech-related tasks. Recently, Sadhu et al. (202T) showed that com- bining consistency loss and contrastive predictive coding improves the results of the wav2vec 2.0 architectures in noisy conditions. Following this idea, we propose to further validate our selection method with extension of wav2vec 2.0 to multitask SSL to demonstrate its scaling capabilities.
8

---

Under review as a conference paper at ICLR 2022
Table 2:
Results observed retraining the Wav2vec2 model with and without weighted pretext tasks using the sparsemax method. "Fr." and "Fine." also respectively refer to Frozen and Finetuned set- tings, Adding selected pretext tasks improves the donwstream performance on all three considered tasks. All models contain 100M neural parameters.
Selections
LibriSpeech (WER (
VoxCeleb1 (EER % (
IEMOCAP (Ace % TV
Fr.
Fine.
Fr.
Fine.
Fr.
Fine.
wav2vee 2.0 BASE
17.93 = 0.33
10.21 + 0.25
7.20 + 0.26
5,35 = 0.22
56.6 1.2
74.0 1.16
wav 2vec 2.0 BASE Naive selection
17.23 + 0.32
10.10 + 0.25
6.80 + 0.25
5.05 0.21
57.4
73.7 1.16
wav2vec 2.0 BASE Sparsemax
16.70 = 0.31
9.18 +: 0.24
6.57 = 0.25
5.30 + 0.22
59.5 = 1.29
74.0 + 1.16
Hence, the training loss is extended to:
LSSL = Lw2v + E 1=1 Az)(
(9)
We use the standard BASE wav2vec 2.0 first described in (Baevski et al., 2020b) as a SSL model and train it with the same Common Voice dataset. The pre-training pipeline implemented within SpeechBrain. The trained BASE model has been compared to one obtained with the official Fairseq implementation from Baevski al. (2020b), and results are strictly equivalent. The entire recipe alongside with the large set of hyperparameters needed to properly train wav2vec 2.0 model are released under our anonymous repository and will be made available with SpeechBrain afterwards.
We follow the SUPERB benchmark conventions Yang et al., 2021) both at the data and downstream architecture levels. Hence, and conversely to the previous experiments, the ASR system solely optimises the CTC criterion over characters. For each of the three tasks (i.e. ASR, SV, ER) compare the standard BASE wav2vec 2.0 model with one trained following the sparsemax selection of multitask SSL. Sparsemax is chosen over softmax because it enforces the sparsity criterion and removes completely few pseudo-labels from the training, which is one of the objectives of this work. Another experiment is led with a "naive" pretext-task selection where constant weight of 0.5 is used across all signal-based pretext-tasks. Like the other experiments, the exact weights of each pseudo-label are reported in Appendix A.IT Each wav2vec 2.0 model required 24 NVIDIA Tesla V100 GPUs to train for 150 epochs (40 hours). Finally, we also propose to compare frozen and unfrozen (i.e. where the wav2vee 2.0 encoder is fine-tuned with the downstream task) SSL models.
It is clear from the results reported in Table 2 that our approach improves the performance over the standard wav2vec 2.0 framework for every considered downstream task. While adding pretext tasks naively improves the final performance, the difference in performance between the naive selection and the sparsemax weighting shows the benefit our method in getting the best downstream perfor- mance. Unsurprisingly this difference is small (though statistically significant in all but one case), as the wav2vec 2.0 BASE is already powerful and the additional workers are anyway useful. Here, is worth noting that the difference in performance compared to the literature mostly comes from the pre-training conditions. For instance, wav2vee 2.0 is commonly pre-trained with larger models on LibriSpeech achieve lower WER this dataset. This comparison can be found in appendix AA88 as it could bias the evaluation of our method with pre-training and target datasets being the same.
7 CONCLUSION
In this work, we introduce method to quickly and simply combine pseudo-labels into useful pretext task for multitask self-supervised learning settings. Our approach allows for an optimal selection of pseudo-labels following a cheap optimisation process drastically decreasing the time and compute needed to design the best performing multitask SSL model. Our method is validated on three speech-related downstream tasks and outperforms common pseudo-label selection strategies when combined with simple and state-of the SSL models. This opens a range of possibilities for finding and selecting new pretext tasks in self- cuff-super learning for speech other types of data.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Robin Algayres, Mohamed Salah Zaiem, Benoît Sagot, and Emmanuel Dupoux. Evaluating the reliability of acoustic speech embeddings. In INTERSPEECH 2020 Annual Conference of the International Speech Communication Association, Shanghai Vitrtual, China, October 2020.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A masiivel--mutilnguaal speech corpus, 2020.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A Theoretical Analysis of Contrastive Unsupervised Representation Learning. 36th International Conference on Machine Learning. ICML 2019, 2119-June9904-999 9923, feb 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning Representations by Maximiz- ing Mutual Information across Views. Curran Associates Inc., Red Hook, NY, USA, 2019.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations, 2020a.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A frame- work for self-supervised learning of speech representations. arXiv preprint arXiv:2006.1147) 2020b.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. lemocap: in- teractive emotional dyadic motion capture database. Language Resources and Evaluation, 42: 335-359, 2008.
Michael Carlin, Samuel Thomas, Aren Jansen, and Hynek Hermansky. Rapid evaluation of speech representations for spoken term discovery. pp. 821-824, 01 2011.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020a. URL
http://proceedingg mlrr wwt.mmm..........mmmm........s.s wwwppppsssssllleeeeeeeepee http://////eeeenicioee html
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. 2020b.
Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and J. Hershey. Speech enhancement and recogni- tion using multi-task learning of long short-term memory recurrent neural networks. In INTER- SPEECH, 2015.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass, An unsupervised autoregressive model for speech representation learning, 2019.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. Un- supervised cross- cross-lingual representation learning for speech recognition. CoRR, 1b3/2006.13979, 2020. URL httpss//aaxx...orrg////////00 13979
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction, 2016.
Carl Doersch, Andrew Zisserman, and Deepmind. Multi-task Self-Supervised Visual Learning. Technical report, 2017.
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi. Michael Auli, Ves Stoy- anov, and Alexis Conneau. Self-training improves pre-training for natural language understand- ing, 2020.
10

---

Under review as a conference paper at ICLR 2022
Kshitij Dwivedi and Gemma Roig. Representation Similarity Analysis for Efficient Task taxon- omy Transfer Learning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-Jume:12373 1tt9:mm::127991338:::333333 apr 2019. URL http://arxiv
org/abs 1904. 11740
Solene Evain, Ha Nguyen, Hang Le, Marcely Zanon Boito, Salima Mdhaffar, Sina Alisamir, Ziyi Tong. Natalia Tomashenko, Mareo Dinarelli, Titouan Parcollet, Alexandre Allauzen, Yannick Esteve, Benjamin Lecouteux, Francois Portet, Solange Rossato, Fabien Ringeval, Didier Schwab, and Laurent Besacier. Lebenchmark: A reproducible framework for assessing self-supervised representation learning from speech, 2021.
Florian Eyben, Martin Wöllmer, and Bjöm Schuller. Opensmile: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM International Conference on Multimedia, MM '10 pp. 1459-1462, New York, NY, USA, 2010. Association for Computing doi: 10. (0.1145//87395511187444 URL https://doi.
Machinery. ISBN 978 605589336.
org/10. 5 8777187788 .1 6874246)
Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu, Exploring wav2vec 2.0 on speaker verification and language identification, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. CoRR, abs/1803.00728, 2018. URL http::/arxiv.orrg /bs/
1803.07722
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8059-8068 2019.
Alex Graves. Connectionist temporal classification. In Supervised Sequence Labelling with Recur- rent Neural Networks, pp. 61-93. Springer, 2012.
Arthur Gretton, Kenji Fukumizu, Choon Hui Teo, Le Song, Bernhard Schölkopf, and Alexander Smola. A kernel statistical test of independence. 01 2007.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.077 07733, 2020.
Michael Gump, Wei-Ning Hsu, and James Glass. Unsupervised Methods for Evaluating Speech Representations. 2020. doi: 10.1143/hncrseecch220-200999 URL http:///a// org/ 10. I0.21447/Interspeech. 2990
Isabelle Guyon and André Elisseeff. An introduction of variable and feature selection. J. Machine Learning Research Special Issue on Variable and Feature Selection, 3:1 157 - 1182, 01 2003. doi: 10.166///53243002255511
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine Learning, 46:389-422, 01 2002. doi:
10.1023/A:101224330279
Hynek Hermansky, Nathaniel Morgan, A. Bayya, and P. Kohn. Rasta- plp speech analysis technique. volume 1, pp. 121 - 124 vol.1, 04 1992. ISBN 0-7803-0532-9, doi: 10. 109/ICASSPP1999 225957.
Nils Holzenberger. Mingxing Du. Julien Karadayi, Rachid Riad, and Emmanuel Dupoux. Learn- ing Word Embeddings: Unsupervised Methods for Fixed-size Representations Variable-length Speech Segments. In Interspeech 2018, Proceedings of Interspeech 2018, Hyderabad, India, September 2018. ISCA. doi: 00.14437/nterspeechh2111 201--2234.
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much
bad teacher benefit asr pre-training? In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6533-6537, 2021. doi: 10.1 0..1009/A 90119999999222229994444466 9414460.

---

Under review as a conference paper at ICLR 2022
Yun-Ning Hung, Yi-An Chen, and Yi-Hsuan Yang. Multitask learning for frame-level instrument recognition, 2019.
Sergey Ioffe.
Probabilistic Linear Discriminant Analysis. In Aleš Leonardis, Horst Bischof, and pp. pp. 531-542, Berlin, Heidelberg, 2006. Springer
Axel Pinz (eds.), Computer Vision ECCV 2006, Berlin Heidelberg, ISBN 978- 777-3-540-33883--0
Dongwei Jiang, Wubo Li, Miao Cao, Ruixiong Zhang, Wei Zou, Kun Han, and Xiangang Li. Speech simelr: Combining contrastive and reconstruction objective for self-supervised speech represen- tation learning, 2020.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Ricard Marxer, and James Glass. convolutional deep markov model for unsupervised speech representation learning, 2020.
Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. CoRR, abs/1802.01880, 2018. URL http::/aaa// org/abs/1802. 01860
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. Generative spoken language modeling from raw audio, 2021.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: lite hert for sule-super learning of language representations. arXiv preprint arXiv: 1942, 2019.
Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning, 2020.
Yazhe Li. Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization, 2021.
Andy T. Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsuper- vised speech representation learning with deep bidirectional transformer encoders. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020. doi: 10. 109/icaspp40776.2002 202.2.002........
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Rwth asr systems for librispeech: Hybrid vs attention. Interspeech 2019, Sep 2019. doi: 10.21457/merrppecch.0111
2tw.:ersppeeeee.211889999 URL http://dxx doi www..oooo....oo.ooooooo
21437/ 21437/Interspeet Interspeech 2019
André T. Martins and Ramón F. Astudillo. From softmax to sparsemax: A sparse model of at- tention and multi-label classification. In Proceedings of the 33rd International Conference on In- ternational Conference on Machine Learning Volume 48, ICML'16, 16. pp. 1614-1623, JMLR.org, 2016.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. pp. 198-502, 08 2017. doi: 10.21 10211437/ntersppecc..017.7
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717 2020.
Peter Murphy and Olatunji Akande. Cepstrum- Based Harmonics-to- No-ios Ratio Measurement in Voiced Speech. In Gérard Chollet, Anna Esposito. Marcos Faundez-Zanuy, and Maria Marinaro (eds.), Nonlinear Speech Modeling and Applications, pp. 199-218, Berlin, Heidelberg. 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31 18666.
12

---

Under review as a conference paper at ICLR 2022
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: A large-scale speaker identifi- cation dataset. Interspeech 2017, Aug 2017. doi: 10.24477/mterppoccc20779999
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles, 2017.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An corpus based on public domain audio books. pp. 520055006550 5210, 04 2015. doi: 10.1109/CCSSPP2015. 7178964.
Santiago Pascual, Mirco Ravanelli, Joan Serrå, Antonio Bonafonte, and Yoshua Bengio. Learning problem-agnostic speech representations from multiple self-supervised tasks, 2019.
Vijayaditya Peddinti, Daniel Povey, and S. Khudanpur. A time delay neural network architecture for efficient modeling of long temporal contexts. In INTERSPEECH, 2015.
Hanchuan Peng, Fuhui Long, and C. Ding. Feature selection based on mutual information criteria of max-dependeecyy max-relevance, and min-redundancy. IEEE Transactions on Puttern Analysis and Machine Intelligence, 27(8): 122--1238, 2005. doi: 10. 109/TPAMI.2005.155
Jacob Peplinski, Joel Shor, Sachin Joglekar, Jake Garrison, and Shwetak Patel. Frill: A non-semantic speech embedding for mobile devices. arXiv preprint arXiv:201 1.000/1..440 1.04609, 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability, 2017.
Alain Rakotomamoniy, Francis Bach, Stephane Canu, and Yves Grandvalet. More efficiency in multiple kernel learning. Proceedings of the 24th International Con-ference on Machine Learning (ICML). 227, 01 2007. doi: 10.1145/1273496.1 1273594.
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition, 2020.
Mirco Ravanelli, Titouan Parcollet, Aku Rouhe, Peter Plantinga, Elena Rastorgueva, Loren Lugosch, Nauman Dawalatabad, Chou Ju-Chich, Abdel Heba, Francois Grondin, William Aris, Chien-Feng Liao, Samuele Cornell, Sung-Lin Yeh, Hwidong Na, Yan Gao, Szu-Wei Fu, Cem Subakan, Re- nato De Mori, and Yoshua Bengio. Speechbrain. https : /ttp://ggttttbttt wwtp.///pppebaaarr.r..... speechbrain. 2021.
Daniel Renshaw, H. Kamper, Jansen, and S. Goldwater. A comparison of neural network meth- ods for unsupervised representation learning on the zero resource speech challenge. In INTER- SPEECH, 2015.
Samik Sadhu, Di He, Che-Wei Huang, Sri Harish Mallidi, Minhua Wu, Ariya Rastrow, Andreas Stolcke, Jasha Droppo, and Roland Maas. wav2vec-C: A Self-Supervised Model for Speech Rep- resentation Learning. In Proc. Interspeech 2021, pp. 711-715, 2021. doi: 10.21437/Interspeech. 2021-717.
Aaqib Saeed, David Grangier, and Neil Zeghidour. Contrastive Leamning of General-Purpose Audio Representations. oct 2020.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. CoRR, abs/2010.03648, 2020. URL https://arxxi
org/abs/ abs/ 2010. 03648
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Em- manuel Dupoux. Evaluating speech features with the Minimal-Pain ABX task: Analysis of the classical MFC/PLP pipeline. INTERSPEECH 2013 14th Annual Conference of the Interna- fional Speech Communication Association, pp. 1-5, Lyon, France, August 2013.
13

---

Under review as a conference paper at ICLR 2022
B. Schuller, B. Vlasenko, R. Minguez, G. Rigoll, and A. Wendemuth. Comparing one and two- stage acoustic modeling in the recognition of emotion in speech, In 2007 IEEE Workshop on Automatic Speech Recognition Understanding (ASRU), pp. 596-600, 2007. 2007. 4430180.
doi: 10.1109/ASRU.
Romain Serizel, Victor Bisot, Slim Essid, and Gael Richard. Acoustic Features for Environmen- tal Sound Analysis. In Tuomas Virtanen, Mark D. Plumbley, and Dan Ellis (eds.), Compula- tional Analysis of Sound Scenes and Events, pp. 71-101. Springer International Publishing AG. URL https: /tttps://hal.... aarchaal.rraaaacccc.c ves ouvertes.
2017. doi: 10. 1001110978---331-6455000 Er
Laurent El Shafey, Hagen Soltau, and Izhak Shafran. Joint speech recognition and speaker diariza- tion via sequence transduction. CoRR, 0etp//077003337 2019. URL http:://axxxi/
abs/1907, 05337
Rajen Shah and Jonas Peters.
The hardness of conditional independence testing and the generalised covariance measure. Annals of Statisties, 48, 04 2018. doi: 10.1214/19-0011857
Sckitoshi Shin ya Yamaguchi, Tetsuya Kanai, Shoichiro Shioda, Ntt Takeda, and Japan Tokyo. Multiple Pretext-Task for Self-Supervised Learning via Mixing Multiple Image Transformations. Technical report.
Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Felix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, and Yinnon Haviv. Towards learning universal non- semantic representation speech. arXiv preprint arXiv:2002. 12764, 2020.
Joel Shor, Aren Jansen, Wei Han, Daniel Park, and Yu Zhang. Universal paralinguistic speech representations using self-supervised conformers. arXiv preprint arXiv:2110.04621 2021.
Abhinav Shukla, Stavros Petridis, and Maja Pantic. Learning speech representations from raw audio by joint audiovisual self-supervision. 07 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
D. Snyder, D.
G. Sell, D. Povey, and S. Khudanpur. X-vectors: Robust dnn embed- dings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5329-5333, 2018. doi: 10.1 10.1109//ASSSP22088...66
David Snyder, Daniel Garcia-Romert and Daniel Povey. Time Delay Deep Neural Network- Based Universal Background Models for Speaker Recognition. 2015 JEEE Workshop on Au- lomatic Speech Recognition and Understanding (ASRU), pp. 92-97, 2015. doi: 10.1 109/asuu
2015.7404779. URL htttt:://pppppp dimensi a /eee/eeelllleeeeee httlss/publllie cublication/pal pub.
1093368586
Xingchen Song, Guangsen Wang, Zhiyong Wu, Yiheng Huang, Dan Su, Dong Yu, and Helen Meng. Speech-xInet: Unsupervised acoustic model pretraining for self-attention networks, 2020.
Sören Sonnenburg, Gunnar Rätsch, Christin Schäfer, and Bernhard Schölkopf. Large scale multiple kernel learning. J. Mach. Learn. Res., 7:1531-1565, December 2006. ISSN 1532-4435.
Johan Sundberg and Maria Nordenberg. Effects of vocal loudness variation on spectrum balance as reflected by the alpha measure long-term-average spectra of speech. The Journal of the Acoustical Society of America, 120:453-7, 08 2006. doi: 10. 10.1122//.220845.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What Makes for Good Views for Contrastive Learning? In Larochelle, Ranzato, R Hadsell, M Balcan, and H Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6838 Curran Associates, Inc., 2020. URL htttps::/proceeeeeee neurips.......p........ cc/ paper 2020 / 4c2e 91 5207 9
M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic. On mutual information max- imization for representation learning. In 8th International Conference on Learning Representa- fions (ICLR), April 2020. htthttts::/ httphtttts:///eeeeee http:///wreeeeeeeeeeeeooooooo/oa
14

---

Under review as a conference paper at ICLR 2022
Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, and Joseph E. Gonzalez. Tafe-net: Task-aware feature embeddings for low shot learning. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition (CVPR), June 2019.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks unlabeled data. CoRR, abs/2010.03622, 2020. BtRphhtts:///xxiiin
abs 2210/2010..m 03622
Ho-Hsiang Wu, Chich-Chi Kao, Qingming Tang. Ming Sun, Brian McFee, Juan Pablo Bello, and Chao Wang. Multi-task self-suuprvised pre-training for music classification, 2021.
Shuwen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-War Wen Li, Shinji Watanabe, AbdeIrah- man Mohamed, and Hung yi Lee. Superb: Speech processing universal performance benchmark, 2021.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B, 68:49-67, 02 2006. doi: 14171/1,667-8888.220000 00532.x.
Amir Roshan Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. CoRR, abs/ 0bbs//804.08328, 2018.
URL httthttp///axiiio http://///gabba/////// 08328
Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang. Quoe V. Le, and Yonghui Wu. Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition. oct 2020.
A APPENDIX
A. DIFERENTIABILITY PROOF
We want to show that the utility estimate differentiable with respect the weighting parameters (A))ejo, Since the final estimate is weighted mean of the in-class independent tests, the problem boils down to showing that within a downstream class HSIC,(X, Zx) is differentiable. Let us recall the definition of the considered quantities:
HSIC.(X,Z2) trace(K.HHLLLt
(10)
n
where Ke and He are independent of À and L. coefficients are defined as:
Lolks = ROF--BBF(BBH.... (Zx), exp(
An Anllan, -2n,l13)
(1I)
20
Therefore for E [0,k]
ØHSIC. Zx)
Ottrace Corrae(K,H,oLollo, LLLois
OAp
Ohp
1
(12)
11
zpoll2 [Lelis
! KJ H )ij
2
202
This allowed us to minimize the conditional -nndeendeneeeeeeeeeeeeee based utility estimator according to the weighting values.
15

---

Under review as a conference paper at ICLR 2022
Table 3: Candidate speech pseudo-labels and descriptions. Description
Feature
Loudness
Intensity & approx. loudness
FO
Fundamental Frequency
Voicing
Voicing Decision
Alpha Ratio (Sundberg & Nordenberg 2006
Ratio of spectrum intensity % 1000 Hz
Zero Crossing Rate
Zero crossing number per frame
RastaSpec Norr
L1 Norm of Rasta Spectrum (Hermansky et al. log of Harmonicity to Noise Ratio
1992)
log HNR (Murphy & Akande 2005
Evolution
Volcing Evolution
logHNR- Evolution
0.082
0.0245
0.0240
0.900
0.078
Valluss
0.076
valuus
0.885
values
0.0290
0.074
0.0225
0.072
0.885
0.0220
0.070
0.880
0.068
100
200400 6001000
20
100 200
400 800,000
200 4.000000000 B00L000
speakers
Nispaabers
Figure 2: Evolution of the CI estimation with different numbers of considered speakers for three pretext tasks: FO, Voicing and logHNR. We can that the values obtained with 20 speakers, while logically exhibiting more variance, are already close to the final values for every pretext task.
A.2 CONSIDERED SIGNAL FEATURES AND DESCRIPTIONS
Table contains the descriptions of the signal features used as labels in this work.
A.3 EFFECTS OF SAMPLING ON THE CONDITIONAL INDEPENDENCE ESTIMATION
Two limitations related to the size of the downstream dataset may be faced using our technique. First, very small downstream datasets could not be sufficient for a good estimate of the conditional independence. Second, very large downstream datasets may render the CI estimation intractable. This section shows experimentally on VoxCelebl that our technique is robust to these two situations. First, we show by taking small subsets of VoxCeleb1, that in case of downstream data scarcity, the CI estimations obtained with our method are close to the final estimations, and the ranking of the pretext tasks is not altered even when we take only 100 speakers among the 1 251 in VC. Second, as one of the main motivations of this work is the reduction of the computation needed to get the best selection of pretext-tasks in self-supervised learning settings, we show that the CI estimation converges quickly with a growing number of speakers considered, and is thus resilient to sampling. Considering one pretext task at time, we consider subsets of VoxCelebl using growing number of considered speakers (total = 1 251). For each of these considered numbers, we run 10 experiments with sampled speakers. We get the CI estimation for every subset and plot the boxplot of the obtained values. Results are shown in Fig. 2 We can see that using only 20 speakers showcases results that are already close to those with 000 speakers. Furthermore, we plot the boxplots of CI values obtained using more than 200 speakers to show the separability between the considered features in Fig. 3 While values for Voicing and Loudness are slightly overlapping, all the other pretext tasks already separated and rankable using only 200 random speakers among the whole dataset.
16

---

Under review as a conference paper at ICLR 2022
Voxceleb- Values distribution per pretext task
Voxceleb- Values distribution per pretext task
0.80
0.06
0.96
valuus
valuues
0.84
0.04
AI Ratlo
Log HNR
ZGR
Loudness
Voicing
Rasta
Fea tures
features
Figure 3: Boxplots of the CI values for every pretext tasks, when more than 200 speakers are con- sidered. Voicing and Loudness are slightly overlapping, but otherwise, the values are separable. We divide the pretext-tasks in two groups according to their CI values for better visualisation of the results.
A.4 SPARSEMAX INITIALIZATION
When initialized with random parameters W, and if one parameter is high enough compared to the other, leading with the Sparsemax function to a weighting value close to 1, we observed that the minimization process falls into local minima selecting only one pscudo-label with weight 1. To avoid this, we initialize all the free parameters W with the same unitary value to which we add some Gaussian noise. Hence, Wint (1) c) with = 0.05.
A.5 TRAINING AND ARCHITECTURES DETAILS
All the considered audio files are sampled at 16kHz. We feed the SSL models with 80-band Mel spectrograms, with 25ms windows and 10ms stride. To every Mel band corresponds a learned vector of size 256 obtained at the output of the SSL model. So if the input spectrogram is of size (N, 80) with N the number of frames, the representation fed to the downstream pipeline is of size (N, 256). All models including SSL and downstream ones developed with SpeechBrain (Ravanelli al.)
2021
A.5.1 PRETRAINING OF THE SSL ENCODER.
The encoder is a succession of 2D CNN layers, LSTM layers and a final dense network. This representation then fed to one dense layer that predict the selected pretext task labels. There are successive CNN blocks containing each 2 CNN layers with kereel size (3, 3) and 128, 200 and 256 channels for each block respectively. No time pooling performed in order to preserve the input sequence length. 5 bidirectional LSTM layers of size 256 are then stacked. Finally, MLP with one hidden layer with 256 neurons. The LeakyReLU activation is used across all the layers except for the L.TTM. We use a dropout rate of 0.15 during the training. The AdaDelta optimizer is used to update the weights with an initial learning rate of 1.0, p = 0.8 and 10-8 For every experiment, the SSL model is trained for 10 epochs Icading to the convergence of the validation loss).
A.5.2 DOWNSTREAM TRAININGS FIRST EXPERIMENTS
Speaker recognition details. VoxCelebl NNgrann al. 2017) is used for the speaker recognition
task. The training set contains 148, 642 utterances from 1. 251 different speakers. To compute the conditional independence estimates while limiting the computational load, restricted ourselves to the utterances of 50 different speakers (the detailed list is given in the released repository). A standard xvector model (Snyder al. 2018) is trained following the available VoxCeleb Speech-
17

---

Under review as a conference paper at ICLR 2022
Brain recipe. The extracted speaker embeddings are tested on the enrol and test splits using PLDA (lofte, 2006) as a similarity metric. Performance is reported in terms of equal error rate (EER). While architecture details are given in appendix A.5 it is worth noticing that the whole pipeline is fully integrated to Speechbrain and can thus easily y be extended.
We train an embedding model (XVector) until the validation loss converges, top of the self pervised representations using 5 successive layers of time-delay neural networks (TDNN) (Peddinti al. 2015). The number of channels is (512, 512, 512, 512, 1500), with kernel sizes of (5, 3, 1) and dilations of (1, 2, 3, 1, 1). The architecture is inspired by successful works on embeddings for speaker recognition (Snyder
al.) 2015). The learned embeddings are therefore used on a list of pairs of samples to predict whether they are from the same speaker or not. The details of the recipe can be found the given GitHub repository. We train every embedding model on 10 epochs with an Adam Optimizer starting with learning rate of 0.001 decaying linearly to 0.0001.
Speech recognition details. ASR is conducted with the 100-hour clean subset of the LibriSpeech dataset (Panayotov et al., 2015) to simulate the low-resource scenario commonly encountered with SSL settings. CI estimations are obtained with word-level alignments from the Montreal Forced Aligner (McAuliffe et al. 2017). The ASR pipeline follows the LibriSpeech recipe of SpeechBrain (Ravanelli et al.) 2021) and therefore contains a CRDNN encoder (i.e. CNN, RNN, DNN) trained jointly with CTC (Graves 2012) and attention (Lüscher et al.] 2019) (details in appendix [A.5) The decoding process is based on beam-search with and without shallow fusion with a pretrained recurrent language model that is publicly available and obtained from SpeechBrain Performance is expressed in word error rate (WER).
The CRDNN starts with three CNN blocks composed each with 2 2D CNN layers, layer- normalisation and (2 maxpooling along the frequency dimension. The filter dimensions for each block are 64, 100, 100. Then, maxpooling of 4 is applied on the time dimension to reduce the se- quence length before being fed to the RNN. The latter is made of 5 bidirectional LSTM layers of 1, 024 neurons. Finally two dense layers are connected (with LeakyReL.U activation function is used across all the layers except for the LSTM. A dropout rate of 0.15 is employed with the encoder. The CTC decoder a simple dense linear layer of size equal to the vocabulary. The vocabulary is obtained with byte pair encoding or sub-words units (BPE) and is of size 1, 000. The attentional decoder is a one-layered location-aware GRU (1, 024 neurons). Then, beam search of depth 60 is applied to obtain the output transcripts. The model is trained for 30 epochs. The learning rate (1.0) is multiplied with a factor of 0.8 every time the validation loss is not decreasing ensure an optimal convergence of all the models.
between). The
A.5.3 SUPERB SETTINGS
SUPERB (Yang et al. 2021 is a recent benchmark for self-supervised representations of speech data. We use this benchmark for our experiments in combining wav2vec with our selected pretext tasks. We detail here the downstream models as detailed in the benchmark paper:
Emotion Recognition. IEMOCAP (Busso al., 2008) is used for the Emotion Recognition (ER) task. 4 classes are considered (neutral, happy, sad, angry), and only the audio data is used. The learned representations are mean- proand then fed a final linear classifier to compute a cross- entropy loss. We cross-validate on five folds of the standard splits. The result shown is the average of the five attempts. The evaluation metric is accuracy (ACC).
Automatic Speech Recognition For ASR, the decoder is vanilla 2-layer 1024-unit BLSTM fed with our self supervisdd representations and optimized by CTC loss on characters. We use the same language model for decoding as in the first experiments. LibriSpeech C0e only is used for downstream training.
Speaker Recognition The model and the dataset splits used in the first experiment correspond to the SUPERB ones, so we kept the same settings. The results are therefore comparable.
1itrr speech
18

---

Under review as a conference paper at ICLR 2022
A.6 INTUITION AROUND THE USE OF CONDITIONAL INDEPENDENCE
To get an intuitive understanding of the motivations of this choice, let us consider the example of image classification as the downstream task, and image colourization as the pretext task. In this case, this pretext task would be suited to the downstream one if the final classification label can help implying the colours. For instance, if there are only two classes "Blue skies" and "Yellow deserts", then colourisation is an interesting pretext task, as knowing the final label helps lot for the pretext task, independently of the image. However, if all the classes share the same colour palette, colourization may not be an interesting task. (In this simple example, we are ignoring the edge detection aspect of colourization, and only focusing on the colour choice part. Obviously the former aspect plays a big part on why the colourization pretext task has been successful.)
Concerning our estimation method, as the pseudo-labels considered in this work are data features, they are indeed functions of the original data samples. This ensures that the data samples are not independent of the pscudo-labels. The idea behind the estimator of conditional independence is that it will test whether this remains true when the considered points share the same downstream class.
A.7 KERNELS USED FOR THE SIMILARITY MATRICES
The computation of the similarity matrices used in our kernel-based independence test, requires fixed-size embeddings for the data speech samples. These embeddings allow the use of classic ker- nels on top. However, in the case of sequential data, as it is the case with audio/speech signals, one may want avoid the additional burden of learning fixed-size embeddings (for possibly variable- length audio sequences). One possible solution to this, which we conveniently exploited in our application to speech data (see Section 5 is the Gaussian Downsampling method (Holzenberger et al., 2018) detailed thereafter. In this instance, after the Mel spectrogram extraction, a speech sam- ple is a sequence of varying length input feature vectors. Therefore, to obtain fixed size embeddings aggregating the input frame-wise Mel spectrum vectors into a fixed number N of input vectors, N being a fixed typer-parameter we first divide the sequence into N equal length segments. Then, in each segment, Gaussian average of the input spectra is computed around the center of the consid- ered segment with the standard deviation Agd being another hyper-parameter. Denoting by D the dimension of the input frame-wist Mel spectrum vectors, this leads, for any speech excerpt, N x D tensor, without any training procedure. As in the work presenting the gaussian downsam- pling method (Holzenberger et al. 2018), we set N = 20 and Ogd = 0.07. For the RBF kernel on the pseudo- labels mean value per file, we fixed he RBF kernel width to = 0.05.
We will know motivate the gaussian downsampling choice. At high-level, we were looking for a method giving fixed size speech embeddings that would not rely on any learning. As we were first working with ASR and Speaker Id, we wanted a representation that captures phonetic and speaker content. The GD has been, quite successfully, applied for unsupervised word discovery (Holzen- berger et al. 2018), thus capturing this phonetic aspect. In unpublished experiments when working on unsupervised work discovery, we also found out that GD holds speaker-related information, this made GD good candidate for our unlearned embedding method. Moreover, after the first another downsampling method 2017 We calculated the CI estimates using GD and SVCCA on the considered pretext tasks for LibriSpeech and Voxceleb1. It resulted in relative difference of 3% between the averaged HSIC. The robustness to this downsampling method change comforted us into using GD.
results,
tested in the beginning of this work SVCCA (Raghu et al.
A.8 IMPACT OF THE PRETRAINING DATASET ON SPEAKER RECOGNITION
We developed in section 5.2 the reasons behind our choice of Common Voice as a pre-training dataset. In this section, we study the impact of this choice on the results. Specifically, it is common in the speech SSL literature to train on LibriSpeech 960 before fine-tuning on LibriSpeech100. As explained before, we believe that this introduces bias due to the closeness of pretraining and fine- tuning data. To verify this, we train our best multitask BASE wav2vee 2.0 architecture with the best performing pretext tasks and their weights on LibriSpeech 960. The model follows the exact same training procedure as for Table 2 We fine-tune the models on LibriSpeech 100 exactly as it has been done with the other models. Table 4 shows the results. Two observations deserve to be noted. First, in this case also, adding a selected set of pretext tasks improves the final downstream performance in the frozen and finetuned cases. Second, as expected, the results obtained after training on Lib-
19

---

Under review as a conference paper at ICLR 2022
Table 4: Results observed retraining the Wav2vec2 model with and without weighted pretext tasks using the sparsemax method. on LibriSpeech 960. "Fr." and "Fine." also respectively refer to Frozen and Finetuned settings. Adding selected pretext tasks still improves the downstream performance. All models contain 100M neural parameters.
Selections
LibriSpeech (WER % +)
Fr.
Fine.
wav2vec 2.0 BASE
9.88
6.33
wav2vec 2.0 BASE + multitask SSL
9.5
6.01
rispeech960 are better than those with Common Voice, reaching the lowest 6.01% of WER with the fine-tuned version compared to 9.18% of WER in table 2
A.9 LINKS WITH FEATURE SELECTION
We also studied the link between classic feature selection and pretext task selection through two periments. The first one was made to check how hard was to estimate the utility of a pseudo-label, computed the mutual information between the pseudo-labels and the downstream labels, and checked how much it would correlate with downstream performance. It led to very low correlation values, with even changing signs between VoxCeleb and LibriSpeech. This seems indicate that Mutual Information is not related directly to self- supessiueriiionn utility.
The maximum relevance minimum redundancy (MRMR) technique (Peng et al. 2005) used baseline in this work relies on the Conditional Independence based estimator. It is a close to a naive selection of the best pretext tasks according to the based criterion, but it furthermore penalizes the mutual information between the selected pretext tasks. More precisely, we select the group of pseudo-labels (Z), E [0.r] maximizing:
ScoreMRMn(Z) =
Σ ( xE(O.p]
Σ
T(Z, Zz) S
Recursive Feature Elimination (RFE) (Guyon al. 2002) relies classifier that provides infor- mation concerning the importance given feature in decision. This classifier first trained with the whole set of pscudo-labels as features, and the least important feature is eliminated. The process is repeated until only pscudo-labels are kept. We use the scikit-learn implementation with the C-Support Vector Classification the the classifier providing the feature importance values. We use the default seikit-learn hyperparameters.
These two baselines perform worse than the proposed techniques. This suggests that despite the apparent similarity, feature selection and self-supervision pretext task design do not necessarily involve the same mechanisms.
A.10
INTERACTIONS.
To understand the interactions between pseudo-labels, studying the evolution of the CI estimate as function of the weights shows which pscudo-labels seem interchangeable, which ones are comple- mentary and which ones seem only harmful to the considered downstream task. Figure the CI estimates for weighted combinations of groups of three pseudo-labels. As the weights sum up to one, two pretext tasks' values are shown on the æ and axes, while the value of the remaining one, whose name is in the title, is equal to 1- z y. For instance, at the origin point (0, 0), only the third pseudo-labe is selected with weight equal to one, while its weight is equal to zero on the hypotenuse of the right triangle. Figure AiguureFillusee iillustratts that the relationship leading utility estimator is not always straightforward For instance, if we consider the second plot on the second row (i.e. a-ratio, FO, logHNR),
lower Cl-based
that selecting only one element is always worse
20

---

Under review as a conference paper at ICLR 2022
CI evolution
0.7
Remainder:
Remainder:
Remainder: Loud
0.6
1.0
1.0
1.0
I0gHNR
Vording
0.5
0.5
logHNR
0.7
Remainder: o-ratio
Remainder:
Remainder:
1.0
1.0
1.0
0.6
Volding
a-ratto
logeNR
0.5
0.5
0.5
logHNR
logHNR
Figure CI-Based utility estimator as function of the weighting for groups of three pseudo-labels. Top line is for Librispeech, while the bottom one is for VoxCeleb. Three pseudo-labels are presented on every plot, one on the z-axis, one on the y-axis and one that equal called the remainder) and whose name is the title. Every point in the triangle corresponds to pretext task that the weighted combination of the three considered pseudo-labels. For instance, in the top left corner, the point (0.5, 0.3) correspond to the CI value of pretext task weighting logHNR with 0.5, a-ratio with 0.3 and FO with 0.2.
x y (hence being
Table 5: Weights for every pretext-tasks in every considered experiment. With techniques only leading to selection of pretext tasks (without weights) unitary weight is assigned for the selected tasks and zero for the non selected. We can see in this table the zeros induced by the Sparsemax function.
Selection
a-zero
FO
Loudness
Audspec Rasta
ZCR
log HNR
Voicing
All
1
1
1
VC RFE
1
1
1
VC MRMR
1
0
0
0
1
VC Sparsemax
0.28
0,26
0
0.4544
VC Softmax
0.27
0.11
0.18
0.04
0.06
0.31
0.03
Libri RFE
1
0
O
1
Libri MRMR
0
1
0
Libri Sparsemax
0.30
0.37
0
0.06
0
0.27
Libri Softmax
0.28
0.47
0.07
0.04
0.02
0.08
0.04
IEMOCAP RFE
0
0
I
1
IEMOCAP MRMR
1
IEMOCAP Sparse.
0.16
0.22
0
0.14
0.12
0.17
0.19
IEMOCAP Softmax
0.29
0.32
0.06
0.24
0.03
0.02
0.03
than selecting a weighted concatenation, because the areas around the origin and the points (1,0) and (0, are brighter than the central area.
A.11 FULL WEIGHTS FOR THE CONSIDERED EXPERIMENTS
Table 5 shows the weights computed using the proposed selection and weighting process for every considered downstream task. also shows the pretext tasks selected by the baseline methods.
21


Reviews:
[{"review_id": "ioziMXkwZSk", "reviewer": "Reviewer_Q1kK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the proposed method for selecting pretext tasks is interesting \n- using this approach the authors show that self-supervised learning could benefit the fine-tuning for the downstream task\n\nWeaknesses: \n- evaluation for the claims are very weak. Authors show, for example, that using a subset of pretext tasks benefits the accuracy of ASR model. However, the experiments are limited to two downstream tasks. Also, results do not show the impact of picking other subsets which are not strongly related to the downstream task.\n- could this approach be applied to other domains? or is it just limited to speech? \n- comparison with wav2vec-2.0 is weak, can this approach beat the SOTA? \n- are the learned features generable? or they fail when the downstream task is changed? \n- how much data is required to learn the weights with HSIC? \n\n", "summary_of_the_paper": "This paper proposes a method to select a group of pretext tasks from a given set of tasks for optimising the network training during self-supervised learning phase.\nThe weights for the given set of tasks are learned based on Hilbert Schmidt Independence Criterion (HSIC) using a few data samples. Using 2 downstream tasks, the authors show that their approach could benefit the learning of features relevant for the downstream tasks. Thus, improves the accuracy of these downstream tasks.", "summary_of_the_review": "Overall, the approach is interesting and could benefit the training during self-supervised learning phase. However, the evaluation requires a significant amount of ablation studies to prove the claims about the effectiveness of this approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RC0YOYH-KcM", "reviewer": "Reviewer_zWkh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) This topic is well timed. Currently, contrastive losses are the best performing, but having SSL targets from a multi-task scenario is also promising. It would be incredibly useful to the community if the weighting of many SSL targets was important, and one could be found to make multi-task training paradigms competitive.\n2) The theoretical result seems interesting. I have not seen Conditional Independence be used as a criteria for weighting SSL targets. If useful, this would be a new way of thinking about tasks in a multi-task training scheme.\n\nWeaknesses:\n1) Empirical results: \n1a) I worry about the comparison in Table 2. To support the claim that the authors’ algorithm is useful, they need to show that their algorithm also improves on wav2vec + naive mixing of multitask SSL. Table #2, as it currently stands, only shows that additional targets can be useful. It does not seem to support the argument that their mixing algorithm was essential to the improvement.\n1b) I worry about the presence of IEMOCAP in experiment #2 but its absence in experiment #1. If the algorithm is as general as is claimed in the paper, I would expect feature selection improvements on IEMOCAP as well, and I would expect to see this dataset in table 1.\n\n2) Unclear algorithm choices: The choice of the Gaussian Downsampling function is essential to computing the CI between X and Z given Y, but is unjustified in the text. Why take a Gaussian average with the hyper parameters used? These seem like essential ingredients to calculating the data component of conditional independence, but they are neither justified theoretically or experimentally (for instance, by determining how sensitive the final Z weights are to these parameters)\n\nAdditional nits: \n1) There are some typos ex “( without weights )” -> “(without weights”\n2) \"The most successful models rely on predictive and contrastive objectives (Baevski et al., 2020; Chung et al., 2019; Saeed et al.,\n2020)...\" https://arxiv.org/abs/2110.04621 shows that \"Saeed et al., 2020\" is not better than other 2020 methods, and in many cases worse. It might be better to be more descriptive than \"successful,\" which doesn't seem strictly true.", "summary_of_the_paper": "This paper attempts to improve self-supervised (SSL) speech representations by determining what linear combination of SSL targets will perform best for downstream tasks (audio SSL targets like fundamental frequency or MFCC). The technique learns to weight these targets according to a Conditional Independence criteria: a target is more valuable if the SSL target is more independent of the data given a label. \n\nThere are two main experimental results on which the usefulness of this work rests. The first is a “sanity check”, which demonstrates that this weighting scheme outperforms two previous ones, from 2002 and 2005, on two tasks: ASR in LibriSpeech and speaker ID in Voxceleb1.  The second is that they manage to improve LibriSpeech, Voxceleb, and IEMOCAP performance over a wav2vec2 model by adding SSL targets weighted according to their algorithm.", "summary_of_the_review": "The work is a very interesting idea in a relevant subdomain of SSL research. The idea of using Conditional Independence to rate SSL targets, and the technique for calculating CI, is novel. However, the empirical results do not support the usefulness of this method (experiment #2) or are not compared to strong baselines (experiment #1). Without stronger empirical results to support the usefulness or relevance of this method, I do not believe this paper is strong enough for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "88v4otCNY4g", "reviewer": "Reviewer_ybV2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a new approach to combine multiple tasks during pre-training for speech representation learning, and it leads to improved performance for downstream tasks including speech, speaker and emotion recognition. Overall, this paper is clearly written. \n\nOne question I think this paper needs to address is how strong baselines are. Though authors mention not to directly compare with downstream tasks' performance reported in literature, it still helps to provide some justification to show the numbers reported in Table 1 & 2 are improvements over strong baselines. Also authors should add error bar to show if difference is statistically significant.", "summary_of_the_paper": "This paper describes authors' proposal for selecting pretext tasks (pseudo-labels) for multitask self-supervised learning, to improve self-supervised learning for speech representations. It leveraged the Hilbert Schmidt Independence Criterion (HSIC) for selecting pseudo-label loss weights and used softmax and its sparsity version to realize it. Experimental results on speech, speaker and emotion recognition showed effectiveness of the proposed approach.", "summary_of_the_review": "The proposed method is technically sound, though additional justification is needed to for experimental results reported in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for selecting a group of pretext tasks out of a set of candidates in order to optimize self training for downstream performance. The method relies on Hilbert Schmidt Independence Criterion (HSIC) and uses a few data samples to select weights for the given set of tasks The paper demonstrates that using the method for task selection can result in learning better representation for downstream tasks improving accuracy on speech, speaker and emotion recognition.

The reviewers had concerns mostly about the strength of the empirical results. In particular, they felt that the baselines are not strong enough. To the authors credit, the paper was augmented with some of the missing experiments that the reviewers pointed out (e.g., wav2vec plus naive task selection), but that did not persuade reviewers to change their recommendations.

The paper still misses the point that self-supervised learning approaches can benefit from training larger models that result in better results. These comparisons are missing from the paper. It is established in other work that findings such as the use of pretext tasks often do not carry over to larger scales. Furthermore, the idea of pretraining a model specific to a downstream task is not inline of the philosophy of self-supervised training that aims to train a single model that can be used for many different tasks.
Decision: Reject