Pretext Tasks Selection for Multitask Self-Supervised Speech Representation Learning

Through solving pretext tasks , self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task . In audio/speech signal processing , a wide range of features where engineered through decades of research efforts . As it turns out , learning to predict such features ( a.k.a pseudo-labels ) has proven to be a particularly relevant pretext task , leading to useful self-supervised representations which prove to be effective for downstream tasks . However , methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly . In fact , the process relies almost exclusively on a computationally heavy experimental procedure , which becomes intractable with the increase of the number of pretext tasks . This paper introduces a method to select a group of pretext tasks among a set of candidates . The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process . The experiments conducted on automatic speech recognition , speaker and emotion recognition validate our approach , as the groups selected and weighted with our method perform better than classic baselines , thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning . 1 INTRODUCTION . Self-supervised learning ( SSL ) methods usually rely on a supervision obtained from the data itself through solving specific pretext tasks leveraging the underlying structure of the considered data ( Doersch et al. , 2016 ; Arandjelovic & Zisserman , 2018 ) . This technique is used in various domains including image processing ( Misra & Maaten , 2020 ; Jing & Tian , 2020 ; Grill et al. , 2020 ) , natural language understanding ( Chen et al. , 2020b ; Du et al. , 2020 ; Lan et al. , 2019 ) or speech and audio processing ( Baevski et al. , 2020b ; Liu et al. , 2020 ; Jiang et al. , 2020 ) . It offers numerous advantages , such as the independence from labeled data , stronger performance on downstream tasks , more robust models and an easier transfer to low-resource setups ( e.g. , low-resource languages ) ( Baevski et al. , 2020b ; Jing & Tian , 2020 ) . The numerous existing SSL approaches are characterized by the nature of the pretext tasks they solve . For instance , common techniques include predictive coding ( Baevski et al. , 2020b ; Liu et al. , 2020 ; Song et al. , 2020 ; Zhang et al. , 2020 ; Hsu et al. , 2021 ) , pseudo-label learning ( Pascual et al. , 2019 ; Ravanelli et al. , 2020 ) , auto-encoding ( Renshaw et al. , 2015 ; Algayres et al. , 2020 ) , tripletloss learning ( Shor et al. , 2020 ; Peplinski et al. , 2020 ) , generative modelling ( Khurana et al. , 2020 ) or contrastive learning ( Saeed et al. , 2020 ; Jiang et al. , 2020 ) . More precisely , these pretext tasks may be defined through the choice of pretext labels , hereafter referred to as pseudo-labels . The automatic extraction of pseudo-labels for SSL ( i.e . from the data itself ) is common in many application domains , such as computer vision ( Noroozi & Favaro , 2017 ; Gidaris et al. , 2018 ) , music processing ( Hung et al. , 2019 ; Wu et al. , 2021 ) and speech processing ( Pascual et al. , 2019 ; Shukla et al. , 2020 ) , and is commonly referred to as multitask self supervised learning . In the specific context of speech processing , the process of designing pseudo-labels may benefit from decades of research in signal processing . For instance , potential candidates are pitch estimators , energy-based features , voicing state and many more . As demonstrated by Pascual et al . ( 2019 ) , multitask speech representation learning is a powerful tool to build representations that are beneficial for a wide range of distinct downstream tasks , by combining different pseudo-labels which “ intuitively ” correspond to these tasks . Unfortunately , there is no clear understanding of how these pseudo-labels may interact when optimised together , and therefore , no common practice of how to select groups of pseudo-labels to obtain better performance on a known downstream task . As a matter of fact , this design process has been essentially driven by empirical validation and there is therefore no evidence that the obtained model is even the best one . This empirical approach can rapidly become intractable with modern SSL architectures which may contain hundreds of millions or billions of parameters trained on thousands of hours of speech , not to mention the carbon footprint of such pseudo-label searches . For instance , the self-supervised training of a single state-of-the-art large wav2vec 2.0 model ( Baevski et al. , 2020b ) on 53.2k hours of speech requires 128 GPUs for 5.2 days . This work aims at providing a clear , efficient and theoretically motivated procedure for pseudolabel group selection and weighting based on conditional independence ( CI ) . The method presented allows one to design ahead of training the most adapted multitask self-supervised speech representation learning model which perfectly suits the considered downstream tasks . Such an approach may also enable researchers to save a substantial amount of time and computation usually devoted to pseudo-label search . Hence , the contributions of this work are fourfold : 1 . Introduce a theoretically motivated and computationally efficient method for the selection of pseudo-label groups among a set of candidates and with respect to the considered downstream tasks ( Sections 3 and 4 ) . 2 . Validate empirically the proposed approach with a first model SSL model relying on different sets of pseudo-labels corresponding to the ones obtained for three considered speech tasks . ( Sections 5 ) . 3 . Extend our method to the SOTA wav2vec 2.0 to enhance its performance ( Section 6 ) . 4 . Release the code base developed with SpeechBrain ( Ravanelli et al. , 2021 ) for replication and to encourage further investigations.1 The conducted experiments demonstrate that the proposed method allows for a more intelligent , i.e . better informed , pseudo-label group selection for multitask SSL settings . Indeed , we find that the models built with the proposed method obtain a word error rate and an equal error rate , respectively , 31.6 % and 27.4 % lower than the baseline , without the need for any empirical search . 2 RELATED WORKS AND MOTIVATIONS . SSL recently became a key component to achieve good performance on downstream tasks especially with low-resource setups , either in speech ( Baevski et al. , 2020b ; Conneau et al. , 2020 ) , natural language processing ( Lan et al. , 2019 ; Chen et al. , 2020b ) or computer vision ( Gidaris et al. , 2019 ; Misra & Maaten , 2020 ; Jing & Tian , 2020 ) . Due to its very nature , SSL relies on large amounts of unlabeled data used to train large deep neural networks over long periods of time . It it thus crucial to understand properly what makes a good SSL model to lower the amount of computation and time needed to obtain the best downstream performance . SSL for Speech . Self-supervised learning for speech has recently enabled researchers to reach stateof-the-art results on various speech processing tasks ( Fan et al. , 2021 ) . The most successful models rely on predictive and contrastive objectives ( Baevski et al. , 2020b ; Chung et al. , 2019 ; Hsu et al. , 2021 ; Shor et al. , 2021 ) performing well across the different tasks even in low-resource settings . This led to the design of different benchmarks evaluating the self-supervised representations in different languages ( Yang et al. , 2021 ; Evain et al. , 2021 ) . However , in contrast to this proposition , these works have not tried to motivate beforehand the different choices made in the self-supervision pipeline . Understanding SSL . A few works have tried to shed some theoretical light on the mainly empirical field of self-supervised learning . Following the different paradigms in SSL , various tracks have 1https : //github.com/iclrsubmitter22/iclr_2022_submission been followed to understand what makes for a good self-supervised representation , exploring different approaches ( Lee et al. , 2020 ; Arora et al. , 2019 ; Wei et al. , 2020 ) . On the one hand , contrastive learning ( Oord et al. , 2018 ; Chen et al. , 2020a ) has been advocated both theoretically and empirically to achieve a balance in the mutual information between alternative representations of the data , keeping just enough shared information to keep the class-related content ( Tschannen et al. , 2020 ; Tian et al. , 2020 ; Bachman et al. , 2019 ) . In a recent work from Li et al . ( 2021 ) , independence testing has been used to produce better transformations in contrastive learning settings for image representations . Predictive learning , on the other hand , requires the model to predict masked elements in sequential data . This technique is powerful on downstream tasks that can be reduced to a masking problem , as suggested by research on language modeling ( Saunshi et al. , 2020 ) . However , all these works have been focusing on computer vision or text-related applications , and none of them addressed the multi-tasked self supervision problem . Multi-task self-supervised learning . While the literature on multi-tasking in self-supervised learning remains scarce , it has been shown in classic supervised learning settings , that through estimates of similarity between tasks or thorough empirical testing , several tasks can take advantage of being solved with a common encoder ( Zamir et al. , 2018 ; Dwivedi & Roig , 2019 ; Shafey et al. , 2019 ; Chen et al. , 2015 ) . More specifically , combining pretext tasks with SSL has been mainly explored in computer vision and speech ( Pascual et al. , 2019 ; Ravanelli et al. , 2020 ) . Pretext tasks such as Jigsaw ( Doersch et al. , 2016 ) , colourisation and rotation ( Gidaris et al. , 2018 ) have been combined successfully to improve downstream performance ( Kim et al. , 2018 ; Shin ’ ya Yamaguchi et al. ) . The two closest works to our line of research are from Lee et al . ( 2020 ) and Doersch et al . ( 2017 ) . The former shows that a theoretical link can be established between conditional independence and an improvement of the performance on the downstream task , while the latter proposes to select layers from a multitask self-supervised encoder according to the pretext task to be solved . However , in both cases , the studies do not offer practical and theoretical solutions to select groups of pseudo-labels to build an adapted SSL model that will perform well on the considered downstream tasks . Group feature selection . Finally , feature selection , and especially group feature selection is another close and inspiring field given the problem we consider . The relationship and interactions between features have been largely investigated in the supervised learning literature ( Guyon & Elisseeff , 2003 ) . This led to multiple solutions to the feature group selection problem , including LASSObased techniques ( Yuan & Lin , 2006 ) , or multiple kernel formulations ( Sonnenburg et al. , 2006 ; Rakotomamonjy et al. , 2007 ) . However , these works do not involve any self-supervision , and links between feature selection and self-supervision design and pretext-task selection are yet to be proved . We will further consider these lines of works for concurrent baselines . With this work , we aim at shortening the process of designing SSL models while giving insights on the pseudo-label importance and the underlying mechanisms between pretext and downstream tasks at the same time . We decided to experiment with speech due to the lack of literature on this domain for multitask SSL , and for the various pseudo-labels available , which are based on decades of signal processing research . The whole pipeline starting from the acoustic feature extraction to the downstream task scoring follows three major steps summarized in Figure 1 . First , for every downstream task , our method produces a pretext task selection and weighting . Then , a SSL model is trained , before being used as a feature extractor front-end to one or many downstream tasks .

Summary: This paper attempts to improve self-supervised (SSL) speech representations by determining what linear combination of SSL targets will perform best for downstream tasks (audio SSL targets like fundamental frequency or MFCC). The technique learns to weight these targets according to a Conditional Independence criteria: a target is more valuable if the SSL target is more independent of the data given a label.   There are two main experimental results on which the usefulness of this work rests. The first is a “sanity check”, which demonstrates that this weighting scheme outperforms two previous ones, from 2002 and 2005, on two tasks: ASR in LibriSpeech and speaker ID in Voxceleb1.  The second is that they manage to improve LibriSpeech, Voxceleb, and IEMOCAP performance over a wav2vec2 model by adding SSL targets weighted according to their algorithm.