Pretext Tasks Selection for Multitask Self-Supervised Speech Representation Learning

Through solving pretext tasks , self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task . In audio/speech signal processing , a wide range of features where engineered through decades of research efforts . As it turns out , learning to predict such features ( a.k.a pseudo-labels ) has proven to be a particularly relevant pretext task , leading to useful self-supervised representations which prove to be effective for downstream tasks . However , methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly . In fact , the process relies almost exclusively on a computationally heavy experimental procedure , which becomes intractable with the increase of the number of pretext tasks . This paper introduces a method to select a group of pretext tasks among a set of candidates . The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process . The experiments conducted on automatic speech recognition , speaker and emotion recognition validate our approach , as the groups selected and weighted with our method perform better than classic baselines , thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning . 1 INTRODUCTION . Self-supervised learning ( SSL ) methods usually rely on a supervision obtained from the data itself through solving specific pretext tasks leveraging the underlying structure of the considered data ( Doersch et al. , 2016 ; Arandjelovic & Zisserman , 2018 ) . This technique is used in various domains including image processing ( Misra & Maaten , 2020 ; Jing & Tian , 2020 ; Grill et al. , 2020 ) , natural language understanding ( Chen et al. , 2020b ; Du et al. , 2020 ; Lan et al. , 2019 ) or speech and audio processing ( Baevski et al. , 2020b ; Liu et al. , 2020 ; Jiang et al. , 2020 ) . It offers numerous advantages , such as the independence from labeled data , stronger performance on downstream tasks , more robust models and an easier transfer to low-resource setups ( e.g. , low-resource languages ) ( Baevski et al. , 2020b ; Jing & Tian , 2020 ) . The numerous existing SSL approaches are characterized by the nature of the pretext tasks they solve . For instance , common techniques include predictive coding ( Baevski et al. , 2020b ; Liu et al. , 2020 ; Song et al. , 2020 ; Zhang et al. , 2020 ; Hsu et al. , 2021 ) , pseudo-label learning ( Pascual et al. , 2019 ; Ravanelli et al. , 2020 ) , auto-encoding ( Renshaw et al. , 2015 ; Algayres et al. , 2020 ) , tripletloss learning ( Shor et al. , 2020 ; Peplinski et al. , 2020 ) , generative modelling ( Khurana et al. , 2020 ) or contrastive learning ( Saeed et al. , 2020 ; Jiang et al. , 2020 ) . More precisely , these pretext tasks may be defined through the choice of pretext labels , hereafter referred to as pseudo-labels . The automatic extraction of pseudo-labels for SSL ( i.e . from the data itself ) is common in many application domains , such as computer vision ( Noroozi & Favaro , 2017 ; Gidaris et al. , 2018 ) , music processing ( Hung et al. , 2019 ; Wu et al. , 2021 ) and speech processing ( Pascual et al. , 2019 ; Shukla et al. , 2020 ) , and is commonly referred to as multitask self supervised learning . In the specific context of speech processing , the process of designing pseudo-labels may benefit from decades of research in signal processing . For instance , potential candidates are pitch estimators , energy-based features , voicing state and many more . As demonstrated by Pascual et al . ( 2019 ) , multitask speech representation learning is a powerful tool to build representations that are beneficial for a wide range of distinct downstream tasks , by combining different pseudo-labels which “ intuitively ” correspond to these tasks . Unfortunately , there is no clear understanding of how these pseudo-labels may interact when optimised together , and therefore , no common practice of how to select groups of pseudo-labels to obtain better performance on a known downstream task . As a matter of fact , this design process has been essentially driven by empirical validation and there is therefore no evidence that the obtained model is even the best one . This empirical approach can rapidly become intractable with modern SSL architectures which may contain hundreds of millions or billions of parameters trained on thousands of hours of speech , not to mention the carbon footprint of such pseudo-label searches . For instance , the self-supervised training of a single state-of-the-art large wav2vec 2.0 model ( Baevski et al. , 2020b ) on 53.2k hours of speech requires 128 GPUs for 5.2 days . This work aims at providing a clear , efficient and theoretically motivated procedure for pseudolabel group selection and weighting based on conditional independence ( CI ) . The method presented allows one to design ahead of training the most adapted multitask self-supervised speech representation learning model which perfectly suits the considered downstream tasks . Such an approach may also enable researchers to save a substantial amount of time and computation usually devoted to pseudo-label search . Hence , the contributions of this work are fourfold : 1 . Introduce a theoretically motivated and computationally efficient method for the selection of pseudo-label groups among a set of candidates and with respect to the considered downstream tasks ( Sections 3 and 4 ) . 2 . Validate empirically the proposed approach with a first model SSL model relying on different sets of pseudo-labels corresponding to the ones obtained for three considered speech tasks . ( Sections 5 ) . 3 . Extend our method to the SOTA wav2vec 2.0 to enhance its performance ( Section 6 ) . 4 . Release the code base developed with SpeechBrain ( Ravanelli et al. , 2021 ) for replication and to encourage further investigations.1 The conducted experiments demonstrate that the proposed method allows for a more intelligent , i.e . better informed , pseudo-label group selection for multitask SSL settings . Indeed , we find that the models built with the proposed method obtain a word error rate and an equal error rate , respectively , 31.6 % and 27.4 % lower than the baseline , without the need for any empirical search . 2 RELATED WORKS AND MOTIVATIONS . SSL recently became a key component to achieve good performance on downstream tasks especially with low-resource setups , either in speech ( Baevski et al. , 2020b ; Conneau et al. , 2020 ) , natural language processing ( Lan et al. , 2019 ; Chen et al. , 2020b ) or computer vision ( Gidaris et al. , 2019 ; Misra & Maaten , 2020 ; Jing & Tian , 2020 ) . Due to its very nature , SSL relies on large amounts of unlabeled data used to train large deep neural networks over long periods of time . It it thus crucial to understand properly what makes a good SSL model to lower the amount of computation and time needed to obtain the best downstream performance . SSL for Speech . Self-supervised learning for speech has recently enabled researchers to reach stateof-the-art results on various speech processing tasks ( Fan et al. , 2021 ) . The most successful models rely on predictive and contrastive objectives ( Baevski et al. , 2020b ; Chung et al. , 2019 ; Hsu et al. , 2021 ; Shor et al. , 2021 ) performing well across the different tasks even in low-resource settings . This led to the design of different benchmarks evaluating the self-supervised representations in different languages ( Yang et al. , 2021 ; Evain et al. , 2021 ) . However , in contrast to this proposition , these works have not tried to motivate beforehand the different choices made in the self-supervision pipeline . Understanding SSL . A few works have tried to shed some theoretical light on the mainly empirical field of self-supervised learning . Following the different paradigms in SSL , various tracks have 1https : //github.com/iclrsubmitter22/iclr_2022_submission been followed to understand what makes for a good self-supervised representation , exploring different approaches ( Lee et al. , 2020 ; Arora et al. , 2019 ; Wei et al. , 2020 ) . On the one hand , contrastive learning ( Oord et al. , 2018 ; Chen et al. , 2020a ) has been advocated both theoretically and empirically to achieve a balance in the mutual information between alternative representations of the data , keeping just enough shared information to keep the class-related content ( Tschannen et al. , 2020 ; Tian et al. , 2020 ; Bachman et al. , 2019 ) . In a recent work from Li et al . ( 2021 ) , independence testing has been used to produce better transformations in contrastive learning settings for image representations . Predictive learning , on the other hand , requires the model to predict masked elements in sequential data . This technique is powerful on downstream tasks that can be reduced to a masking problem , as suggested by research on language modeling ( Saunshi et al. , 2020 ) . However , all these works have been focusing on computer vision or text-related applications , and none of them addressed the multi-tasked self supervision problem . Multi-task self-supervised learning . While the literature on multi-tasking in self-supervised learning remains scarce , it has been shown in classic supervised learning settings , that through estimates of similarity between tasks or thorough empirical testing , several tasks can take advantage of being solved with a common encoder ( Zamir et al. , 2018 ; Dwivedi & Roig , 2019 ; Shafey et al. , 2019 ; Chen et al. , 2015 ) . More specifically , combining pretext tasks with SSL has been mainly explored in computer vision and speech ( Pascual et al. , 2019 ; Ravanelli et al. , 2020 ) . Pretext tasks such as Jigsaw ( Doersch et al. , 2016 ) , colourisation and rotation ( Gidaris et al. , 2018 ) have been combined successfully to improve downstream performance ( Kim et al. , 2018 ; Shin ’ ya Yamaguchi et al. ) . The two closest works to our line of research are from Lee et al . ( 2020 ) and Doersch et al . ( 2017 ) . The former shows that a theoretical link can be established between conditional independence and an improvement of the performance on the downstream task , while the latter proposes to select layers from a multitask self-supervised encoder according to the pretext task to be solved . However , in both cases , the studies do not offer practical and theoretical solutions to select groups of pseudo-labels to build an adapted SSL model that will perform well on the considered downstream tasks . Group feature selection . Finally , feature selection , and especially group feature selection is another close and inspiring field given the problem we consider . The relationship and interactions between features have been largely investigated in the supervised learning literature ( Guyon & Elisseeff , 2003 ) . This led to multiple solutions to the feature group selection problem , including LASSObased techniques ( Yuan & Lin , 2006 ) , or multiple kernel formulations ( Sonnenburg et al. , 2006 ; Rakotomamonjy et al. , 2007 ) . However , these works do not involve any self-supervision , and links between feature selection and self-supervision design and pretext-task selection are yet to be proved . We will further consider these lines of works for concurrent baselines . With this work , we aim at shortening the process of designing SSL models while giving insights on the pseudo-label importance and the underlying mechanisms between pretext and downstream tasks at the same time . We decided to experiment with speech due to the lack of literature on this domain for multitask SSL , and for the various pseudo-labels available , which are based on decades of signal processing research . The whole pipeline starting from the acoustic feature extraction to the downstream task scoring follows three major steps summarized in Figure 1 . First , for every downstream task , our method produces a pretext task selection and weighting . Then , a SSL model is trained , before being used as a feature extractor front-end to one or many downstream tasks .

Summary: This paper describes authors' proposal for selecting pretext tasks (pseudo-labels) for multitask self-supervised learning, to improve self-supervised learning for speech representations. It leveraged the Hilbert Schmidt Independence Criterion (HSIC) for selecting pseudo-label loss weights and used softmax and its sparsity version to realize it. Experimental results on speech, speaker and emotion recognition showed effectiveness of the proposed approach.