Multimodal and Multilingual Embeddings for Large-Scale Speech Mining

1 Introduction . Neural approaches have evolved to the de-facto standard in machine translation ( MT ) , namely textto-text and speech-to-text translation ( Berard et al. , 2016 ; Weiss et al. , 2017 ) , and more recently speech-to-speech translation ( Jia et al. , 2019 ) . While there is very promising research on unsupervised MT , e.g . ( Lample et al. , 2018 ; Artetxe et al. , 2017 ) , or initialization of parts of the MT system by approaches trained in a self-supervised manner , labeled data remains extremely useful to achieve best performance . This labeled data is either used to directly train the MT system end-to-end , if enough data is available , or to fine-tune an MT system which parts have been initialized with some self-supervised methods . This type of approaches is commonly used in speech-to-text translation , see for example ( Tran et al. , 2020 ; Li et al. , 2020 ) . In the case of text-to-text ( t2t ) machine translation , the labeled data consists of source/target sentence pairs , commonly named bitexts or parallel data . There are several international organisations which produce human translation in several languages , best known are probably the European Commission and the United Nations . These human translations have been collected and are distributed to the MT community , e.g . the well known Europarl ( Koehn , 2005 ) and UN corpora ( Ziemski et al. , 2016 ) . There is also a large body of research which aims in finding existing translation in huge collections of monolingual texts , which are abundant on the the Internet for many languages . This bitext mining has provided huge amounts of parallel data , in particular the ParaCrawl project ( Esplà et al. , 2019 ) or the CCMatrix initiative ( Schwenk et al. , 2021 ) . This data enabled training a multilingual t2t 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . MT systems for hundred languages ( Fan et al. , 2020 ) . We are not aware of any attempts to mine for speech translation data . Training speech-to-text ( s2t ) MT systems requires audio in the source language and its translation in the target language . To the best of our knowledge , this type of data is mainly generated by humans volunteers : speech is first transcribed and then translated into a another language – a time and labour extensive process . When translations are not provided at the sentence level , forced alignments are necessary . These methodologies helped creating well-known speech translation corpora , like Must-C ( Di Gangi et al. , 2019a ) , CoVoST ( Wang et al. , 2020a ) and EuroparlST ( Iranzo-Sánchez et al. , 2020 ) . Finally , speech-to-speech translation ( s2s ) is faced with extreme scarcity of labeled data , i.e . speech in the source language and the spoken translation in the target language . Previous work on speech-tospeech translation often use synthesized speech targets to overcome this lack of labeled data . We are only aware of one s2s training corpus ( Wang et al. , 2021 ) which provides speech-to-speech alignments for 16 source languages with 5 target languages . However , the target aligned speech is oral interpretation of the source speech . The quality of these oral interpretations for speech-to-speech translation have not been evaluated yet . In this work , we build on recent advances to perform bitext mining based on a similarity measure in a fixed-size sentence embedding space ( Schwenk , 2018 ; Artetxe and Schwenk , 2019 ; Feng et al. , 2020 ) . The underlying idea of those approaches is to learn a multilingual sentence embedding with the property that sentences with similar meaning , in the same or a different language , are close in the embedding space . Then , mining for translations can be performed by calculating the cosine distance , or a margin-based criterion , in this multilingual embedding space for a large collection of sentences in several languages . Sentence pairs with a distance below a threshold are considered to be parallel . We extend this idea to the speech modality and present a fixed-size embedding of a speech input , of arbitrary length . Our current audio encoder supports speech in five languages , namely English , French , Spanish , German and Russian . We apply a teacher-student training approach which yields speech embeddings which are compatible with an existing multilingual text encoder , namely the freely available LASER encoder . This means that we can mine speech input against the 80 languages which are supported by LASER , as used in the bitext mining project CCMatrix . The contributions of this paper are as follows : • We explore several approaches to learn multimodal ( speech/text ) representations for several languages ; • We use these multimodal embeddings to mine more than twenty thousand hours of speech in four languages from Librivox against texts from Common Crawl ; • We add the automatically mined data to train speech-to-text translation systems and obtain significant improvements with respect to a very strong baseline ; • We also provide a proof of concept that our approach can be used to directly perform speech-to-speech mining , without the need to transcribe and translate the input speech ; The paper is structured as follows . In the next section , we first summarize related work . In Section 3 we describe our approach in detail , followed by a detailed experimental evaluation in Section 4 . The paper concludes with a summary and directions of future research . 2 Related work . Bitext mining . There is a large body of research on bitext mining from monolingual resources , i.e . finding parallel texts in a source and target language . A large variety of approaches has been explored like relying on document meta-information ( Resnik , 1999 ) , cross-lingual document retrieval ( Munteanu and Marcu , 2005 ) or machine translation and information retrieval ( Abdul-Rauf and Schwenk , 2009 ; Bouamor and Sajjad , 2018 ) , just to name a few . Our approach for speech mining is inspired by recent work on massively multilingual sentence embeddings , i.e . fixed-size text representations with the property that similar sentences written in different languages are close in the embedding space . Mining is then performed by simple exhaustive comparison of sentences in different languages , using either a threshold on the cosine distance ( Schwenk , 2018 ) or a margin criterion ( Artetxe and Schwenk , 2018 ; Yang et al. , 2019a ) . The multilingual sentence representations can be learned with neural machine translation framework ( España-Bonet et al. , 2017 ; Schwenk and Douze , 2017 ; Artetxe and Schwenk , 2019 ; Kvapilíková et al. , 2020 ) . Recent works focus on dual encoders , for example ( Yang et al. , 2019a ; Reimers and Gurevych , 2019 ; Yang et al. , 2019b ; Feng et al. , 2020 ) . Finally , knowledge distillation was proposed to extend existing multilingual sentence embeddings to new languages ( Reimers and Gurevych , 2020 ) . Our approach is similar to that work : we extend an existing multilingual sentence embedding to the speech modality . Fixed-size speech representations . Fixed-size speech representations have been mainly studied at the word level for different specific tasks , ranging from spoken term detection , speech pattern discovery , to speech segmentation into words . Different approaches have been studied to extract a fixed-size representation from speech input . Holzenberger et al . ( 2018 ) introduced a method to extract a fixed-size vector from speech inputs using Gaussian downsampling , without the need of any training . Holzenberger et al . ( 2018 ) ; Chung et al . ( 2016 ) introduced an auto-encoder approach based on recurrent neural networks to extract a fixed-size vector between the encoder and the decoder . Some other works ( Settle and Livescu , 2016 ; Riad et al. , 2018 ; Thiolliere et al. , 2015 ) train Siamese networks with a contrastive loss to build a fixed-size representation . Audhkhasi et al . ( 2017 ) studies a keyword search task , building fixed-size representations for audio and words before inputting these representations to a third neural network . At the sentence level , text-audio sentiment analysis often intrinsically introduces a fixed-size crossmodal representation before classifying the input , as in ( Yang et al. , 2020 ; Tsai et al. , 2019 ) . However , such works do not focus on speech/text alignments , but rather take advantage of information coming from both modalities . Khurana et al . ( 2020 ) focuses on speech representation learning with speech translation data and a contrastive loss at the sentence level . The model is first evaluated on a retrieval task but not used for large-scale speech translation mining , the speech encoder is rather used for a phone recognition task . Harwath et al . ( 2018 ) ; Merkx et al . ( 2019 ) ; Ilharco et al . ( 2019 ) ; Harwath et al . ( 2019 ) ; Monfort et al . ( 2021 ) build joint speech/visual embedding spaces at the sentence level and are evaluated with a retrieval task . Speech mining . We are not aware of any approach that performed mining of speech against either texts or even speech in another language . Speech input could be first transcribed by an existing speech recognition system . There are several works which use these automatic transcriptions , often named “ pseudo labels ” , to improve speech recognition . However , we are not aware that automatically transcribed speech was used as input to a text-to-text mining approach with the goal to create a speech translation corpus . One of the challenges are the different conventions : transcriptions are usually all lower case , without punctuation and numbers are spelled out as multiple words . To the best of our knowledge , all existing s2t corpora are using human translations , either volunteers translating sentences , or doing force-alignment when translations are not provided at the sentence level . Must-C ( Di Gangi et al. , 2019a ) provides more than 385 hours of English speech translated in 8 languages , CoVoST ( Wang et al. , 2020a ) provides speech translations in 15 En-X directions and 22 X-En directions , some languages are presented in Table 1 , and EuroparlST ( Iranzo-Sánchez et al. , 2020 ) provides 30 different speech translation directions from 6 European languages . 3 Speech encoder training . Training a multimodal audio/text fixed-size embedding could be motivated by recent research on training Siamese networks with a contrastive loss , e.g . ( Feng et al. , 2020 ) . Instead of two text encoders , one would use one speech encoder and one text encoder . However , since both encoders are trained from scratch , such a procedure would probably require a large amount of labeled multimodal training data . Instead , we apply a teacher/student training framework : we use an existing text encoder as teacher and train an audio encoder to minimize the cosine loss between the two encoder outputs . This architecture is summarized in Figure 1 . It can be trained with two types of labeled data : • Speech transcriptions : both encoders use input in the same language , but differ in the modality ; • Speech translations : since the text encoder is multilingual , we can also minimize the cosine loss of the speech embedding with respect to its written translation in one of the languages supported by the text encoder . Concretely , we use the multilingual LASER sentence encoder which is freely available1 and which was successfully used in large-scale text mining approaches ( Schwenk et al. , 2021 ) . A thorough comparison with other multilingual sentence encoders is left for future research , in particular LaBSE ( Feng et al. , 2020 ) . The LASER encoder is fixed during teacher/student training . Our speech encoder is based on wav2vec 2.0 architecture ( Baevski et al. , 2020 ) , a speech encoder which learns highquality speech representations from unlabelled speech data . It is composed of a CNN feature encoder stacked with a transformer encoder . In recent work , wav2vec 2.0 was extended to multiple languages : the XLSR model ( Conneau et al. , 2020 ) .2 We used the XLSR pre-trained feature encoder , without updating its weights , and fine-tuned the weights of XLSR transformer encoder to obtain our fixed-size audio representation . During fine-tuning , the feature encoder representations are masked with a strategy similar to SpecAugment ( Park et al. , 2019 ) as introduced in wav2vec 2.0 paper . We explored several methods to get a fixed-size speech representation , like max- or mean-pooling of the encoder outputs . Best performance was obtained using the output of the transformer encoder corresponding to a particular begin-of-sentence ( BOS ) vector . This BOS vector is simply a vector filled with ones ( 1.0 ) , added at the beginning of the feature sequence in wav2vec2.0 architecture . This method is inspired by BERT ( Devlin et al. , 2019 ) sentence representations for text , that are often extracted using a CLS token at the beginning of the input sentence , e.g . ( Reimers and Gurevych , 2019 ) .

Summary: The authors aim to augment existing approaches of multilingual text representation, ie. embeddings, to multimodel, ie. speech/text. That is training speech/text encoders that yield the same fixed sentence-level embedding regardless of language or whether spoken or written. The paper demonstrates that the multimodal embedding approach can be used to mine a significant amount of speech in several languages from an open source LIbrivox corpus. The data that is mined was shown to improve the translation BLEU scores over a good baseline, and also yields aligned speech-to-speech data.