Multimodal and Multilingual Embeddings for Large-Scale Speech Mining

1 Introduction . Neural approaches have evolved to the de-facto standard in machine translation ( MT ) , namely textto-text and speech-to-text translation ( Berard et al. , 2016 ; Weiss et al. , 2017 ) , and more recently speech-to-speech translation ( Jia et al. , 2019 ) . While there is very promising research on unsupervised MT , e.g . ( Lample et al. , 2018 ; Artetxe et al. , 2017 ) , or initialization of parts of the MT system by approaches trained in a self-supervised manner , labeled data remains extremely useful to achieve best performance . This labeled data is either used to directly train the MT system end-to-end , if enough data is available , or to fine-tune an MT system which parts have been initialized with some self-supervised methods . This type of approaches is commonly used in speech-to-text translation , see for example ( Tran et al. , 2020 ; Li et al. , 2020 ) . In the case of text-to-text ( t2t ) machine translation , the labeled data consists of source/target sentence pairs , commonly named bitexts or parallel data . There are several international organisations which produce human translation in several languages , best known are probably the European Commission and the United Nations . These human translations have been collected and are distributed to the MT community , e.g . the well known Europarl ( Koehn , 2005 ) and UN corpora ( Ziemski et al. , 2016 ) . There is also a large body of research which aims in finding existing translation in huge collections of monolingual texts , which are abundant on the the Internet for many languages . This bitext mining has provided huge amounts of parallel data , in particular the ParaCrawl project ( Esplà et al. , 2019 ) or the CCMatrix initiative ( Schwenk et al. , 2021 ) . This data enabled training a multilingual t2t 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . MT systems for hundred languages ( Fan et al. , 2020 ) . We are not aware of any attempts to mine for speech translation data . Training speech-to-text ( s2t ) MT systems requires audio in the source language and its translation in the target language . To the best of our knowledge , this type of data is mainly generated by humans volunteers : speech is first transcribed and then translated into a another language – a time and labour extensive process . When translations are not provided at the sentence level , forced alignments are necessary . These methodologies helped creating well-known speech translation corpora , like Must-C ( Di Gangi et al. , 2019a ) , CoVoST ( Wang et al. , 2020a ) and EuroparlST ( Iranzo-Sánchez et al. , 2020 ) . Finally , speech-to-speech translation ( s2s ) is faced with extreme scarcity of labeled data , i.e . speech in the source language and the spoken translation in the target language . Previous work on speech-tospeech translation often use synthesized speech targets to overcome this lack of labeled data . We are only aware of one s2s training corpus ( Wang et al. , 2021 ) which provides speech-to-speech alignments for 16 source languages with 5 target languages . However , the target aligned speech is oral interpretation of the source speech . The quality of these oral interpretations for speech-to-speech translation have not been evaluated yet . In this work , we build on recent advances to perform bitext mining based on a similarity measure in a fixed-size sentence embedding space ( Schwenk , 2018 ; Artetxe and Schwenk , 2019 ; Feng et al. , 2020 ) . The underlying idea of those approaches is to learn a multilingual sentence embedding with the property that sentences with similar meaning , in the same or a different language , are close in the embedding space . Then , mining for translations can be performed by calculating the cosine distance , or a margin-based criterion , in this multilingual embedding space for a large collection of sentences in several languages . Sentence pairs with a distance below a threshold are considered to be parallel . We extend this idea to the speech modality and present a fixed-size embedding of a speech input , of arbitrary length . Our current audio encoder supports speech in five languages , namely English , French , Spanish , German and Russian . We apply a teacher-student training approach which yields speech embeddings which are compatible with an existing multilingual text encoder , namely the freely available LASER encoder . This means that we can mine speech input against the 80 languages which are supported by LASER , as used in the bitext mining project CCMatrix . The contributions of this paper are as follows : • We explore several approaches to learn multimodal ( speech/text ) representations for several languages ; • We use these multimodal embeddings to mine more than twenty thousand hours of speech in four languages from Librivox against texts from Common Crawl ; • We add the automatically mined data to train speech-to-text translation systems and obtain significant improvements with respect to a very strong baseline ; • We also provide a proof of concept that our approach can be used to directly perform speech-to-speech mining , without the need to transcribe and translate the input speech ; The paper is structured as follows . In the next section , we first summarize related work . In Section 3 we describe our approach in detail , followed by a detailed experimental evaluation in Section 4 . The paper concludes with a summary and directions of future research . 2 Related work . Bitext mining . There is a large body of research on bitext mining from monolingual resources , i.e . finding parallel texts in a source and target language . A large variety of approaches has been explored like relying on document meta-information ( Resnik , 1999 ) , cross-lingual document retrieval ( Munteanu and Marcu , 2005 ) or machine translation and information retrieval ( Abdul-Rauf and Schwenk , 2009 ; Bouamor and Sajjad , 2018 ) , just to name a few . Our approach for speech mining is inspired by recent work on massively multilingual sentence embeddings , i.e . fixed-size text representations with the property that similar sentences written in different languages are close in the embedding space . Mining is then performed by simple exhaustive comparison of sentences in different languages , using either a threshold on the cosine distance ( Schwenk , 2018 ) or a margin criterion ( Artetxe and Schwenk , 2018 ; Yang et al. , 2019a ) . The multilingual sentence representations can be learned with neural machine translation framework ( España-Bonet et al. , 2017 ; Schwenk and Douze , 2017 ; Artetxe and Schwenk , 2019 ; Kvapilíková et al. , 2020 ) . Recent works focus on dual encoders , for example ( Yang et al. , 2019a ; Reimers and Gurevych , 2019 ; Yang et al. , 2019b ; Feng et al. , 2020 ) . Finally , knowledge distillation was proposed to extend existing multilingual sentence embeddings to new languages ( Reimers and Gurevych , 2020 ) . Our approach is similar to that work : we extend an existing multilingual sentence embedding to the speech modality . Fixed-size speech representations . Fixed-size speech representations have been mainly studied at the word level for different specific tasks , ranging from spoken term detection , speech pattern discovery , to speech segmentation into words . Different approaches have been studied to extract a fixed-size representation from speech input . Holzenberger et al . ( 2018 ) introduced a method to extract a fixed-size vector from speech inputs using Gaussian downsampling , without the need of any training . Holzenberger et al . ( 2018 ) ; Chung et al . ( 2016 ) introduced an auto-encoder approach based on recurrent neural networks to extract a fixed-size vector between the encoder and the decoder . Some other works ( Settle and Livescu , 2016 ; Riad et al. , 2018 ; Thiolliere et al. , 2015 ) train Siamese networks with a contrastive loss to build a fixed-size representation . Audhkhasi et al . ( 2017 ) studies a keyword search task , building fixed-size representations for audio and words before inputting these representations to a third neural network . At the sentence level , text-audio sentiment analysis often intrinsically introduces a fixed-size crossmodal representation before classifying the input , as in ( Yang et al. , 2020 ; Tsai et al. , 2019 ) . However , such works do not focus on speech/text alignments , but rather take advantage of information coming from both modalities . Khurana et al . ( 2020 ) focuses on speech representation learning with speech translation data and a contrastive loss at the sentence level . The model is first evaluated on a retrieval task but not used for large-scale speech translation mining , the speech encoder is rather used for a phone recognition task . Harwath et al . ( 2018 ) ; Merkx et al . ( 2019 ) ; Ilharco et al . ( 2019 ) ; Harwath et al . ( 2019 ) ; Monfort et al . ( 2021 ) build joint speech/visual embedding spaces at the sentence level and are evaluated with a retrieval task . Speech mining . We are not aware of any approach that performed mining of speech against either texts or even speech in another language . Speech input could be first transcribed by an existing speech recognition system . There are several works which use these automatic transcriptions , often named “ pseudo labels ” , to improve speech recognition . However , we are not aware that automatically transcribed speech was used as input to a text-to-text mining approach with the goal to create a speech translation corpus . One of the challenges are the different conventions : transcriptions are usually all lower case , without punctuation and numbers are spelled out as multiple words . To the best of our knowledge , all existing s2t corpora are using human translations , either volunteers translating sentences , or doing force-alignment when translations are not provided at the sentence level . Must-C ( Di Gangi et al. , 2019a ) provides more than 385 hours of English speech translated in 8 languages , CoVoST ( Wang et al. , 2020a ) provides speech translations in 15 En-X directions and 22 X-En directions , some languages are presented in Table 1 , and EuroparlST ( Iranzo-Sánchez et al. , 2020 ) provides 30 different speech translation directions from 6 European languages . 3 Speech encoder training . Training a multimodal audio/text fixed-size embedding could be motivated by recent research on training Siamese networks with a contrastive loss , e.g . ( Feng et al. , 2020 ) . Instead of two text encoders , one would use one speech encoder and one text encoder . However , since both encoders are trained from scratch , such a procedure would probably require a large amount of labeled multimodal training data . Instead , we apply a teacher/student training framework : we use an existing text encoder as teacher and train an audio encoder to minimize the cosine loss between the two encoder outputs . This architecture is summarized in Figure 1 . It can be trained with two types of labeled data : • Speech transcriptions : both encoders use input in the same language , but differ in the modality ; • Speech translations : since the text encoder is multilingual , we can also minimize the cosine loss of the speech embedding with respect to its written translation in one of the languages supported by the text encoder . Concretely , we use the multilingual LASER sentence encoder which is freely available1 and which was successfully used in large-scale text mining approaches ( Schwenk et al. , 2021 ) . A thorough comparison with other multilingual sentence encoders is left for future research , in particular LaBSE ( Feng et al. , 2020 ) . The LASER encoder is fixed during teacher/student training . Our speech encoder is based on wav2vec 2.0 architecture ( Baevski et al. , 2020 ) , a speech encoder which learns highquality speech representations from unlabelled speech data . It is composed of a CNN feature encoder stacked with a transformer encoder . In recent work , wav2vec 2.0 was extended to multiple languages : the XLSR model ( Conneau et al. , 2020 ) .2 We used the XLSR pre-trained feature encoder , without updating its weights , and fine-tuned the weights of XLSR transformer encoder to obtain our fixed-size audio representation . During fine-tuning , the feature encoder representations are masked with a strategy similar to SpecAugment ( Park et al. , 2019 ) as introduced in wav2vec 2.0 paper . We explored several methods to get a fixed-size speech representation , like max- or mean-pooling of the encoder outputs . Best performance was obtained using the output of the transformer encoder corresponding to a particular begin-of-sentence ( BOS ) vector . This BOS vector is simply a vector filled with ones ( 1.0 ) , added at the beginning of the feature sequence in wav2vec2.0 architecture . This method is inspired by BERT ( Devlin et al. , 2019 ) sentence representations for text , that are often extracted using a CLS token at the beginning of the input sentence , e.g . ( Reimers and Gurevych , 2019 ) .

Summary: This paper proposes a multilingual and multimodal data mining framework that maps text and speech sentences into a shared embedding space of a fixed dimension, where sentences with similar meanings are close to each other and vice versa. Such embeddings can be then used for mining data for speech-to-speech or speech-to-text translation, which often encounter data sparsity issues.   This paper is an extension of LASER, a mining framework for only text. This paper augments LASER with a speech encoder (or multiple speech encoders, one per language) that maps an utterance into a fixed dimension representation shared with text. The authors propose to use a teacher-student framework for training the speech encoder using speech transcriptions and speech translation data. Specifically, a frozen pre-trained LASER text encoder is used, and the speech encoder to be trained is also initialized from a self-supervised speech model, XLSR. The encoder is then trained by regressing the embedding of the paired transcript or translation encoded by the LASER model. Experimental results show that 1) for some languages like Russian, using a shared encoder degrades the performance significantly, and 2) by using mined speech, performance of X-to-En translation can improve from a competitive baseline on the CoVoST2 benchmark.