LeBenchmark 2.0: A standardized, replicable and enhanced framework for self-supervised representations of French speech

Authors:
raw_author_name: Titouan Parcollet
raw_affiliation_string: Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Av., Cambridge, CB3 0FD, United Kingdom; Samsung AI Center Cambridge, 50/60 Station Road, Cambridge, CB1 2JH, United Kingdom
raw_author_name: Ha Nguyen
raw_affiliation_string: Laboratoire Informatique d’Avignon, Avignon Université, 339 Chem. des Meinajariès, Avignon, 84000, France
raw_author_name: Marcely Zanon Boito
raw_affiliation_string: NAVER LABS Europe, France
raw_author_name: Salima Mdhaffar
raw_affiliation_string: Laboratoire Informatique d’Avignon, Avignon Université, 339 Chem. des Meinajariès, Avignon, 84000, France
raw_author_name: Hang Le
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Sina Alisamir
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Natalia Tomashenko
raw_affiliation_string: Laboratoire Informatique d’Avignon, Avignon Université, 339 Chem. des Meinajariès, Avignon, 84000, France
raw_author_name: Marco Dinarelli
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Shucong Zhang
raw_affiliation_string: Samsung AI Center Cambridge, 50/60 Station Road, Cambridge, CB1 2JH, United Kingdom
raw_author_name: Alexandre Allauzen
raw_affiliation_string: ESPCI, CNRS LAMSADE, PSL Research University, France
raw_author_name: Maximin Coavoux
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Yannick Estève
raw_affiliation_string: Laboratoire Informatique d’Avignon, Avignon Université, 339 Chem. des Meinajariès, Avignon, 84000, France
raw_author_name: Mickael Rouvier
raw_affiliation_string: Laboratoire Informatique d’Avignon, Avignon Université, 339 Chem. des Meinajariès, Avignon, 84000, France
raw_author_name: Benjamin Lecouteux
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: François Portet
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Solange Rossato
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Fabien Ringeval
raw_affiliation_string: Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LIG, 38000, Grenoble, France
raw_author_name: Didier Schwab
raw_author_name: Laurent Besacier

Abstract: Self-supervised learning (SSL) is at the origin of unprecedented improvements in many different domains including computer vision and natural language processing. Speech processing drastically benefitted from SSL as most of the current domain-related tasks are now being approached with pre-trained models. This work introduces LeBenchmark 2.0 an open-source framework for assessing and building SSL-equipped French speech technologies. It includes documented, large-scale and heterogeneous corpora with up to 14,000 h of heterogeneous speech, ten pre-trained SSL wav2vec 2.0 models containing from 26 million to one billion learnable parameters shared with the community, and an evaluation protocol made of six downstream tasks to complement existing benchmarks. LeBenchmark 2.0 also presents unique perspectives on pre-trained SSL models for speech with the investigation of frozen versus fine-tuned downstream models, task-agnostic versus task-specific pre-trained models as well as a discussion on the carbon footprint of large-scale model training. Overall, the newly introduced models trained on 14,000 h of French speech outperform multilingual and previous LeBenchmark SSL models across the benchmark but also required up to four times more energy for pre-training.