Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors collected a large French speech dataset which consists of a large variety of speech corpora in French that cover different accents, acted emotions, telephone dialogues, read and spontaneous sentences, broadcast speech, and professional speech. It also presents a diversity of problems, information extracted, and annotated resources. It can serve as a good benchmark for SSL tasks.
strengths: 1. The dataset is quite large and diversified. 
2. The evaluation is sufficient and covers multiple domains.
3. This paper finds a good research problem for a fair comparison of different SSL methods.
4. This paper is well written and organized.
weaknesses: Why did the authors choose French not English for the SSL evaluation? Will it prevent people working on English use your benchmark?
correctness: Yes.
clarity: Yes, it is well writtened. 
relation to prior work: It is well discussed.
additional feedback: N/A
review_title: SSL evaluation benchmark for French

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a benchmark datasets for evaluating self-supervised learning models on the French speech data. The benchmark is documented with large-scale corpora and a collection of pretrained SSL wav2vec 2.0 models. It includes clear evaluation protocols for four downstream tasks: automatic speech recognition, spoken language understanding, automatic speech translation and automatic emotion recognition. 
strengths: The benchmark created from this paper is a good testbed for French speech models, which is an advanced step towards systematic evaluation of speech models using self-supervised learning.
The framework of the benchmark is well executed with clear scoring scripts, documentations, distribution of pretrained checkpoints, corpora, etc.
weaknesses: First, the writing could be largely improved w.r.t. the following aspects:
- As the benchmark proposed is specifically designed for French speech data, it would be better to point this out in the title explicitly. Also, I didn't see how this benchmark contributes to multilingual speech models except for French.

- In the experiment sections as well as the supplementals, different tasks employ different neural models. I'd like to confirm that if these models use representations from different pretrained SSL models (i.e. Wav2Vec 2.0) as the inputs and then finetuned on the labeled downstream datasets. It would be more clear if how different models (downstream models v.s. pretrained models) are used for evaluation.
Also, it would be more clear to have a table that compares different hyperparamters of models (architectures / dimensions, etc).

- I would strongly recommend you to replace (braces) with [brackets] for references. Using braces is quite unconventional and hard to read.

Second, the findings and observations in the experiment section are not surprising. I'd like to know why pretraining on more data (7k) is less beneficial for downstream tasks compared to the 3k model, which is a bit counter-intuitive to me. The other question is, why don't you use the same architecture (e.g. transformers) to perform the four tasks but use different models?
correctness: The evaluation methods and construction of datasets look right to me.
clarity: See weakness above.
relation to prior work: I am not very familiar with speech benchmarks, but it seems that there lacks a section or paragraph that compares with previous speech benchmark datasets, either English or multilingually.
additional feedback: N/A.
review_title: A compiled benchmark for evaluating SSL models on French speech data

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces LeBenchmark, a massive open-source benchmark on eight frameworks on English speech data. The work allows researchers to evaluate speech models on several downstream tasks: Automatic Speech Recognition (ASR), Spoken Language Understanding (SLU), Speech Translation (AST), and Emotion Recognition (AER). In addition, the speech type provided in the benchmark is diverse, making it very useful for evaluating different speech accents and styles.
strengths: - Large benchmark for speech evaluation
weaknesses: - The paper is missing the appendix while the main paper refers to it.
correctness: - It is not clear how to reproduce the results shown in the paper. The authors need to add important details how to reproduce the results.
clarity: In general, the paper is clear and well-written.
relation to prior work: The benchmark focuses on French speech, and it has more variety of datasets.
additional feedback: -
review_title: A good benchmark

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This is a benchmark (i.e., a set of tasks with exactly defined data)
for speech recognition, spoken language understanding, speech
translation, and emotion recognition for self-supervised learners. It
provides pre-trained benchmark models, and scripts to recreate them,
but does not release the data used in the benchmark itself or scripts to 
preprocess the data into the right format.

strengths: With a little bit more work on the part of the authors to make sure
the data preprocessing is reproducible, this work sems likely to be
the foundation of more progress in self-supervised learning for
speech-related tasks.
weaknesses: The largest flaw with this paper is that the results do not appear to
be fully reproducible. In particular, the provided github code does
not show how to extract the corpora that make up LeBenchmark. For example, the \
code is commented out here: https://github.com/LeBenchmark/NeurIPS2021/blob/650\
92828dc8e9fcd71d114f6d0a0f125c4b14d4e/ASR/ETAPE-Hybrid/run.sh#L21

Given that LeBenchmark consists of ~20 individual datasets and is
intended as a leaderboard, it looks like a critical error to this
reviewer that scripts to preprocess datasets exactly in the same way
(text normalization, etc.) are not provided in the open source code.

correctness: It is fishy that the ETAPE speech recognition WER results are so high,
but I am not familiar with how challenging this corpus. It would have
been appreciated to see previous papers' results on this corpus for
comparison (even if it wasn't with self-supervised models), to
understand how challenging the task is.

clarity: This is one of the best written papers I've read in a long time.

It is not clear how the LinTh model is comparable to the GRU model,
given that one works on sequence data, and the other works on a fixed
size input. Were the mel filter banks and/or SSL embeddings averaged
before inputting to the linear model? It isn't clear.

relation to prior work: As the authors mention, self-supervised learning for speech is still
very new; there is no existing benchmark (and very little existing
work) in French for this task

additional feedback: Change "41,5" and "16,8" to "41.5" and "16.8" in section 5.2.
review_title: Good benchmarks, some errors

Meta-review: All reviewers appreciated the contributions of this work, in particular the diversity of data and domains. Two reviewers do note that more information on hyparams etc. ought to be included, in particular for reproducibility purposes. Several reviewers note that the dataset is in French, and that this may limit adoption of the benchmark; on this point I disagree---we need more datasets in languages other than English (even for similarly high resources European languages). To me, it is a strength of the work that it does not focus on English. This being said, it would be nice if you included a bit more on the unique properties of French (speech) that might be relevant for understanding performance on the benchmark (if there are any). All in all, the work is strong and is likely to be recognized as a good quality SSL speech benchmark.
Decision: Accept