Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a new speech recognition dataset under a
CC-BY-NC-SA license explicitly covering the 8 standard subdialects of
Mandarin.

strengths: This is a unique dataset that fulfills a niche not covered before. In
particular, beyond focusing on subdialects, the number of speakers is
unprecedented in existing speech-to-text and speaker recognition
corpora.

weaknesses: Please add a # of hours column to table 2. #Utterances is hard to
interpret on its own.

A serious concern is that there is no comparison of a model trained on
KeSpeech against an existing dataset (AISHELL 1 or 2 is the best
candidate in this case). A common problem with new speech recognition
datasets is that the transcription quality is poor relative to
existing corpora; a good way to verify that this is not the case is,
e.g., to train an ASR model on all dialects of KeSpeech, and then
evaluate it on the AISHELL test set. The word error rate on the AISHELL test set
does not need to be state of the art. It just needs to be reasonable.

Another concern, described below, is that the test sets may not have been created correctly.
correctness: I am suspicious of the baseline evaluations in section 4. In
particular, there is no meaningful description of how the test sets
were created. For example, it is good practice to make sure the same
text does not appear in both the train and test sets. In addition, it
is good practice to make sure the same speaker does not appear in both
the train and test sets. There is no description of how test set
creation was done, however.

The results are definitely thorough, very well described, and convincing. It is just worrisome that 
a new dataset that could have a lot of future impact may have an incorrectly 
prepared test set.
clarity: Phase 2 allegedly occurred two weeks after phase 1, but phase 1 was
stopped when people stopped participating in the task. This
description is quite vague. In particular, someone who volunteered at
the start of phase 1 presumably waited more than two weeks for phase
2.

It is also not clear how many of the same speakers from phase 1 also
appear in phase 2. I understand that it can be hard to clearly
summarize this, but I cannot verify this as a reviewer on my own since
you did not upload the dataset as part of this submission. The number
of speakers who participated in both phases should be straightforward
to compute with a SQL statement.

You claim "The phase 2 task aimed to collect standard Mandarin audio
samples parallel to subdialect ones in phase 1 in terms of some
speaker and some text." It is not clear what "parallel" means in this
case. To create parallel sentences, how did you "translate" text from
a subdialect to standard Mandarin? This seems like a hard thing to do
to me, especially since it would presumably require a human annotator
who knows the subdialect as well as standard Mandarin, but I may be
missing something.

relation to prior work: Good summary of relation to prior work. Good foundational summary for
someone not already with the subdialects of Mandarin.
additional feedback: No additional feedback
review_title: Exciting work, but has some confusing points and potential problems

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This submission consists of a large dataset containing speech recorded in Mandarin and other subdialects. The researchers collected snippets of text and used volunteers located in different cities in China to read these texts. This dataset will be a useful way to make automated speech recognition systems better able to handle different dialectcs.
strengths: The dataset is large and covers an array of different dialects. The researchers have provided other annotations that may be helpful, including information about each speaker.
weaknesses: The dataset is not representative and is skewed towards certain groups. For example, the speakers are disproportionately male. It may be the case that downstream applications perform less accurately for groups less well represented in the dataset.
correctness: The data collection and evaluation procedures appear to be quite thorough. The authors evaluate their dataset using several different comparisons.
clarity: Yes
relation to prior work: The authors cite several related works and distinguish their contribution, although I am not familiar enough with this area to make a definitive judgment.
additional feedback: N/A
review_title: A large dataset of speech in Mandarin and dialects

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Good dataset that will improve our ability to understand dialects of Chinese. Generally this area is underexplored and I'm pretty excited. I think this paper could be a 6-8 with additional work.

10/1/2021 update: Great work addressing my feedback, based on the improvements in the paper and underlying work I would upgrade my  rating to a 7!

Congratulations to the authors, look forward to seeing the data set.
strengths: 1. Dataset offers a large number of speakers from diverse locations in China, about 4X more than existing datasets

2. Dataset offers labels for subdialect. Dealing with dialects is a significant issue and generally doesn't get enough attention. This is very  cool!

3. Dataset offers diverse context, e.g., different recording devices, noisy background noise, etc.

4. Authors performed initial analysis of gender split and speaker location

5. The experiments especially table 3 illustrates the importance of subdialects and the challenge, really nice!
weaknesses: 1. The license is problematic: CC-BY-NC-SA. The non-commercial clause will limit usage by industry researchers.

2. The labels are crowd sourced and therefore of unknown quality, I would like to see this explored more thoroughly.

The authors write: "Each audio was then listened by people to check and correct its transcription, as some of the audio did not actually correspond to the reference text."

Could they elaborate and help me understand this better? I think it's possible this isn't a problem, but need more info. Who listened to check the crowd-sourced work? Were the listeners skilled?

3. The transcription and audio are probably not aligned, which will generally impede use for ASR.

4. I did not see an analysis of how much of the data was noisy versus clean. Also it looks like noisy vs. clean was only labeled on Mandarin.

5. I don't see a hosting/maintenance plan.

6. No noise vs. clean on dialects

If items 2 and 5 are fixed, it would change my rating. 
correctness: The audio and transcripts should be aligned.

I would be happier if we had noise labels on the dialects instead of just Mandarin.
clarity: Mostly.
relation to prior work: Yup.
additional feedback: You should probably cite: https://openreview.net/forum?id=R8CwidgJ0yT

Generally this is really cool work and I'd like for it to get published. Things your team could do that would change my rating  and help push it up above publishing threshold:

1. Align the audio and transcription to enable ASR (or convince me it's not necessary)
2. More detail on label quality to help future users
3. Hosting and maintenance plan
4. Noisy vs. clean on dialects (I know this is a big lift and may not be possible)
5. Use a CC-BY license

This has the possibility of being a 6-8 with some additional work.

I've upgraded to 7.
review_title: Great dataset for subdialects with wide speaker variety

Decision: Accept