Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a dataset for SLU that focuses on voice commands dealing with multi-digit numbers, which often find use in the daily activities of humans. The proposed dataset tries to fill an important gap in current SLU benchmarks that, at best, deal with single-digit scenarios. SLU models are also important in developing and deploying home voice assistants but understanding multi-digit commands often needs access to an internet search. However, having such a dataset should allow the training of models on-the-edge that could try and understand multi-digit voice commands directly. Further, the paper extensively benchmarks different types of relevant baselines on this dataset and provides a hassle-free way to download and use the dataset.

Edit: score updated on July 13th

strengths: 1. Coverage of multi-digit scenarios: Although there is no dearth of such datasets in NLP as they are often used to test models vis-a-vis common sense reasoning and related tasks, the lack of them in the spoken form is an important gap that needs to be filled, in my opinion. Understanding voice commands can often be successfully dealt in two stages by first transcribing the voice and then understanding the transcript, as shown by the multistage baseline, but for that, one first needs a dataset of spoken voice commands from which the transcript can be generated. This paper tries to fill that gap by creating a dataset of real and synthetic voices that cover various scenarios in daily human lives where people deal with multi-digit numbers. 

2. Benchmarking baselines: the paper extensively evaluates different important baselines on the dataset on both the real and synthetic versions of the dataset. This gives insights into the relative usefulness of different baseline types and also shows that authors have done a decent job even with the real data collection despite its limited amount in comparison to the synthetic data. 
weaknesses: Major:
1. As the authors point out, the synthetic version of the dataset is quite artificial in that the speeches are from experts and recorded in very controlled scenarios. However, in the real world it might be very hard to find such controlled environments. If that's the case, why didn't the authors try to collect more real-world data through better forms of crowdsourcing like mechanical Turks, paid volunteers, etc. Even if the models trained mainly on synthetic data generalize very well to the real-world data, the real-world test set should be larger than just 10 speakers and 0.3 hours of audio. Some discussion on this aspect would surely help the paper's cause.

Minor:
1. Page 4, 4.2 Speaker Recruitment: I am not sure what issues there might be with collecting speaker information that helps to identify them if the authors of the dataset keep them anonymous during the dataset release. Like the authors pointed out, that would help them remove the duplicates and make the dataset even cleaner. Besides, asking a volunteer to submit their voice samples only once is a completely legit request.

2, Need for decoupled baseline: in which scenario would one require the decoupled model? If I understand correctly, the only difference between the decoupled model and the multistage model with precomputed transcript is that in the decoupled model, the NLU part of the model has access only to the ground truth transcripts during training, which significantly worsens its performance. However, I can't think of any case where the NLU model won't have access to the ASR outputs during training. Even in privacy preserving cases where distributed or federated learning needs to be used, the NLU model can very well be expected to have access to the ASR outputs during training as they absolutely need those during inference.


P.S.: I am leaning towards acceptance but would like to see the above-listed concerns addressed.
correctness: The claims made in the paper and the supporting experiments/evaluation methods look correct to me.


clarity: The paper is well-written on the whole.

However, some clarifications could improve the paper:
1. Page 4, 4.4 Synthetic Data: the paper talks about 'redundant rows' being part of the .csv file for the sake of balancing. Which .csv file is this (don't think it has been described before)? Also, if the .csv is part of the opensourced dataset, I don't see why one would maintain copies of the same data sample in that. One can always have unique rows in the csv and do the balancing during the data-loading. That allows for different flexibilities: balancing on the fly, weighing the losses instead of balancing the data, etc.

2. Page 7, 5.4, "can overfit to synthetic speech" in the title: why use the term 'overfit' when the direct model also performs better when trained on 'train-real' and tested on 'test-real'?
relation to prior work: The work extensively compares with prior work and tries to address their shortcomings.
additional feedback: None
review_title: An important SLU dataset dealing with multi-digit numbers in realistic contexts

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper provides a dataset for spoken language understanding with numbers, `Timers and Such', and its detailed construction process. Though it lacks a literature search, its necessity is quite well demonstrated, and it is clear that this kind of benchmark has been rarely suggested so far. The paper describes the initiative attempts, how the authors extended their work to a scalable size, and which factors they had taken into account while collecting the volunteered recordings.
strengths: - The proposed benchmark handles numbers in spoken language understanding, which is important for a real world application but has been underestimated
- The categorization of number information, namely SetTimer, SetAlarm, SimpleMath, and UnitConversion, reflects the needs of SLU context
- Design process is demonstrated step by step, from the initiatives to the expansion, telling the readers how the authors reached the desired amount of data with a guaranteed quality
- The collection process describes in detail the considerations conducted while recruiting and cleansing; especially encouraging the community contribution and managing the possibly noisy instances will help future researches collect the dataset with a guaranteed quality
weaknesses: - The paper lacks the literature study. It would be better if there is a survey on the struggles in SLU context that deals with temporal and numeric features, or in other view, MRC materials that handle temporal and mathematical reasoning
- Char level F1 in evaluation section may have to be more elaborately designed, but it is not necessarily the scope of this study 
correctness: I guess the evaluation should be more elaborately designed and described, for this study to be a benchmark. Other parts (motivation, preliminary, construction, experiments) are quite clear.
clarity: The paper is well written and easily understandable.
relation to prior work: The paper lacks the discussion on how the proposed benchmark is related to the literature, but it does not necessarily harm the completeness of the paper.
additional feedback: - Regarding the char F1 evaluation, it is not intuitive that '111' and '210' have same amount of error compared to '110', given that 111 is much more close to 110 than 210 does. 
- Also, given that this benchmark concentrates on numerics, it seems that some kind of normalization or digit shifting is required in the evaluation process.
- It would be nice if there is explanation that reporting SLU WER can compensate the weakness of char level evaluation.
review_title: Dataset for an essential processing of numbers, described with a well-organized building pipeline

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a dataset of spoken English commands involving numbers, aka Timers and Such. Although the real data is small, a supplemental dataset with synthetic speech is also included. The extensive experiment is conducted on this dataset with different kinds of baseline methods, including decoupled, multistage and direct SLU models. 
strengths: 1. A specific dataset for spoken English commands involving numbers.
2. Although the real data is small, but a large synthesis data is included.
3. Extensive experiment is conducted.
weaknesses: 1. The real data is relatively small. 
2. The text is generated with simple grammar, and may not cover different expressions. 
correctness: I still think it is a correct way to use the method mentioned by the author to collect the data， aka, ask the volunteers what they would say to complete a certain task. If it is too random or complex, maybe a part of the data can be collected in this way. 
clarity: This paper is well written.
relation to prior work: Yes, it is clearly discussed the difference from previous contributions.
additional feedback: 1. I think maybe instead of directly train the multistage model from scratch, maybe train the decoupled models first and then use them to initialize the multi-stage model is a good way to try.
2. The SLU output is a structured output. How to make it sequential to train a sequence to sequence SLU model is not mentioned.
review_title: A specific dataset for numbers with small real data and large synthetic data.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The work deals with creating a Spoken Language Understanding dataset. The unique feature of this dataset is that it contains numbers which previous datasets don't contain. This allows the spoken language understanding algorithms to potentially expand to basic numerical operations like unit conversion and simple mathematics. The dataset is collected using a web-based platform. The authors have benchmarked three techniques on the dataset using two metrics. The dataset and the related codebase is released by the authors publicly.
strengths: -- After revision from the authors --
I thank the authors for clarifying the raised points. I believe the newer version of the paper takes care of multiple weaknesses, and thus, I lean more positively.

1. This is the first dataset to contain statements with numbers in it. The statements often also correspond to numerical operations like unit conversion, and models can be trained for such tasks.

2. There are four types of spoken commands collected in the dataset, which relate well to what users use in general.

3. The authors benchmark three techniques on their dataset. They use a decoupled technique where a pre-trained ASR is used to transcribe speech followed the NLU. They also have a multi-stage technique and a direct technique. The authors have used two metrics to report the results.

4. I like how the authors split the train & test set based on demographics and location providing nice unseen variations. 

5. The dataset is public, and the codebase is pretty popular.
weaknesses: 1. The dataset contains only 2.5 hours of real speech (only 1.9 hrs in the train set). The authors use synthetically generated data to augment the dataset which is 68 times more than the real data. Such less amount of real speaker data is a weakness in my view for a dataset paper.

2. The authors don't show a clear comparison table with the previous datasets for this problem. Even though they cite the relevant datasets, a comparison between the number of hours available in the previous dataset and the proposed one will be a good addition to the paper.

3. The synthetic data can be ideally collected with more variations than what is done currently. Multispeaker TTS systems like RTVC(https://github.com/CorentinJ/Real-Time-Voice-Cloning) could have been used to add more variations. 

4. The authors don't identify the speakers of each of the audio files due to the non-collection of name/email id. However, speaker ID is an important feature in such a dataset. The speaker-id could have helped concretely count the number of unique speakers and could have been used for other downstream tasks. Unique information of each person could have been collected which may have been not released. However, the speaker id could have been obtained and released publicly.

5. I think the paper is not well written. Multiple sections need significant improvement. Diagrams should be used to differentiate between the three baseline methods to explain them more clearly. Section 5.1 should also be written more clearly. 

6. Dataset papers should have more insights about the dataset itself. Very little or no information is provided about the demographics of the speakers, gender of the speakers, age of the speakers, vocabulary, audio characteristics like sampling rate, etc. Ideally, the authors should provide more of such information.
correctness: The dataset has been collected reasonably well. I find the dataset collection technique to be fine and have no complaints. According to the authors, the dataset collection could have been done in an alternative way which is better suited for real-world usage but much more complex. 
clarity: I think the paper is not well written. Multiple sections need significant improvement. Diagrams should be used to differentiate between the three baseline methods to explain them more clearly. Section 5.1 should also be written more clearly. Dataset papers should have more insights about the dataset itself. Very little or no information is provided about the demographics of the speakers, gender of the speakers, age of the speakers, vocabulary, audio characteristics like sampling rate, etc. Ideally, the authors should provide more of such information. The authors also don't show a clear comparison table with the previous datasets for this problem. Even though they cite the relevant datasets, a comparison between the number of hours available in the previous dataset and the proposed one will be a good addition to the paper.
relation to prior work: The prior works don't have numerical values spoken in the recorded sentences, which this dataset proposes. However, a table comparing with the previous dataset is missing and is a major weakness, in my opinion.
additional feedback: None
review_title: An interesting dataset requiring more insight.

Meta-review: This paper introduces a speech dataset that involves multi-digit numerics. Particularly, they are looking into four scenarios (set timer, unit conversion, set alarm, simple math operations) that involve multi-digit utterances. Along with a small amount of (2.5 hours of human speech) real speech data, the authors also present large-scale synthetic data. Then, the authors present extensive experiments with decoupled, multistage, and direct SLU models. The dataset fills in the gap of what's missing in current datasets -- which does not cover such numeric expressions. The reviewers all agreed that this can provide useful resources to study the understudied, yet important problem of numeric expression understanding. 

The data collection method is fairly standard, i.e., speaking out what's written, instead of simulating situations and acquiring free speeches. From a language perspective, it's fairly simple -- as it is generated from four simple scenarios/grammar. This is reasonable as the task is focused on numeric understanding, and it's not easy to find such natural texts without access to commercial platforms. Yet, this is somewhat disappointing, and for future revisions of the data, it'll be worthwhile to study a two-stage approach where the user can paraphrase the generated sentence. The reviewers also have concerns about the size of the dataset, but for evaluation purposes, this could be sufficient. 
Decision: Accept