Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper introduces a large-scale acoustic dataset containing around 20 hours of audio recordings of 36 different mosquito species. It also proposes two models based on Bayesian CNNs, one for mosquito event detection (MED), aiming to identify mosquitos from the overall background sound, and one for mosquito species classification (MSC). The dataset is mentioned to be the first large-scale multi-species one tracking mosquitos continuously in free flight. 
strengths: The dataset introduced in this paper seems to bring a significant contribution to the scientific community. Some strengths worth mentioning are the following:

1. The dataset was created in collaboration with mosquito entomologists, making it relevant not only from the machine learning point of view but also from the bio-science one.

2. The authors mention this is the first large-scale multi-species dataset of acoustic recordings of mosquitoes tracked continuously in free flight. 

3. It contains 36 species of mosquitos captured from multiple locations on the Globe, such as the UK, USA, Thailand, or Tanzania. Furthermore, it includes both laboratory culture mosquitoes and wild captured ones, all these aspects providing variety.

4. It contains both noise-free sounds and real-world audio captured with low-cost smartphones and professional devices, allowing the study of the behavior in both scenarios.

5. Extensive experiments are conducted on the dataset using performant audio models for mosquito event detection and mosquito species classification, providing, therefore, a notable benchmark. 
weaknesses: I did not notice any significant weaknesses. 

One thing that would be nice to be added is a pie chart revealing the number of mosquitoes per species or a study of the class balance in the dataset. As the authors mention performing weighted classification instead of the classical one, an additional discussion on this matter would better position the dataset. 
correctness: The dataset is constructed soundly by a team including both machine learning experts and entomology scientists. Therefore, the technical aspects are fully informed.
clarity: The writing of the paper is clear, making it an easy-to-follow one.
relation to prior work: The related work section is well documented, and the advantages of this dataset over the existing ones are clearly described.
additional feedback: None.
review_title: A Large-scale Acoustic Mosquito Dataset

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This contribution presents a dataset that consists of 20 hours of audio recordings of mosquitoes from different species tracked in free flight. Audio recordings have been realised in different locations around the world, with various settings and environments. 

In relation with the dataset, two tasks are suggested, acting like benchmarks to foster research on the topic:
* Identification of mosquitoes, i.e., binary classification of the presence or not of a mosquito in an audio recording.
* Classification of detected mosquitoes into one of the 36 different species.

The elaboration of the dataset is justified as mosquitoes are involved in the propagation of diseases. An automated way to detect and classify mosquitoes would have a significant impact on the management of the risks they pose, and increase the health conditions of affected regions.
strengths: The main strength of the contribution lies in the large scale of the dataset, which includes 20 hours of mosquitoes recordings from different locations, with an average duration of audio samples of 9.7 seconds, which is significantly longer than samples from previous datasets proposed for the same purpose. The number of species is also larger than previous studies. In addition, the audio samples have been recorded in controlled conditions, that increase the reliability of data and labels.

The contribution is accompanied by a comprehensive datasheet that provides valuable information about the dataset, reinforcing the seriousness of the paper and the soundness of the dataset.
weaknesses: My main concern on the paper is about the technical challenges the dataset may bring to the machine learning community. It is not clear what are the limitations of current techniques, in particular those developed for acoustic event detection and sound classification, and in which manner these techniques would not be able to address the tasks discussed in the paper. 

The baseline models provided by the authors seem to indicate that good performance are already possible, and it is not unreasonable to think that extended analysis of the data, using for example more systematic fine-tuning of hyperparameters, data augmentation strategies, expert knowledge, etc., may bring better results relatively easily.

Therefore, the contribution might be more suitable for application domain specialists (such as entomologists) that would have an interest in connecting results obtained using standard machine learning techniques with domain expertise to gain knowledge about the health issues related to mosquitoes.  Machine learning researchers, who constitute the main audience of the NeurIPS conference, might not find in this dataset the technical barriers that will lead to the development of more sophisticated models and techniques.
correctness: The submission consists of a dataset and of two benchmarks. The dataset is constructed in a sound way, and has the potential to lead to the development of machine learning models that may help researchers interested by such application. 

The benchmarks consist in two tasks related to the dataset. The level of details provided and the code accompanying the paper should be sufficient to reproduce the results, even if I haven't tried myself to do it. 
The claims made by the authors seem correct and reasonable, considering the information provided in the submission. 
clarity: The paper is well written, and the content is clear and well-organized.
relation to prior work: The authors included a discussion of related works, and compared the characteristics of their dataset with previous ones.
additional feedback: The paper is a resubmission from Round 1 that has been significantly improved based on the comments of the reviewers of Round 1. In my opinion, the comments have been carefully addressed, and the dataset could be used by researchers to investigate this important problem. My main limitation concerns the relevance of the dataset for the machine learning community as discussed in the Weakness section, and I would be happy to hear from the authors about the technical barriers connected to the dataset they identified in terms of methodology, beyond the application domain.

Update after discussions with authors: I have reconsidered my position regarding the relevance of the dataset for the conference (see comments below).


Other comments
--------------
* The first sentence states that mosquitoes are not present in Antarctica, with a reference to a 2013 publication. Is there any more up-to-date publication to confirm this is still the case with the current rise of temperature?
* According to WHO website, numbers for malaria are '219 millions' and 400 000 deaths, in disagreement with the numbers reported in the introduction.
* Given a system that accurately detect and classify mosquitoes from a sound recording trained using the dataset, what would be the typical use of such a system in real-world conditions? In uncontrolled environment, it seems very unlikely that a mosquito will fly close enough to the microphone to trigger a detection. In that case, would you say the dataset would only be useful in the same conditions as the data collection, i.e., capturing mosquitoes and recording them?
review_title: Robust dataset of audio recordings of mosquitoes

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents an enormously interesting dataset on the problem of mosquito detection and species classification. The dataset is so far the largest public dataset with 20 hours of mosquito samples and 15 hours of background noise, all expertly labeled.
strengths: The HumBugDB dataset is the largest and most diverse dataset currently available, with stronger labels than prior work.

The Authors provide a clear explanation of why mosquito detection and classification is an important task, and presents blueprints of a pipeline of how audio samples of mosquitos can be crowdsourced using smartphone. This is an important tool for more accurately predicting e.g. malaria or dengue outbreaks.

The authors clearly document the entire dataset construction process, the entire benchmarking process and gives insights into the results obtained.

The authors mention that voice activity detection and removal is used in future releases to ensure privacy. In the current release data has been curated to avoid voices unless explicitly allowed by the person in question.

The dataset is a clear contribution to the machine learning field as an interesting application for audio event detection and classification.

weaknesses: The paper solely evaluates using Bayesian Convolutional Neural Networks. This is due to the BNNs fitting well into decision making applications due to the provided uncertainty estimations. However, it would be interesting to see whether performance would improve if classic CNNs are used, at the cost of uncertainty estimation.

The paper evaluates the binary mosquito detection task and the multi-class mosquito species classification using the ROC-AUC. However, the data is stated to be imbalanced. Therefore, the ROC and ROC-AUC is not the appropriate metrics to use. Instead it would be more appropriate to use the Precision-Recall curves (see https://www.biostat.wisc.edu/~page/rocpr.pdf) or a metric such as Matthews Correlation Coefficient which considers the ratio of False/True Positive/Negatives.

It is unclear from the main manuscript whether there is a validation split. When examining the supplementary materials, it appears a random subset is used for validation for mosquito detection, while no subset is used for mosquito species classification. Could the authors explain why a static validation subset is not used, such that everyone uses the same validation data, or why cross-validation was not used

correctness: The claims appear to be correct. The dataset construction process is clearly described and the dataset is benchmarked using three models and two feature descriptors. The benchmarking process is described in detail in the supplementary materials.
clarity: The paper is well written and easy to follow.
relation to prior work: The paper clearly positions itself compared to prior work and explains how it is novel.
additional feedback: it would be beneficial to include visual examples of your data. This could be spectrograms highlighting the differences between audio clips with and without mosquitos, or different kinds of mosquitos. This would fit into the supplementary materials.
review_title: Review of HumBugDB

Meta-review: The data provided in the paper is of relevance for the NeurIPS data track. Based on reviewers opinions, and in particular after discussion with the authors, the paper achieves the minimum score required for publication at NeurIPS data track.
Decision: Accept