Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors present a dataset containing about 20 hrs of acoustic mosquito recordings, out of which 18 hrs are labeled by experts (involving 36 different species). The dataset contains recordings from various locations, including those bred indoors in culture cages and those captured in the wild. The authors have also provided a public codebase that can be used to extract acoustic features and Bayesian NN-based models to separate the mosquito sounds from the background. The dataset can be used by machine learning experts, entomologists, etc., to model mosquito behavior and prevent outbreaks of various mosquito-borne diseases like Malaria.
strengths: 1. The dataset can help in recognizing mosquito species from their sounds and thus can help prevent more dangerous species from breeding. 

2. The metadata for each audio instance also contains location, time, date (seasons, etc. can be inferred), etc. This can help analyze mosquito patterns and can probably be used beyond machine learning.

3. The data is collected in a wide range of conditions including laboratory settings and in-the-wild settings.  

4. The authors use completely unseen test sets which were collected in very different locations under different conditions from what seen by the model during training. The evaluations of future works can potentially be very stable and point towards robustness.

5. The mobile app and database schema is very clearly explained. The codebase and the dataset are public.
weaknesses: 1. The authors should describe the modifications they made to a bed net that attracted mosquitos to fly close to the smartphone. The authors should also describe mosquito behavior that they exploited instead of just citing previous research work. I did not find the previous work online and was not able to verify this step.

2. Multiple parts of the data are actually recorded using a Telinga. Some description about this device and its usage for this project will be necessary for Section 3.

3. The baseline models should be renamed appropriately. More baselines like VGG16 and Inception Net should probably be tried. I also believe, finetuning (transfer learning) on VGG16 trained on other audio-related tasks will be interesting and can improve performance.

4. The performance of the models also varies widely in the two test sets. Detailed experiments should be done to analyze this. Furthermore, ResNet-18 and ResNet-50 are essentially similar architectures, with ResNet-50 being deeper. However, the trend of both of these models reverses in Set B when compared to Set A for ROC-AOC. Why is this happening? 

5. In line 324, the authors mention prior works which were not mentioned in the table. In my view, even they should be mentioned in the table, and their performance should be reported on this dataset.

6. The species distribution is not clear in the graphs given in Figure 6 (supplementary material). Also, it will be really beneficial to have some information about the disease caused by each of these species. By reading the paper, I felt that only malaria-causing species were considered while opportunities existed to focus on other diseases like Dengue.

7. No clear policies and strategies for dataset maintenance and update are mentioned in the paper. The dataset can and should be improved over time with more species, more data per species, etc.

Overall this dataset suffers from baseline evaluations which are necessary to understand the usability of the dataset. I would ideally like to see more rigorous evaluations and more experiments to focus on how the dataset can be used.

correctness: The collection technique and post-processing steps are correct
clarity: The paper is not too well written. A table to compare past datasets with the proposed one will be a great addition. The graphs in supplementary are not easily readable. Overall the writing quality should be improved to make the contributions clearer. 
relation to prior work: Prior work is stated well.
additional feedback: 1. Please put a small table in the related works comparing each dataset in terms of hours, expertize in labelling, background noise present or not, etc. It is very helpful for the reader to get the summary along with the long text in the related works section.

2. I did not understand the usage of Keras BNN and PyTorch ResNet BNN. Both the libraries support similar applications and could have been implemented using a common library.
review_title: An interesting dataset containing recordings from different species of mosquitos. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper provides an acoustic dataset of mosquito sounds and a couple of models for distinguishing background noise from signals. The signals were recorded with wild and culture mosquitos for a large number of species. The suggested models perform well in one test set partition and quite poorly in another one. I believe the main idea of releasing the data is to attract researchers and practitioners to test their models in the challenging partition. 
strengths: The proposal has several strengths, including:

- The application is socially noble since mosquito-borne diseases affect a large number of people.

- Successful models can have a significant impact on Entomology and directly aid people working in mosquito surveillance.

- The data seems well-organised and curated. It results from a significant amount of non-trivial work by a group of people with different skills.

- The paper is well-written with nice illustrations. There is some space for clarity improvement.
weaknesses: The main paper weaknesses are:

- There is a tremendous mismatch between the paper motivation and the technical part. The motivation is about mosquito surveillance, including automatic recognition of species. The benchmark is about differentiating mosquito signals from background noise, a much less interesting task.

- The paper lacks a clear explanation of why Test B is so challenging that makes the tested models fail. How is this data partition relevant?

- It seems to be that text is not fair with the relevant literature. The work of Fanioudakis with pseudo-acoustic sensors produce data without background noise and, therefore, would not need to solve the task the released dataset focuses on solving.
Some very relevant work is missing.

correctness: Overall, the dataset and experiments seem to be well-designed and executed. Regarding the data, I do not understand the passage regarding BNN detection on page 4. It seems the positive signals were automatically segmented what may impose a sampling bias on them. It is unclear if many positive passages were left undetected and may appear as negative samples in the dataset.

Also, the text needs to explain better what makes the B partition challenging. The data partition in training and test sets is unclear. It would be much more interesting if the test data would resemble a scenario of application in the real world.
clarity: The clarity of the paper is good. The text is well-written and organised. The figures and tables are clear. The article is well-organised and presented. 

The main article flaw is a tremendous mismatch between motivation and reality. The motivation regarding mosquito recognition is highly engaging, but the dataset is about background noise identification. There is nothing in the text discussing data, features or models for species identification. 
relation to prior work: The article covers a relevant part of the literature, but it misses a few spots. One example is that the first set of pseudo-acoustic data was released as part of a machine learning competition and is available at the UCR Flying Insect Classification Using Inexpensive Sensors website.

I disagree with the paragraph (85-92) about the work of Fanioudakis et al. Could you explain what "lack of typical background acoustic properties" means? Their work sounds like a better approach for collecting this data since those sensors are insensitive to external factors that generate background noise. Therefore, the problem this paper/benchmark/dataset proposes to solve inexists with data collected with those sensors. Also, the statement about "state-of-the-art" models is very misleading, given the references are for music recognition models, which are naturally designed to work with long samples. However, there are plenty of models that can work with shorter signals. 
additional feedback: See my comments above.
review_title: Review for HumBugDB: a large-scale acoustic mosquito dataset 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper collected a large-scale acoustic mosquito data set, including 18 hours of data and 36 categories. The data and baseline code are publicly available.
strengths: + The collection and labeling of this acoustic mosquito data set are difficult, and it is beneficial for mosquito-borne disease prevention.
+ The paper is well written.
weaknesses: - The quantification and visualization of statistics and comparisons of datasets are relatively limited.
- I believe such a paper can be well published in a mid-ranked conference but I do not see a sufficient novel contribution to NeurIPS. It is one more vanilla dataset.
- Relevant to researchers in subareas only.
correctness: The benchmark experiment is relatively simple.
clarity: The paper is well written.
relation to prior work: There should be more clear quantification and visualization about the difference from prior work.
additional feedback: See above mentioned.
review_title: HumBugDB: a large-scale acoustic mosquito dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper proposes a better dataset for acoustic mosquito studies, including 20 hours of mosquito audio collected from five countries around the world, with curated labels covering 36 different species and background labels. Using this dataset, the authors also conducted an experiment on the mosquito sound classification task and benchmarked many state-of-the-art algorithms. This paper provides extensive materials for utility code, benchmark code, pretrained models, data release, and additional resources for the data collection process.
strengths: 1) this paper is very well-written, easy-to-read, and providing very complete information about the code, data, data collection, statistics, ethical concerns, etc.
2) the authors clearly discussed and compared to previous works, and showed that the proposed dataset has many advantages over previous works: a) longer audio, b) better/more accurate labeling.
3) the dataset marks a great effort since it spans 5 years in time and the researchers have collected the data from 5 countries around the globe. Also, the annotation is non-trivial. The use of algorithms to aid annotation is smart and contributing to the annotation techniques for future works.
4) the dataset itself is of good quality, and are demonstrated to be useful for the task of mosquito sound classification, which is of great importance for detecting mosquito and thus preventing diseases transmission.
weaknesses: I don't have any big concern preventing me from accepting this paper, but below are some minor points if authors want to further improve the work:
1) Can you discuss the data bias? I understand that it's not possible to collect such natural data without any bias. But some discussion about the data bias (e.g. different countries, data imbalance on species), and how would it affect the network training for applications, and how do you practically address these data bias issues, will be very interesting to the readers.
2) Are there other/more applications that your data allows? It would be good to discuss more applications and do more benchmarking on them if possible.
correctness: the claims/dataset construction/evaluation methods are correct to my best judgment.
clarity: yes.
relation to prior work: yes.
additional feedback: no.

Post-rebuttal: I appreciate the authors' great efforts in collecting this data and I think this dataset will be very useful. So I would like to keep my rating.
review_title: An excellent submission with new data and benchmark for acoustic mosquito study

Meta-review: Although the paper proposed a dataset for an extremely important and relevant issue, reviewers felt that the final task the authors wanted to solve was not evaluated. There is a gap between the motivation (insect surveillance) and the task that is evaluated using the dataset (background removal).
Decision: Reject