Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper collected a new dataset of particular matter in Delhi-NCR, which can be used to study air pollution. Such data is collected using vehicle-mounted mobile sensors on buses rather than static sensors.

The contributions of this paper include:

1. The motivation of this paper is good. Air pollution is a big problem in developing countries and raised concerns all over the world. Studying this issue from a machine learning perspective is good and interesting.

2. The data collected process is introduced clearly. The data has been analyzed and compared with that obtained by static sensors.

3. The new dataset introduced new machine learning tasks, including spatio-temporal interpolation and anomaly detection in IoT networks, which are studied in this work.
strengths: Overall, I think this paper studies an interesting problem that connects machine learning with air pollution. 

The strengths include:

+ A new dataset recording the data of particular matter in Delhi-NCR. The collection process is new compared with using static sensors, whose methodology could be further adopted in other countries/areas.

+ A detailed analysis of the dataset, including the comparisons with static sensors, quality of the data, etc.

+ The contribution could be significant since it uses machine learning to solve social problem.

+ Two new tasks are introduced using this dataset.
weaknesses: Although this paper proposes to study two tasks, i.e., spatio-temporal interpolation and anomaly detection in IoT networks, using the new dataset, these tasks seem to be less useful in practice. It may need to predict the PM for other locations, but I might think this dataset can have bigger impacts and solve larger problems than the existing ones. For example, can the data be analyzed with other information (weather, transportation information, etc.) to find important factors that affect the air pollution level. And can the findings suggest ways to control air pollution? I think the longstanding goal is to use machine learning techniques for making the air pollution better.

Another issue is about the generalizability of the proposed datasets/method to other scenarios, e.g., time and location. Since it is collected within a limited period (three month) and only in Delhi-NCR, can it be representive enough?
correctness: The construction process is introduced clearly. I think it is sound.
clarity: Yes, the paper is well written mostly.
relation to prior work: The related work is not introduced. I'm not sure about the comparison with related work. Is there any other dataset doing a similar thing? 
additional feedback: I suggest the authors to further consider more tasks about this dataset to improve the potential impacts.
review_title: A new dataset of particular matter

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper introduces a Particulate Matter (PM) dataset for Indian National Captial Region recorded over three months over a large area using bus-mounted sensors. The dataset captures the spatio-temporal PM trends in one of the most populated regions when the pollution peaks. The paper also benchmarks a few baselines over this dataset. The dataset is super-relevant to the developing world.
strengths: - Public PM datasets are available for only two relatively pollution-free cities. 
- The use of bus-mounted sensors instead of static sensors captures better spatio-temporal data compared to static sensors. They also capture the ground statistics instead of the statistics at the top of the tower.
- Experiments bench-marking different models in Table 3 are also lovely.
- The paper is well-organized.
weaknesses: - One of the major concerns of the dataset is that it only provides PM statistics. Table 3 of [1] contains several different pollutants.  Since the authors intend to add other pollutants such as SOx, NOx, and COx in the future, I feel that this paper submits an ongoing work, and the work is not complete. Therefore, I advise the authors to get the data for all other pollutants and then re-submit the paper. 

- Do authors plan to add the ozone measurement as well as done in [2]?

- Comparison of statistics with other public and non-public air pollution datasets based on the number of pollutants covered, area mapped, sensors/vehicles used, and monitoring days in tabular form would be a nice addition .

- Would you mind adding a plot that compares the pollution levels across different datasets similar to Table 3 of [1]?

- Another potential problem is the number of static towers available to compare with. The authors only found three towers to compare the bus measurements, which is too small to conclude.

- Table 1 only compares baselines over the new NCR dataset. The addition of baselines for publicly available air pollution datasets would be nice. This experiment also lets us infer whether the performance of models depends on the pollution levels. 

- The authors only compare the GPR-based methods with the vanilla neural networks (which do not output variance) with the MSE loss (which is outlier sensitive). As the dataset contains outliers, a more fair baseline to compare would be to use likelihood estimation methods [3,4] that tolerate outliers better and Bayesian neural networks. 

- The authors do not report the variation across the runs in Table 1 by running the experiment multiple times.  

References-
[1] A mobile air pollution monitoring dataset, Adams et al., Data 2019

[2] Sensing the air we breathe, Li et al., AAAI 2012

[3] What uncertainties do we need in computer vision, Kendall et al., NeurIPS 2017

[4] LUVLi face alignment, Kumar et al., CVPR 2020
correctness: Please see the weaknesses section
clarity: Yes, the paper is clear and well-organized. 
relation to prior work: No. Comparison of statistics with other public and non-public air pollution datasets based on the number of pollutants covered, area mapped, sensors/vehicles used, and monitoring days in tabular form is missing.
additional feedback: Since the authors intend to add other pollutants such as SOx, NOx, and COx in the future, I feel that this paper submits an ongoing work, and the work is not complete. Therefore, I advise the authors to get the data for all other pollutants and then re-submit the paper.
review_title: PM Dataset for Indian NCR

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: A particulate matter (PM) dataset is collected with vehicle mounted IoT devices in Delhi, India. Compared with other mobile monitored air pollution works, the proposed dataset has different distribution of PM values as the area under their study is an air pollution hotspot. Due to local constraint, the data were collected using 4G cellular connections on public buses. The dataset is also large in scale and the spatio-temporal characteristics are available. A number of use cases like sample modeling and anomaly detection are discussed. 
strengths: 1. The dataset is large in both spatial and temporal scales. 
2. A wide range of use cases by environmentalists and ML researchers are discussed and analyzed. 
3. The GPS information is provided and the comparison between static and mobile sensors are given. 
weaknesses: 1. There is no statistical comparison with previous PM datasets such as [13] and [14], which makes it difficult to justify the difficulty and significance of the proposed benchmark. 
2. The IoT units were installed in the public buses for data collection. There is no explanation on how they gained the permission. The ethical issues might be a concern. 
3. There is no clear introduction to the IoT devices used and analysis on their accuracy. 
4. The audience is small compared to other CV and NLP based datasets. 
5. It might be better to provide the ground truth of test set separately or create a public submission system/leaderboard for fair comparison.
6. What is the plan for maintenance and licensing?
correctness: The dataset is constructed in a sound way. The evaluation methods make sense. Multiple experimental designs are provided. 
clarity: The paper is well written with clear motivation. I would suggest to create a separate table to compare the statistics with existing public benchmarks. 
relation to prior work: As mentioned in the summary, compared with other mobile monitored air pollution works, the proposed dataset has different distribution of PM values as the area under their study is an air pollution hotspot. Due to local constraint, the data were collected using 4G cellular connections on public buses. 
additional feedback: Please refer to my feedback above.
review_title: Large-scale dataset but no quantitative comparison with other publicly available benchmarks

Meta-review: A strength of the paper is that it provides a new engaging dataset for a pertinent societal problem. However, the dataset is limited and feels like too much of a work in progress. For example, the dataset only covers three months of time for a very specific region. Moreover, the authors have future plans for other associated data to be linked and additional transports to include the sensors. Thus, it looks promising but premature. We encourage the authors to include a solid comprehensive set of data/sensors for a longer period of time and resubmit the work.
Decision: Reject