SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Mean Predictor is the simple mean value of all visible samples which is used as the value of all the held-out locations. The mean value of all visible  $\mathrm{PM}_{2.5}$  locations C is used as the value of the held-out  $\mathrm{PM}_{2.5}$  locations P.
$$
P M _ {2. 5} ^ {p} \leftarrow m e a n \forall p \in P, \text {w h e r e} m e a n \leftarrow \frac {1}{| C |} \sum P M _ {2. 5} ^ {c} \forall c \in C
$$
(b) Inverse Distance Weighting (IDW) is the weighted average value of all visible C samples in terms of distance, which is used as the value of the held-out P locations.
$$
P M _ {2. 5} ^ {p} \leftarrow \sum \frac {P M _ {2 . 5} ^ {c}}{F \left(d _ {c p}\right)} \forall c \in C \forall p \in P, \text {w h e r e F i s a l i n e a r f u n c t i o n o n d i s t a n c e d}.
$$
(c) Random Forest (RF) is a non-linear model capable of modeling complex spaces. It is known to perform efficiently on non-linear regression tasks, using an ensemble of multiple decision trees, taking the final output as the mean of the output from all trees.
(d) XGBoost (XGB) iteratively combines the results from weak estimators. It uses gradient descent while adding new trees during training.
(e) ARIMA or Auto-Regressive Integrated Moving Average is a statistical time-series forecasting model that uses linear regression. It is configured using parameters  $(p, d, q)$  as:  $p$  is the number of lag observations included in the model,  $d$  is the number of times raw observations are differenced, and  $q$  is the size of the moving average window. We use ARIMA with parameters (3, 1, 1).
(f) N-BEATS is Neural Basis Expansion Analysis for Time Series, a deep learning model for zero-shot time-series forecasting [Oreshkin et al., 2020]. We use the code from Python library "Darts".
(g) Non-Stationary Gaussian Process (NSGP) is a recent gaussian processes baseline [Patel et al., 2022]. It learns a non-stationary covariance [Plagemann et al., 2008] for latitude and longitude and locally periodic covariance for time. In general, Gaussian process Ras and Williams [2005] a.k.a. Kriging is a Bayesian non-parametric model known as the best unbiased predictor in spatial interpolation domain. With only three tunable parameters, it is considered a strong baseline.
(h) Graphsage is a graph neural network model to learn and predict values at unknown spatiotemporal locations [Hamilton et al., 2017]. We transform the PM data to a graph, and use Graphsage for interpolation and missing data imputation. Our graph formulation is available in Appendix A.
4.3 Dataset Pre-processing and Evaluation Metrics for the Analysis
To benchmark interpolation and missing data interpolation ML modeling algorithms, we split the data into two parts for visible and held-out/hidden to use the held-out part for testing purpose. For forecasting based ML modeling, the visible/held-out split is not required, as predictions are made for future timestamps for all locations. For the Delhi dataset, we focus on the data collected from Nov 12, 2020, to Jan 30, 2021, excluding the initial days when there were fewer instruments on the buses and limited sample data. Additionally, we exclude the nightly data between 10 PM IST and 5:30 AM IST when buses remain stationary at confined bus-depots. To facilitate analysis, we divide the geographical area into square spatial grids with a side length of  $1\mathrm{km}$ . These grids are further converted into spatio-temporal cells with a time interval of 30 minutes. To obtain representative PM values, we compute the average of all samples within each spatio-temporal cell. Subsequently, we employ  $K$ -fold cross-validation to partition the data into  $K\mathrm{PM}_{2.5}$  sets for each day, out of which 1 set is reserved for test and others used in training. The results obtained from the Delhi dataset are denoted as Delhi (Day) in the generated plots.
Additionally, we utilize two open-sources  $\mathrm{PM}_{2.5}$  datasets, from Hamilton in Ontario, Canada [Adams and Corr, 2019] and from USA [Bhattacharyya et al., 2022]. For the Canada dataset, we process the data from 18 distinct days in the year 2015 using the same methodology. These results are presented as Canada (Day) in the respective experiments. As the data for Canada exhibits temporal sparsity, we project the data for each year onto a single day and treat it as equivalent to 11 days (from 2006 to 2016). The outcomes of this processing approach are depicted as Canada (Year) in the experiments. For the USA data, we use the available  $\mathrm{PM}_{2.5}$  data across 54 cities from Jan 1, 2019 to Dec 11, 2020, and the results are presented as USA (Day). We benchmark the datasets on Nvidia DGX Workstation (with 4X Tesla V100 GPUs) and the benchmarking code is available at https://github.com/sachin-iitd/DelhiPMDatasetBenchmark.
Evaluation Metrics: The Loss is computed as:
$$
R M S E \left(L _ {p} ^ {\prime}, L _ {p}\right) = \sqrt {\frac {1}{N} \sum_ {i = 1} ^ {N} \left(y _ {i} ^ {\prime} - y _ {i}\right) ^ {2}} \tag {1}
$$
where  $y_{i}^{\prime}$  is the predicted and  $y_{i}$  is the true  $\mathrm{PM}_{2.5}$  value, and  $N$  is the total number of samples.
For each of the  $K$  folds, we separately compute RMSE for that fold, and then plot average with standard deviation bars over the  $K$  folds. The lower RMSE being the better.
Notation: We use T (consecutive) days data for the training and take the next day for test/evaluation. For interpolation benchmarking, Fig. 4a denotes the various subsets of this  $\mathrm{T} + 1$  days data as A, B, C and P, for one fold of  $K = 5$  fold validation. The P is the set over which Prediction is to be done. the C set can be understood as the Context for prediction. For a given fold, A is the visible set with  $80\%$  of all T train days data, B is the held-out set with the remaining  $20\%$  of the T train days data. A  $\cup$  B forms the whole dataset for the T train days. Similarly, C is the visible set with  $80\%$  of the test day data, P is the held-out set with the remaining  $20\%$  of the test day and  $\mathrm{C}\cup \mathrm{P}$  forms the whole dataset for the test day. The held-out locations B and P will be different for each of the  $K$  folds. The
(a) An arbitrary split of A, B, C and P sets.
(b)Visible (A) and Held-out (B) sets over map.
Figure 4: PM Data Splits. (a) A  $\cup$  B forms the whole dataset for the T train days, C  $\cup$  P forms the whole dataset for the test day. Spatial locations of C or P are independent of spatial locations of A or B. (b) Set of A and B spatial locations for 3 PM to 4:30 PM on Dec 15, 2020 from all bus-routes.
A and C though both being visible sets, are independent of each other. The exact number of locations in A, B, C and P change across the  $K$  folds. In Fig. 4b, we show set of A and B spatial locations in Delhi dataset for 3 PM to 4:30 PM on Dec 15, 2020.
4.4 Observations and Inferences
Fig. 5, shows the RMSE for interpolation, using 5-fold cross validation for the two training configurations ACT in Fig. 5a and C in Fig. 5b, for 3 training days. ACT uses the visible set from both training and test days, while C uses only the test day's  $\mathrm{PM}_{2.5}$  visible set. The missing data imputation plots are almost identical to the interpolation plots, so we omit these for space constraints.
(a) Train:ACT, Input:ACT, Predict:P
Figure 5: Interpolation RMSE. Training days' data is used by ML model in (a) and not used in (b).
(b) Train:C, Input:C, Predict:P
Observation 1: Delhi dataset is harder to model. All experiments over Delhi data show higher RMSE and all experiments over Canada and USA data show low RMSE, for both interpolation and forecasting, in Figures 5, and 7. This shows that Delhi data is more challenging for ML modeling, than the currently available PM datasets. Also, the majority of samples present in the data is below  $800\mathrm{PM}_{2.5}$ . Omitting the 29 test samples with higher  $\mathrm{PM}_{2.5}$  for the purview of low-cost sensor accuracy, in Fig. 6 we can see the MAE and RMSE for different  $\mathrm{PM}_{2.5}$  levels for Random Forest (RF) algorithm. We observe that the modeling errors increase with increasing  $\mathrm{PM}_{2.5}$  levels. For existing dataset like Canada with a very low  $\mathrm{PM}_{2.5}$  of mean  $15 \pm 13$  (refer the Table 2), the expected modeling error would automatically be low, making the data easier to model. For the Delhi dataset with high  $\mathrm{PM}_{2.5}$  of mean  $208 \pm 114$ , makes the dataset hard to model.
Observation 2: Learning from data helps in modeling the Delhi dataset. All ML based algorithms show significant improvement over Mean Predictor for Delhi data in Figures 5a, whereas improvement for Canada and USA data over Mean Predictor is not significant. In Fig. 5a, all ML algorithms exhibit less than 40 RMSE while Mean Predictor RMSE is 65.80 for Delhi data (best case improvement is  $66.2\%$  for RF and worst case  $39.3\%$  for IDW). For Canada data, best case improvement is  $\sim 27\%$  and worst case sees no improvement, whereas for USA AQI data, improvement is within  $16\% - 26\%$ .
Observation 3: Traditional ML algorithms do as well as the recent models for the Delhi dataset. Learning from data matters, as the ML based models do better than the mean predictor. But the recent
Figure 6: Errors for different  $\mathrm{PM}_{2.5}$  levels (at intervals of 10) for RF algorithm, with the number of test samples. Modeling shows high errors at high  $\mathrm{PM}_{2.5}$ , making Delhi dataset hard to model due to high  $\mathrm{PM}_{2.5}$  levels.
complex Bayesian models like NSGP, and the neural network based models like GraphSage (for interpolation) and N-BEATS (for forecasting), do not outperform powerful traditional ML models like Random Forest. For instance, RF performs best for interpolation (RMSE 22.24 in Fig. 5), and XGBoost performs best for forecasting (RMSE 84.15 in Fig. 7).
Observation 4: Historical training data adds no value for interpolation. For the spatio-temporal interpolation problem, just using data from the visible set C from test day is enough to predict the held-out P data with low RMSE. For example, the RMSE for RF is similar (22.24) for test day only data C in Fig. 5b and with including train day data ACT in Fig. 5a. And XGBoost is better for C with RMSE 29.73 than for ACT with RMSE 33.24. NSGP is the only algorithm, which sees a huge jump in RMSE when not using training data from past days. Thus PM for a given day is mostly unrelated to PM on past days, and using historical training data has no significant impact on interpolation RMSE.
Observation 5: A location's air quality is related to nearby location. In Fig. 5, for Delhi data, we can observe that IDW performs significantly better than the Mean Predictor. Mean Predictor does not take the distance into account, whereas the IDW gives more weightage to nearby locations data. So, the impact of an adjacent location is significant for IDW w.r.t Mean Predictor, pointing that nearby locations air quality impacts the air quality of a location.
Fig. 7 shows RMSE of forecasting. Graphsage does not work in this setting as it requires a subset of test day's data for edge formation to the data being predicted. So we drop Graphsage, and add two forecasting specific baselines: ARIMA and N-BEATS, that are not suitable for interpolation.
**Observation 6:** Forecasting is a harder problem than interpolation. Forecasting RMSEs are significantly higher than interpolation RMSEs. The best model in forecasting is XGBoost in Fig. 7 with RMSE 84.15, whereas the best model for interpolation in Fig. 5 is RF with RMSE 22.24. Higher forecasting RMSE compared to interpolation also supports that previous day's data has less impact on test day's PM data. Hence forecasting using only past days' data for an unseen future test day is hard.
Observation 7: How time is normalized affects forecasting accuracy. In Fig. 7a, time normalization is done across days, i.e. time starts at 0 on first train day and increases to 1 till last train day. ARIMA / N-BEATS don't normalize the time directly, they take all PM values in a sequence corresponding to time from start to end. RF/XGBoost takes input in random sequence and hence takes the time as a state parameter, which can be normalized from start to end, or for each day. Table 7b compares this time normalization across days (T), to normalizing separately for each day. RF, XGBoost and NSGP show lower RMSE for separate normalization for each day, while IDW does better with normalization across days. This pre-processing step of time normalization therefore should be carefully decided based on the ML algorithm.
(a)
Figure 7: Forecasting RMSE. (a) Normalization is done across all days. (b) Comparison of normalization across all days (T) vs normalization over each day.
Algo	Config	Delhi (Day)
Train	Input	Train	Test
IDW	ABT	ABT	-	86.52
AB	AB	-	270.73
RF	ABT	ABT	8.09	110.49
AB	AB	45.85	89.54
XGBoost	ABT	ABT	33.19	102.68
AB	AB	57.12	84.15
NSGP	ABT	ABT	24.34	95.83
AB	AB	51.90	86.34
ARIMA	ABT	ABT	204.87	148.86
N-BEATS	ABT	ABT	-	106.41
We acknowledge this dataset is limited to 3 months, and we are working to scale in another metropolitan city in India for a comprehensive  $6+$  months data collection and analysis. The bus routes selected by us covered the different types of areas, including green cover, residential, commercial etc, termed as static factors. Hence we have tried our best to reduce the bias towards any particular factor. Still, there is limited spatial distribution due to the bus routes over urban arterial roads. An alternative could have been to put sensors in private vehicles, but it is not reasonable to drive cars just to gather the data. Also, commercial cabs can be used for data collection, which we can check in our future work. We are anyhow moving from a high limitation of few data points in the same vicinity to million+ points.
In Appendix G, we also discuss anomaly detection used to fix any faults in the low cost sensors. We check for samples recorded per minute, number of minutes each device is active in an hour, number of active hours in a day, samples recorded per region, inter-sensor PM values variation, and intra-sensor PM values variation, during dataset collection to effectively keep validating the data being collected.
In our future work, we aim to address the problem of recommending suitable locations for installing new expensive sensors effectively within budget constraints, a challenging task in a developing country like India. By leveraging the insights gained from this research, we strive to optimize the allocation of resources and enhance the efficiency of the monitoring network, further strengthening pollution mitigation efforts.
Delhi-NCR, with its notorious air pollution problem, poses a significant health risk to its population of approximately 46 million individuals. In this paper, we present a novel PM dataset AIRDELHI collected from this region using low-cost IoT devices deployed on public buses. This dataset serves as a valuable resource for environmental researchers and medical practitioners, offering insights into ground-level PM exposure for daily commuters and temporal variations in PM levels over days and weeks. It provides a comprehensive view of spatial variations across different locations within the region.
Through thorough statistical analysis and benchmarking studies, we have established that the released dataset is distinct from any other existing pollution dataset. By comparing the performance of machine learning algorithms on the released dataset against the Canada and USA datasets, we have demonstrated the significant differences in characteristics and challenges associated with the Delhi-NCR dataset. This highlights the need for specialized approaches and tailored solutions to address the unique complexities of air pollution in this region.
The low-cost mobile data collection has the potential to complement the expensive static sensor network in the city, empowering citizens to make informed decisions regarding local PM levels. This includes determining the safety of engaging in outdoor activities, choosing appropriate protective measures such as face-masks or air purifiers, and selecting optimal commuting routes and transportation modes to minimize PM exposure. Such considerations are vital for safeguarding public health and promoting environmental sustainability.
To foster further advancements in the field of environmental sustainability, we release both the code and data associated with this study. This allows researchers to build upon our work, explore new avenues of inquiry, and contribute to the collective understanding and management of air pollution-related challenges.
This work was supported by DST-SERB for funding through IMPRINT-II research grant. We also thank the Delhi Integrated Multimodal Transit System (DIMTS) for allowing us to instrument their fleet.
Howard Seltman. Experimental Design and Analysis. Carnegie Mellon University, 2018.
Komal Shukla, Prashant Kumar, Gaurav S. Mann, and Mukesh Khare. Mapping spatial distribution of particulate matter using kriging and inverse distance weighting at supersites of megacity delhi. Sustainable Cities and Society, 54:101997, 2020.
Ronak Sutaria. Delhi plans mesh of sensors to monitor pollution air hot spots, 2022.
CB Tripathi, Prashant Baredar, and Lata Tripathi. Air pollution in delhi. Current Science, 117(7):1153-1160, 2019.
Yi-Ting Tsai, Yu-Ren Zeng, and Yue-Shan Chang. Air pollution forecasting using rnn with lstm. In 2018 IEEE 16th Intl DASC/PiCom/DataCom/CyberSciTech Conf. IEEE, 2018.
UN. The 17 goals, 2015.
Urbanemissions. Air pollution monitoring 101, 2023.
Weather. Delhi pollution: Capital records 'emergency' levels of air pollution, 2020.
Hongmei Yang, Qin Peng, Jun Zhou, Guojun Song, and Xinqi Gong. The unidirectional causality influence of factors on pm2.5 in shenyang city of china. Scientific Reports, 10(1):8403, 2020.
Wei Ying Yi, Kin Ming Lo, Terrence Mak, Kwong Sak Leung, Yee Leung, and Mei Ling Meng. A survey of wireless sensor network based air pollution monitoring systems. Sensors, 15(12):31392-31427, 2015.
Xiuwen Yi, Junbo Zhang, Zhaoyuan Wang, Tianrui Li, and Yu Zheng. Deep distributed fusion network for air quality prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 965-973, 2018.
Yu Zheng, Furui Liu, and Hsun-Ping Hsieh. U-air: When urban air quality inference meets big data. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1436-1444, 2013.
Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In Proceedings of the 21th SIGKDD conference on Knowledge Discovery and Data Mining. KDD 2015, August 2015.
Tongshu Zheng, Michael H. Bergin, Karoline K. Johnson, Sachchida N. Tripathi, Shilpa Shirodkar, Matthew S. Landis, Ronak Sutaria, and David E. Carlson. Field evaluation of low-cost particulate matter sensors in high and low concentration environments. Atmospheric Measurement Techniques, 2018.
T. Zheng, M. H. Bergin, R. Sutaria, S. N. Tripathi, R. Caldwell, and D. E. Carlson. Gaussian process regression model for dynamically calibrating and surveilling a wireless low-cost particulate matter sensor network in delhi. Atmospheric Measurement Techniques, 12(9):5161-5181, 2019.