SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Software & Hardware All of our models, experiments, training procedures and data handling are implemented in Python using PyTorch Paszke et al., 2019 and Lightning Falcon et al., 2019, and logged via Weights and Biases Biewald, 2020. We used NVIDIA A40 GPUs situated in the NAISS cluster Alvis (see acknowledgements). The experiments reported here, including hyperparameter searches, took 1278 hours. With debugging, re-runs due to errors etc., computations took about four times as long.
5.1 Model and training hyperparameters
Unless mentioned otherwise, we use the default hyperparameters of the respective implementations. Loss functions and learning rates are chosen via coarse grid searches. Please see the supplementary material for details. We always use the PyTorch implementation of the AdamW optimizer Loshchilov et al., 2019 with the default parameters  $\beta_{1} = 0.9, \beta_{2} = 0.999, \lambda = 0.01$ . We keep the model with the highest validation loss, evaluated after each training epoch and train for a total of  $10k$  steps.
For the logistic regression, we use a Dice loss with learning rate 0.1. For the U-net, we use the SMP implementation Iakubovskii with a ResNet18 backbone and train it with a Dice loss with learning rate 0.001. For ConvLSTM and UTAE, we use the implementations VSainteuf, 2023 by the UTAE Garnot et al., 2022 authors. For ConvLSTM, we use a Jaccard loss with learning rate 0.01. For UTAE, we use a weighted cross-entropy and learning rate 0.01. The weight for the fire class is the inverse relative frequency of the fire class in the used training set.
5.2 Dataset preparation
We split the data into train/val/test by using two years for the train set and one each for validation and test set. For all experiments, we perform a full 12-fold cross-validation over all possible permutations of years assigned to train/val/test set. This is necessary, because of the large differences between the yearly data distributions (see Table 1 and Table 2, more details in the supplementary material).
For fair evaluation, we need to ensure that the test data is the same across all experiments. However, the number of data points per fire depends on the size of the temporal window  $w_{t}$ , i.e. the number of observed days used as input. E.g. for  $w_{t} = 5$ , we would use the first five days as input, and the binary fire mask of the sixth day as the first target, while for  $w_{t} = 1$ , we would use the second day as our first target, leading to differences in the test set. The reason for this discrepancy is that we only use real data as input to our models, since it is realistic to have access to past data, instead of padding the input with empty frames. In our experiments, we use a maximum  $w_{t} = 5$ , meaning that the first target used in evaluation belongs to day six. For evaluation on the test set to be fair across different values of  $w_{t}$ , we have to leave out initial data points in experiments with  $w_{t} < 5$ , to ensure that all testing starts on the target of day six, for each fire. During training, we instead use all of the data. For details on preprocessing and augmentations, please see the supplementary material.
To evaluate models, we use the test set average precision (AP). It summarizes the precision-recall curve in a single number and is preferable to metrics like AUROC in cases of imbalanced datasets, like WILDFIRESPREADTS. Compared to the F1 score, it considers all possible thresholds instead of just one. For the persistence baseline, we still provide the F1 score, to give readers who are more familiar with this metric an intuition of how good the corresponding AP scores are.
5.3 Fire persistence baseline
As a simple baseline, we use fire persistence, as in Huot et al., 2022. It consists of predicting the same active fire for tomorrow as was observed today. Table 2 shows that the performance of this baseline varies strongly between the different years. Together with the lower number of fires for 2019, this indicates that the difficulty of predicting wildfire spread might vary accordingly.
Since our data includes the time at which each active fire pixel was last detected, we can construct a more fine-grained variant of this baseline, using only pixels from the last detection of the day. If a pixel was detected as active fire early in the day, but not in the last detection of the day, it might have stopped burning. The results in Table 2, however, show a worse performance for this fine-grained baseline. The causes could be: 1. The later detections did not cover the exact same area as the earlier ones, thereby leaving out some burning pixels that were out of view. 2. The detections are noisy and might fail to recognise some of the pixels as burning. This could be caused by different view angles between observations, or temporal variations in fire intensity or cloud/smoke cover.
To investigate how useful different features are, we combine the active fire masks with different feature groups. We train the U-Net on the resulting feature set, varying the numbers of leading days of observations. In preliminary experiments, we found that the U-Net is very easy to optimize, even though it does not explicitly model the temporal dimension. Explicit spatio-temporal models proved difficult to optimize in some cases, which is why we use the U-Net for the ablation studies.
Ablation: Vegetation Table 3 compares the common vegetation indices NDVI and EVI2 with the daily VIIRS reflectance product that we add in this dataset. For one and two days, VIIRS performs better than NDVI, which was the best one-feature solution in Huot et al., 2022, Table V. For three to five days of input observations, VIIRS, NDVI and EVI2 perform similarly, although the exact ranking fluctuates. This strong performance of VIIRS on shorter time series might be caused by the higher temporal resolution, getting daily updates instead of every eight days, and VIIRS bands also offering information about the presence of clouds and smoke. Observing multiple leading days also improves performance for all vegetation features, validating the addition of the temporal dimension.
Ablation: Non-vegetation In Table 4, we test all non-vegetation features in the same setting. We see that topography features perform best, though still worse than the VIIRS features. Again, increasing the number of observed leading days increases the model's performance. Notably this is also true for static features like topography. This indicates that the additional active fire observations contribute some of the improvement, possibly by indicating already burnt areas. Each combination of fire mask and individual feature group and outperforms the persistence baseline that reaches a mean AP of 0.189. Even the worst performing result in this ablation study, ERC & drought, with a single input observation, has a mean AP of 0.291. We also observe that performance can decrease when adding more time steps, even though the model could just learn to ignore the additional data and retain its performance. This could be an issue of optimization difficulty in high-dimensional feature spaces.
Table 3: Ablation: Vegetation We compare EVI2, NDVI and VIIRS reflectance as sources for vegetation data, combined with fire masks. The models predict the wildfire spread 24h ahead, based on one to five days of observations. The performance displayed is the mean test set average precision  $\pm$  the standard deviation, computed in a 12-fold cross-validation over years. Higher is better.
Input days	Features
EVI2	NDVI	VIIRS
1	0.302 ± 0.096	0.298 ± 0.091	0.314 ± 0.093
2	0.313 ± 0.091	0.314 ± 0.085	0.319 ± 0.087
3	0.321 ± 0.092	0.318 ± 0.087	0.320 ± 0.085
4	0.319 ± 0.084	0.325 ± 0.087	0.323 ± 0.091
5	0.318 ± 0.085	0.319 ± 0.080	0.319 ± 0.087
Table 4: Ablation: Non-vegetation We compare all non-vegetation features, individually combined with the fire masks as input features. The models predict the wildfire spread 24h ahead, based on one to five days of observations. The performance displayed is the mean test set average precision  $\pm$  the standard deviation, computed in a 12-fold cross-validation over years. Higher is better.
Input days	Features
ERC, drought	Landcover	Topography	Weather	Weather forecast
1	0.291 ± 0.088	0.301 ± 0.094	0.306 ± 0.087	0.295 ± 0.091	0.296 ± 0.098
2	0.300 ± 0.088	0.311 ± 0.087	0.315 ± 0.087	0.308 ± 0.089	0.296 ± 0.094
3	0.303 ± 0.096	0.314 ± 0.084	0.317 ± 0.082	0.316 ± 0.087	0.292 ± 0.103
4	0.308 ± 0.087	0.307 ± 0.084	0.321 ± 0.089	0.313 ± 0.093	0.289 ± 0.094
5	0.313 ± 0.086	0.318 ± 0.086	0.317 ± 0.080	0.308 ± 0.091	0.293 ± 0.096
Table 5: Baseline results The models predict the wildfire spread 24h ahead, based on one or five days of observations and different input features. Multi includes vegetation, land cover, topography and weather features. The performance displayed is the mean test set average precision  $\pm$  the standard deviation, computed in a 12-fold cross-validation over years. Higher is better.
Model	Input days	Features	Parameters (All)
Vegetation	Multi	All
Persistence	1	0.193 ± 0.065	0.193 ± 0.065	0.193 ± 0.065	0
Log. Regression	1	0.279 ± 0.092	0.288 ± 0.091	0.286 ± 0.092	361
Res18 U-Net	1	0.328 ± 0.090	0.341 ± 0.085	0.341 ± 0.086	14.4M
Res18 U-Net	5	0.333 ± 0.079	0.344 ± 0.076	0.325 ± 0.108	14.7M
ConvLSTM	5	0.306 ± 0.082	0.310 ± 0.085	0.292 ± 0.094	240K
UTAE	5	0.372 ± 0.088	0.350 ± 0.113	0.321 ± 0.135	1.1M
Ablation: Combining features To assess the cumulative contribution of these features, we sequentially combine features. Testing all feature combinations for all time windows is computationally prohibitive. Instead, we sort feature groups in descending order w.r.t. performance on the ablation experiments reported in Table 4, and then add them cumulatively one by one. To reduce experiments, we group all vegetation features together and only compare time windows of one and five days. Table 2 in the supplementary material shows the results in detail. The performance does not always improve with more features or input days. Instead, more features or days sometimes lead to worse results. This is an issue of optimization, possibly made more difficult by the higher input dimensionality, since the model could theoretically just ignore the additional inputs. The best performance is achieved by models using vegetation, land cover, topography and weather data, with five days of input observations. Adding weather forecast, ERC and drought results in worse performance.
5.5 Baseline results
To investigate how different baseline models perform on the dataset, we train them on: a) the best one-feature setting from Table 3 and Table 4, b) the best multi-feature setting (Table 2 in the supplementary material) and c) all available features. The results are displayed in Table 5.
The best result by far is achieved by the spatio-temporal UTAE method, with five input days, using only the vegetation features. This validates moving from mono-temporal methods, that only model spatial interactions, to multi-temporal methods, that model spatial and temporal interactions. The UTAE achieves a maximum AP of 0.372 on the vegetation features (plus the fire masks which are always included). Adding more features reduces the performance to 0.350 for the multi-feature setting and to 0.321 when using all features in the dataset. This hints at optimization difficulties in high-dimensional feature space, similar to what we observed in the ablation studies.
The next-best model is the U-Net with a maximum AP of 0.341 for the mono-temporal and 0.344 for the multi-temporal settings. Notably, the UTAE has about 13 times fewer parameters than the U-Net, but outperforms it by up to  $3.9~\mathrm{pp}$ . The U-net does outperform the ConvLSTM, the second spatio-temporal model we tested, which reaches a maximum AP of 0.310. However, the ConvLSTM also has 61 times fewer parameters than the U-Net. The logistic regression provides a strong baseline, achieving a maximum AP of 0.288. All of the parametric models very clearly beat the parameter-free persistence baseline that has an AP of 0.193, showing that they do indeed learn meaningful features.
6 Opportunities
In this paper, we introduced WildfireSpreadTS, a dataset dedicated to predicting the spread of wildfires based on time-series data. We believe that a wide variety of methods are promising avenues to improve the state of the art of predictions on this important problem.
**Optimization** We have shown that parameter optimization with this high-dimensional, multi-modal dataset can be difficult. While we used standard techniques, more sophisticated methods for optimization with high-dimensional data could improve upon these issues.
Self-supervised pretraining Self-supervised learning (SSL) has proven to be useful for satellite observations Ayush et al., 2021; Gerard et al., 2022; Cong et al., 2023. Additionally, Kang et al., 2021 found that SSL pre-training is beneficial for class-imbalanced datasets. Applying this pretraining could alleviate the optimization difficulties that we encountered.
Noisy labels Since the active fire detections are impacted by various noise sources in the acquisition process, using methods that are tolerant towards label noise Song et al., 2022; Englesson et al., 2021 could improve the robustness of results. Alternatively, uncertainty estimation Gawlikowski et al., 2022 or generative modeling could be approaches to deal with the fact that the process of fire spreading likely contains some stochasticity at the resolution that we are working with.
Separating static and dynamic features While this dataset is structured such that every day provides all features, some of them do not actually change on a daily basis. Treating these static and dynamic features separately, as done by Eddin et al., could facilitate the optimization problem.
7 Access to the dataset
The most up-to-date information on the dataset, as well as the code used in this paper, can be found at https://github.com/SebastianGer/WildfireSpreadTS. This includes a PyTorch dataset class, and a Lightning data module. Our code is available under the MIT license.
The full dataset Gerard et al., 2023 of about 50GB is available as a collection of GeoTIFF files under the CC-BY-4.0 license at https://doi.org/10.5281/zenodo.8006177. For training, we recommend converting the dataset to HDF5. The corresponding code is available in our repository.
This work is funded by Digital Futures in the project EO-AI4GlobalChange. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council through grant agreement no. 2022-06725.
Abatzoglou, John T. (2013). "Development of gridded surface meteorological data for ecological applications and modelling". In: International Journal of Climatology 33.
Artés, Tomás, Duarte Oom, Daniele de Rigo, Tracy Houston Durrant, Pieralberto Maianti, Giorgio Libertá, Jesús San-Miguel-Ayanz (2019). "A global wildfire dataset for the analysis of fire regimes and fire behaviour". In: Scientific Data 6.
Ayush, Kumar, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon (2021). "Geography-Aware Self-Supervised Learning". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10181-10190.
Bali, Shreya, Sydney Zheng, Akshina Gupta, Yue Wu, Blair Chen, Anirban Chowdhury, Justin Khim (2021). "Prediction of Boreal Peatland Fires in Canada using Spatio-Temporal Methods". In: Climate Change AI. ICML 2021 Workshop on Tackling Climate Change with Machine Learning. Climate Change AI. URL: https://www.climatechange.ai/papers/icml2021/12 (visited on 01/19/2023).
Ban, Yifang, Puzhao Zhang, Andrea Nascetti, Alexandre R. Bevington, Michael A. Wulder (2020). "Near Real-Time Wildfire Progression Monitoring with Sentinel-1 SAR Time Series and Deep Learning". In: Scientific Reports 10.1. Number: 1 Publisher: Nature Publishing Group, p. 1322. ISSN: 2045-2322. DOI: 10.1038/s41598-019-56967-x. URL: https://www.nature.com/articles/s41598-019-56967-x (visited on 11/10/2022).
Biewald, Lukas (2020). Experiment Tracking with Weights and Biases. URL: https://www.wandb.com/.
Bolt, Andrew, Carolyn Huston, Petra Kuhnert, Joel Janek Dabrowski, James Hilton, Conrad Sanderson (2022). "A Spatio-Temporal Neural Network Forecasting Approach for Emulation of Firefront Models". In: 2022 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), pp. 110-115. DOI: 10.23919/SPA53010.2022.9927888. arXiv: 2206.08523 [cs]. URL: http://arxiv.org/abs/2206.08523 (visited on 01/16/2023).
Bradshaw, Larry S., John E. Deeming, Robert E. Burgan, Jack D. Cohen (1984). The 1978 National Fire-Danger Rating System: technical documentation. U.S. Department of Agriculture, Forest Service, Intermountain Forest and Range Experiment Station. DOI: 10.2737/int-gtr-169. URL: https://doi.org/10.2737%2Fint-gtr-169.
Burge, John, Matthew R. Bonanni, R. Lily Hu, Matthias Ihme (2022). Recurrent Convolutional Deep Neural Networks for Modeling Time-Resolved Wildfire Spread Behavior. arXiv: 2210.16411 [cs]. URL: http://arxiv.org/abs/2210.16411 (visited on 12/13/2022).
Clough, Shepard A., Mark W. Shephard, Eli J. Mlawer, J. S. Delamere, Michael J. Iacono, Karen Cady-Pereira, Sid-Ahmed Boukabara, Patrick D. Brown (2005). "Atmospheric radiative transfer modeling: a summary of the AER codes". In: Journal of Quantitative Spectroscopy and Radiative Transfer 91.2, pp. 233-244. ISSN: 0022-4073. DOI: 10.1016/j.jqsrt.2004.05.058. URL: https://www.sciencedirect.com/science/article/pii/S0022407304002158 (visited on 06/05/2023).
Cong, Yezhen, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, Stefano Ermon (2023). SatMAE: Pre-training Transformers for Temporal and
Song, Hwanjun, Minseok Kim, Dongmin Park, Yooju Shin, Jae-Gil Lee (2022). "Learning From Noisy Labels With Deep Neural Networks: A Survey". In: IEEE Transactions on Neural Networks and Learning Systems. Conference Name: IEEE Transactions on Neural Networks and Learning Systems, pp. 1-19. ISSN: 2162-2388. DOI: 10.1109/TNNLS.2022.3152527.
Subramanian, Sriram Ganapathi, Mark Crowley (2018). "Using Spatial Reinforcement Learning to Build Forest Wildfire Dynamics Models From Satellite Images". In: Frontiers in ICT 5. ISSN: 2297-198X. URL: https://www.frontiersin.org/articles/10.3389/fict.2018.00006 (visited on 01/16/2023).
Sulla-Menashe, Damien, Josh M. Gray, S. Parker Abercrombie, Mark A. Friedl (2019). "Hierarchical mapping of annual global land cover 2001 to present: The MODIS Collection 6 Land Cover product". In: Remote Sensing of Environment 222, pp. 183-194.
Tavakkoli Pirililou, Sepideh, Golzar Einali, Omid Ghorbanzadeh, Thimmaiah Gudiyangada Nachappa, Khalil Gholamnia, Thomas Blaschke, Pedram Ghamisi (2022). "A Google Earth Engine Approach for Wildfire Susceptibility Prediction Fusion with Remote Sensing Data of Different Spatial Resolutions". In: Remote Sensing 14.3. Number: 3 Publisher: Multidisciplinary Digital Publishing Institute, p. 672. ISSN: 2072-4292. DOI: 10.3390/rs14030672. URL: https://www.mdpi.com/2072-4292/14/3/672 (visited on 08/13/2023).
Tymstra, Cordy, Robert W. Bryce, B. Mike Wotton, Stephen. W. Taylor, O. Bradley Armitage (2010). Development and structure of Prometheus: the Canadian Wildland Fire Growth Simulation Model. Vol. 417. ISSN: 0831-8247. ISBN: 978-1-100-14674-4. URL: http://cfs.nrcan.gc.ca/publications?id=31775 (visited on 01/30/2023).
Vermote, Eric, Belen Franch, Martin Claverie (2016). VIIRS/NPP Surface Reflectance Daily L2G Global 1km and 500m SIN Grid V001. DOI: 10.5067/VIIRS/VNPO9GA.001.
VSainteuf (2023). Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021). URL: https://github.com/VSainteuf/utae-paps (visited on 06/05/2023).
Wooster, Martin J. et al. (2021). "Satellite remote sensing of active fires: History and current status, applications and future requirements". In: Remote Sensing of Environment 267, p. 112694. ISSN: 0034-4257. DOI: https://doi.org/10.1016/j.rse.2021.112694. URL: https://www.sciencedirect.com/science/article/pii/S0034425721004144.
Zhao, Yu, Yifang Ban (2022). "GOES-R Time Series for Early Detection of Wildfires with Deep GRU-Network". In: Remote Sensing 14.17. Number: 17 Publisher: Multidisciplinary Digital Publishing Institute, p. 4347. ISSN: 2072-4292. DOI: 10.3390/rs14174347. URL: https://www.mdpi.com/2072-4292/14/17/4347 (visited on 11/09/2022).