SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SatMAE [15]: This is an unsupervised pre-training framework based on Masked Autoencoders (MAE) for tasks related to remote sensing, specifically for temporal or multi-spectral satellite imagery data. It has been shown to perform well on supervised and transfer learning tasks. We apply transfer learning to our dataset using SatMAE pre-trained model on the fMoW [13] dataset. We use the pre-trained model as a feature extractor and train a dense layer to predict the species encounter rates.
ResNet-18 [27]: CNNs have been widely used as feature extractors for remote sensing imagery. We use ResNet-18 [27] architecture, using the RGB and NIR bands reflectance data as input and we initialize the network with ImageNet pretrained weights. Since ImageNet has only RGB bands, the initialization of the first layer for the NIR band is performed by sampling from a normal distribution parameterized by the mean and standard deviation of the layer's weights for the other bands. We also experiment with RGB true color images as an input and compare the results against reflectance data.
5.2 Including geographical information
We propose to make the CNN baseline model spatially explicit to account for the geographical ranges of some species. For example, Eastern and Western wood p ewes have the same habitat but are found in different regions and do not co-occur. A model blind to geographical information might predict both species in the same place, which is undesirable when producing maps for conservation planning, and could also impede training. We propose to use range maps from eBird, constructing binary masks to zero-out the predicted encounter rates of species when a given location falls outside of their geographical range. For location  $l$ , the range mask  $r_l$  is a vector of size equal to the number of bird species, and  $r_l^i = 0$  if  $l$  is outside of species  $i$  geographical range, and  $r_l^i = 1$  otherwise. If the range map is not available for a species, the predictions are left untouched ( $r_l^i = 1$  for all species). For species  $i$ , if  $z_l^i$  is the logit, the final prediction is then  $\hat{y}_l^{(i)} = \sigma (z_l^i)*r_l^{(i)}$ .
5.3 Evaluation metrics
Beyond optimization for common regression metrics (MSE, MAE), it is desirable for the predicted most likely species in a given hotspot to coincide with those that have been observed most frequently. In fact, while we model many species jointly, our targets are zero-inflated due to the absence of species at many hotspots, which is a frequent challenge in SDM [46, 38]. We therefore report top-10 and top-30 accuracies, representing for  $k = 10$  and  $k = 30$  the number of species present in both the top  $k$  predicted species and the top  $k$  observed species, divided by  $k$ . However, the number of species observed varies considerably across hotspots (Figure 3), unlike in the GeoLifeCLEF challenge for which there is only one ground truth species for each hotspot. We therefore also define an adaptive top-  $k$  accuracy as the top-  $k$  value where  $k$  is the total number of species observed at that hotspot.
6.1 Experimental details
In our experiments, we consider a region of interest of  $640\mathrm{m}^2$ , center-cropping the satellite patches to size  $64\times 64$  around the hotspot and normalizing the bands with our training set statistics. When adding environmental data, we match their resolution to that of the satellite images. Random vertical and horizontal flipping, random noise, blur, contrast and brightness were used for data augmentation. Models are trained with cross entropy loss:  $\mathcal{L}_{CE} = \frac{1}{N_h}\sum_h\mathcal{L}_h = \frac{1}{N_h}\sum_h\sum_{s(\mathrm{species})} - y_h^s\log (\hat{y}_h^s) - (1 - y_h^s)\log (1 - \hat{y}_h^s)$ , where  $N_{h}$  is the number of hotspots  $h$ ,  $y$  the predictions and  $\hat{y}$  the ground truth encounter rates.
Satlas and SatMAE models use the true color images as input. They do not support the use of environmental data. The input is further processed by normalizing its values between 0 and 1. For MOSAIKS, we extracted 1024 features for each true color image, combined them with the environmental data and trained an XGBoost regressor on them. For the ResNet-18-based models, we experiment with different inputs: RGB true color images, RGBNIR reflectance values, and RGBNIR reflectance values combined with the bioclimatic and pedologic data. We normalize the bioclimatic and pedologic data variable-wise with training set statistics, aligning them to the satellite images' resolution and stacking the corresponding patches to the images.
All models were trained with batch size 128 using Adam [37] optimizer. Each experiment is run on 3 seeds and results report the average of 3 seeds on the test set. Training, validation and test sets follow the splitting described in Section 4. All ResNet-18 models were trained for 50 epochs. Satlas and SatMAE were trained for 100 epochs, freezing the pretrained model and training only the last layer. More details about experiments including resources used are available in Appendix B.
6.2 Results & discussion
In this section, we present results for all our baseline models using SatBird-USA-summer and SatBird-USA-winter shown in Tables 2 and 3 respectively. Validation set results for both datasets are reported in Appendix C. We also report baselines for SatBird-Kenya in Appendix D.
We observe that models trained with satellite images only, either the true color image or the reflectance values, do not outperform the environmental baseline. However, combining satellite images with environmental rasters results in a significant improvement, in all metrics, with the highest improvement in the top-30 (13%), top-k (4%), and top-10 (8%) metrics in SatBird-USA-summer and improvement in the top-30 (10%), top-k (4%), and top-10 (5%) metric for SatBird-USA-winter, compared to the environmental baseline. This highlights the value of satellite data for our task, and also the importance of using both environmental and remote sensing data. The models using environmental data along with either RGBNIR reflectance images or RGB visual images achieve nearly-equal competitive results, but our best models use RGB images, along with environmental data, titled ResNet-18 (img+env) in Tables 2 and 3. We suggest incorporating other information as input such as land cover and altitude data which can be obtained from publicly available sources [18, 11] for further improvements.
For the transformer-based models, both Satlas and SatMAE do not outperform ResNet-18 when applied to the true color image only. This observation suggests that we may need further finetuning for these models on our data rather than freezing the model weights. We note that Satlas outperforms
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
Mean encounter rates	3.1 ± 0.0	0.9 ± 0.0	26.9 ± 0.0	38.6 ± 0.0	44.8 ± 0.0
Environmental baseline	2.3 ± 0.0	0.6 ± 0.0	42.9 ± 0.1	57.8 ± 0.1	63.8 ± 0.1
Satlas (img)	2.9 ± 0.01	0.9 ± 0.0	30.5 ± 0.1	46.1 ± 0.0	48.6 ± 0.1
SatMAE (img)	3.0 ± 0.0	0.9 ± 0.0	28.9 ± 0.1	43.5 ± 0.1	46.5 ± 0.2
MOSAIKS (img+env)	2.5 ± 0.0	0.7 ± 0.0	41.9 ± 0.0	56.7 ± 0.0	62.2 ± 0.0
ResNet-18 (img)	2.6 ± 0.02	0.8 ± 0.0	35.5 ± 0.1	52.2 ± 0.2	54.5 ± 0.2
ResNet-18 (refl)	2.6 ± 0.01	0.8 ± 0.0	36.1 ± 0.1	53.3 ± 0.1	55.5 ± 0.1
ResNet-18 (img+env)	2.2 ± 0.02	0.64 ± 0.0	46.3 ± 0.2	65.5 ± 0.2	66.9 ± 0.1
ResNet-18 (refl+env)	2.1 ± 0.01	0.65 ± 0.0	45.8 ± 0.1	65.1 ± 0.2	66.6 ± 0.1
ResNet-18 (refl+env+RM)	2.16 ± 0.0	0.65 ± 0.0	45.6 ± 0.2	65.1 ± 0.1	66.7 ± 0.1
Table 2: Test results on the SatBird-USA-Summer: Best results are shown in bold. img refers to using the RGB true color image, refl refers to using RGBNIR reflectance bands, env refers to using the environmental data (bioclimatic and pedologic variables). RM refers to the use of range maps.
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
Mean encounter rates	2.5 ± 0.0	0.7 ± 0.0	27.6 ± 0.0	45.7 ± 0.0	51.4 ± 0.0
Environmental baseline	1.8 ± 0.0	0.4 ± 0.0	48.7 ± 0.0	63.7 ± 0.0	68.5 ± 0.0
Satlas (img)	2.3 ± 0.0	0.7 ± 0.0	31.6 ± 0.1	51.5 ± 0.1	54.1 ± 0.1
SatMAE (img)	2.4 ± 0.0	0.7 ± 0.0	28.6 ± 0.8	50.2 ± 0.1	52.6 ± 0.1
MOSAIKS (img+env)	1.9 ± 0.0	0.5 ± 0.0	47.8 ± 0.0	62.1 ± 0.0	66.4 ± 0.0
ResNet-18 (img)	2.2 ± 0.2	0.6 ± 0.0	35.3 ± 0.4	54.3 ± 0.2	56.9 ± 0.4
ResNet-18 (refl)	2.2 ± 0.0	0.6 ± 0.0	37.1 ± 0.9	56.5 ± 0.9	58.7 ± 0.9
ResNet-18 (img+env)	1.69 ± 0.0	0.4 ± 0.0	51.1 ± 0.3	69.5 ± 0.24	70.9 ± 0.2
ResNet-18 (refl+env)	1.67 ± 0.0	0.4 ± 0.0	50.9 ± 0.1	69.3 ± 0.0.1	70.8 ± 0.1
ResNet-18 (refl+env+RM)	1.69 ± 0.0	0.4 ± 0.0	51.0 ± 0.1	69.4 ± 0.1	71.0 ± 0.1
Table 3: Test results on the SatBird-USA-winter: Best results are shown in bold. img refers to using the RGB true color image, refl refers to using RGBNIR reflectance bands, env refers to using the environmental data (bioclimatic and pedologic variables). RM refers to the use of range maps.
SatMAE, likely because the former was pre-trained on a very large dataset of Sentinel-2 images which is the same source (and has the same resolution) as our images. For SatMAE, the model was pretrained on the fMoW dataset with a higher resolution  $(0.5\mathrm{m})$  than the images in our dataset. MOSAIKS also performs reasonably well compared to other deep models (ResNet-18 for example) for the true color image and environmental raster input, considering that it is a computationally lightweight model.
We also observe in Tables 2 and 3 that using RGBNIR reflectance bands as input yields better results than using true color images (while not using environmental data), given the information added by the NIR channel. This aligns with previous conclusions [4] that remote sensing indices using the NIR band, such as the NDVI improve the performance of SDMs. Moreover, adding range maps, does not hurt the model but it does not seem to add a significant contribution. Range maps can still be useful in cases where the data is not as abundant as SatBird-USA-summer. In particular, we still recommend to use this approach whenever available with smaller-sized datasets. We had experiments with using latitude and longitude as inputs to a location encoder, concatenating their features with the model features but we found lower performance for this baseline (see Appendix B.5).
In this work, we present and release SatBird, a large-scale dataset for species distribution modeling using remote sensing and citizen science data. SatBird includes datasets in the USA for summer and winter seasons and one dataset in Kenya. We detail the process for building SatBird, as well as using it with different models. We also present benchmark results for the dataset on a wide range of popular remote sensing algorithms. Two limitations of the present work include using a single satellite image to represent a season, while the landscape can change over time, and that satellite images are extracted from one recent year, while eBird data is aggregated across multiple recent years.
We are planning a next version of SatBird that will include multiple satellite images for each hotspot to account for changes over time. We also aim to expand SatBird using citizen science data for other organisms, not merely birds (this is more challenging since most such datasets are presence-only).
As SatBird is intended to directly impact biodiversity monitoring, we look forward to integrating the approaches we have introduced into existing tools for ecology and policy. One immediate application for models trained on SatBird is eBird's existing tool that lists the "likely species" in a given area, to be available for poorly monitored locations. This tool currently relies on encounter rates from past checklists and is therefore only available in well-monitored locations. Models trained on SatBird could estimate such encounter rates for poorly monitored locations via remote sensing. We hope such input will be valuable to researchers seeking to understand biodiversity and climate change, as well as policymakers interested in evaluating conservation priorities across different areas of land.
This research was enabled and funded by Mila (mila.quebec). This project was supported by the Canada CIFAR AI Chairs program and a Google-Mila grant. The authors grateful to Sal Elkafrawy for their contribution in adapting the MOSAIKS baseline code and to the Mila IDT team for technical support.
[1] ABA. American Birding Association checklist: Birds of the continental United States and Canada. 2022.
[2] Carlos Abrahams and Matthew Geary. Combining bioacoustics and occupancy modelling for improved monitoring of rare breeding bird populations. Ecological Indicators, 112:106131, 2020.
[3] Eman A. Alshari, Mohammed B. Abdulkareem, and Bharti W. Gawali. Classification of land use/land cover using artificial intelligence (ann-rf). Frontiers in Artificial Intelligence, 5, 2023.
[4] Salvador Arenas-Castro, Adrián Regos, Ivone Martins, João Honrado, and Joaquim Alonso. Effects of input data sources on species distribution model predictions across species with different distributional ranges. Journal of Biogeography, 49(7):1299-1312, 2022.
[5] T. K. Gottschalk Corresponding author, F. Huettmann, and M. Ehlers. Review article: Thirty years of analysing and modelling avian habitat relationships using satellite imagery data: a review. International Journal of Remote Sensing, 26(12):2631-2656, 2005.
[6] Avibase. Avibase - the world bird database, 2023. Accessed: 2023-06-06.
[7] Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. Satlas: A large-scale, multi-task dataset for remote sensing image understanding. 11 2022.
[8] Sara Beery, Elijah Cole, Joseph Parker, Pietro Perona, and Kevin Winner. Species distribution modeling for machine learning practitioners: A review. In ACM SIGCAS Conference on Computing and Sustainable Societies, pages 329-348, 2021.
[9] Elizabeth H Boakes, Gianfranco Gliozzo, Valentine Seymour, Martin Harvey, Chloë Smith, David B Roy, and Muki Haklay. Patterns of contribution to citizen science biodiversity projects increase understanding of volunteers' recording behaviour. Scientific reports, 6(1):1-11, 2016.
[10] Corey T Callaghan, Alistair GB Poore, Max Hofmann, Christopher J Roberts, and Henrique M Pereira. Large-bodied birds are over-represented in unstructured citizen science data. Scientific reports, 11(1):1-11, 2021.
[11] CGIAR. SRTM - CGIAR-CSI SRTM 90m Digital Elevation Database. https://srtm.csi.cgiar.org/srtmdata/, n.d.
[12] Yilin Chen, Zhiyong Jiang, Ping Fan, Per Ericson, Gang Song, Xu Luo, Fumin Lei, and Yanhua Qu. The combination of genomic offset and niche modelling provides insights into climate change-driven vulnerability. Nature Communications, 13, 08 2022.
A Further Dataset Description
A.1 Number of species per hotspot
A well-known issue in species distribution modeling when looking at many species at a time is the zero-inflated nature of the targets, which also appears in our dataset. Indeed, all 670 considered species in the USA and 1054 species in Kenya are never found in the same place together. Figure 3 shows the distribution of the number of species with non-zero encounter rates per hotspot in the SatBird-US-summer and the SatBird-Kenya datasets.
Figure 3: Distribution of the number of species encountered per hotspot for the USA-summer (left) and Kenya (right) training sets. On average a hotspot in these datasets has resp. 48 and 30 different species reported.
A.2 Dataset splits
Table 4 describes the number of hotspots present in each split for each dataset of SatBird. The splits were obtained by following the process described in Section 4.
USA summer	USA winter	Kenya
train	85553	38241	6988
validation	18511	8323	1658
test	18529	6797	1329
Table 4: Number of samples in each split for the three datasets
A.3 Environmental variables
In Table 5, we provide a description of the environmental variables provided in the SatBird dataset. The bioclimatic data is taken from the highest resolution data available (30 seconds resolution, corresponding to about  $1\mathrm{km}2$ ) from WorldClim [20, 23]. Data for the USA dataset was obtained following the method proposed by [14], and we used WorldClim1.4 which has data aggregated from 1960-1990. For Kenya, we extracted data from WorldClim2.1 which has data aggregated for 1970-2000. The soil data comes from SoilGrids and has resolution  $250\mathrm{m}$  [30].
A.4 Distribution of the targets
Figure 4 shows the distribution of species by average encounter rate over the USA-summer training set hotspots. Most species have a low average encounter rate reflects both the zero-inflated nature of the targets. In Figures 5 and 6, we show the distribution of encounter rates for a very common species all across the USA, the American robin (Turdus migratorius) and the California quail (Callipepla californica) whose range only extends to the west of the USA. If we compare the two distributions,
Bioclimatic variables (30 arcsec resolution)
Name	Description
bio_1	Annual Mean Temperature
bio_2	Mean Diurnal Range (Mean of monthly (max temp - min temp))
bio_3	Isothermality (bio_2/bio_7) (* 100)
bio_4	Temperature Seasonality (standard deviation *100)
bio_5	Max Temperature of Warmest Month
bio_6	Min Temperature of Coldest Month
bio_7	Temperature Annual Range (bio_5-bio_6)
bio_8	Mean Temperature of Wettest Quarter
bio_9	Mean Temperature of Driest Quarter
bio_10	Mean Temperature of Warmest Quarter
bio_11	Mean Temperature of Coldest Quarter
bio_12	Annual Precipitation
bio_13	Precipitation of Wettest Month
bio_14	Precipitation of Driest Month
bio_15	Precipitation Seasonality (Coefficient of Variation)
bio_16	Precipitation of Wettest Quarter
bio_17	Precipitation of Driest Quarter
bio_18	Precipitation of Warmest Quarter
bio_19	Precipitation of Coldest Quarter
Pedologic variables (250m resolution)
Name	Description
orcdrc	Soil organic carbon content (g/kg at 15cm depth)
phiiox	Ph x 10 in H20 (at 15cm depth)
cecsol	Cation exchange capacity of soil (cmolc/kg 15cm depth)
bdticm	Absolute depth to bedrock in cm
clyppt	Clay (0-2 micro meter) mass fraction at 15cm
sltppt	Silt mass fraction at 15cm depth
sndppt	Sand mass fraction at 15cm depth
bldfie	Bulk density in kg/m3 at 15cm depth
Table 5: Description of the bioclimatic and pedologic variables extracted from WorldClim1.4 and SoilGrids.
we see that for the California quail, more hotspots have zero encounter rate which is consistent with the fact that its range is smaller. Table 6 shows in which states the species is found, namely California and neighboring states.
Figure 4: Distribution of species by average encounter rate over the USA-summer training set (left). The figure on the right zooms in the histogram for species with encounter rate  $< 0.1$
Figure 5: Encounter rates distribution of the American robin across the USA-summer training set
Figure 6: Encounter rates distribution of the California quail across the USA-summer training set. We whos the full distribution (left) and the distribution for hotspots with non-zero encounter rates(right).
A.5 Dataset format
For each SatBird dataset subset, we provide a .tar.gz file which is organized as follows:
Dataset name (USA-summer, USA-winter, Kenya)  
| - images/  
| - imagesvisual/  
| - corrected_targets/  
| - environmental_data/  
| - train, valid, test csv files  
| - species_list.txt (file with the scientific name of the considered species)
Each hotspot has one corresponding file in each subfolder. images contains the RGBNIR reflectance data in the format of .tif file, imagesvisual has the RGB true color images in the format of .tif file, environmentemnatal_data has the environmental rasters in the format of .npy file, and corrected_targets have the encounter rates for each hotspot in the format of .json file. We provide more details about the files in the Datasheet for SatBird.
A.6 Where to find the dataset
We have released a project website, where you can find links to access dataset and code. Our dataset is available for download at this link: https://drive.google.com/drive/folders/1eaL2T7U9Imq_CTDSSil1ETSDJ1vxi5Wq?usp=sharing. The companion code for the dataset
State	Number of hotspots
California	3248
Washington	962
Oregon	921
Idaho	384
Nevada	168
Utah	65
Montana	10
Table 6: Distribution of hotspots in the USA-summer training set per state, for which the encounter rate (target) for the California quail is non-zero.
preparation pipeline and benchmark can be found here: https://github.com/RolnickLab/SatBird.
We have presented simple baselines to lay the foundations for the task of regressing encounter rates, trying to cover different data inputs and different categories of models (Gradient-boosted regression, CNNs, transformers).
We used an internal cluster in our organization that has enough GPUs. For the deep vision ResNet baselines, on Nvidia A100 GPU, one epoch takes 54 minutes maximum (for the model Resnet-18(refl+env+RM)), with batch size of 128 and using 16 GB of RAM memory. Other models of Resnet-18 took less time. There are experiments where we used more powerful GPUs, resulting in less than 1 day for training 50 epochs. Each baseline was trained 3 times on 3 different seeds. As for the Satlas and SatMAE baseline, each experiment took two days to train on one GPU. We used Pytorch framework [49] in the implementation of our models.
For the MOSAIKS baseline, we extracted 1,024 features for each  $10\mathrm{m}$ -resolution image from Sentinel-2 and tried both XGBoost and Ridge regression on the combination of the environmental and image features obtained with the Random Kitchen sinks methods proposed in MOSAIKS. The MOSAIKS model [55] was originally trained on around  $1\times 1\mathrm{km}256\times 256$  pixels RGB images (resolution of about  $4\mathrm{m/pixel}$ ) and 8,192 features were extracted for each of the training 100,000 images. In our case, we found XGBoost to perform better on the SatBird-USA-Summer dataset and Ridge regression to perform better on the SatBird-USA-Winter and SatBird-Kenya datasets. Each model was trained with 3 different seeds and results of each baseline are reported on the mean of the 3 experiments.
Below, we add additional experiments on SatBird-USA-summer including: 1) hyperparameter search, 2) using pretrained weights from another remote sensing dataset, 3) using different loss functions, 4) training separate models per bird species.
B.1 Hyperparameter search: learning rate
We conducted a hyperparameter search a posteriori for the learning rate on the ResNet-18 (refl + env + RM) baseline for the SatBird-USA-summer dataset for values in the range of  $\langle 1\times 10^{-4}$ $5\times 10^{-3}\rangle$  , and found that a learning rate of  $1\times 10^{-4}$  gave slightly better results. Hence, we used  $1\times 10^{-4}$  in our reported experiments. We show the loss curves for a few models in our hyperparameter search in Figure 7.
B.2 Initializing with pre-trained weights on a remote sensing dataset
We compare initializing the ResNet-18 (refl + env + RM) model with ImageNet weights (as reported in the paper) with an initialization from Seasonal Contrast weights [44]. We find that the models perform similarly as reported in Table 7.
(a) Training loss vs epochs
(b) Validation loss vs epochs
Figure 7: Loss curves for ResNet-18 (refl+env+RM) models trained on SatBird-USA-summer with different learning rates.
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
ImageNet initialization	2.16	0.65	45.7	65.1	66.8
Seasonal Contrast initialization	2.2	0.65	45.6	65.1	66.6
B.3 Comparing different loss functions
Throughout the main paper, our experiments used cross entropy as the loss function as mentioned in Sec 6. We have conducted further experiments using other regression loss functions such as L1 loss and L2 loss. We have also extended our cross entropy loss to focal loss [39] to address the class imbalance. Table 8 shows that cross entropy loss achieves the best accuracies, and even the best MSE over the test set.
B.4 Could the model be better at predicting only one bird instead of the whole distribution?