SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mélisande Teng
tengmeli@mila.quebec
Amna Elmustafa
Mila, Stanford University amna.elmustafa@mila.quebec
Benjamin Akera
Mila, McGill University  
akeraben@mila.quebec
Mila, Google Research, Google Deepmind
hugolarochelle@google.com
Hager Radi Abdelwahed
hager.radi@mila.quebec
David Rolnick
drolnick@cs.mcgill.ca
Biodiversity is declining at an unprecedented rate, impacting ecosystem services necessary to ensure food, water, and human health and well-being. Understanding the distribution of species and their habitats is crucial for conservation policy planning. However, traditional methods in ecology for species distribution models (SDMs) generally focus either on narrow sets of species or narrow geographical areas and there remain significant knowledge gaps about the distribution of species. A major reason for this is the limited availability of data traditionally used, due to the prohibitive amount of effort and expertise required for traditional field monitoring. The wide availability of remote sensing data and the growing adoption of citizen science tools to collect species observations data at low cost offer an opportunity for improving biodiversity monitoring and enabling the modelling of complex ecosystems. We introduce a novel task for mapping bird species to their habitats by predicting species encounter rates from satellite images, and present SatBird<sup>1</sup>, a satellite dataset of locations in the USA with labels derived from presence-absence observation data from the citizen science database eBird, considering summer (breeding) and winter seasons. We also provide a dataset in Kenya representing low-data regimes. We additionally provide environmental data and species range maps for each location. We benchmark a set of baselines on our dataset, including SOTA models for remote sensing tasks. SatBird opens up possibilities for scalably modelling properties of ecosystems worldwide.
Climate change is a major driver of biodiversity loss, affecting ecosystem services which, in turn, impacts various human wellbeing aspects [32, 45]. It is crucial to understand the changing distributions of species globally to inform policy decisions, for example in shaping land use and land conservation choices. However, there remain large knowledge gaps about species distribution, due principally to the amount of effort and expertise required for traditional field monitoring. Furthermore, mapping species habitats at scale can be challenging as habitats have varying scales depending on species.
<sup>1</sup>Project Website: https://satbird.github.io/
For example, song sparrows are found in a wide variety of open habitats, including tidal marshes, agricultural fields, overgrown pastures, forest edges, and suburbs whereas Kirtland's warblers only breed in young jack pine forests. Traditional methods for species distribution models (SDMs) use environmental data to predict the distribution of species across geographic space. However, they generally focus either on narrow sets of species or narrow geographical areas, while species interact with each other, and multi-species modelling is needed for understanding ecosystems. The limitations of traditional methods lie in their computational cost, the insufficient ability of models to account for complex relationships between different variables, and the limited availability of the type of data used [50, 47], at a fine-grained resolution.
Machine learning algorithms for remote sensing have increasingly seen wide applicability across sustainability-related domains (see e.g. [55]) and have been suggested as a promising tool for SDMs [8]. Moreover, the surge of data collection on citizen science platforms along with improved data quality validation processes in recent years offers tremendous opportunity for scientific research, as species observation records from these sources can cover a larger temporal and geographic extent at a finer resolution and at a lower cost than traditional sampling methods. Indeed, citizen science data and remotely sensed ecosystem attributes such as vegetation indices have been shown to improve the performance of models, especially for less widespread species [4]. Recently, the GeoLifeCLEF challenge [41] was introduced with the goal of directly predicting plant and animal abundance from 1m-resolution aerial images, using deep computer vision. However, the GeoLifeCLEF benchmark has proven extremely challenging, potentially since only one species (out of 17,000 overall) is associated with each location, even though numerous other species could likely be found in the same place.
We propose to use remote sensing to infer the joint distribution of many species for a given location, using publicly available citizen science observation records as ground truth. Our approach leverages the fact that a species' presence or absence at a location depends on the ecosystem present there, and therefore the abundances of different species are highly correlated. We present SatBird, a dataset and benchmark for the task of jointly predicting the encounter rates of bird species. Encounter rates are a widely used measure in species distribution modeling, and correspond to the probability of an observer to encounter this species if they visit a given location.
- Proposing and framing the task of predicting bird species encounter rates jointly at a specific location using remote sensing data (Section 2)
- We introduce SatBird, a dataset for the above task, obtained from publicly available bird observation and satellite data sources (Section 4), composed of the following sub-datasets: (i) USA summer dataset, generally corresponding to the breeding season, (ii) USA winter dataset, the nonbreeding season, (iii) Kenya dataset, as an example of a low-data regime.
- We benchmark a variety of popular models on SatBird and show the efficacy of deep computer vision methods for this task (Section 5).
We release the dataset and code for dataset preparation and benchmark making it possible to easily extend the dataset to other regions in the world.
2 Task definition
We propose to predict bird encounter rates from remote sensing data with the goal of completing species distribution mapping in places that have not been surveyed. We use bird sighting records from the citizen science database eBird [36], which has 80 million records of almost all 10000 global bird species. In particular, we leverage observation reports called complete checklists. Each location (termed a hotspot) in the eBird database is associated to a number of complete checklists containing all species a birdwatcher was able to observe at a specific date and time. Complete checklists are presence-absence data -reflecting both the presence of reported species and the absence of nonreported species - and therefore may be considered almost as informative as expert field surveys. If  $h$  is a hotspot, and  $s_1, \ldots, s_n$  the species of interest, then our goal is to build a machine learning model that takes as input a satellite image of  $h$  (and optionally other data) and predicts the vector  $\mathbf{y}^h = (y_{s_1}^h, \dots, y_{s_n}^h)$ , where  $y_s^h$  is the number of complete checklists reporting species  $s$  at  $h$  divided by the total number of complete checklists at  $h$ .
This ratio  $y_{s}^{h}$  can be understood as an encounter rate, the probability for a visitor to observe a species if they visit the same location (hotspot). We aim to jointly predict this quantity for all relevant bird
Figure 1: An overview of data streams for SatBird, as well as inputs and outputs for the task for predicting species encounter rates. Sentinel-2 10m-resolution satellite data can be used along low resolution environmental data as input to a model after matching their resolutions. Labels are derived from eBird complete checklists. Observations of vagrants (migrating birds) in the labels are corrected with range maps from eBird, which can also be incorporated in the model to make it geography-aware.
species. Thus, our task can be considered as a supervised multi-output regression problem. We chose to predict encounter rates as they are an important quantity in ecology, and they are also used extensively in the form of "hotspot bar charts" in the eBird platform in order to summarize the species in a given location for birdwatchers and ornithologists. These bar charts are used to suggest to users what species may be expected in a particular location. However, they rely only on past data, and the suggestions cannot be extended to places that have not yet been visited. Moreover, encounter rates from eBird have not been modeled jointly to take the interaction between species in the same habitat into consideration, which is important to inform species presence estimates. Our goal is to bridge this gap by modeling the encounter rates of many species jointly and predict them in locations with few or no observations. An overview of the interaction between our data and models is shown in Figure 1.
There have been an increasing number of citizen science initiatives to collect species observation data for a variety of taxa in the past decade, from butterflies, to plants, to sharks [26, 58, 51]. Indeed, by crowdsourcing data collection efforts, it is possible to gather data not only over a larger temporal and geographic extent but also at a fine resolution [16]. As a result, the number of papers using citizen science for SDMs has increased at approximately double the rate of the overall number of SDM papers [22]. eBird data has been increasingly used for scientific research on birds, from assessing climate change-driven vulnerability of species, to modelling population change to predicting virus transmission [59, 35, 12]. The eBird status and trends project [24] from the eBird team combines satellite images with raw eBird data and uses statistical models and machine learning to build visualizations and tools to better understand migration, abundance patterns, range boundaries, and other patterns around the world. Other large scale citizen science databases used for SDM include iNaturalist [31] in which users record observations of species by taking geolocated pictures. In total, it has 113 million species observations. However, because it is limited to observations with pictures, it is more prone to sampling biases, both geographically and towards certain species (e.g. larger,
more brightly coloured species [9, 10]). It is also presence-only – that is, while there is data on the presence of certain species at a given location, it is assumed that other species may also be present.
Remote sensing data has been used for a variety of biodiversity monitoring applications, including predicting land cover classes for downstream ecological modeling [3], measuring the size of groups of animals [63], identifying tree species from crowns [61], and localizing bird nesting sites [29]. Bioacoustics has been used for bird monitoring [2] but one of the main limitations relates to the detectability of species, for example in densely populated areas with many anthropogenic sound sources [25]. Therefore, we will focus on discussing related work with remotely sensed imagery. In the GeoLifeCLEF 2020 challenge [14], they proposed a task combining remote sensing and citizen science data, where the goal is to predict the localization of plant and animal species using 1.6M geo-localized observations from France and the USA of 17,000 plant and animal species from aerial images and environmental features. The labels are derived from citizen science databases of presence-only species observations and each location is associated with only one species, limiting the ability to accurately model multiple species at a time. A newer version (2023 edition) of this classification challenge proposes to focus only on plant species and to use Sentinel-2 satellite images rather than aerial images. While each location is still associated with one species, a validation set with presence-absence labels is provided, highlighting the importance of presence-absence data, at least for the evaluation of such species modeling tasks. Methods deriving information from satellite images have also been developed in the context of avian studies but they usually use a suite of carefully selected measures calculated from the imagery rather than leveraging its full potential [5, 21].
A particular field of application that saw the development of large remote sensing images datasets is agriculture, with an emphasis on the temporal resolution for crop monitoring [28, 44, 62]. Recently, a number of methods have been introduced to learn robust representations of remote sensing data, with the goal of using them for a variety of downstream tasks [44, 52, 53]. In particular, MOSAIKS [55] proposes to use a single encoding of satellite imagery for diverse prediction tasks, and Satlas [7] and SatMAE [15] provide pre-trained models on a large number of remote sensing images, arguing that they can be useful feature extractors for remote sensing tasks.
Compared to existing datasets and methods for SDM, our proposed approach has several advantages:
- We fully leverage presence-absence data from a large citizen science database and compute encounter rates such that biases due to the different number of observers visiting locations are mitigated (see Section 2).
- We enable modelling of a wide range of species at a time across large geographical areas. To our knowledge, this is the first attempt at predicting encounter rates for many species jointly, building on recent advances in machine learning for remote sensing.
- Our proposed pipeline makes it easily extendable to other regions in the world as all data sources are open; eBird [36] data is available in all countries in the world and satellite Sentinel-2 data is publicly available.
4 SatBird Dataset
We introduce SatBird, a dataset for the task of predicting encounter rates defined in Section 2 from remote sensing images and environmental data with labels derived from eBird observation reports, with the continental USA and Kenya as regions of interest. Observations in eBird are particularly skewed towards North America, and we choose the USA as a first region of interest, given the extensive and reliable data available on bird observations. We consider the summer season (June-July) and the winter season (December-January) separately. For most North American bird species, the summer reflects breeding season range, while the winter reflects nonbreeding range. Note that we omit the spring and fall migration seasons, in which species distributions may vary dramatically, and which are also often of less interest to ecologists than breeding habitat. In Kenya, we consider the distribution of birds all-year-round as migration is negligible.
In this section, we detail the data sources and data preparation for SatBird for the USA summer and winter seasons and Kenya. Table 1 summarizes the data for each of the considered subsets SatBird-USA-summer, SatBird-USA-winter and SatBird-Kenya. Figure 1 summarizes the different sources of data for SatBird.
USA-summer	USA-winter	Kenya
Number of hotspots	122593	53361	9975
Number of species	670	670	1,054
Satellite RGBNIR reflectance	✓	✓	✓
Satellite true color image	✓	✓	✓
Bioclimatic rasters	✓	✓	✓
Pedologic rasters	✓	✓
Range maps	✓(586 species)	✓(620 species)
Table 1: Summary of the data provided for each of the SatBird subsets: USA-summer, USA-winter, and Kenya.
4.1 Bird species data
We used complete checklists from eBird [36]. For USA, we extracted checklists recorded from 2010 to 2023 at hotspots in the continental USA in June-July and December-January for the summer and winter seasons respectively. We filtered out hotspots with fewer than 5 complete checklists recorded, to ensure more meaningful encounter rates. We also excluded locations in the sea using the 5-meter resolution US nation cartographic boundaries provided by the Census Bureau's MAF/TIGER [48]. We included all regularly occurring species in the region, as denoted by ABA Codes 1 and 2 [1], which include regular breeding species and visitors. Code-1 species are more widespread, while Code-2 species can have restricted range or occur in lower density, bur are still widespread. We omit species found only in Hawaii and Alaska, as well as one Code-2 seabird species with rare oceanic observations, leaving a total of 670 species.
For Kenya, we extracted all complete checklists recorded from 2010 to 2023, limiting ourselves to the 1,054 species found regularly in Kenya according to Avibase [6]. Due to the scarcity of data in this region, we applied no criteria regarding the minimum number of checklists per hotspot or the month in which an observation was made. There are also a larger number of non-migratory birds present in Kenya than the USA, further motivating our choice to consider records aggregated across seasons.
An additional preparation step was merging hotspots with different IDs in eBird that had the same latitude and longitude, pooling the checklists together, unless all checklists were reported by the same observer on the same day, in which case we dropped the hotspot. We then computed the encounter rates for each hotspot as explained in Section 2 and corrected for vagrants (species seen in hotspots outside of their geographical range) by using range maps from eBird to set the target encounter rates to zero in these hotspots when they were available. We aggregated the species observations over 13 years, only considering seasonal change in distributions in the USA and leaving annual temporal change for future iterations of this work. As we aggregated data per hotspot, we did not take into account the reporting patterns of birders. Previous work [34] demonstrates that estimating the observer expertise can improve species distributions from eBird data. This may also be considered for future iterations of this work.
4.2 Environmental data
Following the GeoLifeCLEF 2020 dataset [14], we extracted 19 bioclimatic variables rasters of size  $50 \times 50$  and  $6 \times 6$  for the USA and Kenya datasets respectively and of spatial resolution approximately  $1\mathrm{km}$  centered on each hotspot from WorldClim 1.4. Bioclimatic variables are often used to model species distributions as a function of climate [33]. They represent annual trends for temperature, precipitation, solar radiation, wind speed and water vapor pressure. Additionally, for the USA dataset, we also extracted 8 pedologic (soil) variables with resolution 250 meter from SoilGrids [30], which provides global soil properties maps, including pH, soil organic carbon content and stocks. These maps are obtained with machine learning methods trained on soil profile observations and environmental covariates derived from remote sensing. We provide more details on the environmental variables in Appendix A.
(a) SatBird-USA-summer
(b) SatBird-USA-winter
Figure 2: Distribution of hotspots across the training, validation, and test sets.
(c) SatBird-Kenya
4.3 Satellite data
For each hotspot, we extracted data at 10 meter resolution for RGB and NIR reflectance data from Sentinel-2 satellite tiles covering a square of about  $5\mathrm{km}^2$  centered around the hotspots. Additionally we also extracted the true color image RGB bands. We extracted images with cloud coverage of at most  $10\%$ , keeping the least cloudy image in the time windows (June 1 - July 31, 2022) for USA-summer, (December 1, 2022 - January 31, 2023) for USA-winter, and (January 1, 2022 - January, 1 2023) for Kenya datasets respectively. In the case where no image was found for a hotspot with these criteria, we composed mosaics with images with up to  $25\%$  cloud coverage in the considered time window - and taking the median values. We associate one image per hotspot, considering that a recent satellite image is best representative of our species data, since there are more checklists in recent years compared to earlier years.
4.4 eBird range maps
We extracted range maps from the eBird Status and Trends [24] for the summer season in the USA from eBird by filtering ranges between June and July 2021 which are available for 586 of the 670 species considered in the USA. For the winter dataset, we were able to extract range maps for 620 out of 670 species using the same process but filtering ranges between December and January using the R package ebirdst [17]. For each hotspot  $l$ , we construct a binary mask  $m_{l}$  of size 670 where  $m_{l}^{(i)} = 1$  if  $l$  is within the range of species  $i$ . Range maps were not available for Kenya data.
4.5 Aligning data
Finally, we aligned the satellite and the species observation data to obtain the final set of hotspots included in the SatBird dataset, as some satellite images were not found (for example if the cloud cover criterion was not met). Only hotspots with a corresponding satellite image with height and width greater than 128 pixels were kept, with their associated the environmental data rasters. In this way we obtained 122,593 hotspots for the USA summer season, 53,361 for the USA winter season and 9,975 in Kenya. In summary, each hotspot is associated with the following input data: 1 satellite image with RGB & NIR reflectance  $(4) +$  RGB true color bands (3), bioclimatic rasters (19), and pedologic rasters (8). The targets are computed from the aggregated checklists, and correspond to species encounter rates. Figure 3 shows the distribution of the number of species per hotspot.
4.6 Dataset splits
In order to account for spatial auto-correlation and overfitting that can arise from random splits of geospatial data [54], we use the scikit-learn DBSCAN algorithm [19] implementation to cluster hotspots. DBSCAN finds core samples and expands clusters from them, and we specify core samples to have at least 2 hotspots with a maximum allowed distance between them of  $5\mathrm{km}$ . We obtained 12,650 clusters which were then randomly assigned to train, validation and test splits such that they respectively contained  $70\%$ ,  $15\%$ , and  $15\%$  of the hotspots for the US summer dataset. We kept the same split assignment for winter US locations also present in the summer US locations, assigned the
rest in a  $70\%$ ,  $15\%$ ,  $15\%$  repartition in the train, validation and test sets. The same procedure was followed for the data in Kenya. Figure 2 shows the spatial distribution of hotspots for the dataset and we provide number of hotspots per split in Appendix A.
In this section, we present the baselines used to benchmark our task, predicting species encounter rates. We choose a wide range of baselines covering simple regression trees, deep CNNs and transformers.
Mean encounter rates: We propose a first basic baseline which is simply the average of encounter rates over the training set for each species.
Environmental baseline: Following the GeoLifeCLEF [41] challenge, we propose an environmental baseline, using Gradient Boosted Regression Trees on the bioclimatic and pedological variables extracted at each of the hotspots.
MOSAIKS [55]: This model was proposed as an accessible method that generalizes well across a wide range of tasks at significantly lower computational cost compared to training a deep neural network for each task. It performs a one-time unsupervised image featurization by convolving a set of randomly selected patches with the input satellite image. These features can then be reused for an unlimited number of tasks. We propose to use the MOSAIKS method to extract features from our satellite images dataset and combine them with the environmental variables before training a regressor to predict the encounter rates at each hotspot.
SATLAS pretrained transformer [7]: The literature suggests pretraining Swin-v2 [40] transformer on a large number of remote sensing images to target distribution shifts in remote sensing tasks. The model is pre-trained on Satlas, a dataset of 1.3 million satellite images from different sources, and is proven to perform well in both in-distribution and out-of-distribution settings. They provide weights for models pre-trained on low-resolution and high resolution images. We freeze the Satlas parameters and train a final fully connected layer to predict our target.