SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our goal with the Satbird dataset and task is to predict many birds at once, and we expect that having information about other birds allows a single model to learn from features that are common across many bird species and that it may indeed benefit predictions for the less common birds. We hypothesize that training a single model to predict all species is better when modeling species distribution to capture interactions and co-occurrences among species. To verify this, we conduct experiments where we train ResNet-18 model (refl+env+RM) on 20 random species (including the most common and least common species), a model on each of these species (total 20 models) and
Table 7: Test results on the SatBird-USA-summer dataset for ResNet-18 (refl+env+RM) models initialized with ImageNet and Seasonal Contrast weights
Model	MAE [1e-2]	MSE [1e-2]	Top-10	Top-30	Top-k
Cross Entropy	2.16	0.65	45.74	65.13	66.78
L1 Loss	1.86	0.77	43.58	60.86	61.13
L2 Loss	2.64	0.66	44.92	64.03	65.49
Focal Loss	2.93	0.75	39.43	57.99	60.68
Table 8: Using different loss functions for Resnet-18 baseline: this table compares the performance of Resnet-18 baseline when using different loss functions. Cross entropy Loss outperforms all other losses.
Figure 8: Training separate models per one species vs. training all species: Resnet18 - 670 species refers to a model trained over all bird species. Resnet18 - 20 species refers to a model trained over 20 bird species only. Resnet18 - 1 species refers to a model trained over each single species only as specified on the x-axis.
one model on all the 670 species. Then, we compare the predictions of these models over the 20 species to the ground truth and report the mean absolute error (MAE). The difference in performance is, as predicted, quite small; with models trained on more classes (either 20 species or all 670 species) performing slightly better. Figure 8 shows that models trained on 20 or all species can be better than training a model per single species. Even if it may be better to train on less number of species, we suggest that the gain is much less than the compute overhead we will have when training many separate models per species.
B.5 Incorporating location information
Besides the range map masking method to make the CNN models spatially explicit, we also propose to train a location encoder taking as input latitude and longitude information jointly with the image encoder. We used the same architecture as the one proposed in [42] for the location encoder's architecture, removing the final classification layer, and instead concatenating the obtained features to those obtained from the ResNet image encoder, before passing them to a fully-connected layer to obtain the predicted encounter rates. More specifically, we first map each spatial coordinate  $x^l$  (latitude and longitude) to  $[\cos(\pi x^l), \sin(\pi x^l)]$ . We end up with a input vector of size 4, which is passed through an initial fully connected layer, followed by 4 residual blocks, each consisting of two fully connected layers with a dropout layer in between. The performance of initial experiments with this setting on a subset of the SatBird-USA-summer dataset was not better than what we obtained with range maps masking, and thus we did not explore further this method for this benchmark. However, it could be that the location encoder's architecture or the fusion with the satellite image features could be improved. We show results obtained on a small subset of an early version of the SatBird-USA-summer dataset (9k hotspots) which led us to prefer exploring the range maps masking method in Table 9.
Model	MAE [1e-2]	MSE [1e-2]	Top-10	Top-30	Top-k
Mean encounter rate	2.9 ± 0.0	0.7 ± 0.0	26.45 ± 0.0	43.91 ± 0.0	51.46 ± 0.0
Env baseline	2.05 ± 0.00	0.5±0.0	43.1 ± 0.1	62.3 ± 0.03	68.86 ± 0.0
ResNet-18 (refl)	1.98 ± 0.00	0.5±0.0	41.8 ±0.3	60.1 ±0.2	67.0 ±0.0
ResNet-18 (refl+env)	1.8 ± 0.0	0.5±0.0	48.0 ±0.0	67.0 ±0.00	72.9 ±0.2
ResNet-18 (refl+env+LE)	1.7± 0.0	0.5 ± 0.1	44.5 ±0.7	65.0 ±0.0	72.0 ± 0.1
ResNet-18 (refl+env+RM)	1.8± 0.0	0.5± 0.0	48.2 ±0.2	67.7 ±0.1	73.4 ± 0.1
C Further Analysis on Satbird-USA-summer results
C.1 Validation set results
We report the performance of baselines on the validation set of the SatBird-USA-summer dataset in Table 10, and SatBird-USA-Winter in Table 11. The results on the validation set for both datasets are consistent with the reported test results.
Table 9: Results for baseline models on the test set of a subset of SatBird-US-summer. env indicates the use of environmental variables.  $RM$  and  $LE$  refer to the range maps and the location encoder respectively.
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
Mean encounter rates	3.1 ± 0.0	0.9 ± 0.0	27.1 ± 0.0	38.6 ± 0.0	44.86 ± 0.0
Environmental baseline	2.3 ± 0.0	0.6 ± 0.0	43.1 ± 0.0	57.7 ± 0.0	63.9 ± 0.0
Satlas (img)	2.8 ± 0.0	0.9 ± 0.0	30.9 ± 0.1	46.3 ± 0.0	48.9 ± 0.0
SatMAE (img)	3.0 ± 0.0	0.9 ± 0.0	29.4 ± 0.1	43.8 ± 0.1	46.9 ± 0.2
MOSAIKS (img+env)	2.5 ± 0.0	0.7 ± 0.0	42.1 ± 0.0	56.6 ± 0.0	62.2 ± 0.0
ResNet-18 (img)	2.6 ± 0.0	0.8 ± 0.00	35.9 ± 0.29	52.6 ± 0.1	54.8 ± 0.1
ResNet-18 (refl)	2.6 ± 0.0	0.79 ± 0.0	36.2 ± 0.03	53.8 ± 0.05	56.01 ± 0.03
ResNet-18 (img+env)	2.1 ± 0.0	0.6 ± 0.0	46.8 ± 0.14	66.2 ± 0.15	67.6 ± 0.14
ResNet-18 (refl+env)	2.1 ± 0.01	0.64 ± 0.0	46.4 ± 0.2	65.9 ± 0.1	67.3 ± 0.1
ResNet-18 (refl+env+RM)	2.1 ± 0.01	0.6 ± 0.0	46.3 ± 0.12	65.9 ± 0.1	67.4 ± 0.0
Table 10: Validation results on the SatBird-USA-Summer: Best results are shown in bold. img refers to using the RGB true color image, refl refers to using RGBNIR reflectance bands, env refers to using the environmental data (bioclimatic and pedologic variables). RM refers to the use of range maps.
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
Mean encounter rates	2.5 ± 0.0	0.7 ± 0.0	28.2 ± 0.0	47.5 ± 0.0	52.7 ± 0.0
Environmental baseline	1.8 ± 0.0	0.5 ± 0.0	49.2 ± 0.0	63.9 ± 0.0	68.5 ± 0.0
Satlas (img)	2.3 ± 0.0	0.7 ± 0.0	32.5 ± 0.1	53.2 ± 0.0	55.4 ± 0.0
SatMAE (img)	2.4 ± 0.0	0.7 ± 0.0	29.7 ± 0.3	51.9 ± 0.1	54.0 ± 0.0
MOSAIKS (img+env)	2.0 ± 0.0	0.5 ± 0.0	47.8 ± 0.0	62.1 ± 0.0	66.1 ± 0.0
ResNet-18 (img)	2.2 ± 0.0	0.6 ± 0.0	37.6 ± 0.0	56.3 ± 0.0	58.6 ± 0.0
ResNet-18 (refl)	2.2 ± 0.0	0.6 ± 0.0	38.2 ± 0.0	57.4 ± 0.0	59.5 ± 0.0
ResNet-18 (img+env)	1.7 ± 0.0	0.4 ± 0.0	51.45 ± 0.0	69.59 ± 0.0	71.0 ± 0.0
ResNet-18 (refl+env)	1.7 ± 0.0	0.5 ± 0.0	51.59 ± 0.0	69.68 ± 0.0	71.2 ± 0.0
ResNet-18 (refl+env+RM)	1.7 ± 0.0	0.4 ± 0.0	51.55 ± 0.0	69.61 ± 0.0	71.2 ± 0.0
Table 11: Validation results on the SatBird-USA-Winter: Best results are shown in bold. img refers to using the RGB true color image, refl refers to using RGBNIR reflectance bands, env refers to using the environmental data (bioclimatic and pedologic variables). RM refers to the use of range maps.
Species scientific name	Training occurrences	MSE ranking	MSE
Calcarius lapponicus	1	1	7.46597932e-08
Geopelia striata	1	2	7.48139554e-08
Rallus obsoletus	1	3	7.48282276e-08
Synthloramphus scrippsii	2	4	7.48671356e-08
Ardenna tenuirostris	1	5	7.48679727e-08
Calidris maritima	4	5	7.49105737e-08
Spizelloides arborea	1	7	7.49142886e-08
Aethia cristatella	1	8	7.49201557e-08
Table 12: Top predicted species on the USA-summer test set according to MSE. Occurrences refers to the number of training hotspots for which the encounter rate (target value) is non-zero.
Species scientific name	Training occurrences	MSE ranking	MSE
Agelaius phoeniceus	52630	670	0.09383779
Turdus migratorius	68198	669	0.07773756
Zenaida macroura	48582	668	0.07559232
Melospiza melodia	52107	667	0.06899277
Hirundo rustica	48299	666	0.06486867
Corvus brachyrhynchos	59977	665	0.0647007
Geothlypis trichas	38993	664	0.0632578
Cyanocitta cristata	47654	663	0.05804076
Cardinalis cardinaliss	48687	662	0.05713348
Passer domesticus	41578	661	0.05654651
Table 13: Bottom 10 predicted species on SatBird-USA-summer test set according to MSE in descending order. Occurrences refers to the number of training hotspots for which the encounter rate (target value) is non-zero.
C.2 Analysis of results
On the test set of SatBird-USA-summer, the top 50 species with lowest test-MSE are unsurprisingly species with very low number of occurrences in the training set  $(< 11)$  (so the model can predict a zero encounter rate and have good MSE for these species). Among those species we find species breeding in arctic regions (e.g. Bombycilla garrulus, Calcarius lapponicus) and seabirds (e.g. Uria lomvia). By design of our dataset, we don't have many observations of seabirds since we excluded all hotspots outside the boundaries of the continental US). Species with largest MSE have  $>30\mathrm{k}$  hotspots where they were observed in the training set. Tables 12 and 13 show the best and worst predicted species on the USA-summer test set according to MSE.
We show the squared error distribution by target value (encounter rates) for the USA-summer test set in Fig. 10. We binned the target values 19 bins and for each bin averaged the squared errors. The squared error is higher on average for higher target values. In Fig. 9, we show the predicted values against the target values. The target values were binned in 19 bins. The model generally underestimates the target values.
We show the geographic distribution of the top-k error in the predictions for the USA-summer test set in Fig. 11
In Appendix E, we show sample predictions of our benchmark model (ResNet-18 with refl+env+RM) on the SatBird-USA-summer dataset. The example in Fig. 12 shows that the model is able to predict some of the rare species that are very habitat specific commonly found in certain areas, such as the Verdin, which is mainly restricted to scrublands.
D Results on Kenya dataset
SatBird-Kenya dataset was created with the motivation of transferring a model trained on a large dataset (SatBird-USA-summer) for the task of predicting species encounter rates to a region with less data available. We reported the baselines as done on SatBird-USA.
Figure 9: Predicted vs target values (binned in 19 bins) for the USA-summer test set.
Figure 10: MSE value distribution by target value (binned in 19 bins) for the USA-summer test set. The bars show the minimum and maximum squared error for targets in a given bin.
Transfer learning: We experiment with transfer learning from models trained on SatBird-USA data to SatBird-Kenya, a low-data region compared to USA. With a Resnet-18 model trained on SatBird-USA-summer, we transfer the weights and fine-tune on SatBird-Kenya (finetune-USA). We also experiment with freezing the weights and only update the last classification layer (freeze-USA). All these models are compared to training on SatBird-Kenya from scratch with ResNet-18. We report results for the SatBird-Kenya dataset in Table 14.
Weighted Loss: In addition to the above experiments, we tried incorporating the number of complete checklists to the training strategy for Kenya. This was motivated by the observation that Kenya has has relatively fewer hotspots and some hotspots are over represented by the number of complete checklists reported. We address this by balancing the cross entropy loss by the number of complete checklists per hotspot. Our hypothesis is that by balancing the cross entropy by the number of complete checklists per hotspot, our model will learn a more balanced representation of the encounter rates across the whole dataset. eBird, just like many other large biodiversity datasets may have biased estimates of species diversity, as species occurrence generally follow a log-normal
Figure 11: USA-summer test set hotspots colored by adaptive top-k performance for the ResNet-18 (refl+env+RM) model.
distribution with a long tail [56, 57, 43, 60]. Because of this inherent bias, it is unsurprising that the data distribution in Kenya, and parts of the USA will have fewer hotspots with complete checklists.
We define the weighted loss, similar to our original loss, but with an additional weight as follows:
$$
\mathcal {L} _ {W C E} = \frac {1}{N _ {h}} \sum_ {h} w _ {h} \mathcal {L} _ {h} = \frac {1}{N _ {h}} \sum_ {h} w _ {h} \sum_ {s (\text {s p e c i e s})} - y _ {h} ^ {s} \log \left(\hat {y} _ {h} ^ {s}\right) - \left(1 - y _ {h} ^ {s}\right) \log \left(1 - \hat {y} _ {h} ^ {s}\right)
$$
where:  $N_{h}$  is the number of hotspots  $h$ .  $w_{h}$  is the weight corresponding to the number of complete checklists for hotspot  $h$ .  $y$  are the predictions and  $\hat{y}$  are the ground truth encounter rates. The summation over  $s$  represents a sum over all species.
In Table 14, we note that all ResNet-18 models perform similarly well with the use of cross-entropy as a loss function. One thing to note is that we did not perform any hyperparameter tuning so there is room for improvement for these baselines. Moreover, the sparse nature of the data and the large number of species (1054) to predict, which make this task particularly challenging might also contribute to the limited performance of the models. ResNet-18 initialized with the weights from the model trained on the USA data performed better on some metrics (MSE and MAE), highlighting the relevance of transferring knowledge from one geographical region and set of species to another. Satlas and SatMAE models seem to perform well overall. The MOSAIKS baseline performed the worst. A first path to improving this baseline would be to experiment with a different number of features generated from MOSAIKS. We also note that the images we considered were  $64*64$  with resolution  $10\mathrm{m/pixel}$ , when the original implementation was used for  $256*256$  images with  $4\mathrm{m/pixel}$  resolution.
As for using weighted loss in ResNet-18 baselines, it did not yield significant improvement to the performance on the Kenya dataset. We think that this is as a result of the fewer number of complete checklists and hotspots overall in Kenya. The task of extending these models to low-data regimes such as in Kenya presents a substantial and important challenge in species distribution modelling. This task is complex, and we think that it would be one direction to expand upon in future work.
E Prediction samples
All predictions reported are those obtained with the ResNet-18 (refl+env+RM) model. We sampled randomly hotspots from the SatBird-USA-summer test set and evaluated the model's performance in predicting the top-10 bird species at each location. We selected hotspots from geographically distinct
Model	MAE[1e-2]	MSE[1e-2]	Top-10	Top-30	Top-k
Mean encounter rates	3.5 ± 0.0	1.7 ± 0.0	14.0 ± 0.0	19.2 ± 0.0	23.4 ± 0.0
Environmental Baseline	3.8 ± 0.0	1.9 ± 0.00	10.6 ± 0.1	18.7 ± 0.0	24.2 ± 0.1
Satlas (img)	3.1 ± 0.0	1.7 ± 0.0	12.6 ± 0.1	29.5 ± 0.2	23.9 ± 0.1
SatMAE (img)	3.6 ± 0.2	1.7 ± 0.0	14.0 ± 0.8	28.8 ± 0.4	23.8 ± 0.1
MOSAIKS (img+env)	4.8 ± 0.0	2.1 ± 0.0	8.9 ± 0.0	15.2 ± 0.0	19.7 ± 0.0
ResNet-18(scratch)	3.9 ± 0.4	1.7 ± 0.0	18.5 ± 1.4	35.1 ± 0.1	28.6 ± 0.4
ResNet-18(finetune-USA)	3.8 ± 0.4	1.7 ± 0.0	17.7 ± 0.0	34.6 ± 0.3	28.4 ± 0.1
ResNet-18(freeze-USA)	3.3 ± 0.1	1.6 ± 0.0	18.3 ± 0.2	33.8 ± 0.4	27.7 ± 0.1
WL-ResNet-18(scratch)	3.6 ± 0.0	1.7 ± 0.0	17.4 ± 0.9	32.7 ± 1.2	27.2 ± 0.9
WL-ResNet-18(finetune-USA)	3.6 ± 0.1	1.6 ± 0.0	17.4 ± 2.5	33.0 ± 0.1	27.4 ± 0.4
WL-ResNet-18(freeze-USA)	3.6 ± 0.1	1.7 ± 0.0	16.6 ± 1.5	31.4 ± 0.3	26.1 ± 0.3
Table 14: Test results on the SatBird-Kenya: img refers to using the RGB true color image. All ResNet-18 baselines use  $refl + env$ , which corresponds to using RGBNIR reflectance bands and the environmental data (bioclimatic and pedologic variables), and WL refers to the weighted Loss. scratch refers to training ResNet-18 model from scratch. finetune-USA refers to fine-tuning a ResNet-18 model with weights transferred from SatBird-USA-summer. freeze-USA refers to using ResNet-18 weights transferred from SatBird-USA-summer while training the last layer only.
regions, featuring different land cover types, including the Pacific Coast (California), Northeast (Maine) and Central US (Ohio). The following figures illustrate the results from these regions.
F Authors' statement
The authors bear all responsibility in case of violation of rights.
G Licenses
The SatBird Dataset is released under a Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License (https://creativecommons.org/licenses/by-nc/4.0/). Additionally, use of our dataset should comply with the eBird Terms of Use (https://www.birds.cornell.edu/home/terms-of-use/) and Terms of Use of the eBird Status and Trends Products (https://science.ebird.org/en/status-and-trends/products-access-terms-of-use).
US-CA-L654261
Species	Predicted probability
Verdin	0.410
Mourning Dove	0.394
Gambel's Quail	0.333
House Finch	0.322
Great-tailed Grackle	0.310
White-winged Dove	0.284
Eurasian Collared-Dove	0.241
Common Raven	0.239
Turkey Vulture	0.239
Black-tailed Gnatcatcher	0.184
Species	Groundtruth probability
Ash-throated Flycatcher	0.625
Abert's Towhee	0.5
Verdin	0.5
Mourning Dove	0.375
White-winged Dove	0.375
House Finch	0.375
Ladder-backed Woodpecker	0.375
Gambel's Quail	0.25
Western Screech-Owl	0.25
Hooded Oriole	0.25
Figure 12: Hotspot and species at Desilt Wash, San Bernardino Couty in California: Our model correctly identified the Verdin in this region in the top-10 species predicted list. It's worth noting that the Verdin's habitat is a fairly rare species whose habitat is mainly restricted to shrublands.
US-CA-L1516413
Figure 13: Hotspot and species in Lampson Reservoir, Ashtabula County, Ohio: Our model predicts the presence of various species in this region, including the song sparrow, red-winged blackbird and gray catbird. These species are consistent with the ground truth and are highly reported on eBird for this location.
Species	Predicted probability
American Robin	0.785
Red-winged Blackbird	0.753
Song sparrow	0.727
Gray Catbird	0.686
Northern Cardinal	0.676
American Goldfinch	0.603
Common Yellowthroat	0.556
Yellow Warbler	0.520
Mourning Dove	0.491
American Crow	0.476
Species	Groundtruth probability
Song Sparrow	0.757
American Robin	0.703
Red-winged Blackbird	0.649
Canada Goose	0.649
Great Blue Heron	0.622
Eastern Kingbird	0.540
Gray Catbird	0.513
Yellow Warbler	0.513
Wood Duck	0.405
Mourning Dove	0.378
US-ME-L608588
Figure 14: Hotspot and species in Kennebago River-Boy Scout Road, Maine: while our model identifies correctly the species with highest ground truth encounter rates, it underestimates the encounter rates values.
Species	Predicted probability
White-throated Sparrow	0.553
Northern Parula	0.542
Red-eyed Vireo	0.526
Black-capped Chickadee	0.471
Red-breasted Nuthatch	0.457
Common Yellowthroat	0.455
Blue Jay	0.449
Swainson's Thrush	0.431
American Robin	0.414
Magnolia Warbler	0.414
Species	Groundtruth probability
Northern Parula	0.876
White-throated Sparrow	0.826
Red-breasted Nuthatch	0.815
American Robin	0.765
Blue Jay	0.754
Black-capped Chickadee	0.749
Golden-crowned Kinglet	0.731
Blue-headed Vireo	0.718
Ovenbird	0.718
Veery	0.702