SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training. For our feature representation  $\mathcal{X}$  (discussed in Sec. 3.1), we use a learned spatial implicit neural representation obtained from a pretrained neural network. Specifically, we train a fully connected network  $f$  with presence-only data using the  $\mathcal{L}_{\mathrm{AN - full}}$  loss proposed in [13]. We use the same original set of publicly available data from iNaturalist [5] but retrain the model from scratch after removing species that are in our test sets. This results in 44,181 total species. The linear classifiers corresponding to these species form the set of candidate models  $\mathcal{H}$  used by our approach. Each model in  $\mathcal{H}$  has a dimensionality of 256, i.e., each is a logistic regressor with 256 weights. During the active learning phase, every time a labeled observation is obtained, for models that require a linear model to be estimated (e.g., LR models or models with  $h_{\mathrm{online}}$ ), we update the classifier parameters  $\theta$  for the species in question using the cross-entropy objective in Eqn. 1. We do not update the parameters of the feature extractor  $f$ , which are kept frozen. In later experiments, we quantify the impact of different feature representations by changing the feature extractor, e.g., by comparing models trained with environmental features as input or randomly initialized networks.
Datasets. We make use of two expert curated sources of range maps for evaluation and to generate labels for observations during the active learning process: International Union for Conservation of Nature (IUCN) [2] and eBird Status and Trends  $(S\& T)$  [21]. We use the procedure outlined in [13] to preprocess the data and randomly select 500 species from each data source for evaluation. We apply an ocean mask to remove any cell that overlaps with the ocean to make the active learning procedure more "realistic" as we do not expect naturalists interested in improving a land based species range to explore the ocean to ensure that the species is not present there. This data represents our expert-grade presence-absence evaluation range map data. Unlike the IUCN data which has a confirmed presence or absence state for each location on the earth, the  $S\& T$  data does not cover the entire globe. In the case of  $S\& T$ , the valid region is different on a species by species case and is biased towards the Americas. As a result, during  $S\& T$  evaluation, cells which are not included within the valid region for each species are excluded from sampling and evaluation. The distribution of each test set is illustrated in the appendices.
Evaluation metrics. Each experiment consists of comparisons between different active learning strategies where we start with the same initial set of two randomly selected observations (i.e., one absence and one presence). We report performance using mean average precision (MAP), where a model is evaluated at the centroid of each valid H3 cell (i.e., limited to cells that only overlap with land) and is compared to "ground truth" presence-absence obtained from the test set expert-derived
Figure 2: Comparison of different active learning querying methods for the task of species range estimation. For each of the two datasets, we display the performance of different active learning methods over time, where higher values are better. The results depict the average of three runs for each method. We observe that methods that use a weighted combination of the candidate models result in the best performance (e.g., WA versus LR) particularly in early timesteps. Our WA_HSS+ approach performs best overall in both datasets. (Left) IUCN results, where we observe a large difference between methods. (Right) S&T results, where the performance difference is smaller but our approach is superior in the earlier timesteps.
range maps. During active learning, we measure average precision across all species in the test set at each time step and average them to obtain a single MAP score.
Baselines. In addition to evaluating our WA_HSS+ approach, we compare to several baseline active learning strategies. These include random and uncertainty sampling. For each strategy, we evaluate models resulting from a weighted combination from the candidate set (WA) and conventional logistic regressors (LR). We also compare to AN_FULL_E2E, the end-to-end trained implicit network method from [13] using their  $\mathcal{L}_{\mathrm{AN - full}}$  loss trained with all available data for each species. While this model is trained only on presence-only data using pseudo-negatives, it has access to many more observations per species (i.e., a minimum of 50 presence observations per species) and is not restricted to only sampling data from land. Additional comparisons are provided in the appendices.
Comparison of active learning strategies. In Fig. 2 we present quantitative results on the IUCN and S&T datasets, and show qualitative results in Fig. 4. We compare our WA_HSS+ approach to multiple different baseline active learning methods and observe superior performance compared to each. The performance on individual species can vary a lot, but the average MAP across all test species is similar from run to run.
There is a clear advantage when using our active sample selection method (i.e., comparing WA_HSS+ to random sampling in WA_random) on the IUCN test set. The addition of the online model estimate also is beneficial (i.e., comparing WA_HSS+ to WA_HSS). Estimating model parameters as a weighted combination of the hypothesis set (i.e., WA methods) is significantly better than conventional logistic regression (i.e., LR methods). The performance improvement on the  $S \& T$  test set is less dramatic. Here, even the random selection baseline performs well. This can possibly be attributed to the limited valid locations to explore for the species, the relative larger sizes, and the American focus of its ranges compared to the more globally distributed IUCN data. However, we still observe an advantage when using WA methods in the early time steps. As an additional baseline, we also compare to an end-to-end trained presence-only model from [13] which has access to significantly more observation data at training time. We can see that our WA_HSS+ method approaches, and then exceeds, the performance of this baseline in fewer than 40 time steps for the IUCN species.
Impact of the feature space. Our approach makes use of a weakly supervised pretrained feature extractor  $f$  to generate the input features  $x$ . Here, we explore different features spaces: nn loc feats are the standard deep network features from [13] that uses location only inputs, nn env feats are deep network features that use location and 20 additional environmental covariates (i.e., features that describe local properties such as amount of rainfall and elevation) from [20], and nn rand feats are from a randomly initialized and untrained version of the standard network. We also
Figure 3: Ablations of our approach on the IUCN dataset. (Top Left) Impact of the feature space used to represent  $x$ . (Top Right) Impact of the number of species in the candidate set  $\mathcal{H}$ . (Bottom Left) Impact of observation noise by introducing false negatives, where higher label noise amounts indicate that the observer is more likely to incorrectly predict that a species is not present at a location. (Bottom Right) Impact of candidate model weighting during active query selection.
perform experiments using the same input features but without a deep feature extractor (i.e., loc feats and env feats). In this setting, the candidate models are just logistic regression classifiers operating directly on the input features. In Fig. 3 (top left) we observe that the deep feature extractors significantly outperform non-deep ones. This highlights the power of transfer learning from disjoint species in the context of range estimation. This has not been explored in the exiting literature, where conventional approaches typically train models on environmental covariate input features [18]. Interestingly, the fixed random higher dimensional projection of nn rand feats performs better than using only the low dimensional location or environmental features, loc feats and env feats. This is likely because the low dimensional features are not sufficiently linearly separable.
Impact of number of candidate ranges. One of the strengths of our approach is that it can leverage representations learned from disjoint species. In this experiment we quantify the impact of reducing the number of species available when training the backbone feature extractor which also results in a reduction of the size of the candidate set  $\mathcal{H}$ . Specifically, we train different feature extractors from scratch, where each model has access to a different subset of the data during training, e.g., WA_HSS, 1000 sp is trained on data from only 1,000 species. In Fig. 3 (top right) we unsurprisingly observe a drop in MAP when the size of  $\mathcal{H}$  is small (e.g.,  $< 1,000$  species). As the size increases, the model performance improves. This illustrates the benefit of larger and more diverse candidate sets. This reduction has a larger impact on WA_HSS compared to WA_HSS+ due to the inclusion of  $h_{\mathrm{online}}$  in the case of the latter.
Impact of observation noise. Here we simulate the impact of observation noise by exploring the setting where the species is actually present at the location but the observer misses it. Thus we obtain false negative observations, but not false positives. We simulate different noise rates where the observer misses the species at each time step with a probability sampled from  $\{0\%, 10\%, 25\%, 50\}$ . In Fig. 3 (bottom left) we observe that our approach is surprisingly robust to non-trivial amounts of observation noise, and only starts to fail when the noise level becomes very large (i.e.,  $50\%$ ). In contrast, the best logistic regressor baseline is much more impacted by such label noise.
Impact of hypothesis weighting. Unlike traditional query by committee methods where each committee member is trained on a subset of the available data [43], our hypotheses represent models
Figure 4: Predicted range maps for the Yellow-footed Green Pigeon for three different active learning strategies, illustrated over three different time steps. We also display the expert range map centered on India and Southeast Asia, inset in the top left. The queried locations are marked with  $\times$ , with green representing present locations and red representing absent ones. WA_HSS+ quickly identifies both the Indian and Southeast Asian part of the species' range.
trained on different species. Thus, no individual model will necessarily have the same range as the test species. To compensate for this, we weight each  $h \in \mathcal{H}$  by the likelihood of that hypothesis,  $P(h|S^t)$ , as indicated in Eqn. 5. In Fig. 3 (bottom right) we observe that removing this weighting term (no weighting) significantly reduces the performance of WA_HSS+, particularly at later time steps where  $S^t$  contains enough samples to effectively identify and downweight hypotheses that are no longer consistent with the data. On the other hand, averaging the "soft votes" of each hypothesis (i.e., multiplying by  $h_k(\pmb{x})$  as in Eqn. 5) (WA_HSS+), or averaging the "hard votes" (i.e.,  $[h_k(\pmb{x}) > 0.5]$  used in WA_HSS+ hard labels) as in traditional query by committee methods [24], results in similar performance. The uncertainty of any one model is not particularly important, potentially due to the large number of hypotheses that can contribute to sample selection.
5 Limitations
In spite of the significant improvement in performance our approach provides, it does have some limitations which we leave for future work. Firstly, all active learning strategies we consider query single points rather than diverse batches [9]. This might limit the use of our strategies in situations where there are multiple observers available to gather data simultaneously. The geographic accessibility of specific locations is another limitation as we currently do not consider the practical difficulties associated with reaching certain geographic areas, e.g., remote regions. We also do not explore other forms of observation noise outside of failing to detect the species of interest when it is present. However, errors associated with misidentifying species are mitigated on platforms such as iNaturalist [4] who use a community consensus mechanism to assign labels to observations.
The performance of our approach is limited by the expressiveness of the provided learned feature space and the diversity of species present in the candidate hypothesis set. We are also impacted by any inaccuracies in the candidate ranges that may result from their inability to capture very high spatial resolution range characteristics, in addition to any class imbalance or geographic sampling biases in the data used to train the underlying models. However, our method will further benefit from advances in handling noise and bias in the context of training joint range estimation models [11], in addition to improvements in architectures and location encodings [39]. Finally, the expert-derived evaluation data we use represents the best available source, but it is still not guaranteed to be free from errors as characterizing a species' true range is a challenging problem.
Broader impacts. The models trained as part of this work have not been validated beyond the use cases explored in the paper. Thus care should be taken when using them for making any decisions about the spatial distribution of species. We only performed experiments on publicly available species observation data, and caution should be exercised when using these models to generate range maps for threatened species.
Obtaining a sufficient number of on the ground observations to train species range estimation models is one of the main limiting factors in generating detailed range maps for currently unmapped species. To address this problem, we presented one of the first investigations of active learning in the context of efficiently estimating range maps from point observation data. We also proposed a new approach that makes use of a pre-existing learned environmental feature representation and a set of range maps from disjoint species to enable efficient querying of locations for observation. Even though this feature representation is derived from a model trained on weakly labeled presence-only data, we show that it is still very effective at transferring to new species. Our approach is computationally efficient as all of our operations are linear with respect to the input features which ensures that we can easily scale to tens of thousands of species. Through extensive quantitative evaluation, we demonstrated that it is possible to obtain a significant improvement in the quality of the resulting range maps with much fewer observations when compared to existing active learning methods.
Acknowledgements. This project was funded by the Climate Change AI Innovation Grants program, hosted by Climate Change AI with the support of the Quadrature Climate Foundation, Schmidt Futures, and the Canada Hub of Future Earth. Funding was also provided by the Cornell-Edinburgh Global Strategic Collaboration Awards.
[1] H3. https://h3geo.org/. Accessed on 2023-05-16.
[2] The IUCN Red List of Threatened Species. Version 2022-2. https://www.iucnredlist.org. Accessed on 2023-05-16.
[3] PlantNet. https://plantnet.org. Accessed on 2023-05-16.
[4] iNaturalist. https://www.inaturalist.org, . Accessed on 2023-05-16.
[5] iNaturalist Open Data. https://github.com/inaturalist/inaturalist-open-data., Accessed on 2023-01-13.
[6] N. Abe. Query learning strategies using boosting and bagging. ICML, 1998.
[7] S. Beery, E. Cole, J. Parker, P. Perona, and K. Winner. Species distribution modeling for machine learning practitioners: a review. In SIGCAS Conference on Computing and Sustainable Societies, 2021.
[8] C. Botella, A. Joly, P. Bonnet, P. Monestiez, and F. Munoz. A deep learning approach to species distribution modelling. Multimedia Tools and Applications for Environmental and Biodiversity Informatics, 2018.
[9] K. Brinker. Incorporating diversity in active learning with support vector machines. In ICML, 2003.
[10] W. Cai, Y. Zhang, and J. Zhou. Maximizing expected model change for active learning in regression. In ICDM, 2013.
[11] D. Chen and C. P. Gomes. Bias reduction via end-to-end shift learning: Application to citizen science. In AAAI, 2019.
[12] D. Chen, Y. Xue, S. Chen, D. Fink, and C. Gomes. Deep multi-species embedding. arXiv:1609.09353, 2016.
[32] C. J. Marsh, Y. Gavish, M. Kuemmerlen, S. Stoll, P. Haase, and W. E. Kunin. Sdm profiling: A tool for assessing the information-content of sampled and unsampled locations for species distribution models. Ecological Modelling, 2023.
[33] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaepau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 2011.
[34] S. J. Phillips, M. Dudík, and R. E. Schapire. A maximum entropy approach to species distribution modeling. In ICML, 2004.
[35] S. J. Phillips, M. Dudík, J. Elith, C. H. Graham, A. Lehmann, J. Leathwick, and S. Ferrier. Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data. Ecological applications, 2009.
[36] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang. A survey of deep active learning. ACM computing surveys, 2021.
[37] A. C. Rodríguez, S. D'Aronco, K. Schindler, and J. D. Wegner. Mapping oil palm density at country scale: An active learning approach. Remote Sensing of Environment, 2021.
[38] H. Rosner-Katz, J. L. McCune, and J. R. Bennett. Using stacked sdms with accuracy and rarity weighting to optimize surveys for rare plant species. Biodiversity and Conservation, 2020.
[39] M. Rußwurm, K. Klemmer, E. Rolf, R. Zbinden, and D. Tuia. Geographic location encoding with spherical harmonics and sinusoidal representation networks. arXiv:2310.06743, 2023.
[40] E. G. Ryan, C. C. Drovandi, J. M. McGree, and A. N. Pettitt. A review of modern computational algorithms for bayesian optimal design. International Statistical Review, 2016.
[41] B. Settles. Active learning literature survey. University of Wisconsin, Madison, 2010.
[42] B. Settles, M. Craven, and S. Ray. Multiple-instance active learning. NeurIPS, 2007.
[43] H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In  $COLT$ , 1992.
[44] B. L. Sullivan, C. L. Wood, M. J. Iliff, R. E. Bonney, D. Fink, and S. Kelling. eBird: A citizen-based bird observation network in the biological sciences. Biological conservation, 2009.
[45] M. Teng, A. Elmustafa, B. Akera, H. Larochelle, and D. Rolnick. Bird distribution modelling using remote sensing and citizen science data. Tackling Climate Change with Machine Learning Workshop, 2023.
[46] R. C. Whytock, J. Świeżewski, J. A. Zwerts, T. Bara-Słupski, A. F. Koumba Pambo, M. Rogala, L. Bahaa-el din, K. Boekee, S. Brittain, A. W. Cardoso, et al. Robust ecological analysis of camera trap data labelled by a machine learning model. Methods in Ecology and Evolution, 2021.
[47] J. N. Williams, C. Seo, J. Thorne, J. K. Nelson, S. Erwin, J. M. O'Brien, and M. W. Schwartz. Using species distribution models to predict new occurrences for rare plants. Diversity and Distributions, 2009.
[48] Y. Xue, I. Davies, D. Fink, C. Wood, and C. P. Gomes. Avicaching: A two stage game for bias reduction in citizen science. In International Conference on Autonomous Agents and Multiagent Systems, 2016.
A Additional comparisons
In Table A1 we report the numerical performance of our approach compared to multiple different active learning strategies, including additional ones that are not present in the main paper. Performance is reported using per-time step mean average precision (MAP), in addition to area under the MAP curve (MAP-AUC). We also illustrate a subset of these methods in Fig. A1. We observe that our WA_HSS+ approach results in superior performance.
Table A1: Numerical results on the IUCN dataset reported at the end of four different time steps (5, 10, 20, or 50). Each row is a different active querying method. We report results as the average of three different runs.
Time step 5	Time step 10	Time step 20	Time step 50
MAP	MAP-AUC	MAP	MAP-AUC	MAP	MAP-AUC	MAP	MAP-AUC
WA_HSS+	0.65	0.61	0.69	0.64	0.73	0.68	0.77	0.72
WA Uncertain+	0.61	0.58	0.64	0.60	0.68	0.63	0.73	0.68
WA_random+	0.57	0.57	0.58	0.57	0.60	0.58	0.64	0.61
WA_positive+	0.58	0.57	0.58	0.58	0.58	0.58	0.58	0.58
WA_HSS	0.64	0.61	0.68	0.63	0.71	0.67	0.73	0.70
WA Uncertain	0.60	0.58	0.64	0.60	0.67	0.63	0.70	0.66
WA_random	0.57	0.57	0.58	0.57	0.60	0.58	0.63	0.60
WA_positive	0.58	0.57	0.58	0.57	0.58	0.58	0.59	0.58
LR_HSS	0.36	0.38	0.38	0.38	0.38	0.38	0.39	0.39
LR Uncertain	0.48	0.47	0.52	0.49	0.65	0.54	0.74	0.64
LR_random	0.48	0.46	0.50	0.48	0.53	0.50	0.58	0.53
LR_positive	0.39	0.40	0.43	0.41	0.48	0.43	0.50	0.47
LR_QBC	0.49	0.47	0.53	0.49	0.65	0.54	0.75	0.64
LR_EMC	0.47	0.46	0.51	0.47	0.61	0.52	0.71	0.62
Figure A1: Comparison of additional active learning methods for the task of species range estimation on the IUCN dataset. We display the performance of different active learning methods over time, where higher values are better. This figure is similar to Fig. 2 (left) in the main paper, but includes additional comparisons.
B Visualizations
B.1 Learned representations
In Fig. A2 (left), similar to the visualizations in [13], we display a visualization of the learned features from a neural network backbone. We use ICA to project the 256D vector to 3D and only display non-ocean features. Fig. A2 (right), we show a 2D TSNE visualization of the candidate model hypothesis set  $\mathcal{H}$ , where each point represents a different species weight vector  $\theta$ . We show the resulting range predictions (i.e.,  $\sigma(\pmb{\theta}_i^\top \pmb{x})$ ) for three different species ( $h_1 =$  Ludwig's Bustard,  $h_2 =$  Namib Rock Agama, and  $h_3 =$  Black-headed Ibis), where we observe that the two species with similar ranges ( $h_1$  and  $h_2$ ) have similar low-dimensional embeddings.
In Fig A3 we illustrate the learned combination of species' weight vectors that are used to generate predictions for three unseen test species. We observe that highly weighted species from  $\mathcal{H}$  often form distinct clusters in the TSNE representation, where each cluster is comprised of species whose ranges form one part of the test set species range. This helps explain why it is meaningful to learn a linear combination of pretrained ranges as in our method. With very few observations species with similar ranges can be identified and exploited to create a good range map for the species of interest.
Figure A2: (Left) 3D ICA projection of features for an nn env feats neural network backbone. (Right) 2D TSNE visualization of  $\mathcal{H}$  for a nn coord feats neural network backbone with predicted range maps shown for 3 species. Here,  $h_1 =$  Ludwig's Bustard,  $h_2 =$  Namib Rock Agama, and  $h_3 =$  Black-headed Ibis. Points close in the hypothesis space represent similar ranges.
Figure A3: Visualization of the learned combination of species' weight vectors that are used to generate predictions for three unseen test species, each shown on a different row. On the left we show a TNSE visualization of the hypothesis set, where brighter colours indicate species that have a greater weight at the end of active learning for our WA_HSS+ approach. For each of the species visualized, we indicate the highest weighted candidate species from a set of highly weighted locations that are indicated using a red, blue, or green marker on the left.
B.2 Additional qualitative results