SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christian Lange<sup>1</sup> Elijah Cole<sup>2,3</sup> Grant Van Horn<sup>4</sup> Oisin Mac Aodha<sup>1</sup>
<sup>1</sup>University of Edinburgh
$^{2}$ Altos Labs  $^{3}$ Caltech
$^{4}$ UMass Amherst
We propose a new active learning approach for efficiently estimating the geographic range of a species from a limited number of on the ground observations. We model the range of an unmapped species of interest as the weighted combination of estimated ranges obtained from a set of different species. We show that it is possible to generate this candidate set of ranges by using models that have been trained on large weakly supervised community collected observation data. From this, we develop a new active querying approach that sequentially selects geographic locations to visit that best reduce our uncertainty over an unmapped species' range. We conduct a detailed evaluation of our approach and compare it to existing active learning methods using an evaluation dataset containing expert-derived ranges for one thousand species. Our results demonstrate that our method outperforms alternative active learning methods and approaches the performance of end-to-end trained models, even when only using a fraction of the data. This highlights the utility of active learning via transfer learned spatial representations for species range estimation. It also emphasizes the value of leveraging emerging large-scale crowdsourced datasets, not only for modeling a species' range, but also for actively discovering them.
Understanding the geographic range that a biological species occupies is a fundamental piece of information that drives models for ascertaining how vulnerable a species is to extinction threats [2], for quantifying how they respond to climate induced habitat change [29], in addition to being important for assessing biodiversity loss. Estimated range maps are typically generated from statistical or machine learning-based models that are parameterized from sparsely collected in situ observations [7]. Such in situ observations can be obtained through various means, e.g., via experts conducting detailed field surveys to record the presence or absence of a particular species in a defined geographic area [27], from community scientists that record incidental observations in a less structured manner [44], or from autonomous monitoring solutions such as camera traps [46]. One of the major limiting factors in scaling up the generation of reliable range maps to hundreds of thousands of species is the underlying collection of data.
Recently, community science-based platforms such as iNaturalist [4], eBird [44], and PlantNet [3] have proven to be an appealing and scalable way to collect species observation data by distributing the effort across potentially millions of participants around the world. However, due to the incidental nature of the observations collected on these platforms (i.e., users are not directed towards a specific geographic location and asked to survey it exhaustively for the presence of a species of interest), there can be strong biases with respect to the places people go and the types of species they report [11]. Additionally, a conflict exists in that the ranges of rare species are the hardest to characterize owing to the lack of data, but estimating their ranges would actually provide the most useful data for conservation purposes [30]. In this work, we explore an alternative collection paradigm for efficiently estimating geographic ranges of previously unmapped species inspired by ideas from
Figure 1: Our goal is to estimate the geographic range of a species (i.e., the locations where the species can be found during its lifetime) from a small number of actively selected in situ observations. (Left) We assume we have access to a hypothesis set of candidate species range estimation models, where each member of the set encodes the range for the species it was trained on. Intuitively, models that are close in this space encode similar spatial ranges (e.g.,  $h_i$  and  $h_j$ ). (Right) At each time step in our active learning-based species range estimation paradigm, we sample a geographic location, a human then goes there to determine if the species is present or not, and then we update our predicted range based on the new data. The hypothesis space on the left is used to guide the selection of locations via our active querying approach.
active learning [41]. Our objective is to minimize the number of geographic locations that need to be sampled to accurately estimate a species' geographic range. To achieve this, we leverage recent advances in deep learning methods for joint species range estimation [13].
Existing work in species range estimation typically focuses on the offline setting whereby the entire set of observations is assumed to be available at training time [18; 12; 13]. As a result, the range estimation model is trained once offline and not updated. There have been attempts to use the trained models to predict which geographic location to investigate next to determine if a species might be present [38; 32]. However, these methods have typically not been developed, or quantified, in a fully online setting where new data is requested sequentially to update the model. Our approach instead makes the assumption that the geographic range of an unmapped species can be represented by a weighted combination of existing ranges. Through iterative active querying guided by these existing ranges, we efficiently estimate these weights online using only a small number of observations. An overview of this process is illustrated in Fig 1.
Our core contribution consists of a novel active learning-based solution for efficient species range estimation. We quantitatively evaluate the performance of our approach and compare it to alternative active learning methods on two challenging datasets containing expert-derived evaluation range data from one thousand species. We obtain a  $32\%$  improvement in mean average precision after ten time steps compared to a conventional active learning baseline on our globally evaluated test set. This translates to a significant reduction in the number of locations that need to be sampled to obtain a reliable estimate of a species range. Code for reproducing the experiments in our paper can be found at https://github.com/Chris-lange/SDM.active_sampling.
2.1 Species distribution modeling
Species distribution modeling refers to a family of methods that avail of species observation and geospatial data to estimate the geographic range of a species [17]. These models typically take some encoding of a spatial region of interest as input and output a numerical score indicating how likely it is for a particular species to be found there. There is a large body of work on this topic, but existing methods can be broadly categorized based on the type of data they train on (e.g., presence-absence or presence-only data) and the output quantity they attempt to predict (e.g., species presence or abundance) [7].
In the case of presence-absence data, at training time we have access to observations containing a set of locations where a species of interest is confirmed to be present and also locations where it has been confirmed to be absent. The advantage of this setting is that it is easy to fit standard supervised classification methods to the training data, e.g., decision trees [19]. This type of data
can be obtained from skilled observers that fill out detailed checklists recording the species they observe in a given area and time [44]. In contrast, in the presence-only setting, we do not have access to confirmed absences. Existing methods either factor this into the model [34] or synthetically generate pseudo-absence data for training [35]. Presence observations are easier to collect than presence-absence data as they are gathered opportunistically by observers [4]. Like all data-driven methods, these approaches have difficulties in estimating accurate ranges in the low data regime [30].
While traditional machine learning solutions have been shown to be effective for range estimation [18], recently we have observed a growing interest in the application of deep learning methods to the task [12; 8; 31; 16; 45; 13; 15]. In contrast to the majority of existing single species methods, these deep approaches jointly model multiple species in the same model, thus exploiting the fact that some species share similar range characteristics. In this work, we do not innovate on the underlying range estimation models used, but instead we show that existing methods that are trained on readily available weakly supervised presence-only data can provide transferable representations that can be used by active learning-based methods.
2.2 Active learning and species range estimation
Motivated by the fact that obtaining sufficiently diverse and informative training data can be one of the main bottlenecks when applying machine learning solutions to many real-world problems, active learning attempts to frame the data collection process as an optimization problem [41]. Through interaction with a user, the goal is to minimize the number of datapoints that need to be labeled while still obtaining high performance on the task of interest, e.g., in our case, estimating a species' range. A variety of active querying strategies have been proposed in the literature, including uncertainty sampling [28], committee-based methods [43], and methods that seek to label the datapoint that would have the largest impact on the model's parameters [42], among others.
The optimal design of spatial surveys for sampling a geographic region of interest when estimating spatially varying quantities is a central question in geostatistics [14] and has applications in tasks such as sensor placement [26]. Both non-adaptive and sequential methods have been explored for related tasks in the context of tractable parametric models [40]. In the case of deep learning [36], active sampling methods have also been shown to be effective for predicting spatially varying quantities in the context of remote sensing data [37]. However, the application of active sampling-based methods for large-scale species range estimation remains under explored. While we focus on estimating species ranges, our approach is applicable to any spatially varying quantity where it is possible to obtain a set of possible related predictions. Potential alternative applications include sensor placement, geographic priors for image classification, and disease modeling, in addition to related tasks such as active data collection for training image classifiers.
Xue et al. [48] explored different methods for guiding observers to under sampled locations to address the issue of spatial data bias in crowd collected datasets. However, their focus is not on estimating ranges, but instead to attempt to counteract data sampling biases. There have been attempts to use species distribution models to help guide the search for under observed species [25; 47; 23]. Rosner-Katz et al. [38] proposed a spatial sampling approach that involves stacking the outputs from a small number of range estimation models in order to test if locations that are estimated to contain rare species are good locations to check for other rare species. Marsh et al. [32] outlined an approach for defining a utility score for unsampled locations by measuring the difference in the output of a distribution model that is updated assuming the species of interest occurs there, or does not. This score can be used for suggesting locations to sample. However, it is expensive to compute as it requires updating the model for each possible location of interest. Different from these existing works, we propose a new efficient approach for the sequential acquisition of species presence or absence observations to efficiently parameterize models for range estimation.
3 Active species range estimation
3.1 Species range estimation
The goal of species range estimation is to predict the geographic range that a species occupies given a sparse set of observations at training time. Specifically, in the presence-absence setting [7], we are given a dataset  $S = \{(x_{i},y_{i})\}_{i = 1}^{N}$ , where each  $x_{i}$  is a feature vector encoding a geographic location
of interest (i.e., a specific latitude and longitude). In this work we focus on spatial observations, but it is possible to extend our models to spatio-temporal data. The corresponding  $y_{i} \in \{0,1\}$  is a binary label indicating if the species has been observed to be present ( $y_{i} = 1$ ) or found to be absent ( $y_{i} = 0$ ) at the geographic location represented by  $x_{i}$ .
The input features  $\pmb{x}$  can simply represent transformed latitude and longitude values (i.e.,  $\pmb{x} = [lat,lon]$ ), a set of environmental features that encode information about the climate and/or habitat present at the location (i.e.,  $\pmb{x} = [e_1,e_2,\dots ,e_d]$ ) [12], or could be a set of latent features extracted from a geospatially aware deep neural network (i.e.,  $\pmb{x} = f([lat,lon])$ ) [31]. In our binary classification setting, the goal is to learn the parameters of a model  $h$  such that it correctly classifies the observed data. This can be achieved by minimizing the cross-entropy loss with respect to the training data:
$$
L _ {C E} = - \sum_ {(\boldsymbol {x}, y) \in \mathcal {S}} y \log (h (\boldsymbol {x})) + (1 - y) \log (1 - h (\boldsymbol {x})). \tag {1}
$$
Here, we represent a prediction from the model for one species as  $h(\pmb{x}) \in [0,1]$ . In the simplest case,  $h$  could be a logistic regressor with  $h(\pmb{x}) = \sigma (\pmb{\theta}^{\top}\pmb{x})$ , where  $\sigma$  is the sigmoid function and  $\pmb{\theta}$  is the parameter (i.e., weight) vector that is learned for each species. Once trained, the model can be densely evaluated across all locations of interest (e.g., the entire globe) to generate a predicted range map for one species.
3.2 Active location sampling
In the previous section, we assumed we had access to a labeled training set  $S$  to train the range estimation model  $h$ . As noted earlier, obtaining this data on a global scale for rare or difficult to observe species can be extremely challenging as well as time consuming, even with a spatially distributed set of observers. Given this, it would be highly desirable to target limited observation resources to geographic locations that would result in observations that would be most informative for estimating the parameters of the model.
By framing our task as an active learning problem [41], our goal is to efficiently select and prioritize geographic locations for observation in order to improve the performance of the range estimation model for a given species while minimizing the total number of required observations. The active learning process can be broken down into three main steps; querying/sampling: determining the next location to observe, labeling: obtaining the observation label for the sampled location (i.e., is a species present or absent over a specified temporal interval), and updating: changing the model parameters based on the newly acquired data. Greedy active learning approaches simply query the next datapoint to be sampled one instance at a time. In this greedy setting, after a datapoint is observed and labeled, the model is updated. This process continues until some termination criteria is met, e.g., a maximum number of iterations or only a minor change in the model parameters is obtained.
One of the most commonly used strategies for greedy active learning is uncertainty sampling [28]. This involves sampling the feature that represents the location  $\pmb{x}$  that the current model is most uncertain about:
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x}} {\arg \min } | 0. 5 - P (y = 1 | \boldsymbol {x}, \mathcal {S} ^ {t}) |. \tag {2}
$$
Here,  $P(y = 1|\pmb{x},\mathcal{S}^t) = h^t(\pmb{x})$  indicates the output of the model at the time step  $t$ , and  $\mathcal{S}^t = \{(x_1,y_1),(x_2,y_2),\ldots,(x_t,y_t)\}$  is the set of training observations acquired up to and including time  $t$ . Once sampled,  $\pmb{x}^*$  is labeled (i.e., an individual goes to the location represented by  $\pmb{x}^*$  and determines if the species is present there, or not) and then added to the updated set of sampled observations  $\mathcal{S}^{t+1} = \mathcal{S}^t \cup (\pmb{x}^*,y^*)$ . Finally, the current model is updated by fitting it to the expanded set of sampled observations. Later in our experiments, we compare to uncertainty sampling, in addition to other baseline active learning methods, and show that they are not as effective for the task of species range estimation in early time steps.
3.3 Leveraging candidate range estimation models
Unlike conventional classification problems found in tasks such as binary image classification, the ranges of different species are not necessarily distinct, but can instead have a potentially large amount of overlap (see Fig. 1 (left)). Inspired by this observation, we propose a new active sampling strategy that makes use of a set of candidate range estimation models.
We assume that we have access to a set of candidate pretrained models  $\mathcal{H} = \{h_1, h_2, \dots, h_k\}$ . Each individual model encodes the spatial range for a different species, with one model per species. In practice, these can be a set of linear models (e.g., logistic regressors) that operate on features from a geospatially aware deep neural network, where each model has an associated weight vector  $\theta_k$ . Having access to a large and diverse  $\mathcal{H}$  may initially seem like a strong assumption, however in practice, it is now feasible to generate plausible range maps for thousands of species thanks to the availability of large-scale crowd collected species observation datasets [45; 13]. Importantly, we assume the species that we are trying to model using active learning is not already present in  $\mathcal{H}$ .
Assuming the candidate models are linear classifiers, we can represent our range estimation model  $h$  for an unseen species, as a weighted combination of the candidate models' parameters:
$$
\boldsymbol {\theta} ^ {t} = \sum_ {k = 1} ^ {| \mathcal {H} |} P \left(h _ {k} \mid \mathcal {S} ^ {t}\right) \boldsymbol {\theta} _ {k}. \tag {3}
$$
By averaging the parameters of our candidate models we generate a compact linear classifier that can be used to predict the presence of the new species, without requiring access to all models in  $\mathcal{H}$  during inference. When a new observation is added to  $S^t$ , the posterior  $P(h_k|S^t)$  is updated based on the agreement between a model  $h_k$  and the observed data in  $S^t$ . Assuming a uniform prior over the candidate models in  $\mathcal{H}$ , we model the posterior as  $P(h_k|S^t) \propto P(S^t|h_k)$ , and the resulting likelihood is represented as:
$$
P \left(\mathcal {S} ^ {t} \mid h _ {k}\right) = \prod_ {\left(\boldsymbol {x}, y\right) \in \mathcal {S} ^ {t}} y h _ {k} (\boldsymbol {x}) + (1 - y) \left(1 - h _ {k} (\boldsymbol {x})\right). \tag {4}
$$
This is related to the query-by-committee-setting in active learning [43], where our candidate models represent the individual 'committee' members. However, instead of being trained on subsets of the sampled data online during active learning, our candidate models are generated offline from a completely different set of observation data from a disjoint set of species.
3.3.1 Active learning with candidate range estimation models
Given our estimate of the model parameters from Eqn. 3, we can apply any number of active learning query selection methods from the literature to perform sample (i.e., location) selection. However, while our candidate models in  $\mathcal{H}$  provide us with an effective way to estimate the model parameters online, we can also use them to aid in the sample selection step. Specifically, given the likelihood of an observation from a model, weighted by  $P(h_{k}|S^{t})$ , we choose the location  $\boldsymbol{x}^*$  to query that is the most uncertain:
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x}} {\arg \min } \left| 0. 5 - \frac {1}{| \mathcal {H} |} \sum_ {h _ {k} \in \mathcal {H}} h _ {k} (\boldsymbol {x}) P \left(h _ {k} \mid \mathcal {S} ^ {t}\right) \right|. \tag {5}
$$
The intuition here is that we would like to sample the location that is currently most uncertain across all the candidate models. Our approach is motivated by the desire to make use of existing well characterized ranges when modeling the range of a less well described species, where observations may be very limited, to reduce the sampling effort needed.
The success of our approach hinges on the hypothesis set  $\mathcal{H}$  containing a sufficiently diverse and representative set of candidate models. If the target test species' range is very different from any combination of the ones in  $\mathcal{H}$ , our ability to represent its range will be hindered. Additionally, even if the test species' range is broadly similar to some weighted combination of the models in  $\mathcal{H}$ , some finer-grained nuances of the range might be different. To address this, we extend the hypothesis set by adding an additional model  $h_{\mathrm{online}}$ .  $h_{\mathrm{online}}$  is also a logistic regressor, but it is trained on the progressively accumulated observations from the test species that are obtained during active learning using a cross-entropy objective from Eqn. 1. Adding  $h_{\mathrm{online}}$  ensures that we also have access to the maximum likelihood model according to the observed data.
We refer to our approach as WA_HSS, which indicates that it uses a Weighted Average of the hypothesis set to represent the model weights, and performs Hypothesis Set Selection during active querying. Similarly, WA_HSS+ is our approach with the inclusion of  $h_{\mathrm{online}}$ .
In this section we quantitatively evaluate our active learning approach on two existing benchmark datasets which we adapt to our online learning setting.
4.1 Implementation Details
Experimental paradigm. Our experiments are designed to compare different active learning strategies using synthetic presence-absence data drawn from expert-derived range maps. We begin by discretizing the world into a set of valid explorable locations which correspond to the centroids of all land overlapping resolution five H3 cells [1]. Here, each cell encompasses an area of  $252\mathrm{km}^2$  on average. At each time step, and for each active learning strategy, we select one location  $\pmb{x}^{*}$  to query from any valid location that has not yet been sampled for the presence (or absence) of the species of interest. The corresponding species presence or absence label  $y^{*}$  for  $\pmb{x}^{*}$  is then generated according to the "ground truth" range map. In the real world, this step represents an observer visiting the queried location and searching for the species of interest. While our model could evaluate any continuous location, using H3 cell centroids ensures that queryable locations are equally spaced across the surface of the globe. We commence each experiment by randomly sampling one presence and one absence observation for each species. Experiments are performed independently for each species in the test set, and there is no interaction between species during the observation process. This is realistic, as in practice it would be unreasonable to assume that a single observer would be able to confirm the presence of more than a small number of species owing to their finite knowledge of different biological organisms. We repeat each experiment three times using different initial labelled examples and show standard deviations as error bars. We provide additional implementation details and a description of each of the baseline methods in the appendices.