Title: Active Learning-Based Species Range Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: species range estimation

active learning

implicit networks
Abstract: We propose a new active learning approach for efficiently estimating the geographic range of a species from a limited number of on the ground observations. We model the range of an unmapped species of interest as the weighted combination of estimated ranges obtained from a set of different species. We show that it is possible to generate this candidate set of ranges by using models that have been trained on large weakly supervised community collected observation data. From this, we develop a new active querying approach that sequentially selects geographic locations to visit that best reduce our uncertainty over an unmapped species’ range. We conduct a detailed evaluation of our approach and compare it to existing active learning methods using an evaluation dataset containing expert-derived ranges for one thousand species. Our results demonstrate that our method outperforms alternative active learning methods and approaches the performance of end-to-end trained models, even when only using a fraction of the data. This highlights the utility of active learning via transfer learned spatial representations for species range estimation. It also emphasizes the value of leveraging emerging large-scale crowdsourced datasets, not only for modeling a species' range, but also for actively discovering them.

Active Learning-Based Species Range Estimation

Abstract

We propose a new active learning approach for efficiently estimating the geographic range of a species from a limited number of on the ground observations. We model the range of an unmapped species of interest as the weighted combination of estimated ranges obtained from a set of different species. We show that it is possible to generate this candidate set of ranges by using models that have been trained on large weakly supervised community collected observation data. From this, we develop a new active querying approach that sequentially selects geographic locations to visit that best reduce our uncertainty over an unmapped species' range. We conduct a detailed evaluation of our approach and compare it to existing active learning methods using an evaluation dataset containing expert-derived ranges for one thousand species. Our results demonstrate that our method outperforms alternative active learning methods and approaches the performance of end-to-end trained models, even when only using a fraction of the data. This highlights the utility of active learning via transfer learned spatial representations for species range estimation. It also emphasizes the value of leveraging emerging large-scale crowdsourced datasets, not only for modeling a species' range, but also for actively discovering them.

1 Introduction

Understanding the geographic range that a biological species occupies is a fundamental piece of information that drives models for ascertaining how vulnerable a species is to extinction threats [2], for quantifying how they respond to climate induced habitat change [29], in addition to being important for assessing biodiversity loss. Estimated range maps are typically generated from statistical or machine learning-based models that are parameterized from sparsely collected in situ observations [7]. Such in situ observations can be obtained through various means, e.g., via experts conducting detailed field surveys to record the presence or absence of a particular species in a defined geographic area [27], from community scientists that record incidental observations in a less structured manner [44], or from autonomous monitoring solutions such as camera traps [46]. One of the major limiting factors in scaling up the generation of reliable range maps to hundreds of thousands of species is the underlying collection of data.

Recently, community science-based platforms such as iNaturalist [4], eBird [44], and PlantNet [3] have proven to be an appealing and scalable way to collect species observation data by distributing the effort across potentially millions of participants around the world. However, due to the incidental nature of the observations collected on these platforms (i.e., users are not directed towards a specific geographic location and asked to survey it exhaustively for the presence of a species of interest), there can be strong biases with respect to the places people go and the types of species they report [11]. Additionally, a conflict exists in that the ranges of rare species are the hardest to characterize owing to the lack of data, but estimating their ranges would actually provide the most useful data for conservation purposes [30]. In this work, we explore an alternative collection paradigm for efficiently estimating geographic ranges of previously unmapped species inspired by ideas from

Observation

Estimated

hi

hj

hk

time 0

Species is

present

Species is

absent

time 1

time t

✓

✓

✓

✓

✘

✘

✘

✘

✘

✘

✓

✓

✘

✓

✓

✘

✓

active learning [41]. Our objective is to minimize the number of geographic locations that need to be sampled to accurately estimate a species' geographic range. To achieve this, we leverage recent advances in deep learning methods for joint species range estimation [13].

Existing work in species range estimation typically focuses on the offline setting whereby the entire set of observations is assumed to be available at training time [18; 12; 13]. As a result, the range estimation model is trained once offline and not updated. There have been attempts to use the trained models to predict which geographic location to investigate next to determine if a species might be present [38; 32]. However, these methods have typically not been developed, or quantified, in a fully online setting where new data is requested sequentially to update the model. Our approach instead makes the assumption that the geographic range of an unmapped species can be represented by a weighted combination of existing ranges. Through iterative active querying guided by these existing ranges, we efficiently estimate these weights online using only a small number of observations. An overview of this process is illustrated in Fig 1.

Our core contribution consists of a novel active learning-based solution for efficient species range estimation. We quantitatively evaluate the performance of our approach and compare it to alternative active learning methods on two challenging datasets containing expert-derived evaluation range data from one thousand species. We obtain a 32% improvement in mean average precision after ten time steps compared to a conventional active learning baseline on our globally evaluated test set. This translates to a significant reduction in the number of locations that need to be sampled to obtain a reliable estimate of a species range. Code for reproducing the experiments in our paper can be found at https://github.com/Chris-lange/SDM_active_sampling .

2 Related work

2.1 Species distribution modeling

Species distribution modeling refers to a family of methods that avail of species observation and geospatial data to estimate the geographic range of a species [17]. These models typically take some encoding of a spatial region of interest as input and output a numerical score indicating how likely it is for a particular species to be found there. There is a large body of work on this topic, but existing methods can be broadly categorized based on the type of data they train on (e.g., presence-absence or presence-only data) and the output quantity they attempt to predict (e.g., species presence or abundance) [7].

In the case of presence-absence data, at training time we have access to observations containing a set of locations where a species of interest is confirmed to be present and also locations where it has been confirmed to be absent. The advantage of this setting is that it is easy to fit standard supervised classification methods to the training data, e.g., decision trees [19]. This type of data

can be obtained from skilled observers that fill out detailed checklists recording the species they observe in a given area and time [44]. In contrast, in the presence-only setting, we do not have access to confirmed absences. Existing methods either factor this into the model [34] or synthetically generate pseudo -absence data for training [35]. Presence observations are easier to collect than presence-absence data as they are gathered opportunistically by observers [4]. Like all data-driven methods, these approaches have difficulties in estimating accurate ranges in the low data regime [30].

While traditional machine learning solutions have been shown to be effective for range estimation [18], recently we have observed a growing interest in the application of deep learning methods to the task [12; 8; 31; 16; 45; 13; 15]. In contrast to the majority of existing single species methods, these deep approaches jointly model multiple species in the same model, thus exploiting the fact that some species share similar range characteristics. In this work, we do not innovate on the underlying range estimation models used, but instead we show that existing methods that are trained on readily available weakly supervised presence-only data can provide transferable representations that can be used by active learning-based methods.

2.2 Active learning and species range estimation

Motivated by the fact that obtaining sufficiently diverse and informative training data can be one of the main bottlenecks when applying machine learning solutions to many real-world problems, active learning attempts to frame the data collection process as an optimization problem [41]. Through interaction with a user, the goal is to minimize the number of datapoints that need to be labeled while still obtaining high performance on the task of interest, e.g., in our case, estimating a species' range. A variety of active querying strategies have been proposed in the literature, including uncertainty sampling [28], committee-based methods [43], and methods that seek to label the datapoint that would have the largest impact on the model's parameters [42], among others.

The optimal design of spatial surveys for sampling a geographic region of interest when estimating spatially varying quantities is a central question in geostatistics [14] and has applications in tasks such as sensor placement [26]. Both non-adaptive and sequential methods have been explored for related tasks in the context of tractable parametric models [40]. In the case of deep learning [36], active sampling methods have also been shown to be effective for predicting spatially varying quantities in the context of remote sensing data [37]. However, the application of active sampling-based methods for large-scale species range estimation remains under explored. While we focus on estimating species ranges, our approach is applicable to any spatially varying quantity where it is possible to obtain a set of possible related predictions. Potential alternative applications include sensor placement, geographic priors for image classification, and disease modeling, in addition to related tasks such as active data collection for training image classifiers.

Xue et al. [48] explored different methods for guiding observers to under sampled locations to address the issue of spatial data bias in crowd collected datasets. However, their focus is not on estimating ranges, but instead to attempt to counteract data sampling biases. There have been attempts to use species distribution models to help guide the search for under observed species [25; 47; 23]. Rosner-Katz et al. [38] proposed a spatial sampling approach that involves stacking the outputs from a small number of range estimation models in order to test if locations that are estimated to contain rare species are good locations to check for other rare species. Marsh et al. [32] outlined an approach for defining a utility score for unsampled locations by measuring the difference in the output of a distribution model that is updated assuming the species of interest occurs there, or does not. This score can be used for suggesting locations to sample. However, it is expensive to compute as it requires updating the model for each possible location of interest. Different from these existing works, we propose a new efficient approach for the sequential acquisition of species presence or absence observations to efficiently parameterize models for range estimation.

3 Active species range estimation

3.1 Species range estimation

The goal of species range estimation is to predict the geographic range that a species occupies given a sparse set of observations at training time. Specifically, in the presence-absence setting [7], we are given a dataset S = { ( x i , y i ) } N i =1 , where each x i is a feature vector encoding a geographic location

of interest (i.e., a specific latitude and longitude). In this work we focus on spatial observations, but it is possible to extend our models to spatio-temporal data. The corresponding y i ∈ { 0 , 1 } is a binary label indicating if the species has been observed to be present ( y i = 1 ) or found to be absent ( y i = 0 ) at the geographic location represented by x i .

The input features x can simply represent transformed latitude and longitude values (i.e., x = [ lat, lon ] ), a set of environmental features that encode information about the climate and/or habitat present at the location (i.e., x = [ e 1 , e 2 , . . . , e d ] ) [12], or could be a set of latent features extracted from a geospatially aware deep neural network (i.e., x = f ([ lat, lon ]) ) [31]. In our binary classification setting, the goal is to learn the parameters of a model h such that it correctly classifies the observed data. This can be achieved by minimizing the cross-entropy loss with respect to the training data:

Here, we represent a prediction from the model for one species as h ( x ) ∈ [0 , 1] . In the simplest case, h could be a logistic regressor with h ( x ) = σ ( θ ⊤ x ) , where σ is the sigmoid function and θ is the parameter (i.e., weight) vector that is learned for each species. Once trained, the model can be densely evaluated across all locations of interest (e.g., the entire globe) to generate a predicted range map for one species.

3.2 Active location sampling

In the previous section, we assumed we had access to a labeled training set S to train the range estimation model h . As noted earlier, obtaining this data on a global scale for rare or difficult to observe species can be extremely challenging as well as time consuming, even with a spatially distributed set of observers. Given this, it would be highly desirable to target limited observation resources to geographic locations that would result in observations that would be most informative for estimating the parameters of the model.

By framing our task as an active learning problem [41], our goal is to efficiently select and prioritize geographic locations for observation in order to improve the performance of the range estimation model for a given species while minimizing the total number of required observations. The active learning process can be broken down into three main steps; querying/sampling : determining the next location to observe, labeling : obtaining the observation label for the sampled location (i.e., is a species present or absent over a specified temporal interval), and updating : changing the model parameters based on the newly acquired data. Greedy active learning approaches simply query the next datapoint to be sampled one instance at a time. In this greedy setting, after a datapoint is observed and labeled, the model is updated. This process continues until some termination criteria is met, e.g., a maximum number of iterations or only a minor change in the model parameters is obtained.

One of the most commonly used strategies for greedy active learning is uncertainty sampling [28]. This involves sampling the feature that represents the location x that the current model is most uncertain about:

Here, P ( y = 1 | x , S t ) = h t ( x ) indicates the output of the model at the time step t , and S t = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x t , y t ) } is the set of training observations acquired up to and including time t . Once sampled, x ∗ is labeled (i.e., an individual goes to the location represented by x ∗ and determines if the species is present there, or not) and then added to the updated set of sampled observations S t +1 = S t ∪ ( x ∗ , y ∗ ) . Finally, the current model is updated by fitting it to the expanded set of sampled observations. Later in our experiments, we compare to uncertainty sampling, in addition to other baseline active learning methods, and show that they are not as effective for the task of species range estimation in early time steps.

3.3 Leveraging candidate range estimation models

Unlike conventional classification problems found in tasks such as binary image classification, the ranges of different species are not necessarily distinct, but can instead have a potentially large amount of overlap (see Fig. 1 (left)). Inspired by this observation, we propose a new active sampling strategy that makes use of a set of candidate range estimation models.

We assume that we have access to a set of candidate pretrained models H = { h 1 , h 2 , . . . , h k } . Each individual model encodes the spatial range for a different species, with one model per species. In practice, these can be a set of linear models (e.g., logistic regressors) that operate on features from a geospatially aware deep neural network, where each model has an associated weight vector θ k . Having access to a large and diverse H may initially seem like a strong assumption, however in practice, it is now feasible to generate plausible range maps for thousands of species thanks to the availability of large-scale crowd collected species observation datasets [45; 13]. Importantly, we assume the species that we are trying to model using active learning is not already present in H .

Assuming the candidate models are linear classifiers, we can represent our range estimation model h , for an unseen species, as a weighted combination of the candidate models' parameters:

By averaging the parameters of our candidate models we generate a compact linear classifier that can be used to predict the presence of the new species, without requiring access to all models in H during inference. When a new observation is added to S t , the posterior P ( h k |S t ) is updated based on the agreement between a model h k and the observed data in S t . Assuming a uniform prior over the candidate models in H , we model the posterior as P ( h k |S t ) ∝ P ( S t | h k ) , and the resulting likelihood is represented as:

This is related to the query-by-committee-setting in active learning [43], where our candidate models represent the individual 'committee' members. However, instead of being trained on subsets of the sampled data online during active learning, our candidate models are generated offline from a completely different set of observation data from a disjoint set of species.

3.3.1 Active learning with candidate range estimation models

Given our estimate of the model parameters from Eqn. 3, we can apply any number of active learning query selection methods from the literature to perform sample (i.e., location) selection. However, while our candidate models in H provide us with an effective way to estimate the model parameters online, we can also use them to aid in the sample selection step. Specifically, given the likelihood of an observation from a model, weighted by P ( h k |S t ) , we choose the location x ∗ to query that is the most uncertain:

The intuition here is that we would like to sample the location that is currently most uncertain across all the candidate models. Our approach is motivated by the desire to make use of existing well characterized ranges when modeling the range of a less well described species, where observations may be very limited, to reduce the sampling effort needed.

The success of our approach hinges on the hypothesis set H containing a sufficiently diverse and representative set of candidate models. If the target test species' range is very different from any combination of the ones in H , our ability to represent its range will be hindered. Additionally, even if the test species' range is broadly similar to some weighted combination of the models in H , some finer-grained nuances of the range might be different. To address this, we extend the hypothesis set by adding an additional model h online . h online is also a logistic regressor, but it is trained on the progressively accumulated observations from the test species that are obtained during active learning using a cross-entropy objective from Eqn. 1. Adding h online ensures that we also have access to the maximum likelihood model according to the observed data.

We refer to our approach as WA_HSS , which indicates that it uses a W eighted A verage of the hypothesis set to represent the model weights, and performs H ypothesis S et S election during active querying. Similarly, WA_HSS+ is our approach with the inclusion of h online .

4 Experiments

In this section we quantitatively evaluate our active learning approach on two existing benchmark datasets which we adapt to our online learning setting.

4.1 Implementation Details

Experimental paradigm. Our experiments are designed to compare different active learning strategies using synthetic presence-absence data drawn from expert-derived range maps. We begin by discretizing the world into a set of valid explorable locations which correspond to the centroids of all land overlapping resolution five H3 cells [1]. Here, each cell encompasses an area of 252 km 2 on average. At each time step, and for each active learning strategy, we select one location x ∗ to query from any valid location that has not yet been sampled for the presence (or absence) of the species of interest. The corresponding species presence or absence label y ∗ for x ∗ is then generated according to the 'ground truth' range map. In the real world, this step represents an observer visiting the queried location and searching for the species of interest. While our model could evaluate any continuous location, using H3 cell centroids ensures that queryable locations are equally spaced across the surface of the globe. We commence each experiment by randomly sampling one presence and one absence observation for each species. Experiments are performed independently for each species in the test set, and there is no interaction between species during the observation process. This is realistic, as in practice it would be unreasonable to assume that a single observer would be able to confirm the presence of more than a small number of species owing to their finite knowledge of different biological organisms. We repeat each experiment three times using different initial labelled examples and show standard deviations as error bars. We provide additional implementation details and a description of each of the baseline methods in the appendices.

Training. For our feature representation x (discussed in Sec. 3.1), we use a learned spatial implicit neural representation obtained from a pretrained neural network. Specifically, we train a fully connected network f with presence-only data using the L AN -full loss proposed in [13]. We use the same original set of publicly available data from iNaturalist [5] but retrain the model from scratch after removing species that are in our test sets. This results in 44,181 total species. The linear classifiers corresponding to these species form the set of candidate models H used by our approach. Each model in H has a dimensionality of 256 , i.e., each is a logistic regressor with 256 weights. During the active learning phase, every time a labeled observation is obtained, for models that require a linear model to be estimated (e.g., LR models or models with h online), we update the classifier parameters θ for the species in question using the cross-entropy objective in Eqn. 1. We do not update the parameters of the feature extractor f , which are kept frozen. In later experiments, we quantify the impact of different feature representations by changing the feature extractor, e.g., by comparing models trained with environmental features as input or randomly initialized networks.

Datasets. We make use of two expert curated sources of range maps for evaluation and to generate labels for observations during the active learning process: International Union for Conservation of Nature ( IUCN ) [2] and eBird Status and Trends ( S&T ) [21]. We use the procedure outlined in [13] to preprocess the data and randomly select 500 species from each data source for evaluation. We apply an ocean mask to remove any cell that overlaps with the ocean to make the active learning procedure more 'realistic' as we do not expect naturalists interested in improving a land based species range to explore the ocean to ensure that the species is not present there. This data represents our expert-grade presence-absence evaluation range map data. Unlike the IUCN data which has a confirmed presence or absence state for each location on the earth, the S&T data does not cover the entire globe. In the case of S&T , the valid region is different on a species by species case and is biased towards the Americas. As a result, during S&T evaluation, cells which are not included within the valid region for each species are excluded from sampling and evaluation. The distribution of each test set is illustrated in the appendices.

Evaluation metrics. Each experiment consists of comparisons between different active learning strategies where we start with the same initial set of two randomly selected observations (i.e., one absence and one presence). We report performance using mean average precision (MAP), where a model is evaluated at the centroid of each valid H3 cell (i.e., limited to cells that only overlap with land) and is compared to 'ground truth' presence-absence obtained from the test set expert-derived

IUCN

Time

0

10

20

30

40

50

0.3

0.4

0.5

0.6

0.7

0.8

MAP

AN_FULL_E2E

WA_HSS+

WA_HSS

WA_uncertain

WA_random

LR_HSS

LR_uncertain

LR_random

0

10

20

30

40

50

0.3

0.4

0.5

0.6

0.7

0.8

MAP

AN_FULL_E2E

WA_HSS+

WA_HSS

WA_uncertain

WA_random

LR_HSS

LR_uncertain

LR_random

SNT

Time

range maps. During active learning, we measure average precision across all species in the test set at each time step and average them to obtain a single MAP score.

Baselines. In addition to evaluating our WA_HSS+ approach, we compare to several baseline active learning strategies. These include random and uncertainty sampling. For each strategy, we evaluate models resulting from a weighted combination from the candidate set ( WA ) and conventional logistic regressors ( LR ). We also compare to AN_FULL_E2E , the end-to-end trained implicit network method from [13] using their L AN -full loss trained with all available data for each species. While this model is trained only on presence-only data using pseudo-negatives, it has access to many more observations per species (i.e., a minimum of 50 presence observations per species) and is not restricted to only sampling data from land. Additional comparisons are provided in the appendices.

4.2 Results

Comparison of active learning strategies. In Fig. 2 we present quantitative results on the IUCN and S&T datasets, and show qualitative results in Fig. 4. We compare our WA_HSS+ approach to multiple different baseline active learning methods and observe superior performance compared to each. The performance on individual species can vary a lot, but the average MAP across all test species is similar from run to run.

There is a clear advantage when using our active sample selection method (i.e., comparing WA_HSS+ to random sampling in WA_random ) on the IUCN test set. The addition of the online model estimate also is beneficial (i.e., comparing WA_HSS+ to WA_HSS ). Estimating model parameters as a weighted combination of the hypothesis set (i.e., WA methods) is significantly better than conventional logistic regression (i.e., LR methods). The performance improvement on the S&T test set is less dramatic. Here, even the random selection baseline performs well. This can possibly be attributed to the limited valid locations to explore for the species, the relative larger sizes, and the American focus of its ranges compared to the more globally distributed IUCN data. However, we still observe an advantage when using WA methods in the early time steps. As an additional baseline, we also compare to an end-to-end trained presence-only model from [13] which has access to significantly more observation data at training time. We can see that our WA_HSS+ method approaches, and then exceeds, the performance of this baseline in fewer than 40 time steps for the IUCN species.

Impact of the feature space. Our approach makes use of a weakly supervised pretrained feature extractor f to generate the input features x . Here, we explore different features spaces: nn loc feats are the standard deep network features from [13] that uses location only inputs, nn env feats are deep network features that use location and 20 additional environmental covariates (i.e., features that describe local properties such as amount of rainfall and elevation) from [20], and nn rand feats are from a randomly initialized and untrained version of the standard network. We also

0

10

20

30

40

50

Time

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Feature Space

WA_HSS+, nn loc feats

WA_HSS+, nn env feats

WA_HSS+, nn rand feats

WA_HSS+, loc feats

WA_HSS+, env feats

0

10

20

30

40

50

Time

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Hypothesis Set Size

WA_HSS+, 44181 sp.

WA_HSS, 44181 sp.

WA_HSS+, 10000 sp.

WA_HSS, 10000 sp.

WA_HSS+, 1000 sp.

WA_HSS, 1000 sp.

WA_HSS+, 100 sp.

WA_HSS, 100 sp.

WA_HSS+, 10 sp.

WA_HSS, 10 sp.

0

10

20

30

40

50

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

WA_HSS+, noise = 0.0

LR_uncertain, noise = 0.0

WA_HSS+, noise = 0.1

LR_uncertain, noise = 0.1

WA_HSS+, noise = 0.25

LR_uncertain, noise = 0.25

WA_HSS+, noise = 0.5

LR_uncertain, noise = 0.5

0

10

20

30

40

50

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Hypothesis Weighting

WA_HSS+

WA_HSS+, hard votes

WA_HSS+, no weighting

WA_HSS+, hard votes, no weighting

Time

Time

perform experiments using the same input features but without a deep feature extractor (i.e., loc feats and env feats ). In this setting, the candidate models are just logistic regression classifiers operating directly on the input features. In Fig. 3 (top left) we observe that the deep feature extractors significantly outperform non-deep ones. This highlights the power of transfer learning from disjoint species in the context of range estimation. This has not been explored in the exiting literature, where conventional approaches typically train models on environmental covariate input features [18]. Interestingly, the fixed random higher dimensional projection of nn rand feats performs better than using only the low dimensional location or environmental features, loc feats and env feats . This is likely because the low dimensional features are not sufficiently linearly separable.

Impact of number of candidate ranges. One of the strengths of our approach is that it can leverage representations learned from disjoint species. In this experiment we quantify the impact of reducing the number of species available when training the backbone feature extractor which also results in a reduction of the size of the candidate set H . Specifically, we train different feature extractors from scratch, where each model has access to a different subset of the data during training, e.g., WA_HSS, 1000 sp is trained on data from only 1,000 species. In Fig. 3 (top right) we unsurprisingly observe a drop in MAP when the size of H is small (e.g., < 1,000 species). As the size increases, the model performance improves. This illustrates the benefit of larger and more diverse candidate sets. This reduction has a larger impact on WA_HSS compared to WA_HSS+ due to the inclusion of h online in the case of the latter.

Impact of observation noise. Here we simulate the impact of observation noise by exploring the setting where the species is actually present at the location but the observer misses it. Thus we obtain false negative observations, but not false positives. We simulate different noise rates where the observer misses the species at each time step with a probability sampled from {0%, 10%, 25%, 50%}. In Fig. 3 (bottom left) we observe that our approach is surprisingly robust to non-trivial amounts of observation noise, and only starts to fail when the noise level becomes very large (i.e., 50%). In contrast, the best logistic regressor baseline is much more impacted by such label noise.

Impact of hypothesis weighting. Unlike traditional query by committee methods where each committee member is trained on a subset of the available data [43], our hypotheses represent models

WA_HSS+

nn

loc

feats

WA_random

nn

loc

feats

LR_uncertain

nn

loc

feats

Time step 0

Time step 4

Time step 8

trained on different species. Thus, no individual model will necessarily have the same range as the test species. To compensate for this, we weight each h ∈ H by the likelihood of that hypothesis, P ( h |S t ) , as indicated in Eqn. 5. In Fig. 3 (bottom right) we observe that removing this weighting term ( no weighting ) significantly reduces the performance of WA_HSS+ , particularly at later time steps where S t contains enough samples to effectively identify and downweight hypotheses that are no longer consistent with the data. On the other hand, averaging the 'soft votes' of each hypothesis (i.e., multiplying by h k ( x ) as in Eqn. 5) ( WA_HSS+ ), or averaging the 'hard votes' (i.e., J h k ( x ) > 0 . 5 K used in WA_HSS+ hard labels ) as in traditional query by committee methods [24], results in similar performance. The uncertainty of any one model is not particularly important, potentially due to the large number of hypotheses that can contribute to sample selection.

5 Limitations

In spite of the significant improvement in performance our approach provides, it does have some limitations which we leave for future work. Firstly, all active learning strategies we consider query single points rather than diverse batches [9]. This might limit the use of our strategies in situations where there are multiple observers available to gather data simultaneously. The geographic accessibility of specific locations is another limitation as we currently do not consider the practical difficulties associated with reaching certain geographic areas, e.g., remote regions. We also do not explore other forms of observation noise outside of failing to detect the species of interest when it is present. However, errors associated with misidentifying species are mitigated on platforms such as iNaturalist [4] who use a community consensus mechanism to assign labels to observations.

The performance of our of approach is limited by the expressiveness of the provided learned feature space and the diversity of species present in the candidate hypothesis set. We are also impacted by any inaccuracies in the candidate ranges that may result from their inability to capture very high spatial resolution range characteristics, in addition to any class imbalance or geographic sampling biases in the data used to train the underlying models. However, our method will further benefit from advances in handling noise and bias in the context of training joint range estimation models [11], in addition to improvements in architectures and location encodings [39]. Finally, the expert-derived evaluation data we use represents the best available source, but it is still not guaranteed to be free from errors as characterizing a species' true range is a challenging problem.

Broader impacts. The models trained as part of this work have not been validated beyond the use cases explored in the paper. Thus care should be taken when using them for making any decisions about the spatial distribution of species. We only performed experiments on publicly available species observation data, and caution should be exercised when using these models to generate range maps for threatened species.

6 Conclusion

Obtaining a sufficient number of on the ground observations to train species range estimation models is one of the main limiting factors in generating detailed range maps for currently unmapped species. To address this problem, we presented one of the first investigations of active learning in the context of efficiently estimating range maps from point observation data. We also proposed a new approach that makes use of a pre-existing learned environmental feature representation and a set of range maps from disjoint species to enable efficient querying of locations for observation. Even though this feature representation is derived from a model trained on weakly labeled presence-only data, we show that it is still very effective at transferring to new species. Our approach is computationally efficient as all of our operations are linear with respect to the input features which ensures that we can easily scale to tens of thousands of species. Through extensive quantitative evaluation, we demonstrated that it is possible to obtain a significant improvement in the quality of the resulting range maps with much fewer observations when compared to existing active learning methods.

Acknowledgements. This project was funded by the Climate Change AI Innovation Grants program, hosted by Climate Change AI with the support of the Quadrature Climate Foundation, Schmidt Futures, and the Canada Hub of Future Earth. Funding was also provided by the Cornell-Edinburgh Global Strategic Collaboration Awards.

References

H3. https://h3geo.org/ . Accessed on 2023-05-16.

The IUCN Red List of Threatened Species. Version 2022-2. https://www.iucnredlist.org . Accessed on 2023-05-16.

PlantNet. https://plantnet.org . Accessed on 2023-05-16.

iNaturalist. https://www.inaturalist.org , . Accessed on 2023-05-16.

iNaturalist Open Data. https://github.com/inaturalist/inaturalist-open-data , . Accessed on 2023-01-13.

N. Abe. Query learning strategies using boosting and bagging. ICML , 1998.

S. Beery, E. Cole, J. Parker, P. Perona, and K. Winner. Species distribution modeling for machine learning practitioners: a review. In SIGCAS Conference on Computing and Sustainable Societies , 2021.

C. Botella, A. Joly, P. Bonnet, P. Monestiez, and F. Munoz. A deep learning approach to species distribution modelling. Multimedia Tools and Applications for Environmental and Biodiversity Informatics , 2018.

K. Brinker. Incorporating diversity in active learning with support vector machines. In ICML , 2003.

W. Cai, Y. Zhang, and J. Zhou. Maximizing expected model change for active learning in regression. In ICDM , 2013.

D. Chen and C. P. Gomes. Bias reduction via end-to-end shift learning: Application to citizen science. In AAAI , 2019.

D. Chen, Y. Xue, S. Chen, D. Fink, and C. Gomes. Deep multi-species embedding. arXiv:1609.09353 , 2016.

E. Cole, G. Van Horn, C. Lange, A. Shepard, P. Leary, P. Perona, S. Loarie, and O. Mac Aodha. Spatial implicit neural representations for global-scale species mapping. ICML , 2023.

N. Cressie. Statistics for spatial data . John Wiley & Sons, 2015.

C. L. Davis, Y. Bai, D. Chen, O. Robinson, V. Ruiz-Gutierrez, C. P. Gomes, and D. Fink. Deep learning with citizen science data enables estimation of species diversity and composition at continental extents. Ecology , 2023.

R. De Lutio, Y. She, S. D'Aronco, S. Russo, P. Brun, J. D. Wegner, and K. Schindler. Digital taxonomist: Identifying plant species in community scientists' photographs. ISPRS journal of photogrammetry and remote sensing , 2021.

J. Elith and J. R. Leathwick. Species distribution models: ecological explanation and prediction across space and time. Annual review of ecology, evolution, and systematics , 2009.

J. Elith, C. H. Graham, R. P. Anderson, M. Dudík, S. Ferrier, A. Guisan, R. J. Hijmans, F. Huettmann, J. R. Leathwick, A. Lehmann, et al. Novel methods improve prediction of species' distributions from occurrence data. Ecography , 2006.

J. Elith, J. R. Leathwick, and T. Hastie. A working guide to boosted regression trees. Journal of animal ecology , 2008.

S. E. Fick and R. J. Hijmans. Worldclim 2: new 1-km spatial resolution climate surfaces for global land areas. International Journal of Climatology , 2017.

D. Fink, T. Auer, A. Johnston, M. Strimas-Mackey, O. Robinson, S. Ligocki, W. Hochachka, L. Jaromczyk, C. Wood, I. Davies, M. Iliff, and L. Seitz. ebird status and trends, data version: 2020; released: 2021. Cornell Lab of Ornithology, Ithaca, New York , 10, 2020.

M. Fois, G. Fenu, A. C. Lombrana, D. Cogoni, and G. Bacchetta. A practical method to speed up the discovery of unknown populations using species distribution models. Journal for Nature Conservation , 24:42-48, 2015.

M. Fois, A. Cuena-Lombraña, G. Fenu, and G. Bacchetta. Using species distribution models at local scale to guide the search of poorly known species: Review, methodological issues and future directions. Ecological Modelling , 2018.

Y. Freund, H. S. Seung, E. Shamir, and N. Tishby. Selective sampling using the query by committee algorithm. Machine Learning , 1997.

A. Guisan, O. Broennimann, R. Engler, M. Vust, N. G. Yoccoz, A. Lehmann, and N. E. Zimmermann. Using niche-based models to improve the sampling of rare species. Conservation biology , 2006.

A. Krause, A. Singh, and C. Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. JMLR , 2008.

P. Legendre, M. R. Dale, M.-J. Fortin, J. Gurevitch, M. Hohn, and D. Myers. The consequences of spatial structure for the design and analysis of ecological field surveys. Ecography , 2002.

D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In SIGIR Conference on Research and Development in Information Retrieval , 1994.

S. R. Loarie, P. B. Duffy, H. Hamilton, G. P. Asner, C. B. Field, and D. D. Ackerly. The velocity of climate change. Nature , 2009.

A. Lomba, L. Pellissier, C. Randin, J. Vicente, F. Moreira, J. Honrado, and A. Guisan. Overcoming the rare species modelling paradox: A novel hierarchical framework applied to an iberian endemic plant. Biological Conservation , 2010.

O. Mac Aodha, E. Cole, and P. Perona. Presence-only geographical priors for fine-grained image classification. In ICCV , 2019.

C. J. Marsh, Y. Gavish, M. Kuemmerlen, S. Stoll, P. Haase, and W. E. Kunin. Sdm profiling: A tool for assessing the information-content of sampled and unsampled locations for species distribution models. Ecological Modelling , 2023.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 2011.

S. J. Phillips, M. Dudík, and R. E. Schapire. A maximum entropy approach to species distribution modeling. In ICML , 2004.

S. J. Phillips, M. Dudík, J. Elith, C. H. Graham, A. Lehmann, J. Leathwick, and S. Ferrier. Sample selection bias and presence-only distribution models: implications for background and pseudo-absence data. Ecological applications , 2009.

P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang. A survey of deep active learning. ACM computing surveys , 2021.

A. C. Rodríguez, S. D'Aronco, K. Schindler, and J. D. Wegner. Mapping oil palm density at country scale: An active learning approach. Remote Sensing of Environment , 2021.

H. Rosner-Katz, J. L. McCune, and J. R. Bennett. Using stacked sdms with accuracy and rarity weighting to optimize surveys for rare plant species. Biodiversity and Conservation , 2020.

M. Rußwurm, K. Klemmer, E. Rolf, R. Zbinden, and D. Tuia. Geographic location encoding with spherical harmonics and sinusoidal representation networks. arXiv:2310.06743 , 2023.

E. G. Ryan, C. C. Drovandi, J. M. McGree, and A. N. Pettitt. A review of modern computational algorithms for bayesian optimal design. International Statistical Review , 2016.

B. Settles. Active learning literature survey. University of Wisconsin, Madison , 2010.

B. Settles, M. Craven, and S. Ray. Multiple-instance active learning. NeurIPS , 2007.

H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In COLT , 1992.

B. L. Sullivan, C. L. Wood, M. J. Iliff, R. E. Bonney, D. Fink, and S. Kelling. ebird: A citizen-based bird observation network in the biological sciences. Biological conservation , 2009.

M. Teng, A. Elmustafa, B. Akera, H. Larochelle, and D. Rolnick. Bird distribution modelling using remote sensing and citizen science data. Tackling Climate Change with Machine Learning Workshop , 2023.

R. C. Whytock, J. ´ Swie˙ zewski, J. A. Zwerts, T. Bara-Słupski, A. F. Koumba Pambo, M. Rogala, L. Bahaa-el din, K. Boekee, S. Brittain, A. W. Cardoso, et al. Robust ecological analysis of camera trap data labelled by a machine learning model. Methods in Ecology and Evolution , 2021.

J. N. Williams, C. Seo, J. Thorne, J. K. Nelson, S. Erwin, J. M. O'Brien, and M. W. Schwartz. Using species distribution models to predict new occurrences for rare plants. Diversity and Distributions , 2009.

Y. Xue, I. Davies, D. Fink, C. Wood, and C. P. Gomes. Avicaching: A two stage game for bias reduction in citizen science. In International Conference on Autonomous Agents and Multiagent Systems , 2016.

Appendix

A Additional comparisons

In Table A1 we report the numerical performance of our approach compared to multiple different active learning strategies, including additional ones that are not present in the main paper. Performance is reported using per-time step mean average precision (MAP), in addition to area under the MAP curve (MAP-AUC). We also illustrate a subset of these methods in Fig. A1. We observe that our WA_HSS+ approach results in superior performance.

0

10

20

30

40

50

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Performance of WA Baseline Strategies

WA_HSS+

WA_HSS

WA_uncertain+

WA_uncertain

WA_random+

WA_random

WA_positive+

WA_positive

0

10

20

30

40

50

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Performance of LR Baseline Strategies

LR_HSS

LR_uncertain

LR_EMC

LR_random

LR_positive

Time

Time

B Visualizations

B.1 Learned representations

In Fig. A2 (left), similar to the visualizations in [13], we display a visualization of the learned features from a neural network backbone. We use ICA to project the 256D vector to 3D and only display non-ocean features. Fig. A2 (right), we show a 2D TSNE visualization of the candidate model hypothesis set H , where each point represents a different species weight vector θ . We show the resulting range predictions (i.e., σ ( θ ⊤ i x ) ) for three different species ( h 1 = Ludwig's Bustard, h 2 = Namib Rock Agama, and h 3 = Black-headed Ibis), where we observe that the two species with similar ranges ( h 1 and h 2 ) have similar low-dimensional embeddings.

In Fig A3 we illustrate the learned combination of species' weight vectors that are used to generate predictions for three unseen test species. We observe that highly weighted species from H often form distinct clusters in the TSNE representation, where each cluster is comprized of species whose ranges form one part of the test set species range. This helps explain why it is meaningful to learn a linear combination of pretrained ranges as in our method. With very few observations species with similar ranges can be identified and exploited to create a good range map for the species of interest.

Neural Network Backbone Features (

nn env feats

)

Candidate Hypothesis Space

Predicted Ranges

h1

h2

h3

House Finch

AmericanWhitePelican

MournfulSphinx

StiltSandpiper

Diamond Dove

Malleefowl

Buff-bandedRail

Six-BandedArmadillo

GiantAmeiva

BrownWidow

TSNE

Range Prediction

Candidate Species 1

Candidate Species 2

Candidate Species 3

B.2 Additional qualitative results

In Figs. A4 to A6 we display additional qualitative results. In each figure, we compare the outputs of different active sampling strategies across different time steps for the same species. In general, we observe that our WA_HSS+ approach results in predictions that are closer to the expert range map (shown in the top left of each figure). However, Fig. A4, illustrates an interesting failure case where our approach fails to predict the species presence outside the region of the initial presence observation. However, from Fig. A11 we can see that the predictions for most species results in higher MAP scores for WA_HSS+ .

WA_HSS+

nn loc feats

WA_HSS

nn loc feats

WA_random

nn loc feats

LR_uncertain

nn loc feats

Time step 0

Time step 4

Time step 8

C Implementation details

C.1 Dataset details

In Sec. 4.1 in the main paper we noted that the S&T dataset does not span the entire globe and is more spatially biased compared to the IUCN dataset. In Fig. A7 we visualize the number of coarse locations that have a species presence for both datasets. We can see that the IUCN dataset on the left is more globally distributed, compared to the S&T dataset on the right. In addition, in the case of the S&T dataset, it is only possible to evaluate on a subset of possible locations as opposed to the IUCN dataset where we can evaluate on all possible locations on the earth's surface. In Table A2 we summarize the statistics of both datasets.

WA_HSS+

nn loc feats

WA_HSS

nn loc feats

WA_random

nn loc feats

LR_uncertain

nn loc feats

Time step 0

Time step 4

Time step 8

C.2 Input features

In our experiments we compare different input feature encodings x to represent locations on the earth's surface. Unless otherwise specified, we use a four dimensional trigonometric encoding of the scaled latitude and longitude (either directly or via a neural network-based feature extractor). This encoding is used to prevent discontinuities at the connected edges of the world map [31]:

In the results in Fig. 3 (top left) in the main paper we compare to models that use features derived from a neural-network backbone (i.e., those with nn in the name). If a neural network backbone is not specified, the input features are instead passed directly to the classifier. These neural network feature extractors are either randomly initialized using location only features and not trained (i.e., rand ), or are trained using the procedure outlined in [13] with only location features (i.e., loc ) as input or location and environmental covariates (i.e., env ).

Environmental covariates are commonly used for species distribution models as it allows them to attempt to learn the 'fundamental ecological niche' of the species of interest [7]. By also including the encoded coordinates we make our models spatially explicit and allow them to distinguish between locations that have similar environmental characteristics but are spatially distant. We use 20 environmental covariates (including elevation) generated from elevation and bioclimatic rasters at a five arc minute spatial resolution, taken from the widely used Worldclim 2.1 dataset [20]. Each covariate is normalized by subtracting the mean and dividing by the standard deviation, and NaN values are replaced with 0. Bilinear interpolation is used to generate corresponding environmental covariates for any given location. Models that take environmental features as input therefore accept 24 dimensional inputs, with four dimensions encoding location information and the remaining 20 representing the bilinearily interpolated environmental covariates at the location.

WA_HSS+

nn loc feats

WA_HSS

nn loc feats

WA_random

nn loc feats

LR_uncertain

nn loc feats

Time step 0

Time step 4

Time step 8

IUCN

10

20

30

40

50

number of present species

S&T

0

50

100

150

200

250

number of present species

0

C.3 Backbone feature extractor

In addition to the input encoding, our method involves extracting features from a deep species range estimation model trained on community collected presence-only data. This network uses the same architecture, training procedure, and hyperparameter settings as the L AN -full model from [13]. We also use the same training data from iNaturalist [5], but we remove classes (i.e., species) that are present in the full set of species (i.e., not just the 1,000 in our subset of the test species) in either of our test datasets. This reduces the number of species to 44,181.

The deep species range estimation model is a fully connected neural network consisting of an input layer, then four residual layers, each comprised of two fully connected layers with skip connections between them, and finally a classification head consisting of a linear classifier per-species, each of which predicts the presence or absence of a single species using a sigmoid output. It can be considered a location encoder f that feeds into set of single layer classifiers H which all share the

same input features. ReLU nonlinearities are used between layers, and each hidden layer consists of 256 neurons. Dropout layers are also utilized. This model is trained for 10 epochs using Adam optimizer with a dropout probability of 0.5 and a batch size of 2048. Training the feature extractor takes 2.5 hours on an NVIDIA RTX A6000 GPU with 48GB of RAM.

After training, the frozen network f can then be used as a feature extractor, generating 256 dimensional embeddings used for hypothesis set selection and for training new classifiers. Each per-species classifier h can be used as the set of hypotheses H in our WA approaches. As f is frozen during active sampling and the locations that can be sampled are limited to the centroid of resolution five H3 cells, features can be generated ahead of time by applying the feature extractor to each centroid and storing the results. Thus f is not required during the active sampling process.

C.4 Active sampling

In each experiment, we apply the chosen active sampling strategy on 500 held-out test species randomly selected from each of the International Union for Conservation of Nature ( IUCN ) [2] and eBird Status and Trends ( S&T ) [21] datasets. We use the same 500 species from each dataset for each experiment, and use the same initial samples consisting of a single presence and absence observation for each experiment. The species set in each dataset is not the same, e.g., the S&T dataset contains only birds. Each experiment is repeated three times with different seeds and initial samples. The same frozen feature extractor f is used across all experiments except 'Impact of Feature Space' experiments in Fig. 3 in the main paper and the additional ablations in this appendix.

We evaluate our performance using per-time step mean average precision (MAP), and using area under the MAP curve (MAP-AUC). As we synthetically generate data from the underlying expert range maps provided, we do not have a specified train test split and merely evaluate at each valid H3 cell centroid and measure performance as the difference between a model's predictions and the expert-derived range map. Unlike [13], we evaluate our models only on land and not also over the ocean.

The logistic regression range estimation model h we optimize during active sampling is implemented using scikit-learn's linear_model.LogisticRegression class with default settings [33]. If the features from f have been stored ahead of time, then each experiment can be performed without a GPU as it consists only of training and evaluating a logistic regression model. A single experiment using our WA_HSS+ method involving 500 species across 50 time steps takes 4 hours using unoptimized code on an AMD EPYC 7513 32-Core Processor.

C.5 Baseline strategies

Here we provide additional information about the baseline active sampling strategies we evaluated. In all cases, the set of possible locations from which to select a sample x ∗ consists of all valid resolution five H3 cell centroids (i.e., those that are located over land and have not yet been sampled, and for the S&T dataset only those where the ground truth is provided). The LR strategies below update the range estimation model at each time step by minimizing the cross entropy loss in Eqn. 1 from the main paper.

LR_random. x ∗ is selected randomly from all valid H3 cell centroids.

LR_uncertain. x ∗ is determined by Eqn. 2, and corresponds to the location where the model's probability of presence is closest to 0.5. This is one of the most well established and commonly used methods of active sampling for classification.

LR_EMC. x ∗ is selected to maximize the absolute gradient of the model parameters θ with respect to the loss function, taking an expectation over the current distribution of class labels. We are looking for the sample that will result in the largest expected model change. The intuition is that we expect this location will cause our model to 'learn' the most [10].

LR_HSS. x ∗ is selected according to Eqn. 5 in the main paper, where the weights of the hypothesis set H are determined by agreement between the hypotheses and the sampled data, and the location that the weighted hypotheses are most uncertain about is then selected. However unlike WA_HSS , h is not updated by averaging these hypotheses, but by updating the weights for a logistic regression classifier to minimize the cross entropy loss.

LR_QBC. This 'query by committee' method [43; 24] is similar to LR_HSS above but H is now a set of classifiers trained on subsets of the currently available data S t as in [6]. Due to the small size of S t , especially in early time steps, subsets are created by removing a single data point from S t , and then discarding subsets that only contain points from a single class.

LR_positive. x ∗ is selected as the location that has the highest probability of presence according to h . This querying method has been used within the ecology literature as a method for evaluating generated species distribution models or as a 'guided search' method [22; 23].

We also compare our WA_HSS+ strategy to others that use weighted averaging of H to update h , but do not use hypothesis set selection to determine x ∗ . The following WA strategies can also be modified by the inclusion of h online, leading to WA_random+ , WA_uncertain+ , and WA_positive+ baselines.

WA_random. x ∗ is selected randomly from all valid H3 cell centroids.

WA_uncertain. x ∗ is determined by Eqn. 2 in the main paper, and corresponds to the location where the model's probability of presence is closest to 0.5.

WA_positive. x ∗ is the location that has the highest probability of presence according to h .

D Additional ablations

Here we provide the results of additional ablation experiments.

D.1 Impact of label noise

In Fig. A8 we demonstrate the impact of adding label noise for WA_HSS+ compared to WA_HSS . This expands on the results in Fig. 3 (bottom left) in the main paper. While WA_HSS+ outperforms WA_HSS in the presence of modest label noise, as more noise is added, the WA_HSS strategy performs slightly better than WA_HSS+ .

0

10

20

30

40

50

Time

0.3

0.4

0.5

0.6

0.7

0.8

MAP

WA_HSS+, noise = 0.0

WA_HSS, noise = 0.0

WA_HSS+, noise = 0.1

WA_HSS, noise = 0.1

WA_HSS+, noise = 0.25

WA_HSS, noise = 0.25

WA_HSS+, noise = 0.5

WA_HSS, noise = 0.5

D.2 Impact of hypothesis set size

In Fig. A9 (left) we illustrate the impact of removing species from the training set when training the backbone feature extractor. We compare the performance of WA_HSS using different backbone models trained with a limited number of classes to WA_HSS where we remove classifiers from H but keep the same feature extractor trained on all classes. Fig. 3 (top right) in the main paper shows a large impact from using backbones trained with fewer classes. Here we observe that in almost all cases the model using a backbone trained on all classes performs best, and the performance difference increases when more classes are removed.

In Fig. A9 (right) we further show how using backbones trained on fewer classes impacts the baseline active sampling strategy. The change in performance for the LR_uncertain strategy underlines that jointly modeling many species in the backbone is vital for good performance due to the features

extracted becoming more useful for range estimation, even when the predicted species ranges are not directly used, as they are in WA_HSS .

0

10

20

30

40

50

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Feature Extractor Training

WA_HSS, 44181 sp.

WA_HSS, 10000 sp.

WA_HSS, 10000 hypothesis sp.

WA_HSS, 1000 sp.

WA_HSS, 1000 hypothesis sp.

WA_HSS, 100 sp.

WA_HSS, 100 hypothesis sp.

0

10

20

30

40

50

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Hypothesis Set Size for LR Strategies

LR_UNC, 44181 sp.

LR_UNC, 22000 sp.

LR_UNC, 10000 sp.

LR_UNC, 1000 sp.

LR_UNC, 100 sp.

LR_UNC, 10 sp.

Time

Time

D.3 Impact of feature space

In Fig. A10 we explore the impact of using learned input features for our WA_HSS+ method and the logistic regression baseline LR_uncertain . The feature spaces are a subset of those shown in Fig. 3 (top left) from the main paper. We observe that using the features generated by the deep feature encoder ( nn loc feats ) is vital for good performance for both the LR_uncertain and WA_HSS+ strategies. In contrast, only using location ( loc feats ) or location and environmental features ( env feats ) directly with no neural network results in poor performance.

0

10

20

30

40

50

Time

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

MAP

Impact of Feature Space

WA_HSS+, nn loc feats

LR_uncertain, nn loc feats

WA_HSS+, loc feats

LR_uncertain, loc feats

WA_HSS+, env feats

LR_uncertain, env feats

D.4 Per-species performance

In Figs. A11 to A14 we display histograms of the per-species performance across six different time steps (2, 5, 10, 20, 30, and 50) on the IUCN dataset. Each individual histogram displays the distribution of MAP scores for each of the species in the IUCN dataset. In general, we observe a trend that performance improves over time, i.e., the mass of the histograms move the right. Our WA_HSS+ approach in Fig. A11 is noticeably better than the naive random selection using logistic regression baseline in Fig. A14.

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 2

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 5

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 10

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 20

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 30

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 50

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 2

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 5

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 10

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 20

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 30

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

Frequency

Timestep 50

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 2

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 5

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 10

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 20

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 30

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0

5

10

15

20

25

30

35

40

Frequency

Timestep 50

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 2

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 5

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 10

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 20

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 30

0.0

0.2

0.4

0.6

0.8

1.0

Average Precision

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Timestep 50

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of species distribution modeling from limited observation data. An active learning approach is proposed that makes the assumption that an unmapped species can be represented by a weighted combination of candidate species models (previously trained offline on a disjoint dataset). A sampling strategy is described that uses the aggregate uncertainty of the candidate models to select the next location to query. For representing location, a pre-existing feature representation is derived from a model trained on weakly labeled presence-only data.  Results on two benchmark datasets outperform baselines and an ablation study is presented.\n\nSTRENGTHS:\n- Addresses an important and interesting problem. Species distribution modeling has received a great deal of attention in the literature lately, especially related to fine-grained visual categorization. This work is of the first to explore how active learning learning can be applied to this problem to estimate range models from limited amounts of observation data.\n\n- The proposed approach has technical novelty. The key idea is to take advantage of an existing set of species range models as a hypothesis set. During training, a range model for an unseen species is estimated by a weighted average of the candidate models. To capture fine-details, this candidate set is augmented with a model trained on the accumulated observations of the new species. In addition, a modified form of uncertainty sampling is also used which leverages the candidate models.\n\n- Extensive evaluation, including an ablation study. The results indicate that design decisions directly lead to performance improvements, for example the hypothesis set sampling approach and the inclusion of the online model in the candidate set. \n\n- Well written, easy to follow.\n\n- Sufficient detail provided that the proposed method is likely reproducible.\n\nWEAKNESSES:\n- The proposed approach is a relatively straightforward application of active learning beyond the introduction of the candidate models.\n\n- The approach is highly dependent on the hypothesis set (candidate models) being diverse such that a new species can be represented by some combination of them (L190). To partially address this, the authors add a model trained on the accumulated observations to the hypothesis set. \n\n- The evaluation is largely limited to simple baselines in the form of\n  a self-ablation (incrementally adding to an active learning baseline).  Only one recent paper is compared against (not active learning-based), with diverging results across the two benchmark datasets. That said, it appears there are no obvious other baselines to compare against.\n\nQUESTIONS:\nMy initital rating is a weak accept. This paper is exceptionally well written and was fun to read. The use of active learning for species range modeling has real-world utility. The proposed approach, while simple, has novelty in its construction and the results improve over baselines. I think this paper will be of interest to the community.\n\nSuggestions:\n\n- An interesting byproduct of the proposed approach would be to show which species contributed (and their range models) the most in the weighted average.\n\n- The annotations on Fig. 4 are hard to see without zooming in.\n\nLIMITATIONS:\nLimitations sufficiently addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Paper Summary**\n\nThis paper tackles the data-driven estimation of geographical habitat area for a given species; that is, given a certain geographic location (i.e., a \"point on the map\"), the goal is to predict how likely a certain species is present in this location. Prior efforts train models in an \"offline\" setting, assuming a finite training data set that indicates (for a fixed set of different species) the presence (or absence) of the species. Datasets that signal the presence (and possibly absence) of a large number of species are readily available. \n\nThis paper points out that dataset expansion for species not covered by existing datasets would be cumbersome- we would need to collect presence indication signals for new (or not yet recorded) species worldwide by possibly uniformly sampling presence indication evidence, which can then, in turn, be used to update/re-train existing models.\n\nThis paper investigates an interesting angle of this general problem: can we leverage *existing* (already collected) datasets to obtain a \"weak\" classifier for a new species, which can then guide the data collection efforts to regions, such that with as few samples we can decrease the overall presence entropy *as much as possible*? \n\nThis is akin to active learning, where the idea is to label as-informative-as-possible recorded data points. Different to that, in this case, the model suggests geographic regions in which we should sample the data for the new species (i.e., obtain presence/absence signals), such that the range estimates for new species will be as accurate as possible with as few data points (i.e., manual collections) as possible. \n\nThe overall idea behind the proposed work is to leverage the similarity between the (new) query species and existing species: this is based on the intuition that \"similar species\" (in terms of similarity in the feature space) will likely habitat similar territories. Such an approach can then be applied iteratively by incorporating newly-obtained presence/absence signals, suggesting increasingly more informative locations for the data collection. \n\nThe idea behind the approach is simple and intuitive: assuming an ensemble of (trained) linear models for a set of species, a model for a new species can be estimated as a weighted combination of existing linear models (where the model weight is proportional to the similarity in the feature space between the new (query) species and species database). This model then provides a \"heatmap\" over geographic locations, and guides the data collection -- new data points can be collected in the areas where the model is most uncertain about the new species' presence. \n\n**Core Contributions/Insights**\n\nThe core contribution of this work is the insight that techniques similar to active learning can be used to streamline the data collection process. This is based on the key insight that existing datasets contain a sufficiently large number of species samples, such that a linear combination of already trained linear classifiers provide a reasonably good predictor for the presence of new species. \n\n**Key Results**\n\nThe paper convincingly demonstrates that the proposed hypothesis ensembling outperforms logistic regression, and adding an online-trained model to the ensemble further improves the performance. I cannot comment on whether any relevant baselines are missing, as I am not actively working in species range estimation or active learning, and I am, therefore not familiar with related work. I hope my colleagues can comment on this. \n\nExperimental validation also discusses the impact of the number of existing species (models) for model averaging on the final performance and establishes that the proposed approach is useful when a large number of specie presence models is readily available. \n\n**Potential for Impact**\n\nSpecie habitat estimation has the potential to assist scientists in monitoring how the habitats of species are changing over time, which is especially important in the era in which global warming presents an existential threat to several species. Therefore, data-driven methods that can provide reasonable estimates based on (I assume difficult to obtain) evidence of species have important practical implications. As monitoring (and consequentially, data collection) is a continual process, developing models that can guide data collection efforts using active learning techniques makes sense. \n\n**Rating Justification**\n\nI think this paper presents an interesting, well-motivated, and well-executed work with direct practical relevance to monitoring species habitats. At this point, I see no obvious objections to why this paper should not be accepted.\n\nThis is a low-confidence review written by a reviewer entirely unfamiliar with bird-species range estimation and only weakly familiar with the field of active learning. \n\n**Post Rebuttal Comments**\n\nI want to thank the authors for their thorough responses. All my questions were addressed, and I see other reviewers are also positive about this paper. I am upgrading my rating to accept.\n\nSTRENGTHS:\n* Application relevance: this work has the potential to guide future data collection efforts for species range estimation. Monitoring species range is important for understanding how the earth's biosphere evolves under rapidly changing environmental conditions. \n* Paper presentation: this paper is very well written and summarizes well both the application domain (species range estimation), as well as existing methods that are commonly being used, together with simple and intuitive taxonomy of prior efforts. Core contributions are presented clearly, well-motivated, and experimentally validated. As someone reading a paper on species range estimation for the first time, I really appreciate Sec. 3, which summarizes how this problem is typically tackled in the literature. \n* Motivation: the paper makes a convincing case for investigating which areas should be used to collect data in the next training cycles using active learning techniques. \n* Results: I find the paper's claims convincingly validated.\n\nWEAKNESSES:\nIn addition to the weaknesses discussed in the limitation section, it would be fair to point out that the proposed approach is most likely *only* applicable to new species that are sufficiently similar to species for which data is readily available. I assume this approach would fail in the case of potentially newly-discovered species that would hypothetically be quite unlike anything that was observed before. As I am not familiar with the biological aspect of this task, I am not sure if this is a practical concern.\n\nQUESTIONS:\nDid the authors perform any kind of analysis on whether the intuition that most related species are a good indicator for the presence of new species holds? I.e., for a query species, are highly-weighted classifiers indeed representing most related species, and their presence is a good indicator? Did the authors observe any failure cases? For example, the presence of domestic cat, which is related to wild cats, is probably not a good presence indicator for, e.g., lion.\n\nLIMITATIONS:\nThe paper thoroughly discusses the limitations in a designated section. I find this section insightful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores a method for geographical range estimation of a species.  One of the primary challenges in range estimation is the scarcity and costliness of data collection, particularly for rare species. To address this issue, the paper introduces an active learning framework as a solution. The proposed method offers two key advantages. Firstly, it utilizes a joint feature representation that combines information from multiple species, enhancing the accuracy of the range estimation. Secondly, it employs an active learning scheme to efficiently collect and learn from data to estimate a geographical range of new species. The paper also proposes a linear combination of learned classifiers to develop a new classifier for each new species. The effectiveness of the proposed method is evaluated on two datasets: the IUCN dataset and the S&T dataset.\n\nSTRENGTHS:\nThe paper is written clearly. The description of the problem setup and the technical approach is intuitive, allowing readers to grasp the concepts easily.\n\nThe paper makes a significant contribution to the field of species range estimation. It achieves a notable performance increment and thoroughly explores the problem with a comprehensive analysis.\n\nFurthermore, the paper provides a comprehensive analysis of the proposed method, particularly focusing on the active learning scheme. This includes multiple stages such as initial representation learning, acquisition function, and model training. The authors thoroughly examine and analyze the proposed active learning framework, providing a comprehensive understanding for each components.\n\nWEAKNESSES:\nWhile the paper thoroughly analyzes its proposed method and provides empirical results for the specific problem of range estimation, it does not explicitly highlight the unique machine learning challenges of this problem or how they connect to the technical contribution of the paper. Range estimation can be seen as sharing commonalities with other common machine learning problem settings. For instance, the presence-absence data setting can be considered a supervised classification problem, while the presence-only data setting resembles one-class learning or semi-supervised learning, where the labeled set only contains positive samples.\n\nThe paper primarily focuses on the application aspect by developing an efficient active learning framework for the range estimation problem. However, it does not provide a detailed explanation of why the chosen solution, which employs widely used machine learning techniques, was devised among numerous alternative approaches. The paper would benefit from explicitly stating the unique machine learning challenges posed by the range estimation problem and how the proposed method effectively addresses those challenges. Additionally, a more comprehensive discussion of the motivation behind the chosen approach would enhance the clarity and understanding of the paper.\n\nQUESTIONS:\nWhat is the unique machine learning challenge of the range estimation problem that distinguishes it from other machine learning problems or applications?\n\nEquation (3) averages the classifier parameters. How does the model perform if averaging is applied to the output rather than to the parameters? Why did the authors decide to apply the average on parameters rather than output?\n\nEquation (5) uses average over outputs while equation (3) averages over parameters. Why should uncertainty computation (Eq. 5) be different from inference computation (Eq. 3)? Is there an intuition behind this?\n\nWhat happens if the pretrained classifiers are not used, but instead train a new species classifier from scratch on S_t? This is an important baseline but is missing in the paper.\n\nIt is interesting that nn_rand performs way better than the other two features; loc_feats and env_feats. What would be the reason for this? Is nn_rand trained on S_t? Or is nn_rand fixed as the random initial weights?\n\nLIMITATIONS:\nThe paper addresses the limitations of the proposed method and its broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an active learning framework for estimating the geographical range of species from sparse observations. The core idea is to exploit a large set of pretrained range estimation models from many different species. This paper also provides extensive experiments including detailed ablation studies and comparisons with baselines, demonstrating the effectiveness of the proposed active learning approach.\n\nSTRENGTHS:\n1. It is technically sound to exploit a large set of pretrained range estimation models from diverse species to reduce the required observations for training a new model.  \n2. This paper provides detailed ablation studies and comparisons with existing methods and baselines and convincingly demonstrates superior performance.\n3. This paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The technical novelty is limited. It is reasonable yet straightforward to exploit a set of pretrained range estimation models for training a model for a target species.\n2. The exact reason why it is meaningful to learn a linear combination of pretrained range estimation models needs further clarification. Is this because the range of different species is highly correlated and the learned weights indicate some correlation among different species, or such a linear combination indicates some general habitat candidates? A baseline probably stronger than the LR_random (Fig. 2) is to train a general classifier on joint presence data from all available species. Such a general model may indicate whether a given location is a good habitat for birds or other animals.   \n3. Is it visualize the weights in the linear combination for some target species? Do the weight visualizations indicate any correlations among species? If so, is such a correlation related to species taxonomy? It might also be useful to incorporate the species taxonomy as input to the model.\n\nQUESTIONS:\n1. Could you elaborate on the details of LR_HSS? For HSS (hypothesis set selection)? If logistic regression instead of weighted averaging is applied during training, how to apply HSS (Eqn. 5) at the test time? It looks like the weights per hypothesis H is required during active querying.  \n2. Typos: #199-#200, WA_HHS should be WA_HSS.\n\nLIMITATIONS:\n1. Limited technical novelty. The core idea of this paper is to exploit a set of pretrained range estimation models to reduce the number of observations required for training a new model for target species. This problem studied in this work is very specific to the range estimation problem. \n2. A stronger baseline that trains a general classifier should be considered as described above.  \n3. The role of weighted averaging needs more clarification.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

comments: SUMMARY:
This paper proposes to:
1) Use crowdsourced data to obtain species range maps for a large set of data-rich species, and then
2) Learn a linear combination of these maps to represent the range of other, data-poor, species, while minimizing the number of required presence/absence queries via active learning.

STRENGTHS:
The paper is well motivated, well structured, reads well and provides a good overview of the existing literature.
The proposed method is simple and intuitive, and the performance improvements quite substantial.

WEAKNESSES:
Although I'm quite positive about the paper, I do have some small doubts about whether a more general context placement would be required for a better fir in NeurIPS. I'm no expert, but it seems to me that the authors could connect the work to methods beyond species range estimation in order to connect better with the ML community. After all, the proposed method seems to perform reconstruction with a predefined basis. Otherwise, it seems like a paper more oriented exclusively towards the SDM community.

QUESTIONS:
As reflected in the weaknesses and limitations sections:
1) What other fields of application and methods in the ML community would have connections to the presented methodology?
2) Would this method eventually be feasible in reality?

LIMITATIONS:
Most limitations are properly addressed. The only thing I missed would be the feasibility of obtaining presence/absence queries. Is this something that has been done? Obtaining a presence given that the species is present is one thig, but obtaining an absence could be very challenging. Would this method eventually be feasible in reality?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-14