# Expert Elicitation, Uncertainty, and the Value of Information in Controlling Invasive Species

Fred A. Johnson, Brian J. Smith, Mathieu Bonneau, Julien Martin, Christina

Romagosa, Frank Mazzotti, Hardin Waddle, Robert N. Reed, Jennifer

Kettevrlin Eckles, Laurie J. Vitt

<figure><img src="image_1.png" /></figure>

## To cite this version:

Fred A. Johnson, Brian J. Smith, Mathieu Bonneau, Julien Martin, Christina Romagosa, et al.. Expert Elicitation, Uncertainty, and the Value of Information in Controlling Invasive Species. Ecological Economics, 2017, 137, pp.83-90. 10.1016/j.ecolecon.2017.03.004. hal-01514233

HAL Id: hal-01514233

https://hal.science/hal-01514233

Submitted on 25 Apr 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

Analysis

# Expert Elicitation, Uncertainty, and the Value of Information in Controlling Invasive Species

<figure><img src="image_4.png" /></figure>

CrossMark

Fred A. Johnson<sup>a,*</sup>, Brian J. Smith<sup>b</sup>, Mathieu Bonneau<sup>b</sup>, Julien Martin<sup>a</sup>, Christina Romagosa<sup>b</sup>, Frank Mazzotti<sup>c</sup>, Hardin Waddle<sup>d</sup>, Robert N. Reed<sup>e</sup>, Jennifer Kettevlin Eckles<sup>c</sup>, Laurie J. Vitt<sup>f</sup>

## ARTICLE INFO

**Article history:**

Received 31 August 2016

Received in revised form 30 January 2017

Accepted 2 March 2017

Available online xxxx

**Keywords:**

Expert elicitation

Invasive species

Matrix model

Stochastic dominance

Uncertainty

Value of information

## ABSTRACT

We illustrate the utility of expert elicitation, explicit recognition of uncertainty, and the value of information for directing management and research efforts for invasive species, using tegu lizards (*Salvator merianae*) in south- ern Florida as a case study. We posited a post-birth pulse, matrix model in which four age classes of tegus are rec- ognized: hatchlings, 1 year-old, 2 year-olds, and 3 + year-olds. This matrix model was parameterized using a 3- point process to elicit estimates of tegu demographic rates in southern Florida from 10 herpetology experts. We fit statistical distributions for each parameter and for each expert, then drew and pooled a large number of rep- licate samples from these to form a distribution for each demographic parameter. Using these distributions, as well as the observed correlations among elicited values, we generated a large sample of matrix population models to infer how the tegu population would respond to control efforts. We used the concepts of Pareto effi- ciency and stochastic dominance to conclude that targeting older age classes at relatively high rates appears to have the best chance of minimizing tegu abundance and control costs. We conclude that expert opinion com- bined with an explicit consideration of uncertainty can be valuable in conducting an initial assessment of what control strategy, effort, and monetary resources are needed to reduce and eventually eliminate the invader. Sci- entists, in turn, can use the value of information to focus research in a way that not only increases the efficacy of control, but minimizes costs as well.

Published by Elsevier B.V.

## 1. Introduction

The demography of a species in a novel environment is usually un- known or highly uncertain and this can seriously compromise the con- trol of invasive species, particularly in the early stages of the invasion. Rather than postpone analyses of control actions until more is known, one possible approach is to elicit judgements from experts to form a basis for decision making (Canessa et al., 2015a; Converse et al., 2013; McBride et al., 2012; Runge et al., 2011). This so-called collective wisdom can be an effective method for forming accurate judgements in cases of high uncertainty (Lyon et al., 2015). In the case of invasive species, experts might be those familiar with the species' demography in its native range, those knowledgeable about closely related species, or those engaged in research or management of the invasive species in

situ. Although it is common practice to convene a panel of experts and ask them to produce consensus judgements about the values in ques- tion, preserving the diversity of opinions among experts is seen as more representative of the actual state of knowledge (Morgan, 2014). Demographic information, whether from expert opinion or empirical data, is then used to parameterize models of the invasive species, which in turn are used to predict how abundance, and perhaps spatial distribution, is likely to change over time in response to dynamic envi- ronmental conditions and specific control actions. In most cases these predictions will be highly uncertain, especially if based on expert opin- ion. Decision analysis that accounts for the probabilistic nature of these predictions is technically straightforward, but it depends critically on understanding the risk attitude of the decision maker (Canessa et al., 2016). For example, some outcomes may be so undesirable that the de- cision maker may accept a lower net benefit overall if it minimizes the chance of the most undesirable outcomes. There are also decision-mak- ing tools for cases of deep uncertainty – those in which possible out- comes lack any (reliable) stochastic structure (Johnson and Williams,

<footnote><sup>a</sup> U.S. Geological Survey, Wetland and Aquatic Research Center, 7920 NW 71 St., Gainesville, FL 32653, United States</footnote>

<footnote><sup>b</sup> Department of Wildlife Ecology and Conservation, University of Florida, PO Box 110430, Gainesville, FL 32611, United States</footnote>

<footnote><sup>c</sup> Department of Wildlife Ecology and Conservation, University of Florida, 3205 College Ave., Fort Lauderdale, Florida 33314, United States</footnote>

<footnote><sup>d</sup> U.S. Geological Survey, Wetland and Aquatic Research Center, 700 Cajundome Blvd., Lafayette, AL 70506, United States</footnote>

<footnote><sup>e</sup> U.S. Geological Survey, Fort Collins Science Center, 2150 Centre Ave., Bldg. C, Fort Collins, CO 80526, United States</footnote>

<footnote><sup>f</sup> Sam Noble Museum and Biology Department, University of Oklahoma, Norman, OK 73072, United States</footnote>

<footnote>* Corresponding author.
E-mail address: fjohnson@usgs.gov (F.A. Johnson).</footnote>
2015). In either case, demographic models must make predictions about the consequences of control actions in terms that are relevant to the decision maker's objectives. At a minimum, these objectives are likely to include the desire to minimize abundance of the invasive species and the costs of control. Other common objectives include minimizing the impacts of control on non-target species and protecting sites of high value (e.g., conservation reserves) from invasion.

A useful tool for addressing questions about the nature and implications of uncertainty is the expected value of information (VOI) (Canessa et al., 2015b; Clemen, 1996; Runge et al., 2011; Williams et al., 2011; Williams and Johnson, 2015a, 2015b). In particular, the expected value of perfect information (EVPI) expresses the gain in the value expected from optimal management if uncertainty were to be eliminated. Obviously, uncertainty can never be eliminated in resource management problems, but EVPI nonetheless provides a useful heuristic for determining the extent to which uncertainty is relevant to management decisions. EVPI is simply the difference between the objective value expected if there were no uncertainty and the best that could be expected with values that are averaged over uncertain outcomes. Also of potential use in the design of research and management programs is the expected value of partial information, in which the value of eliminating one of multiple sources of uncertainty is assessed. The value of information is often expressed monetarily, but any relevant performance metric will suffice. Expressing VOI in dollars is useful, however, for determining what managers should be willing to spend on monitoring and other data-collection programs designed to reduce uncertainty.

We illustrate the utility of expert elicitation, explicit recognition of uncertainty, and the value of information for directing management and research efforts for an invasive species of lizard in southern Florida. Argentine black and white tegus (Salvator merianae) were likely introduced from the pet trade in Miami-Dade County around 2000 (Krysko et al., 2011), and by 2010 were established as a breeding population (Pernas et al., 2012). Tegus are large, long-lived, fecund and omnivorous lizards (Fitzgerald, 1994). As such, they present a growing threat to ecological resources of concern in southern Florida, including the nests of crocodilians (Crocodilia), sea turtles (Chelonioida), and groundnesting birds (Mazzotti et al., 2015). While some early detection – rapid response efforts were attempted, it is clear that this population has grown rapidly and the focus has now shifted to containing (or eliminating) the population to prevent expansion into areas of high ecological value, including Everglades National Park (ENP), the Florida Keys, and critical habitat for American crocodiles (Crocodylus acutus) at the Turkey Point Nuclear Generating Station near Homestead, Florida.

There have been concerted efforts to control tegus in the natural areas just east of ENP (Southern Glades Wildlife and Environmental Area), but there is concern that these efforts are not meeting the identified management objective of containing the population. Current efforts to address the tegu invasion include trapping, as well as research projects to characterize movements, brumation, reproduction, habitat use, and diet, and to develop more efficient control methods. However, given limited resources, managers need to consider how to optimize the allocation of resources for management and research. To this end, a decision-making workshop was held in August 2015 to help identify management objectives and available actions, to link research and modeling efforts to management needs, and to identify the capacity and constraints of interested parties. This workshop was attended by representatives from various state and federal resource agencies, as well as university scientists. During and following the workshop, we elicited estimates of key demographic parameters of the tegu population in southern Florida from ten experts, who had both general knowledge and field experience with tegus in their native range or in southern Florida. We then considered the experts' uncertainty about these parameters to generate inferences concerning population growth rate and other demographic characteristics of interest. Finally, we used these inferences to examine control strategies intended to minimize tegu abundance and control costs.

## 2. Methods

### 2.1. Expert Elicitation

At the workshop, experts were provided the limited information available concerning survival and fecundity of wild tegus in South America (Fitzgerald, 1994). We then used a 3-point process (Soll and Klayman, 2004; Speirs-Bridge et al., 2010) to elicit from each expert their best guess (median) and its 95% confidence interval for tegu demographics in southern Florida. These estimates were collated and then shared with the entire group. The experts were then allowed to revise their estimates if they desired based on discussions within the group. Several months after the workshop, we contacted the experts via email to allow them to revise their estimates again if they wished. We encouraged the experts to draw on any published or unpublished information that they believed to be relevant. Although we encouraged experts to discuss their estimates, we did not use the Delphi process (or its variants), in which experts are encouraged to develop consensus on elicited values (Morgan, 2014). The concern is that such consensus is not based on genuine agreement, but rather is the result of strong group pressure (Woudenberg, 1991). Indeed, we were interested in assessing the full range of experts' opinions about the uncertain parameters (Clemen and Winkler, 1999; Morgan, 2014).

There is an extensive literature on how to combine the diverse opinions of experts (Budescu and Rantilla, 2000; Lyon et al., 2015). A general conclusion from this literature is that some form of averaging is almost always optimal. The most common averaging approaches involve weighting experts equally or weighting experts by the precision of their elicited values. Giving more weight to more precise judgments can give good results (Yaniv and Foster, 1997), but more recent literature indicates that overconfidence tends to predominate in interval judgments (Morgan, 2014; Soll and Klayman, 2004). Several methods have been suggested to reduce overconfidence, including the 3-point elicitation process we used (Soll and Klayman, 2004). Nonetheless, we believed it prudent to weight experts equally because we could not be sure that precision was an accurate reflection of an expert's actual knowledge about the parameter in question. Equal weighting of experts can better account for outlying opinions, is guaranteed to be no less accurate than the typical individual judgments, and has been shown to be surprisingly accurate (Lyon et al., 2015; Morgan, 2014).

Most experts' confidence intervals were asymmetric, and this requires special consideration in aggregating estimates (O'Leary et al., 2015). For each expert and for each parameter, we fit an appropriate statistical distribution to the three elicited values depending on the nature of the demographic parameter. We used a beta distribution for rates that are naturally constrained between zero and one (e.g., survival). We used a log-normal distribution for parameters that were bounded by zero and infinity (e.g., clutch size). Distributions were fit using the rriskDistributions package (Belgorodski et al., 2015) of the open-source computing language R (RCoreTeam, 2016). We then used the fitted distribution for each expert and for each parameter to generate 10,000 samples of the parameter in question. The samples for a given parameter were then pooled across experts (equal weighting of experts) to produce a single discrete distribution for each parameter. We then generated 50,000 complete sets of all parameters by drawing samples from the cumulative distribution functions for the pooled samples. Because considerable correlation existed between some pairs of elicited demographic parameters (see Supplementary Material, Fig. S1), we used a Cholosky decomposition (Morris and Doak, 2002) to preserve this correlation structure. We then confirmed that the marginal distributions of parameters and the correlation matrix were similar to the original data. These 50,000 sets of demographic parameters were then used to draw inference from a population model.
## 2.2. Population Model

Matrix models can be effective for understanding the growth potential of an invasive population, the life stages with the most influence on population growth, and the importance of transient dynamics (i.e., those occurring prior to reaching the asymptotic growth rate) in predicting abundance and responses to control efforts (Morris et al., 2011; Shea and Kelly, 1998). We posited a post-birth pulse model in which four age classes of tegus are recognized: hatchlings, 1 year-old, 2 year-olds, and 3 + year-olds. Only tegus aged three years or older are assumed to breed regularly, although we acknowledge that size may be a more important determinant of sexual maturity than age (Fitzgerald et al., 1993). We parameterized survival (S) and fecundity (F) using estimates elicited from our 10 experts. Fecundity was defined as the product of clutch size and egg survival divided by 2, as we assumed a 1:1 sex ratio in the population. We emphasize that this model has no spatial aspect and thus assumes there is a single population in Miami-Dade County. The life cycle diagram and matrix model are provided in Fig. 1.

The matrix model was parameterized using the set of 50,000 demographic parameters described above. To understand the population level consequences of the tegu life cycle we used the popdemo package of R (Stott et al., 2014) to calculate the finite growth rate (λ), stable age distribution, damping ratio (d), and elasticities (e) of demographic rates for each of the 50,000 matrices. We chose elasticities rather than sensitivities because proportional (rather than additive) perturbations were more in line with the type of management intervention we examined (i.e., removal rate) (Caswell, 2001). Convergence time to the stable age distribution was calculated using the mean matrix and a range of plausible initial conditions; these computations were also conducted using the popdemo package. We calculated tegu lifespan ($I_{max}$) as the number of years required for 99% of an initial population of hatchlings to die (Slade et al., 1998).

## 2.3. Tegu Control Strategies

We first calculated the removal rate ($h$) required to stabilize the population, assuming that the same removal rate would be applied to all age classes, as $h = (λ - 1)/λ$ (Slade et al., 1998). Then we used the method of Hauser et al. (2006) to calculate combinations of removal rates for hatchlings, 1, 2, and 3 + year-olds that would stabilize the population. We used the mean matrix (i.e., the mean of the 50,000

<figure><img src="image_5.png" /></figure>

$$A = \begin{bmatrix} 0 & 0 & FS_2 & FS_3 \\ S_h & 0 & 0 & 0 \\ 0 & S_1 & 0 & 0 \\ 0 & 0 & S_2 & S_3 \end{bmatrix}$$

<caption>**Fig. 1.** Post birth-pulse, life-cycle diagram (top) and corresponding matrix model (bottom) for Argentine black and white tegus in southern Florida. Age classes are hatchlings (H), and 1, 2, and 3 + year-olds. Demographic rates are survival (S) and fecundity (F = hatchlings produced per breeding female).</caption>

matrices) for these calculations. The purpose of these calculations was to demonstrate how various combinations of age-specific removal rates could prevent further population growth, and to emphasize that a lack of control over age-specific removal rates can lead to unpredictable population trajectories (Hauser et al., 2006).

The principal method of tegu control currently available is to trap and remove individuals with some level of effort, which is summarized by a removal rate, $h$ (i.e., the proportion of an age class removed per year). The time of year when trapping occurs affects the age classes available for capture and this can be used to target specific age classes. We examined control strategies for reducing tegu population size by examining a range of removal rates targeted at certain age classes. As a basis for comparison, strategy A consisted of varying removal rates targeted at all age classes equally (here and elsewhere we assume that trap mortality is additive to other sources of mortality), such that the projection matrix with control was:

$$A_a = \begin{bmatrix} 0 & 0 & (1-h)FS_2 & (1-h)FS_3 \\ (1-h)S_h & 0 & 0 & 0 \\ 0 & (1-h)S_1 & 0 & 0 \\ 0 & 0 & (1-h)S_2 & (1-h)S_3 \end{bmatrix}$$

We used a coauthor's (FM) empirical data on catch per trap day ($ω = 0.034$), total trap days in a season ($δ = 192$; February–October), and an approximate cost per trap per season ($θ = \$1000USD$). The total annual cost (C) associated with this (and the following) strategies was calculated by first determining the total number of removals (R) for a simulated population vector ($\bar{N}$) and age-specific removal rate ($h$). Thus, for strategy A, removals were:

$$R = \begin{bmatrix} h & 0 & 0 & 0 \\ 0 & h & 0 & 0 \\ 0 & 0 & h & 0 \\ 0 & 0 & 0 & h \end{bmatrix} \cdot \bar{N}$$

Thus, total annual cost (C) was:

$$C = \frac{R}{ωγ}θ$$

Strategy B consisted of varying removal rates for animal aged 1 + years (i.e., no hatchlings), such that the projection matrix with control was:

$$A_b = \begin{bmatrix} 0 & 0 & (1-h)FS_2 & (1-h)FS_3 \\ S_h & 0 & 0 & 0 \\ 0 & (1-h)S_1 & 0 & 0 \\ 0 & 0 & (1-h)S_2 & (1-h)S_3 \end{bmatrix}$$

Catch per trap day ($ω = 0.043$), trap days per season ($δ = 131$), and cost per trap per season ($θ = \$683USD$) were derived from trap statistics during the March–August period when few hatchlings are caught.

Strategy C consisted of varying rates of nest destruction, such that the projection matrix with control was:

$$A_c = \begin{bmatrix} 0 & 0 & (1-h)FS_2 & (1-h)FS_3 \\ S_h & 0 & 0 & 0 \\ 0 & S_1 & 0 & 0 \\ 0 & 0 & S_2 & S_3 \end{bmatrix}$$

The number of nests destroyed was a function of the removal rate and the number of 3 + year-olds alive at the time of breeding:

$$Nests = h \left( \frac{n_2 S_2 + n_3 S_3}{2} \right)$$

where $h$ = removal rate, $n_2$ and $n_3$ are the number of 2 and 3 + year-olds at the start of the annual cycle, respectively, and $S_2$ and $S_3$ are the
annual survival rates of 2 and 3 + year-olds, and division by 2 is because only females nest. Empirical data on the cost of nest searching are not available, but we thought it was important to see how this strategy might perform using assumed, but plausible, trapping statistics. We assumed conservatively that one search crew could find 3 nests in the March–May period (δ=92 days, ω=3/92=0.03) and that the cost of nest searching was twice that of trapping (θ=$958USD).

Finally, strategy D consisted of varying removal rates of breeding-aged adults just prior to nesting in the spring (δ=65days; March–May), such that the projection matrix with control was:

$$A_d = \begin{bmatrix} 0 & 0 & (1-h)FS_2 & (1-h)FS_3 \\ S_h & 0 & 0 & 0 \\ 0 & S_1 & 0 & 0 \\ 0 & 0 & (1-h)S_2 & (1-h)S_3 \end{bmatrix}$$

Catch per day (ω=0.44) and cost per trap per season (θ=$338USD) were estimated from empirical trap data.

## 2.4. Simulating Age-specific Control Strategies

We simulated application of the four control strategies (A, B, C, and D) with removal rates varying from h = 0.0 to h = 0.6 in increments of 0.1; thus, there was a no-removal strategy and 6 possible removal rates for each of the 4 strategies (25 combinations in all). We initialized each simulation of the 50,000 population matrices with a draw from the distribution of the elicited population size in 2008 and the stable age distribution associated with each matrix. Note that the simulation of each matrix is deterministic (no environmental variation), so that each only needed to be run once. We used the elicited size of the tegu population in 2008 because of uncertainty about the year of introduction and the size of the founding population. We assumed no tegu removals until 2016, and then simulated the removal strategy for 10 years. We then examined the performance of each removal strategy by plotting medians of the final population size and average annual cost of removals. The goal here was to identify strategies that were Pareto-efficient (Kennedy et al., 2008) and most likely to produce both low tegu abundance and cost of removals.

After using the above exercise to identify a subset of Pareto-efficient alternatives, we normalized final population size and cumulative costs for each matrix model, such that 0 was the worst performing strategy and 1 was the best performing strategy in terms of each management objective. We then combined the two management objectives into a single objective function, using a proportional weight w to specify the relative importance of annual removal cost:

$$V(\pi) = w(C|\pi, m) + (1-w)(N|\pi, m)$$

where the value Vof a removal strategy π is the sum of the weighted, normalized cumulative cost C and the weighted, normalized, ending population size N, given a particular matrix model m. To identify the optimal removal strategy we used the concept of stochastic dominance to account for the full range of uncertainty concerning strategy performance (Canessa et al., 2016). This approach first examines the cumulative distribution function (cdf) of objective values (over all matrix models) for each strategy being investigated. The strategy associated with the lowermost cdf is the optimal choice if, and only if, it does not cross the cdf for any of the remaining strategies (called first-order dominance). If cdf's cross, then it is necessary to know the general risk attitude of the decision maker to identify an optimal strategy. We assumed a risk averse (rather than risk seeking) decision maker, so the ascending integrals of the cdf's were examined. An ascending integral depicts the area under the cdf from 0 to increasingly larger values of x (the objective value, V(π), in this case). The strategy associated with the lowermost ascending integral is the optimal choice for a risk-averse decision maker if, and only if, it does not cross the ascending integral for any of the remaining strategies (called second-order

dominance). We calculated ascending integrals of the cdf's using numerical integration.

## 2.5. Value of Information

We calculated the expected value of perfect information (EVPI) based on our subset of Pareto-efficient alternative control strategies. EVPI is the difference between the best performance (in objective function value) expected if all model uncertainty could be eliminated (the first term) and the best performance expected in the face of continued model uncertainty (the second term):

$$EVPI = E_m \left[ \max_{\pi} (V(\pi, m)) | m \right] - \max_{\pi} E_m [V(\pi, m) | m]$$

Both terms involve maximizing and averaging, but in different orders. In the first term, we are averaging the value of the optimal strategy for each model. In the second term we are maximizing the model-averaged performance of each strategy. We examined EVPI for varying weights on cost from 0 to 1, and expressed EVPI as the proportional gain in expected management performance.

We also calculated the expected value of partial information (EVPXI) (Williams and Johnson, 2015b). We focused on the potential gain in performance that could be expected by eliminating uncertainty about fecundity and survival rates. We again used our subset of Pareto-efficient alternative control strategies and, for each demographic parameter, began by assigning the 50,000 potential demographic rates into one of 5 bins based on the 20% quantiles of their empirical distribution. The probability associated with each of the bins was thus 0.20. The second term of EVPXI is identical to that of EVPI, and represents the best performance expected in the face of continued uncertainty. The first term of EVPXI was calculated by first averaging the performance of each strategy in each of the 5 bins. Then for each bin, we took the maximum performance across strategies. Finally these values were averaged using the bin probability as weights. This first term thus expresses the expected performance that could be expected if uncertainty about the most appropriate bin for the demographic rate in question were known, while accounting for the residual uncertainty in other demographic rates. EVPXI is thus:

$$EVPXI = E_b \left[ \max_{\pi} E_i (V_i(\pi, b)) | b \right] - \max_{\pi} E_m [V(\pi, m)]$$

where b = a parameter bin and i = an individual model within a bin. We calculated EVPXI for fecundity and survival rates only for the case where ending population size and cumulative costs were equally weighted.

# 3. Results

As expected, empirical distributions for parameters based on expert opinion reflected a high degree of uncertainty (Table 1; Supplementary Material, Figs. S2–S8). Survival rates tended to be bimodal, with the exception of survival of 1-year olds. Mean lifespan was $l_{max}$=11.7,

<caption>**Table 1**</caption>

Summary statistics for demographic parameters of Argentine black and white tegus in South Florida elicited from 10 experts.

<table><thead><tr><th>Parameter</th><th>Mean</th><th>Median</th><th>90% credible interval</th></tr></thead><tbody><tr><td>Hatchling survival, S<sub>h</sub></td><td>0.273</td><td>0.211</td><td>0.082–0.664</td></tr><tr><td>1 year-old survival, S<sub>1</sub></td><td>0.501</td><td>0.502</td><td>0.269–0.738</td></tr><tr><td>2 year-old survival, S<sub>2</sub></td><td>0.616</td><td>0.631</td><td>0.380–0.811</td></tr><tr><td>3+ year-old survival, S<sub>3+</sub></td><td>0.726</td><td>0.762</td><td>0.511–0.899</td></tr><tr><td>Lifespan, l<sub>max</sub></td><td>11.7</td><td>8.9</td><td>4.0–24.9</td></tr><tr><td>Clutch size, c</td><td>26.5</td><td>25.8</td><td>20.2–34.0</td></tr><tr><td>Egg survival, S<sub>e</sub></td><td>0.813</td><td>0.894</td><td>0.007–1.000</td></tr><tr><td>Population size in 2008</td><td>768</td><td>500</td><td>75–2483</td></tr></tbody></table>
comparable to a value of $l_{max}$ = 13 reported from the tegu's native range (Fitzgerald, 1994). Clutch size exhibited a fairly smooth distribution, but the distribution of egg survival reflected widely varying judgements among experts. Initial population size (i.e., population size in 2008) was 768 tegus (90% Credible Interval: 75–2483).

Based on the 50,000 alternative matrix models, the mean finite growth rate was $\lambda$ = 1.29 (90% Credible Interval: 0.461–2.313) (Supplemental Material, Fig. S9). A population exhibiting this constant growth rate will double every 2.8 years. The mean stable age distribution was 69% hatchlings, 13% 1 year-olds, 5% 2 year-olds, and 13% 3+ year-olds. Due to a highly skewed distribution, we report the median damping ratio as $d$ = 1.58 (90% Credible Interval: 1.30–11.2), suggesting the population can exhibit significant transient dynamics (Caswell, 2001). For example, based on the mean matrix, populations not at the stable age distribution typically take $\ge$ 10 years for the age ratio to stabilize. Mean elasticity was highest for survival of 3+ year-olds, followed by fecundity, hatchling survival, and survival of 1 year-olds (Table 2). Survival of 2 year-olds had the lowest elasticity.

The mean removal rate required to stabilize the population, assuming all age classes are removed at the same rate, was $h$ = 0.19 (90% Credible Interval: 0.00–0.54). When no hatchlings can be removed, the removal rates of the other age classes necessary to stabilize the population generally were >0.25 (Fig. 2). In contrast, if high removal rates of hatchlings (or nests) could be achieved (e.g., >0.4), removal rates for the other age classes could be reduced substantially.

Based on simulations, the uncontrolled tegu population potentially grew from a median of 500 individuals (90% Credible interval: 75–2473) in 2008 to a median of 2483 individuals (90% Credible Interval: 2–410,800) in 2016. Of the possible removal strategies, removing all age classes at the same rate (strategy A) performed well in terms of mean tegu abundance, but was the most costly on average (Fig. 3). The removal strategies affiliated with nest destruction (strategy C) were of low to moderate costs, perhaps not surprisingly so due to our conservative cost assumption. However, the strategy C alternatives varied widely in terms of minimizing tegu abundance. To make this a more viable alternative, nest searching would have to be more efficient than we assumed. Removal strategies focused on either removing all age classes except hatchlings (strategy B) or removing breeding-aged adults in the spring (strategy D) were most cost-effective overall. We chose four of these along the Pareto frontier for analysis of stochastic dominance: (1) strategy B with $h$ = 0.5; (2) strategy B with $h$ = 0.6; (3) strategy D with $h$ = 0.5; and (4) strategy D with $h$ = 0.6. With equal weights on the objectives to minimize cost and tegu abundance, strategy D with $h$ = 0.6 had the highest mean value $V(\pi)$ = 0.77, closely followed by strategy B with $h$ = 0.6, which had a mean of $V(\pi)$ = 0.73 (recall that $V(\pi)$ = 1 represents the best possible outcome). Without considering the range in objective values ($V(\pi)$), a risk-neutral decision maker would choose one of these two strategies. Examining medians and interquartile ranges (IQR) because of the highly variable and skewed distributions, strategy D with $h$ = 0.6 produced a median population size of 158 tegus after 10 years of control (IQR: 0–16,930), at a median

<caption>**Table 2**</caption>

Elasticities (and SD) and the value of partial information (EVPXI) for demographic parameters of Argentine black and white tegus in South Florida as elicited from 10 experts. Elasticities represent the proportional change in the finite growth rate expected with a proportional change in a demographic rate. EVPXI is the proportional increase in management performance (i.e., objective value) that could be expected as a result of eliminating uncertainty about a demographic rate.

<table><thead><tr><td>Demographic parameter</td><td>Elasticity (SD)</td><td>EVPXI</td></tr></thead><tbody><tr><td>Fecundity, F</td><td>0.208 (0.064)</td><td>0.123</td></tr><tr><td>Hatchling survival, S<sub>h</sub></td><td>0.208 (0.064)</td><td>0.134</td></tr><tr><td>1 year-old survival, S<sub>1</sub></td><td>0.208 (0.064)</td><td>0.146</td></tr><tr><td>2 year-old survival, S<sub>2</sub></td><td>0.118 (0.031)</td><td>0.175</td></tr><tr><td>3+ year-old survival, S<sub>3+</sub></td><td>0.259 (0.217)</td><td>0.142</td></tr></tbody></table>

annual cost of $8641 (IQR: $580–$166,500). The median number of tegus removed annually was 73 (IQR: 5–1406).

None of the subset of 4 Pareto-efficient strategies had first-order dominance (Fig. 4, left panel), meaning that the optimal strategy cannot be identified without knowing the risk attitude of the decision maker. The risk-averse decision maker, however, should have a clear preference for strategy D with $h$ = 0.6, as this is second-order dominant to the other 3 alternatives (Fig. 4, right panel).

The expected value of perfect information varied depending on the relative weight of cost in the objective function, with the highest EVPI (a 25% gain in performance) at a cost weight of $w$ = 1.0 (Fig. 5). In this case, of course, cost is the sole objective and the gain in performance simply represents the cost savings when model uncertainty is eliminated; tegu abundance is irrelevant. At a cost weight of $w$ = 0.0, EVPI = 0, meaning that the expected performance is equivalent in the case of model certainty or model uncertainty (i.e., the optimal strategy for all models is the one that minimizes tegu abundance, regardless of cost; strategy B with $h$ = 0.6). Beyond this, EVPI increases rapidly up to a cost weight of $w$ = 0.42, after which EVPI goes up more slowly with further increases in cost weight. Most of the potential performance gain from eliminating model uncertainty is achieved when there is approximate parity in the relative importance of the two objectives. At a cost weight of $w$ = 0.5, the expected value of partial information was more homogeneous among fecundity and survival rates than was the case for elasticities (Table 2). Eliminating uncertainty about survival of 2 year-olds had the highest EVPXI (18%), but the lowest elasticity. Fecundity had the lowest EVPXI (12%), but an intermediate elasticity.

## 4. Discussion

As a benchmark, our analyses suggested that a removal rate of $h$ = 0.19 on all age classes each (and every) year would be required to stabilize the tegu population in southern Florida. Assuming trap efficiency is constant, removal rates higher than this may be feasible (by increasing the number of traps) and potentially affordable, and could potentially reduce tegu abundance. We urge caution, however, as there is a great deal of uncertainty associated with this value. Based on our elicited demographic rates, there is a 50% chance that the removal rate needed to stabilize the population could be higher, and possibly much higher. Moreover, it is not unusual for trapping efficiency to decline with effort and decreasing populations (Epanchin-Niell and Wilen, 2012), which would make higher removal rates less feasible. Finally, even if population size were stabilized, tegus would likely continue to expand their range; yet our models do not account for dispersal into areas not subject to control.

Targeting older age classes (strategies B and D) appears to have the best chance of minimizing both tegu abundance and control costs. Current trapping efforts (which span 192 days) tend to produce more of the older (or at least larger) age classes, and it appears additional cost savings could be incurred by limiting trapping to those periods that produce the oldest individuals. Whatever the targeted removal rates, we stress that the removal rate of each class must be at least partially controlled (i.e., removal rates vary about some mean rate that does not vary). Otherwise, even a constant removal rate overall will lead to unpredictable changes in tegu abundance (Hauser et al., 2006).

We also wish to emphasize that each of our simulated populations was initialized in 2008 at its stable age distribution as based on the corresponding matrix model. We believe this to be likely if and only if tegus began breeding immediately after the initial release(s) around 2000. Otherwise, tegu abundance would likely be very different than in the year we began simulated control (2016). For example, a population consisting of mostly breeding-age adults in 2008 would have resulted in a much larger population in 2016 than we simulated and, thus, our control costs and final tegu abundance would be negatively biased. Finally, we remind the reader that our matrix models exhibited no density-dependent growth, as seems likely at some (unknown) level of tegu
<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Removal rates of Argentine black and white tegus in southern Florida required to stabilize the population, as based on expert elicitation. Each panel represents a different (fixed) removal rate of hatchlings. The x and y axes represents removal rates of 1 and 2 year-olds respectively. Colors represent the required removal rates of 3 + year-olds. White space in the top right of the graphs represent combinations that would result in a declining, rather than stable, population. White space in the bottom left of the graphs represent conditions for which a solution to stabilize the population does not exist.</figcaption></figure>

abundance. Accounting for density dependence may lead to lower values of removal rates to stabilize the population and therefore may result in more optimistic outcomes in terms of population control.

With respect to the value of information, some authors (e.g., (Moore and McCarthy, 2010; Walters, 1986) have observed that EVPI is often low in practice. Indeed, our finding of EVPI ≤ 25% is relatively large compared to some reports in the natural resource literature (Johnson et al., 2014; Williams and Johnson, 2015b), but nonetheless comparable to others (Runge et al., 2011). EVPI will be low if uncertainty is low or if optimal management actions are insensitive to model choice. In some cases, management may be constrained (e.g., by laws or cultural norms) in such a way that it is not possible to capitalize on what is learned. Clearly, EVPI will be low where time horizons are short (Hauser and Possingham, 2008), or where the future is heavily discounted (Moore et al., 2008). We also note that the magnitude of EVPI in our study was variable, depending on the relative emphasis placed on minimizing tegu control costs. In fact, EVPI shrinks to zero when control cost is of no consequence. This is because a sole objective to minimize tegu abundance will always suggest the highest available removal rate, regardless of how the population model is parameterized (i.e., there is no value in reducing model uncertainty). The appropriate trade-off of the objectives to minimize cost and minimize tegu abundance is necessarily a value judgement of the decision maker. These

analyses can help inform this decision, but cannot identify the “correct” tradeoff because there is no objective basis for doing so.

Tegu population growth in southern Florida appears to be most sensitive to proportional changes in survival of 3 year-olds, perhaps not surprisingly because tegus are relatively long-lived. Interestingly, EVPXI was highest for survival of 2 year-olds, yet this demographic rate had the lowest elasticity. In general, the range of EVPXI values for all demographic parameters was smaller than that of elasticities. These patterns are perhaps not surprising given that elasticity and EVPXI address different questions. Elasticity suggests which demographic rates could be perturbed to produce the greatest change in growth rate of the tegu population. In a management setting, that information has value only to the extent that control efforts could feasibly target that particular demographic rate. EVPXI, on the other hand, characterizes the loss in management performance resulting from uncertainty about the various demographic rates. In this light, we suggest EVPXI is a better metric than elasticity for focusing research and monitoring efforts because it is a metric tied explicitly to management performance, which must consider cost as well as tegu abundance. Ongoing telemetry and camera-trapping efforts in southern Florida could contribute to meeting this need. Without empirical data, managers run considerable risk of allocating too little effort to control the expanding tegu population.
<figure><img src="image_7.png" /><figcaption>**Fig. 3.** Simulated performance of control strategies for Argentine black and white tegus in southern Florida, as based on expert elicitation. Strategy A consisted of varying removal rates targeted at all age classes equally. The digit following the letter represents the removal rate times 10 (e.g., 1 stands for a removal rate of $h = 0.10$). Strategy B consisted of varying removal rates for animal aged 1+ years (i.e., no hatchlings). Strategy C consisted of varying rates of nest destruction. Strategy D consisted of varying removal rates of breeding-aged adults just prior to nesting in the spring. Values on the x and y axes are medians from 50,000 population matrices simulated for 10 years of control. Strategies performing the best on both objectives are those in the bottom left of the graph.</figcaption></figure>

This study represents the first attempt to model population demography of Argentine black and white tegus in southern Florida. Given the absence of empirical data, it was not surprising that experts varied widely in their judgements about demographic rates of this newly established invader. This variation produced rather extreme variability in simulated population sizes, and thus in control efficacy and costs. Nonetheless, quantification of these expert opinions, as well as their

<figure><img src="image_8.png" /><figcaption>**Fig. 5.** The expected value of perfect information (EVPI), expressed as a proportional gain in management performance, for controlling Argentine black and white tegus in southern Florida. EVPI is the expected increase in objective value that could be attained by eliminating all uncertainty about tegu demography as expressed by a panel of experts. The x axis represents the weight placed on the objective to minimize annual control cost. The weight on the objective to minimize tegu abundance after 10 years is the compliment of the cost weight.</figcaption></figure>

associated uncertainty, is a critical first step in developing an informed control strategy. Acting more aggressively to control tegus before empirical data are available may be prudent, particularly given the population's estimated doubling time of about three years. We note that the parameters of our population model can be treated as prior distributions, which can then be updated as empirical information from field studies becomes available (Royle and Dorazio, 2008). Finally, we note that a separate population of tegus is also established near Tampa, Florida (Engeman et al., 2011), and there have been multiple tegu sightings at locations across peninsular Florida (https://www.eddmaps.org/distribution/viewmap.cfm?sub=18346). We thus believe this study could help provide preliminary guidance for controlling what appears to be a growing threat throughout the state.

By definition, invasions of exotic plants and animals occur in novel environments, where demography may be poorly understood even if the species is well studied in its native range. Expert opinion, combined with suitable techniques like stochastic dominance to explicitly consider uncertainty, can be a valuable approach for an initial assessment of what control strategy, effort, and monetary resources are needed to reduce and eventually eliminate the invader. In these situations, like with the tegu, any honest effort to characterize uncertainty is likely to lead to a level of precision that is less than satisfying. However, such quantification of uncertainty can be useful to management because those outcomes with apparently low probability, but with highly undesirable consequences, may need to be considered in addition to more likely outcomes. And the attitude towards that risk must ultimately reflect social values and norms, which are the purview of managers rather than scientists. Scientists, in turn, can use the value of information to focus research in a way that not only increases the efficacy of control, but minimizes costs as well.

Adaptive management has also been increasingly suggested as a way to reduce uncertainty and improve management performance (Williams, 2011). Doremus (2011) made an effective case that adaptive management is an information problem, in that the key question to be addressed is whether the lack of information about ecological processes and system responses to human intervention is the principal impediment to decision making and effective management. Decision makers thus require some way to identify pertinent and reducible uncertainties so they can determine whether a particular resource conservation issue

<figure><img src="image_9.png" /><figcaption>**Fig. 4.** Cumulative distribution function (cdf) of objective values (left panel) and ascending integrals of those cdf's (right panel) for 4 strategies to control Argentine black and white tegus in southern Florida, as based on simulation of 50,000 population matrices for 10 years of control. Strategy B consisted of varying removal rates for animal aged 1+ years (i.e., no hatchlings). Strategy D consisted of varying removal rates of breeding-aged adults just prior to nesting in the spring. The digit following the letter represents the removal rate times 10 (e.g., 5 stands for a removal rate of $h = 0.5$).</figcaption></figure>
is a good candidate for adaptive management, whether learning through management is possible, and whether an effective adaptive management program can be designed. Adaptive management can be expensive, and decision makers naturally want some assurance that those costs can be offset by improvements in management performance. Quantifying the value of perfect and partial information can be a powerful tool for helping address these issues.

## Acknowledgments

We thank all those that attended the tegu-control workshop in August 2015, as well as Tylan Dean and the National Park Service for sponsoring that event. In addition to many of the coauthors, we thank Kyle Allen, Bryan Falk, Michelle McEachern Collier, Lindsey Garner, and Mike Rochford for participating in the expert elicitation. Funding for this research was provided by the U.S. Geological Survey, Greater Everglades Priority Ecosystem Science program. Any use of trade, product, or firm names in this article is for descriptive purposes only and does not imply endorsement by the U.S. Government.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at http://dx.doi.org/10.1016/j.ecolecon.2017.03.004.

## References

Belgorodski, N., Greiner, M., Tolksdorf, K., Schueller, K., 2015. *rriskDistributions: Fitting Distributions to Given Data or Known Quantiles*.

Budescu, D.V., Rantilla, A.K., 2000. *Confidence in aggregation of expert opinions. Acta Psychol.* **104**, 371–398.

Canessa, S., Converse, S.J., West, M., Clemann, N., Gillespie, G., McFadden, M., Silla, A.J., Parris, K.M., McCarthy, M.A., 2015a. *Planning for ex situ conservation in the face of uncertainty*. *Conserv. Biol.* **30**, 599–699.

Canessa, S., Guillera-Arroita, G., Lahoz-Monfort, J.J., Southwell, D.M., Armstrong, D.P., Chadès, I., Lacy, R.C., Converse, S.J., 2015b. *When do we need more data? A primer on calculating the value of information for applied ecologists*. *Methods Ecol. Evol.* **6**, 1219–1228.

Canessa, S., Ewen, J.G., West, M., McCarthy, M.A., Walshe, T.V., 2016. *Stochastic dominance to account for uncertainty and risk in conservation decisions*. *Conserv. Lett.* **9**, 260–266.

Caswell, H., 2001. *Matrix Population Models: Construction, Analysis, and Interpretation*. second ed. Sinauer Associates, Inc., Sunderland, MA.

Clemen, R.T., 1996. *Making hard decisions. An Introduction to Decision Analysis*, second ed. Duxbury Press, Pacific Grove, CA.

Clemen, R.T., Winkler, R.L., 1999. *Combining probability distributions from experts in risk analysis*. *Risk Anal.* **19**, 187–203.

Converse, S.J., Moore, C.T., Folk, M.J., Runge, M.C., 2013. *A matter of tradeoffs: reintroduc- tion as a multiple objective decision*. *J. Wildl. Manag.* **77**, 1145–1156.

Doremus, H., 2011. *Adaptive management as an information problem*. *N. C. Law Rev.* **89**, 1455–1495.

Engeman, R., Jacobson, E., Avery, M.L., Meshaka Jr., W.E., 2011. *The aggressive invasion of exotic reptiles in Florida with a focus on prominent species: a review*. *Curr. Zool.* **57**, 599–612.

Epanchin-Niell, R.S., Wilen, J.E., 2012. *Optimal spatial control of biological invasions*. *J. Environ. Econ. Manag.* **63**, 260–270.

Fitzgerald, L.A., 1994. *The interplay between life history and environmental stochasticity: implications for the management of exploited lizard populations*. *Am. Zool.* **34**, 371–381.

Fitzgerald, L.A., Cruz, F.B., Perotti, G., 1993. *The reproductive cycle and the size at maturity of Tupinambis rufescens* (Sauria: Teiidae) in the Dry Chaco of Argentina. *J. Herpetol.* **27**, 70–78.

Hauser, C.E., Possingham, H.P., 2008. *Experimental or precautionary? Adaptive manage- ment over a range of time horizons*. *J. Appl. Ecol.* **45**, 72–81.

Hauser, C.E., Cooch, E.G., Lebreton, J., 2006. *Control of structured populations by harvest*. *Ecol. Model.* **196**, 462–470.

Johnson, F.A., Williams, B.K., 2015. *A decision-analytic approach to adaptive resource management*. In: Allen, C.R., Garmestani, A.S. (Eds.), *Adaptive Management of So- cial-ecological Systems*. Springer, Netherlands, pp. 61–84.

Johnson, F.A., Hagan, G., Palmer, W.E., Kemmerer, M., 2014. *Uncertainty, robustness, and the value of information in managing a population of northern bobwhites*. *J. Wildl. Manag.* **78**, 531–539.

Kennedy, M.C., Ford, E.D., Singleton, P., Finney, M., Agee, J.K., 2008. *Informed multi-objec- tive decision-making in environmental management using Pareto optimality*. *J. Appl. Ecol.* **45**, 181–192.

Krysko, K.L., Burgess, J.P., Rochford, M.R., Gillette, C.R., Cueva, D., Enge, K.M., Somma, L.A., Stabile, J.L., Smith, D.C., Wasilewski III, J.A., G. N. K., Granatosky, M.C., Nielson, S.V., 2011. *Verified non-indigenous amphibians and reptiles in Florida from 1863 through 2010: outlining the invasion process and identifying invasion pathways and stages*. *Zootaxa* **3028**, 1–64.

Lyon, A., Wintle, B.C., Burgman, M., 2015. *Collective wisdom: methods of confidence in- terval aggregation*. *J. Bus. Res.* **68**, 1759–1767.

Mazzotti, F.J., McEachern, M., Rochford, M., Reed, R.N., Eckles, J.K., Vinci, J., Edwards, J., Wasilewski, J., 2015. *Tupinambis merianae* as nest predators of crocodilians and tur- tles in Florida, USA. *Biol. Invasions* **17**, 47–50.

McBride, M.F., Garnett, S.T., Szabo, J.K., Burbidge, A.H., Butchart, S.H.M., Christidis, L., Dutson, G., Ford, H.A., Loyn, R.H., Watson, D.M., Burgman, M.A., 2012. *Structured elic- itation of expert judgments for threatened species assessment: a case study on a con- tinental scale using email*. *Methods Ecol. Evol.* **3**, 906–920.

Moore, A.L., McCarthy, M.A., 2010. *On valuing information in adaptive-management models*. *Conserv. Biol.* **24**, 984–993.

Moore, A.L., Hauser, C.E., McCarthy, M.A., 2008. *How we value the future affects our desire to learn*. *Ecol. Appl.* **18**, 1061–1069.

Morgan, M.G., 2014. *Use (and abuse) of expert elicitation in support of decision making for public policy*. *Proc. Natl. Acad. Sci.* **111**, 7176–7184.

Morris, W.F., Doak, D.F., 2002. *Quantitative Conservation Biology*. Sinauer, Sunderland, Massachusetts, USA.

Morris Jr., J.A., Shertzer, K.W., Rice, J.A., 2011. *A stage-based matrix population model of invasive lionfish with implications for control*. *Biol. Invasions* **13**, 7–12.

O'Leary, R.A., Low-Choy, S., Fisher, R., Mengersen, K., Caley, M.J., 2015. *Characterising un- certainty in expert assessments: encoding heavily skewed judgements*. *PLoS ONE* **10**, e0141697.

Pernas, T., Giardina, D.J., McKinley, A., Parns, A., Mazzotti, F.J., 2012. *First observations of nesting by the Argentine black and white tegu, Tupinambis merianae*, in south Florida. *Southeast. Nat.* **11**, 765–770.

RCoreTeam, 2016. *A Language and Environment for Statistical Computing*. R Foundation for Statistical Computing, Vienna, Austria.

Royle, J.A., Dorazio, R.M., 2008. *Hierarchical Modeling and Inference in Ecology*. Academic Press, London, England.

Runge, M.C., Converse, S.J., Lyons, J.E., 2011. *Which uncertainty? Using expert elicitation and expected value of information to design an adaptive program*. *Biol. Conserv.* **144**, 1214–1223.

Shea, K., Kelly, D., 1998. *Estimating biocontrol agent impact with matrix models: Carduus nutans* in New Zealand. *Ecol. Appl.* **8**, 824–832.

Slade, N.A., Gomulkiewicz, R., Alexander, H.M., 1998. *Alternatives to Robinson and Redford's method of assessing overharvest from incomplete demographic data*. *Conserv. Biol.* **12**, 148–155.

Soll, J.B., Klayman, J., 2004. *Overconfidence in interval estimates*. *J. Exp. Psychol. Learn. Mem. Cogn.* **30**, 299.

Speirs-Bridge, A., Fidler, F., McBride, M., Flander, L., Cumming, G., Burgman, M., 2010. Re- ducing overconfidence in the interval judgments of experts. *Risk Anal.* **30**, 512–523.

Stott, I., Hodgson, D., Townley, S., 2014. *popdemo: Provides Tools for Demographic Modelling Using Projection Matrices, R package Version 0.1-4 ed.*

Walters, C.J., 1986. *Adaptive Management of Renewable Resources*. MacMillan Publishing Co., New York, NY.

Williams, B.K., 2011. *Adaptive management of natural resources—framework and issues*. *J. Environ. Manag.* **92**, 1346–1353.

Williams, B.K., Johnson, F.A., 2015a. *Value of information and natural resources decision- making*. *Wildl. Soc. Bull.* **39**, 488–496.

Williams, B.K., Johnson, F.A., 2015b. *Value of information in natural resource manage- ment: technical developments and application to pink-footed geese*. *Ecol. Evol.* **5**, 466–474.

Williams, B.K., Eaton, M.J., Breininger, D.R., 2011. *Adaptive resource management and the value of information*. *Ecol. Model.* **222**, 3429–3436.

Woudenberg, F., 1991. *An evaluation of Delphi. Technol. Forecast. Soc. Chang.* **40**, 131–150.

Yaniv, I., Foster, D.P., 1997. *Precision and accuracy of judgmental estimation*. *J. Behav. Decis.* **Mak.* **10**, 21–32.