Number range CHAPTERNovember 14, 2002

# MEMORANDUM

SUBJECT: Transmittal of Revised Meeting Minutes for the FIFRA Scientific
Advisory Panel Meeting Held October 1, 2002

TO: Marcia E. Mulkey, Director

> Office of Pesticide Programs

FROM: Steven M. Knott, Designated Federal Official

> FIFRA Scientific Advisory Panel
>
> Office of Science Coordination and Policy

Attached, please find revised meeting minutes for the FIFRA Scientific
Advisory Panel open meeting held in Arlington, Virginia on October 1,
2002. This report addresses a set of scientific issues being considered
by the Environmental Protection Agency regarding projecting domestic
percent-crop-treated with pesticides for dietary risk assessment.

Subsequent to issuance of the final minutes, Dr. Anne Koehler, one of
the October 1^st^ Panelists, noted an error in one of the equations on
page 18. Therefore, the minutes have been revised as follows:

on page 18 of the final minutes, the equation

$$\text{Cov}\left\{ (y_{n + 1,},y_{n + 2})|y_{1},y_{2},\ldots,y_{n},\ell_{0},\alpha,\sigma \right\}$$

$$\text{Cov}(y_{n + 1},y_{n + 2}) = E(\text{αε}_{n + 1}^{2} + \varepsilon_{n + 1}\varepsilon_{n + 2}) = \text{ασ}^{2}$$

has been changed to

$$\text{Cov}\left\{ (y_{n + 1,},y_{n + 2})|y_{1},y_{2},\ldots,y_{n},\ell_{0},\alpha,\sigma \right\}$$

=
$\text{Cov}(y_{n + 1} - \ell_{n},y_{n + 2} - \ell_{n}|y_{1},y_{2},\ldots,y_{n},\ell_{0},\alpha,\sigma) = E(\text{αε}_{n + 1}^{2} + \varepsilon_{n + 1}\varepsilon_{n + 2}) = \text{ασ}^{2}$

This revised expression of the equation is more precise and should avoid
potential misreading of the equation as the covariance for consecutive
values of the time series (as opposed to the correct interpretation of
covariance of forecast errors made from the same time period).

Attachment

cc:

Stephen Johnson

Susan Hazen

Adam Sharp

James Jones

Janet Andersen

Debbie Edwards

Anne Lindsay

Steve Bradbury

Denise Keehner

Linda Moos

Lois Rossi

Frank Sanders

Margaret Stasikowski

William Jordan

Antonio Bravo

Douglas Parsons

David Deegan

Vanessa Vu (SAB)

OPP Docket

FIFRA Scientific Advisory Panel Members

Dr. Ronald J. Kendall, FIFRA/SAP Chair

Dr. Geoffrey Allen

Dr. Bruce Babcock

Dr. Frederick Collopy

Dr. Dale Hattis

Dr. Barry Jacobsen

Dr. Anne Koehler

Dr. George Norton

Dr. Nu-May Ruby Reed

Dr. Mark Whalon

Dr. Larry Wilhoit

Number range CHAPTERNovember 7, 2002

# MEMORANDUM

SUBJECT: Transmittal of Meeting Minutes of the FIFRA Scientific Advisory
Panel Meeting Held October 1, 2002

TO: Marcia E. Mulkey, Director

> Office of Pesticide Programs

FROM: Steven M. Knott, Designated Federal Official

> FIFRA Scientific Advisory Panel
>
> Office of Science Coordination and Policy

THRU: Larry C. Dorsey, Executive Secretary

> FIFRA Scientific Advisory Panel
>
> Office of Science Coordination and Policy
>
> Joseph J. Merenda, Jr., Director
>
> Office of Science Coordination and Policy

Attached, please find the meeting minutes of the FIFRA Scientific
Advisory Panel open meeting held in Arlington, Virginia on October 1,
2002. This report addresses a set of scientific issues being considered
by the Environmental Protection Agency regarding projecting domestic
percent-crop-treated with pesticides for dietary risk assessment.

Attachment

cc:

Stephen Johnson

Susan Hazen

Adam Sharp

James Jones

Janet Andersen

Debbie Edwards

Anne Lindsay

Steve Bradbury

Denise Keehner

Linda Moos

Lois Rossi

Frank Sanders

Margaret Stasikowski

William Jordan

Antonio Bravo

Douglas Parsons

David Deegan

Vanessa Vu (SAB)

OPP Docket

FIFRA Scientific Advisory Panel Members

Dr. Ronald J. Kendall, FIFRA/SAP Chair

Dr. Geoffrey Allen

Dr. Bruce Babcock

Dr. Frederick Collopy

Dr. Dale Hattis

Dr. Barry Jacobsen

Dr. Anne Koehler

Dr. George Norton

Dr. Nu-May Ruby Reed

Dr. Mark Whalon

Dr. Larry Wilhoit

##### SAP Minutes No. 2002-07

**October 1, 2002**

**FIFRA Scientific Advisory Panel Meeting,**

**held at the Sheraton Crystal City Hotel,**

**Arlington, Virginia**

**A Set of Scientific Issues Being Considered by the**

**Environmental Protection Agency Regarding:**

Projecting Domestic Percent-Crop-Treated with

> Pesticides for Dietary Risk Assessment

####### NOTICE

These minutes have been written as part of the activities of the Federal
Insecticide, Fungicide, and Rodenticide Act (FIFRA), Scientific Advisory
Panel (SAP). These minutes have not been reviewed for approval by the
United States Environmental Protection Agency (Agency) and, hence, the
contents of these minutes do not necessarily represent the views and
policies of the Agency, nor of other agencies in the Executive Branch of
the Federal government, nor does mention of trade names or commercial
products constitute a recommendation for use.

The FIFRA SAP was established under the provisions of FIFRA, as amended
by the Food Quality Protection Act (FQPA) of 1996, to provide
independent advice, information, and recommendations to the Agency
Administrator on pesticides and pesticide-related issues regarding the
impact of regulatory actions on health and the environment. The Panel
serves as the primary scientific peer review mechanism of the EPA,
Office of Pesticide Programs (OPP) and is structured to provide balanced
expert assessment of pesticide and pesticide-related matters facing the
Agency. Food Quality Protection Act Science Review Board members serve
the FIFRA SAP on an ad-hoc basis to assist in reviews conducted by the
FIFRA SAP. Further information about FIFRA SAP minutes and activities
can be obtained from its website at
[http://www.epa.gov/scipoly/sap/]{.underline} or the OPP Docket at (703)
305-5805. Interested persons are invited to contact Larry Dorsey, SAP
Executive Secretary, via e-mail at [dorsey.larry@.epa.gov]{.underline}.

######## CONTENTS

[PARTICIPANTS 5](#__RefHeading___Toc24430690)

[INTRODUCTION 6](#__RefHeading___Toc24430691)

[PUBLIC COMMENTERS 6](#__RefHeading___Toc24430692)

[CHARGE 7](#__RefHeading___Toc24430693)

[SUMMARY OF PANEL DISCUSSION AND RECOMMENDATIONS
8](#__RefHeading___Toc24430694)

[PANEL DELIBERATIONS AND RESPONSE TO CHARGE
10](#__RefHeading___Toc24430695)

[REFERENCES 26](#__RefHeading___Toc24430696)

######### SAP Minutes No. 2002-07

**October 1, 2002**

**FIFRA Scientific Advisory Panel Meeting,**

**Held at the Sheraton Crystal City Hotel,**

**Arlington, Virginia**

A Set of Scientific Issues Being Considered by the Environmental
Protection Agency Regarding:

Projecting Domestic Percent-Crop-Treated with

Pesticides for Dietary Risk Assessment

**Steven M. Knott, M.S. Ronald J. Kendall, Ph.D.**

**Designated Federal Official FIFRA SAP, Chair**

**FIFRA Scientific Advisory Panel FIFRA Scientific Advisory Panel**

**Date:** November 6, 2002 **Date:** November 6, 2002

**Federal Insecticide, Fungicide, and Rodenticide Act**

**Scientific Advisory Panel Meeting**

**October 1, 2002**

**Projecting Domestic Percent-Crop-Treated with**

**Pesticides for Dietary Risk Assessment**

[]{#__RefHeading___Toc24430690 .anchor}**PARTICIPANTS**

**FIFRA SAP, Chair**

Ronald J. Kendall, Ph.D., Professor and Chairman, Department of
Environmental

Toxicology, Director, The Institute of Environmental and Human Health,
Texas Tech

University, Texas Tech University Health Sciences Center, Lubbock, TX

**Designated Federal Official**

Mr. Steven M. Knott, FIFRA Scientific Advisory Panel Staff, Office of
Science Coordination and Policy

**FQPA Science Review Board Members**

Geoffrey Allen, Ph.D., Professor and Chairman, Department of Resource
Economics, University of Massachusetts, Amherst, MA

Bruce Babcock, Ph.D., Director, Center for Agricultural and Rural
Development, Professor, Department of Economics, Iowa State University,
Ames, IA

Frederick Collopy, Ph.D., Professor and Department Chair, The
Weatherhead School, Case Western Reserve University, Cleveland, OH

Dale Hattis, Ph.D., Research Professor, George Perkins Marsh Institute,
Clark University, Worcester, MA

Barry Jacobsen, Ph.D., Professor, Department of Plant Sciences and Plant
Pathology, Montana State University, Bozeman, MT

Anne Koehler, Ph.D., George and Mildred Panuska Professor of Business
Administration, Miami University, Oxford, OH

George Norton, Ph.D., Professor, Department of Agricultural and Applied
Economics, Virginia Tech University, Blacksburg, VA

Nu-May Ruby Reed, Ph.D., Staff Toxicologist, California Environmental
Protection

Agency, Department of Pesticide Regulation, Sacramento, CA

Mark Whalon, Ph.D., Professor, Center for Integrated Plant Systems,
Michigan State University, East Lansing, MI

Larry Wilhoit, Ph.D., Senior Environmental Research Scientist,
California Environmental Protection Agency, Department of Pesticide
Regulation, Sacramento, CA

[]{#__RefHeading___Toc24430691 .anchor}**INTRODUCTION**

The Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA),
Scientific Advisory Panel (SAP) has completed its review of the set of
scientific issues being considered by the Agency pertaining projecting
domestic percent-crop-treated with pesticides for dietary risk
assessment. Advance notice of the meeting was published in the *Federal
Register* on August 23, 2002. The review was conducted in an open Panel
meeting held in Arlington, Virginia, on October 1, 2002. Dr. Ronald
Kendall chaired the meeting. Mr. Steven Knott served as the Designated
Federal Official.

David Widawsky, Ph.D. (Office of Pesticide Programs, EPA) began the
Agency presentations by providing an introduction and background on the
role of percent crop treated information in dietary risk assessments.
This was followed by a presentation from Arthur Grube, Ph.D. (Office of
Pesticide Programs, EPA) describing the current methods for estimating
percent crop treated. Mr. Philip Villanueva (Office of Pesticide
Programs, EPA) ended the Agency presentations with a description of the
proposed methodology for projecting percent crop treated. Mr. Villanueva
also presented on an evaluation of the proposed methodology. Ms. Denise
Keehner (Director, Biological and Economic Analysis Division, Office of
Pesticide Programs, EPA) provided an introduction to the session and
also participated in the meeting.

In preparing these meeting minutes, the Panel carefully considered all
information provided and presented by the Agency presenters, as well as
information presented by public commenters. These meeting minutes
address the information provided and presented at the meeting,
especially the response to the charge by the Agency.

[]{#__RefHeading___Toc24430692 .anchor}**PUBLIC COMMENTERS**

**Oral statements were made by:**

Mr. Leonard Gianessi on behalf of the National Center for Food and
Agricultural Policy

Jennifer Phillips, Ph.D., Bayer CropScience, on behalf of CropLife
America

**Written statements were received as follows:**

Written copies of the oral statements made by Mr. Gianessi on behalf of
the National Center for Food and Agricultural Policy and Dr. Phillips on
behalf of CropLife America were received. No other written comments were
received.

[]{#__RefHeading___Toc24430693 .anchor}**CHARGE**

**Issue 1**

The Office of Pesticide Program's (OPP's) focus in developing a
forecasting methodology for percent crop treated (PCT) has been
univariate methods, which seems most appropriate given the available
data. In the best cases, historical information on PCT consists of no
more than 15 observations. Such a limited number of data points excludes
the use of complex forecasting models, such as Box-Jenkins.

> **Question 1.a**. Keeping this in mind, please comment on the
> strengths and weaknesses of exponential smoothing as a tool for
> forecasting PCT.
>
> **Question 1.b**. Please provide any comments or suggestions for
> enhancing the proposed forecasting methodology that is based on
> exponential smoothing.
>
> **Question 1.c**. Please comment on OPP's assessment of other
> forecasting methods, including whether there are other methods that
> would be suitable for producing three to five year forecasts of PCT
> with the available number of data observations.

# Issue 2

Generally OPP will not perform a risk assessment for a pesticide every
year. Therefore estimates of percent crop treated (PCT) may need to
account for pesticide use for the next three to five years. For
assessing risk due to acute exposure to a pesticide, OPP relies on an
estimate of the maximum PCT. For a given forecasting model, a prediction
interval (i.e. interval forecast) could serve as an upper bound for an
*individual* point forecast. However, to estimate an upper bound for
*multiple* point forecasts, OPP proposes the calculation of
*simultaneous* prediction intervals. The probability that all of the
future observations being forecasted will be less than that bound is
assumed to be the product of the individual probabilities that each of
the future observations will be less than that bound (i.e. the
individual probabilities are assumed to be independent).

> **Question 2.a**. Please comment on the reasonableness of the
> assumption of independence between the observations.
>
> **Question 2.b**. What consequences would a violation of this
> assumption have on the simultaneous prediction interval being
> calculated?
>
> **Question 2.c**. Please provide comments and recommendations on
> refinements for calculating simultaneous prediction intervals.

[]{#__RefHeading___Toc24430694 .anchor}**SUMMARY OF PANEL DISCUSSION AND
RECOMMENDATIONS**

The following is a brief summary of the FIFRA SAP discussion and
recommendations concerning the proposed method for projecting percent
crop treated (PCT) with pesticides for dietary risk assessment.

#### Strengths and Weaknesses of Exponential Smoothing

The Panel members observed that the exponential smoothing methods
proposed by OPP are more flexible than the methods currently used. In
addition, Panel members felt that exponential smoothing methods are
reasonably parsimonious, appropriate for short series, expressible as
special cases of autoregressive integrated moving average models and
state-space formulations, and thereby have well-understood properties
with internally consistent formulations for prediction intervals. On the
other hand, like any statistical model, exponential smoothing methods
assume the future will be a continuation of the past, and selecting the
most accurate model from the family of exponential smoothing models is
difficult with a short data series.

# Suggested Improvements for the Proposed Methodology

**Transformation of the Data**

Since the values of a proportion are bounded by zero and one (or 0 and
100% when expressed as percentages as for PCT), two questions must be
answered. Will the forecasts remain within the possible values of zero
and one? Is it reasonable to assume that the one-period-ahead forecast
errors are normally distributed? If the answer is "no" to either of
these questions, transformations that are designed to deal with these
problems should be considered.

**Explore the Choice of Algorithm**

The Panel members recommended that OPP select a random sample of PCT
series and compare results using a standard algorithm such as is found
in a commercial software package and the one they propose, to determine
how much the choice of algorithm matters.

# Assessment of Other Forecasting Methods

The Panel members recommended that the evaluation OPP has conducted on
17 selected PCT series be extended to examine a reasonably large number
of representative series. Panel members also suggested using the
naïve-no-change method (which is optimal for a series generated by a
random walk) as the baseline method.

# Independence of Observations

The Panel members agreed with the OPP that the historical values of PCT
are not independent and they suggested several biological and behavioral
arguments as to why PCT might be positively or negatively correlated.
However, the question that was intended with respect to constructing
valid simultaneous prediction intervals is whether or not the forecast
errors for different horizons are independent. It can be readily
demonstrated that forecast errors at different horizons are correlated
when an exponential smoothing model is the correct model for the data.
In addition, errors caused by using a misspecified model and by having
to estimate unknown parameters have impacts on forecast error
distributions beyond those accounted for in the prediction interval
formulas.

# Consequences of Lack of Independence of Forecast Errors

The Panel members concluded that the presence of the problems referred
to in the previous paragraph would result in prediction intervals that
are too narrow. Evidence from the published literature supports this
conclusion (see for example Chatfield, 1993). The prediction interval
horizon by horizon will be too narrow and so will a simultaneous (single
value) prediction interval.

# Simultaneous Prediction Interval Refinements and Recommendations

To provide valid upper bounds, the Panel members recommended that OPP
must either adopt a calibration formula or some form of Monte Carlo
simulation. Both of these approaches should be developed and tested on a
sufficiently large, randomly drawn set of PCT series so that there is
sound justification for the final procedure that OPP adopts.

# Other Issues

**Data Quality**

Panel members expressed concern about the overall quality of PCT data,
though this was not part of the charge to the SAP. The Panel members
recommended that questions about the origin, the quality, and the
weighting of the values combined in computing the historical annual PCT
should be the subject of another FIFRA Scientific Advisory Panel
meeting. The Panel members also recommended that construction of the
data series be as transparent as the forecasting procedures used on
them.

**Judgmental Forecasts or Adjustments to Forecasts**

Unaided judgmental forecasts done by an expert in an area are useful in
their own right. Where OPP wishes to take information from an expert and
use that to judgmentally adjust statistical forecasts produced by the
proposed method, such adjustments should (1) be made in advance of the
statistical forecast, not as a reaction to it, and (2) be transparent;
that is, OPP should state explicitly how each piece of information
determines the amount and direction of adjustment to the forecast PCT.

[]{#__RefHeading___Toc24430695 .anchor}**PANEL DELIBERATIONS AND
RESPONSE TO CHARGE**

The specific issues addressed by the Panel are keyed to the Agency\'s
background documents, dated August 2002 and the Agency's charge
questions.

**Issue 1**

The Office of Pesticide Program's (OPP's) focus in developing a
forecasting methodology for percent crop treated (PCT) has been
univariate methods, which seems most appropriate given the available
data. In the best cases, historical information on PCT consists of no
more than 15 observations. Such a limited number of data points excludes
the use of complex forecasting models, such as Box-Jenkins.

> **Question 1.a**. Keeping this in mind, please comment on the
> strengths and weaknesses of exponential smoothing as a tool for
> forecasting PCT.

# Strengths of Exponential Smoothing

The reports from the OPP are very thorough and explain well the current
and proposed methods for forecasting the domestic percent crop treated
(PCT). The OPP is proposing a method that would choose among three
different models for exponential smoothing. The current methodology of
finding a point prediction for future values of the PCT by computing a
weighted average of the historical values of the PCT is already similar
to using one of the models, namely simple exponential smoothing. The
current weights, which decrease as the time period is further away from
the present, are close to being the weights for a simple exponential
smoothing parameter $\alpha$ of .224. The proposed method will allow for
much greater flexibility in that the forecaster may choose both the
model and the values of the smoothing parameters for the models that fit
the historical data well. It is not always true that tighter fits of the
historical data produce better forecasts. However, the OPP has provided
a very limited study on 17 time series for PCT that gives some evidence
that more accurate forecasts are produced by the exponential smoothing
models than by the current method.

The OPP has observed that there is an upward or downward trend to the
values of the PCT for some of the crops and pesticides. While the point
forecasts from the current method do not incorporate this trend, the OPP
has recognized that the trend should not be ignored. Hence, the current
methodology includes accounting for the trend in the approximate
prediction intervals. The current methodology fits a deterministic trend
line in order to get a value to be used as a bound on the forecast
error. The bound on the forecast error from the regression for the last
time period is simply added to the weighted average (i.e. the point
forecast) to obtain an upper limit for a prediction interval. In the
proposed methodology, the OPP will get both the point forecasts and the
upper limits for the prediction intervals from the same model. The
choice of models includes models that allow for trend. Specifically, the
models include simple exponential smoothing (SES), which allows for a
changing mean but no trend; linear exponential smoothing (LES), which
allows for a trend that can change; and damped trend exponential
smoothing (DES), which allows for a trend that can change but dampens
the trend as the length of the forecasting horizon increases. Choosing
among these three models has several important advantages over the
current methodology. If a trend exists an appropriate procedure can be
chosen, better choices for the weighting of the historical values can be
selected, and prediction intervals can be based on the selected model.

Exponential smoothing is transparent, accessible, and reproducible. It
allows for a variety of trends, for dampening of trends, and should
provide more reliable forecasts than previous weighted moving average
procedures used to predict PCT. It passes the parsimony test by no being
more complicated than is necessary to obtain a credible forecast.
Exponential smoothing requires fewer observations than other forecasting
methods that allow for changing levels and trends.

# Weaknesses of Exponential Smoothing

Choosing the right exponential smoothing model and estimating the
parameters is difficult even in exponential smoothing with so few data
points (10 to 15). Since the main reason for using the Bayesian
Information Criterion (BIC) is to decide if a trend should be included
in the forecasts, one could limit the number of models to two simple
models in place of the proposed three choices. The two models would be
simple exponential smoothing and linear exponential smoothing with a
fixed growth rate. The second model is

> $y_{t} = \ell_{t - 1} + b + \varepsilon_{t}$
>
> $$\ell_{t} = \ell_{t - 1} + b + \text{αε}_{t}$$

This model allows for what appears to be a changing trend in a plot of
the data even though the growth rate *b* is fixed. It is a special case
of linear exponential smoothing where the trend smoothing parameter is
assumed to be zero. Alternatively, it adds a constant drift to simple
exponential smoothing. The difference from linear exponential smoothing
is that the trend cannot change directions. There is only one smoothing
parameter to estimate in the linear exponential smoothing with a fixed
growth rate. To use the BIC, one should count the initial level and
$\alpha$ as two free parameters in the BIC for simple smoothing and the
initial level, the initial growth rate, and $\alpha$as three free
parameters in the BIC for linear exponential smoothing with a fixed
growth rate. These two models might be sufficient for forecasting PCT.

As in any statistical forecasting method, exponential smoothing assumes
that the historical pattern continues into the future. Exponential
smoothing excludes the possibility of including other factors that may
influence pesticide use. The assumption that the future is like the past
may not hold when broad-spectrum pesticides are cancelled or
significantly mitigated under FQPA. With the demise of a whole class of
chemicals, as with organophosphates under FQPA, pests may rebound very
quickly, as may have happened with the glassy-winged sharpshooter. Both
these problems would have to be addressed by allowing for judgmental
forecasting as part of the process. The issue of using professional
judgment will be addressed later under the heading, Other Issues.

> **Question 1.b**. Please provide any comments or suggestions for
> enhancing the proposed forecasting methodology that is based on
> exponential smoothing.

**Transformation of the Data**

There is a limit to the range of possible values for the annual PCT. As
percentages, the values are restricted to being from 0 to 100, including
0 and 100. One possible solution is to transform the data. For the rest
of the discussion about transformations, the Panel members assumed that
PCT is expressed as a decimal rather than a percentage so that the
values of PCT would be restricted to being from 0 to 1. The purpose of
the transformation would be two-fold. The first goal is to prevent
forecasts from exceeding 1 and falling below 0. The second goal is for
the one-period-ahead forecast errors to have a normal distribution. The
second goal is important in justifying, for example, the use of 1.64 for
computing one-sided upper 95% prediction intervals or in justifying the
use of normally distributed error terms in a simulation of prediction
intervals. One would need to examine one-period-ahead-forecast errors
(i.e. the residuals) from the selected model to see if the normality
assumption is reasonable. This may be very difficult with only 10 to 15
values in the time series. Perhaps the one-period-ahead forecast errors
from similar time series could be examined together.

One suggestion from a Panel member for a transformation is the logistic
transformation

> $$\ell n\left\lbrack \frac{\text{PCT}}{1 - \text{PCT}} \right\rbrack$$

This transformation would be appropriate when the PCT plotted over time
approximates a logistic curve

> $$\text{PCT} = \frac{e^{(a + \text{bt})}}{1 + e^{a + \text{bt}}}$$

It could not be used for data that has values of 0 or 1. Otherwise it
could meet both goals of transformation. This logistic curve has 0 and 1
as limiting values. A second suggestion is

> $$- \ell n(1 - \text{PCT})$$

This transformation would be appropriate when the PCT plotted over time
approximates the function

> $$\text{PCT} = 1 - e^{- (a + \text{bt})}$$

This curve has as a limiting value of 1. This transformation will not
help with the first goal if the PCT can have values trending down toward
0. A third suggestion from another Panel member is the arcsine function

> $$\text{arcsin}(\text{PCT})$$

This transformation would be appropriate if the PCT plotted over time
approximates the sine curve

> $$\text{PCT} = \text{sin}(a + \text{bt})$$

This transformation would not help with the first goal because it is
still bounded, but it might help with the second goal. A log transform
would indicate exponential growth of the PCT which is not likely with
the limited range of values. It may be that the one-period-ahead
forecast errors are normally distributed without any transformation of
the PCT values. It also may be that for most of the thousands of time
series, the PCT values are sufficiently far away from 0 that no
transformation is necessary to avoid forecasts outside of the range of 0
to 1. However, the need for a transformation should be considered.

**Model selection**

Selecting the best-fitting forecasting model sounds like an appealing
strategy; however, the evidence for its success is decidedly limited.
There is always the danger that when the forecaster chooses the model
that best fits the available sample, there is a tendency to over fit.
For example, a particular time series appears to display a downward
trend so that a model such as linear exponential smoothing is favored
over simple exponential smoothing. The truth may be that the population
from which the sample is drawn does not have a trend. Selecting the best
model for forecasting a series out of all the models tested does not
lead to the most accurate forecasts often enough for it to be
recommended as a sound strategy. One Panel member suggested that some
strategies worth investigating are: (1) group similar series according
to some criterion and use the model that works best for the group on
each series or (2) use two or three methods, preferably of different
classes of model, and average the forecasts from them to get a composite
or combined forecast.

**Relating the PCT Forecasts to Monitoring Data (e.g. Data from the
Pesticide Data Program)**

While the actual application of forecasted PCT in the context of dietary
exposure assessment is not a part of the current discussion, an analysis
of the relationship between the historical PCT and the pesticide residue
monitoring data could provide an additional tool for evaluating the
forecasting methods. Specifically, how closely are the historical PCT
data reflected in the residue data? Some of the questions could be:

Is the percentage of non-detect samples consistently lower than the
estimated PCT at both the national and state levels, and especially for
those commodities with a higher percentage of residue detection?

In cases when the historical PCTs show significant isolated peaks of
increase, do the monitoring data also show corresponding decrease in the
percentage of non-detect samples?

Does the peak in PCT correlate better with the peak or average residue
levels rather than with the percentage of non-detect samples?

With respect to peak PCT in the historical data, one concern often
raised in dietary exposure assessment is whether the estimated range of
PCT generally encompasses any sudden increase of pesticide use, e.g., in
pest infestation episodes. It appears that whether the proposed
forecasting method can sufficiently address this concern would somewhat
depend on whether the historical PCT data include such episodes and
whether they are generally sufficient for anticipating possible rises in
pesticide use. Thus, one Panel member suggested that a greater
confidence on the estimated upper PCT intervals could be assured by
systematically identifying the key contributing factors to fluctuating
pesticide uses, and the magnitude of their effects on the PCT levels.
When historical PCT data that form the basis of forecasting are judged
insufficient for anticipating these rises in PCT, it may be prudent to
include an additional factor in the forecasting procedure.

# Choice of Algorithm

The algorithm OPP proposes to use to estimate the exponential smoothing
models is a fairly primitive one. The initial level and trend values are
estimated from regression against time and the smoothing parameters
estimated by a brute-force search over all possible values. With such
short series, choice of initial level and trend are likely to be
important and these parameters are not optimized during estimation. With
only one or two initial values to estimate, one can include these values
in the optimization process along with the smoothing parameters.
However, as stated earlier, a better historical fit does not guarantee
better forecasts. Standard computer software packages employ other
methods. For example, the time-series module of SAS® uses backcasting,
first estimating the trend by linear regression for the initial value of
the trend, second taking that value and the final observation as the
initial level, then working with the series in reverse order to find the
smoothing parameter values with minimum sums of squares, and finally
forecasting level and trend for one-step ahead. These forecasts become
the initial level and trend at time zero, the period before the first
observation. Estimation of optimal smoothing parameter values then
proceeds in the forward direction. The steps are sufficiently well
described in the manuals accompanying the software that OPP's concern
for transparency can be met. One Panel member recommended that the OPP
select a random sample of series and compare the results of using an
algorithm such as described above with results from the one they propose
in order to determine how much the choice of algorithm matters. The OPP
might also find it helpful to examine other optimization routines that
have already been developed.

> **Question 1.c**. Please comment on OPP's assessment of other
> forecasting methods, including whether there are other methods that
> would be suitable for producing three to five year forecasts of PCT
> with the available number of data observations.

The OPP is correct that the exponential smoothing method is a very good
choice among the possible choices for forecasting methods. Because of
the changing nature of the level and trend for the time series of
interest to the OPP, one would not want to use a deterministic method,
such as fitting a mean value or a linear trend line. The OPP is correct
that the Box-Jenkins methodology requires more observations to estimate
autocorrelations and partial autocorrelations than they have available.
In the Box-Jenkins methodology, before estimating an autoregressive
integrated moving average (ARIMA) model, one must first answer the
following questions: is the mean changing, does a trend exist, and if
the trend exists, is it changing? These are the same questions that are
addressed with exponential smoothing. If a trend exists, it tends to
dominate the other patterns that the Box-Jenkins method would pick up
even if more data were available. In the past the exponential smoothing
methods have not been based on statistical models in an obvious way. It
has long been known that each additive exponential smoothing method
(e.g. level and trend components are added to get forecasts) has a
corresponding ARIMA model. The OPP has based its proposed procedures on
recent work in exponential smoothing that provides an integrated
framework for the exponential smoothing methods. This framework combines
the original exponential smoothing methods with single source of error
state space models.

Several Panel members indicated that the evaluation study should be
extended to examine a reasonably large number of representative series.
The series should be randomly selected from the thousands of time series
that need to be forecasted. The random walk (or naïve no-change method)
should be added to the methods already included in the evaluation study.
If a method does not perform better than a random walk then either there
is no information in the data to support better forecasts or the
information in the data is not being used effectively.

Combining forecasts from several methods was suggested as another
possible procedure for forecasting PCT. Another potential method is
unaided expert judgment.

# Issue 2

Generally OPP will not perform a risk assessment for a pesticide every
year. Therefore estimates of percent crop treated (PCT) may need to
account for pesticide use for the next three to five years. For
assessing risk due to acute exposure to a pesticide, OPP relies on an
estimate of the maximum PCT. For a given forecasting model, a prediction
interval (i.e. interval forecast) could serve as an upper bound for an
*individual* point forecast. However, to estimate an upper bound for
*multiple* point forecasts, OPP proposes the calculation of
*simultaneous* prediction intervals. The probability that all of the
future observations being forecasted will be less than that bound is
assumed to be the product of the individual probabilities that each of
the future observations will be less than that bound (i.e. the
individual probabilities are assumed to be independent).

> **Question 2.a**. Please comment on the reasonableness of the
> assumption of independence between the observations.

The representatives from the OPP acknowledged that asking about
independence of the observations was the wrong question with respect to
Question 2. However, since the exponential smoothing methods are
appropriate when the observations are correlated, the comments on
independence of the observations support the use of these methods. The
Panel members agreed that the data are not independent. Examination of
the within-sample observations of the first nine of the 17 series
analyzed by OPP (those with a starting year of 1987) shows that only two
of them have sample correlation coefficients not different from zero, at
the 5% level of significance. Independent random variables are
uncorrelated, and although we cannot say that uncorrelated variables are
necessarily independent, we can assert that correlated variables are not
independent.

Several Panel members advanced reasons why lack of independence would be
expected. One expressed the opinion that correlations among yearly PCT
values would be sometimes negative and sometimes positive depending on
the crop, pesticide, geographic area, and environmental conditions (the
seven significant correlations referred to above were both positive and
negative). For example, for some crops and pests, PCT is related to
regional pest population size. It is probably true in some cases that
pest population is negatively correlated from year to year; a higher
than average population size one year results in higher natural enemy
populations or overexploitation of resources that then leads to lower
populations the next year. If current pest levels are the dominant
factor in determining current PCT, we would then expect PCT to be
negatively correlated from year to year. In addition, a delayed response
from growers (i.e., using pest incidence in the previous year as a guide
to pesticide use in the current year) could again lead to negative
correlations between yearly PCT.

On the other hand, other processes might lead to positive correlations.
For example, for some pests, higher populations one year might result in
greater overwintering populations, which then lead to higher populations
next year, as long as population size is not near its environmental
carrying capacity; similarly lower populations one year might usually be
followed by lower populations next year. Another situation where
positive year-to-year correlation of PCT could occur is when pesticide
use is more dependent on past practice than on pest populations. For
example, if perceived effectiveness of a pesticide is determined
primarily by word of mouth from grower to grower, then with few users,
PCT stays low, whereas many users reinforce the message of effectiveness
and keep PCT high.

There was some discussion among Panel members about the forces that
might cause PCT to shift markedly from year to year. For example, for
pesticides such as fungicides, insecticides, and miticides that are used
within Integrated Pest Management (IPM) systems, variance in pest
pressure, pest resistance and weather will make PCT highly variable.
Conversely, preplant herbicides and nematicides used on row crops tend
to be applied consistently, regardless of pest levels. Having a high
variance is different from being correlated over time, although variance
does appear in the denominator of the formula for the correlation
coefficient.

**Are Forecast Errors Independent?**

Panel members agreed that the answer to this question is no. This
question is also subject to reinterpretation. If exponential smoothing
is the true model (that which generated the observations) then the
within-sample one-step-ahead prediction errors will have a mean of zero
and be normally distributed, homoscedastic and uncorrelated; that is the
model will pass misspecification tests on the residuals. Further, as
long as the data series continues into the future without breaks, the
forecast error *added* at each step ahead is independent.

In the OPP's procedure to find simultaneous prediction intervals, it is
assumed that forecast errors for different future time periods are
independent. If an exponential model is the correct model, the forecast
errors for different future time periods will be correlated. This
correlation can be illustrated for the first two forecast errors made
from the same time period using the simple exponential smoothing method.
Assume that simple exponential smoothing is the correct model for the
time series and that we have observed the first *n* values of the time
series, $y_{1},y_{2},\ldots,y_{n}$. Also assume that we have no error in
estimating the initial level $\ell_{0}$ and the smoothing parameter
$\alpha$.

Given the simple exponential smoothing model

> $$y_{t} = \ell_{t - 1} + \varepsilon_{t}$$
>
> $$\ell_{t} = \ell_{t - 1} + \text{αε}_{t}$$
>
> then
>
> $$y_{n + 1} = \ell_{n} + \varepsilon_{n + 1}$$

and
$y_{n + 2} = \ell_{n + 1} + \varepsilon_{n + 2} = \ell_{n} + \text{αε}_{n + 1} + \varepsilon_{n + 2}$

Hence,

$$\text{Cov}\left\{ (y_{n + 1,},y_{n + 2})|y_{1},y_{2},\ldots,y_{n},\ell_{0},\alpha,\sigma \right\}$$

> =$\text{Cov}(y_{n + 1} - \ell_{n},y_{n + 2} - \ell_{n}|y_{1},y_{2},\ldots,y_{n},\ell_{0},\alpha,\sigma) = E(\text{αε}_{n + 1}^{2} + \varepsilon_{n + 1}\varepsilon_{n + 2}) = \text{ασ}^{2}$

Thus, the larger the smoothing parameter $\alpha$ becomes, the larger is
the correlation between the first two forecast errors made from the same
time period. And since both the smoothing parameter and the variance are
non-negative, positive correlation will exist. However, the impact on
the upper prediction interval (UPI) is not clear. At least one panelist
believes that the assumption will make the UPI too small.

In practice, even if the correct model is applied to the data, estimates
of the initial values and smoothing parameters are calculated. These are
random variables and represent an additional source of error. Second,
and more importantly, the process that generated the data is unknown and
must be approximated by a relatively simple mathematical model, creating
another source of error. Several Panel members argued that since the
model fitted is not actually the data generating process (DGP) then the
errors *added* into the prediction interval formula are not independent.
Positive correlation should be expected in the errors for pesticides
that are used preventatively before a pest outbreak, rather than in
response to a pest outbreak. Preventative use rises during a forecast
period in response to marketing efforts, pricing decisions, and crop
planting decisions. Positive correlation may not be as big a problem for
pesticides that are used in response to pest outbreaks, if those pest
outbreaks themselves are independent over time. In addition, errors from
a misspecified model are not likely to be normally distributed.

> **Question 2.b**. What consequences would a violation of this
> assumption have on the simultaneous prediction interval being
> calculated?

Panel members proceed from the argument made above that successive
forecast errors are not independent to conclude that prediction
intervals (pi's) will be too narrow. Evidence from the published
literature supports this conclusion (see for example Chatfield, 1993).
Since the pi horizon by horizon will be too narrow, so will a
simultaneous (single value) pi.

Whether the conclusion follows from the premise is an interesting
question. The random walk model can be viewed as a special case of
simple exponential smoothing with the smoothing parameter, α = 1. The
variance of an h-steps-ahead forecast for simple exponential smoothing
is given by the formula

![](media/image1.wmf)

Since the prediction interval for any horizon depends directly on v~h~,
the random walk has a wider prediction interval than any simple
exponential smoothing model with α\<1. In a different context, Hahn
provided tables for the simultaneous confidence interval for k student-t
variates with correlations between zero and 0.5. The critical values for
a 1 -- α prediction interval increase as the correlation falls from 0.5
to zero, so that the bounds specified for the interval increase
similarly (Hahn and Hendrickson, 1971). \[Note that the α in the 1 -- α
prediction interval refers to the probability of a Type I error and has
no connection whatsoever with the smoothing parameter. Unfortunately,
the same symbol is conventionally used in the respective literatures for
both these purposes.\]

Hahn also notes that Bonferroni approximations will be as satisfactory
as the tabulated values except for small n (sample size), large m
(future values) and large level of significance (Hahn, 1969).

One Panel member observed that departure from one or more of the
assumptions of normally and independently distributed (NID) can affect
both the level of significance and the sensitivity of whatever test one
is using to determine significance, or in this case, the probability
that an individual PCT observation in the future will exceed the
simultaneous prediction interval. Experience indicates that the biases
resulting from failure of the data to fulfill the NID assumptions are
unimportant except:

where n is small

a\. How one defines "small" is not clear. Certainly when the PCT series
for a new use of a pesticide on a crop covers only a very few years, as
when FQPA alternatives are introduced, the inferential consequences need
to be carefully considered. Dealing with potential biases in UPI in such
a situation is crucial to national crop production security for the
future and should in due course be addressed more completely by the EPA.

b\. A second situation where use data may be limited in both quantity
and quality is for minor crops, which do not receive grower or
manufacturer encouragement for data reporting compared with major crops.
Some members of the panel are concerned about the adequacy of federal
funding to enable USDA-NASS to sample sufficient FQPA-targeted fruits
and vegetables.

where m is large:

a.  This condition is not likely since m will not exceed 3-5 years in
    OPP's analysis. Although as NASS Minor Crop data deteriorates, even
    this short time horizon in the forecast may become problematic, and
    again, the Agency should estimate the consequences of these data
    reductions.

where α is large or correspondingly the (1 -- α) probability is small:

a.  As long as the OPP uses an interval of 95% or larger, this should
    not be a problem.

In summary, if the assumptions are violated, the regulators essentially
believe that they are using a putative 5% level when in fact the level
has eroded to 7% or 8%. Loss of sensitivity for tests of significance
and estimation occur since a more powerful test could be constructed if
the exact mathematical model were known. However, for most types of
pesticide data, experience indicates that the usual disturbances
resulting from failure of the data to fulfill the NID assumptions are
unimportant and contribute less than a 1% error. Yet, the Agency should
develop a system of checking a series of data to determine if this
experiential observation is true.

> **Question 2.c**. Please provide comments and recommendations on
> refinements for calculating simultaneous prediction intervals.

Evidence from the selected small number of PCT series analyzed by OPP
showed that the proposed simultaneous prediction interval formula
results in a "likely upper bound" that is too low. This outcome is not
unexpected, and will almost certainly be repeated if a larger randomly
selected sample of series is tested. Panel members made a number of
suggestions for improving the calculation procedure.

OPP could check for model misspecification based on within-sample
one-step-ahead residuals from the fitted model. Only for series where a
given model passes these tests can it be regarded as the true model and
the pi as reasonable. Since a forecast must be made for each series
using one of the proposed Exponential Smoothing methods, even if none of
them is well specified, this process provides a warning of a problem,
but it does not provide a solution.

OPP could use a Bonferroni inequality for the simultaneous prediction
interval. Its big advantage is that it is valid even if the observations
are not independent, so it is more appropriate in the PCT situation.
Applying Bonferroni's inequality to this situation gives:

(Probability that PCT is less than some bound for all 5 years) ≥

1 -- \[(1 - p~1~) + (1 - p~2~) + (1 - p~3~) + (1 - p~4~) + (1 - p~5~)\],

where p~i~ is the probability that PCT is less than the bound during
year i. If all p~i~ values are high, then this gives a joint probability
close to the value from multiplying all p~i~ values. For example, if
each p~i~ = 0.99, then multiplying itself 5 times gives 0.95099, but
Bonferroni's inequality gives 0.95. The formula used in the OPP proposal
for the simultaneous pi is a Sidak inequality and the Bonferroni
inequality is more conservative.

Since a forecast must be made, even if the model used is misspecified, a
better solution would be an empirically based calibration. Panel members
suggested the following:

a.  Use a compound distribution function and corresponding critical
    value based on work pioneered by Shlyakhter in the analysis of the
    distribution of actual uncertainties in the physicists' measurements
    of physical constants for elementary particles, relative to
    uncertainties previously estimated on the basis of standard
    statistical procedures that capture only random error (Shlyakhter
    and Kammen, 1992). Essentially, the thrust of Shlyakhter's
    observations are that (1) real uncertainties in measurements have a
    distribution that is closer to exponential than Gaussian in
    shape---producing a relatively larger amount of underestimation of
    uncertainty as the number of standard errors from the central
    estimate increases, and (2) a reason for this is likely to be that
    in addition to random error, real measurements incorporate
    unsuspected systematic error. Shlyakther defined a constant, u, to
    be the ratio of unsuspected systematic error (measured as a standard
    deviation) to random error. The empirical finding was that for
    physical constants measured by physicists, u is about 1; for the
    errors of forecasters---such as demographers placing uncertainty
    bounds on future population sizes, u appeared to be about 3.
    Empirical testing is needed to determine the degree of systematic
    underestimation of forecasting errors for the PCT exponential
    smoothing methods. An illustration of the application of this type
    of correction is contained in Hattis and Burmaster (Hattis and
    Burmaster, 1994.). From that paper, the following table gives
    expectations for needs to expand confidence intervals to achieve
    better calibration to real uncertainty for u = 1 and u = 2:

Table 1. Contrast Between Gaussian Confidence Limits and Confidence
Limits Calculated According to the Revised Shlyakhter Compound
Distribution Function

  ------------ -- -------------- ----------------- ----------------- -----------------
                                 **Number of                         
                                 Stated Standard                     
                                 Errors from the                     
                                 Mean Needed to                      
                                 Achieve the                         
                                 Confidence                          
                                 Level**                             

  2-tailed                       standard gaussian compound function compound function
  confidence                     distribution      with u=1a         with u=2a
  Level                                                              

  50%                            0.674             1.12              1.51

  80%                            1.282             2.27              3.26

  90%                            1.644             3.07              4.61

  95%                            1.96              3.87              5.99

  98%                            2.326             4.91              7.84

  99%                            2.576             5.70              9.25

  99.9%                          3.29              8.3               \>\>10
  ------------ -- -------------- ----------------- ----------------- -----------------

aSource: A. Shlyakhter, personal communication.

In other words, to achieve a real one-tailed 95% upper confidence limit,
as desired for the OPP's Upper Prediction Intervals for single years,
one would need to add not 1.644 standard deviations, but 3.07 standard
deviations to the mean for u =1, and about 4.6 standard deviations if an
empirical comparison of actual versus expected forecast errors indicates
that u is close to 2.

A problem with implementing this method for a simultaneous prediction
interval is that the probability values required will need to be
interpolated from the table. Even so, we recommend that OPP investigate
this method on a sample of PCT series to test the adequacy of the
published critical values.

b.  Use a Monte Carlo approach. Monte Carlo simulation covers a wide
    variety of generating techniques and Panel members did not provide
    detailed suggestions for how MC simulation would arrive at the
    essential distribution.

The fundamental choices are

i.  Draw synthetic samples based only on the information contained in
    one series and perform repeated estimations of point and interval
    forecasts so that parameter uncertainty, misspecification error and
    random error are all incorporated.

ii. Use information from all series (or from a group of series judged to
    have similar properties)

E.g., Calculate the z score for each forecast error

> ![](media/image2.wmf) h = 1, . . .,5
>
> where Y~t+h~ is the actual h-step-ahead value, Ŷ~t+h~ is the value
> forecasted at time t for h steps ahead and v~h~ is the h-steps-ahead
> variance calculated using the true model formula.
>
> Do this for every series, e.g., suppose you have 900 series. Now you
> have 900 z~1~, 900 z~2~, that can be compared.
>
> For a given series, create a distribution of synthetic values by
> adding Ŷ~t+1~ to each z~1~ value, giving 900 one-step-ahead forecasts.
> Add Ŷ~t+2~ to each z~2~ value, giving 900 two-step-ahead forecasts,
> and so on.
>
> Draw one value randomly from the one-step-ahead forecasts, another
> from the two-steps-ahead forecasts, and so on to generate an h-tuple
> $y_{n + 1},y_{n + 2},\ldots,y_{n + h}$(e.g. h=5).
>
> Choose the maximum value. Repeat for a desired number of replications
> (e.g., 1000). Rank order the maximum values from all the replications
> then choose the 95% percentile (or any desired percentile) of these
> maximum values for the UPI.
>
> This example is intended to illustrate a possible Monte Carlo approach
> that will deal with estimation and misspecification errors as well as
> non-normality in the forecast error distributions. Other Monte Carlo
> approaches with the same benefits could certainly be developed. The
> important question that must be answered is whether or not the upper
> bound value derived is sufficiently high so that it is well
> calibrated; that is, actual exceedences occur with expected
> probability.

c.  For each time series find the UPI by the following simulation.
    Assume that the final estimates for the smoothing parameters, level,
    trend, and $\sigma$are true values for the model. This procedure
    would not incorporate the estimation error. Using the model (the
    observation and state equations) that was selected for the time
    series and used to get the estimates, generate several thousand
    h-tuples $y_{n + 1},y_{n + 2},\ldots,y_{n + h}$(e.g. h=5). In each
    h-tuple, choose the maximum value. Then choose the 95% percentile
    (or any desired percentile) of these maximum values for the UPI.
    Values for
    $\varepsilon_{n + 1},\varepsilon_{n + 2},\ldots,\varepsilon_{n + h}$
    will have to be selected randomly from a normal distribution or some
    other distribution. Hence, one problem is whether it is proper to
    use the normal distribution for the error terms (see the discussion
    about transformations of the PCT).

One Panel member suggested to (1) empirically examine the degree of
correlation of forecast errors in adjacent years (perhaps for data from
particular types of crops or pesticides), and (2) model the effects of
the empirically determined degree of correlation on multi-year
upper-prediction intervals using Monte Carlo simulations. If the form of
the distribution of forecast errors is known (again, from empirical
testing) and the degree of correlation is known, then existing Monte
Carlo simulation programs such as Crystal Ball or \@Risk make provisions
for correlated variables and can easily handle simulations of the
combined effects on the probability that any particular value will be
exceeded one or more times in a multi-year period.

The Panel members recommend that OPP develop a method analogous to one
of the above and test its performance on a sample of PCT series.

In summary, the Panel is in agreement that blind application of a
true-model prediction-interval formula will, for reasons of model
misspecification, omission of parameter estimation errors, and possibly
other sources of error, result in serious underestimation of
simultaneous upper prediction interval bounds, as has previously been
reported in the literature. To provide valid upper bounds, OPP must
either adopt a calibration formula or some form of Monte Carlo
simulation. Both of these approaches should be developed and tested on a
sufficiently large, randomly drawn set of PCT series so that there is
sound justification for the final procedure that OPP adopts.

# Other Issues

In addition to the issues addressed by the charge questions, the Panel
members also addressed two issues that would apply to all methods that
might be employed for projecting PCT. These issues are: the quality of
the PCT data (i.e. the historical annual values for PCT) and the use of
expert judgment in forecasting.

### Data Quality

The success of any forecasting method depends on the trustworthiness of
the input data itself. In the case of the annual PCT different data
sources are combined to obtain the historical annual PCT. Concern about
these values for the PCT was expressed by both the Panel members and in
the two written statements from the public comments. It seemed to some
Panel members that the questions about the origin, the quality, and the
weighting of the values combined in computing the historical annual PCT
should be the subject of another FIFRA Scientific Advisory Panel
meeting. There was a request for transparency of the input data as well
as the forecasting procedures. One Panel member observed that the
historical annual PCT values are themselves estimates. The estimation
error that corresponds to these estimates is not accounted for in the
"likely maximum PCT." There was concern expressed by a Panel member
about changes in how the National Agricultural Statistics Service plans
to limit their sample frame and the effect this might have on the way in
which the historical annual values of PCT are calculated.

One point that should be kept in mind is that problems with data quality
and problems with forecast accuracy are different. Obtaining better
quality data, that is, data that measure what is purported to be
measured, with more accuracy than was previously possible, will not
necessarily lead to more accurate forecasts. If the more accurate data
show as much variability as the poorer quality data, and systematic
patterns are hard to detect (the data contain a lot of noise) then no
forecasting method will deliver great accuracy. One Panel member
observed that best method used on the 17 series selected by OPP to
illustrate the proposed methodology had a mean absolute percentage
forecast error (MAPE) of around 50%. For four of the series, the
forecasted trend appears to have dropped below zero percent and was
constrained to zero. Four of the series displayed large breaks, apparent
from inspection of the data without even viewing a graph, and these
breaks appeared to occur immediately after the span of data used to
estimate the model. No statistical forecasting method can hope to
perform well under these conditions. A key question for OPP is whether
the series were deliberately selected to challenge the proposed
methodology, or whether the existence of such breaks is commonplace
among the entire set of series.

OPP apparently collects PCT at a state or regional level of aggregation.
Related to the question of data collection is whether the forecasts
should be done on regional data or on aggregated data. One Panel member
recommended that OPP forecast at the disaggregated level (i.e., state or
region) and then aggregate the forecasts to the national level (this was
referred to as bottom-up forecasting). Although the evidence is not
strong, it does favor the bottom-up method, when data availability makes
it possible (See for example, Dangerfield and Morris, 1992).

**Judgmental Forecasts or Adjustments to Forecasts**

Professional judgment in the development of the forecasts is another
area that received multiple comments. The OPP is combining professional
judgment with its current methodology and would continue to do so with
any new methodology. Statistical forecasting methods assume that the
future is like the past. Hence, incorporating expert judgment is
important for many reasons, including changes in prices, new pests, pest
resistance and cross-resistance, invasive species, new products, and
weather changes. One Panel member stated that the most important thing
that an analyst can do is hypothesize about the direction of the trend
to be expected given knowledge about underlying forces other than
historical trend. Developing rules for implementing judgmental forecasts
and when they are applied would aid the analyst in providing forecasts
and the public in understanding these forecasts. It needs to be clear if
the judgment is used to make the forecast with no statistical forecast,
to improve the choice of a statistical model, or to alter a statistical
forecast. One Panel member suggested that "expert opinion" be used while
the data is still disaggregated rather than have an analyst work on the
back end. Another Panel member noted that adjustments do not always
improve forecasts and that all adjustment should be documented and then
audited to see the impact on forecasting accuracy. There were
suggestions for the OPP to improve the judgmental aspects of their
forecasting by using publicly available resistance databases and to
commission a regular series of market research reports on segments of
the pesticide industry. In addition, one Panel member suggested the
following reference on judgmental forecasting: Sanders, Nada R. and
Ritzman, Larry P. \"Judgmental Adjustment of Statistical Forecasts,\"
pages 405-416 in J. Scott Armstrong's (editor) *Principles of
Forecasting: A Handbook for Researchers and Forecasters*, Kluwer
Academic, Boston, MA, 2001.

[]{#__RefHeading___Toc24430696 .anchor}**REFERENCES**

Chatfield, Christopher, "Calculating Interval Forecasts," *Journal of
Business and Economic Statistics*, 11, pp. 121-135, 1993.

Dangerfield, B.J., and Morris, J.S., "Top-down or bottom-up: Aggregate
versus disaggregate extrapolation," *International Journal of
Forecasting*, 8, pp. 233-241, 1992.

Hahn, Gerald J., "Factors for Calculating Two-Sided Prediction Intervals
for Samples from a Normal Distribution," *Journal of the American
Statistical Association*, 64,

pp. 878-888, 1969.

Hahn, Gerald J. and Richard W. Hendrickson, "A Table of Percentage
Points of the Distribution of the Largest Absolute Value of *k* Student
*t* Variates and Its Applications," *Biometrika*, 58, pp. 323-332, 1971.

Hattis, D. and Burmaster, D. E. "Assessment of Variability and
Uncertainty Distributions for Practical Risk Analyses" [Risk
Analysis,]{.underline} Vol. 14, pp. 713-730, 1994.

Sanders, Nada R. and Ritzman, Larry P. \"Judgmental Adjustment of
Statistical Forecasts,\" pages 405-416 in J. Scott Armstrong (editor)
*Principles of Forecasting: A Handbook for Researchers and Forecasters*,
Kluwer Academic, Boston, MA, 2001.

Shlyakhter, A. I., and Kammen, D. M., "Sea-Level Rise or Fall?"
[Nature]{.underline}, Vol. 253, p. 25, May 3, 1992.