# Presentation: 020365

## Cumulative Risk Assessment for Pesticide Regulation: A Risk Characterization Challenge

- Mary A. Fox, PhD, MPH
- Linda C. Abbott, PhD
- USDA Office of Risk Assessment and Cost-Benefit Analysis

## Cumulative Risk Assessment for Pesticide Regulation

- Debut of multi-chemical assessment of pesticide exposure through food, water, and residential uses
- Highly refined dose-response and exposure assessment
- Nationally representative dietary assessment
- What do we know about risk characterization for such complex assessments?

**Notes:**
Novelty of the cumulative assessment

Many technical refinements, especially in DR and EA

The dietary assessment incorporates a number of USDA datasets – food survey and pesticide residues – hence our interest in the modeling effort as well as the ultimate decision-making on pesticide uses  

As I learned about the technical refinements in DR and EA that are the cumulative OP risk assessment I wondered if there was guidance on how to do RC for such a complex assessment. I thought the question was worthy of attention and I’m pleased that SRA thought so too.

## Risk Characterization DefinedNAS 1996

- From _Understanding Risk:_
- A synthesis and summary of information about a potentially hazardous situation that addresses the needs and interests of decision makers and interested and affected parties
- Analytic-deliberative process
- The process of organizing, evaluating, and communicating ...

**Notes:**
Just so we’ re all on the same page...this is what I’m thinking about when I say RC

I like the NAS definition of risk characterization because it includes not only what a good risk characterization should contain – the synthesis and summary- but also acknowledges the importance how one does it ( the process) and the various audiences – decision makers and interested and affected parties.  

## Outline

- Identify key elements of risk characterization for probabilistic assessments
- Evaluate the risk characterization chapter of the revised organophosphate (OP) assessment
- Review example highlighting importance of uncertainty and sensitivity analyses

**Notes:**
My first task today is to review guidance on risk characterization and distill out the key elements that I will then apply to the revised OPCRA released over the summer.  I will conclude with an example that underscores the complexity of the cumulative model and the importance of an analytic risk characterization.

## Resources

- Presidential/Congressional Commission on Risk Assessment/Management, 1997
- US EPA Guidance
- Principles for Monte-Carlo Analysis, 1997
- Risk Characterization Handbook, 2000
- US EPA Revised OP Cumulative Risk Assessment, 2002
- DEEMTM and DEEM-FCID TM
- Data files for methamidophos

**Notes:**
My synthesis of the guidance comes mostly from the Commission and the Guiding Principles for MCA. The RC Handbook is not as detailed regarding complex model driven assessments.

Warning – please read these documents.  I am glossing over many details

The modeling experiment I will present compares assessments for one OP more on this later.

## Presidential Commission, 1997

- Quantitative and qualitative descriptions of risk
- Summarize weight of evidence
- Include information on the assessment itself
- Describe uncertainty and variability
- Use probability distributions as appropriate
- Use sensitivity analyses to identify key uncertainties
- Discuss costs and value of acquiring additional information
- _Did not recommend:_
- Use of formal quantitative analysis of uncertainties for routine decision-making (i.e. local, low-stakes)

**Notes:**
Highlights of the Commissions recommendations on risk characterization:

Quantitative and qualitative
Describe the assessment 
Use sensitivity analyses to identify important uncertainties
Describe uncertainty and variability
Use distributions as appropriate

Commission made clear that uncertainty analysis was not recommended for routine decision-making....
I would argue that the OPCRA is not routine decision making and uncertainty analysis is appropriate and necessary

## Excerpts from Guiding Principles of Monte Carlo Analysis, US EPA 1997

- Selecting Input Data and Distributions
- Conduct preliminary sensitivity analyses
- Evaluating Variability and Uncertainty
- Separate variability and uncertainty to provide greater accountability and transparency.
- Presenting the Results
- Provide a complete and thorough description of the model.  The objectives are transparency and reproducibility.

**Notes:**
Begin to see the common elements of risk characterization for probabilistic assessments:

16 principles – 3 categories

Again sensitivity analysis (in contrast to the Commission, this EPA guidance emphasizes using sensitivity analysis early on to learn about the model and important inputs)

Separate and evaluate variability and uncertainty

Thorough description of the model and equations to promote transparency and reproducibility

## Risk Characterization Handbook, 2000

- Transparency
- Explicitness
- Clarity
- Easy to understand
- Consistency
- Consistent with other EPA actions
- Reasonableness
- Based on sound judgment

**Notes:**
I found that the RC Handbook was a bit generic for the task of evaluating a probabilistic assessment but the criteria for transparency echo the Commission and the principles for MCA

Describe assessment and assumptions

## Transparency Criteria

- Describe assessment approach, assumptions
- Describe plausible alternative assumptions
- Identify data gaps
- Distinguish science from policy
- Describe uncertainty
- Describe relative strengths of assessment

**Notes:**
These criteria echo the important points from the other documents

## Key Elements of Risk Characterization

- Separately track and describe uncertainty and variability
- Conduct sensitivity analyses
- Conduct formal uncertainty analyses
- Transparency and reproducibility
- Model components
- Basic operational details

**Notes:**
From that whirl-wind tour of recent guidance I extracted these 4 points as the keys to the characterization of probabilistic risk assessments

Review first 3 points

Provide enough detail on the model and its basic operations to ensure transparency and reproducibility

## Evaluation of the Revised OP Cumulative Assessment

- Track and describe uncertainty and variability
- Sensitivity analyses
- Uncertainty analyses
- Yes, but ...spotty, qualitative, not comprehensive
- Transparency/reproducibility – No
- Significance of many inputs unknown
- No mention of random seed, # iterations used

**Notes:**
The risk characterization for the cumulative assessment gets so-so marks on these criteria:

Distinctions between uncertainty and variability are clear but selected
Sensitivity and uncertainty, inconsistent,  mostly qualitative
I don’t think the OP cumulative risk characterization meets expectations for transparency and reproducibility

Many inputs are not discussed at all in the risk characterization or in other sections – the example that follows is one of these

I could not find a record of the random number seed used for the dietary analysis, but the number of iterations (1000) does appear in an appendix

Model limitations do not help with rigorous analysis as outlined by Commission

There is more we can and must do, as example illustrates

## Recipes – essential to dietary model

- Break down foods reported in dietary recall records to commodities that can be matched with pesticide residue data
- Recipes are ‘representative’ with nutritional basis
- May not accurately reflect commodities eaten
- E.g. beef stew with vegetables – recipe includes carrots but could be broccoli or leafy greens
- DEEM TM – proprietary recipes
- DEEM-FCID TM – EPA & USDA collaboration
- Policy relevant

**Notes:**
Recipes are an input that is not discussed in the revised cumulative risk characterization.

 Recipes have a fundamental role in the dietary model  - they are the bridge between a dietary recall record and the agricultural commodities for which we have pesticide residues  SHOW NEXT SLIDE – TOMATO SOUP

Our concern about how the recipes contribute to the exposure assessment stems from their ‘representative’ nature because getting the right commodities is essential

Two sets of recipes have been used in risk ass’t and management for the OP pesticides... Describe models – then policy approach
	Dietary Exposure Evaluation Model
	Food Commodity Intake Database
US EPA’s approach to risk assessment and management of the OPs used both single chemical assessments done with DEEM and the dietary analysis for the cumulative assessment with FCID.  

Both the nature of the recipes and how they have been applied to the OP assessment lead us to look at their influence on exposure estimates

## Tomato Soup Recipe

## Experiment to examine importance of recipes

- Focus on one chemical- methamidophos
- Look at dietary exposure using DEEM TM and DEEM-FCID TM
- Forty 1000 iteration replicates with different random number seeds
- 1-6 year olds, 99.9th %ile, exposures in mg/kg-day

**Notes:**
To focus in on the recipes and to avoid the complication of different toxicological potency among the OP pesticides we looked at methamidophos

We ran DEEM and DEEM-FCID with the same set of methamidophos residue data files

40 1000 iterations replicates

Children 1-6 is population of interest, 99.9th %, the exposure estimates in the following slides are in units of mg/kg body weight-day

## Between Model Exposure Variability Forty 1000-Iteration Replicates, Different Random Number Seeds

|  | DEEM TM Estimate | DEEM-FCID TM
Estimate | Difference | % Difference |
| --- | --- | --- | --- | --- |
| Minimum | 7.43 x 10-4 | 8.54 x 10-4 | 1.02 x 10-4 | 13.56 % |
| Maximum | 7.63 x 10-4  | 8.80 x 10-4 | 1.32 x 10-4 | 17.74 % |
| Mean | 7.53 x 10-4 | 8.69 x 10-4 | 1.17 x 10-4 | 15.48 % |

**Notes:**
This table show the summary statistics from the model runs

We found that the recipes do influence the exposure estimates 

DEEM with FCID estimates are 15% higher on average

We suspect that FCID recipes may refer to more commodities than the DEEM recipes, altering the progression of random numbers – which in turn alters the draws from the residue distributions 

## Within Model Exposure Variability Forty 1000-Iteration Replicates, Different Random Number Seeds

|  | DEEM TM | DEEM-FCID TM |
| --- | --- | --- |
| Within model exposure variability | 2.69 % | 3.04 % |

*On par with US EPA findings for 1000-iteration runs*

**Notes:**
The within model variability gives some context for 15% difference

This is the same series of runs summarized in the previous slide

Here you see that within each model exposure estimates differ by about 3 % - this is on par with US EPA’s experience with 1000 iteration runs

## Exposure variability findings in contextPreliminary data files, Children 1-2, Single 1000 iteration runs

| Commodities | DEEM-FCID TM
Estimate | Difference | % Difference from complete model |
| --- | --- | --- | --- |
| Exclude grapes | 0.00157 | 0.00024 | 15.29 % |
| Exclude apples | 0.00163 | 0.00018 | 11.00 % |
| All included | 0.00181 |  |  |

*Average DEEM vs. FCID difference is 15%*

**Notes:**
This example comes from another experiment that we did using the preliminary OPCRA files – these are one-time 1000 iteration runs of the preliminary multiple pesticide assessment,  random seed = 7000

Differences we found due to the recipes in the methamidophos example are akin to complete removal of a major kid’s food commodity from the cumulative assessment.

## Risk Metric Comparison – 15% Difference

- Margin of Exposure (MOE) = _Toxicological Benchmark _
-                                                  Exposure Estimate
- Revised OPCRA Tox. Benchmark for dietary = 0.08 mg/kg-d
- MOE average exposure DEEM = 0.08 / 0.000753 = 106
- MOE average exposure FCID =  0.08 / 0.000869  =   92

**Notes:**
Example: Does 15% have regulatory significance?  Yes!  The larger exposure estimate from the FCID model gives an MOE below 100 – indicates health concerns and triggers further review.

## Conclusions

- Risk characterization is incomplete
- Good guidance on risk characterization for complex models
- Continue to work and share findings

**Notes:**
A lot yet to learn about the cumulative model

Computational limitations of the model mean we need to be more creative in figuring things out 

Encouraged that we have guidance on what to strive for