# Presentation: 789010

## Focusing on the Risk Decision: Improving Communications between Risk Assessors, Risk Managers, and Research Scientists

**D. Warner North**
- President and Principal Scientist, NorthWorks, Inc. Consulting Professor, Department of Management Science and Engineering, Stanford University
**E-mail: **_[northworks@mindspring.com](mailto:northworks@mindspring.com)_
**Web: **_[www.northworks.net](http://www.northworks.net/)__ _
**ORACBA September Risk Forum**** **
- Monday, September 19, 2005

## Overview: Two Themes

**Risk analysis is as much an art as a science** (W. North, Tuskegee University Int’l S/PS Symposium August 12, 2005:  web:  _[http://compepid.tuskegee.edu/RiskConference/Warner%20North.ppt#2](http://compepid.tuskegee.edu/RiskConference/Warner%20North.ppt#Overview: Two Themes)_)
**Risk assessment and risk management should be ****conceptually distinct****.  These two processes should ****NOT**** be ****separated**** organizationally and ****carried out sequentially in time****.**  Rather, there should be ongoing good communication between risk assessors and risk managers.  It may be advisable to view the process as an iterative, back-and-forth alternation between analysis and deliberation.  (W. North, “Reflections on the Red/Mis-Read Book, 20 Years After,” HERA 9:1145-54, 2003; National Research Council, *Understanding Risk*, 1996)

## Quote: Theme 1

-      Risk analysis provides a **useful tool kit** for dealing with **uncertainty and complexity**, but it does not make difficult problems simple, especially where societal values are involved. For the foreseeable future, risk analysis is going to be **as much of an art as a science**.  **Skilled practitioners will not be plentiful**, especially in accomplishing effective two-way risk communication.
- -- D. Warner North, from First International Symposium, Risk Assessment for Veterinary Biologicals, Washinton D.C., December 5, 1994.  (Page 921 of Limitations, definitions, principles, and methods of risk analysis,” *Rev. sci. tech. Off. int. Epiz. (Scientific and Technical Review of the Office International des Epizooties),* **14**(4):913-923, 1995

## Music School: Moscow 

## How Do You Get to ... ?

## The Answer: 

**Practice! Practice!**

## Vision and Mission Statements:   S/PS Risk “Art School” 

**Vision**:  **successful, effective decision support** in highly uncertain and complex SPS regulatory decisions
**Mission (Tools and Training):**  Learn technique, master the media (materials), evaluate your audience, learn from your colleagues, recognize that the profession is highly competitive, maintain your integrity.

##  Curriculum for SPS Risk “Art School” 

**Risk Analysis 101**:  Risk 101: Use of probabilities and consequences to describe risk, in context of a regulatory decision.  How to achieve good documentation and **effective communication.**
**Risk Analysis 201:** Ongoing and adaptive risk management through evolving regulation, with a proactive scientific research program in support of decision improvements.  Use of  value-of-information calculations to guide research, allocate research funds.  How to achieve  good documentation and **effective communication**.

## Risk 101

- Use of probabilities and consequences to describe risk, in context of a regulatory decision.  How to achieve good documentation and effective communication
**Main Tools**:
- Systems analysis/modeling: fate and transport
- Sensitivity analysis
- Assessment of probabilities for most important uncertainties (based on data where available – and/or expert judgment)
- More sensitivity analysis
- Careful documentation: an easy-to-read short summary, comprehensive and transparent backup, sources of data and judgment specified.

## Risk 101 Example: Microbes on Mars

- Do we infect Mars with terrestrial microbes by landing a spacecraft on its surface?

## New Methodology for Assessing the Probability of Contaminating Mars  

- D. W. North, B.R. Judd, and J.P. Pezier, ***Life Sciences and Space Research XIII***, P. A. Sneath, ed., Akademie-Verlag, Berlin, pp. 103-109, 1975.
- “Limitations, Definitions, Principles, and Methods of Risk Analysis,” Risk Assessment for Veterinary Biologicals, special issue, **Office International des Epizooties, Scientific and Technical Review**, Vol. 14, pp. 913-923, 1995.  (on website, _[http://www.northworks.net/limitations.pdf](http://www.northworks.net/limitations.pdf)_ )
- 1974 Final Report to NASA: much longer, available on request

## Mission Contamination Model Results

## Mission Contamination Model Marginal Sensitivity Analysis

|  |  |  |  |  | Probability of Contamination  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Contamination Model Variables  |  |  | Values  |  |  | Units: = 10**-6 |  |
|  | Extreme  | Intermed. |  | Intermed. | Extreme  | Nominal: 5.9  |  |
|  | Low  | Low  | NOMINAL | High  | High  | Low | High  |
| Bio-Burden Variables  |  |  |  |  |  |  |  |
| 1. bio External  | 2.2 | 5.5 | 11 | 22 | 55 | 5 | 10.7 |
| 2. bio Covered | 3.2 | 8 | 16 | 32 | 80 | 3.1 | 20.2 |
| 3. bio Encapsulated | 4,000 | 10,000 | 20,000 | 40,000 | 100,000 | 5 | 10.4 |
|  |  |  |  |  |  |  |  |
| Release Variables |  |  |  |  |  |  |  |
| 1. rel Hard Landing Probability  | 0.0004 | 0.001 | 0.002 | 0.004 | 0.01 | 5.2 | 9.6 |
| 3. rel Newly Exposed/Hard, Encaps | 0.0001 | 0.0002 | 0.001 | 0.005 | 0.01 | 5.4 | 10.9 |
| 4. rel Implanted, Soft | 0.0001 | 0.0002 | 0.001 | 0.005 | 0.01 | 5.7 | 8.7 |
| 6. rel VTO/Vibration | 0.001 | 0.002 | 0.01 | 0.05 | 0.01 | 5.4 | 11.1 |
| 9. rel VTO/Erosion, Encaps | 0.00001 | 0.00002 | 0.0001 | 0.0005 | 0.001 | 5.4 | 10.9 |

## Mission Contamination Model Marginal Sensitivity Analysis -2

|  |  |  |  |  | Probability of Contamination  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Contamination Model Variables  |  |  | Values  |  |  | Units: = 10**-6 |  |
|  | Extreme  | Intermed. |  | Intermed. | Extreme  | Nominal: 5.9  |  |
|  | Low  | Low  | NOMINAL | High  | High  | Low | High  |
| Transport Variables |  |  |  |  |  |  |  |
| 1 tra Survive Transit | 0.001 | 0.002 | 0.01 | 0.05 | 0.1 | 2.2 | 45.2 |
| 2 tra Find Water | 0.0005 | 0.001 | 0.005 | 0.025 | 0.05 | 1.5 | 49.9 |
| 4 tra Water Deposition | 0.00005 | 0.0001 | 0.0005 | 0.0025 | 0.005 | 5 | 15.2 |
| 5 tra Stay Lodged | 0.1 | 0.2 | 0.5 | 0.8 | 0.9 | 5.5 | 10 |
|  |  |  |  |  |  |  |  |
| Reproduction Variables  |  |  |  |  |  |  |  |
| 1 rep Psychrophilic, Anaerobic  | 0.005 | 0.01 | 0.05 | 0.1 | 0.25 | 0.6 | 29.6 |
| 2 rep Availability of Nutrients  | 0.01 | 0.02 | 0.1 | 0.2 | 0.5 | 0.6 | 29.6 |

## National Academy of Sciences, Viewpoint -1992 

-     “... it is the unanimous opinion of the task group that terrestrial organisms have almost no chance of multiplying on the surface of Mars and in fact have little chance of surviving for long periods of time, especially if they are exposed to wind and to UV radiation.”
-      ---- Space Studies Board, National Research Council,   *Biological Contamination of Mars, *1992, page 49.* *

## When the Uncertainties Matter, and Data Are Not Available

- Elicitation of Expert Judgment:
**History**: Bayesian school of probability
- Extensive **research in social sciences**: people are not good at processing information about uncertainty.
**Need careful approach**: trained elicitors, careful framing of questions, good technique

## References on Probability: Human Fallibility, Good Assessment Technique

- D. Kahneman, Paul Slovic, and A. Tversky, *Judgment under Uncertainty: Heuristics and Biases*, Cambridge Univ. Press, 1982.
- M. G. Morgan and M. Henrion, *Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Public Policy Analysis*, Cambridge Univ. Press, 1990
- C. S. Spetzler and C.A. Staël von Holstein, “Probability Encoding in Decision Analysis,” *Management Science* **22**: 340-352, 1975
- T. S. Wallsten and D. V. Budescu, “Encoding Subjective Probabilities: A Psychological and Psychometric Review,” *Management Science* **29**:151-173, 1983
- M. G. Morgan and D. W. Keith, “Subjective Judgments by Climate Experts, *Environmental Science and Technology* **29**(10):468A-476A, 1995

## Range of Expert Opinion: Climate Change

- Source:
- Morgan
- and Keith,
- ES&T,
- 1995

## Risk 101, Advancing toward Risk 201: Value of Information

- Even if probability assessments are debatable, so that best immediate regulatory decision is not clearly evident, there may be an opportunity for **robust conclusions on the value of better information via research**.
**Two examples**:
- Air Quality: Sulfur Oxides and Fine Particulate Matter (Omitted: ask me via e-mail if interested)
- Weather Modification on Hurricanes
**Reference on Value of Information**:  F. Yukota and K. Thompson, “The Value of Information in Environmental Health Risk Management Decisions: Past, Present, Future,” *Risk Analysis* **24**(3):635-650, 2004

## Expected Value of Further Information

**Concept:**
- Expected value of further information is the
- expected value taken over possible outcomes from research of the further information – of best decision taken based on this new information
-   less
- expected value with best decision taken on the basis of current information
-  (can generalize to expected utility).  Standard concept or “tool”  in decision analysis textbooks.  Example: R.T. Clemen, *Making Hard Decisions: An Introduction to Decision Analysis*, Duxbury Press, 1991.

## VOI Example:  The Decision to Seed Hurricanes

***Science, *****176**, 1191-1202, 1972
- _[http://northworks.net/hurricanes.pdf](http://northworks.net/hurricanes.pdf)__ _

*R.A. Howard, J.E. Matheson, D.W. North*

## New York Times, Thursday, October 3, 2002 

**Thousands Seek Safety as Hurricane Nears Gulf Coast**** **By JEFFREY GETTLEMAN (NYT) 1308 words Late Edition - Final , Section A , Page 24 , Column 3
- ABSTRACT - About half million people flee southern Louisiana and Texas with approach of **Hurricane Lili**, which is heading toward Gulf Coast with winds of more than **140 miles per hour, making it daunting Category 4 storm**; map; chart of five most intense hurricanes to hit US since 1928; photos (M) Highways across southern Louisiana and Texas were solid columns of steel today as more than half a million people grabbed their valuables and fled their homes, looking for higher, safer ground before Hurricane Lili hit.
- The exodus of cars and trucks, some with furniture lashed down on their roofs, began in low-lying areas but quickly spread inland **as the storm intensified and threatened to become the worst natural disaster here in decades.**

## Value of A Seeding Experiment(Government Responsibility Cost Included)

**Value = $2.96 million**
**Actual cost = $0.25 million**

- $103.44

- 	+32	%
- 	+16
- 	0
- 	-16
- 	-34
- 	+32%
- 	+16
- 	0
- 	-16
- 	-34
- 	+32%
- 	+16
- 	0
- 	-16
- 	-34
- 	+32%
- 	+16
- 	0
- 	-16
- 	-34

**Outcomes**

- 	$503.7
- 	248.4
- 105.0
- 46.7
- 16.3
- 503.7
- 248.4
- 105.0
- 46.7
- 16.3
- 503.7
- 248.4
- 105.0
- 46.7
- 16.3
- 503.7
- 248.4
- 105.0
- 46.7
- 16.3

**Total Cost**
**(millions of dollars)**

**Operational Seeding Decision**

- $103.44

- $116.00

- Seed

- Do Not Seed

- $110.92

- $116.00

- Seed

- Do Not Seed

- $110.92

- $107.96

- $110.92

- 0.038

- Perform Experiment

- Do not Perform Experiment

**Results of Experiment**

**Choice of Whether to Perform Experiment**

- Expected Value (M$)

- $87.83

- $116.00

- $116.00

- $116.00

- 0.143

- 0.392

- 0.255

- 0.172

## Summary of the Value of Additional Information on the Effect of Seeding (Values in Millions of Dollars)

- Considering only the 50% of hurricanes that are assumed to be possible candidates for seeding because of tactical consideration. If all hurricanes are assumed to be candidates for operational seeding, the figures of the last two columns should be doubled.

## Findings
1.	Current meteorological and economic information indicates that the seeding alternative stochastically dominated the non-seeding alternative
2.	No firm legal basis for operational seeding appears to exist
3.	The decision to seed a particular hurricane should take into account its specific characteristics
4.	Resolving meteorological uncertainty on the effect of hurricane modification is worth over $20 millions/year

- 1.	Current meteorological and economic information indicates that the seeding alternative stochastically dominated the non-seeding alternative
- 2.	No firm legal basis for operational seeding appears to exist
- 3.	The decision to seed a particular hurricane should take into account its specific characteristics
- 4.	**Resolving meteorological uncertainty on the effect of hurricane modification is worth over $20 millions/year**

- Recommendations
- 1.	The present policy prohibiting seeding any hurricane threatening the U.S. should be rescinded
- 2.	A hurricane modification agency with authority to seed operationally should be established
- 3.	Decision procedures supported by further analysis should be developed
- 4.	**Modification experiments should be conducted on an expanded scale to provide a more refined basis for making each operational seeding decision**

*Findings and Recommendations** *

## Risk 201:  Adaptive Management under Uncertainty, with Directed Research 

- Most SPS decisions are not isolated decisions at one point in time, but ongoing issues of managing safety in international trade in plant and animal products
- Value of information from research:  What is the expected value from research that might show that a pest in crop C from country A will not be present in export shipments? Or cannot survive and replicate in the climate of a country B that imports crop C from country A?
- How can regulatory authorities in countries A and B work together to assure that the appropriate research is done and properly used?

## A Success Story: Hass Avocados

- Anastrepha ludens fruit fly:
- “Not all alien invaders
- are from outer space” - APHIS

## Hass Avocado Chronology

- 1914: US quarantine of Mexican avocados into the lower 48, under Food Protection Act of 1912.
- 1994: Mexico proposal to APHIS to amend regulation to allow import of Hass avocados from approved orchards in Michoacán Province.
- January 1997: Workshop by Harvard Center for Risk Analysis, funded by grant from California Avocado Commission (“CAC”).
- February 5, 1997: USDA APHIS Final Rule allowing Hass avocados into 19 northern states + DC during November through February.
- October 1999: Mexico asks for modification: longer time, more states.
- November 1, 2001: APHIS Final “Expansion” Rule: Adds 12 states, time of Oct 15 – April 15.
- January 31, 2005: APHIS new Final Rule: all months, all states but California, Florida, Hawaii; add these 3 in two years.
- May 18, 2005: Litigation resolved. Federal Judge Robert Coyle finds legal challenges by CAC to USDA moot.  Decision not appealed, suit not re-filed by CAC.

## My Viewpoint as an “Expert Witness”

- Testimony in June 2002, August 2002 in support of adequacy of APHIS pest risk analysis, on behalf of Mexican growers.
- More testimony, March 2004 supporting what became the Jan 31 2005 APHIS Rule: all states, all times of year.
- Comments to APHIS on draft 2003 draft pest risk assessment.
- Key points:  (1) strength of Mexican research by Dr. Martin Aluja: Hass avocado not a host to Anastrepha fruit flies.  (2) Inspection of over 10 million fruit: no pests found. (3) Pest risk draft good -- but could be improved, especially communication aspects and some technical details.

## Pest Risk Assessment Guidelines

- Reference:
- George M. Gray, Jon C. Allen, David E. Burmaster, Stuart H. Gage, James K. Hammitt, Stanley Kaplan, Ralph L. Keeney, Joseph G. Morse, D. Warner North, Jan P. Nyrop, Alina Stahevitch, and Richard Williams, Principles for Conduct of Pest Risk Analysis: Report of an Expert Workshop, *Risk Analysis* **18**(6):773-780, 1998.

## Risk 201:  Adaptive Management under Uncertainty, with Directed Research 

- Most SPS decisions are not isolated decisions at one point in time, but ongoing issues of managing safety in international trade in plant and animal products
- Value of information from research:  What is the expected value from research that might show that a pest in crop C from country A will not be present in export shipments? Or cannot survive and replicate in the climate of a country B that imports crop C from country A?
**How can regulatory authorities in countries A and B work together to assure that the appropriate research is done and properly used?**

## National 

- National
- Academy
- Press, 1996;
- Catalog 5138

## Guidance from Understanding Risk: Analytical-Deliberative Process

- Inform the interested and affected parties among the public
- Respect differences in values and cultures
- Seek flexibility and maintain choices
- Seek progress in stages, rather than seeking overall solutions.
- Source:  National Research Council, *Disposition of High-Level Waste and Spent Nuclear Fuel: The Continuing Societal and Technical Challenges*, National Academy Press, 2001, page 130.   _[http://books.nap.edu/catalog/10119.html](http://books.nap.edu/catalog/10119.html)__ _

## Coming Next in the National Academies Series ...

- Panel on Public Participation in Environmental Assessment and Decision Making.  Last Panel Meeting, Sept 16-17, 2005.  Expected date for report: mid-2006. Web: _[http://www7.nationalacademies.org/hdgc/Public_Participation.html](http://www7.nationalacademies.org/hdgc/Public_Participation.html)_
- Website for workshop papers from Feb 2005 Workshop: _[http://qp.nas.edu/QuickPlace/publicparticipation/Main.nsf/h_Toc/4df38292d748069d0525670800167212/?OpenDocument](http://qp.nas.edu/QuickPlace/publicparticipation/Main.nsf/h_Toc/4df38292d748069d0525670800167212/?OpenDocument)_ You may need to register first – see instructions. Then click on “MATERIALS” before “Page.” Then go to TAB 12 for:
-      D. Warner North and Ortwin Renn, “Decision Analytic Tools and  Participatory Decision Processes,” Background “State of the Science” Paper - Tab 12.

## Theme #2: Back to 1983, The Red Book (or “Mis-Read” Book)

- National Research Council,
*Risk Assessment in the Federal Government:*
*Managing the Process,* 1983 ( available at _[www.nas.edu](http://www.nas.edu/)__ )_
- 20th Anniversary Special Issue: *Human and Ecological Risk Assessment* (HERA), Vol. 9, No. 5, August 2003.  See W. North paper, other papers, editors’ comments
- (journal not available on line; copies of my paper available today, or by request)

## “Red Book” Recommendations 

- “Conceptual distinction” ( not separation) between risk assessment and risk management – and ongoing communication between those responsible for risk assessment and those responsible for risk management.
- Risk assessments should be published as publicly available documents
- These documents should be reviewed by a group of scientists from outside the agency preparing the document
- Results of this peer review should be published and available

## Risk Assessment: “Figure 1”

- Source: National Research Council, *Risk Assessment in the *
*Federal Government: Managing the Process *(“Red Book”), 1983, page 21

## 1983 Red/Mis-Read Book: What Went Right – and Wrong?

**Right:**
- Formalization of Risk Assessment – Figure 1/Chapter 1 endorsed by National Academies ...
-  Guidelines for risk assessment: OSTP for carcinogens, EPA for carcinogens, ...
**Wrong:**
- EPA Administrator William Rickelshaus’ speeches/papers:  “Academy told us to separate risk assessment from risk management ...”  Committee did **NOT** say this.
- More general problem: agency leaders want appearance of objective science, carried out via standardized process
**PLEASE READ** THE 1983 BOOK!  Also 1996 book, and maybe also the 1994 book, *Science and Judgment in Risk Assessment.  *

## Optimize the Tool Kit?Optimize the Risk Team! 

-      Risk analysis provides a **useful tool kit** for dealing with **uncertainty and complexity**, but it does not make difficult problems simple, especially where societal values are involved. For the foreseeable future, risk analysis is going to be **as much of an art as a science**.  **Skilled practitioners will not be plentiful, especially in accomplishing effective two-way risk communication. **
- -- D. Warner North, from First International Symposium, Risk Assessment for Veterinary Biologicals, Washinton D.C., December 5, 1994.  (Page 921 of Limitations, definitions, principles, and methods of risk analysis,” *Rev. sci. tech. Off. int. Epiz. (Scientific and Technical Review of the Office International des Epizooties),* **14**(4):913-923, 1995
**     ****Prepare! Practice! Improve with practice!  ****Be as good as you can be**** in carrying out the ****art**** and the ****science**** of risk analysis – especially ****interactive communication**** with management (and stakeholders, research scientists).**
- -- D. Warner North, International Conference on Sanitary and Phytosanitary Risk Assessment Methodology August 9, 2005, Expanded for September 19 talk.

## Concluding Quote 

-    If a man will **begin**** ****with certainties** he shall **end in doubts**, but if he will be content to **begin with doubts**, he shall **end in certainties**.
- -- Francis Bacon *The Advancement of Learning*, 1605.