Executive Office of the President Executive Office of the President Office of Management and Budget Office of Science and Technology Policy

September 19, 2007

M-07-24

MEMORANDUM FOR THE HEADS OF EXEC UTIVE DEPARTMENTS AND AGENCIES

FROM: Susan E. Dudley Administrator, Office of Info rmation and Regulatory Affairs, Office of Management and Budget

Sharon L. Hays

Associate Director and Deputy Director for Science, Office of Science and Technology Policy

SUBJECT: Updated Principles for Risk Analysis

Federal agencies take a variety of actions to improve public health, safety, and the environment. Agency activities designed to re duce risks are influenced by numerous factors, including Congressional priorities, informati on on the degree of risk faced by different populations, entities, or individua ls, resources available, and the ease of implementing chosen priorities. Development of these actions often begins with an assessment of the risks posed under certain conditions, as well as assessments of the potential cha nges in risk achievable due to different policy options.

In 1995, an interagency working group, co-chair ed by the Office of Management and Budget (OMB) and the Office of Science and Technology Policy (OSTP), developed a set of principles to guide policymakers in assessing, mana ging, and communicating policies to address environmental, health, and safe ty risks (the 1995 Principles). 1 The 1995 Principles, shared with regulatory agencies in a memorandum from Sally Katzen, then Administrator of OMB’s Office of Information and Regulatory Affairs (OIRA), remain valid today.

This Memorandum reinforces the 1995 Principles with reference to more recent guidance from the scientific community, the Congress, a nd the Executive Branch. This Memorandum also

benefits from feedback received on OMB’s Pr oposed Risk Assessment Bulletin issued in 2006 (Proposed Risk Assessment Bulletin). 2

1 U.S. Office of Mgmt. and Budget (OMB), Memorandum for the Regulatory Working Group, Principles for Risk Analysis (1995), available at http://www.whitehouse.gov/omb/inforeg/regpo l/jan1995_risk_analysis_principles.pdf

. 2 OMB, Proposed Risk Assessment Bulletin , (2006) [hereinafter Proposed Risk Assessment Bulletin], available at http://www.whitehouse.g ov/omb/inforeg/proposed_risk_ assessment_bulletin_010906.pdf

.2

In January 2006, OIRA, in consultation with OSTP, released the Proposed Risk Assessment Bulletin for public comment and asked the National Academy of Sciences (NAS) to conduct an expert peer review. 3 The NAS issued its report on the Proposed Risk Assessment Bulletin in 2007 (the 2007 NAS Report on the Pr oposed Risk Assessment Bulletin). 4 While supportive of the goal of “increasing the quality and objectivity of risk assessment in the federal government,” the NAS recommended an approach that would “ outline goals and general principles of risk

assessment.” 5 After carefully evaluating these constr uctive recommendations from the NAS, as well as feedback from a rigorous inte ragency review, and public comments, 6 we have decided

not to issue the bulletin in final form. Rather , we are issuing this Memorandum to reinforce generally-accepted principles for risk analysis upon which a wide consensus now exists.

Recognizing the diversity of documents that stem from risk analys is techniques, this Memorandum reinforces generally-accepted principles for risk analysis related to environmental, health, and safety risks. 7 As a whole, the Memorandum endeavors to enhance the scientific quality, objectivity, and utility of Agency risk analyses and the comple mentary objectives of improving efficiency and consistency among the Federal family. 8 The general principles presented here should continue to assist and gu ide agencies as they conduct risk analyses, and thereby should enhance the multiple deci sions that are based upon these analyses.

The 1995 Principles were divided into five parts: general prin ciples, principles for risk assessment, principles for risk management, pr inciples for risk communication, and priority setting. 9 This Memorandum reiterates each of th ese principles (in bold text) and, where appropriate, highlights and references more recen t guidance (in plain text). OMB and OSTP will

3 See Press Release, OMB, OMB Requests Peer Review of Proposed Risk Assessment Bulletin (Jan. 9, 2006),

available at http://www.whitehouse.gov/omb/pubpress/2006/2006-01.pdf

. 4 National Research Council, National Academy of Sciences, Scientific Review of the Proposed Risk Assessment Bulletin from the Office of Management and Budget (2007) [hereinafter 2007 NAS Report on the Proposed Risk Assessment Bulletin].

5 Id. at 6 - 7.

6 OMB received 79 public comments on the Proposed Risk Assessment Bulletin. Thes e comments are posted on OMB’s website, available at http://www.whitehouse.gov/omb/infor eg/comments_rab/list_rab2006.html

. 7 While many of the principles presented in this Memorandum may be relevant to other fields, such as financial or information technology risk analyses, the focus of this Memorandum is on those risk analyses related to environmental, health, and safety risks.

8 The enhancement of information quality, objectivity, tran sparency, and reproducibility is addressed in other OMB guidance as well. See OMB, Circular A-4 for Regulatory Review (2003) [hereinafter Circular A-4] (includes requirements for regulatory analysis as it relates to risk management decisions, particularly those required under Executive Order 12866), available at http://www.whitehouse.gov/omb/circulars/a004/a-4.pdf . In addition, pursuant to what is commonly referred to as the Information Qua lity Act (Sec. 515 of the Treasury and General Government Appropriations Act for FY 2001, Pub. L. No. 106-554), OMB issued government-wide Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies (2002), 67 Fed. Reg. 8452 (Feb. 22, 2002) [hereinafter Information Quality Guidelines], available at http://www.whitehouse.gov/omb /fedreg/reproducible2.pdf ; and its Final Information Quality Bulletin for Peer Review (2004), 70 Fed. Reg.2664 (Jan. 14, 2005) [hereinafter Peer Review Bulletin], available at http://www.whitehouse.gov/omb/memoranda/fy2005/m05-03.pdf . These documents were each issued first as a proposal for public comment. In addition, the Information Quality Guidelines were then implemented by agencies in their own agency-specific information quality guidelines . This Memorandum is intended to complement and support the Information Quality Guidelines.

9 This framework is consistent with the long-standing distinction between risk assessment, risk management, and

risk communication. See National Research Council, National Academy of Sciences, Risk Assessment in the Federal Government: Managing the Process (1983).3

work with Federal agencies to ensure consiste ncy with the principles in this memorandum. Agencies should review their current risk anal ysis practices and guidelines and incorporate these principles as they develo p, update, and issue risk an alyses and guidelines.

Should you have any questions regarding this Memorandum, or would like to discuss it, please do not hesitate to contac t either of us or our sta ff at 202-395-4852 (OIRA) or 202-456- 7116 (OSTP).

General Principles

1. These Principles are intended to be goals fo r agency activities with respect to the assessment, management, and communication of environmental, health, and safety risks. Agencies should recognize that ri sk analysis is a tool—one of many, but nonetheless an important tool—in the regulatory tool kit. These Principles are intended to provide a general policy framework for evaluating and reducing risk, while recognizing that risk analysis is an evolving process and agencies must retain sufficient flexibility to incorporate scientific advances.

The 2007 NAS Report on the Proposed Risk Assessment Bulletin recommended that OMB “develop goals for risk assessment that emphasize the central objective of enhanced scientific quality and the complementary objectives of efficiency and consistency among agencies evaluating the same or similar risks.” 10 Recognizing that “[r]isk assessment is not a monolithic process or a single method,” and that “[a]ll risk assessments share some common principles, but their application varies widely among domains,” the NAS recommended that “affected federal agencies develop their own technical risk assess ment guidelines that are consistent with the OMB general principles.” 11

2. The Principles in this document are inte nded to be applied and interpreted in the context of statutory polici es and requirements, and Administration priorities.

3. As stated in Executive Order No. 12866, “ In setting regulatory priorities, each agency shall consider, to the extent reasonable, the degree and nature of the risks posed by various substances or activities within its ju risdiction” [Section 1( b)(4)]. Further, in developing regulations, federal agencies sh ould consider “...how the action will reduce risks to public health, safety, or the enviro nment, as well as how the magnitude of the risk addressed by the action relates to other risks within the jurisdiction of the agency” [Section 4(c)(1)(D)]. 12

Agencies should refer to Circular A-4 for expanded and updated guidance regarding best practices for agency regulatory analysis. 13

10 2007 NAS Report on the Proposed Risk Assessment Bulletin, supra note 4, at 110.

11 Id. at 106, 110-11.

12 See Exec. Order No. 12866, 58 Fed. Reg. 51,735 (Oct. 4, 1993), available at http://www.whitehouse.gov /omb/inforeg/eo12866.pdf

. See also Exec. Order No. 13132, 72 Fed. Reg. 2763 (Jan. 23, 2007), available at http://www.whitehouse.gov/omb/info reg/eo12866/index_eo12866.html

. 13 See footnote 8 . Circular A-4 refined OMB’s “Economic Analysis of Federal Regulations Under Executive Order 12866” (1996) (a “best practices” guide to preparing a regulatory analysis), available at4

4. In undertaking risk analyses, agencies should establish and maintain a clear distinction between the identification, quantification, and characterization of risks, and the

selection of methods or mechanisms for mana ging risks. Such a distinction, however, does not mean separation. Risk manage ment policies may induce changes in human

behaviors that can alter risk s (i.e., reduce, increase, or change their character), and these linkages must be incorporated into evaluations of the effectiveness of such policies.

5. The depth or extent of the analysis of the risks, benefits and costs associated with a

decision should be commensurate with the nature and significance of the decision.

Subsequent reports have re-affirmed this pr incipal. The 1997 Pres idential/Congressional

Commission on Risk Assessment and Risk Mana gement (referred to as the Presidential Commission on Risk) issued a two-volume report that stated as follows:

The level of detail considered in a risk assessment and includ ed in a risk characterization should be commensurate with the problem’s impo rtance, expected health or environmental impact, expected economic or social impact, ur gency, and level of controversy, as well as with the expected impact a nd cost of protective measures

. 14

A 2007 NAS report on global change assessments eval uated lessons learned from relevant past global change assessments. 15 The NAS, drawing on its analys is and the relevant literature,

identified essential elements of effective a ssessments and included elements relating to the adequacy of funding and the balance between be nefits of the assessment and the opportunity costs of producing it. 16

http://www.whitehouse.gov/o mb/inforeg/riaguide.html . The “best practices” guide wa s issued as guidance in 2000, OMB’s Memorandum for the Heads of Departments and Agenci es on Guidelines to Stan dardize Measures of Costs

and Benefits and the Format of Accounting Statements (2000), available at http://www.whitehouse.gov/omb/memoranda/m00-08.pdf ; and reaffirmed in 2001, OMB’s Memorandum for the President’s Management Counc il on Presidential Review of Agency Rulemaking (2001), available at http://www.whitehouse.gov/omb/inf oreg/oira_review-process.html

. Before finalization, Circular A-4 went through public comment, interagency review, and peer review.

14 Presidential/Congressional Commission on Risk Assessment and Risk Management, Framework for Environmental Health Risk Management , 1 Final Report 25 (1997) [hereinafter Risk Commission Report I].

15 See National Research Council, National Academy of Sciences, Analysis of Global Cha nge Assessments: Lessons

Learned ( 2007) [hereinafter 2007 NAS Report on Global Change Assessments].

16 Id.

, supra note 15, at 4 (two essential elements of effectiv e assessments are “[a]dequate funding that is both commensurate with the mandate and effectively managed to ensure an efficient assessment process” and a “balance between the benefits of a particular assessment and the opportunity costs (e.g., commitments of time and effort) to the scientific community”).5

Principles for Risk Assessment

1. Agencies should employ the best reasonab ly obtainable scientific information to assess risks to health, safe ty, and the environment.

Risk analyses should be based upon the best av ailable scientific methodologies, information, data, and weight of the available scientific evidence. 17 The Presidential Commission on Risk

observed:

Because so many judgments must be based on limited information, it is critical that all

reliable information be considered. Risk assessors and economists are responsible for providing decision-makers with the best technical informa tion available or reasonably attainable, including evaluations of the weight of the evidence that supports different assumptions and conclusions. 18

Congress emphasized using the best available scie ntific evidence for risk information in the 1996 amendments to the Safe Drinking Water Act (SDWA). 19 Pursuant to the SDWA, an agency is directed “to the degree that an agency action is based on science,” to use:

(i) the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and (ii) data collected by accepted methods or best available methods (i f the reliability of the method and the nature of the decision justifies use of the data). 20

Agencies have adopted or adapted this SDWA st andard in their Information Quality Guidelines with regard to the analysis of risks to human health, safet y, and the environment. 21

2. Characterizations of risks and of changes in the nature or magnitude of risks should be both qualitative and quantitative, consistent with available data. The characterizations should be broad enough to inform the range of policies to reduce

risks.

In the 1996 SDWA amendments, Congress adop ted a basic quality standard for the dissemination of public information about risks of adverse health eff ects. Under the 1996

17 In this Memorandum, “scientific” in formation includes information related to applied sciences (such as engineering) and technical inform ation related to these fields.

18 Risk Commission Report I, supra note 14, at 38. The Risk Commission Report I provides examples of the kinds of considerations entailed in making judgments on the basis of the weight of the scientific evidence in a toxicity study: quality of the toxicity study, ap propriateness of the toxicity study methods, consistency of results across studies, biological plausibility of statistical associations, an d similarity of results to responses and effects in humans

. 19 42 U.S.C. § 300g-1(b)(3)(A,B).

20 42 U.S.C. § 300g–1(b)(3)(A).

21 Links to agency specific information quality guidelines can be found in the OMB Draft 2007 Report to Congress on the Costs and Benefits of Federal Regulations, Appendix C, available at http://www.whitehouse.gov/omb/inforeg /2007_cb/2007_draf t_cb_report.pdf . For specific examples of agency guidelines, see http://dms.dot.gov/submit/DataQualityGuidelines.pdf (Department of Transportation guidelines), http://aspe.hhs.gov/infoquality/Guidelines/index.shtml (Department of Health and Human Services guidelines), http://www.sti.nasa.gov/qualinfo.html (National Aeronautics and Space Administration guidelines), http://www.dol.gov/informationquality.htm (Department of La bor guidelines).6

SDWA amendments, the Environmental Protection Agency is directed “to ensure that the presentation of information on public health effects is comprehensive, informative, and

understandable.” 22 The Information Quality Guidelines adapt this language and further direct the agencies to:

in a document made available to the public in support of a regulation [to] specify, to the extent practicable— (i) each population addressed by any es timate [of applicable risk effects]; (ii) the expected risk or central estimate of risk for the specific populations [affected]; (iii) each appropriate upper-bound or lower-bound estimate of risk; (iv) each significant uncertainty identified in the process of the assessment of [r isk] effects and the

studies that would assist in resolving the uncertainty; a nd (v) peer-reviewed studies known to the [agency] that support, are directly relevant to, or fail to support any estimate of [risk] effects and the methodology used to reconcile inconsistencie s in the scientific

data. 23

Agencies have adopted or adapted this standard in their Information Qu ality Guidelines with regard to analysis of risks to human health, safety, and the environment. 24

In addition, the Information Quality Guidelines state that agency disseminations should be objective; these guidelines, as we ll as individual agency inform ation quality guidelines, provide further discussion of objec tivity and its application in agency disseminations. 25 While risks should not be minimized nor exaggerated, the 2007 NAS Report on the Proposed Risk Assessment Bulletin stated that “[i]nformation on the variability of effects across potentially affected populations is e ssential to decision-making.” 26

When characterizing risk in its 1996 report on understanding risk and how it informs decisions, the NAS stated that “quantitative models to orga nize and interpret data ar e particularly important

to risk characterization.” 27 When a risk analysis is influential, 28 increased efforts to provide useful quantitative estimates of risk are particularly important. Du e to the inherent uncertainties associated with estimates of risk, presentati on of a single estimate may be misleading and

22 42 U.S.C. § 300g-1(b)(3)(B).

23 See Information Quality Guidelines, supra note 8.

24 See footnote 21.

25 See Information Quality Guidelines, supra note 8. On a substantive level, objectivity ensures accurate, reliable

and unbiased information.

26 See 2007 NAS Report on the Pr oposed Risk Assessment Bulletin, supra note 4, at 63 (in referring to the Proposed

Risk Assessment Bulletin’s statement that risk assessmen ts should be presented such that they are “neither minimizing nor exaggerating the nature and magnitude of risks,” the NAS stated that this “could, however, degrade risk analysis if it were interpreted so as to depriv e decision-makers of important information on sensitive subpopulations on the grounds that such information may generate risk estimates considerably higher than a central tendency or general population estimates.”).

27 National Research Council, Na tional Academy of Sciences, Understanding Risk: Informing Decisions in a Democratic Society 99-100 (1996) [hereinafter 1996 NAS Report on Understanding Risk and Informing Decisions].

28 In this Memorandum, the term “influential” is defined in the same way as it is defined in the Information Quality Guidelines. Influential means that “the agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on importa nt public policies or important private sector decisions.” In their information quality guidelines, agencies have defined “influential” in ways appropriate for them given the nature and multiplicity of issues for which the agencies are responsible.7

provide a false sense of precision. 29 Expert panels agree that when a quantitative characterization of risk is provi ded, a range of plausible risk estimates should be provided. 30
When something more than a superficial anal ysis can be conducted, quantitative uncertainty analysis, sensitivity analysis, and a discussion of model uncertainty can greatly inform risk management decisions. 31

Experts have recognized that when presenting ri sk information qualitatively, or quantitatively, agencies’ methodological approaches will likely vary and will depend upon the context for which the analysis is used. 32 These methodologies are continuing to develop as the science associated with quantitative uncertainty analysis advances. As technical guidance continues to develop, so too should agenci es’ presentation of quantitative risk information.

29 See National Research Council, National Academy of Sciences, Models in Environmental Regulatory Decision Making 136 (2007) [hereinafter 2007 NAS Report on Environmental Decision Making] (“there are substantial problems in reducing the results of a large-scale study with many sources of uncertainty to a single number or even a single probability distribution. We contend that such an approach draws the line between the role of analysts and the role of policy makers in deci sion making at the wrong place.”); Id. at 7 (“Effective decision making will require providing policy makers with more than a single probability distribution for a model result (and certainly more than just a single number, such as the expected net benefit, w ith no indication of uncertainty ). Such summaries obscure

the sensitivities of the outcome to individual sources of uncer tainty, thus undermining the ability of policy makers to make informed decisions and constraining the efforts of stakeholders to understand the basis for the decisions.”).

See also 1996 NAS Report on Understanding Risk and Informing Decisions, supra note 27, at 66 (“Risk characterizations often fail because they a ttribute meaning to scientific estimates in ways that mislead participants in the risk decision process or ar e incomprehensible to them.”).

30 See 2007 NAS Report on the Pr oposed Risk Assessment Bulletin, supra note 4, at 37 (“The committee agrees with OMB that in some cases ‘presentation of single estimat es of risk is misleading’ and that ranges of ‘plausible risk’ should be presented; however, the challenge is in the operational definitions of such words as central

, expected

, and plausible .”) (emphasis in original).

31 See 2007 NAS Report on Environmental Decision Making, supra note 29, at 6 (“A wide range of possibilities is available for performing model uncertainty analysis. At one extreme, all model uncertainties could be represented probabilistically, and the probability distribution of any model outcome of interest could be calculated. However, in assessing environmental regulatory issues, these analys es generally would be quite complicated to carry out convincingly, especially when some of the uncertainties in critical parameters have broad ranges or when the parameter uncertainties are difficult to quantify. Thus, a lthough probabilistic uncertainty analysis is an important tool, requiring EPA to do complete probabilistic regulatory analyses on a routine basis would probably result in superficial treatments of many sources of uncertainty. The practical problems of performing a complete probabilistic analysis stem from models that have large numbers of parameters whose uncertainties must be estimated in a cursory fashion. Such problems are compounded when models are linked into a highly complex system, for example, when emissions and meteorological model results are used as inputs into an air quality model.”); Id. at 7 (“It is not necessary to choose between purely probabilistic approaches and deterministic approaches. Hybrid analyses combining aspects of probabilistic and deterministic approaches might provide the best solution for quantifying uncertainties, given the finite resources available for any analysis. For example, a sensitivity analysis might be used to determine which mode l parameters are most likely to have the largest impacts on the conclusions, and then a probabilistic analysis could be used to quantify bounds on the conclusions due to uncertainties in those parameters.”).

32 Id. at 7-8 (“In some cases, presenting results from a sm all number of model scenarios will provide an adequate uncertainty analysis (for example, cases in which the stakes are low, modeling resources are limited, or insufficient information is available). In many instances, however, probab ilistic methods will be necessary to characterize properly at least some uncertainties and to communicate clearly the overall uncertainties.”).8

3. Judgments used in developing a risk assessm ent, such as assumptions, defaults, and uncertainties, should be stated explicitly. The rationale for these judgments and their

influence on the risk assessment should be articulated.

If important judgments are supported by, or conflic t with, empirical data, that information should

be discussed. The discussion s hould address the range of scientif ic and/or technical opinions regarding the likelihood of plausible alternate judgments and the direction and magnitude of any resulting changes that might arise in the analysis due to changes in key judgments. 33 Every effort should be made to perform a quantit ative evaluation of r easonable alternative assumptions. 34 When an analysis combines multiple assumptions, the basis and rationale for combining the assumptions should be explicitly described. 35

Critical judgments are often made when choosi ng and presenting study resu lts. Results based on

different effects and/or different studies should be presented to convey how the choice of effect and/or study influences the an alysis. The presentation of in formation regarding different scientifically plausible endpoints should allow for a robust discussion of the available data, associated uncertainties, and underlying science. In its 2007 report eval uating global change assessments, the NAS recommended that “[t]here should be a deliberate effort to clarify the importance of alternative assumptions and to illustrate the impacts of uncertainties.” 36 When relying on data from one study ove r others, the agency should provi de a clear rationale and/or

scientific basis for its choice. 37

4. Risk assessments should encompass all appropriate hazards (e.g., acute and chronic risks, including cancer and non-cancer risk s, to human health and the environment). In addition to considering th e full population at risk, attention should be directed to subpopulations that may be particularly su sceptible to such risks and/or may be more highly exposed.

A good risk analysis should cl early summarize the scope of the assessment, including a description of: the agent, technology and/or activity that is the subj ect of the analysis; the hazard of concern; the affected entities (populations, subpopulations, individuals , natural resources, ecosystems, critical infrastructure, or othe r) that are the subject of the assessment; the exposure/event scenarios relevant to the objectives of the asse ssment; and the type of event-

33 See 1996 NAS Report on Understanding Risk and Informing Decisions, supra note 27, at 53 (“Simplifying assumptions generate especially seriou s problems when some of the assumptions are unreasonable in the face of information available to people outside the analytical process.”).

34 Id. at 99 (“Without good analysis, deliberative processe s can arrive at agreements that are unwise or not feasible.”). See also National Research Council, National Academy of Sciences, Health Risks from Dioxin and Related Compounds 7 (2006) (“Although EPA [in its dioxin assess ment] addressed many sources of variability and uncertainty qualitatively, the committee noted that the Rea ssessment would be substantially improved if its risk characterization included more quantitative approaches. Failure to characterize variability and uncertainty thoroughly can convey a false sense of precision in the conclusions of the risk assessment.”).

35 See Circular A-4, supra note 8, at 39 (“Inferences and assumptions used in your analysis should be identified, and your analytical choices should be explicitly evaluated and adequately justified.”).

36 See 2007 NAS Report on Global Change Assessments, supra note 15, at 126.

37 See Circular A-4, supra note 8, at 39.9

consequence or dose-response relati onship for the hazard of concern. 38 In the 2007 NAS Report on the Proposed Risk Assessment Bulletin, the NAS reaffirmed that including this information would improve the clarity of a risk analysis a nd is consistent with the recommendations of previous expert reports. 39

More recent reports have reaffirmed that, in addi tion to considering the fu ll population or entities at risk, the risk analysis s hould also consider subpopulations or sub-entities that may be particularly susceptible to such risks and/or may experience greater exposures. 40 If a risk analysis is to address only sp ecific subpopulations, th e scope should be very clear about this

limitation.

Where there are known differences in risk fo r different individuals, subpopulations or ecosystems, analysts should characterize this vari ability. Risk managers will be better informed when an understanding of variability and the key c ontributors to the cause of this variability are presented in the risk analysis. 41 As guidance on the presentation of risk information to the public continues to develop, so too will agencies’ presentation and discussion of variability.

The President’s Commission on Risk stated: “A good risk management decision is based on a careful analysis of the weight of scientific evidence th at supports conclusions about a problem’s potential risks to human h ealth and the environment.” 42 This may include consideration of both positive and negative studies, in light of each stud y’s technical quality. Agencies and the risk assessment community are continuing to develop tec hniques for weight of evidence evaluations.

Agencies should consider confounding and/or synerg istic factors. The sc ientific process of considering these elements may assist policy make rs in developing a broader sense of how risk can be reduced significantly and th e range of decision options that need to be considered in developing risk management approaches. 43

38 See Risk Commission Report I, supra note 14, at 24 (a list of questions should be addressed in a risk characterization).

39 See 2007 NAS Report on the Pr oposed Risk Assessment Bulletin, supra note 4, at 62.

40 See Risk Commission Report I, supra note 14, at 24 (a list of questions that includes asking about individuals or groups that are at risk and if some people are more likely to be at risk than others).

41 See 2007 NAS Report on the Pr oposed Risk Assessment Bulletin, supra note 4, at 80 (“information on the variability of effects across potentially affected populations--due to differences in sensitivity, exposure, or both--is essential to decision-making.”).

42 See Risk Commission Report I, supra note 14, at 23; Id. at 23-24 (“It is important th at risk assessors respect the objective scientific basis of risks and procedures for making inferences in the absence of adequate data. Risk assessors should provide risk managers and other stakeholders with plausible conclusions about risk that can be made on the basis of the available information, along with evaluations of the scientific weight of evidence supporting those conclusions and descriptions of majo r sources of uncertainty and alternative views.”).

43 Id. at 24 (a list of questions that includes asking about “[w]hat other sources cause the same type of effects or risks.”).10

5. Peer review of risk assessments can ensure that the highest professional standards are maintained. Therefore, agencies should develop policies to maximize its use.

Agencies should refer to the P eer Review Bulletin for updated gui dance regarding agency best practices for peer review. 44

6. Agencies should strive to adopt consistent approaches to evaluating the risks posed by hazardous agents or events.

Principles for Risk Management

1. In making significant risk management decisions, agencies should analyze the

distribution of the risks and the benefits and costs (both direct and indirect, both quantifiable and non-quantifiable) associated with the selection or implementation of risk management strategies. Reasonably feas ible risk management strategies, including regulation, positive and negative economic incentives, and other ways to encourage behavioral changes to reduce risks (e.g., information dissemination), should be evaluated. Agencies should employ the be st available scientific, economic and policy analysis, and such analyses should include explanations of significant assumptions, uncertainties, and methods of data development.

Agencies should refer to Circular A-4 for update d guidance regarding agency best practices for regulatory analysis. 45

2. In choosing among alternative approaches to reducing risk, agencies should seek to offer the greatest net improvement in tota l societal welfare, accounting for a broad range of relevant social and economic consid erations such as e quity, quality of life, individual preferences, and the magnitude and distribution of benefits and costs (both direct and indirect, both quantif iable and non-quantifiable).

Agencies should refer to Circular A-4 for update d guidance regarding agency best practices for regulatory analysis. 46

Principles for Risk Communication

1. Risk communication should involve the open, two-way exchange of information between professionals, including both policy makers and “experts” in relevant disciplines, and the public.

In describing its approach to risk charact erization in the 1996 NAS Report on Understanding Risk and Informing Decisions, NAS stated that “[t]he responsible orga nization’s staff should describe the stated and implicit purposes of the decision-making activity, the type of decision and general aims furthered by the activity, and th e intended users of the risk characterization.” 47

44 See Peer Review Bulletin, supra note 8.

45 See Circular A-4, supra note 8.

46 Id.

47 See 1996 NAS Report on Understanding Risk and Informing Decisions, supra note 27, at 146; Id. (“Different types of decisions may require different types of knowledge and perspectives and hence require different participants in the analytic-deliberative process--both inside and outside the organization.”).11

Agencies should provide this information as it will assist the readers and users in better understanding the questions that the analysis sought to answer and will help to ensure that the risk analyses are used for their intended purposes. This information is particularly important in cases where likely users of the risk analyses are not the original intended audience for the

document.

2. Risk management goals should be stated clearly, and risk assessments and risk management decisions should be communi cated accurately and objectively in a meaningful manner.

An executive summary that discloses the objectives and scope, the key findings of the analysis, and the key scientific limitations and uncerta inties can be an important part of risk

communication. 48 In the 2007 NAS Report on the Proposed Risk Bulletin, the NAS commented that the inclusion of an executive summary c ould improve the clarity of risk assessments. 49
Presentation of information in a helpful and c oncise introductory section of the report will not only foster improved communication of the findings, but will also help ensure that the risk analysis is appropriately interp reted by diverse end users.

To maximize public understanding and particip ation in risk-related decisions, agencies

should:

a. explain the basis for significant assumpti ons, data, models, and inferences used or relied upon in the assessment or decision;

A high degree of transparency with respect to data, assumptions, and methods will increase the credibility of the risk analysis, and will allow inte rested individuals, internal and external to the agency, to understand better the technical basis of the analysis. 50

b. describe the sources, extent and magnitude of significant uncertainties associated with the assessment or decision;

In the 1996 NAS Report on Understanding Risk and Informing Decisions, the NAS observed that “[t]here is strong agreemen t that risk analysts should expl icitly summarize uncertainty, and

48 Key limitations are those that are most likely to affect significantly the dete rminations and/or estimates of risk presented in the analysis.

49 See 2007 NAS Report on the Pr oposed Risk Assessment Bulletin, supra note 4, at 64 (“These qualitative standards . . . [presented in an executive summary] could improve the clarity of risk assessment in the federal government if risk assessments do not implement them already (although the existence of such problems is not established by the bulletin.”)).

50 See 1996 NAS Report on Understanding Risk and Informing Decisions, supra note 27, at 67; Id. at 100 (in a good quantitative analysis, “[a]ny assumptions used are clear ly explained, used consistently, and tested for reasonableness,” “[c]alculations are presented in such a fo rm that they can be checked by others interested in verifying the results,” and “[u]ncertainties are indicated, including those in data, models, parameters, and calculations.”). See 2007 NAS Report on Global Change Assessments, supra note 15, at 126 (NAS recommends that “[t]here should be a deliberate effo rt to clarify the importance of alternative assumptions and to illustrate the impacts of uncertainties.”). See also U.S. Environmental Protection Agency, Science Policy Council Handbook,
Risk Characterization 15 (2000) [hereinafter EPA Risk Characterization Handbook] (“[Transparency] ensures that any reader understands all the steps, logic, key assumptions, limitations, and decisions in the risk assessment, and comprehends the supporting rationale that lead to the ou tcome.”). In the EPA Risk Characteriza tion Handbook,

EPA lists 10 elements of full disclosure. Id. at 15-16.12

there are methods for doing so.” 51 In the 2007 NAS Report on Global Change Assessments, when referring to critical elements of a credib le, legitimate and salient assessment, the NAS

included as one of four central elements:

Deliberative and consistent methods of tr eating and communicati ng uncertainties add credibility and salience. Rega rdless of method (statistics, sensitivity analysis, scenario development, or expert judgment), each m easure must be defined and communicated in a

consistent manner. 52

In the same report the NAS also stated:

An effective characterization of uncertainty in assessments requires determining what sorts of uncertainty information would be useful for decision makers as well as developing quantitative or qualita tive measures of uncertainty . . . . The manner in which uncertainties are acknowledged and characterized will affect both the salience and credibility of the assessment.” 53

The agency also should identify the sources of the underlying informa tion (consistent with sensitive information and confidentiality protecti ons) and the supporting data and models, so that the public can evaluate whet her there may be some reas on to question objectivity. 54 Data should be accurately documented, and error sources aff ecting data quality should be identified and

disclosed. 55

c. make appropriate risk comparisons, taking into account, for example, public attitudes with respect to voluntar y versus involuntary risk; and,

When making risk comparisons, agencies should be careful to consider the perspectives, assumptions, attitudes and context that the public associates with each risk. Agencies may want

to consult the risk communicat ion literature when considering appropriate comparisons. Although the risk assessor has considerable latitude in making ri sk comparisons, the fundamental point is that risk should be placed in a contex t that is useful and relevant for the intended

audience. 56 Furthermore, effective communication of risk information can assist the public in

51 See 1996 NAS Report on Understanding Risk and Informing Decisions, supra note 27, at 107 (“When uncertainty is recognizable and quantifiable, the language of probability can be used to describe it.”). See also EPA Risk Characterization Handbook, supra note 50, at 41 (“While it is generally preferred that quantitative uncertainty analyses are used in each risk characterization, there is no single recognized guidance that currently exists on how to conduct an uncertainty analysis. Nonetheless, risk a ssessors should perform an uncertainty analysis . . . . Uncertainty analysis should not be rest ricted to discussion of precision and accuracy, but should include such issues as data gaps and models.”).

52 See 2007 NAS Report on Global Change Assessments, supra note 15, at 60.

53 Id . at 126.

54 See Information Quality Guidelines, supra note 8.

55 See OMB, Statistical Policy Working Paper No. 31, Measuring and Reporting Sources of Error in Surveys, 1-5 (2001) (“The measuremen t and reporting of error sources is important for everyone who uses statistical data. For the analyst, this information helps data analyses thr ough an awareness of the limitations of the data.”), a vailable at http://www.fcsm.gov/01papers/SPWP31_final.pdf

. 56 See National Research Council, National Academy of Sciences, Improving Risk Communication 165-79 (1989).

See also Risk Commission Report I, supra note 14, at 4 (the problems a regulation is intended to address should be placed in their “public health and ecological context.”).13

balancing benefits and risks. 57 As our understanding regard ing the presentation of risk information to policy makers and the public continues to develop, so too will agencies’ presentation and discussion of risk comparison information.

d. provide timely, public access to relevan t supporting documents and a reasonable opportunity for public comment.

Agencies should refer to OMB’s Final Bulletin for Agency Good Guidance Practices, as well as the Peer Review Bulletin, for updated guidance regarding best practices for increasing public access and public comment concerning guidan ce documents and influential scientific

information. 58 In addition, as noted in the OMB Inform ation Quality Guidelines, influential risk analyses should be reproducible. 59 For guidance on how to provide the public with timely access to government information, Agencies sh ould refer to OMB’s Circular A-130, which addresses the management of Federal information resources, and OMB Memorandum 06-02, which addresses improving public access to Federal information. 60

Principles for Priority Se tting Using Risk Analysis

1. To inform priority setting, agencies should seek to compare risks, grouping them in broad categories of concern (e.g., high, moderate, and low).

2. Agencies should set priorities for managing risks so that those actions resulting in the greatest net improvement in societal welfa re are taken first, accounting for relevant management and social considerations such as different types of health or environmental impacts; individual preferences; the feasibility of reducing or avoiding risks; quality of life; environmental justice; and the magnitude and distribution of both short- and long-term benefits and costs.

3. The setting of priorities should be inform ed by internal agency experts and a broad range of individuals in state and lo cal government, industry, academia, and nongovernmental organizations, as well as the public at large. Where possible, consensus views should be reflected in the setting of priorities.

4. Agencies should attempt to coordinate risk reduction efforts wherever feasible and appropriate.

57 See Institute of Medicine, National Academy of Sciences, Seafood Choices: Balancing Benefits and Risks 207 (2006) (“In the committee’s judgment, it is important to conduct substitution analyses of the potential impacts of changes in consumption despite the uncertainties about the underlying nutrient and contamination levels.”). Id. at 231 (in referring to guidance and information that can si mplify such tradeoffs, the NAS lists considerations for guidance development).

58 See OMB, Final Bulletin for Agency Good Guidance Practices (2007), available at http://www.whitehouse.gov/omb/fedreg/2007/012507_good_guidance.pdf , and Peer Review Bulletin, supra note 8.

59 See Information Quality Guidelines, supra note 8, 67 Fed. Reg. at 8460 (independent reanalysis of the original or supporting data using the same methods would generate simila r analytical results, subject to an acceptable degree of precision).

60 See OMB, Circular A-130, Management of Federal Information Resources (2000), available at http://www.whitehouse.gov/omb/c irculars/a130/a130trans4.pdf ; OMB Memorandum No. 06-02, Improving Public Access to and Dissemination of Government Information and Using the Federal Enterprise Architecture Data Reference Model (2005), available at http://www.whitehouse.gov/omb/memoranda/fy2006/m06-02.pdf

.