SCIENCE IN REGULATION:
A STUDY OF AGENCY
DECISIONMAKING APPROACHES

Wendy Wagner
University of Texas School of Law
Feb. 18, 2013
This report was prepared for the
consideration of the Administrative
Conference of the United States. The views
expressed are those of the author and do not
necessarily reflect those of the members of
the Conference or its committees.

Science in Regulation:
A Study of Agency Decisionmaking Approaches
Wendy Wagner, University of Texas School of Law *
Final Draft: February 18, 2013

Executive Summary
This Executive summary sets out the focus of the study, the methods used, the
general findings, and outlines recommendations.
Purpose and Focus of the Study
For the last three decades, federal agencies have been criticized for not being clear
about the role that science played in their decision-making processes. In response to
these criticisms, a number of efforts have been made by the Executive Branch and
Congress to provide mechanisms that attempt to shore up the quality of the scientific
process undergirding agency decision-making. Most recently, in 2009 President Obama
issued a memorandum to the agencies directing that “To the extent permitted by law,
there should be transparency in the preparation, identification, and use of scientific and
technological information in policymaking.” At base, these initiatives demand
heightened transparency of the agencies’ use of science, a demand that is central to
ensuring the basic accountability of agency regulation.
This study examines the basic decision-making processes employed by agencies
in science-intensive regulatory projects. Illuminating basic agency decision processes
helps provide greater transparency about the agencies’ use of science since these
processes reveal the points at which agencies look to external peer reviewers, the public,
and other entities for critical feedback and advice. The elucidation of agency decisionmaking processes – essentially re-creating the agencies’ underlying flow charts for how
they integrate science into regulation -- also provides information on how the agencies
assemble and assess the available technical evidence relevant to a decision, how the
agencies integrate that evidence into a larger policy decision, and how they explain their
work to the larger public and scientific community. Indeed, precisely because the line
between science and policy is blurred, it is critically important that the underlying
decision-processes themselves be both analytically robust and transparent. It is no
coincidence, in fact, that many of the worst examples of politicized science have occurred
in settings where agency decision processes lacked both transparency and rigor.
*

This report was prepared for the consideration of the Administrative Conference of the United States. The
views expressed are those of the author and do not necessarily reflect those of the members of the
Conference or its committees.

1

Methods
This study of the agency’s integration of science into regulation examines the
processes used by three agencies (namely the Environmental Protection Agency, the Fish
and Wildlife Service, and the Nuclear Regulatory Commission) in five different
regulatory programs. Some of these regulatory programs are regarded as exemplary
while others are viewed as problematic; the diversity in the programs thus offers a rich
basis for understanding how agencies integrate science into their decision-making
processes. Moreover, while the programs reviewed in this study are not comprehensive
and do not cover the full spectrum of possible agency approaches, the breadth of
programs reviewed is sufficiently broad to offer a preliminary glimpse of some of the
various challenges and creative adjustments that agencies have made in their integration
of science into regulation.
Findings
The study concludes that the agencies have made considerable strides in ensuring
the rigor and transparency of their integration of science into regulation. For example,
some agencies explain how policy and science intersect in their regulatory projects in
sophisticated, yet accessible ways. Likewise, some agencies have established processes
that ensure both expert and internal review of their work, and, in connection with these
processes, are providing a public record of the changes they make in response to peer
review and public comment. Some agencies have also established integrity policies that
allow their staff to raise scientific differences with supervisors through various informal
and formal mechanisms. Finally, a number of agencies increasingly use the Internet to
post the reference list used in their decision-making and even to make copies of
documents consulted during that process readily available through databases and
hyperlinks.
In addition to identifying these positive practices, a series of recommendations are
developed for four separate areas of agency processes. First, the study recommends that
innovative practices for incorporating science into the regulatory process be shared and
that best practices be established. The study finds that while some agencies are
innovative in their use of science, little of this innovation is recorded or shared with other
agencies. In addition to recommending that information about innovative practices be
systematically collated and disseminated, the report also identifies a number of “best
practices” for integrating science into regulation as well as a number of other innovative
approaches that deserve further study and experimentation. In identifying these best
practices, the report also underscores that because there is such a wide variation in
regulatory projects, a one-size fits-all approach to the integration of science in regulation
is ill-advised.

2

Second, the study exposes a number of external constraints on agency decisionmaking processes that limit the ability of the agencies to improve their decision processes
in keeping with the President’s scientific integrity initiative. A second set of
recommendations suggests that these various external constraints should be tracked more
vigorously to identify their impact on agency science-intensive projects and ultimately
reformed, if needed.
Third, because agencies necessarily must rely on scientific research produced by
regulated parties to support in a number of health and environmental program, the study
recommends that this privately produced research be subjected to the same types of
requirements imposed on research published in scientific journals and on research
produced with federal monies. At present this research is not subject to a vigorous
oversight processes.
Finally, the investigation highlights the need for additional study of regulatory
science in the future. The agencies’ use of external peer reviewers, for example, is vital
to ensuring that the agencies’ integration of science into policy is both rigorous and
transparent, yet very little is understood about this feature of agency decision-making.
Indeed, just in this study alone, external peer reviewers were employed by the agencies in
very different roles (e.g., collaborative and iterative versus providing a one-shot technical
evaluation of a decision), were utilized at different points in the decision-making (e.g.,
early, continuous, or at the end), and in different ways (e.g., individual reviewers versus
panels). Without some appreciation of the breadth or variety of the use of external exert
reviewers, it is difficult to identify the most effective process moving forward. The
agencies’ variable use of external peer reviewers as well as other topics are recommended
for future study in the final section of the report.
Specific Recommendations
Specific recommendations emerging from the research in the report are provided
below.
I.

Capitalizing on Agency Successes

A. Sharing Information
1. Agencies should provide the public with an accessible description of the process
that they utilize for integrating science into their decisions for each of their
science-intensive programs. This includes a statement of how an agency
evaluates the scientific information used in its analysis; how the agency makes
that information available to reviewers and the public; how the analysis is
reviewed by experts and interested parties; and how the agency ensures that the
final decision can be compared against the scientific record.
3

2. OSTP, a standing NAS Committee, or some other body should take responsibility
for identifying and publicizing the innovations developed by agencies for
transparently incorporating science into their regulatory decisions.
B. Best Practices
3. In supporting its science-based regulatory decision, an agency should identify and
make publicly available a list of the scientific literature it consulted, including
even the literature it rejected when it is material to the scientific analysis, as well
as the literature it relied upon. This reference list should be posted online
whenever possible.
When an agency relies on studies that are not published, it should post the
studies on its website as soon as is practicable, subject to copyright and other
legal restrictions. When this public transparency is not possible, these restrictions
should be explained in the agency’s individual analyses and possibly more
generally in describing its regulatory program for the public.
4. Consistent with President Obama’s directive, agencies should be encouraged, not
impeded, from having their scientific analyses reviewed by other experts, even if
this oversight occurs wholly inside the agency. Any limitations on an agency’s
ability to have scientific work reviewed by scientific experts should be actively
discouraged. Additionally and when possible, agencies should endeavor to
explain how they ensured the rigorous review of their scientific products for each
regulatory project.
5. Agencies should resist applying deliberative process protections to documents and
communications that influenced the development of science-based regulatory
projects. To the extent agencies do invoke the deliberative process privilege, they
should justify so doing with respect to each document that is withheld from the
public. Draft science-policy analyses, such as draft papers, can be made public
with the disclaimer that they do not necessarily represent the policy or scientific
position of the agency. Agencies should prepare an administrative record that
advances this transparency goal by ensuring that the documents, meetings, and
other deliberations that resulted in potentially significant changes to scientific
assumptions or interpretations are made part of the administrative record. These
administrative records should be posted on the internet when possible.

4

C. Innovations that Should be Considered by Agencies but that are Currently too
Novel to be regarded as Best Practices
6. All significant science-policy choices made by an agency in reaching a decision
should be identified and explained in clear and understandable terms. In order to
provide this heightened level of transparency, agencies should consider following
an analytical process that: a) identifies the policy-relevant questions that can be
informed by science; b) identifies in advance a study design, such as criteria for
weighting individual studies, as well as identifying other a priori analytical
choices, like stopping rules; c) provides a synthesis of the available evidence and
relevant literature guided by this study design; and d) identify other significant
assumptions, choices of analytical techniques, and remaining uncertainties and
how different plausible choices might change the resulting policy decision. If
possible, the agency should also follow the model of the NAAQS policy
assessment in bridging science and policy in a final report, although this final step
will likely involve more effort and experimentation.
Making these analytical steps explicit may not be practicable in some
science-policy decisions and may not be practicable in other regulatory settings.
This recommendation simply encourages agencies to consider this staged
approach in their processes. Ultimately, with experience, this analytical approach
may develop into a best practice. Until then, agencies are strongly encouraged to
consider this analytical approach in conducting their work.
7. In regulatory settings, particularly in cases when agencies are not bound by
judicially enforceable deadlines, the agencies should be encouraged to establish
explicit stopping rules on regulatory projects, both with regard to when they will
close their consideration of emerging research and when they chose to close
currently unresolvable scientific debate in order to reach a decision. External peer
review bodies may be particularly useful to agencies in establishing scientifically
credible points at which debate should cease.
8. For science-intensive rules, an agency should identify specific types of future
research projects that will advance understanding on the regulatory issue. This
identification of research questions and priorities should influence the agency’s
research agenda as well as provide a benchmark for measuring regulatory
progress. While agencies may need to experiment with when and how they
incorporate “research checkpoints” into their regulatory analysis, the most fruitful
time may occur at the point the record is closed, as determined by stopping rules.
9. Agency staff plays an important role in producing the agency’s analyses. When
practicable and appropriate, agency managers should consider providing staff
5

with some form of consensual authorship right or attribution for reports or
analyses to which they contribute in a significant way. All staff authors who
contributed in a significant way to a technical or scientific report should be
acknowledged, including economists, lawyers, and other nonscientists. In a
similar vein, reviewers and other contributors should also be identified by name
and general contribution.
10. Agencies should encourage vigorous debate among agency scientists that may
include developing written policies that allow agency staff to dissent or express
their non-concurrence on a technical analysis to which they contributed. In cases
where written dissents are permitted, agency managers should take seriously a
staff member’s request to place a dissent or non-concurrence into the public
record. Dissenting employees should also be allowed and encouraged to publish
these dissenting positions in the peer reviewed literature, provided that
confidential governmental deliberations are not compromised. In all cases and
regardless of the public availability of these discussions, dissenting staff members
should be protected from reprisals.
II.

External Constraints and Influences on Science-Based Regulation

A. External Constraints
11. There are statutory and regulatory constraints that limit the ability of the agencies
to ensure that their decisions are scientifically robust and transparent in keeping
with the President’s Directive. The agencies should identify these legal barriers
that impede public access to the scientific information underlying agency analyses
or otherwise block the agencies’ development of scientifically robust decisionmaking processes and OSTP or other centralized agencies can take responsibility
for collecting information and proposing government-wide recommendations.
B. OIRA Review
12. Under Section 2(a) and 6(a) of Executive Order 12866, the agencies are
responsible for
interpreting and complying with Section 6(a). The agencies’ compliance under
Section 6(a) should include at the very least:
1) documentation of the major changes made at the suggestion or
recommendation of OIRA at any point in the lifecycle of the regulation as
required by Section 6(a)(3)(E)(iii) and 6(a)(3)(F). If there are no major changes,
then the agency should provide a statement to that effect;
2) an identification of all substantive changes made between the draft
submitted to OIRA for review and the action subsequently announced in
6

compliance with Section 6(a)(3)(F)(ii). This includes but is not limited to a redlined version of the document undergoing OIRA review;
3) for both #1 and 2, the agencies should provide a “complete, clear, and
simple” identification and explanation of each major change in “plain,
understandable language” for the public. Explication of these major changes
should be accessible to the public – through for example a cover memorandum --and not buried in hundreds of pages of red-lined documents. Although the
Executive Order technically requires this accessible explication of all changes
(and not simply the major changes) made at the suggestion of OIRA, a disclosure
of the major changes is considerably less burdensome and appears consistent with
the thrust of the Executive Order;
4) a library of all significant, substantive documents exchanged between
OIRA and the agency throughout the life cycle of the regulatory action to ensure
that the agency is in full compliance with Section 6(a)(3)(E)(ii) and (iii).
5) centralized public access to the information specified above to ensure
practical, as opposed to merely theoretical, compliance with the general
requirements of Section 6(a)(3)(E) and (F). Both reginfo.gov and regulations.gov
should link to or provide the public with document libraries that enable simple
access to and searching of documents required under Section 6.
Agencies should apply these same requirements of Section 6(a)(3)(E), as
interpreted above, to all significant science-intensive regulatory actions, including
agency guidances and other standards and policies, whether or not they are
published in the Federal Register, as well as to all significant, supporting studies
and projects that inform science-intensive agency rules, guidances, policies and
related products. The requirements should also apply to all rules that are
withdrawn, whether ORIA has reviewed them or not.
C. Changes made at the Suggestion of other Agencies and White House Offices
13. The agency should disclose material changes made at the suggestion or
recommendation of White House offices or other agencies, consistent with
Section 6(a)(3)(E)(iii), when doing so does not impair the deliberative processes.
III.

Enhancing the Transparency of Research produced by Regulated
Parties

14. Agencies should require conflict disclosures on all future research submitted to
inform an agency’s licensing, regulatory, or other decision-making process -whether published and unpublished. This conflict of interest disclosure should be
similar to the conflict of interest disclosure required by the biomedical journals.
See See ICJME Standards. The regulatory conflict of interest disclosure should
7

also, where possible, identify whether a sponsor reserved the right to participate in
the design of the research; the collection of data; the interpretation of data; the
writing or disseminating the report, or any other material features of the research.
Finally, “[a]gencies and scientific advisory committees should be extremely
skeptical of a scientific study unless they are sure that the principal investigator(s)
(as opposed to the sponsor or funder) had ultimate control over the design and
publication of the study.”*
15. “Studies used in the formulation of regulation should be subject to data access
requirements equivalent to those under the Data Access Act (Shelby Amendment)
and its implementing circular (OMB Circular A-110) regardless of who funded
the study. If a study is used by an agency to inform the development of a
regulation, then the same kinds of information about that study should be
available upon request, regardless of whether the study was funded by the federal
government, industry, or some other entity.”**
16. “Confidential Business Information (CBI) claims can . . . make it difficult for the
interested public to evaluate studies that contribute to regulatory policy.” ***
Agencies that provide CBI protections for studies or data that inform regulation
should ensure that the CBI claims are justified. Given the strong incentives to
regulated parties for overclaiming CBI protection and the resultant costs from this
overclaiming to public health protection and research, it is important that the
agencies’ CBI programs not provide a safe haven for unjustified suppression of
relevant regulatory research. To that end and as a first step, the agencies should
review their CBI programs to ensure that there is rigorous oversight of CBI and
related trade secret claims on health and environmental research. Agencies
should, where possible, penalize those CBI claims that, upon review, appear
unjustified.

*

Recommendation copied verbatim from BiPartisan Policy Center, Improving the Use of Science in
Regulatory Policy 42 (Aug. 2009).
**
Id. at 43.
***
Id.

8

IV.

Specific Issues Recommended for Further Research

17. There are a number of topics recommended for research. They include:
 The role(s) of advisory groups and external peer reviewers
 Challenges to evaluating the reliability of data and studies submitted
by applicants, particularly studies that are unpublished and/or
otherwise publicly inaccessible
 The role of expert elicitation
 Agency outreach and decision-making during the pre-NPRM Stage
 The privileged role of science in regulation

9

Table of Contents
I.

Background: The Illusive Line between Science and Policy ............................................... 15

II.

Methods ............................................................................................................................ 19
A.

Decision Processes as a Diagnostic Tool for Studying the Agencies’ Use of Science ...... 19

B.

Methods ........................................................................................................................ 21
1.

Identifying the Types of Science-based Regulatory Projects most in need of Study..... 22

2.

Which Agencies ......................................................................................................... 22

3.

Which Programs ........................................................................................................ 23

4.

What Factors.............................................................................................................. 25

5.

Sources of information about the Agency Decision-making Processes ........................ 27

III.

Findings ........................................................................................................................ 28

A.

The Incorporation of Science into Specific Regulatory Programs ................................... 29
1.

The Environmental Protection Agency (EPA) ............................................................ 29

2. The Department of Interior’s Fish and Wildlife Service: Listing Endangered and
Threatened Species and Designating Critical Habitat ......................................................... 57
3. A Birds’ Eye View of the Nuclear Regulatory Commission’s Incorporation of Science
into its Regulatory Decision-making .................................................................................. 65
4. Review by the Office of Management and Budget ......................................................... 75
B.

Agency Efforts to Improve the Scientific Integrity of their Processes ............................. 91
1.

Scientific Integrity Policies in response to the White House Initiative ......................... 92

2.

NRC’s Procedures to Enhance the Scientific Integrity of its Decisions and its Staff .... 97

C.

Summary of the Findings ............................................................................................. 104

IV.

Analysis and Recommendations................................................................................... 106

A.

Capitalizing on Agency Successes ............................................................................... 108
1.

Sharing Information on Agency Approaches ............................................................ 108

2.

Developing a Set of Best Practices for Science-Intensive Regulation ........................ 111

3.

Agency Innovations that may develop into Best Practices with Experience ............... 119

B. Interagency and External Constraints on Agencies that might impair the Rigorous and
Transparent Incorporation of Science into Regulation .......................................................... 135
1. Identifying and Redressing External Constraints that Impede an Agency’s Efforts to
Improve the Rigor and Transparency of its Use of Science ............................................... 135
2.
C.

Review by OIRA and Other White House Offices and Agencies .............................. 139
Enhancing the Transparency of Research produced by Regulated Parties ..................... 146
10

1. Disclosure of Conflicts of Interest .............................................................................. 148

D.

2.

Data Disclosures in line with OMB Circular A-110 for All Regulatory Research ....... 149

3.

Minimizing Unsupported and Outdated Confidential Business Information Claims .. 151
Future Questions for the Study of Regulatory Science .................................................. 152

Regulatory science is often under fire, particularly when agency decisions are
hotly contested. For at least the last three decades, federal agencies have been criticized
in particular for not being clear about the role that science played in their decisionmaking process. This problem has been identified as one in need of reform by bipartisan,
respected organizations like the National Academy of Sciences 1 and the Bipartisan Policy
Center.2 The agencies’ failure to explain their work is one of the most common bases for
remands. 3 It is also tied to more fundamental concerns about how a lack of scientific
transparency in the agencies can fuel the politicization of science.4
It is not surprising that regulatory science presents difficult challenges for the
administrative state. Evaluating the rigor of a scientific analysis requires expert training,
often in the discrete area under study. It is thus difficult for agency decision processes,
which depend heavily on public comment and institutional checks by nonscientific
entities, to ensure that science has been used properly. The newsworthy examples within
administrative practice when the scientific analyses were not conducted rigorously or,
even worse, were manipulated to justify a particular result serve as a testament to the
1

See, e.g., NATIONAL RESEARCH COUNCIL, REVIEW OF THE E NVIRONMENTAL PROTECTION AGENCY’S
DRAFT IRIS ASSESSMENT OF FORMALDEHYDE (2011) [hereinafter NAS, FORMALDEHYDE REPORT];
COMMITTEE ON RISK ASSESSMENT OF HAZARDOUS AIR POLLUTANTS, NATIONAL RESEARCH COUNCIL,
SCIENCE AND JUDGMENT IN RISK ASSESSMENT (1994); NATIONAL RESEARCH COUNCIL, RISK ASSESSMENT
IN THE FEDERAL GOVERNMENT : MANAGING THE PROCESS (1983) [hereinafter NAS, RISK ASSESSMENT ].
2
BiPartisan Policy Center, Improving the Use of Science in Regulatory Policy 15-16, 41-42 (Aug. 2009);
see also Advancing the Public Interest through Regulatory Reform: Recommendations for President-Elect
Obama and the 111th Congress, c/o OMB Watch 26, 34, 47 (Nov. 2008).
3
Nearly forty percent of the vacaturs of agency regulations apparently occur because the agency failed to
adequately explain or document its reasoning. See, e.g., Patricia M. Wald, Judicial Review in the Time of
Cholera, 49 ADMIN. L. REV. 659, 665 (1997); see also Christopher H. Schroeder & Robert L. Glicksman,
Chevron, State Farm and the EPA in the Courts of Appeals in the 1990s, 31 Environmental L. Rep. (ELI)
10371, 10405 (April 2001) (describing a decade of cases in which EPA rules were remanded for failure to
support the agency’s reasoning). For an early example of these opinions, see Ethyl Corp. v. EPA, 541 F.2d
1, 68 (D.C. Cir. 1976) (Bazelon, concurring) (It is not enough for an agency to prepare a record compiling
all the evidence it relied upon for its action; it must also organize and digest it, so that a reviewing court is
not forced to scour the four corners of the record to find that evidence for itself. . . . . In informal rulemaking, the record should clearly disclose when each piece of new information is received and when and
how it was made available for comment.”)
4
Shortly after taking office, for example, President Barack Obama observed that “we have watched as
scientific integrity has been undermined and scientific research politicized in an effort to advance
predetermined ideological agendas.” President Barack Obama, Remarks at the National Academy of
Sciences (Apr. 27, 2009).

11

possibility that existing administrative processes are not adequate to police the quality
and transparency of agency science. 5 Equally serious, when a controversial decision is
made that depends heavily on science, it is difficult for nonscientific participants to tell
where the science leaves off and the policy choices begin. In this setting, agency officials
and even the President can dodge accountability by pretending that the “science made me
do it” when nothing could be further from the truth.
In response to these dual problems of using science robustly and transparently for
public decisions, a number of efforts have been made by the Executive Branch and
Congress to shore up the quality of the science undergirding regulatory products. Most
recently, President Obama issued a memorandum to the agencies directing that “To the
extent permitted by law, there should be transparency in the preparation, identification,
and use of scientific and technological information in policymaking.” 6 This
memorandum was further elaborated by the Director of the Office of Science and
Technology Policy (OSTP), John Holdren, who directed the agencies to “communicate
scientific and technological findings by including a clear explication of underlying
assumptions; accurate contextualization of uncertainties; and a description of the
probabilities associated with both optimistic and pessimistic case projections . . . ”7.
Because occasional lapses in the transparency and rigor of the agencies’ use of
science have been spotlighted in the news and political process so regularly over the last
three decades, however, there is reason to believe that the problems require more
fundamental changes to agency processes. It is possible, for example, that at least some
of these problems originate in decision-making structures that are deeply embedded in
agency practice and are not easy to change with well-intended directives. It is also
possible that in some cases the lack of transparency and rigor in the agencies’ scientific
analyses has more to do with forces outside the agencies’ control. Hard constraints on
agency decision-making imposed by Congress or interference with agency processes
from the White House can also contribute to reduced transparency and rigor in the
agencies’ scientific analyses. If these outside sources are the primary causes of these
recurring problems, then commanding the agencies to provide more robust analyses may
be preaching to the wrong choir.8
5

See generally Holly Doremus, Scientific and Political Integrity in Environmental Policy, 86 TEX. L. REV.
1601 (2008) (describing these problems in the natural resource field); Union of Concerned Scientists,
Federal Science and the Public Good (Dec. 2008); see also infra note365.
6
Memorandum on Scientific Integrity from the Administration of Barack H. Obama for the Heads of
Executive Departments and Agencies (Mar. 9, 2009), http://www.gpo.gov/fdsys/pkg/DCPD200900137/pdf/DCPD-200900137.pdf.
7
Memorandum on Scientific Integrity from John P. Holdren for the Heads of Executive Departments and
Agencies (Dec. 17, 2010), at pt. V [hereinafter John Holdren Scientific Integrity Memo] available at
http://www.whitehouse.gov/sites/default/files/microsites/ostp/scientific-integrity-memo-12172010.pdf .
8
The reforms to date seem to assume that agencies need only understand they need to do a better job
“showing their work” and that if problems are occurring with the agencies’ scientific integrity, they are
internal problems and are not caused by external forces.

12

This study looks behind agency work products to examine the agencies’ actual
decision-making processes themselves – the flow charts that show how the agency
incorporates science into its regulatory products. These flow charts reveal the points at
which agencies look to external peer reviewers, the public, and other entities for critical
feedback and advice. The flow charts also reveal internal oversight processes that are
intended to increase scientific and other sources of engagement in the agency’s sciencebased regulatory projects.
A comparative investigation of the decision-making structures that the agencies
use to incorporate science into regulatory projects helps provide purchase on this
challenging topic of regulatory science in several ways. First and perhaps most
important, there has been little to no attention to the decision-processes used by agencies
to incorporate science, particularly at a level that goes beyond the study of a specific
program. The well-known NRC and Carnegie Commission reports that provide the
foundation for much of our understanding of regulatory science focused primarily on
other types of challenges, such as the transparency of risk assessments.9 Understanding
the basic flow charts or processes by which agencies integrate science is thus largely
unexplored territory. 10 Indeed, precisely because they are often not well described, these
basic decision-making structures are particularly promising in their potential to spotlight
areas of innovation and also areas that might benefit from reform.
Second, and perhaps equally important, some of the most publicized problems in
agency science in recent years, ranging from the prominent Department of Interior (DOI)
scandal over high level manager, Julie MacDonald’s, scientific misconduct 11 to the more
run-of-the-mill charges by the National Academies of Sciences (NAS) that the
Environmental Protection Agency’s (EPA) assessments are unduly voluminous and
difficult to understand,12 all arguably originate with problems in the agency processes.
DOI’s Inspector General report that documents the manipulation of science at DOI traced
many of the examples of that manipulation to “enormous” gaps in guidelines governing
the processes that the FWS uses to integrate science into its decision processes. 13 Even
9

See, e.g., NRC reports cited in supra note 1; see also CARNEGIE COMMISSION, RISK AND THE
ENVIRONMENT: IMPROVING REGULATORY DECISION MAKING (June 1993); CARNEGIE COMMISSION, A
SCIENCE AND TECHNOLOGY AGENDA FOR THE NATION: RECOMMENDATIONS FOR THE PRESIDENT AND
CONGRESS (December 1992).
10
Unfortunately, because an exploration of the decision-making process or flow chart for integrating
science into regulation has been largely ignored, there also is no readily accessible theoretical or normative
framework that can serve as a foundation for this study. Instead, the basic norms of science – largely
extracted from the philosophy, sociology and history of science – serve as general guideposts for assessing
agency flow charts for integrating science into their decisions. See infra Sections III.A. and B.4.
11
See, e.g., U.S. Department of Interior Office of Inspector General, Report of Investigation: The
Endangered Species Act and the Conflict between Science and Policy, Dec. 15, 2008 [hereinafter OIG
MacDonald Report]; see also infra note 222 (providing an overview of the MacDonald scandal).
12
See, e.g., NAS, FORMALDEHYDE REPORT, supra note 1.
13
See OIG MacDonald Report, supra note 11, at pg. 2 of cover letter. [All signals (see, see, e.g., etc.)
should be italicized. There are many cases where they are not. Please review all footnotes and ensure that
all signals are italicized.]

13

more subtle problems originate from the structures that agencies use for making
decisions. For example, high level EPA staff concedes that its Integrated Risk
Information System (IRIS) risk assessments are lengthy and unwieldy due in part, and
perhaps in large part, to the convoluted decision processes that those assessments
undergo.14 The possibility that decision-processes may be to blame for some of the
continued dissatisfaction with how well agencies show their work and protect science,
then, seems inescapable.
Third, a process-focus allows a legal analyst to make a useful contribution to
assessing the reliability and transparency of the agencies’ use of science. While legal
analysts cannot identify, or at least will have a difficult time identifying agency analyses
that are incomplete in their use of the scientific literature or in their explanation of
alternative interpretations or assumptions, a legal analyst can trace the process by which
the science enters the regulatory process.15 This diagrammatic study not only provides a
helpful basis for comparing very different types of regulatory programs, but it helps
identify the role of institutional actors outside the agency, which can also influence how
the agencies use science.
Finally, understanding how the science is used by the agencies is a fundamental
first step in identifying ways to improve agency processes. If an agency isolates the role
scientific information plays in its ultimate decision and explains how it ensured that
scientific information was rigorous, then the public has a basis against which it can
evaluate both the scientific and policy judgments embedded in the agency’s decision.
This transparency allows those outside the agency to assess whether the agency’s use of
science comports with the authorizing law, the larger scientific record, and political
preferences. Distinguishing the role science plays in informing decisions from the role
played by policy judgments is in fact central to ensuring the accountability of scienceintensive regulations.16 In addition, clearly explicating how science informed a policy
decision advances other institutional and scientific goals, such as making agency
decisions accessible to a broad range of stakeholders, providing a bulwark against some
of the risks of the politicization of science, and identifying promising areas for future
research. A clear description of how the agency used science over the course of its
decision process may even provide the courts with a record that reduces the risk of
judicial challenge.
14

Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
15
Cf. Jamie Conrad, comments on draft ACUS Outline on Science in the Administrative Process project,
Dec. 5, 2011 (raising this concern).
16
Interestingly, although it seems well settled that transparency is fundamental to ensuring the rigorous
incorporation of science into policy, there has been little effort to consider this transparency from the
standpoint of the integration of science over the life cycle of a rule. Yet in some ways this chronological
view of science informing the decision is more illuminating than an agency’s (often post hoc) explanation
of what it selected as a default juncture at various points in the process. Somewhat similarly, the role of
various governmental scientists and other actors as authors in a report as largely escaped notice in the larger
regulatory science literature. See Section IV.A.3. infra.

14

As discussed in more detail below, a focus on the chronology of the
decisionmaking process (or flow chart) reveals that the agency that initiates a sciencebased rule is not the only party in control of how science is integrated. Potentially
significant constraintson agencies’ efforts to develop a more rigorous decision-making
process for incorporating science (e.g., relying on more robust external peer review;
providing more transparency in the analytical process; protecting authorship rights) can
arise from outside the agencies. In several important areas of regulatory decisionmaking, the agencies enjoy only partial control over their decision-making processes, and
their efforts to develop more robust external peer review or more transparent analyses
were effectively blocked by forces outside their control. Thus, while agencies certainly
can improve their processes at the margin, some of the most substantial impediments to
further improvements in the rigor and transparency of the agencies’ use of science must
come from outside the agencies.
The study of the agencies’ use of science is developed in four sections. The first
section provides a brief overview of the basic challenges caused by the frequent
conflation of science and policy and discusses how these challenges complicate a study
of the agencies’ use of science. The second section details the methods used in this
study. The third, more detailed and lengthy section then presents the findings; namely
the decision-making processes agencies use to integrate science in five separate
regulatory settings. In the fourth and final section, these findings are analyzed and this
analysis provides the basis for a series of recommendations.
I.

Background: The Illusive Line between Science and Policy

Most of the clashes in the use of science for policy arise from the difficulty of
determining where the science leaves off and the policymaking begins. There is no clear
point of demarcation. Scientists can credibly count many of the judgments and
assumptions interlaced in a computational model as “scientific judgments”; yet
policymakers can also credibly argue that many of these same choices – which essentially
select among plausible options – are better understood as policy judgments that fill in the
many gaps that science leaves behind. The line between science and policy is so
contested that the battles over it have a name in the social studies of science – “boundary
work.”17 There are multiple, historical accounts, spanning back centuries, over whether
scientists on the one hand or religious leaders or politicians on the other should make the
important choices needed to fill the cracks in scientific evidence, models, and
predictions. 18

17

See, e.g., Thomas F. Gieryn, Boundary-Work and the Demarcation of Science from Non-Science: Strains
and Interests in Professional Ideologies of Scientists, 48 AM. SOC. REV. 781, 782 (1983).
18
Id.

15

The deeply intertwined roles of science and policy in the development of
regulation lie at the core of many of the clashes over the agency’s use of science, but this
feature also complicates efforts to study science-policy. It is difficult to assess decisions
and choices that defy exclusive ties to either science or policy and thus lack a disciplinary
home. Before explaining how the methods of this study have been designed to take these
challenges into account, this first section offers a brief tutorial on why this line between
science and policy is so difficult to pin down.
In the realm of science-policy, scientific information is available to test narrow
and discrete hypotheses or collect observations, but to make the evidence useful to
policy, extensive extrapolation beyond the study is generally necessary. The analyst must
also choose between competing models or analytical tools in making these extrapolations,
must identify basic assumptions or defaults in order to make the models run, and
encounters considerable uncertainty in the resulting findings. As a result of these and
other judgment-laden steps, an analyst encounters a veritable landmine of choices that
need to be resolved to bridge existing evidence about, for example, the toxicity of a
chemical to its potential effects on an endangered species. Since these choices are not
purely scientific and since they often have considerable implications with respect to their
public consequences, the choices require input from scientists, policymakers,
stakeholders, and the general public. 19
To make this abstract concept of the intermingling of science and policy more
concrete, consider one of the controversies that arose in a regulatory program covered by
this study. 20 Currently the U.S. Fish and Wildlife Service (FWS) and the Environmental
Protection Agency (EPA) are at loggerheads on how to use the best available scientific
evidence to predict the adverse impacts of individual pesticide products on endangered
species. The agencies reach very different conclusions from the data about these
potential adverse impacts, as illustrated in the text box below. In comparing the
agencies’ answers to these science-policy questions, consider their very different
statutory instructions for assessing risks. The FWS is tasked with preventing the
extinction of endangered species, and when a species may be adversely affected by a
federal activity, the Endangered Species Act requires the FWS to use the best available
evidence in a way that gives the endangered species the benefit of the doubt.21 By
contrast, in its regulatory assessment of a pesticide registration, the EPA is required to
19

Exacerbating these challenges for the scientific analyst is ensuring that these assumptions and related are
communicated successfully to the decisionmakers and other attentive (nonscientific) public. As one EPA
manager noted, “with the complexity of scientific information ever increasing, limited time on the calendar
for policymakers to be briefed, and ‘junk science’ (a buzzword to discredit scientific information, among
other disparaging characterizations such as claims of bias), it is difficult for even the best communicators
among scientists to effectively share what may be nuanced interpretations of available data and associated
uncertainties.” Email from undisclosed EPA manager, March 5, 2012.
20
The nature of this controversy is summarized briefly in the agencies’ charge to the NAS Committee
examining “Ecological risk Assessment under FIFRA and ESA.” This Statement of Task is available at
http://dels.nas.edu/Study-In-Progress/Ecological-Risk-Assessment-Under-FIFRA/DELS-BEST-11-01.
21
16 U.S.C. § 1533(b).

16

balance the benefits of a pesticide against its costs to human health and environment. 22
This net balancing produces a much more open-ended framework that does not afford
species the benefit of the doubt. Instead, the species’ risks are compared against the
benefits of the pesticide.
Text Box 1: Comparison of FWS vs. EPA judgments in assessing pesticide risks to
endangered species (these differences are inferred from documents and interviews and
are illustrative only). 23
Questions arising in the
scientific analysis

FWS’s Answers

EPA’s Answers

Should a study with
methodological problems be
excluded from the analysis?
(e.g., what is the definition of
“best available science”?)

Not if part of the study does
not suffer from the
methodological problems and
the findings of that part of the
study suggest risks to
endangered species.
Sub-lethal, indirect and
cumulative effects on species
must be considered.

Yes. Standard exclusion
criteria exclude studies that
have methodological flaws
that cause the studies to be
unreliable.

What types of endpoints 24
should be measured?

How should chemical
mixtures be assessed?

What types of assumptions
should be included in the
models?

The effects of chemical
mixtures, as well as inactive
ingredients, are critical to an
assessment of risks to a
species.
Liberal spray drift25
assumptions must be factored
into an exposure model.

How should the species’ range

The species’ range should be

Only endpoints that can be
measured with some precision
can be included in the
analysis.
There is so much variation in
mixtures that they cannot be
included in a reliable model.

Reasonable spray drift
assumptions should be
factored into an exposure
model.
Population models need to

22

EPA must ensure that the pesticide does not present “unreasonable adverse effects on the environment.”
7 U.S.C. § 136a(a).
23
These differences are drawn largely from Statement of Task to the NAS Committee, supra note 20; from
letters from EPA to NMFS regarding draft biological opinions on various pesticide decisions, see Letters at
http://www.epa.gov/espp/litstatus/effects/epa-to-nmfs.pdf (page 3 and 4);
http://www.epa.gov/espp/litstatus/11-18-08-nmfs-biop.pdf (page 2); and Interview with FWS Staff,
Endangered Species Program, Jan. 26, 2012.
24
An endpoint is the adverse effect that a researcher measures in a toxicity study. Mortality is one of the
most straightforward endpoints. Other endpoints include various measures of neurological effects (e.g.,
spontaneous locomotion of a mouse in an open field), tumors (e.g., benign and malignant), reproductive
and development effects (e.g., brain weights of offspring at birth), etc. The challenge in toxicology is
identifying one or more endpoints for a study that can be measured reliably. Behavioral change in animals,
for example, is a much more difficult endpoint to measure as compared with mortality.
25
Spray drift refers to how far the pesticide sprays into the environment (and beyond the target) when it is
applied. Spray drift is affected by a number of factors, including the contents of the pesticide product, its
method of application, and wind speed.

17

be determined?

measured by assuming the
most expansive range.

How extensively should
possibilities of pesticide
misuse (beyond the label) be
considered?

Pesticide misuse should be
factored into the model in all
cases.

adopt reasonable assumptions
and require documentation for
all assumptions.
Pesticide misuse should not be
considered unless there is
evidence of that misuse.

As the table of disagreements reveals, there are important judgments at each point
in the process of assessing pesticide risks to endangered species. At the first step, the
agency must determine which of the existing studies inform the regulatory project and
which do not. While one might imagine that generic “exclusion/inclusion” criteria could
be designed to sort out the available research, even decisions about how to use the
literature depends on whether the agency seeks to afford every benefit of the doubt to the
species or instead simply to produce a replicable, “mean” answer to a question. Choices
also arise in identifying the parameters that will be used in a model. For example, what
effects should be considered in predicting adverse impacts (e.g., sub-lethal effects or
easily measured mortality) and what pesticides should be included (e.g., the entire
chemical mix or one pesticide at a time)? Choices arise again in determining how to
account for various scenarios, such as assumptions regarding spray drift, species’ range,
and even the misuse of pesticides during application. All of these decisions are informed
by scientific and technical judgments about plausible options, yet none is resolved by
them. While the text box extracts only a handful of these choices, in science-policy work
ordinarily done by agencies there are dozens, and according to one classic NAS report,
often as many as fifty significant choices that can punctuate any given effort to
characterize the risks of a product.26
No wonder, given these different statutory directions, that the two agencies’
approaches to the scientific literature and related analytical steps are divergent and have
been the source of continued technical disagreements and interagency strife. 27 Yet
drawing out some of the disagreements – that the agencies have sent to the National
Academies for guidance – also illuminates the types of embedded science-policy choices
that are commonplace in the agencies’ use of science. This illustration also underscores
that there is clearly no one size that fits all with respect to either the questions or the
answers, and thus calls for generic risk assessment guidelines may be misplaced if not
mistaken. 28

26

See, e.g., NAS, RISK ASSESSMENT, supra note 1, at 29-33.
See, e.g., Interview with FWS Staff, Endangered Species Program, Jan. 26, 2012; Interview with EPA
Staff, Office of Pesticide Programs, August 5, 2011.
28
But see Nicholas Bagley & Richard L. Revesz, Centralized Oversight of the Regulatory State, 106
COLUMBIA L. REV. 1260, 1320-24 (2006) (recommending the centralization of risk assessment guidelines).
27

18

II.

Methods

These intertwined science-policy choices help to highlight the methodological
benefits of focusing on the process by which agencies make science-intensive decisions.
Such a process orientation avoids the need to draw lines between “science” or “policy”
and focuses the analysis instead on whether the agency’s decision process, including the
assumptions, framing, and integration of science and policy, is conducted in a way that
helps ensure it will be both robust and transparent. Staff papers that identify and interpret
the available scientific evidence, or that place that evidence into risk assessment models
are not simply “science”; they involve choices made necessary by limits in information.
The goal, then, is to develop a process that helps ensure the resulting analysis will state
these core assumptions, uncertainties, and framing assumptions clearly and subject them
to scientific and public review.
This section first describes in more detail why the decision-making process is a
particularly good vehicle for assessing the scientific rigor and transparency of the
agencies’ use of science. The remainder of the section then outlines the more detailed
methods used to examine these processes in this study.
A. Decision Processes as a Diagnostic Tool for Studying the Agencies’ Use of
Science
Adherence to a basic, well-established process is one of the cornerstones of
rigorous science. For example, standard scientific practices insist, at a minimum, that
research be rigorously peer reviewed and that the methods be communicated in a way
that allows the research to be replicated. 29 Over time, sectors within science, particularly
editors of biomedical journals, have also insisted that the underlying data be shared;30 the
researchers’ affiliations and biases be openly disclosed;31 and that authorship be certified
to ensure that an author has the right to make all final decisions on a manuscript, which

29

See, e.g., HELEN E. LONGINO, SCIENCE AS SOCIAL KNOWLEDGE 8 0 (1990); Philip Kitcher, Patterns
of Scientific Controversies, in PETER MACHAMER, MARCELLO PERA, & ARISTIDES BALTAS, EDS.,
SCIENTIFIC CONTROVERSIES: PHILOSOPHICAL AND HISTORICAL PERSPECTIVES 31-35 (2000) (on methods
and theories); AAAS, SCIENCE FOR ALL AMERICANS: A PROJECT 2061 REPORT ON LITERACY GOALS IN
SCIENCE, MATHEMATICS, AND TECHNOLOGY 28 (1989).
30
See, e.g., NATIONAL RESEARCH COUNCIL, NATIONAL ACADEMIES OF SCIENCES, SHARING
P UBLICATION -R ELATED DATA AND MATERIALS: R ESPONSIBILITIES OF AUTHORSHIP IN THE LIFE
SCIENCES 4 (2003) (advocating a “uniform principle for sharing integral data and materials
expeditiously” or UPSIDE); NATIONAL RESEARCH COUNCIL, N ATIONAL ACADEMIES OF SCIENCES,
RESPONSIBLE SCIENCE 1 1 (1992) (noting that scientists “are generally expected to exchange research
data as well as unique research materials that are essential to the replication or extension of
reported findings”); ROBERT K. M ERTON, THE SOCIOLOGY OF SCIENCE ( 1 9 7 3 , ed. Norman Storer).
31
See, e.g., SHELDON KRIMKSY, SCIENCE IN THE PRIVATE INTEREST 125-40 (2003); C. D. DeAngelis, P.
B. Fontanarosa, & A. Flanagin, Reporting Financial Conflicts of Interest and Relationships
between Investigators and Research Sponsors, 2 86 JAMA 89 (2001).

19

also serves as an endorsement by the scientist that the research findings and statement of
methods are correct.32
Just as processes and decision-making structures serve as a proxy for ensuring a
minimum level of quality and transparency in research science, so they would seem to
provide an equally if not more valuable proxy in assessing the agencies’ use of science
for regulation. When scientific oversight processes within science do not include these
bare minimum qualities, scientists generally draw an adverse inference about the quality
of the resulting product. Again, these same measures seem to be fairly used to evaluate
regulatory science.
Due to differences between research science and regulatory science, the basic
design of the study – applying standard scientific conventions regarding ideal decisionprocesses to the regulatory arena – requires at least one significant adjustment, however.
As just described, the agency’s use of science involves many more judgments,
assumptions, and uncertainties than most basic research, and each of these major choices
is likely to affect a range of parties. The agency is thus developing policy at the same
time it is attempting to use the available evidence in a robust way. As a result, the
agency’s decision-making process must ensure not only that the science is used
rigorously but also that important choices are exposed in the analysis. Expert peers are
not the only important reviewer of agency analyses, and stakeholders and the public must
also review this work. As a result, iterative reviews may be needed to draw out all these
intertwined, yet very different science and policy choices and decisions.
While a process-based examination of the agencies’ use of science sheds light on
the extent to which the process itself ensures that these basic principles are given some
weight, a study of flow charts and agency processes is not the most direct way to examine
the agency’s use of science. The National Academies of Sciences (NAS) panels that
examine all facets of an agency’s scientific work – from the agency’s identification and
interpretation of the literature through the use of models to its ultimate explication of how
it conducted the analysis – for example, will provide both more complete and more
robust accounts of the actual quality of the agency’s work. Yet the chief strength of these
NAS studies – a deep, detailed account of the agency’s use of science – is also a
limitation, since by their nature the studies can typically only drill down deeply into a few
regulatory projects. These substantive studies also generally do not investigate why the
agency might be missing opportunities for the robust utilization of science in its products;
NAS reports focus primarily on ways that the agency’s substantive discussions and
analyses could be improved.
32

For example, the Journal of the American Medical Association (JAMA) requires as a condition to
publication that “[f]or all reports (regardless of funding source) containing original data, at least 1 named
author (eg, the principal investigator) who is independent of any commercial funder or sponsor must
indicate that she or he “had full access to all the data in the study and takes responsibility for the integrity
of the data and the accuracy of the data analysis.” JAMA Instructions for Authors, Data Access and
Responsibility, available at http://jama.ama-assn.org/site/misc/ifora.xhtml#DataAccessandResponsibility.

20

The tack taken in this study is thus a complement to a detailed substantive study
of the agency’s science since this study looks at these same regulatory projects from a
process perspective. The NAS report highlights ways that the agency’s report could be
better, but there is little to no attention given to the actual flow chart or process by which
science is incorporated by the agency. Indeed, this feature is generally bracketed.
Examining the actual processes themselves, then, hopefully illuminates process or related
features that may be causing repeat problems in agency work products. Until fixed, these
process problems may stand as an impediment to the agencies’ ability to increase the
rigor and transparency of regulatory science.
In focusing exclusively on agency decision-making processes, this study
necessarily ignores other, equally important factors that inevitably impact the quality, and
transparency of the agency’s use of science. The focus on decision-making process thus
illuminates some important features of the agency’s use of science at the expense of
eclipsing or even obscuring other features. This is an analytical hazard that seems
unavoidable, at least to the extent that the study does not provide a reconnaissance
inventory of potential problems and issues. 33 Indeed, even with respect to examining
only this one specific feature – agency decision-making processes – the study must
bracket several process features that could ultimately alter its conclusions. Two of these
features deserve mention because they are particularly critical to the analysis. First, it is
assumed that when external peer review is done, it is done in a way that is both
competent and fairly neutral. Potential problems with the selection and use of reviewers,
for example, are not explored in this study but left for another day. 34 It is also assumed
that an agency’s scientific staff is much like the scientific staff one might find in
academic or other research settings in terms of competence, expertise, and ethical
commitments to do excellent work. In this study, these two important features of the
analysis were not investigated in any detail. 35 Indeed, if the staff is not sufficient or the
peer review selection is badly biased, the best response is to repair them since both are
fundamental to so much of the work that the agency does.
B. Methods
Given the vastness of the science-based regulatory universe, the study required
substantial narrowing to keep it to a manageable size. This entailed several consecutive
“cuts” or screening steps. These screening decisions and related methodological choices
are explained below.
33

The Regulation Advisory Committee recommended a narrower study as opposed to reconnaissance
research.
34
See infra Section IV.C. (recommending this topic for further study).
35
It may be helpful to note that at least with respect to the competency and strength of the agencies’
scientific and technical staff, the interviewees and documentary evidence was generally favorable with
regard to the quality of the agency’s scientific work. Thus at least the evidence collected here did not
undermine the assumption nor given an evidence that the ethics and competence of agency scientific work
is relatively strong.

21

1. Identifying the Types of Science-based Regulatory Projects most in need of
Study
The first step in the screening process was to identify which part of the universe
of science-based regulation to examine. Many and perhaps the majority of agency
decisions involve some type of scientific or factual information that counts as scientific,
but given the breadth and variation in this universe of science-based regulatory projects,
further focus is necessary.
This study examines only those agency decisions that require mastery of a large
body of scientific literature (natural sciences and engineering) as applied to a particular
policy question. Identifying and interpreting the diverse studies, applying them through
models to reach predictions, and then explaining the limitations of these processes and
what they imply for policy are all critical features of the regulatory projects selected here
for in-depth study. By contrast, the study does not examine regulatory work that involves
much more limited use of scientific or technical information – such as the use of medical
information in entitlement hearings or the use of engineering data in transportation
decisions. The study also does not examine the agencies’ use of social science.
2. Which Agencies
The second screening step involved identifying which agencies and processes to
examine within this still very large set of possible science-intensive regulatory programs.
Because so little is known about agency decision-making structures, an examination of
diverse approaches among the agencies was expected to yield more insights than an
examination of a single set of regulations within one agency. To that end, three agencies
were selected for study that engage in very different types of science-based decisionmaking projects.
The three agencies occupy different points within a larger matrix of agency
science-based regulation. The first agency – the EPA – generally develops regulations
that protect the public health and environment from pollution and dangerous products. A
great deal of the science that informs EPA’s work comes from toxicology, epidemiology,
and ecological sciences. Because EPA covers a broader range of issues than other
science-intensive agencies like the Food and Drug Administrative, it was selected as the
best candidate for studying agency processes in the general area of environmental and
public health protection.36 The second agency – the Fish and Wildlife Service (FWS) of
the Department of the Interior – protects certain natural resources, including endangered
36

The Food and Drug Administration (FDA) was actually one of the agencies originally slated for
investigation in this Study. In part because the nature of FDA’s regulatory work parallels that of the EPA
and in part because the scope of this Study was already substantial, FDA was ultimately dropped from the
Study. At the point that FDA was dropped, Wagner had already conducted one set of interviews and
collected a small stack of documents on the agency’s decision-making. Thus if the FDA becomes a focus
in further research, there is an initial set of research materials available on the agency.

22

species. The sciences the FWS utilizes for its science-intensive regulation is more
heavily based in taxonomy, animal behavior, ecology, and related environmental studies.
The FWS, perhaps more than other natural resource agencies, has come under political
fire repeatedly over the last few decades, particularly with respect to its endangered
species decisions. Because of this more intense scrutiny, the FWS was selected as the
agency that would provide a particularly useful window into the decision processes used
by natural resource agencies (as contrasted to health and environmental protection
agencies like EPA) to incorporate science into publicly important decisions. The third
agency – the Nuclear Regulatory Commission (NRC) – develops requirements for
nuclear safety, including licensing nuclear reactors. Its formal and informal rulemakings
rely on radiation sciences, but also often involve issues that are informed by complex
engineering and operational sciences. The NRC is an independent agency, which
provides a valuable point of contrast with the other two agencies, particularly with
respect to Office of Management and Budget (OMB) review.
Although these three agencies engage in quite different types of science-based
regulatory work, it is surely the case that they do not represent the waterfront of agency
approaches to science, nor may they even offer windows into what are likely to be the
worst practices or possibly even the best practices among the agencies (although the latter
seems less likely as discussed below).37 As such, then, this first foray into agency
decision-making processes represents only the tip of the iceberg in terms of identifying
regulatory features worthy of study and selecting the most varied and interesting
processes within the agencies for investigation.
3. Which Programs
Because individual programs within a single agency can sometimes vary
dramatically, the third step involved choosing specific programs within each agency to
examine in greater detail. In the case of EPA, in particular, the agency’s decision-making
processes vary considerably from one regulatory program to another, although there are
some common themes that run through the programs. Regulatory programs were
selected within EPA that represent some of this variation. The first program selected for
study is EPA’s regular review of the six national air quality criteria pollutant standards
(NAAQS), which are conducted through large, informal (only in the APA sense)
rulemakings. This NAAQS program is renowned for its scientific quality and also for the
extraordinary size of the literature that informs EPA’s review. A second EPA program –
the review of registrations for conventional pesticides – involves a much higher
regulatory throughput (1000 chemicals as opposed to standards for six pollutants) and is
conducted as a licensing decision. The nature of the scientific analyses is more uncertain
and the data is much thinner as compared with the NAAQS reviews. The third EPA
37

In the analysis section, the larger literature that bears on agency science and decision-making processes is
integrated with this more grounded research to provide additional perspective on the extent to which the
regulatory programs under study are wholly unique or relatively typical of other regulatory areas.

23

program selected for study – the EPA’s Integrated Risk Information System (IRIS) –
produces reference values and cancer potency estimates for exposure to toxic substances,
but the assessments are informational only; these assessments are not required by statute
and are not judicially reviewable. As a scientific matter, the analyses are narrower in
scope than pesticide registration reviews, since they do not involve exposure estimates,
but the available literature is generally even more incomplete than is the case even for
pesticide registration reviews; thus scenarios and uncertainties are many and complicated
and permeate these hazard and dose-response assessments.38 These EPA programs also
offer an in-depth view of science-based decisions at three different points in the
administrative process spectrum (e.g., informal rules; licensing decisions; and nonbinding
risk assessments) and vary with respect to the nature of the available evidence and the
type of analytical models needed to reach a decision.
FWS conducts a wide range of science-based regulatory projects, but because it
uses a relatively truncated approach for incorporating science into policy, it does not
appear that, as compared with EPA, there is nearly as much variation from program to
program in the decision-making steps that the FWS follows. In any event, due to an
effort to keep the study to a manageable size, only one program was examined in the Fish
and Wildlife Service that appears relatively representative – the listing of endangered
species and the designation of critical habitat. These particular decisions are heavily
informed by scientific analyses of the species and their need for survival, but the latter
decisions on critical habitat must also be informed by economic and related
considerations. FWS’s decisions on listing and habitat designations are also published as
informal rulemakings, and this more standard type of regulatory output should allow for
easier comparisons to other agency informal rules.
Finally, the NRC, an independent agency, conducts a wide range of informal
rulemakings (e.g., to establish requirements governing nuclear waste and nuclear safety)
and licensing decisions. Like the FWS and perhaps even more so, NRC does not appear
to have sharp differences in its approach to the incorporation of science from one type of
program or regulatory decision to the next, as is the case with EPA. Thus the decisionmaking processes at NRC seem largely generalizable on an agency-wide level. It is
assumed that NRC’s informal rulemakings may be the most useful feature to study to
allow for a comparison with other agencies’ work, but given the importance of licensing
decisions at NRC, some attention is also given to these decisions.

38

The NRC refers to four stages of risk assessment, and IRIS captures only the first two stages: hazard ID
and dose-response. Various EPA program offices will then apply exposure scenarios depending on the
regulatory issue at hand to develop standards or related risk estimates.

24

4. What Factors
To ensure the various regulatory programs were consistently evaluated, several
key principles were identified to frame the analysis. The principles emerged by melding
the practical realities of regulatory science with established scientific norms and goals,
such as skepticism and disinterestedness. A discussion of each of these principles
follows:
1. Transparency: In using science, agencies should explain their use of the existing
evidence in as robust a way as practicable. This includes identifying the overarching
values and assumptions that influenced the technical decision-making (as discussed
above with pesticides and endangered species); detailing the literature consulted;
explaining how or why they weighted or excluded a study bearing on a policy question;
how the use of different assumptions and models might alter conclusions; areas of
uncertainty that limit the evidence; and what the underlying policy questions were and
how the framing of the questions themselves affected the integration of scientific
evidence for the issue at hand.39 This transparency makes it possible for others to
evaluate and replicate the analysis and is thus critical to the rigorous use of science.
2. Disinterestedness: Agencies should, where-ever possible, attempt to conduct their
initial analysis of the scientific literature and evidence bearing on a policy question
without being influenced by a preferred policy outcome. 40 While the complete separation
of science from policy is not possible, a rigorous and candid explication of how the
existing evidence intersects with the policy question(s) will help separate the analysis
from the decision-making. 41 Ideally, this is accomplished by a first step that provides a
statement of the general policy questions and an analysis of the evidence and alternative
applications of that evidence to a decision, including a robust statement of uncertainties,
in relation to those questions. A second step then selects the best policy choice from the
resulting options. While this approach does not exactly map against the “findings of
facts” and “conclusions of law” bifurcation used in trial courts, the basic idea is that a
vigorous and robust airing of the facts, including uncertainties and their implications,

39

See supra note 29 (citing sources); see also infra note 452 and accompanying text (identifying these
same principles in President Obama’s Memorandum on Scientific Integrity).
40
See, e.g., SHELDON KRIMKSY, SCIENCE IN THE PRIVATE INTEREST 125-40 (2003).
41
See, e.g., Bipartisan Policy Center, supra note 2, at 15 (2009) (recommending that “[t]he Administration
needs to devise regulatory processes that, in as many siutations as possible, could help clarify for both
officials and the general public which aspects fo disputes are truly about scientific results and which
concern policy”).

25

provides a clearer record against which subsequent policy considerations can then be
judged. 42
3. Skepticism: Skepticism from a diverse set of experts is vital to robust science and also
improves the transparency of the analysis. 43 There are at least two features necessary to
develop this vigorous skepticism
a. Peer (expert) review or related oversight. Agencies conducting science-based
decisions should insist on a rigorous expert review of their scientific analyses. 44 This
expert skepticism should be applied first and foremost to the judgments embedded in the
agency’s scientific analysis and in the agency’s explication of its assumptions. This can
be done through a range of techniques that include periodic audits of technical analyses;
routine intra-agency review processes; or the solicitation of expert peer reviewers that can
provide input individually or through a Federal Advisory Committee Act (FACA) panel.
b. Internal debate and dissent. There should also be constant vigilance against
the tendency to devolve into a “group think” mentality, which can stem from a number of
diverse, but reinforcing forces within the agency including: 1) career staff scientists who
work together over years or decades and seek to maintain a united front against outside
criticism; 2) a perception that a management-held position should prevail when at all
possible; and 3) direct intervention from supervisors or strong-willed colleagues. To
protect against these risks, individual scientists should have a right to dissent, which can
range from removing their name from authored reports to lodging formal dissents on
decisions within the agencies.
4. Use of Stopping Rules: Science-based dialog, if done in keeping with the norms of
science, has no clear stopping point. Science is continually evolving and, in theory,
policy should be constantly evolving with it. This creates the need for artificial and
explicit deadlines, usually based on policy, for completing the analysis associated with
any given decision (referred to as “stopping rules” throughout the paper). Stopping rules
thus identify the point at which the scientific record will be closed and disagreements and
debates bracketed for purposes of reaching a decision. Since scientific questions will
never be resolved completely, a decision subjected to stopping rules will be revisited
again later, through an adaptive process that considers the new evidence and adjusts the
original rule or decision as necessary. 45 While stopping rules can be amended, decision

42

Id. (recommending that “the Administration should require that a section of the Federal Register notie for
any proposed guidance or rule that is informed by scientific studies describe the primary scientific
questions and the primary policy questions that needed to be answered in drafting the rule”).
43
Longino, Science as Social Knowledge, supra note 29, at 80.
44
See supra note 29.
45
See generally Sheila Jasanoff, Transparency in Public Science: Purposes, Reasons, Limits, 69 LAW AND
CONTEMPORARY PROBLEMS, Summer 2006, at 22, 37-39 (introducing the concept of stopping rules to the
science and law literature).

26

theory suggests that it is best that they be established in advance of a process, with an
opportunity to make exceptions later. 46
5. Setting the Policy Choices against the Evidence: Those making the ultimate policy
decision should be challenged on their use of science, and the decision-maker should
explain his/her science-policy choices against the scientific record. This provides a
mechanism for “policy” transparency. Rather than simply dropping out studies or
invisibly changing assumptions or the algorithms of complex computational models, the
decision-maker must be forced to explicate both technical and policy decisions and
explain why they were reached.
6. Utilitarian considerations and other policy considerations, such as equitable impacts,
irreversible effects, statutory requirements, etc.: These ideals or goals for scientific
transparency must be balanced against other goals and principles. Resources, time, or
other features may simply not justify an agency’s ability to ensure that all five principles
for scientific transparency enumerated above are met in a given decision-making process.
The programs varied, sometimes considerably, in their commitment to these
principles. These differences are discussed in detail in the analysis section.
5. Sources of information about the Agency Decision-making Processes
Agency documents and interviews provided the primary source of information
about the agencies’ decision-making processes. Agency documents ranged from publicly
available emails to short summaries of decision processes posted on the internet to
lengthy decision documents. Interviews were conducted primarily by phone and
typically with top managers within the agency; more than thirty current and former
agency officials and staff at EPA, FWS, the Department of Interior (DOI), NRC, the
Office of Management and Budget (OMB), the Office of Science and Technology Policy
(OSTP), and the National Academies of Science (NAS) were interviewed over a seven
month period.47 This information was supplemented by over a dozen interviews with
various stakeholders;48 a search of a few individual rules to provide samples of the

46

For a discussion of setting stopping rules ex ante in decision theory, see WARD EDWARDS, ET AL.,
DECISION SCIENCE AND TECHNOLOGY: REFLECTIONS ON THE CONTRIBUTIONS OF WARD EDWARDS 86
(James Shanteau, Barbara Mellers & David Schum eds. 1999) (describing “simple stopping rules” in
decision theory that specify computationally simple conditions for halting the gathering of more
information). For a discussion about their ex ante use in regulatory science with regard to identifying a
point at when clinical trials can be stopped because adequate information has been collected, see Nigel
Stallard, et al., Stopping rules for phase II studies, 51 BRITISH J. OF CLINICAL PHARMACOLOGY 523 (2001).
47
Several academics who have worked with or researched the agencies’ use of science were also
interviewed.
48
The organizations interviewed generally, although not always, have PhD scientists on staff who are
deeply involved in oversight of the agency’s science-based decisions. The stakeholders interviewed from
the public interest community included staff from: Beyond Nuclear; Environmental Defense; the Natural
Resources Defense Council, PEER, Pesticide Research Institute, and the Union of Concerned Scientists.

27

agency’s work; and surveying Governmental Accountability Office (GAO) reports,
Inspector General reports, and congressional oversight hearings. While an attempt was
made to document every important process step with written records to the extent
possible, for some features of agency processes, agency interviews provided the only
source of information on how science is integrated into the decision-making process.
This “non-transparent” feature of the agencies’ decision-making process is taken up again
in the analysis and recommendations section.
As noted, the basic study design focuses on discrete programs within three
agencies. This narrow focus makes it difficult to generalize beyond the specific programs
under study. To work around this practical limitation, the analysis and recommendations
identify general themes that run through many or most of the programs and suggest best
practices that operate presumptively and can be rebutted when circumstances make them
impracticable or otherwise ill-advised. When there is supporting evidence in the
literature that a problem may be occurring more widely within the government, this
information is also included in the analysis.
III.

Findings

This section, which details the findings, is divided into two parts. The first part
provides a detailed discussion of the decision-making approaches or flow charts used in
the five regulatory programs under study. These descriptions identify comparable
features of the processes used across agencies, as well as key differences.
The second part focuses more specifically on the agencies’ scientific integrity
policies used to protect the autonomy of scientific staff. These policies also attempt to
encourage internal skepticism within the agency and provide mechanisms for internal
dissent on science-intensive analyses and decisions. Because these scientific integrity
programs are implemented agency-wide, they were investigated at the Departmental
level. For example, rather than examine specific regulatory programs within EPA, the
entire agency was studied holistically. Similarly, overall Department of Interior policies
were studied rather than those specific to the Fish and Wildlife Service.
Together the two parts provide a relatively complete picture of how the agencies
incorporate science into policy, at least for the regulatory programs selected for study.

Stakeholders interviewed from industry included: the American Chemistry Council, and the Center for
Regulatory Effectiveness, and Exponent.

28

A. The Incorporation of Science into Specific Regulatory Programs
1. The Environmental Protection Agency (EPA)
a. The Review of National Ambient Air Quality Standards (NAAQS)
The findings begin with a particularly detailed examination of the NAAQS review
process because it presents the equivalent of a five-star process for incorporating science
into regulatory policy. Indeed, the extraordinarily elaborate, five year NAAQS review
process appears unprecedented, and it is difficult to imagine any other regulatory setting
where such an extravagant science-policy process may be necessary. On the other hand,
precisely because the NAAQS decision-making process is so exemplary, it offers
numerous lessons for simpler, low-budget agency processes with respect to developing a
rigorous analytical approach to science-policy.
The initial section on the NAAQS process concludes with a discussion of some of
its most innovative features. This discussion spotlights features that then inform the
analysis of other agency programs.
The Law of NAAQS
Under Section 109 of the Clean Air Act, EPA is required to review at regular,
five-year intervals the standards, as well as the supporting scientific assessment criteria,
for six criteria or general pollutants that EPA has identified under Section 108 of the
Act.49 Section 109 of the Clean Air Act not only sets specific deadlines for EPA’s
staggered review of the criteria pollutant standards, but it also provides several
substantive and procedural constraints on that decision-making process.50 First, Congress
required that a primary NAAQS standard be set at a level that is “requisite to protect the
public health” with “an adequate margin of safety.”51 The Supreme Court has confirmed
that this mandate allows EPA to consider only scientific and not economic factors in
setting the standards for these criteria pollutants.52 Second, Congress required extensive
involvement in the standard-setting exercise by a seven-member expert panel that
“includ[es] at least one member of the National Academies of Sciences, one physician,
and one person representing State air pollution control agencies.” 53 By statute, this panel
49

42 U.S.C. § 7409(d)(1). This review process of National Ambient Air Quality Standards (NAAQS) is
mandatory, and EPA has been sued numerous times for missing its deadline. See, e.g., Comm. for a Better
Env’t v. U.S. E.P.A., No. C 07-03678 JSW, 2008 WL 1994898, at *1 (N.D. Cal. May 5, 2008) (bringing
suit to compel the EPA to perform its past due, mandatory review duties). Indeed, EPA’s review and
reform of the NAAQS process in 2006, discussed below, was triggered by a realization that EPA was
growing only further and further behind in meeting its statutory deadline. Interview with EPA Staff,
National Center for Environmental Assessment, Office of Research and Development, Jan. 18, 2012.
50
42 U.S.C. § 7409(d)(1).
51
Id. at § 7409(b)(1).
52
Id. at § 7409(b)(1); see Whitman v. American Trucking Associations, Inc., 531 U.S. 457 (2001).
53
Id. at § 7409(d)(2)(A)-(B).

29

is authorized to make recommendations to the Administrator on possible revisions, as
well as to review EPA’s scientific assessments. In practice, the panel tends to focus its
role on reviewing EPA’s various scientific and policy documents, although the panel can
make recommendations independent from these documents.54 In response, EPA created
the Clean Air Scientific Advisory Committee (CASAC), a standing committee chartered
under the Federal Advisory Committee Act (FACA), which plays an important role in the
EPA’s NAAQS review process. Finally, EPA’s decision to revise or retain a NAAQS
must undergo a public notice and comment period.55
In the forty years that have followed passage of the Clean Air Act, EPA’s
implementation of its NAAQS reviews has evolved over time. The scientific analyses in
these reviews have grown from short, relatively simple assessments to encyclopedic
assessments that even experts sometimes labeled as impenetrable.56
EPA’s challenges in conducting NAAQS reviews are made still more daunting
because the literature that EPA must consider during a review period is substantial. A
single NAAQS review can involve the analysis of thousands of studies. 57 As a result of
this highly complex and involved analysis, EPA has experimented over the decades with
different techniques for explicating its judgments as well as for managing the huge and
growing scientific literature. Although it has not been easy, EPA appears to have finally
developed a transparent process that produces analyses that are accessible to expert
onlookers and that manages successfully to bridge science and policy in ways that appear
worthy of replication.
Steps in the NAAQS Revision Process
To gain control over the sprawling NAAQS process and bring its NAAQS
reviews back on schedule and limit future litigation, Administrator Johnson under
President George W. Bush empaneled a “top-to-bottom review” of the NAAQS process
in 2006. 58 The EPA staff recommendations from that initiative have now been largely
implemented,59as illustrated in the figure below. 60

54

Id.
Id. at § 7409(b)(1), referencing (a)(1)(A).
56
EPA, Review of the Process for Setting National Ambient Air Quality Standards, March 2006, Review at
E-1, available at http://www.epa.gov/ttnnaaqs/pdfs/naaqs_process_report_march2006.pdf.
57
Interview with EPA Staff, Office of Air and Radiation, July 26, 2011; Interview with EPA Staff, Office
of Air Quality Planning and Standards, Jan. 17, 2012; Interview with EPA Staff, National Center for
Environmental Assessment, Office of Research and Development, Jan. 18, 2012.
58
EPA, Review of the Process for Setting National Ambient Air Quality Standards, March 2006, at E-1
[hereinafter EPA, NAAQS Review], available at
http://www.epa.gov/ttnnaaqs/pdfs/naaqs_process_report_march2006.pdf. Although the primary impetus
for this review was concern about the ability of EPA to meet its statutory (and judicial) deadlines, two of
the four priorities were to improve the transparency of EPA’s use of science, particularly in relation to
policy. Specifically, among the four key charges, the team was asked to determine ways to clarify the
distinctions between science and policy judgments and to identify and characterize uncertainties in
55

30

scientific information. See Memo from George Gray and William Wehrum to Marcus Peacock, April 3,
2006, at page 1 (memo available at
http://www.epa.gov/ttnnaaqs/pdfs/naaqs_process_report_march2006_cover.pdf); see also EPA, NAAQS
Review, supra, at Appendix 2 (setting out more elaborate bullet points on the priorities for the review team)
(appendices available at
http://www.epa.gov/ttnnaaqs/pdfs/naaqs_process_report_march2006_attachments.pdf). Interestingly, some
of this enhanced transparency, at least initially, was also expected to result from transferring responsibility
from EPA staff to the political management for producing key reports, particularly the policy assessment.
As discussed below, this particular tact did not ultimately work; many observers believed improved the
scientific rigor or transparency of the document suffered and authorship was returned to EPA staff.
Interestingly, some of this enhanced transparency, at least initially, was also expected to result from
transferring responsibility from EPA staff to the political management for producing key reports,
particularly the policy assessment. As discussed below, this particular tact did not ultimately work; many
observers believed improved the scientific rigor or transparency of the document suffered and authorship
was returned to EPA staff.
59

EPA’s own documentation of this process is somewhat patchy. For its public overview, see EPA,
Process of Reviewing the National Ambient Air Quality Standards, last updated on August 25, 2011,
available at http://www.epa.gov/ttnnaaqs/review.html. Beyond this general statement, the next best sources
to understand EPA’s process is the review report (which obviously was prepared prior to implementation),
see EPA, NAAQS Review, supra note 58, and by accessing EPA’s individual reports prepared after the
2006 review. (For particulates, for example, see EPA, Particulate Matter (PM) Standards, last updated on
November 8, 2011, available at http://www.epa.gov/ttnnaaqs/standards/pm/s_pm_index.html. After the
2006 review, Administrator Jackson made some additional adjustments to implementation of the NAAQS
process; the most significant change is discussed at the end of this Section and concerns authorship of the
staff policy assessment. See Process for Reviewing National Ambient Air Quality Standards memorandum
from Lisa Jackson to Elizabeth Craig and Lek Kadeli, May 21, 2009, available at
http://www.epa.gov/ttnnaaqs/pdfs/NAAQSReviewProcessMemo52109.pdf.
60
Excerpted from id. at 3.

31

As this flow chart shows, the current NAAQS decision-making process involves
four separate staff-authored reports, each of which each constitutes a distinct stage in the
process. This flowchart is a simplified representation of the process; the steps in the
process are described in greater detail below (in some cases providing additional substeps not included in the flowchart). Cumulatively these reports evaluate the scientific
evidence to determine whether a revised standard is needed. A discussion of each stage
follows.
1. The Planning Report (Integrated Review Plan)
The first stage – the development of a planning report - begins with a “kick-off”
workshop that solicits comments from the public and scientific community (including
invited scientists) about developments in the science and policy that should frame EPA’s
review. 61 The workshop focuses specifically on scientific discoveries and related
developments occurring over the past five years that should inform EPA’s review of the
standard and hence deserve careful scientific review.
After the workshop, the EPA staff from the Office of Research Development
(ORD) and the Air Office prepares an integrated review plan. The primary purpose of
this planning document is to frame “key policy-relevant issues that would generally be
used to frame the science assessment, risk/exposure assessment, and policy assessment.”
The report also sets a timetable for completing subsequent stages of the process.62
The planning report is integral to enhancing transparency of the NAAQS
review. 63 By framing the relevant science-policy questions, the planning report provides
a focus for the remaining four years of EPA’s analysis.64

61

See EPA, Generic NAAQS Review Process, March 2007, available at
http://www.epa.gov/ttnnaaqs/pdfs/peacock_4_17_07_attachment2.pdf.
62
EPA, NAAQS Review, supra note 58, at 24.
63
Id. The planning report is thus not a trivial step in the process; EPA allocates almost a year to its
finalization and the planning reports alone are substantial in length – the particulates planning document
was eighty-five pages in length, for example. A draft planning report is reviewed internally and then by
CASAC and the public. Based on this feedback, EPA revises the planning document. For a sample
planning document, see EPA, Integrated Review Plan for the National Ambient Air Quality Standards for
Particulate Matter, March 2008, available at
http://www.epa.gov/ttnnaaqs/standards/pm/data/2008_03_final_integrated_review_plan.pdf. See in
particular pages 18-21 of id (stating policy-relevant questions for primary PM NAAQS that expand on the
excerpts provided above in the text).
64
It is hoped that “[i]n discussing policy-relevant issues, this plan could help clarify appropriate
distinctions between science and policy judgments and/or elaborate on important concepts and terms that
have both science and policy components.” Id. Examples of policy-relevant questions identified by EPA
for its particulate matter NAAQS process are:


Has new information altered conclusions from previous reviews regarding the plausibility of
adverse health effects associated with exposures to PM2.5, PM10, PM10-2.5, or alternative PM
indicators that might be considered?

32

2. Integrated Scientific Assessment Report
At the next step of the NAAQS review, EPA compiles an integrated scientific
assessment (ISA) of the existing scientific literature. This is effectively a review of all of
the scientific evidence bearing on the discrete policy questions identified in the Planning
Report.65
Generally, staff in the National Center for Environmental Assessment in the
EPA’s Office of Research and Development, supplemented by contracts with academics
for specific topics, identify and evaluate thousands of studies published since the prior
NAAQS review and then draft individual chapters of an ISA. The EPA staff interprets
and integrates the scientific evidence to draw key conclusions and the basis for the
conclusions. There are multiple points of review (at least three) from intra-agency
reviewers, CASAC and the public before the ISA is considered final. 66 While the
resulting ISAs are hailed as vastly more focused and concise than their predecessor
(criteria) documents, they still are quite long and technical, and EPA has included, with
CASAC encouragement, an Executive Summary in recent ISAs to aid communication of
the findings to policymakers and stakeholders. In at least some cases these reports are
more than 1000 pages in length, not counting the appendices.67
3. Risk/Exposure Assessment Report
Based on its analysis of the scientific evidence in the ISA, staff in the EPA’s
Office of Air Quality Planning and Standards then prepares a separate risk assessment
report that uses this evidence to model and predict the effects of alternate standards on
public health. While these risk assessments are constrained by the available air quality
data and concentration-response data, the goal at this stage of the process is to employ



What evidence is available from recent studies focused on specific size fractions, chemical
components, sources, or environments (e.g., urban and non-urban areas) of PM to inform our
understanding of the nature of PM exposures that are linked to various health outcomes?
To what extent is key scientific evidence becoming available to improve our understanding of the
health effects associated with various time periods of PM exposures, including not only short-term
(daily or multi-day) and chronic (months to years) exposures, but also peak PM exposures (less
than 24-hour)?

Id. at 18-19.
65
In stark contrast to EPA’s earlier version of this assessment in previous NAAQS processes (i.e., the
infamous “criteria document”), the new and improved integrated scientific assessment is more concise and
focuses the assessment on the specific questions framed in the planning report. More detailed information
is reserved for annexes, which can sometimes be longer than the body of the report. While the ISA and the
“criteria” document overlap significantly in content, the other two reports were not contained in the criteria
document, by and large, and hence go beyond the information that EPA provided in earlier approaches.
66
For a flowchart for production of the scientific assessment, see Figure 4.1 in EPA, Integrated Review
Plan for the National Ambient Air Quality Standards for Particulate Matter, March 2008, at 25, available at
http://www.epa.gov/ttnnaaqs/standards/pm/data/2008_03_final_integrated_review_plan.pdf.
67
See, e.g., EPA, Integrated Science Assessment for Particulate Matter, Final Report Dec. 2009, available
at http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=216546.

33

multiple models to produce quantitative risk estimates, accompanied by expressions of
the underlying uncertainties and variability, for various endpoints, such as the impacts of
a pollutant on susceptible populations and ecosystems. 68 The risk assessment process
itself begins with a planning/scoping stage, which again involves CASAC review and
public comment, followed by two more periods of intra-agency, CASAC, and public
comment on the draft risk assessment reports.69
4. Policy Assessment Report
The last document in the process is a policy assessment that “bridges” these more
science-intensive (ISA and risk assessment) reports to the policy questions at hand. The
policy assessment is, in and of itself, an extensive document (in the EPA’s review of the
particulate matter standard, the policy assessment was over 450 pages in length, including
appendices), but the discussion is written for nonscientists who do not have an extensive
background in the relevant science. 70
In this final staff report, the scientific literature is summarized in a way that
relates to the overarching policy questions. The report then offers alternative health
protection scenarios (and standards), as well as alternative welfare protection scenarios
and secondary standards, that are supported by the evidence and risk assessments. The
policy analysis also provides a discussion of remaining questions and key uncertainties
and identifies priorities for further data collection. 71
The policy assessment is typically reviewed by internal EPA staff, the public and
by CASAC twice to ensure it is faithful to the scientific assessments and that important
scientific information is not lost in translation.72 It is worth noting that even at this late
stage, CASAC review and comment is rigorous and extensive. For example, the second
CASAC review of EPA’s policy assessment for the review of particulate NAAQS
consists of over 70 pages of single-spaced comments.73
5. Inter-agency Review

68

See, e.g., EPA, Planning report for Integrated Assessment, supra note 66, at 41 (describing this goal of
the risk assessment).
69
See, e.g., id. at 54.
70
For a sample of a policy assessment, see EPA, Policy Assessment for the Review of the Particulate
Matter National Ambient Air Quality Standards, April 2011, available at
http://www.epa.gov/ttnnaaqs/standards/pm/data/20110419pmpafinal.pdf.
71
See Appendix C for an excerpt from a policy assessment report.
72
For a very brief summary of CASAC input, see id. at 2-100 through 2-101 (summarizing CASAC
advice).
73
For the second CASAC review of EPA’s policy assessment for particulates, see Letter from Dr. Jonathan
M. Samet, Chair, CASAC, to Lisa P. Jackson, September 10, 2010, available at
http://yosemite.epa.gov/sab/sabproduct.nsf/CCF9F4C0500C500F8525779D0073C593/$File/EPA-CASAC10-015-unsigned.pdf.

34

After completing this extensive analytical work-up, EPA officials select a
proposed standard and write a proposed rule. The agency’s proposed rule then goes to
OMB. During this process EPA meets with OMB and other agencies and apparently
receives comments, both written and oral, on its proposal.
According to EPA staff, the oral communications are protected by the deliberative
process privilege and are not made public, although interagency comments received in
writing may be posted in the rulemaking docket after the rule is signed.74 The idea
behind this deliberative process exception to the Freedom of Information Act (FOIA) is
to allow the free and uninhibited exchange of ideas and positions within government and
thus it is a discretionary claim. 75 Nevertheless, OMB currently appears to apply the
deliberative process exemption liberally. 76 In the course of this review,77 OMB not only
offers comments and suggestions on the EPA’s rule, but some EPA staff members
maintain that OMB determines the point at which a proposed or final rule responds
adequately to all of the comments and can exert significant influence over the content of
the rule.78
6. The Final Rulemaking Process
EPA generally hires a contractor to summarize the public comments, which are
collected in a Background Information Document (“BID”) document. EPA staff then
74

Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012. As discussed
infra Section III.A.4., in some cases an email exchange between EPA and other agency, most commonly
OIRA, is posted but the attachment that contains comments or edits is not posted. It is not clear whether
EPA would claim that this information is protected by the deliberative process privilege in a request under
the Freedom of Information Act.
75
Although it was initially a common law creation, the deliberative process privilege is most commonly
invoked as an exemption to FOIA, which allows an agency to withhold “inter-agency or intra-agency
memorandums or letter which would not be available by law to a party other than an agency in litigation
with the agency.” 5 U.S.C. § 552(b)(5); see generally Shilpa Narayan, Proper Assertion of the Deliberative
Process Privilege: The Agency Head Requirement, 77 FORDHAM L. REV. 1183 (2009) (describing the
history and developing of the deliberative process privilege over time).
The burden is on the government in this regard. As the Supreme Court has stated, FOIA’s “basic
policy of full agency disclosure unless information is exempted under clearly delineated statutory language,
indeed focuses on the citizens’ right to be informed about what their government is up to. Official
information that sheds light on an agency’s performance of its statutory duties falls squarely within that
statutory purpose.” U.S. Dept. of Defense v. Fed. Labor Relations Authority, 510 U.S. 487, 495-96 (1994).
Indeed, the courts will require release of withheld deliberative documents if they find that the public
benefits to disclosure outweigh the harm to the government. U.S. v. AT&T, 86 F.R.D. 603, 609 (D.D.C.
1979).
76
Nina Mendelson, Disclosing “Political” Oversight of Agency Decision Making, 108 MICH. L. REV. 1128,
1157 (2010) (finding no reference to OMB review, even though it occurred and changes were made as a
result); see also Stephanie Tatham, unpublished paper on OMB’s Assertion of the Deliberative Process
Privilege in Science-based Rulemakings (on file with author) (discussing OMB’s extensive use of
deliberative process privilege and finding that over 90% of OMB’s denial of FOIA claims invoked this
exemption); see also infra notes 484-485and accompanying text.
77
See infra Section =.
78
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012; Interview with
EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012. See also Section III.A.4., infra.

35

determine which comments warrant changes and which comments warrant discussion in
the preamble of the final rule.79 EPA staff also report that, after the notice and comment
period has concluded, OMB reviews the draft final rule, just as it did the draft proposed
rule. Again, OMB review may result in changes to the substance of the final rule. 80
Other characteristics of the NAAQS Process
Beyond these basic steps to EPA’s NAAQS decision-making process, there are
other features of EPA’s NAAQS process that deserve mention. These features bear on
both the rigor and transparency of EPA’s use of science in setting the NAAQS standards.
1. Attribution and authorship
One important feature of the NAAQS process is EPA’s reliance on staff to author
or in the case of the ISA, co-author with external experts, the analytical reports that
inform its process. Specifically, teams of EPA staff produce the four reports that lead up
to the proposed rule for a revised NAAQS. 81 While EPA management is briefed through
“information sessions” on the contents of these reports, there is no editing of the report by
management. (EPA staff could, however, point only to tradition and not to any rule that
explicitly precludes this type of editing.) 82 Indeed, at least some of the draft NAAQS
reports contain the disclaimer that the “findings, opinions, conclusions or
recommendations” reflect those of the authors and do not necessarily represent the views
of EPA.83
The team of staff authors is also listed by name in the acknowledgment section of
the final report. This detailed acknowledgement section links each EPA staff to his/her
specific contributions to individual chapters. An agency staff member has the right to
remove his/her name from this acknowledgement section if he/she disagrees with the
final version of the chapter (presumably the issue has not yet arisen). Although there is
dim recollection of this withdrawal occurring at least once, the fact that staff are allowed
to withdraw their names provide an indicia of authorship on the supporting NAAQS
reports.

79

Id.
See Section III.A.4., infra.
81
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
82
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012.
83
See, e.g., EPA, Draft Policy Assessment for Particulate Matter, September 2009, available at
http://www.epa.gov/ttn/naaqs/standards/pm/data/PreliminaryDraftPA091609.pdf. Note in this document
that the names of individual staff are not listed, however. This is different from the final report which
includes a detailed acknowledgement section that lists staff and reviewers by name and identifies their
specific contributions to the report.
80

36

2. Availability of the Supporting Literature
EPA is highly committed to making the literature upon which it relies publicly
accessible. In the revisions to its NAAQS process in 2006, EPA developed an elaborate
database (Health and Environmental Research Online or HERO) to make the scientific
research used in the NAAQS reviews accessible. Peer reviewers can access the entire
version of each of the tens of thousands of referenced studies used in NAAQS reviews
through this database, including copyrighted publications. 84 The public can access at
least summaries of these documents, as well as the full citations. 85 Providing this type of
fingertip access to the enormous library of research that supports the NAAQS allows for
more rigorous review of EPA’s scientific work by those inside the agency, CASAC, and
stakeholders.86
Summary and General Observations
The process that EPA has developed over the last forty years to review the
standards for the NAAQS appears to be highly regarded.87 As one interviewee stated, “It
is a process that delivers a credible standard.”88 Another EPA staff remarked that, based
on the current design, “I don’t know how we could be more transparent.” “In a general
sense, the process serves the agency well.” 89 The NAAQS process was also touted as a
model of scientific transparency in a recent National Academy of Science report that
suggested it should be adapted to other EPA programs.90
There are several features of the revised NAAQS process that offer useful insights
to other agencies working to incorporate science into their decisions in a rigorous and
transparent way. The first and perhaps most significant innovation is EPA’s decision to
break out the component parts of its analyses into separate steps (or in the case of
NAAQS, actual reports) that can be reviewed on their own terms.91 As just explained,
there is a planning document that identifies the policy-relevant questions; an assessment
of the available evidence in the ISA; an application of the evidence in the risk
assessment; and a translation of the evidence and models, and their limitations, in the
policy assessment.
84

Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012.
85
See the Health and Environmental Research Online (HERO) Database, last updated on February 8, 2012,
available at http://hero.epa.gov/.
86
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012.
87
See also Interview with Staff of American Chemistry Council, July 29, 2011.
88
Interview with EPA Staff, Office of Air and Radiation, July 26, 2011.
89
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
90
See NAS, FORMALDEHYDE REPORT, supra note 1, at 120-121.
91
See EPA, NAAQS Review, supra note 58, at 22 (see figure)

37

A second, innovative feature of the NAAQS process is the iterative involvement
of the CASAC, which is itself a prominent science advisory body. 92 In multiple, detailed
reviews of each of the four EPA reports, CASAC scrutinizes the assumptions and
alternate characterizations of the relevant scientific information. This provides rigorous
external review of the agency’s use of science. 93 CASAC also plays an important role in
determining when a revised report has adequately responded to comments, as elaborated
below.94
Third, the NAAQS process is developed in a way that offers multiple
opportunities (at least seven) for public comment. In such a process, there seems to be
less chance of group think or tunnel vision emerging in EPA’s assessment of the
evidence. Because of these elaborate outreach efforts, EPA presumably also encounters
few surprises during notice and comment on the proposed rule, at least to the extent that
the proposed rule follows from the staff reports. From the standpoint of stakeholders, the
iterative comment process on individualized stages of EPA’s analysis would seem to
make the commenters’ work more manageable and focused.
Fourth, EPA has instituted the equivalent of “stopping rules” 95 that allow it to put
an end to debate and close the record with regard to new scientific discoveries. 96 CASAC
historically acted as the referee on when debate could effectively close by declaring an
EPA report essentially ready for finalization after specific changes were made. 97 While
this “closure” role was criticized by some EPA staff members and temporarily
suspended, in at least one recent case it appears that CASAC still continues to identify
when it believes that a report is nearly complete and can be finalized and when yet
another review is necessary. 98 CASAC’s ability to determine a convenient point for
closing the record, or to set “stopping rules,” is taken up again in the analysis section.
92

Of all the science advisory bodies at EPA, CASAC seems to have consistently received the highest marks
in terms of its balance, leadership, and quality of its work in the science-policy literature. See, e.g., SHEILA
JASANOFF, THE FIFTH BRANCH: SCIENCE ADVISERS AS P OLICYMAKERS (1990) (discussing virtues of
CASAC in a chapter-length case study); MARK R. POWELL, SCIENCE AT EPA: INFORMATION IN THE
REGULATORY PROCESS 43 (1999) (reporting on how persons interviewed for the study on science at EPA
‘gave SAB and CASAC credit for improving EPA's acquisition and use of science‘); U.S. Envtl. Protection
Agency, Safeguarding the Future: Credible Science, Credible Decisions 38 (1992) (noting positive effect of
CASAC on EPA’s decisions).
93
CASAC, last updated on February 8, 2012, available at
http://yosemite.epa.gov/sab/sabpeople.nsf/WebCommittees/CASAC.
94
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
95
For a discussion of “stopping rules” see supra notes 45-46 and accompanying text.
96
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012; Interview with
EPA Staff, National Center for Environmental Assessment, Office of Research and Development, Jan. 18,
2012. Specifically, EPA will consider new studies up until the second ISA. After that point, all new
evidence will be bracketed and reserved for the next five year NAAQS review. Id.
97
Id.
98
Cf. CASAC Draft Letter to EPA regarding Second Draft of ISA for Ozone, 2/8/12, at 1, available at
http://yosemite.epa.gov/sab/sabproduct.nsf/264cb1227d55e02c85257402007446a4/FC9FAB17E2B06D4D
8525799E0069EF5B/$File/draft+ozone+SERD+ISA+letter-020812.pdf (recommending that EPA prepare
a revised draft of the ISA and submit it to CASAC for a third review opportunity).

38

EPA also has stopping rules for emerging science. If a new study emerges that is
relevant to NAAQS, but it is not available until after the draft ISA has been peer
reviewed, EPA will not consider it until the next NAAQS review five years later. 99
Fifth, the current NAAQS reports provide attribution to named staff through a
detailed acknowledgements section. The acknowledgement section identifies their
specific contributions to the report and, if staff members disagree with the contents of the
report they can ask that they not be acknowledged. 100 There was a sense among agency
interviewees that it would make little sense not to acknowledge authorship. Doing so is
believed to result in more robust and scientifically credible reports and to provide
scientific staff with deserved credit for their work.101
Sixth and finally, during the throes of its evolution, the NAAQS process shed
some light on the tradeoffs between scientific staff and political management in preparing
technical documents. One of these revelations occurred when EPA placed responsibility
for editing/finalizing the policy assessment with political rather than EPA staff. While
staff helped draft the policy assessment, ultimate editing and decision-making authority
was reserved for political management and the assessment was published as an Advance
Notice of Proposed Rulemaking, thus also requiring OIRA clearance.102 The resulting
policy assessment, prepared under the supervision of political management and reflecting
management’s views, was harshly criticized by both EPA’s Office of Research
Development and by CASAC as compared to the prior staff- authored reports of the same
type.103 CASAC in particular noted that this management-supervised policy assessment
departed from and arguably ignored scientific recommendations of CASAC; did not
connect the options suggested to the scientific literature; and presented options as equally
plausible, despite their very different scientific underpinnings. 104 In response to this

99

In cases of new scientific discoveries, EPA sometimes does prepare a “provisional science” report near
the end of the NAAQS review that considers whether certain new and emerging science would have
materially altered its analysis or conclusions. In cases where EPA has done this additional provisional
science report, it has consistently concluded that the new science would not materially alter its assessment.
One EPA staff scientist suggested that this is because the supporting studies are so numerous that additional
research is unlikely to significantly alter the assessment’s conclusions. Interview with EPA Staff, Office of
Air Quality Planning and Standards, Jan. 17, 2012; Interview with EPA Staff, National Center for
Environmental Assessment, Office of Research and Development, Jan. 18, 2012.
100
Exchange with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, March 11, 2012.
101
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012; Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan.
17, 2012.
102
Since the assessment was published as an Advanced Notice of Proposed Rulemaking, it required OMBclearance. See EPA, Advanced Notice of Proposed Rulemaking: National Ambient Air Quality Standards
for Lead, 72 Fed. Reg. 71488 (2007).
103
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
104
See Letter from Dr. Rogene Henderson, Chair of CASAC to Administrator Johnson, Jan. 22, 2008, at 1
(condemning the ANPR for lead prepared by EPA management as “unsuitable and inadequate” because it
“does not provide the underlying scientific justification for the range of options for standard-setting that the

39

controversy, Administrator Jackson ultimately determined that the policy assessment
should reflect the views of EPA staff, not political management.105 Employees and
CASAC report a high level of satisfaction with the change. 106 At least in this case, the
staff-authored report provided a more robust, nuanced, and complete statement of the
scientific information as it informs policy in comparison to a report that was within the
ultimate control of EPA’s political management.
In sum, there are a number of features of the NAAQS process that appear both
innovative and promising for science-policy analysis. 107 While extrapolating from this
expensive,108 time-consuming,109 and science-intensive NAAQS process to more
mundane science-based regulatory projects is difficult, the process still appears to serve
as a useful model for at least identifying the key steps necessary for the development of
rigorous and transparent science-based regulation.
b. The Review of Conventional Pesticide Registrations
Much like the NAAQS standard-setting process, the review of existing,
conventional pesticide registrations has undergone a major revision in the wake of the
1996 amendments to the Food Quality Protection Act. 110 Since its authorizing statute
now requires EPA to review over 1000 pesticides every fifteen years,111 the process EPA
agency is currently considering” and providing substantial details in the remainder of the letter regarding
these concerns).
105
See Jackson memorandum, May 21, 2009, supra note 59.
106
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012; Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan.
17, 2012.
107
Interestingly, none of these features of the NAAQS process was mentioned by the NAS panel in touting
it as a model for IRIS assessments. See NAS, FORMALDEHYDE REPORT, supra note 1.
108
Although cost estimates are not available, the staff suggested that the NAAQS revision process is likely
quite expensive. Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
109
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012; Interview with
EPA Staff, National Center for Environmental Assessment, Office of Research and Development, Jan. 18,
2012.
110
After a series of fits and starts, for more than two decades, EPA struggled to review and re-register
“grandfathered” pesticides that had been on the market at the time the Federal Insecticide and Fungicide
Act (FIFRA) was passed in 1972. The delays and unequal treatment of these old pesticides as compared to
new pesticides not only sparked harsh criticism of EPA’s pesticide program, but it prompted a series of
congressional amendments intended to rectify the delay. Congress now requires EPA to review all the
registrations (or licenses) of existing pesticides on the market every fifteen years. Specifically,
amendments to FIFRA in 1996 require EPA to develop “a procedure for accomplishing the periodic review
of registrations” and to ensure that “pesticide's registration [is reviewed] every 15 years.” Section 3(g) of
FIFRA, 7 U.S.C. § 136a(g). This statutorily mandated time frame is intended to ensure that pesticide
registrations will not fall too far behind scientific and technological developments and that EPA provides
all marketed pesticides with a rigorous review.
111
EPA, “Registration Review Highlights”, available at
http://www.epa.gov/oppsrrd1/registration_review/highlights.htm. Specifically, EPA is reviewing the
registrations of all pesticides registered at the time the law was passed, in August 1996. See EPA,
“Agrichemicals: Food Quality Protection Act – Pesticide Reviews,”
www.epa.gov/agriculture/factsheets/epa-305-f-00-006ag.html. This requires EPA to review more than 50

40

has employed to incorporate science into the revision process offers a useful point of
comparison with the EPA’s review of the NAAQS. To meet its deadline, the agency has
set a goal of completing a review of approximately 60–70 pesticide products a year, as
opposed to the review of about one NAAQS standard over the same time frame. The
statutory mandate is also more open-ended and gives the agency discretion to consider
both scientific and economic factors when making its decisions. Specifically, EPA may
register a pesticide if the EPA is satisfied that the pesticide “will not generally cause
unreasonable adverse effects on the environment.”112
Despite the very different workload, EPA’s incorporation of science into its
review of pesticide registrations tracks the NAAQS process in several important ways.
First, like in NAAQS, EPA’s pesticide office usually develops a series of separate reports
– a planning report, a risk assessment report, and a proposed decision – for each pesticide
reviewed. EPA also solicits public comment on each of these three documents. Finally,
like NAAQS, EPA’s analytical process stretches over five or six years for each
pesticide.113 In contrast to the NAAQS process, however, EPA does not engage external
peer reviewers for individual pesticide registrations except in highly unusual cases.

chemicals a year in order to stay on schedule to complete the reregistration of 1100 pesticides by 2022.
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011. See also EPA,
“Accomplishments under the Food Quality Protection Act,”
http://www.epa.gov/pesticides/regulating/laws/fqpa/fqpa_accomplishments.htm.
112
7 U.S.C. § 136a(c)(5)(D). The pesticide must also satisfy several other statutory conditions, but they are
not relevant to this Study. These additional requirements are that a) “its composition is such as to warrant
the proposed claims for it”; b) “its labeling and other material . . . comply with the requirements of this
subchapter”; and c) “it will perform its intended function without unreasonable adverse effects on the
environment.” Id.
Congress provided EPA with several additional tools to assist it in this science-based decisionmaking. First, EPA has the authority to require manufacturers to conduct additional testing needed to
evaluate a pesticide registration (termed “call-in” authority). Id. at § 136a(c)(2)(b). In the data call, EPA
can require manufacturers to submit (or, if necessary conduct) studies on a variety of effects, such as
toxicological and ecological effects. See, e.g., EPA Staff Background Paper #3.1, TRAC 5/27/98, available
at www.epa.gov/oppfead1/trac/dci.htm. Second, EPA may solicit expert scientific advice from EPA’s
standing Science Advisory Panel (SAP), established under 25(d) of FIFRA and chartered under FACA.
(For the 2006 charter of the SAP, see http://www.epa.gov/scipoly/sap/pubs/charter.pdf.) The SAP is
statutorily required to review notices of intent to cancel or reclassify pesticide regulations, as well as other
specifically identified regulations, but the Panel can also be used by EPA to evaluate other aspects of its
decision-making, including difficult registration review decisions.
113
Five or six years does not seem to be an unusually long time for the development of technically
complicated informal rule at EPA. IRIS assessments, as described Section III.A.1.a. infra, can take a
decade to finalize. MACT standards, which are technology-based emissions standards for air toxics
promulgated for various individuals sectors of industry, take about 5 and half years, on average, to
promulgate as final rules. See Wendy Wagner, Katherine Barnes, and Lisa Peters, Rulemaking in the
Shade: An Empirical Study of EPA's Air Toxic Emission Standards, 63 ADMINISTRATIVE LAW REVIEW 99,
144-45 (2011). And of course at other agencies the informal rulemaking process for science-intensive
regulations can stretch still longer. At OSHA, workplace standards are promulgated very slowly, if at all
and appear to stretch over years or even decades. See OSHA Website, “History of Health Standards and
Need to Revise PELs,” available at
http://www.osha.gov/pls/oshaweb/owadisp.show_document?p_table=PREAMBLES&p_id=768
(conceding that “OSHA has issued only 24 substance-specific health regulations since its creation [in

41

The Steps to the Review of Pesticide Registrations
There are four steps to EPA’s registration review process for each pesticide, three
of these steps involve the preparation of a staff report that is subjected to notice and
comment.114
1. The Planning or Summary Document. This document describes the existing literature
that bears on the pesticide under study and solicits additional information on it.115 The
Summary Document also identifies in specific terms the data EPA needs to complete its
assessment.116
2. The Data Call-in. EPA regulations establish the test protocols and risk assessment
models it will use to evaluate each pesticide registration.117 Since it is not uncommon for
some of the information needed to complete these assessments to be missing, EPA issues

1970]. It has not been able to review the many thousands of currently unregulated chemicals in the
workplace or to keep up with reviewing the several thousand new chemicals introduced since its creation”).
While there is no clear point for comparison at NRC, even the renewal of the license of a nuclear plant
stretches to nearly two years per license. See OIG, Audit of NRC’s License Renewal Program, Sept. 6,
2007, at 4. This offers a stark point of comparison with the FWS’s very short deadlines for listing and
habitat designation, discussed Section III.A.2, infra.
114
The registration review process assesses the risks of pesticides already registered and in use. Perhaps
because of this, the process is designed like an informal rulemaking process, without adversarial hearings
but instead relying on notice and comment at several distinct points in the process. See generally 40 C.F.R.
§§ 155.40-58 (setting out the pesticide registration review process).
115
This document is the rough equivalent of the planning document described above for NAAQS. In
contrast to the NAAQS planning document, the precise questions in need of public input are not always
identified explicitly or in an accessible way, although in some cases, the proposed work plan provides very
specific questions for the subsequent scientific analysis. See, e.g., EPA, Summary Report for Butylate,
June 2009, downloadable at http://www.epa.gov/oppsrrd1/registration_review/butylate/index.html. EPA
also provides a detailed fact sheet that summarizes the existing information available on the pesticide in the
Summary Document. Although not all of these summary documents include complete citations or
accessible links, they provide a basic overview of the information within the agency’s files that is relevant
to the pesticide review. For example, for Acephate, EPA’s summary report does not always contain
citations to the literature. There does not appear to be a link to all the underlying studies either. See EPA,
Summary Report for Acepthate, available at at http://www.regulations.gov/#!documentDetail;D=EPA-HQOPP-2008-0915-0003;oldLink=false. Stakeholders did not seem to feel handicapped by the inaccessibility
of the studies, even studies done by manufacturers that are not available. Interview with Pesticide Research
Institute (a public interest organization), Aug. 1, 2011. This may be attributed in part to limited time and
resources to review the studies, however. Finally, the work plan includes a timeline for the review.
116
For a general description of the Summary Report see EPA’s overview of the re-registration steps at
http://www.epa.gov/oppsrrd1/registration_review/reg_review_process.htm. After soliciting public
comment on the draft Summary Document, EPA prepares a final version of the Summary document or, in
some cases, simply prepares a short addendum that summarizes the comments and EPA’s responses. The
final Summary document is posted in a pesticide’s electronic file and serves as the work plan that guides
the registration review process for a pesticide.
117
For a web-based list of the test protocols, go to http://www.epa.gov/pesticides/science/guidelines.htm;
for pesticide models, go to http://www.epa.gov/pesticides/science/models_db.htm.

42

a “data call-in” to the manufacturers that requires them to conduct this additional testing
to fill the gaps. 118
3. The Risk Assessment. Once the additional data has been submitted by the
manufacturers, the EPA pesticide team conducts a risk analysis to assess the risks of the
pesticides through various exposure pathways, from drinking water to worker exposure to
food. The guidelines and models used for these assessment have been peer reviewed by
the Science Advisory Panel (SAP), which is an external EPA science advisory group
established by statute that EPA can use to review particularly important pesticide-related
decisions, 119 or an equivalent peer review body (e.g., the NAS). 120 The draft risk
assessment, prepared by staff, is again subject to public notice and comment, and in some
cases may be subjected to multiple public comment periods.121 The risk assessments can
also be quite lengthy – 1200 or more pages in some cases - although the length of the
assessment may vary considerably depending on the pesticide being reviewed.122
118

It appears from a cursory review of recent Summary Documents and staff interviews that the need for
some data call-ins in the course of a registration review decision is not unusual, and may be the norm.
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011. EPA’s data call-ins require OMB
clearance before they can be issued since they are considered “information collection requirements” under
the Paperwork Reduction Act, 44 U.S.C. § 3501 et seq. While this OMB clearance can involve
considerable transaction costs between EPA and OMB (for example, OMB currently reviews each
individual data call-in before allowing it to be sent), EPA reports that to date the agency has succeeded in
gaining authorization to request all of the data they deem necessary to conduct their risk assessment.
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012. Since OMB currently has decided
it must review the call-ins individually, the process of working through the Paperwork Reduction Act with
OMB can add as much as six months delay to the registration review and can involve considerable staff
resources, however. Id. After a data call-in is issued, there is occasionally a second process of negotiation
with the manufacturer with respect to satisfying the call-in demands. These settlements are similar to those
occurring in civil enforcement cases. Rather than taking a manufacturer to court for noncompliance with a
data call-in, EPA will agree to a somewhat different submission provided it generally meets EPA’s data
needs. Id.
Limited lab space and a variety of other factors constrain the manufacturers’ ability to produce all
of the requested data in a timely fashion. Manufacturers may also identify existing studies that appear to
resolve some or even all of EPA’s data needs. Interview with EPA Staff, Office of Pesticide Programs,
Jan. 27, 2012. EPA may alter its call-in request based on these discussions, while ensuring that all basic
data needs are satisfied. The resulting changes between what EPA initially requested in a call-in and what
it ultimately received is detailed in the draft Risk Assessment. For a sample, see Appendix A.1. of the EPA,
Chlorpyrifos Preliminary Human Health Risk Assessment for Registration, available at regulations.gov
(document ID number: EPA-HQ-OPP-2008-0850-0025); Interview with EPA Staff, Office of Pesticide
Programs, Jan. 27, 2012.
119
25(d) of FIFRA, 7 U.S.C. § 136w(d).
120
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
121
For a sample, see, e.g., multiple references to various risk assessments in Readers Guide for Methyl
Bromide, June 3, 2009, Docket EPA-HQ-OPP-2005-0123, available at regulations.gov; see also comments
filed on multiple drafts of the assessment by industry association, the American Chemistry Council, at
http://www.regulations.gov/#!documentDetail;D=EPA-HQ-OPP-2005-0123-0444.
122
The EPA team follows the NAS Silver Book in conducting their risk assessments, which includes
following NAS’s recommendations for explicating uncertainty factors, assumptions and other important
judgments. Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011. In a recent draft risk
assessment for Chlorpyrifos, for example, EPA provides a frank discussion of data needs, changes in
information over ten years, uncertainty factors, and other assumptions in the first 10 pages of the risk

43

4. Proposed Decision on Registration. The pesticide team uses the risk assessment to
draft a proposed decision with regard to registration of the pesticide. 123 The proposed
decision summarizes the scientific evidence, EPA’s assessment of the evidence, and
provides a proposal for re-registration, which could involve changes to the label; the
legally approved uses; partial or complete cancellation; or full registration of the
pesticide.124 This proposed decision is also subject to notice and comment. OMB is not
involved in reviewing these proposed rules, presumably because they are licensing
decisions.
5. The Final Decision. EPA ultimately issues a final decision that responds to the
information received through notice and comment.125 This final rule is signed by the
EPA director of the pesticide registration review program (a career staff position),
although in more complex or controversial cases, high levels of management, including
the Administrator, may be briefed on the decision. 126 Like the proposed rule, the final
rule does not require interagency clearance or OMB review.

assessment. Chlorpyrifos Preliminary Human Health Risk Assessment, supra note 118, at pages 2-16. It is
not clear whether this type of accessible explication is unique in the agency’s pesticide assessments, yet the
format and approach in this sample risk assessment has the flavor of the NAAQS staff policy and planning
reports with regard to more explicit and frank admissions of limitations in the available science.
123
For some pesticide reviews, it is not necessary to conduct a new risk assessment (e.g., if a pesticide is
withdrawn), in which case the staff skips the risk assessment step and moves straight to the proposed
decision. Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
124
This proposed decision is again subject to notice and comment. The format of the proposed decisions
generally provides a discussion of the available information and then offers EPA’s proposed decision based
on that information. See, e.g., outline in EPA, “Bromine Final Registration Decision,” at
http://www.regulations.gov/#!documentDetail;D=EPA-HQ-OPP-2009-0167-0010;oldLink=false. Unlike
the NAAQS policy assessment report, however, the proposed decision does not include a section that
bridges the scientific evidence with the overarching policy questions; instead the most important judgments
and assumptions would seem accessible only to experts. See, e.g., id. (providing a sample of this technical
quality of the assessments). EPA staff opined that the agency perhaps could do better in providing clearer
discussions of its exclusion/inclusion criteria, uncertainties, and other technical discussions in this proposed
decision, but they believe that much of these risk-related explications are provided in a clear format in the
earlier, risk assessment document. Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
Moreover, it would be costly for the agency to do this more elaborate discussion. Interview with EPA Staff,
Office of General Counsel, Jan. 13, 2012. The staff also reiterated that pesticide assessments are much
more complicated than most other risk assessments. In contrast to the NAAQS, for example, there are
multiple endpoints in pesticide assessments, which include assessment workplace risks, bystander risks,
risks on food, risks in drinking water, and a host of environmental impacts, including on plants, mammals,
birds, fish, and endangered species. Human exposures, moreover, occur by ingestion, inhalation, and
dermal exposure. Providing an accessible explanation of the estimates for all of these endpoints, with an
explication of the accompanying uncertainties and assumptions, would be extremely costly and timeconsuming. Id.
125
See 40 C.F. R. § 155.58.
126
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.

44

Other Characteristics of Pesticide Registration Review
As with the NAAQS process, there are other features of pesticide registration
reviews that deserve mention with respect to the integration of science into the decision.
1. Authorship
Much like the NAAQS process, a multidisciplinary team of EPA staff works
together on all phases of the registration process, and authorship rights are afforded to
these staff with respect to their analyses. Specifically, staff members are listed by name
in the front matter of each of the pesticide registration reports.127 Because EPA staff
members serve as authors, they both gain credit and bear responsibility for the quality of
the analysis presented in the document.128
While these documents are subjected to some intra-agency review before they are
shared with the public, they are not subjected to management-initiated edits or revisions
without the staff authors’ assent.129 EPA’s approach to authorship in pesticide reviews
thus seems to parallel the NAAQS process with respect to producing reports in a team or
consensus-based fashion.130
2. Accessibility of the Literature
Also like the NAAQS process, EPA endeavors to provide a comprehensive
bibliography of all the literature it relied upon in its draft risk assessment.131 While the
literature cannot be accessed through an internet database, as is the case with the HERO
database used in NAAQS, EPA provides complete citations where possible.132 EPA also
prepares web pages for each pesticide133 and a “readers’ guide” to the docket to walk the
reader through the various key decisions and stages of the registration review process. 134

127

See, e.g., Chlorpyrifos Preliminary Human Health Risk Assessment, supra note 118, at 1.
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
129
Id.
130
In contrast to NAAQS, in which the proposed and final rules are reviewed and potentially revised by
management, the proposed and final decisions for pesticide review decisions are subject to a highly
delegated decision making structure. The director of the conventional pesticide unit is the final signatory of
all registration decisions. Id. Higher level management of EPA is not in the line of command and their
assent is not necessary for pesticide approval.
131
See Chlorpyrifos Preliminary Human Health Risk Assessment, supra note 118, at 102-110.
132
See, e.g., id.
133
See http://www.epa.gov/oppsrrd1/registration_review/chlorpyrifos/index.htm for information on this
pesticide, including the summary document and all other reports and decisions.
134
See, e.g., Readers Guide for Acephate, March 18, 2009, docket number EPA-HQ-OPP-2008-0915-0002,
available on regulations.gov.
128

45

EPA does not make the manufacturers’ safety testing data and studies publicly
available in the course of its registration reviews because of a requirement in the Federal
Insecticide, Fungicide, and Rodenticide Act (FIFRA) (Section 10(g)) that limits general
public access to the data to ensure that the safety and related test data are not released to
pesticide manufacturers in other countries. 135 (The justification and continued necessity
of this requirement is not developed here except to note that questions have been raised
about its benefits, particularly in light of the costs to transparency with respect to basic
toxicity and related information regarding registered pesticides). To gain access to this
information, a person must certify that he/she does “not seek access to the data for
purposes of delivering it or offering it for sale to any such business or entity or its agents
or employees and will not purposefully deliver or negligently cause the data to be
delivered to such business or entity or its agents or employees.”136 FIFRA also requires
EPA to keep a record of the “names of persons to whom data are disclosed” and must
“inform the applicant or registrant of the names and affiliations of such persons.” 137 Most
problematic, the information cannot be accessed until after a registration decision. 138
Because it is partly classified information under FIFRA (there is public access, but it is
limited), the data must also be viewed in EPA offices. 139
In an effort to provide maximum transparency of this manufacturer data in spite
of the restrictions of Section 10(g), EPA does post summary tables of each of these
manufacturer-produced studies. It also employs teams of EPA scientists to ensure that
the studies are done well. 140 These summary tables provide outsiders with at least some
window into the nature of the scientific information supporting a decision.
3. External Peer Review
EPA does not routinely seek external peer review of its individual pesticide
registration decisions or supporting risk assessments.141 Instead, the opportunity for
135

7 USC § 136h(g)(1).
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012. The person seeking access must
certify that they are not an “employee or agency of any business or other entity engaged in the production,
sale or distribution of pesticides in countries other than the U.S. . .”. Id. Although this certification appears
light for most persons, there is evidence that EPA may be denying FOIA requests for this information. A
propublica investigation revealed that EPA denied 38 FOIA claims seeking this information from 20082009. Obviously further investigation is needed regarding the nature of these claims before it can be
determined whether there is a relatively substantial impediment to accessing this information. See Jennifer
LaFleur, FOIA Eyes Only: How Buried Statutes are Keeping Information Secret, available at
http://www.propublica.org/article/foia-exemptions-sunshine-law. For the specific investigation under
136h(g), go to http://projects.propublica.org/foia-exemptions/statutes/113.
137
7 U.S.C. § 136h(g)(2).
138
7 U.S.C. § 136a(c)(2)(A).
139
Id. at § 136h(g)(2).
140
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012.
141
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012. When asked whether additional
external review would be beneficial to EPA’s process, one staffer vigorously defended the existing
approach. Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012. The staffer argued that to
assemble peer review panels for each of the 1000 pesticide reviews would be very costly and would not
136

46

multiple points of public input, coupled with the interdisciplinary teams and intra-agency
review processes, are believed to be sufficient to provide rigorous oversight of the
agency’s scientific assessments and analyses. 142
EPA does consult its standing Science Advisory Panel (SAP) for particularly
difficult scientific questions, including registration cases, as in the reregistration of
atrazine. 143 Generally, though, these consultations with SAP over registration reviews
are the exception; indeed EPA staff could only recall one case where the SAP was
involved in a conventional pesticide registration review since the 1996 amendments to
the program. The SAP also reviews all models and methods used by EPA in its pesticide
reviews; in fact, this appears to be the primary source of external oversight over EPA’s
pesticide registration decisions. 144
4. Inter-agency Coordination
The role of other federal agencies in pesticide decisions is different from the
NAAQS in several ways. First, and as a point of sharpest contrast, OMB’s involvement
in pesticide registration reviews, according to EPA staff, has been limited exclusively to
its data call-in oversight under the Paperwork Reduction Act. OMB is not involved in
reviewing proposed decisions for registration, even in high level cases. 145
Also in contrast to NAAQS, USDA takes considerable interest in the review of at
least certain pesticide registrations and is likely to be an active participant in these
cases.146 USDA occasionally files comments on pesticide registration reports and
proposed decisions during notice and comment, although EPA staff report that USDA’s
involvement can also involve phone calls that are not recorded in the record. 147 Finally,
because pesticides potentially affect endangered species, EPA must formally consult with
the authorized agencies (National Marine Fishery Service and the U.S. Fish and Wildlife
have benefits that outweigh the costs of the panels, not to mention the potential added delays. Id. Panelists
with the greatest knowledge are often conflicted out of serving, and more general reviewers may lack the
needed familiarity with the research (much unpublished and provided by the manufacturer) to provide a
robust review unless they actually reviewed all of the studies themselves. Id.
142
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012.
143
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012. This consultation is iterative and is
still ongoing. See Atrazine Evaluation Process, available at
http://www.epa.gov/opp00001/reregistration/atrazine/atrazine_update.htm. More generally, EPA calls
upon the SAP for consultation rather regularly. One fact sheet suggests that EPA has consulted the SAP 58
times over the last ten years. EPA, “Accomplishments under the Food Quality Protect Act, Aug. 3, 2006,”
available at http://www.epa.gov/pesticides/regulating/laws/fqpa/fqpa_accomplishments.htm. For more
general information about the SAP, see http://www.epa.gov/scipoly/sap/basic.htm.
144
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011.
145
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011. The reason for this is not
altogether clear, although it appears to be a relic of the “Tozzi rule” that OIRA would refrain from being
involved in reviewing licensing decisions, which presumably sweep in these registration reviews that look
more like informal rules than adjudications.
146
Id.; Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011.
147
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012.

47

Service) if there is a possibility that a pesticide may adversely affect endangered or
threatened species. 148 This consultation step has been fraught with difficulty and delay
and has sparked lawsuits and congressional oversight hearings. 149
5. Transparency
Stakeholders were generally satisfied with the transparency of EPA’s pesticide
assessments.150 Despite their praise for the transparency of EPA’s reports, stakeholders
still identified numerous contestable points in EPA’s analyses. These disagreements with
EPA often focused on one of the following: EPA’s use of models (e.g., EPA ignored
factors that should have been considered and included, such as temperature, wind speed,
etc.); EPA’s use of uncertainty factors (e.g., the agency erroneously considered some data
and used it to adjust the uncertainty factors in contestable ways); or EPA’s explanation of
the results of studies (e.g., it misrepresented incident reports). These issues are
considered again in more detail in the analysis section.
c. EPA’s Integrated Risk Information System (IRIS)
The Integrated Risk Information System (IRIS) is a database of human health risk
assessments that calculates reference doses for inhalation and ingestion for human

148

16 U.S.C. § 1536.
A review of the correspondence to date between FWS and EPA on individual pesticide decisions reveals
credible and difficult technical disagreements over critical judgments that should be used in making these
assessments. See supra notes 20-27 and accompanying text. For example, FWS questions the viability of
models used by EPA to estimate risks to endangered species or to calculate their ranges, see Memo from
Marjorie Nelson, FWS to EPA, 2/11/2008, available at
http://www.epa.gov/oppfead1/endanger/litstatus/effects/atrazine/2008/fws-nonconcur.pdf, and EPA
questions the basis for various conclusions in the Biological Opinion about adverse effects resulting from
the use of pesticide products. See, e.g., Memo from Stephen Bradbury, EPA, to NMFS, 8/16/2010,
available at http://www.epa.gov/oppfead1/endanger/litstatus/effects/final-biop-ltr.pdf and Memo from
Stephen Bradbury, EPA, to NMFS, 6/14/2011, available at
http://www.epa.gov/oppfead1/endanger/litstatus/nmfs-draft-4-1comment.pdf. Some progress has been
made in bridging these different risk assessment approaches. In at least one set of pesticide reviews, EPA
adjusted its registration requirements in accord with these agencies’ biological opinions. See Memo from
Richard Keigwin, EPA, to NMFS, 7/10/2009 at http://www.epa.gov/oppfead1/endanger/litstatus/11-18-08nmfs-biop.pdf. Additionally, the agencies collectively sent the NAS a set of questions regarding their
differences. The National Academies, “Statement of Task: Ecological Risk Assessment under FIFRA and
ESA,” available at http://dels.nas.edu/resources/static-assets/best/miscellaneous/DELS-BEST-11-01statement-of-task.pdf. There is some hope that the Academy can offer illumination that will begin to
expedite future ESA consultations under FIFRA.
150
They indicated that they were generally able to trace EPA’s use of the available literature, understand
the assumptions it was making, and could recreate the agency’s analysis. In this regard, one industry
stakeholder noted that “you can generally figure out how EPA used the science and got from one point to
another.” Interview with Staff of Exponent, an industry consulting group, Dec. 20, 2011. The public
interest stakeholders generally agreed. For example, one public interest scientist said: “If you are a
toxicologist, you can generally understand what EPA did with the science.” Interview with Pesticide
Research Institute (a public interest organization), Aug. 1, 2011.
149

48

exposure to hundreds of chemicals. 151 The assessments are not binding regulations, yet
they can be and are used by EPA and other federal and state governments to support
regulation. For example, IRIS is used to assess residual risk in the air toxics program,
which in turn can lead to enforceable and costly emissions reductions from industry.152
Thus, while there is no statutory mandate under which EPA prepares these assessments
(and thus no mandated time frames or procedural or substantive directions), there is
nevertheless a great deal of interest in these risk assessments from a wide range of
stakeholders because of their use in various regulatory contexts.
Background of IRIS
In IRIS assessments, EPA attempts to identify a quantitative reference (roughly
analogous to what lay persons call safe) dose for inhalation and ingestion for over 500
chemicals through a risk assessment.153 The end goal for this reference dose is generally
the “no observed adverse effects level,” at which there is no biologically or statistically
significant evidence of adverse effects in an exposed group as compared with a control
group, divided by uncertainty factors to reflect extrapolation from animal data to humans,
interindividual differences in response to exposure to a pollutant, etc.154 This welldefined objective makes completing IRIS assessments more straightforward than
completing pesticide assessments, which must consider a number of different targets
(e.g., workers, consumers, bystanders, plants, animals, and ecosystems) and endpoints
(lethal and sub-lethal) that occur through multiple routes of exposure (e.g., water, air,
dermal). 155 IRIS assessments are also more straightforward than the NAAQS standardsetting assessments for the same reasons (although there are fewer exposure routes and
endpoints in NAAQS as compared to pesticides).
Over the last fifteen years, EPA’s IRIS assessments have been criticized on
several grounds. 156 First, EPA’s progress in producing assessments has been very slow,
averaging between six to eight years per assessment.157 In fact, in 2009 GAO listed the
IRIS program as one of the areas of high risk inside government for waste and
151

EPA, “Integrated Risk Information System: What is IRIS,” available at
http://www.epa.gov/IRIS/help_ques.htm#whatiris.
152
GAO, CHEMICAL ASSESSMENTS: CHALLENGES REMAIN WITH EPA’S INTEGRATED RISK INFORMATION
SYSTEM 6 (Dec. 2011).
153
More specifically, EPA is only assessing hazard and dose-response assessment and not exposure
assessment and risk characterization, and does not attempt to develop risk management information.
154
See, e.g, EPA, “Integrated Risk Information System: What is IRIS,” available at
http://www.epa.gov/IRIS/help_ques.htm#whatiris.
155
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012.
156
Some of these problems and EPA’s reforms are further elaborated in Testimony by John Stephenson,
Director of Natural Resources and Environment, GAO, Scientific Integrity: EPA’s Effort to Enhance the
Credibility and Transparency of its Scientific Process, June 9, 2009, available at
http://www.gao.gov/assets/130/122677.pdf.
157
Several stakeholders have criticized these delays, as has GAO in a series of reports. See, e.g., GAO,
CHEMICAL ASSESSMENTS, supra note 152, at 2-4 (summarizing this critical review), available at
http://www.gao.gov/assets/130/122677.pdf.

49

mismanagement.158 Second, IRIS assessments are subjected to two separate rounds of
interagency review, with the scientific assessment sandwiched between these interagency
stakeholder discussions.159 Finally, the fact that OMB has historically managed the
interagency review of IRIS assessment has led some to question the scientific credibility
of the resulting assessments.160
In response to these concerns, Administrator Jackson made changes to the IRIS
assessment decision-making process in 2009 with the dual goals of making the
assessment process more transparent and reducing the time spent on IRIS reviews to less
than two years per chemical. 161 Most significantly, as a result of these changes, EPA now
takes the lead on the assessments; all written interagency comments must be submitted on
the record.162

158

See GAO, HIGH-RISK SERIES: AN UPDATE, GAO-09-271 (Jan. 22, 2009).
See, e.g., GAO, CHEMICAL ASSESSMENTS: LOW PRODUCTIVITY AND NEW INTERAGENCY REVIEW
PROCESS LIMIT THE USEFULNESS AND CREDIBILITY OF EPA’S INTEGRATED RISK INFORMATION SYSTEM 57
(March 2008), available at http://www.gao.gov/assets/280/273184.pdf. There are also indications that
inter-agency review has influenced the assessment process, potentially significantly in some cases. See
GAO, CHEMICAL ASSESSMENTS, supra note 152, at Appendix III (describing on a chemical-by-chemical
basis the role of agencies like OMB and DOD on EPA’s assessments).
160
See GAO, LOW PRODUCTIVITY, supra note 159, at 56-58 (describing and criticizing OMB’s role in IRIS
assessments).
161
See Appendix E for the evolution of EPA’s current IRIS process through four separate flow charts. The
revised process focuses primarily on limiting interagency review to two discrete points in the process and
requiring that all comments from the agencies be formal and on the record, including comments from
OMB. See EPA, IRIS Progress report, Aug., 2011, at 4, available at
http://www.epa.gov/IRIS/pdfs/irisprogressreport2011.pdf; GAO, CHEMICAL ASSESSMENTS, supra note 152,
at 10. Time limits were also placed on these interagency reviews. Administrator Jackson also was able to
double the IRIS budget and add 25% more staff to the program. IRIS Progress report, supra, at 6.
162
See, e.g., id.
159

50

Steps to an IRIS Assessment
An IRIS assessment follows several separate steps:
1. Identification of relevant scientific evidence. The first task, assigned to a
contractor, is to collect all of the scientific research available on the chemical
being assessed.163 After the contractor completes its work, EPA issues a
Federal Register notice (“FRN” on flowchart) that announces the availability
of this contractor-prepared literature search for a chemical undergoing an IRIS
assessment and solicits from the public additional information relevant to the
assessment.164

163

See, e.g., EPA, NCEA Policy and Procedures for Conducting IRIS Peer Reviews 4 (July 30, 2009),
available at http://www.epa.gov/IRIS/pdfs/Policy_IRIS_Peer_Reviews.pdf.
164
See, e.g., EPA, Announcement of availability of literature searches for IRIS
assessments; request for information, 76 Fed. Reg. 13402 (2011). EPA does not provide a deadline for the
submission of this information. See literature assessments posted at
http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=187215. The literature assessments are
arranged by topic (e.g., chemical properties, toxicokinetics) but do not contain narratives; simply a listing
of relevant studies alphabetized by author. The literature assessments also generally cite to published
literature or literature with links, although some studies are not publicly available and presumably must be
acquired via FOIA. EPA is currently attempting to “HERO-ize” the IRIS literature database to make it
more accessible. Interview with EPA Staff, National Center for Environmental Assessment, Office of
Research and Development, Jan. 18, 2012.

51

2. Preparation of the Draft Risk Assessment. EPA then assembles an
interdisciplinary team (paralleling the FIFRA process) that, with considerable
help from the contractor, reviews the literature and prepares a draft hazard and
dose-response assessment that can include development of a reference dose
and reference concentration for non-cancer effects, and a weight of evidence
classification and potency estimates for cancer effects. . To provide a more
robust internal scientific review, the draft risk assessment is then subjected to
a staff level agency review using staff from appropriate offices and
disciplines.165 Based on this review, modifications are made, and the final
draft assessment is made a part of the public record.
3. Interagency Review. EPA leads a process of interagency review on the draft
assessment,166 which includes not only written comments placed on the record
but other discussions by phone and meetings that are not on the record.167
EPA sets deadlines for these comments, and the entire review is scheduled to
last no more than 45 days. 168 EPA then revises the assessment based on this
review.
4. External Peer review and Public Comment. After completing an open
meeting on the assessment, EPA subjects the revised draft assessment to
public comment and external peer review (the latter process is elaborated
below).169
5. Revised Assessment and Second Round of Interagency Review. Based on
input from the peer reviewers and the public, EPA revises the assessment and
subjects it to another round of internal peer review within the agency and then

While this first stage of the IRIS process – the assembly of relevant information -- is not as
elaborate as the planning documents under FIFRA or NAAQS programs, this step does provide an initial
opportunity for stakeholders and the general public to supplement the scientific record on a chemical. Note
that GAO recommends that EPA give two years of advance notice of chemical review to provide private
parties with opportunities to supplement the scientific record. GAO, CHEMICAL ASSESSMENTS, supra note
152. EPA has not yet implemented this step, but suggests it will. One can imagine both advantages and
disadvantages to this advanced notice.
165
The Chemical Manager, the Contractor, and these internal EPA reviewers are listed in the front of the
draft assessment. See, e.g., EPA, Toxicological Assessment of Acrylamide, March 2010, at xxi-xxiii,
available at http://www.epa.gov/iris/toxreviews/0286tr.pdf.
166
As mentioned, prior to 2009, OMB took the lead on soliciting interagency comments on the draft
assessment and the entire inter-agency dialog, including changes, would be protected as deliberative
process. See GAO, LOW PRODUCTIVITY, supra note 159, at 56-58.
167
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012. Other agencies comment not only on the draft assessment, but the draft charge
to the external peer reviewers. See, e.g., EPA, EPA’s Integrated Risk Information System: Assessment
Development Process, May 2009, available at
http://www.epa.gov/iris/pdfs/2009_IRIS_PROCESS_FINAL_05_19_09.PDF.
168
Id.
169
Id.

52

to interagency review. Written comments by other agencies, including OMB,
are again incorporated into the public record.170
6. Final Assessment. Based on these comments, EPA revises its draft
assessment and posts a final assessment.171
Characteristics of IRIS Assessments
1. The Transparency of IRIS Assessments
IRIS assessments are viewed as being less transparent than the NAAQS. This
may stem in part from the fact that IRIS assessments are technical and are often quite
long. For example, one of the recent assessments – on dichloromethane – was over 550
pages.172 More important than their length, however, is the perception that the
discussions are quite technical and not framed in ways that expose the major assumptions
and alternative choices for nonscientists. 173 At least some of EPA’s NAAQS reports, by
contrast, are purposely designed to communicate the results of EPA’s complex risk
assessments in a way that makes them accessible to non-scientists.
Concerns about the transparency of IRIS assessments were raised more concretely
in a spring 2011 NAS review of the EPA’s IRIS formaldehyde assessment. Specifically,
the panel observed that the assessments were unduly lengthy and repetitive and that it
was difficult to understand EPA’s assumptions and analysis at a number of points. 174 The
NAS panel did not, however, offer comments on the EPA’s overarching process for
developing assessments, such as its use of peer review, public comment, or interagency
review. 175
A recent GAO report conducted six months after release of the NAS
formaldehyde report provides a more favorable account of EPA’s efforts to make its
assessments transparent: “it appears that EPA has begun to enhance the readability of its
assessments by making changes that appear to be in line with the suggestions made by
170

Id.
Id.
172
See EPA, Toxicological Review of Dichloromethane (Methylene Chloride), Nov. 2011, at
http://www.epa.gov/iris/toxreviews/0070tr.pdf.
173
See id.
174
See NAS, FORMALDEHYDE REPORT, supra note 1, at chapter 7.
175
Id.; see also Section II.A., supra. In response to the NAS report, EPA agreed to make a number of
changes to its future reviews to make them more transparent. See, e.g., EPA, Progress Report, 2011, supra
note 161, at 11. In future assessments, EPA plans to include a short (15 page) executive summary and
short extended scientific summary (50 pages) to precede the longer report. Interview with EPA Staff,
National Center for Environmental Assessment, Office of Research and Development, Jan. 18, 2012. EPA
also intends to include graphics and other summary devices borrowed from the ISA of the NAAQS process.
Id. One of the first IRIS assessments in this new format is trichloroethene TCE. Id. EPA is also
investigating adding an early stage of external peer review and is developing a standing committee that
operates like CASAC. See, e.g., GAO, CHEMICAL ASSESSMENTS, supra note 152, at 44 (Letter from Paul
Anastas, Assistant Administrator of EPA to GAO, Nov. 22, 2011).
171

53

the National Academies.” 176 The GAO report was skeptical of EPA’s ability to complete
assessments within two years, however. 177 Indeed, much of the GAO report documented
continued delays in EPA’s assessment that nearly doubled the time originally allocated to
the projects. 178 GAO also indicated that EPA may be experiencing delays and turf
battles with OMB over the interagency review process, including the Data Quality Act.179
2. External Peer Review
As mentioned, after the draft assessment has been subjected to the first round of
interagency review, it is then subject to some form of external peer review in accord with
the EPA’s IRIS guidelines.180 The form of the external peer review depends on the
nature and extent of scientific evidence and the level of controversy. 181 External peer
review can be conducted by the NAS for highly controversial assessments; EPA’s
Science Advisory Board (SAB) for medium controversy assessments (this usually is
limited to about 4 chemicals/year);182 and contractor-selected individualized peer review
panels for the remaining IRIS assessments (which one EPA official estimated covered
about eighty percent of the total assessments).183 Regardless of which group conducts the
peer review, there are minimal standards for those serving as reviewers, such as standard
conflict of interest requirements adopted by the agency. 184 The peer review contractor
sets up and manages this external peer review when it is not conducted by EPA’s SAB or
the NAS.185
Of these three peer review processes, the most problematic (and the most
common, because the SAB has limited time to conduct reviews) is the review by external

176

See, e.g., id. at 22.
GAO noted that although EPA attempts to set rigid deadlines initially for these assessments, it appeared
to have limited authority to enforce them, particularly against other agencies. Id. at 18. In its recent report,
GAO also identified continuing, significant delays in IRIS assessments, although GAO also questioned
whether EPA’s two year goal was feasible, at least for moderately complex assessments. See GAO,
CHEMICAL ASSESSMENTS, supra note 152, at 27-28.
178
See, e.g., id. at 8
179
Apparently, several Data Quality Act correction requests have been filed on draft IRIS assessments. See
generally Rena Steinzor et al., Corrective Lenses for IRIS (Oct. 2010), available at
http://www.progressivereform.org/articles/IRIS_1009.pdf.
180
See IRIS Flowcharts at Appendix E. Note that for IRIS assessments, there will always be external peer
review. Presumably this ensures complete compliance with OMB’s peer review guidelines since even
potentially nonsignificant agency decisions will be subjected to external peer review.
181
The Chemical Manager in consultation with the NCEA Director and other career managers determines
which level of peer review is needed and prepares a publicly available peer review plan. See, e.g., NCEA
Policy Report, supra note 163, at 5-6.
182
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
183
See, e.g., GAO, CHEMICAL ASSESSMENTS, supra note 152, at 8.
184
See, e.g., id. at 5-6 and 7-10.
185
See id. at 7. For a list of “highly influential assessments” under IRIS, see EPA, Peer Review Agenda, at
http://cfpub.epa.gov/si/si_public_pr_agenda.cfm.
177

54

reviewers selected by an EPA contractor.186 Since the comments come from individual
scientists who have not conferred (this process of independent review is used in the
majority of cases where compliance with FACA appears too onerous), EPA finds that
responding to the comments can be quite challenging. 187 It is not uncommon, for
example, that one reviewer will offer suggestions on issues for which the other reviewers
are silent (thus leaving open the possibility that the other reviewers would disagree). In
addition, it is not uncommon for the reviewers to offer comments that conflict with one
another or to have reviewers focus on completely different feature of the assessment.
These problems are exacerbated because interagency review occurs after external
peer review and the other agencies can seize on comments that favor their own positions
and insist they be addressed. 188 The large range of individualized external review
comments thus provides fodder for even more debate and discussion during the second
stage of interagency review, with little hope of expeditious closure.
Thus, in contrast to CASAC’s consensus peer review process and role in
“closing” debate in the NAAQS process, the EPA faces a relatively large set of scientific
questions and queries that seem to actually expand rather than narrow with each round of
interagency review and peer review. EPA staff suggests that risk assessments for
chemicals like formaldehyde (recall this assessment was the subject of a critical NAS
report) are unwieldy and unfocused in large part because of this decision-making process.
By subjecting assessments to multiple rounds of stakeholder and interagency discussion
and depriving the agency of the authority to effectively “close” the debate, agency reports
can be filled with “facts that go nowhere” and “hypothetical analyses that don’t need to
be run.”189
3.

Interagency Review

EPA staff report that interagency review remains a significant feature of the IRIS
process.190 The GAO report and interviews indicate that other agencies take a keen
interest in EPA’s IRIS assessments because of their potential liability and compliance
costs, which can hinge on IRIS values. As a result, interagency review generally comes
much closer to supplementing the public comment from stakeholders rather than serving

186

For an example of external peer review comments, see the Consolidated Comments from External Peer
Review for Acrylamide, posted (with other items) at
http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=187729.
187
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
188
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
189
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
190
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012. .

55

as a form of expert peer review.191 In contrast to public commenters and even external
peer reviewers, however, EPA is expected – perhaps by OMB or as a matter of
interagency courtesy – to provide responses to all of these federal agency comments and
concerns in the assessment itself. Indeed, one EPA staff suggested that this need to
respond to the extensive interagency comments causes EPA to drill down into details at
the expense of dedicating staff resources to providing a strong, coherent draft on the most
important issues. 192
As the decision process outlined above indicates, EPA dedicates two separate
windows of time to interagency review – one before EPA’s draft assessment is released
for public comment and peer review and a second after both peer review and public
comment have concluded. In other agency decision-making processes, by contrast,
interagency review is merged with the public comment process. Moreover, although
OMB historically managed the interagency review process, EPA now manages this
review, and the agencies’ comments are placed in the record.193 Considerable
interagency review still takes place by phone and meetings, however, and these
discussions are protected as deliberative and are not on the public record.194
4. Access to the Literature
EPA posts a list of references for each of the chemicals going through the
assessment process. EPA also includes this reference list in its draft assessments, and in
the most recent assessments even includes hyperlinks to each study cited in the reference
list.195 Although it is still being developed, EPA is attempting to build a web database of
these studies, modeled after the NAAQS HERO database. When completed, this
database will facilitate public and peer reviewer access to the literature that forms the
basis for IRIS decisions. 196
5. Authorship and Attribution
The EPA and EPA-contractor authors, EPA internal reviewers, and external peer
reviewers are each listed by name in the first section of an IRIS assessment.197
191

Although some health agencies were initially involved in interagency discussions, they have largely
dropped out of the process. Id.
192
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
193
Cf.., GAO, CHEMICAL ASSESSMENTS, supra note 152, at 19 (NASA requested OMB to hold an
interagency workgroup to discuss EPA’s short comment deadlines).
194
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012. .
195
See, e.g., EPA, “IRIS Agenda and Literature Searches,” avaialable at
http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=187215.
196
See, e.g., EPA, “IRIS Progress Report 2011,” Aug., 2011, at 6, available at
http://www.epa.gov/iris/pdfs/irisprogressreport2011.pdf.
197
See, e.g., EPA IRIS Home Page, available at http://www.epa.gov/IRIS/ (providing links to recent
assessments with this attribution in the front matter of the assessment).

56

Attribution and authorship rights are considered a basic feature of the IRIS assessment
process.198 EPA staff report that this authorship gives staff credit for their work while
ensuring that they are accountable for the same. The list of internal and external
reviewers also gives these scientists a stake in the assessment. While not formally stated,
it is generally understood that, if the authors are not comfortable with the document, they
can remove themselves as authors and simply be acknowledged as contributing to the
report.199
2. The Department of Interior’s Fish and Wildlife Service:
Listing Endangered and Threatened Species and Designating
Critical Habitat
Although the U.S. Fish and Wildlife Service (FWS) engages in a number of
science-based decisions, 200 this study focuses entirely on the FWS’ species listing and
critical habitat determinations. Species and habitat designation decisions – the primary
tools for preventing species extinction – were selected in part because they attract more
public attention than other science-based decisions, and consequently more information is
available on them. These decisions also appear to be fairly representative of other FWS
science-based processes with respect to how science is incorporated into policy. 201
The approach to science-policy decision-making used by the FWS to list species
and critical habitat under the Endangered Species Act 202 is a very different approach from
that of the EPA. As described above, the NAAQS reviews begin with staff-authored
documents (with attribution to individual scientists in the acknowledgement section) that
undergo multiple points of contact with the public and peer reviewers before the
proposed rule is published.
The FWS’s approach to species listing and habitat designations, by contrast, is
much more abbreviated, limits authorship, and is less transparent. The FWS’s
assessment of the evidence, its technical analysis, and its decision in terms of applying
the policy factors to the scientific evidence is contained in a single proposed rule that is
drafted by staff and management working together. In addition, instead of listing
individual scientists as authors, attribution for the technical analysis is generally given to

198

Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
199
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
200
These include five-year reviews of threatened and endangered species, emergency listings, the
development of recovery plans, interagency consultations regarding jeopardy, delisting species, and the
development of habitat conservation plans.
201
It should be noted that the OIG report of Julie MacDonald identifies irregularities across a number of
different programs. See generally OIG, MacDonald Report, supra note 11. The discussions of transparency
problems in the listing and habitat designation decisions thus seem to carry over to these other areas.
202
16 U.S.C. § 1533.

57

a field staff office.203 Furthermore, virtually all internal drafts and assessments
supporting this single decision document, with a few exceptions discussed below, are
classified as deliberative privilege until the proposed rule is published. 204
Endangered Species Act (ESA) Listings and Critical Habitat Determinations: The
Law
The FWS and the National Marine Fisheries Service (NMFS) (which was not
included in the study), are responsible for determining whether a species or subspecies is
endangered or threatened under the Endangered Species Act.205 If based “solely on the
basis of the best scientific and commercial data available,” 206 a species is threatened or
endangered with extinction (or likely to be so) “throughout all or a significant portion of
its range,” then the FWS must list the species. 207
In making this determination, FWS is statutorily required to consider five factors,
any of which can trigger listing. 208 Much like the NAAQS determinations, the FWS is
prohibited from considering economic impacts in listing species.209 The designation of
critical habitat, however, must take economic and related considerations into account as
required by statute.210

203

See Attribution subsection on FWS, infra.
This truncated approach to listing is an explicit policy of the FWS. As the Director instructed staff in
2006:
Premature release of drafts, scientific information or briefings can significantly
undermine the confidence in the process by the public (through the Administrative
Record) as well as our ability to have free and open debate on data interpretation. Failure
to maintain a culture of "in Service scientific debate" prior to forming conclusions can
significantly undermine the credibility placed with the science as we and the Department
engage in policy or decision-making discussions. In order to ensure the integrity of this
process, it is imperative that all documents, assessments and drafts remain inside the
Service, except for discussions as appropriate with your recognized federal and state
peers. Any requests for such release or premature briefings should be forwarded to this
office for appropriate action.
Memorandum from FWS Director H.Dale Hall to FWS Directorate, Feb. 3, 2006, at page 1,
available at http://www.fws.gov/endangered/esalibrary/pdf/Science_in_DecisionMaking_2_3_06.pdf.
205
The split jurisdiction is based on the different expertise of the staff. The FWS has jurisdiction over
freshwater and land animals. NMFS has jurisdiction over marine and anadromous animals. See, e.g.,
http://www.nmfs.noaa.gov/pr/laws/esa/ (describing this jurisdictional split).
206
16 U.S.C. § 1533(b)(1)(A).
207
Id. at § 1532(6).
208
Id. at § 1533(a)(1)(A)-(E). The 5 factors are: a) “the present or threatened destruction, modification, or
curtailment of its habitat or range;” b) “overutilization for commercial, recreational, scientific, or
educational purposes;” c) “disease or predation;” d) “the inadequacy of existing regulatory mechanisms;”
or e) “other natural or manmade factors affecting its continued existence.” Id.
209
Id. at §1533(b)(1)(A).
210
Id. at § 1533(b)(2) (in making critical habitat designations the Secretary shall take into consideration
“the economic impact, and any other relevant impact, of specifying any particular area as critical habitat”).
204

58

Steps for Listing and Habitat Designations
A listing/habitat decision typically follows a three-step process, with each step
published in the Federal Register:
1) A 90-day decision on a petition. Listing analyses are often triggered by citizen
petitions that demand that a particular species be listed as a threatened or
endangered species. 211 The FWS has only ninety days by statute to determine
whether a petition presents “substantial scientific or commercial data that the
petitioned action may be warranted.”212 The FWS’s decision typically involves a
relatively extensive scientific analysis that relies largely on evidence submitted by
a petitioner.213 The FWS details its analysis with respect to each of the five
statutory factors and publishes its full analysis and decision in the Federal
Register.214
2) A 12-month finding that listing is not warranted or that proposes listing and
solicits notice and comment on that proposal. After completing the 90 day
decision the FWS then has twelve months (by statute) to issue a proposed finding
with regard to whether a listing is warranted.215 The analysis in the 12-month
finding (whether it be to list or not to list) is based on a comprehensive analysis of
the best available evidence. 216 This analysis in the proposed rule is conducted by
staff and management working together.

211

In recent years, the FWS reports that its priorities are largely driven by petitions filed by nonprofit
groups for the simple reason that the agency’s resources are very limited and the agency’s response to
petitions is statutorily required. Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
212
16 USC § 1533(b)(3)(A); 50 CFR § 424.14(b). This is a lighter evidentiary standard than a conclusion
(during the next review) that listing is ultimately warranted based on the “best scientific and commercial
data” standard. Id. at § 1533(b)(1)(A).
213
In ruling on this petition, the FWS relies primarily on “the information provided by the petitioner”,
FWS, Endangered Species Petition Management Guidance 10 (July 96), available at
http://www.nmfs.noaa.gov/pr/pdfs/laws/petition_management.pdf, with attention to ensuring the reliability
of that information, id. at Appendix A, but it also considers “information already available in the Service’s
files.” Id. at 10.
214
16 U.S.C. § 1533(b)(3)(C)(ii). For a sample, see FWS, Endangered and Threatened Wildlife and
Plants; 90-Day Finding on a Petition To List the Humboldt Marten as Endangered or Threatened, 77 Fed.
Reg. 1900, 1904-08 (2012) (undertaking this analysis). At this point, the FWS will also include a notice in
the same published decision that solicits from the public information on the species and its habitat, often
including very specific questions for which it seeks information. Id. at 1900-01. Much like the approach in
all the other agencies, this initial solicitation of information is intended to ensure that the agency’s files are
as complete as possible.
215
16 U.S.C. § 1533(b)(5)(A)(i). A six month extension can be justified based on “substantial disagreement
regarding the sufficiency or accuracy of the available data.” Id. at § 1533(b)(6)(B)(i). The FWS has been
held to these timelines in litigation by nonprofit group. For a list of judicially imposed deadlines for critical
habitat designations, see http://www.fws.gov/endangered/esa-library/pdf/ch-actions.pdf.
216
16 U.S.C. § 1533(b)(1)(A).

59

If the FWS concludes that listing is warranted,217 then its decision takes
the form of a proposed rule upon which it solicits public comment. As described
below, with the help of a contractor, the FWS also solicits external, independent
peer review on its proposal. In the same proposed rule, the FWS is required by
statute to propose the designated critical habitat for the species. 218 Although the
listing proposal does not require OMB review, the critical habitat designation
must be cleared by OMB.219
3) A final rule. Like the proposed rule, the final rule is the result of a collaboration
between staff and management within the FWS and the Department. Also like the
proposed rule, for critical habitat designations the final rule must be cleared by
OMB.

217

If the FWS concludes that listing is not warranted or is warranted but precluded, the FWS issues its
decision as a finding without notice and comment. A negative decision (i.e., “not warranted”) is considered
a final action subject to appeal. Id. at § 1533(b)(3)(A). The FWS will conclude that a listing is not
warranted if “convincing data on biological vulnerability and threat are not available to support a proposed
to list.” Petition Guidance, supra note 213 at 13-14. By contrast, listing is warranted when there is
“convincing evidence” in its favor. Id. There is also a third, “warranted but precluded” category in cases
where there are inadequate resources to list relative to other, higher priority species. Id. Species falling
into this category are reconsidered annually. Id.; see also 16 U.S.C. § 1539(j)(2)(C)(i). For the priority
system that the FWS has developed, see FWS, 1983 Priority Guidelines, 49 Fed. Reg. 43098 (1983). The
cutoff for where how these priority rankings affect whether a species is in the warranted but precluded
category are variable and depend in part on workload and resources. See Stanford Environmental Law
Society, The Endangered Species Act 47-49 (2001) (describing this feature in more detail with examples).
The statute is read to create a presumption in favor of listing. Interview with FWS Staff, Endangered
Species Program, Jan. 9, 2012.
218
The critical habitat designation includes areas considered necessary to protect survival at status quo
levels and does not include what may be necessary for recovery. 16 U.S.C. § 1532(5); 50 C.F.R. §
424.12(b). As mentioned, critical habitat designations must include economic considerations to the extent
that the habitat extends beyond a biologically determined “core” area essential to prevent extinction. 16
U.S.C. § 1533(b)(2). Although habitat designations are required to be issued with the species listing
decisions, the statute does allow the FWS to delay the decision if the designation at the time is “not
determinable” and to avoid it altogether if the designation is “not prudent.” Id.
219
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.

60

From FWS, Listing a Species as Threatened or Endangered, June 2011.

General Characteristics of Listing and Habitat Designations
1. Public access to technical assessments supporting FWS decisions
As mentioned, the FWS follows an abbreviated process for incorporating science
in which both management and staff work together to produce the analysis that supports a
decision.220 The proposed rule and accompanying technical documents that comprise the
220

An email from Dale Hall, the FWS Director in 2006, posted on the FWS site in its ESA policy library
fleshes out the agency’s expectations regarding this collaboration between staff and management in
formulating listings and other decisions in the FWS. See email from Dale Hall to FWS Directorate and
Deputies, dated Feb. 8, 2006, available at http://www.fws.gov/endangered/esalibrary/pdf/Directions_for_Directorate.pdf. In listing and related decisions, the Director will become
involved early in the process. “[A] written briefing” to the FWS director is required just at the point that
the “field is beginning to write the document”. “The real value here is to give advice and suggestions to the
field so they can assist in providing information in the draft to answer expected questions.” Hall’s
directions conclude with:
“(6) The discussions between you in the A/S [Assistant Secretary] office and me will
focus on policy direction or policy decision-making. Identification of other weaknesses
in the draft are welcomed, but will be given to me as the responsible person in the
Service to make necessary corrections or improvements. . . . This will be tricky until we
get better at it, but we will keep working it until a solid process emerges.”
In this directive, Hall is attempting to separate and protect the field’s scientific analysis from the
policy decisions made in the Assistant Secretary’s office based on that analysis. See OIG MacDonald
Report, supra note 11, at 15 (citing FWS employee as saying that Hall drew a “line in the sand” that

61

listing and critical habitat packages originate with field staff but are then reviewed and
revised by a number of regional and Washington offices and ultimately are sent to the
DOI Office of the Solicitor, the FWS Director, and the Assistant Secretary for Fish,
Wildlife, and Parks.221
In theory, the extensive review of the proposed rule by management creates
opportunities for the scientific analysis to be altered, although whether or how much this
occurs cannot be determined. An IG report produced to investigative the misuse of
authority by Assistant Secretary Julie MacDonald did criticize the FWS for not having
guidelines relating to how scientific evidence and technical analysis should be
incorporated into decisions during this internal review process.222 In the case of critical
habitat designations, for example, the IG identified 14 different versions of a FWS policy
that purported to guide critical habitat designations over a three year period (between
2003 and 2006).223 Because the guidelines to FWS personnel on how to assess and report
on scientific evidence are so ambiguous, the IG concluded that the transparency and
scientific rigor of the FWS’s supporting analyses were at risk of being compromised.224
Although the internal deliberations of the FWS are protected as deliberative
process before the proposed rule is published, after the rule is published, the FWS does
typically make the administrative record available from its field office.225 A FWS staff
MacDonald could not change the science coming from the field). To the extent that this approach prevails,
it would seem to improve the scientific integrity of the decision-making within the Department. Yet the
informality of Hall’s directions (an email) leaves open the possibility that future FWS Directors may take
the opposite tack. [Perhaps this could form the basis of a recommendation to FWS? Perhaps they should
formally embrace this policy.] The Hall email openly concedes in fact that “There is almost never one
clear answer to ESA, FERC or other questions, and our objective is to ensure we have as clear of an
understanding of the range of options as we can have.” Hall, supra. Yet how those range of options is to
be expressed as against the evidence and interpretation of that evidence is not spelled out.
221
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
222
Assistant Secretary MacDonald allegedly bullied staff and manipulated the scientific record in order to
undermine recommendations for listing; as a result of her activities, a number of proposals for listing were
dropped. In investigating her case, the Office of the Inspector General concluded that the FWS’s own lack
of guidelines was partly to blame for her ability to abuse this authority. OIG MacDonald Report, supra
note 11, at 7. Specifically, the OIG found:
While the ESA affords the Secretary great discretion in several areas – exclusions of
habitat being one example – the absence of policy guidelines in exercising that discretion
has resulted, in MacDonald’s case, in a wholesale lack of consistency, a process built on
guess-work, and decisions that could not pass legal muster. [Indeed, the resulting
ambiguity created “an enormous policy void, which MacDonald was able to readily
exploit.”] This dearth of policy and guidance seems less than coincidental. For many
years, through several administrations, this appears to be an area of intentional failure to
clarify, in order to maximize the agenda du jour.
OIG MacDonald Report, supra note 11, summary letter at 2, available at
http://www.lb9.uscourts.gov/webcites/10documents/HomeBuilders_conflict.pdf
223
OIG MacDonald Report, supra note 11, at 130.
224
Id.
225
No documents, except for a possible bibliography and two critical habitat reports, required under NEPA
and Executive Order 12866, are posted online. [It is not quite clear how Executive Order 12,866 requires
this. Please elaborate.] One of the two critical habitat reports is a supporting environmental assessment

62

member indicated, however, that there can be considerable variation in how these records
are prepared. To the extent that there is guidance to assist with this task, it is badly
dated.226 Thus some administrative records that support listing will be carefully prepared
to include the full range of documents that led up to the FWS’ proposed rule, including
significant changes and discussions. 227 Other administrative records may be less
comprehensive, mostly because drafts are not saved or collected and the record is
generally prepared at the end of the process. 228 A staff member also indicated that the
agency’s approach to claiming deliberative process protections for internal documents
has varied over time and generally lacks clear guidance. 229
2. External Peer Review
The FWS voluntarily solicits independent peer review relating to its proposed
listing decision from at least three independent specialists. 230 While the reviewers are
invited to comment on any issues in FWS’s analysis, they are asked in particular to
provide opinions on the species’ taxonomy and biology. 231
The FWS’s decision to solicit external peer review on its species listing decisions
is the result of formal FWS policy, but it is not legally required. 232 Perhaps in part
because it is discretionary, FWS’s decision to use external peer review for listings, as
well as its selection of reviewers, has been controversial. The FWS’s actual selection of
reviewers in individual cases has been criticized by members of Congress, who, for
example, argue that the FWS selects reviewers in ways that stack the deck in favor of a

that explores the consequences of the critical habitat designation, required under NEPA. The second is a
draft economic assessment of alternative critical habitat designations. The economic assessment of critical
habitat designations not only informs the Secretary regarding the benefits and costs of alternative habitat
designations as required by the ESA, but also allows the Service to address the requirements of Executive
Orders 12866 and 13211, and the Regulatory Flexibility Act (RFA), as amended by the Small Business
Regulatory Enforcement Fairness Act (SBREFA). Both of these additional assessments are made available,
with the proposed rule, for public viewing and are considered part of the larger package subjected to notice
and comment. It is not clear whether they must also pass through the same proposed rule review chain or
whether they are considered the equivalent of staff documents that support a larger critical habitat
designation.
226
Interview with FWS Staff Member, Field Office, Feb. 15, 2012.
227
Id.
228
Id.
229
Id.
230
See, e.g., Joy Nicholopoulos, The Endangered Species Listing Program, 24 ENDANGERED SPECIES
BULLETIN, Nov./Dec. 1999, at 5, 9.
231
Id. at 9.
232
FWS, Endangered and Threatened Wildlife and Plants: Notice of Interagency Cooperative Policy for
Peer Review in Endangered Species Act Violations, 59 Fed. Reg. 34270 (1994). A search through past
FWS proposals reveals that this peer review is also occasionally sought for critical habitat designations. See
FWS Letter soliciting external peer review on the Gopher Frog, Nov. 11, 2011, Document Number FWSR4-ES-2010-0024-0024, available at regulations.gov.

63

desired outcome.233 Other stakeholders question whether peer review is even necessary
at all and have expressed concern that it might reduce the FWS’s discretion, delay the
listing process, and tax the limited resources of staff. 234 According to FWS officials,
however, peer review has been helpful and provides a robust check on the FWS’s
scientific and technical analyses.235 This peer review can occasionally alter the FWS’s
analysis in substantive ways. 236
3. Attribution and the Release of Staff Reports
Attribution for the FWS’s technical analysis is given to a field office at the end of
the Federal Register publication (e.g., “The primary authors of this notice are the staff
members of the [fill in the blank] office.”). According to FWS staff, there is no
individualized attribution or scientific authorship for the analyses that support the
proposed rule, as is the case at EPA.237 This is likely due in large part to the collaborative
nature of the FWS’s analysis and the fact that it is reviewed and edited by multiple
offices as a draft proposed rule.238
4. Bibliography and Public Access to the Supporting Literature
The extent to which the bibliographic information on which decisions are based is
publicly available is inconsistent. For example, while references in abbreviated form are
cited in the FWS’s Federal Register preamble, the full bibliography is not published in
the proposed rule and is not always posted on regulations.gov.239 In cases when the
bibliography is not posted, commenters are invited to contact the designated contact
person in the appropriate field office. It appears that external peer reviewers must follow
this same process to acquire the literature supporting the FWS’s analysis. 240

233

Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012. [Can we cite to something other
than an interview for this point? Couldn’t we cite to the legislative record or some other source to
document this criticism?]
234
See STANFORD ENVIRONMENTAL LAW SOCIETY, supra note 217, at 53 (reporting on this).
235
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
236
Id. The FWS ultimately revises the proposed rule and reopens public comment after explaining the
changes it made in response to peer review. See, e.g., FWS, Endangered and Threatened Wildlife and
Plants: Designation of Critical Habitat for Mississippi Gopher Frog, 77 Fed. Reg. 2254 (2012).
237
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
238
Id.; Interview with FWS Staff Member, Field Office, Feb. 15, 2012.
239
See, e.g., Proposed Endangered Status for the Chupadera Springsnail (Pyrgulopsis chupaderae) and
Proposed Designation of Critical Habitat, 76 Fed. Reg. 46218, 46233 (2011) (guiding readers to
regulations.gov for the bibliography).
240
In at least one solicitation to peer reviewers pulled from a docket, the reviewers were given only the
published federal rule. See, e.g., Letter to Peer Reviewers for the Gopher Frog, supra note 232, at 2
(advising reviewers that “[a] list of the References Cited in the proposal- Copies of the references cited are
available from our files if you wish to review them.”).

64

In the bibliographies that were available online, 241 only some of the cited
literature was published. Other references included unpublished studies and internal
memoranda. Presumably these documents would also need to be requested from the field
office that serves as the designated contact point for listings. 242
5. The Role of OMB and Interagency Review
As discussed, OMB does not review listing decisions, 243 but decisions regarding
critical habitat designations do require OMB clearance.244 As with OMB’s review of
EPA’s proposed rules, the internal deliberations are considered deliberative process and
are therefore privileged, although the changes that result from formal review are required
by Executive Order 12866 to be disclosed to the public.245 Note that while critical habitat
designations are often published in the same proposed rule as species listings, these two
elements of the rule are discussed separately in the preamble. Therefore, it is presumably
not difficult for these two types of decisions to be separated for the purposes of OMB
review.
3. A Birds’ Eye View of the Nuclear Regulatory Commission’s
Incorporation of Science into its Regulatory Decisionmaking
The Nuclear Regulatory Commission (NRC) provides yet a third point of contrast
to the FWS and EPA regulatory programs. As discussed in Section II, supra, the primary
reason that the NRC was selected for study is that the NRC is independent and thus
presents a slightly different regulatory process. Rather than responding to Executive
Branch appointees and undergoing mandatory OMB clearances (as well as other methods
of coordination), the NRC is governed by a five member board of Commissioners
appointed by the President for five year, staggered terms. No more than three of the
Commissioners may be affiliated with the same political party. 246 As a consequence, the
NRC is a bipartisan body. The internal politics and external mechanisms of pressure at
the NRC are thus different than at an Executive Branch agency.
241

See, e.g., Reference List for the Chupadera Springsnail, Document ID FWS-R2-ES-2011-0042,
available at regulations.gov.
242
Given the limited timeframe and resources of the FWS, it might not be possible to make supporting
documents more readily available. However, in a policy email from Dale Hall to the FWS Directorate,
supra note 220, point 3 at page 1, Hall indicates that: “If literature cited in the document is in electronic
form in the field office, that will be forwarded with the draft [to the Director]. If not, we agreed that an
intern could be assigned to find the citations and either print them off or put them in electronic form.” It is
not clear why this internally collected research cannot be made available in the docket, but this was not
asked of the interviewees in this study.
243
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012. Officials in FWS were not sure
why OMB refrains from reviewing species listing decisions.
244
Id.
245
See infra Section III.A.4.
246
See Energy Reorganization Act of 1974, Sec. 201.

65

The NRC’s technical decisions are generally based on more limited scientific
information than the information upon which EPA and the FWS typically rely. For
example, the NRC regulatory issues often concern technical issues and modeling
assumptions that require considerable engineering expertise. 247 Grounded operational
experience is also viewed as a necessary disciplinary perspective to include on most
expert advisory groups because the technical issues can be closely tethered to specific,
applied issues and challenges. 248
Finally, NRC’s statutory mandates generally do not impose many deadlineforcing requirements on NRC. Consequently, the NRC is not subject to the deadline
pressure that appears to be a significant factor in EPA’s NAAQS and pesticide reviews
and in FWS’s ESA determinations.
Roland Frye, a Senior Attorney at NRC, was conveniently on detail to ACUS
during the preparation of this report. Mr. Frye contributed most of the information
detailed in this and the next section on NRC. Specifically, Mr. Frye conducted all of the
interviews with the NRC personnel and contributed virtually all of the documentation and
other substantive information used in this report on informal rulemakings and on NRC’s
scientific integrity programs. Mr. Frye also prepared two stand-alone white papers on
NRC’s use of science advisory bodies and its use of expert elicitation, which are
available in Appendices A and B. Any errors in transcribing and analyzing this
information for purposes of the report, however, are attributable solely to the author.
a. Informal Rulemakings
The the NRC engages in a diverse set of rulemakings and licensing decisions, but
most of the NRC’s regulatory activities involve some form of oversight or restrictions on
nuclear operations, such as waste disposal, worker safety, or operations and
maintenance.249 This study placed primary, but not exclusive, emphasis on examining
NRC’s informal rulemaking projects (particularly in nuclear material and waste
rulemakings) so that these processes can be compared with EPA’s NAAQS review
process.250
The NRC’s actual process or flowchart for undertaking its technically-based,
informal rulemakings is not explicated in the NRC fact sheets, on the NRC’s webpage, or
even in technical documents or directives (at least that the author could find). Like the
other agency programs studied in this report, the NRC’s process for promulgating
247

Interview with the NRC Staff, Office of General Counsel, Nov. 15, 2011.
Id.
249
See first four horizontal tabs on NRC’s home page at U.S. NRC, last visited on February 2, 2012,
available at http://www.nrc.gov/ (list).
250
Since the processes examined in the FWS and other program offices of EPA were relatively product- or
issue-specific (e.g., akin to a licensing decision), the NRC’s broader rulemaking orientation also provides a
more diverse view of rulemaking activities in different agencies.
248

66

technical rules was reconstructed from interviews with staff and stakeholders, the
synthesis of various documents, and sampling ongoing and concluded rulemakings
posted on several tabs of the NRC website. 251 While the process outlined below for
technical rulemakings may not describe all the NRC rulemakings, it appears to be the
general approach used in most cases.
In general, NRC’s informal rulemaking for technical rules proceeds as follows:
1. Trigger for Action. The need for a rule or guidance is generally triggered by a
statute or by Commission priority, a staff recommendation, a recommendation
from the Advisory Committee on Reactor Safeguards (ACRS), or a petition from
the outside.252 In some cases, the decision to proceed with a regulatory project, in
and of itself, may result from give-and-take between staff and the Commission. 253
2. Possible Workshop. Staff or the Commission occasionally suggest a stakeholder
workshop early in the rulemaking process. These workshops are public and in
some cases are lengthy in duration and broad in scope.254 In the workshops, the
staff solicits information and guidance on the rulemaking project.
3. Staff Analysis. The staff then provides the Commission with one or more technical
papers (called a Commission or SECY paper)255 that offer analysis and sometimes
251

The following rulemakings were selected from the NRC’s website, in part because they are currently
ongoing or recently concluded: a) a revision of 10 CFR Part 61 (low level nuclear waste), see U.S. NRC,
Potential Revision of 10 CFR Part 61, last updated on April 6, 2011, available at http://www.nrc.gov/aboutnrc/regulatory/rulemaking/potential-rulemaking/potential-part61-revision.html#background; b) U.S. NRC,
Specific Analysis Rulemaking (Unique Waste Streams), last updated on June 20, 2011, available at
http://www.nrc.gov/about-nrc/regulatory/rulemaking/potential-rulemaking/uw-streams.html; c) rulemaking
regarding uranium mill tailings, see U.S. NRC, Docket Folder Summary, No. NRC-2010-0075, available at
http://www.regulations.gov/#!docketDetail;dct=SR%252BFR%252BPR;rpp=10;po=0;D=NRC-2010-0075;
d) U.S. NRC, Requirements for Maintenance of Inspections, Tests, Analyses, and Acceptance Criteria, No.
NRC-2010-0012, available at
http://www.regulations.gov/#!docketDetail;rpp=10;so=ASC;sb=postedDate;po=0;D=NRC-2010-0012; e)
U.S. NRC, Proposed Security Rulemaking for Independent Spent Fuel Storage Installations, last updated on
September 16, 2011, available at http://www.nrc.gov/about-nrc/regulatory/rulemaking/potentialrulemaking/isfsi-security.html; f) U.S. NRC, Options to Revise Radiation Protection Regulations and
Guidance, April 9, 2011, available at http://www.nrc.gov/about-nrc/regulatory/rulemaking/potentialrulemaking/opt-revise.html.
252
Interview with NRC Staff, Nov. 2011. A more extensive discussion of the role of the ACRS appears
below in section III.a.3.c.
253
Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011.
254
See, e.g., U.S. NRC, Potential Revision of 10 CFR Part 61, last updated on April 6, 2011, available at
http://www.nrc.gov/about-nrc/regulatory/rulemaking/potential-rulemaking/potential-part61revision.html#background (includes discussion of public workshop); U.S. NRC, Site-Specific Analysis
Rulemaking (Unique Waste Streams, last updated on June 20, 2011, available at http://www.nrc.gov/aboutnrc/regulatory/rulemaking/potential-rulemaking/uw-streams.html (includes a public meeting at the
beginning of the process); U.S. NRC, Options to Revise Radiation Protection Regulations and Guidance,
last updated on April 9, 2011 available at http://www.nrc.gov/about-nrc/regulatory/rulemaking/potentialrulemaking/opt-revise.html (same).
255
See, e.g., NRC, Commission Papers (SECY), available at http://www.nrc.gov/reading-rm/doccollections/commission/secys/. It is not clear from NRC’s website whether “SECY” is an acronym or
simply the proper name for the papers.

67

options for the regulatory project in question. 256 SECY papers used to be
extremely long and detailed. As these staff analyses generally grew in size and
became increasingly complicated, the Commission demanded that they not exceed
ten pages. 257 SECY papers now are relatively short and to the point, although the
technical rulemakings often include multiple attachments.258 In some cases,
documents are also included in the record that either precede or follow issuance of
the SECY papers.259
4. Draft proposal. If the Commission decides to move forward, it directs the staff to
draft a proposed rule and also a preamble, which is termed a “Statement of
Consideration.”260 This draft Statement of Consideration is deliberative and the
Commission therefore does not, at least initially, make it available to the
public.261 The proposed rule and its accompanying Statement of Consideration
are circulated internally to the Office of General Counsel (OGC) for legal
concurrence. 262 Once OGC states that it has “no legal objection” to this package
of documents, they are then submitted to the Commissioners, who may edit and
revise the documents. The Commissioners’ alterations to the draft proposed rule
and Statement of Consideration are also considered part of the deliberative
process and are therefore exempt from public release, although the
Commissioners’ decisions on staff papers are made public. 263
5. No OMB Clearance. Because the NRC is an independent agency, the proposed
rule does not need to be cleared through OMB. OMB’s role is limited primarily
to its statutory responsibilities in authorizing information collection requirements
under the Paperwork Reduction Act 264 and in determining whether an NRC
256

Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011. For samples, see supra note 251.
See also U.S. NRC, Proposed Security Rulemaking for Independent Spent Fuel Storage Installations, last
updated on September 16, 2011, available at http://www.nrc.gov/aboutnrc/regulatory/rulemaking/potential-rulemaking/isfsi-security.html; U.S. NRC, Docket Folder Summary,
No. NRC-2010-0012, available at
http://www.regulations.gov/#!docketDetail;rpp=10;so=ASC;sb=postedDate;po=0;D=NRC-2010-0012;
U.S. NRC, Docket Folder Summary, No. NRC-2010-0075, available at
http://www.regulations.gov/#!docketDetail;dct=SR%252BFR%252BPR;rpp=10;po=0;D=NRC-2010-0075.
257
Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011.
258
Id.
259
See, e.g., US. NRC, Docket Folder Summary, No. NRC-2011-0012, available at
http://www.regulations.gov/#!docketDetail;dct=FR%252BPR%252BN%252BO%252BSR;rpp=10;so=AS
C;sb=postedDate;po=0;D=NRC-2011-0012; U.S. NRC, Docket Folder Summary, No. NRC-2010-0075,
available at
http://www.regulations.gov/#!docketDetail;dct=SR%252BFR%252BPR;rpp=10;po=0;D=NRC-2010-0075;
U.S. NRC, Docket Folder Summary, No. NRC-2010-0012, available at
http://www.regulations.gov/#!docketDetail;rpp=10;so=ASC;sb=postedDate;po=0;D=NRC-2010-0012.
260
Interview with NRC Staff, Nov. 2011.
261
Id.
262
Id.
263
Statement of NRC Staff, Feb. 1, 2012. For the Commission decisions, see the Staff Requirements
Memoranda recording these decisions, available by data at http://www.nrc.gov/reading-rm/doccollections/commission/srm/.
264
44 U.S.C. § 3501, et seq.

68

regulation constitutes a “major rule” under the definition section of the
Congressional Review Act.265
6. Notice and comment and possible external peer review. The proposed rule is
published in the Federal Register and is subject to notice and comment. It might
also be subject to external peer review, particularly if it falls under the jurisdiction
of one of the Commission’s expert advisory panels.
7. Final rule. The development of the final rule follows an internal review process
very similar to the one outlined in the steps above and is published in the Federal
Register.266
The Availability of Initial, Staff Analyses
Most of NRC’s technical analyses are contained, or at least summarized, in the
SECY papers, which provide the equivalent of EPA’s staff analyses. 267 In virtually all
cases, except when the papers expose personnel, proprietary information, or security

265

5 U.S.C. § 804(2). NRC’s Office of General Counsel report that “neither OMB nor the White House
has served as a general-purpose editor/reviewer of NRC regulations (in contrast to their review of
regulations from Executive agencies). Nor does OMB resolve disputes between the NRC and other
agencies with respect to the issuance of NRC regulations.” Statement of NRC Staff, OGC, Jan. 25, 2012.
266
Interview with NRC Staff, Nov. 2011.
267
The importance of these staff analyses as providing a backdrop and information foundation to
Commission decisions is reinforced by a decision-making structure that attempts to maintain a relatively
strict separation between the Commissioners and the general NRC staff (those staff who do not serve
Commissioners directly). While there is still give-and-take between staff and Commissioners, NRC
processes are relatively hierarchical and arranged to generally make it clear where staff analyses leave off
and the Commissioner decision-making begins.
There have been some interactions that potentially blur these traditionally distinct roles of staff
versus the Commissioner, however. Paralleling the Julie MacDonald scandal in DOI, see supra, there are
current allegations that Commission Chairman Jackzo pressured staff on at least one occasion to withhold
and/or “clear” a SECY paper through him before the paper could be shared with the other Commissioners.
While the staff papers at issue were not concerned exclusively with technical issues, the unorthodox
process of intervening in communications between the staff and other Commissioners set off a firestorm
within NRC. As discussed below, the noncurrence process brought some of these staff-management
disagreements to light. These allegations of interference have triggered an OIG report, see NRC OIG,
“NRC Chairman’s Unilateral Decisoin to Terminate NRC’s Review of DOE Yucca Mountain Repository
License Application,” Report No. 11-05 (Jun. 6, 2011); a congressional hearing, see House Committee of
Oversight and Government Reform, “Hearing on ‘The Leadership of the Nuclear Regulatory
Commission’”, Dec. 14, 2011, available at
http://oversight.house.gov/index.php?option=com_content&view=article&id=1536:12-14-2011-qtheleadership-of-the-nuclear-regulatory-commissionq&catid=12&Itemid=1; an investigative report authored
by Congressman Darrell Issa (R-CA), see Majority Staff Report, U.S. House of Representatives,
Committee on Oversight and Government Reform, “A Crisis of Leadership” (Dec. 13, 2011), and media
coverage. A report by Congressman Ed Markey (D-MA) attempts to rebut the allegations. See Edward J.
Markey, Regulatory Meltdown (Dec. 9, 2011). See also Appendix D, nonconcurrence statement on Yucca
Mountain; Press Release, Barbara Boxer, U.S. Sen., Boxer Opening Statement on NRC Hearing, December
15, 2011, available at
http://epw.senate.gov/public/index.cfm?FuseAction=Majority.PressReleases&ContentRecord_id=426c2a48
-802a-23ad-432a-c3b0a1258452&Region_id=&Issue_id.

69

information, SECY papers are available to the public.268 Outsiders interested in the
NRC’s policies can thus track the scientific assumptions and use of the technical
literature in virtually all of the NRC’s rulemakings by comparing the staff report (and
potentially other technical documents) with the proposed rule. The NRC also generally
includes other technical documents and reports in its public docket before the proposed
rule is published.
Attribution and Authorship
The NRC does not appear to give attribution or authorship rights to the staff
preparing the staff SECY papers. A review of SECY papers in recent years reveals that
the papers are signed by higher level supervisors, without acknowledgement or
attribution to staff. 269 Presumably this practice is intended to signal to the Commissioners
that the papers have been reviewed rigorously within the agency and approved (and
hence) signed by a higher-level supervisor. Perhaps like the FWS, the tendency to resist
staff attribution is thus balanced against the need to provide the decision-makers with a
document that formally represents the agency’s final technical position on a particular
matter. It is not clear, however, whether listing the individual staff as contributors or in
an acknowledgement section would impede these goals. At the same time this gesture
towards authorship could produce countervailing benefits, a possibility that is considered
again in the analysis section.
Availability of the Supporting Literature
To supplement the references cited in the preamble of the proposed rule, the NRC
maintains a public database (Agency wide Documents Access and Management System
or ADAMS) that is available on its website. The documents cited in the literature
reviews for SECY Papers and other technical documents are, according to the NRC staff,
generally available in this docket or on ADAMS.270
b. Overview of the NRC’s Licensing Renewal Decisions
Public interest stakeholders identified NRC’s licensing process, such as the
licensing renewal process, as more problematic in ensuring rigorous and transparent
regulatory products as compared with NRC’s informal rulemakings. While an
examination of NRC’s entire licensing program is beyond the scope of this study, two

268

U.S. NRC, Commission Papers (SECY), last updated on January 9, 2012, available at
http://www.nrc.gov/reading-rm/doc-collections/commission/secys/. The NRC website also provides an
accessible portal to these SECY papers. Id. See also Statement of NRC Staff, Feb. 1, 2012.
269
This statement is based on a review of ten different SECY papers posted in 2011 at
http://www.nrc.gov/reading-rm/doc-collections/commission/secys/.
270
Statement of NRC Staff, Feb. 1, 2012.

70

Office of the Inspector General (OIG) reports deserve mention because they document
problems with the NRC’s use of technical information.271
The NRC must renew some operator licenses and make amendments to other
licenses over time and as the need arises. With regard to license renewal, although a
nuclear reactor operating license expires in forty years, the NRC can extend this license
for an additional twenty-years if it believes the plant can continue to be operated
safely. 272 The NRC has developed technical directions for reviewing these license
renewal applications. 273 The NRC’s flow chart for this decision-making process is
provided below. 274

The OIG’s review, which focused on the second and third steps (columns) of
Figure 1 (i.e., the staff’s technical reviews, on-site inspections and audits, and resulting
staff evaluation reports and letters), found the staff analyses to lack scientific and
technical rigor.275 For example, more than thirty-five percent of the more than 450
passages examined by the OIG lacked specific support for the conclusions or a mention
of the methodology used to conduct the technical review, 276 and more than sixty percent
of the passages simply repeated the operators’ technical and factual assertions in support
of staff conclusions, without any indication of independent validation. 277 OIG found that
the lack of guidance to staff in conducting the assessments, coupled with inadequate
quality control, to be at the root of much of this lack of rigor and transparency in the

271

The first of these two reports was mentioned by a public interest stakeholder in interviews as
representative of larger problems at the NRC. Interview with Staff, Beyond Nuclear, Jan. 23, 2012.
272
10 C.F.R. Part 54.
273
See OIG, Audit of NRC’s License Renewal Program, Sept. 6, 2007, at 1 (summarizing these
regulations).
274
Id. at 3.
275
The OIG performed a content analysis of randomly selected audit reports, inspection reports, and safety
evaluation reports in license renewal applications over a six-year period. Id. at pp. 45-47 in Appendix C.
276
An explicit methodology was also found lacking for the post-renewal inspection process. See id. at 29.
277
In addition, less than three percent of the passages contained detailed information to support the
conclusions. Id. at 46-47.

71

NRC’s safety evaluations.278 The OIG also found that inspectors and auditors were
prohibited from removing operator documents on-site, which made it more difficult for
them to conduct the analysis and write the report.279 This prohibition also made it
effectively impossible for others in or outside the agency to validate or even spot-check
the quality of these foundational reports used to renew licenses for aging reactors. 280
A second OIG report is less overtly critical of the technical rigor and transparency
of NRC’s licensing decisions, but identifies similar types of problems in NRC’s safety
evaluations of the operators’ applications for license amendments. 281 In reviewing these
license amendments, the OIG found that NRC’s licensing process was not supported by
sufficient documentation or validation of the applicant’s statements.282 While the OIG
found no instances of flawed safety evaluations, the underlying absence of documentation
would seem to limit the OIG’s ability to evaluate whether the safety evaluations were in
fact robust and reliable.
Public stakeholders suggested that NRC’s incomplete documentation and
explanation of its methods and technical conclusions, spotlighted in these two OIG
reports, has been a continuing source of concern, not only in licensing decisions but in its
enforcement of regulations.283 In essence, these concerns translate to a lack of rigor and
transparency in the documentation and explanation of agency decisions. In the case of
license renewals and amendment decisions, moreover, it is difficult to imagine how the
278

Id. at 11-13. The NRC auditors and inspectors also were not instructed on how and whether to seek
verification of operator-supplied information, and thus they tended to accept this operator-supplied
information without critical scrutiny. There were various examples in the OIG report of how this
information required cross-checking and follow-up inquiry regarding past lapses in operator performance,
Id. at 18-23, and how accepting the operators’ generic statements missed a number of important safety
concerns.
279
Id. at 14-17
280
Id.
281
U.S. Nuclear Reg. Comm’n, Review of NRC’s License Amendment/Safety Evaluation Process,
September 18, 2001, available at http://pbadupws.nrc.gov/docs/ML0127/ML012770353.pdf. The OIG
report was commissioned by Congress in 2000 after the failure of a steam pipe that had been approved one
year earlier. Id. at 1. Since nuclear reactor licenses are very specific, virtually any change to a plant, even
simple changes in the organizational chart, requires an approval of an amendment to a license. At the time
of the OIG report, in 2001, the NRC was receiving 1,500 applications per year for the review and approval
of these amendments. Id. at 1.
282
Specifically, the OIG found that NRC’s “process does not provide adequate controls to ensure that all
process steps are completed and supported by adequate documentation.” Id. at i-ii. The OIG did note,
however, that the NRC’s assessment process was “well-thought out, thorough and provides all the
necessary steps for ensuring the staff perform the required technical reviews,” id. In their examination of
four requested amendment approvals, for example, NRC staff had failed to provide any documentation for
“defin[ing] regulatory requirements, policies [and] applicable precedents.” In three of the four cases, there
was no documentation for the step “Notify public and complete no significant hazards signification.” Id. at
6.
283
See, e.g., Inspector General, NRC’s Regulation of Davis-Besse regarding Damage to the Reactor Vessel
Head, Dec. 30, 2002, at 14 (describing the failure of NRC to document its controversial decision to reverse
course and decline to shut down Davis-Besse in light of evidence of significant safety concerns); see also
GAO, NRC Needs to More Aggressively and Comprehensively Resolve Issues Related to the Davis-Besse
Nuclear Power Plant’s Shutdown, May 2004.

72

public or even those within the agency could oversee the licensing decisions when the
underlying documents are not available and the methods and technical bases for the
conclusions are generally not explained.
c. External Peer Review
Science advisory groups appear to play an important role in a number of the
NRC’s rulemaking processes. In some cases these groups even have the authority to
suggest projects and comment on NRC’s priorities. Although it is not clear whether the
advisory groups are realizing their full potential in ensuring that NRC’s use of science is
rigorous and transparent, their heavy engagement in NRC’s work 284 suggests that they are
a critical feature of the decision-making structure, at least for science-intensive issues.285
In some rulemaking settings and in virtually all license decisions, peer review by
one of the two expert advisory groups is required by statute.286 In other cases, the NRC
requests an existing expert advisory group to review a rule or otherwise provide guidance
on a regulatory project. In either case, when this review of a rulemaking occurs, the
review is generally conducted by either the NRC’s Advisory Committee on Reactor
Safeguards (ACRS)287 or the Advisory Committee on Medical Uses of Isotopes
(ACMUI), both of which are chartered under FACA.
The advisory group with the most expansive jurisdiction is ACRS, which was
established by the Atomic Energy Act of 1954 to provide advice to the the NRC on
licensing, with particular attention to nuclear reactor safety. 288 The NRC by rule has

284

Members also commit to considerable service on these committees. ACRS requires 100-120 days a year
from each member, with full committee meetings running three days and with 2-3 subcommittee meetings
each month. Interview with NRC Staff, Oct. 26, 2011.
285
See Roland Frye, The United States Nuclear Regulatory Commission’s Use of Scientific and Technical
Advisory Committees, Dec. 2011, at Appendix B.
286
As detailed below, the jurisdiction of these two advisory groups is focused on the topic areas denoted in
their respective names, although the ACRS in particular appears to have an increasingly expansive
jurisdiction, which now includes radioactive waste. See Id. at 3. In cases where there is not a requirement
for peer review, NRC staff and the Commission can still assemble a panel of independent reviewers or
experts who are not full-time employees of the NRC for these other regulatory projects. It is not clear
whether or how often this type of additional peer review occurs or whether the experts are convened under
FACA, however. [Have you checked with Rollie on how frequently this type of additional peer review
occurs?] Interviews also revealed that for at least some of these more specific issues, it was difficult to
identify neutral reviewers (this is a problem that EPA staff also suggested could occur in pesticide peer
review). Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011. As mentioned at the end of
this report, however, the challenges associated with developing robust peer review are left for future ACUS
projects.
287
The ACRS is comprised of a maximum of fifteen members, who consist primarily of academics,
scientists and regulated parties in the private sector with diverse, relevant expertise. Interview with NRC
Staff, Oct. 26, 2011. Members are selected to provide broad expertise, and some attention is also given to
ensuring that at least some members have actual operational experience. Id.
288
42 U.S.C. § 2039. For fuller discussion of the creation and history of this advisory committee, see Frye,
supra note 285, at 3-5.

73

charged ACRS with a number of responsibilities. 289 Among these responsibilities is the
mandatory review of all informal rules on nuclear safety. ACRS typically reviews the
proposed rules either after public comment or simultaneously with public comment and
then again, as an optional matter, once the final rule is drafted.290 The NRC staff
typically provides a written response to each of the advisory committee’s comments,
although historically this was not always the case. 291 Both the ACRS review and staff
responses are generally placed in the public record.
Over the more than 50 years of interaction between ACRS and the NRC staff,
there have been some vigorous disagreements. 292 In several cases, in fact, the NRC staff
did not agree with or accept ACRS’ recommendations. 293 Interviewees report, however,
that ACRS has never disagreed with staff with regard to the granting of a license or
construction permit. 294
Beyond its formal role in reviewing staff rules, technical documents, and licenses,
the ACRS also reports to the Commission at least annually.295 The ACRS may also, on
its own initiative, “conduct reviews of specific generic matters or nuclear facility safetyrelated items.”296 It may even recommend that the Commission initiate a rulemaking – a
formal recommendation that requires a response from the Commission in 90 days. 297
ACRS’s scope of powers thus appears quite expansive and may be broader than EPA’s
CASAC.

289

10 C.F.R. § 1.13 (listing the responsibilities). The responsibilities include, for example, reviewing and
reporting on “[e]ach application for a construction permit or an operating license for a facility which is of a
type described in [10 CFR] 50.21(b) or 50.22, or for a testing facility.” Id. at § 50.58(a). NRC regulations
also require the staff to involve ACRS in informal rulemakings regarding nuclear safety. NRC, Final Rule,
ACRS Participation in NRC Rulemaking, 46 Fed. Reg. 22,358 (1981), as amended, NRC, Electronic
Availability of NRC Public Records and Ending of NRC Local Public Document Room Program, 64 Fed.
Reg. 48,948 (1999).
290
Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011. For technical rules, the ACRS
either reviews proposed rules and provides detailed comment simultaneously with public comment or it
provides its review on the proposed rule after the NRC has responded in a draft to public comment. Id.
ACRS also reviews license decisions, including license renewals, based on the staff’s safety reports. These
ACRS reviews can be detailed and are always part of the public record. Id.
291
Id.
292
See Frye, supra note 285, at 6-7.
293
For instance, in 1959, the ACRS adamantly opposed a staff recommendation regarding standards for
locating nuclear power reactors in or near population centers. Similarly, in 1965, the ACRS opposed a
related recommendation by the regulatory staff to prohibit the location of power reactors in metropolitan
areas. Id. at 7.
294
Interview with NRC Staff, Oct. 26, 2011.
295
10 C.F.R. § 1.11(c); Interview with NRC Staff, Oct. 26, 2011. For example, the ACRS makes
occasional oral presentations directly to the Commission. See Frye, supra note 285, at 7.
296
10 C.F.R. § 1.13.
297
Id. at § 2.809. It is not clear whether or how often this occurs. Such recommendations must be posted
on the NRC website, but in a quick search there did not appear to be a central repository for such postings.

74

The second advisory committee, ACMUI,298 plays an important but narrower role
in the NRC decision-making as compared with ACRS. ACMUI was created an NRC
regulation299 and reports to staff as needed on medical questions that are referred to it. 300
The Commission may also request advice and expert opinions from this body. 301 The
ACMUI receives informational copies of all proposed regulations under its purview,302
yet unless specifically requested, the ACMUI is not expected to conduct reviews of these
rules, although it is allowed to do so. Like the ACRS, there are occasions on which
ACMUI and the NRC staff disagree on key issues.303
Since the 1980’s, the NRC has also employed expert elicitation to develop expert
recommendations on particularly difficult technical issues. 304 Unlike peer review or
expert advisory boards that comment on technical analyses, expert elicitation involves the
selection of experts who attempt to arrive at empirical estimates based on very limited
information and often in circumstances where even computational models cannot be
developed to handle relevant variables in a robust way. 305 Roland Frye has prepared a
memo providing a detailed analysis of that program, which is included in Appendix B.
4. Review by the Office of Management and Budget
Some science-intensive rules not only pass through the levels of agency staff and
political management, but are subjected to review by the Office of Information and
Regulatory Affairs. Under Executive Order 12866, only regulations with the most
significant implications are selected out of the agencies’ larger set of rules and formally
reviewed, 306 but when this review occurs, the literature indicates that the underlying
changes associated with OIRA may be material and yet can be difficult to assess. 307
298

Members of ACMUI come from diverse sectors and appear selected to ensure broad disciplinary
representation. Although they were originally expected to include only physicians and scientists,
disciplinary representation is much broader in practice. See Frye, supra note 285, at 10.
299
10 C.F.R. § 1.19(a).
300
Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011; Interview with NRC Staff, Oct.
26, 2011.
301
Id.
302
Interview with NRC Staff, Office of General Counsel, Nov. 15, 2011.
303
See Frye, supra note 285, at 11-12.
304
Roland Frye, Use of Expert Elicitation at the Nuclear Regulatory Commission, Feb. 16, 2012, included
as Appendix B.
305
NRC identifies expert elicitation as “a well-recognized technique for quantifying phenomenological
knowledge when modeling approaches or data are insufficient.” NUREG-1829, Vol. 2, Estimating Loss-ofCoolant Accident (LOCA) Frequencies through the Elicitation Process: Appendices A through M (Apr.
2008) (ML081060300), at v (cited in Frye, supra note 304, at 1).
306
See Exec. Order No. 12,866 §§ 3(f), 58 Fed. Reg. 51,735, 51,742 (Oct. 4, 1993). As Section 3(f) makes
clear, OMB’s review attaches to numerous regulatory projects, and economically significant rules
constitute only a fraction of the rules undergoing OMB review.
307
As the GAO observed in their 2003 study, “the OIRA regulatory review process is not well understood
or documented, and the effect that OIRA’s reviews have on individual rules is not always easy to
determine.” GAO, OMB’s Role in Review of Agencies’ Draft Rules and the Transparency of Those
Reviews, Sept. 2003, at 110.

75

Since this study examines the processes the agencies use to develop science-based
rules and assesses the transparency of each step in a rule’s development, changes made to
science-intensive rules that occur as a result of OIRA review cannot be ignored. If
substantive changes are being made to science-intensive rules as a result of OIRA review
and if those changes are not identified, then a potentially important step in the
development of science-intensive rules may not be accessible to the general public or to
scientists who review the rule. [Note that there are other ways that OIRA’s
responsibilities may allow it to alter the substance of science-intensive rules. For
example, under the Paperwork Reduction Act, OIRA must review and ultimately approve
agency requests for information from private parties, including scientific and technical
information. 308 Even though this clearance role provides OIRA with the ability to
influence the substance of science-intensive projects, for example by affecting the types
of information an agency can request in the development of its rule, these additional
OIRA responsibilities are not examined here since they fall outside the scope of this
study – namely the examination of agencies’ decision-making processes.]
Executive Order 12866 imposes specific disclosures requirements on the agencies
with respect to OIRA’s review, yet despite these requirements a number of interviewees
and the literature more generally – including two studies by the GAO – highlight the lack
of transparency of OIRA-induced changes to science-intensive rules. 309 In order to gain
more empirical purchase on the agencies’ disclosure of OIRA-inspired changes, the only
two subsets of rules in this study subjected to formal OIRA review – the EPA’s NAAQS
rules and FWS’s ESA habitat designations (sixteen rules in total)-- were examined to
assess compliance with the transparency requirements of Section 6(a)(E).310
a. The Transparency of the Changes made during OIRA’s Review
Although in theory the agencies might claim the deliberative process privilege
with regard to the disclosure of the results of OIRA’s review of agency rules, President
Clinton imposed several transparency requirements on the agencies under Executive
Order 12866 that override these claims. Specifically, under Executive Order 12866 the
agency originating a rule is required to “[i]dentify for the public, in a complete, clear, and
simple manner, the substantive changes between the draft submitted to OMB for review
and the action subsequently announced” and also to “[i]dentify for the public those
changes in the regulatory action that were made at the suggestion or recommendation of
308

This role is discussed above, for example, in OIRA’s review and clearance of EPA’s requests for
information from pesticide manufacturers under the Paperwork Reduction Act. See supra Section III.A.1.b.
309
See the remainder of this Section for support for this statement (Section III.A.4., infra).
310
The Nuclear Regulatory Commission is an independent agency and is not (yet) subject to the review
requirements of Executive Order 12866. As discussed, the FWS’ listing of endangered species has
historically been exempt from OMB review under the Executive Order, as have pesticide approvals under
FIFRA. See supra Section III.A. Finally, since IRIS assessments are not rulemakings, they also are not
technically covered by Executive Order 12866.

76

OMB.”311 OMB for its part must “make available to the public all documents exchanged
between OMB and the agency during the review.” 312 All of this information must be
“provided to the public . . . in plain, understandable language.” 313
OMB and the agencies appear to interpret these Executive Order transparency
requirements narrowly, in a way that that generally limits the transparency of the changes
made at the suggestion of OIRA.314 Specifically, for at least some of the rules examined
here, the changes made during OIRA review were difficult to identify and impossible to
understand. This lack of disclosure appears in direct conflict with Section 6 of the
Executive Order and President Obama’s memorandum directing agencies to maximize
the transparency of their regulatory activities.
1. Documents Exchanged between OIRA and the Agencies
Executive Order 12866 requires that after a rule is published or an agency
announces that it will not move forward with a regulation, “OIRA [must] make available
to the public all documents exchanged between OIRA and the agency during the review
by OIRA under this section.” 315 However, OMB reads the documentation requirements
of section 6(b) to apply only to documents exchanged between the agency and the branch
levels of OMB. 316 All staff exchanges are treated by OMB as exempt from the reach of
this requirement. No justification could be found for this interpretation of the Executive
Order, and at least the GAO has been critical of it. 317 No documents underlying OIRA’s
review of the rules in this study – other than return letters -- were posted on OIRA’s
website or otherwise made publicly available by OIRA. The absence of documents is
consistent with OIRA’s narrow interpretation of its responsibilities.

311

Exec. Order No. 12,866 §§ 6(a)(3)(E)(ii)–(iii)
Specifically, the EO states: “After the regulatory action has been published in the Federal Register or
otherwise issued to the public, or after the agency has announced its decision not to publish or issue the
regulatory action, OMB shall make available to the public all documents exchanged between OMB and the
agency during the review by OMB under this section.” Id. at § 6(b)(3).
313
Id. at § 6(a)(F).
314
See infra Sections III.A.4.a.1-3.; see generally GAO 2003, supra note 307, at 119 (“the difficulties that
we experienced during this review clearly demonstrated that OIR A’s reviews are not always transparent to
the public.”).
315
Section 6(b)(4)(D) of Executive Order 12866.
316
See, e.g., GAO, IMPROVEMENTS NEEDED TO MONITORING AND EVALUATION OF RULES DEVELOPMENT
AS WELL AS TO THE TRANSPARENCY OF OMB REGULATORY REVIEWS at 32, n.40 (April 2009) ( reporting
that “[a]ccording to OMB representatives, the requirement to make available “all documents exchanged
between OMB and the agency” issuing the regulation only applies to exchanges made by OMB staff at the
branch chief level and above, not documents exchanged between OMB desk officers and staff in regulatory
agencies.”); see also GAO 2003, supra note 307, at 57 (reporting on the OIRA document policies that read
6(b) in a limited fashion). This express statement could not be located on OMB’s site or in its various
memos, however.
317
See id.
312

77

(i)

Identification of all Substantive Changes occurring during
OIRA’s Review

Section 6(a)(3)(E)(ii) of the Executive Order requires the agency originating the
regulatory action to “[i]dentify for the public, in a complete, clear, and simple manner,
the substantive changes between the draft submitted to OIRA for review and the action
subsequently announced.” For both the NAAQS and ESA rules, the agencies’
compliance with the Executive Order requirements consisted, at best, of posting a redlined version of the rule produced during OIRA’s formal review. This documentation
was incomplete across the rules in the study, however; for one out of every three rules,
there was no red-lined document available, and for one proposed and final rule (lead)
there was no documentation of the rule prior to OIRA review. 318
Even when the agencies did provide red-lined documents of changes occurring
during OIRA’s formal review, however, this documentation did not “[i]dentify for the
public, in a complete, clear, and simple manner” and in “plain, understandable language”
“the substantive changes” made during OIRA’s formal review as required by Section
6(a)(3)(E)(ii) and (F). In each of these red-lined documents, substantive changes
occurring during OIRA review were not separated out of the dozens of grammatical,
formatting, and purely editorial and stylistic changes. Instead the changes that appear
likely to be substantive were discovered only by reading through each page of the redlined documents that often run several hundreds of pages long. 319 The GAO noted
similar compliance lapses by other agencies in its 2009 study.320
In addition, locating the red-lined documents was far from “simple” as required
by the Executive Order.321 While OMB’s Reginfo.gov contains a detailed search engine
that pulls up information on each rule that is or has undergone OIRA review, this
database does not post or link to the red-lined documents.322 In this study, the red-lined
documents for the EPA and FWS rules had to be located on a rule-by-rule basis by

318

Red-lined documents were not available for: The Canadian Lynx critical habitat final rule; the Primary
NAAQS review for Lead (proposed and final); the Primary NAAQS review for Nitrogen Dioxide
(proposed and final). For three of these five rules, the pre-OMB draft rule was available, however. The
level of compliance was thus higher here than observed by GAO in 2009; they report that 62% of the
agencies documented at least some features of OMB’s formal review. GAO 2009, supra note 316, at 13.
319
The reader can view the red-lined documents located for the NAAQS rules and habitat designation rules
canvassed in this study at /wewagner/upload for acus. The “re-created” red-lined for the proposed and final
primary NAAQS nitrogen dioxide review is available at /wewagner/nitrogen dioxide change docs.
320
GAO observed a wide range of agency compliance with Section 6, ranging from no compliance to
detailed memoranda that detailed OIRA changes. Id. It appears that the majority of agencies adopt
something like the approach here, which is simply posting a single red-lined copy of the draft rule as a
result of OMB review. Id.
321
Section 6(a)(3)(E)(ii). GAO encountered the same difficulties in its 2009 study. See GAO 2009, supra
note 316, at 34.
322
See http://www.reginfo.gov/public/do/eoAdvancedSearchMain

78

running searches for the documents in the agency’s individual rulemaking dockets. 323
This search was further complicated by the fact that the name that the EPA assigns to
these red-lined document is not regularized;324 in some cases the red-lined document is
located only after opening dozens of posted documents that contain OMB or Office of
Management and Budget in the title of a rulemaking docket, which in turn can contain
hundreds of entries. 325 In the case of the FWS, the search is still more complicated since
the FWS did not maintain online dockets for habitat designations under the Endangered
Species Act. Instead, one must call the FWS field office that originated those rules
during office hours (this office phone number is listed at the end of the published rule in
the Federal Register).326 The Field Office personnel then attempt to identify the
appropriate documents and provide them electronically or by mail.
In sum, the evidence of agency compliance with Section 6(a)(3)(E)(ii)’s
requirement to identify the substantive changes occurring during OIRA’s formal review
generally consists, at best, of the posting of a redlined version of the changes that
occurred during formal OIRA review in a rulemaking docket. Once found, this document
does not identify the substantive changes made by OIRA nor explain them; one must read
the hundreds of pages of red-lined to pull out changes that might be important.
(ii)

The Identification of changes made at the Suggestion or
Recommendation of OIRA

Executive Order 12866 also requires the agency originating a regulatory action to
“[i]dentify for the public those changes in the regulatory action that were made at the
suggestion or recommendation of OIRA;”327 yet the agencies did not provide this
documentation for any of the rules in this study. Again, this suggests EPA and the FWS
are not in compliance with the Executive Order 6(a)(3)(E) requirements for at least some
rules. The fact that the red-lined documents reveal hundreds of changes raise the
likelihood that OIRA suggested at least some of them during its formal review. Yet there
was no effort by the EPA or the FWS to identify which of these redlined changes were

323

The dockets were searched for “OMB” and “Office of Management and Budget.” Searches for “OIRA”
and “Office of Information and Regulatory Affairs” typically produced no documents. OMB uses a RIN
number to identify rules; agencies use docket numbers. To track down the agency’s identification of
changes resulting from OMB review, one must locate the agency’s docket number by entering the unified
regulatory agenda using the RIN number. In cases where the docket number is still not provided, further
tracking and linking to find the agency’s docket is necessary.
324
Again, GAO encountered the same difficulties in its 2009 study. See GAO 2009, supra note 316, at 34
(describing the differences in agency labeling practices).
325
See the OIRA-EPA exchanges for each of the NAAQS rules in the data compiled for this study at
/wewagner/Document Inventory of OMB review.
326
The docket index was only available for the rule that was undergoing litigation and it seemed from the
discussions with several field offices that the preparation of a docket index might be the exception rather
than the rule.
327
Section 6(a)(3)(E)(iii) of Executive Order 12866.

79

made “at the suggestion or recommendation of OIRA” as opposed to other agencies or
the originating agencies themselves.
An agency might maintain that all changes made in a red-lined version that
documents the changes made during OIRA’s review were also changes “made at the
suggestion or recommendation of OIRA.” But even if that were true with respect to the
changes made during OIRA’s formal review, it misses the changes made at the
suggestion of OIRA that were made outside this short timeframe. To get a sense of
OIRA’s review, I assembled all available documentation of OIRA’s exchanges cataloged
in the EPA NAAQS and FWS habitat designation dockets. Although OIRA does not
make the correspondence between its staff and the agency publicly available, the
agencies do docket some of this written information consistent with their responsibilities
under Section 6(a)(3)(E). 328 The figure below provides a summary of these docketed
agency-OIRA communications for the NAQQS rules canvassed in this study. 329 As this
figure indicates, the docketed exchanges vary a great deal from rule to rule, but for some
rules, relatively extensive exchanges occurred between the agency and OMB outside the
formal review period.330

328

EPA did create a site dedicated to tracking rules through the rulemaking process, including those going
through OMB review. http://yosemite.epa.gov/opei/RuleGate.nsf/ In a search of the archived rules,
however, very little and perhaps none of the information collected in this study on OMB review of NAAQS
rules was available. That is not a criticism of the site; only an acknowledgement that the site is not
complete.
329
The excel lists for each of the rules in this study where dockets were available is provided at
/wewagner/Document Inventory of OMB review.
330
See id.; see also Appendix G.

80

In the NAAQS rules in particular, nearly half of the communications between OIRA
and EPA occurred outside the formal window of OMB review. In the lead primary
NAAQS review all of the nearly 120 docketed exchanges between OIRA and EPA
occurred outside the very short (1 day) OIRA formal review window. 331 (Readers can
access each of the documents by following the hyperlinks to the documents detailed in
the footnote.)332 See also Appendix H. Yet precisely because many exchanges occurred
331

See Docket EPA-HQ-OAR-2006-0735 at /wewagner/Document Inventory of OMB review
To view the individual documents, go to /wewagner/Document Inventory of OMB review and click a
docket number. This will list the OIRA-EPA exchanges, along with some details. In the last column, there
332

81

outside the formal review window, the ability of the public to assess the actual changes to
a rule “made at the suggestion or recommendation of OIRA” as required by Section
3(a)(3)(E)(iii) is effectively impossible. At best, one can piece together scattered
docketed email exchanges between OMB and the agency to identify potential changes,
and then compare those potential changes against the published rule. 333 Even then, the
agency dockets are incomplete. Some docket exchanges consist only of the short cover
emails, and omit the attachment that contain the substantive comments and suggested
edits. In its report, GAO similarly noted extensive OMB engagement with agency rules
outside of the formal review process that were not accounted for by agencies in
compliance with the Executive Order.334
In sum, while Section 6(a)(3)(E)(iii) requires the agencies to “[i]dentify for the
public those changes in the regulatory action that were made at the suggestion or
recommendation of OMB,” the EPA and FWS did not identify these changes for any of
the rules in this study. 335 While it is possible that this simply means that there were no
changes in any of the rules, the red-lined documents and other documents posted,
particularly by EPA, belie that possibility. Indeed, given the likelihood that no
identification of changes will be interpreted by the public as signifying that OIRA offered
no changes, agencies should either provide the identification of changes for each rule or
indicate that there were no changes made at the suggestion or recommendation of OIRA.
It is not clear why the agencies did not provide this identification of changes as
required by Executive Order. One partial explanation for this noncompliance is the
possibility that the agencies read a cryptic 1993 OMB directive to suggest that Section
6(a)(3)(E)(iii) should be read as a subset of Section 6(a)(3)(E)(ii). 336 Yet even in this
statement, the agencies are still directed by OMB to identify the changes “made at the
is a hyperlink to the original document which is posted at regulations.gov. In the longterm, I hope to gather
more detailed information about the nature of these exchanges, both in substance and in relation to formal
review. At this point, however, the reader must assess for themselves the nature of these various
communications.
333
GAO 2003, supra note 307, at 111 (reaching a similar conclusion).
334
Id. at 5.
335
The same finding appears to be drawn from the GAO study, which found very few agencies able to
attribute the changes to OMB. See, e.g., id. at 111.
336
See, e.g., Memorandum to Executive Heads and Departments form Leon Panetta, Oct. 12, 1993, at 9,
available at
http://www.whitehouse.gov/sites/default/files/omb/assets/inforeg/eo12866_implementation_guidance.pdf.
The 1993 memorandum provides only this one sentence guidance on Section 6(a)(3)(E):
In addition, after the regulatory action has been published in the Federal Register or
otherwise issued to the public, each agency is to identify for the public, in a complete,
clear, and simple manner, the substantive changes that it made to the regulatory action
between the time the draft was submitted to OIRA for review and the action was
subsequently publicly announced, indicating those changes that were made at the
suggestion or recommendation of OIRA (Sec. 6(a) (3)(E) (ii) & (iii)).
See also GAO 2003 Study, supra note 307, at 56-57, 95 (discussing difficulty with OMB’s
apparent legal interpretation).

82

suggestion or recommendation of OIRA.” 337 OIRA’s interpretation could also be read to
require the agencies to identify changes “made at the suggestion or recommendation of
OMB,” be limited only to formal OIRA review, but again OIRA’s interpretation does not
go that far.338 Regardless, since the agencies, not OIRA, are tasked with responsibility
for meeting the requirements of Section 6(a) of the Executive Order,339 OMB’s
interpretation – even if it did preclude agency compliance with (E)(iii) which it does not
– will not insulate the agencies from their important disclosure requirements. Indeed, the
responsibility to identify changes resulting from OIRA review was likely placed on the
agencies precisely in order to provide an independent check on this review.
Equally problematic in terms of the agency’s compliance with Section (E)(iii),
OIRA review can sometimes result in the suspension or even termination of rules; yet
beyond the fact that a rule is withdrawn (identified on OIRA’s website), there is no
record of what happened during this suspension period or what OIRA suggestions led to
the withdrawal or suspension of the proposed rule. The GAO raised concerns about this
as well and issued recommendations for heightened transparency about rule withdrawals.
As just one example, from a rule in this study - the primary NAAQS for nitrogen dioxide
- EPA initiated a review process with OIRA in January 2009, but withdrew the proposed
rule two weeks later without public explanation. 340 It took EPA another four months to
337

Id.
OMB directs agencies to identify changes made between the “time the draft was submitted to OIRA for
review” and when the regulatory action was subsequently publicly announced, which could include both
formal reviews and more informal reviews occurring outside the formal review period. In any event,
reading the (E)(iii) disclosures as limited to only the formal review period is not a reasonable reading of the
Executive Order and hence, by extension, it is not a reasonable reading of OMB’s passage either.
Collapsing Sections (ii) and (iii) of the Executive Order together to require disclosure of changes occurring
only during formal review conflicts with basic rules of legal interpretation that afford each section an
independent meaning; moreover, subsection (iii) refers to an identification of all “changes [made] in the
regulatory action,” while subsection (ii) refers only to “substantive changes” made “between the draft
submitted to OIRA for review and the action subsequently announced.” Collapsing the two sections to
preclude an identification of changes occurring outside the formal review is also inconsistent with the
structure and intent of Section 6, which seeks to provide basic transparency of the results of OIRA review,
as well as with the President Obama’s memorandum on transparency. Finally and perhaps most important,
to limit disclosures of OIRA-induced changes only to the formal review period risks missing most, if not
all, of the changes made at the suggestion of OIRA, in at least some rules. Perhaps in 1993 when OIRA
issued this statement, its influence was limited to the formal review period. Twenty years later, it is clear –
on OIRA’s own admission -- that many changes made to rules at the suggestion or recommendation of
OIRA occur outside the formal review period and the interpretation makes disclosure badly incomplete.
See, e.g., GAO 2003, supra note 307, at 36-37 and 113-114.
339
See Section 2(a) and Section 6(a) of the Executive Order. Section 2(a) for example directs that:
“Because Federal agencies are the repositories of significant substantive expertise and experience, they are
responsible for developing regulations and assuring that the regulations are consistent with applicable law.
the President's priorities, and the principles set forth in this Executive order.” Section 6(a), entitled
“Agency Responsibilities” reinforces that compliance is the responsibility of the agencies.
340
See http://www.reginfo.gov/public/do/eoDetails?rrid=116780 (the record of EPA’s withdrawal of the
proposed rule after a two week period at OMB). A review of the more than 400 documents in the EPA’s
docket did not reveal any documents that explained the reason for withdraw or even the fact that the initial
draft had been submitted. To this extent, the EPA is arguably in violation of Section 6(a)(3)(E)(i) to the
338

83

return to OIRA with a new proposed rule. 341 Yet there are no docketed exchanges
regarding this brief unconcluded OMB review that explain what transpired or how or
whether OMB suggested or recommended changes that influenced EPA’s subsequent
draft proposed rule. 342 While it is possible that the initial submission to OIRA and
subsequent withdrawal did not lead to any changes to the rule made at the suggestion or
recommendation of OIRA, this seems unlikely. Several agency officials interviewed for
this study also suggested that OIRA is becoming increasingly active in pre-review in an
effort to avoid these withdrawals; OIRA simply urges the agency not to send a rule for
OIRA review so that a public record of withdrawal is not created. The potential
significance of these types of informal or non-formal review exchanges was further
underscored by a recent Washington Post article that reported an apparent request by
OMB that EPA hold a publicly important rule and not submit it to OMB for review. 343
To the extent that these pre-review activities and withdrawals ultimately lead to
changes “at the suggestion or recommendation of OMB,” they should be disclosed and
explained by the agency. Additionally and in order to remain faithful to the spirit of the
President’s memorandum on transparency, the agencies should document these changes
even if the regulatory action is never published. Otherwise these withdrawals and holds
provide a major loophole in ensuring the basic transparency of OIRA review as promised
by Section 6(a) of E.O. 12866.
(iii)

No Enforcement of Agency or OMB Compliance

Across the rules investigated here, there was no centralized effort to ensure
agency compliance with Section 6(a)(3)(E) nor were sanctions levied against
noncomplying agencies. OMB concedes that it does not monitor the agencies’

extent that it doesn’t provide the draft that it submitted to OMB at the initiation of this, ultimately
withdrawn rule.
341
See http://www.reginfo.gov/public/do/eoDetails?rrid=117155 (the record of EPA’s second submission
of the proposed rule to OMB). The draft submitted and then withdrawn in January 2009 could not be
located on EPA’s docket. To the extent that changes were made to that proposal as a result of OIRA’s
suggestions, they do not appear to be provided in the public record.
342
Since this discovery of the withdrawal was accidental, this type of concluded review may be a far more
frequent occurrence. A search of EPA rules that have been withdrawn over time yielded a list of 162 rules
since 1981. (To replicate this search, go to http://www.reginfo.gov/public/do/eoAdvancedSearchMain.
Select EPA as the main agency and “withdrawn” as the “concluded action”. Also click “concluded
reviews” rather than the default of pending reviews (located on the right column of the search terms). GAO
also indicated concern about withdrawn rules and the lack of documentation of OMB’s influence in the
process. See, e.g., GAO 2003, supra note 307, at 7.
343
Juliet Eilperin, Obama administration slows environmental rules as it weighs political cost, Feb. 12,
2012, Washington Post, available at http://www.washingtonpost.com/national/health-science/obamaadministration-weighs-political-cost-of-environmental-rules/2012/02/07/gIQAvJzx8Q_story.html. An
effort to locate documentation of these discussions was not successful, however. The docket on the Tier III
rules was empty. EPA-HQ-OAR-2011-0135 . (See http://www.epa.gov/rfa/tier3.html for the discussion of
the rule and the docket number).

84

compliance with Section 6(a)(3)(E) of the Executive Order; 344 OMB also does not
reference or link to the agency’s redlined document in its own database, reginfo.gov, to
make it easier for the public to evaluate agency compliance. Similarly, within the
agencies there did not appear to be a coordinated system to ensure compliance with
section 6(a)(3)(E) or even to issue agency-wide interpretations and directions for what
this section requires.
b. Scientific Significance of the Changes that result from Formal OIRA
Review345
A number of interviewees maintained that OMB did make changes to the
scientific and technical characterizations of some of EPA rules (and the NAAQS rules in
particular),346 but evidence of this influence was not provided and thus the nature and
significance of these changes was controverted. To gain a preliminary assessment of the
nature of changes made during OIRA’s formal review, I coded all non-editorial
changes347 in each of the available red-lined documents for the rules canvassed in this
study based on whether the changes appeared technical or non-technical. 348 Technical
changes are those that involved alterations to technical discussions in the preamble or
rule, such as qualifying scientific discussions, adding scientific or technical citations,
revising the numbers in tables, and recommending different substantive standards
altogether that would seem to benefit from further scientific review and discussion. Nontechnical changes may be significant, but they concern legal issues, economic issues, or
other types of changes that do not affect the characterization of the scientific record or
make technical or scientific changes to the rule or its preamble. For example, the
following red-lined excerpt – which occurred to EPA’s rule during OIRA review -would be counted as a technical change since it involves changes to the enforceable rule
that occurred during OMB review.

344

GAO 2009, supra note 316, at 32.
As discussed above, there is no way to identify the changes made at the suggestion of OIRA throughout
the rulemaking record.
346
See, e.g., infra notes 590-591 and accompanying text.
347
See Appendix G for further definitions of this term.
348
See Appendix G. A proposed rule might record thousands of changes in the red-lined reviewing pane,
but less than one-hundred of these changes appear non-editorial and are thus coded as either technical or
non-technical. The coded changes then constitute only a fraction of the changes made during OMB review.
Note that the methods used here are different from those employed by GAO in its 2003 and 2009 study in
which GAO categories the cumulative changes made by OIRA to a rule on a spectrum ranging from
significant to nonexistent. See, e.g., GAO 2003, supra note 307, at 29-30.
345

85

An Excerpt of the Redlined Changes Occurring during OIRA’s Formal Review of EPA’s
Proposed Rule for the Sulfur Dioxide NAAS349

A review of the available red-lined documents for the NAAQS reviews revealed
that in most rules the majority of the non-editorial changes were technical in nature. See
Table below. The changes made to FWS rules, by contrast, were about equally divided
between technical and more substantive (often legalistic) changes and were, overall,
fewer in number.350

349

Red-lined of the Proposed NAAQS Standard for Sulfur Dioxide at pages 244-46, Docket id EPA-HQOAR-2007-0352-1347, available at regulations.gov.
350
See Appendix G .

86

Title
Secondary
Secondary
Lead
Monitoring
Ozone
Network
Design
Sulfur Dioxide
Sulfur Dioxide
Monoxide
Monoxide

# Non-editorial
Changes
44
21

#/% of
noneditorial
Changes
that were
technical
35 (80%)
13 (62%)

Proposed
Proposed

24
12

15 (62%)
6 (50%)

Proposed
Proposed
Final
Proposed
Final

5
49
16
26
14

0 (0%)
42 (86%)
10 (62%)
13 (50%)
10 (72%)

Proposed/Final
Proposed
Final

While it was not possible without considerably more expert assistance and
detective work to determine the significance of most of these various technical changes,
for more than half of the NAAQS rules examined in this study, the science-based rules
did change during OIRA’s review in ways that affect the enforceable requirements of the
regulation.351 For example, the required number of ambient monitors to measure both
carbon monoxide and nitrogen dioxide concentrations in the ambient air 352 were reduced
by about twenty percent during OIRA’s review (since EPA did not identify the changes
made at OIRA’s suggestion, it is not possible to determine whether the change was
actually made at the suggestion of OIRA). 353 During OIRA review, EPA changed the
rule for the secondary standard for the oxides of nitrogen and sulfur to adopt the primary
standard (EPA had initially declined to set a secondary standard) (the same qualification
applies to this change).354 At OIRA’s recommendation, EPA also loosened the secondary
351

In its 2009 report, the GAO identified about 30% of the rules it reviewed as involving significant
changes that “revised the scope, impact, or costs and benefits of the rules”. GAO 2009, supra note 316, at
5.
352
For the WORD-generated track change document that highlights this change in the nitrogen dioxide
rule, see the final red-lined rule posted at /wewagner/nitrogen dioxide change docs. For the change in the
carbon monoxide final rule, go to pages 100- 101 of the red-lined final rule for carbon monoxide, which is
listed as EPA-HQ-OAR-2008-0015-0200A track final rule and is posted at /wewagner/upload for acus.
353

If the reduced number of monitors falls comfortably within some range agreed upon by the scientists,
meteorologists and statisticians, and has no scientific costs or downsides, then these changes may not
intersect with scientific issues. If however there are technical implications of a reduced number of required
ambient monitors, then these changes do occur at the intersection of science and policy.
354
EPA apparently preferred to provide no secondary standards; during OMB review the primary standard
was inserted.

87

standard for ozone (this change was not located through examination of the red-lined
documents but instead is reported in the literature, which is why OIRA’s role is
clearer).355 OIRA’s influence could not be identified in the lead proposed and final rules,
but the number of substantive documented exchanges (N=119) occurring between OIRA
and EPA give an outsider the impression that OIRA’s influence on the lead rule may
have been significant.356 See also Appendix H and hyperlinked documents in the
footnote. Finally, OMB returned two of the NAAQS rules regarding ozone NAAQS in
2011 due primarily to concerns about the implementation of the NAAQS rule on the
economy, where the rules remain to this day. 357
Since the red-lined documents provide only the changes occurring during formal
OIRA review, moreover, they miss all of the changes made at the suggestion or
recommendation of OIRA occurring at other points in the regulatory life cycle. A
review of some of the scattered documents included in EPA’s NAAQS rulemaking
dockets reveal that these pre-formal review exchanges also involve disagreements about
interpretation of scientific evidence. See Appendices F and H. While the examples in
NAAQS documents are necessarily incomplete, they do indicate that OIRA is not
reluctant to get “deep” in the science, as various interviewees suggested. 358
A few complementary studies also find that OIRA involved in technical and
scientific features of the agencies’ regulatory actions. In their detailed study of OIRA
involvement in EPA’s IRIS profiles during the Bush Administration, for example, the
Majority Staff of the House Subcommittee on Investigations and Oversight concluded
that OIRA played a central role in “challenging the science being done at regulatory
agencies.”359 “[T]he Subcommittee has ample documentation showing that OIRA’s staff
scientists did far more than merely coordinate and facilitate science discussions across
agencies. OIRA’s staff scientists directly challenged the science put forward by EPA
IRIS staff in very detailed peer review-type comments.”360
The Majority Staff include in their report extensive documentation of
communications between OIRA and the agencies that provide numerous examples of
355

See, e.g., Robert Percival, Who’s in Charge? Does the President Have Directive Authority over Agency
Regulatory Decisions?, 79 Fordham Law Review 2487, 2520-21 (2011); Juliet Eilperin, Ozone Rules
Weakened at Bush’s Behest, Washington Post, March 14, 2008, at A1., available at
http://www.washingtonpost.com/wp-dyn/content/article/2008/03/13/AR2008031304175.html
356
See the documents posted in EPA’s docket, cited at supra note 331
357
See Ozone return letter which put an end to the Network Design rule and the Ozone primary NAAQS
review, available at
http://www.whitehouse.gov/sites/default/files/ozone_national_ambient_air_quality_standards_letter.pdf
358
See infra note 591.
359
Majority Staff of the Subcommittee on Investigations and Oversight, Nipping IRIS in the Bud:
Suppression of Environmental Science by the Bush Administration’s Office of Management and Budget at
2 (June 11, 2009), available at http://www.ucsusa.org/assets/documents/scientific_integrity/miller-irisreport-june-09.pdf.
360
Id. at 5.

88

substantive editorial change(s) by OIRA to EPA’s characterization of the science. As just
one representative example, OIRA staff inserted the following bold passage to EPA’s
summary of the scientific literature in an IRIS profile:
this may imply that different activities may expose different age
groups more than others, or that some PBDE congeners may
accumulate differently with age, however the sample size here is
very small and firm conclusions cannot be made. 361
This type of editorial insert is similar to a number of the technical changes
identified in the NAAQS red-lined documents examined in this study; the technical edits
frequently qualify the characterization of the relevant literature in ways that alter the
weight or credibility to be attached to them. 362 In GAO’s 2003 study there are similar
references to substantive, technical changes by OIRA to EPA rules. For example, OIRA
suggested adding the qualification that EPA was adopting “plausible but highly uncertain
assumptions” to EPA’s preamble;363 in another rule, GAO identified how OIRA
successfully convinced EPA to remove manganese from the list of hazardous constituents
in its final rule.364
c. Implications
The risk that science-intensive rules can be changed in ways that are at odds with
public understanding or the scientific record are heightened when the rulemaking process
includes opportunities for agency officials to change the rules without public disclosure
or expert review. While the most salient example of this type of manipulation in recent
years did not involve OIRA or the White House, but instead involved actions by a high
level DOI appointee, Julie MacDonald, 365 a close examination of the McDonald
experience and other examples in which science has been politicized reveal a set of
circumstances – often present during OIRA review -- that seem to make rules particularly
susceptible to these behind-the-scenes manipulations. Specifically, in most if not all
cases of politicization of science, the process governing the integration of science into
regulation tends to involve: 1) a regulatory decision that involves high stakes; 2) highlevel interventions or related pressures occurring at a point in the decision process where
361

See id. at 7. Excerpted from interagency documents provided in full at Appendix A.
See Red-lined Drafts of Proposed and Final NAAQS and Habitat Designation rules at /wewagner/upload
for acus
363
GAO 2003, supra note 307, at 79.
364
See id. at 9.
365
During her tenure, MacDonald managed to alter the scientific record supporting species listing and
habitat designations through a variety of techniques. The result in a number of cases was alterations in the
agency’s rules. While MacDonald’s actions may have been motivated by a policy agenda, the changes she
made were to the characterization of the scientific evidence. See, e.g., Office of Inspector General, U.S.
Department of Interior, “The Endangered Species Act and the Conflict between Science and Policy”
(2008). This type of political meddling into the scientific record to alter decisions on policy grounds is
only one of many examples of the politicization of science by government officials. See, e.g., Holly
Doremus, Scientific and Political Integrity in Environmental Policy, 86 TEXAS LAW REVIEW 1601 (2008)
362

89

there is limited to no transparency regarding the changes, and 3) the absence of rigorous
scientific oversight or peer review of the changes, including changes to the
characterization or use of the relevant science. 366
Experience thus reveals significant risks associated with decision-making
processes that allow last-minute, high level changes to an agency’s characterization of
scientific or technical evidence, especially where these changes are not recorded,
justified, or subjected to scientific or technical review. Section 6(a) attempts to counter
some of these risks through its direction to the agencies to document the changes made as
a result of OIRA’s review, both formal and informal. Yet compliance with Section 6(a)
by the agencies in the rules studied here was incomplete. While the preliminary study of
NAAQS and ESA rules conducted here illuminates some features of OMB review in
need of further study, it also highlights areas worthy of preliminary recommendations by
ACUS.
First and as mentioned above, EPA and FWS, for the rules studied here, did not
identify or explain the “changes made at the suggestion or recommendation of OIRA” as
required by Section 6(a)(3)(E)(iii) of Executive Order 12866. For some rules, numerous
exchanges occurred between OMB and EPA both inside and outside of the formal review
process and yet none of the resultant changes made at the suggestion of OIRA to its
regulatory actions were identified by the EPA. The 2009 GAO study similarly concludes
from interviews of OIRA officials that changes, particularly during informal review, can
be significant and yet are undisclosed. 367

366

See, e.g., id. at 1640 (concluding that “[t]he single biggest contributor to the lack of political integrity in
this Administration’s environmental-policy decisions is the absence of barriers between political appointees
who view their mission as the single-minded advancement of the President’s policy agenda and career
employees charged with providing scientific advice or analysis.”). Julie MacDonald, for example, edited
decisions in ways that were largely invisible to outsiders and that often occurred at the “eleventh hour.”
See, e.g., OIG Report, supra note 365, at 1 and 16. MacDonald also had conflicts of interest on some of the
decisions that were unknown to others and influenced her decisions. See id. at 3. One of the case studies
elaborates on these features that are particularly likely to lead to the politicization of science. In one case:
The [FWS] rule drafted by the field biologists represented the views and
recommendations of the biologists, and as the rule for the CHD traveled through the
region, fairly few edits were made to the recommendations of the biologists. However,
once the rule was forwarded to Washington, D.C., [a FWS biologist] said the rule
underwent significant editing [by MacDonald].
Id. at 21. In another example, “Macdonald . . . changed the text of the final rule the day before it was
completed.” Id. at 33. In yet another “MacDonald recommended numerous edits to the gulf sturgeon rule
that were different than what the FWS recommended” and directed changes to be made “despite concerns
expressed by region biologists and solicitors that the record did not support” them. Id. at 56.
367
GAO 2003, supra note 307, at 95 (observing that in some of the rules reviewed, “OIRA suggested
significant changes prior to formal submission of the rule to OIRA). See also id. at 113 (concluding that
“restricting the transparency requirements in Executive Order 12866 only to a brief period of formal review
seems antithetical to the intent of those requirements.”); id. at 114 (“it is not clear who OIRA believes that
the executive order’s transparency requirements should not cover the part of the review period when the
most important changes can occur.”).

90

Second, even with respect to the agency’s responsibility for identifying the
changes occurring during OIRA’s formal review, compliance by the agency proposing
the rule is incomplete on several levels. For about one-third of the NAAQS and ESA
rules, there was no explication of the substantive changes that occurred during OIRA
formal review. In addition, for the remaining rules, the agency’s red-lined version of the
proposed or final rule did not “[i]dentify for the public, in a complete, clear, and simple
manner, the substantive changes between the draft submitted to OIRA for review and the
action subsequently announced” as required by subsection (E)(ii) (emphasis added).
These red-lined documents were also very difficult to locate within the larger, individual
rulemaking dockets and not provided in a “simple” manner.
More complete agency compliance with the requirements of Section 6(a)(3)(E)
for science-intensive rules would help abate the risk that these rules can be changed in
ways that escape not only understanding by the public, but by scientists familiar with the
technical details of the rule. Recommendations in the final section identify how this
transparency of these changes can be increased in the future.
B. Agency Efforts to Improve the Scientific Integrity of their Processes
In recent years, the suppression and editing of staff technical analyses by political
appointees in ways that substantially change the analyses have received a great deal of
attention.368 Much like the more general criticisms of agency transparency, however,
these politicization problems with agency science are not well characterized. To be sure,
there are publicized accounts of staff suppression and the editing of technical memoranda
by agency management in ways that alter the characterization of the scientific
information, but they are relatively few in number. 369 Likely more pervasive, but even
more difficult to document, are concerns by agency personnel and outsiders that there is a
risk of group think and the discouragement of dissenting views in some agency settings.
This type of group think or top-down narrowing of acceptable views may not be
malicious or even conscious, but if it occurs it can impair the quality of the underlying
technical analysis by limiting the vigorous skepticism afforded the agency’s analysis. 370
The President’s directive on scientific integrity identifies all of these scientific
challenges, but it focuses most intently on the need to halt the overt manipulation of
agency science by supervisors and political appointees. 371 According to one former

368

See, e.g., supra note 5; Testimony of Jeff Ruch, PEER, Endangered Species Act Implementation:
Science or Politics, before the House Natural Resources Committee, May 9, 2007, available at
http://www.peer.org/news/news_id.php?row_id=854.
369
See, e.g., Doremus, supra note 5; OIG MacDonald Report, supra note 11.
370
See infra Section IV.B.3.
371
Interviews with past OSTP employees confirmed that the laser-like focus from the Obama Scientific
Integrity directive was on this politicization of science, with less focus on other problems with regulatory
science that were known to be problematic, including regulatory review, the quality of regulatory science,

91

Office of Science and Technology Policy (OSTP) official, there was strong support
within all the agencies for this narrower focus. Agency scientists were purportedly
“jubilant” at the prospect of finally addressing this issue formally within the Executive
Branch. 372 Not surprisingly, the agencies’ science integrity guidelines promulgated in
response to the President’s and OSTP’s directives primarily address the “science
politicization” problem and defer or only lightly address the challenge of encouraging
diverse views and internal skepticism. In particular, the signature feature of most of these
agency and departmental scientific integrity guidelines is the establishment of a scientific
misconduct process. A number of agencies’ scientific integrity guidelines also include
codes of conduct and communications policies, so that agency scientists can publish,
speak with the press, and express their views without clearance requirements. 373 While
the current guidelines are still developing and thus are inevitably incomplete, they
constitute a solid foundation for future integrity programs. Regardless, the purpose of
this report is to provide only a brief synopsis of these emerging programs without a
detailed critique of their details.
A review of these recent scientific integrity policies, as well as other relevant
programs in place at DOI and EPA at the time of the study, is provided below. The final
section then considers NRC’s approach to these problems, which have been evolving for
several decades and have been implemented independently of the recent White House
initiative.
1. Scientific Integrity Policies in response to the White House Initiative
a. The Department of the Interior
DOI was the first agency to develop a final science integrity policy in response to
OSTP Director John Holdren’s memorandum requesting these polices. Recall that
President Obama requested OSTP to develop a government-wide program to enhance the
scientific integrity of agency work, and Director Holdren’s subsequent memorandum set
the parameters for that program.374 DOI officials indicate that this integrity policy is only
the first in a series of policies that they intend to promulgate to address issues of scientific
integrity. 375 DOI is currently working on a communications policy in response to the
and how to set policy on scientific communications from an agency. Interview with former OSTP
employee, Feb. 14, 2012; Interview with former OSTP employee, Jan. 23, 2012.
372
Interview with former OSTP employee, Jan. 23, 2012.
373
See generally OSTP, Scientific Integrity Policies Submitted to OSTP, available at
http://www.whitehouse.gov/blog/2011/08/11/scientific-integrity-policies-submitted-ostp.
374
See supra notes 6-7 and accompanying text.
375
Interview with Department of Interior Staff, Aug. 29, 2011. There was some internal disagreement
about the strategy of developing guidelines that addressed only one issue at a time. A decision was
ultimately made to start with the low hanging fruit and gradually develop policies on these other, more
difficult areas as consensus builds. Without this more incremental approach, there was also a concern
about paralysis and/or extended delays in which the DOI would operate with absolutely no integrity
guidelines in place. Id.

92

scientific integrity initiative and has plans to address a succession of other issues over
time.376
The goal of DOI’s current integrity policy, which is focused primarily on policing
misconduct and establishing codes of conducts, is to provide agency scientists with
autonomy and the support of the organization if they believe they have been coerced or
otherwise bullied in ways that go beyond basic minimum standards of scientific
integrity. 377 The hope is that this policy will allow problems within the Department to be
identified and corrected at an early, internal stage of the deliberations, before they grow
into larger public controversies. 378
In its initial scientific integrity policy, DOI has established a relatively elaborate
scientific misconduct process.379 Under the policy, DOI staff and outside parties can
make allegations of scientific misconduct against a staff or official within DOI. 380 The
scientific misconduct process thus provides a mechanism that is new to DOI policy for
formally reporting and punishing scientific fabrication, including editing technical reports
in ways that are not supported by the evidence. 381 While the definition of misconduct is
relatively narrow and includes only intentional or reckless “fabrication, falsification, or
plagiarism” in scientific activities,382 DOI’s policy also encompasses “intentionally
circumventing policy that ensures the integrity of science and scholarship” or in ways
that “compromise scientific and scholarly integrity.”383
DOI’s process involves the initial submittal of a non-anonymous allegation of
misconduct by anyone, including persons outside the agency. 384 If the allegation is
determined to be credible, then an inquiry will be conducted.385 If it is ultimately
376

Id.
Id.
378
Id.
379
Department of Interior, Departmental Manual 305 DM 3, Jan. 28, 2011, available at
http://www.whitehouse.gov/sites/default/files/microsites/ostp/DOI-DM-sci-integ.pdf [hereinafter DOI
Scientific Integrity Guidelines]. A critical OIG report, published in April 2010, undoubtedly provided an
added urgency; this April 2010 report focused exclusively on the failure of the DOI to develop a scientific
misconduct program and was published only one year after the large, OIG investigation of the Julie
MacDonald case. OIG, DOI, Evaluation Report: Interior Lacks a Scientific Integrity Policy, Report No.
WR-EV-MOA-0014-2009 (April 2010); see also supra note 222.
380
DOI Scientific Integrity Guidelines, supra note 379.
381
DOI’s misconduct provisions borrow from the HHS, Office of Research Integrity regulations on
scientific misconduct, 42 C.F.R. Part 93. It may be the case that at least some of the DOI’s integrity
regulations were overdue. In 2000 OSTP issued a policy requiring federal agencies to adopt scientific
misconduct regulations following ORI’s model for extramural and intramural research. See
http://ori.hhs.gov/federal-policies. The extent to which DOI’s 2011 policy simply satisfies this earlier 2000
command would benefit from further research.
382
DOI Scientific Integrity Guidelines, supra note 379, at 3.5M; see also id. 3.8A (adding the intentional
and reckless requirement and cautioning against using the process for honest differences of opinion).
383
Id.
384
Id. at 3.8A(1).
385
This inquiry is conducted by the Department or Bureau Scientific Officers working with the responsible
manager and an assigned Servicing Human Resources Officer. See id. at 3.8B through F
377

93

determined that intentional or reckless misconduct occurred, then sanctions – spanning
the range from termination to reprimand – may be issued.386 As mentioned, DOI’s
integrity policy also offers codes of conduct and principles to guide staff behavior,
although these objectives appear difficult to enforce, in part because of their ambiguous
definitions.387
DOI’s scientific integrity policy has been criticized by several public interest
groups for failing to establish a public tracking system for the misconduct complaints
filed in the Department.388 DOI has also been criticized for failing to address other
important science integrity issues in its first set of guidelines, such as enhanced
whistleblower protections.389 DOI admits that its first policy has a limited range in
386

Sanctions are issued at the discretion of by the manager and the Servicing Human Resources Officer.
See id. at 3.8G. DOI is currently developing a training and outreach program to educate staff about the
program and how they can report misconduct within the Department. Interview with Department of
Interior Staff, Aug. 29, 2011. Within the Department, there is a collaborative learning network (not
available to those outside the agency) that provides staff with an online training tool, including case studies
and other features. The training is considered a key feature of the new integrity initiative at DOI. Id.
Presumably broad outreach also helps deter abuses of scientific integrity within the agency by advertising
the potentially high costs of this abuse.
387
For example, one provision directs staff to “[d]ocument the scientific and scholarly findings considered
in decision making and ensure public access to that information and supporting data through established
Departmental and Bureau procedures – except for information and data that are restricted from disclosure
under procedures established in accordance with statute, regulation, Executive Order, or Presidential
Memorandum.” DOI Scientific Integrity Guidelines, supra note 379, at 3.4.C. Other, even more ambitious
(but unenforceable) goals include a requirement that employees commit to “clearly differentiate among
facts, personal opinions, assumptions, hypotheses, and professional judgment in reporting the results” and
to “fully disclose methodologies used, all relevant data, and the procedures for identifying and excluding
faulty data.” See id. at 3.7A.(7) and 3.7.B.(3). See also id. at 3.7A.(9) and 3.7.B. (1)-(4). The DOI
integrity rules close with guidelines for staff serving on the boards for professional societies. Employees
serving in these roles in their personal capacity do not appear to require approval, but they are urged to
consult with ethic’s officers to ensure their compliance with conflicts of interest and ethical standards. Id. at
3.9.B.(1). Approval is required of employees who wish to engage in these activities in their official
capacity. See generally id. at 3.9.
388
Interview with Staff, Union of Concerned Scientists, June 28, 2011; Interview with Staff, PEER, July
27, 2011; Interview with Department of Interior Staff, Aug. 29, 2011. Currently, all of the misconduct
proceedings remain internal to the agency and are not publicly available or summarized. Interview with
Department of Interior Staff, Aug. 29, 2011. DOI defends its position as necessary to protect the
confidentiality of the accused and as a way to encourage employees to feel freer to identify problems early
in the process, before they become media events. Id.
389
Stakeholders argue that a fundamental mechanism for ensuring that the misconduct program will
actually be utilized is the institutionalization of parallel protections for complainants through heightened
whistleblower protections. Without heightened protections against retaliation, there are substantially
reduced incentives for employees to report misconduct and a corresponding risk that the complaint process
will be underutilized. See, e.g., Statement of Francesca Griffo, UCS, Interior Department’s New Scientific
Integrity Policy Must Trigger Significant Changes To Be Effective, Feb. 1, 2011, available at
http://www.ucsusa.org/news/press_release/interior-departments-new-SI-policy-0495.html. Even in
universities, incidents of research misconduct are grossly under-reported, presumably because of scientists’
fears of retaliation as well as a perception of being viewed as non-collegial. See, e.g., Bob Montgomerie &
Tim Birkhead, A Beginner’s Guide to Scientific Misconduct, 17 ISBE Newsletter, May 2005, at 16.
Finally, the public interest groups have criticized DOI for failing to adopt a more liberal communications
policy that allows Department staff and officials to publish studies or speak with the media. They point to
the FWS’s more liberal publication policies, which allow FWS to speak openly provided they include a

94

addressing problems of scientific integrity. 390 However, DOI officials maintain that if one
must start with the most serious problems, this policy is a first, strong step in the right
direction.391
It is worth noting that after publication of the DOI policy, the National Oceanic
and Atmospheric Administration (NOAA) published a parallel scientific integrity policy
that is generally viewed as similar, but more beneficial. 392 Although NOAA is not within
the three agencies under study here, because its guidelines build heavily on DOI’s
guidelines, because the National Marine Fisheries Service within NOAA is also
responsible for the protection of endangered species, and finally because NOAA is
viewed by public interest groups as considerably more effective, they are discussed
briefly in closing.393 NOAA’s policy, like DOI’s, establishes a scientific misconduct
program, 394 but NOAA ultimately departs from DOI’s policy in several important ways.
First, NOAA commits to an annual, public reporting of misconduct allegations and

disclaimer that they are not speaking for the agency. FWS allows its scientists to publish without
management approval provided they add a sentence that the contents “do not necessarily represent the
views of the U.S. [FWS].” Fish and Wildlife Service, Information and Expression: Part 117
Communicating Scientific Information, January 26, 2010, at § 1.4, available at
http://www.fws.gov/policy/117fw1.pdf. The FWS’ policies, they maintain, should serve as an example of
what the entire Department of Interior should develop for staff scientists. DOI officials indicate that they
are currently working on such a policy. Interview with Department of Interior Staff, Aug. 29, 2011.
390
Specifically, misconduct is narrowly defined. Suppression of science and even bullying of staff to alter
their findings or analysis might not be covered by the policy if these acts are not clearly intentional or
reckless, yet these negligent acts are likely to be more prevalent than outright misconduct. The policy also
does nothing to encourage good faith disagreements and debates among scientists within the Department.
At the time of this report, there was only one known allegation and it was filed by PEER, a
watchdog group, based on personnel actions taken against a scientist researching the Arctic polar bear
(dubbed Polarbeargate). Suspended Polar Bear Research Defended by Advocates, ScienceInsider, July 29,
2011, available at http://news.sciencemag.org/scienceinsider/2011/07/suspended-polar-bearresearcher.html. This incident is still under investigation by the Department’s OIG. For a recent critical
account of that investigation, see PEER, Polar Bear Probe Careens in New Directions, Oct. 26, 2011, at
http://www.peer.org/news/news_id.php?row_id=1527. The allegations in PEER’s case do not concern
management’s manipulation of the technical or scientific analysis, but rather the halting of a research
biologist’s work without cause and stigmatizing his research through a criminal investigation of
undisclosed “integrity” issues. See PEER complaint, available at
http://www.peer.org/docs/doi/7_28_11_Scientific_Misconduct_Complaint.pdf.
391
Interview with Department of Interior Staff, Aug. 29, 2011.
392
See, e.g., UCS, NOAA Boosts Scientific Integrity with New Policy, Dec. 7, 2011, available at
http://www.ucsusa.org/news/press_release/noaa-boosts-scientific-integrity-1357.html. It is not clear why
NOAA, rather than the Department of Commerce, developed these guidelines. The most likely answer is
that NOAA was determined to put into place integrity guidelines and did not want to wait for the
Department’s leadership on this issue.
393
In the development of its scientific integrity policy, NOAA solicited and responded to more than one
thousand different, substantive comments on its draft policy. See NOAA’s Disposition of Comments
Received on Draft Scientific Integrity Guidelines 3 (Nov. 21, 2011), available at
http://nrc.noaa.gov/Public_Comments_Disposition.pdf.
394
See NOAA Scientific Integrity Policy, NAO 202-736D, Dec. 7, 2011, available at
http://www.corporateservices.noaa.gov/ames/administrative_orders/chapter_202/202-735-D.html.

95

proceedings occurring within the agency. 395 This addresses some of the most vigorous
public interest group complaints regarding DOI’s policy. Second, NOAA attempted to
draft its policy in a way that not only develops ambitious codes of conduct, but makes
them easier to enforce because of their specificity and because the code of conduct is
explicitly referenced as enforceable.396 Finally, NOAA’s integrity policy references a
parallel communications policy that should allow agency scientists to speak and publish
freely, provided they provide disclaimers when they are not acting in their professional
capacity. 397
b. EPA
In contrast both to NOAA and DOI, EPA has had a scientific misconduct program
in place since 2000 (EPA’s program was established in response to an OSTP federal
policy developed under President Clinton).398 A detailed line-by-line comparison of
EPA’s program with those developed more recently by DOI and NOAA was not
attempted in this study, but EPA’s program appears relatively complete 399 and the Health
and Human Services (HHS) Office of Research Integrity site refers to EPA’s program as
a finalized program. 400
395

See § 10.04 of id. (“NOAA’s Chief Scientist . . . will provide annual public reporting . . . of the
aggregate number of misconduct cases, the areas of concern, the affiliation of the individuals involved, how
much accusations were investigated, and the number of findings of misconduct.”). NOAA staff indicates
that ultimately, because of the lack of a Chief Scientist, “the Chair of the NOAA Research Council will
make this annual reporting available.” Statement of NOAA Staff, Jan. 26, 2011.
396
Violations must be intentional and reckless and constitute some form of scientific misconduct to be
enforceable. See §§ 1.01(b) and 8.01 of NOAA Scientific Integrity Policy, supra note 394. Nevertheless,
NOAA evinces an intent to include as scientific misconduct, violations of its code of conduct. See § 5.01
of id. (stating that “[a]ll staff identified in Section [3].02 must uphold the fundamental Principles of
Scientific Integrity [Section 4], the Code of Scientific Conduct [Section 6], and the Code of Ethics for
Science Supervision and Management [Section 7] . . .”) (emphasis added); see also § 1.01 of NOAA,
Procedural Handbook for Scientific Integrity, available at
http://www.corporateservices.noaa.gov/ames/administrative_orders/chapter_202/Procedural_Handbook_N
AO_202-735D_31Jan_2012.pdf (noting that “[a] finding of Scientific and Research Misconduct requires a
determination by the Determining Official”; and under § 1.01(a), the determination is based on significant
departures from the Code of Scientific Conduct or Code of Ethics for Science Supervision and
Management). Since NOAA’s policy contains ambitious principles for its code of conduct, this is an
important feature of the guidelines. See NOAA Scientific Integrity Policy, §§ 4-7, supra note 394.
397
See id. at § 4.05 (noting that “NOAA scientists may freely speak to the media and the public about
scientific and technical matters based on their official work”); id. at § 4.06 (stating that “NOAA scientists
are free to present viewpoints . . . that extend beyond their scientific findings to incorporate their expert or
personal opinions, but . . . must make clear they are presenting their individual opinions”).
398
Apparently, neither NOAA nor the DOI complied with this 2000 policy, although presumably their
recent programs now satisfy this directive. See HHS, Summary of Agency Misconduct Policies, available
at http://ori.hhs.gov/federal-research-misconduct-policy. EPA simply cites to this existing program in its
draft guidelines. EPA Scientific Integrity Policy at 10, available at
http://www.epa.gov/osa/pdfs/epa_scientific_integrity_policy_20120115.pdf.
399
EPA Scientific Misconduct Policy, available at http://ori.hhs.gov/sites/default/files/epapolicy.pdf .
400
HHS Summary of Misconduct Policies, available at http://ori.hhs.gov/federal-policies. EPA’s
misconduct program was criticized in July 2011 by EPA’s OIG, however, for inadequate training of EPA
staff and insufficient updating and monitoring of the program. See EPA Office of Inspector General,

96

Also in contrast to DOI and NOAA, EPA has not been as diligent in preparing its
science integrity guidelines. EPA’s guidelines were distributed in draft in August 2011,
six months after DOI’s guidelines were final. 401 It is worth noting that there are relatively
few accounts of overt suppression and scientific manipulation at EPA, 402 at least as
compared with DOI’s travails. 403 Although it may not fully explain EPA’s apparent
apathy with regard to complying with the White House’s scientific integrity initiative,
EPA may in part lack a sense of urgency with respect to this particular problem.
2. NRC’s Procedures to Enhance the Scientific Integrity of its Decisions and its
Staff404
The NRC has had relatively elaborate policies in place for several decades to
preserve the scientific integrity of its decision-making process and staff; these policies
were thus developed completely independently from President Obama’s scientific
integrity initiative.405 The policies include opportunities for the airing and resolution of
internal disagreements through informal and formal channels. 406 As elaborated below,
Office of Research and Development Should Increase Awareness of Misconduct Policies (July 2011),
available at http://www.epa.gov/oig/reports/2011/20110722-11-P-0386.pdf. Although EPA reports that it
receives very few allegations of misconduct from within the agency, the OIG warned that this low rate
could well be due to this insufficient outreach and training by the agency. Id. at 8.
401
While the fate of these guidelines is an open question, the draft guidelines are very general and do not
appear to create enforceable responsibilities or provide meaningful additions to EPA’s existing
commitments to scientific integrity, except perhaps for the appointment of a Scientific Integrity Officer and
committee within the agency. EPA Scientific Misconduct Policy, supra note 399, at 2 and 9-10. EPA’s
draft policy also seems to studiously avoid offering specifics on communications and whistleblower
policies, although there is a reference to the ability of EPA staff to publish and speak to the press, provided
they indicate that it does not state the views of the agency. See id. at 5.
402
EPA staff was apparently prevented from conducting an assessment of the adverse consequences of coal
ash [when did this take place?], for example. Interview with PEER Staff, July 27, 2011. See EPA OIG,
Agency Handling of Coal Ash, Sept. 8, 2008, available at
http://peer.org/docs/epa/7_18_11_EPA_sci_suppression.pdf; (discussing EPA staff that report that EPA
suppressed the risks of coal ash and lacked a robust process for reviewing drafts on the subject posted); see
also EPA OIG Audit Report, EPA Promoted the Use of Coal Ash Products with Incomplete Risk
Information, March 23, 2011, available at http://www.epa.gov/oig/reports/2011/20110323-11-P-0173.pdf;
and PEER press release on the issue. http://www.commondreams.org/newswire/2011/07/18. Sixty Minutes
also ran an investigation of these and related issues concerning the lax regulation of coal ash. See video
clip at http://www.huffingtonpost.com/2009/10/04/coal-ash-on-60-minutes-un_n_309268.html.
403
Interview with Former EPA Staff, Dec. 21, 2011. Note that “interference” with EPA science from OMB
is, however, considered a potentially significant problem. See infra Section IV.A.1.
404
This section benefitted significantly from comments and research assistance provided by Roland Frye.
405
Recall that as an independent agency, the NRC is not bound by the President’s scientific integrity
initiative; NRC has in fact indicated it will not issue its own scientific integrity policies in response to the
OSTP memorandum. Interview with NRC Staff, Oct. 26, 2011.
406
Although communications policies and whistleblower protections were not studied here because they
are significant enough issues to warrant a separate study of their own, it is worth noting in passing that
NRC also appears to have a relatively liberal communications policy. According to interviewees and
written (but somewhat ambiguous) policies, members of the NRC staff are allowed to speak with the media
and publish their work without clearance, provided they do not ascribe their private views to NRC.
Interview with NRC Staff, Oct. 26, 2011. The applicable Directive – Directive 3.9 – is unclear on this
point, however. [Can we cite this directive?] It suggests clearance is needed to discuss NRC work, but

97

the NRC’s approach to scientific integrity may well provide a model or best practices that
could be used by other agencies for a broader range of issues, particularly with respect to
encouraging diverse views and skepticism within the government.407
a. Encouraging Dissenting and Diverse Views
In order to facilitate the free exchange of ideas from within the agency, NRC has
developed an elaborate “Collaborative Work Environment Program.” This program
consists of three separate policies, two of which are more than three decades old. These
cumulative policies are intended to create an environment in which employees can
disagree and publicly question decisions by those higher in the chain of command. 408
Open Door Policy
NRC’s first initiative in this program – the open door policy – was initially
created in 1976409 to provide staff with an opportunity to meet with supervisors beyond
their immediate supervisor to discuss disagreements over technical issues and other
matters.410 The current directive provides that “[a]ny employee may initiate a meeting
with an NRC manager or supervisor, including a Commissioner or the Chairman of NRC,

then provides a disclaimer in settings where the publication or speech is not cleared by NRC. See NRC
Management Directive 3.9 Handbook at 6, available at
http://pbadupws.nrc.gov/docs/ML1100/ML110070679.pdf. NRC Management Directive 7.3 Handbook, at
7, available at http://pbadupws.nrc.gov/docs/ML0414/ML041410583.pdf only further muddies the waters.
It states “All speeches, papers, or journal articles prepared by an employee for a professional organization
that relate to NRC technical, legal, or policy issues should be reviewed in accordance with MD 3.9.(a).”
This includes information about technical issues unless the information is deliberative or confidential. By
contrast, if staff members purport to speak on behalf of NRC, their statements must be cleared through the
agency. Management Directive 3.9. Staff members are also encouraged to participate in professional
societies and other similar activities, although participation through NRC (under salary) requires supervisor
approval. See Handbook for Directive 7.3 at p.6. In cases when employees participate in their private
capacity, they may again proffer whatever technical or opinion statements they wish, provided they “make
clear to the organization that the views expressed by the employee in the course of participation are not
necessarily those of the NRC.” Id.
407
The DOE appears to have a differing professionals program as well, at least for technical issues. See
DOE P 442.1 “Differing Professional Opinions on Technical Issues”. Before concluding that the NRC
program provides the most complete model, further research is recommended into both DOE and other
agencies not studied in this report.
408
The NRC has a webpage dedicated to this Collaborative Work Environment Program. The website
states that “[i]n some organizations, being a “team player” means accepting management's preliminary
views during the decision-making process and not “rocking the boat.” Being an NRC Team Player does not
mean those things. NRC holds its employees to a higher standard of involvement and responsibility for the
decisions that are made.” http://www.nrc.gov/about-nrc/values/open-work-environment.html.
409
NRC Office of Enforcement, Review of the NRC Differing Professional View/Opinion Program,
ML082190414, at 1 (2007) [hereinafter DPO 2007 Review].
410
NRC, Management Directive 10.160, “Open Door Policy”, available at
http://pbadupws.nrc.gov/docs/ML0414/ML041490186.pdf. The policy indicates it was approved in 1991
and revised in 1997.

98

to discuss any matter of concern to the employee.” 411 Managers are required to honor an
employee’s request for confidentiality unless the manager is a Commissioner or unless
specific explicit exceptions make the promise of confidentiality impracticable. 412
Managers are also prohibited from retaliating against employees who utilize the
program. 413 There is no formal tracking of the use of this policy, but there are now
relatively vigorous agency efforts to educate staff about its existence.
Differing Professional Opinions
In 1980,414 the NRC developed the “differing professional opinions” (DPO)
program415 to provide a formal process for “expressing differing professional
opinions . . . concerning issues directly related to the mission of NRC” and to provide
prompt resolution of these disagreements through an impartial review by knowledgeable
personnel. 416 Staff members are allowed to prepare formal statements of dissent against
decisions that have already cleared staff review and are effectively conclusive decisions
of the NRC.417 The dissent is not only placed in the record, but it is actually adjudicated
to determine whether the official position of the agency should be adjusted.418 The
adjudication is conducted by a ad hoc panel. 419 The DPO submitter recommends potential
candidates for this ad hoc panel, but panels are ultimately selected by the Office Director
or Regional Administrator of the office of the submitter under guidelines that require one
of the panelists to be an employee recommended by the submitter.420 If an adjustment is
411

NRC, Open Door Policy, Handbook 10.160, at 1, available at
http://pbadupws.nrc.gov/docs/ML0414/ML041490186.pdf.
412
Id.
413
Open Door Policy, Directive 10.160, id., at 2.
414
DPO 2007 Review, supra note 409, at 1.
415
NRC Management Directive 10.159, Differing Professionals Opinions Program [hereinafter DPO
Program]. This Directive indicates it was initially approved in 1999 and revised in 2004.
416
Id. at 1 of Directive.
417
NRC Management Directive 10.158, Non-concurrence Process, at 1 of the Handbook, available at
http://pbadupws.nrc.gov/docs/ML0706/ML070660506.pdf [hereinafter NCP] (“The DPO Program applies
only to positions that are no longer under staff review, and has certain prerequisites and exclusions that do
not apply to the NCP.”). The DPO process is only available to those who make the case that the issues
cannot be resolved through informal channels and also cannot be used to raise grievances covered under
other programs. DPO Program, supra note 415, at 2 of DPO Handbook (listing “issues that do not qualify”
for DPOs). There is a formal screening step that involves culling out the nonqualifying DPOs by NRC
staff. Id. at 4-5. While there are provisions for protecting the confidentiality of the submitter of a DPO, id.
at 4, the DPO process is not available to anonymous filers. Id. As with the open door policy, users of the
DPO program are supposed to be protected from retaliation. Id. at 14-15. DPO submitters may also
receive a certificate of appreciation at the end of the year for raising issues of concern. DPO 2007 Review,
supra note 409, at v.
418
Not surprisingly, the composition of this panel remains controversial in individual case and appears to
be a continuing challenge in implementation of the program. Id. at 8-9. The DPO submitter recommends
potential candidates for this ad hoc panel, but panels are ultimately selected by the Office Director or
Regional Administrator of the office of the submitter under guidelines that require one of the panelists to be
an employee recommended by the submitter. DPO Program, supra note 415, at 6-7 of Handbook.
419
DPO review, supra note 409, 2007 at 6. Not surprisingly, the composition of this panel remains
controversial in individual case and appears to be a continuing challenge in implementation of the program.
NRC 2007 review at 8-9.
420
DPO Program, supra note 415, of Handbook at 6-7.

99

required, this adjustment occurs through a formal hearing as well as an appeal process. A
flow chart provides the numerous, discrete steps for the process (CDs=calendar days).421

Although the volume of DPOs in the NRC is not terribly high, 422 the NRC
appears to take the program seriously and has struggled over the years to make it
credible. An audit report by the NRC Office of the Inspector General in 2000 produced a

421

The figure was copied from id. at Exhibit 2. While the DPO process is designed to allow issues to move
expeditiously through the agency, in practice the process can take much longer. Generally, the process
takes over six months and in at least one case dragged out over nearly two years. DPO 2007 Review, supra
note 409, at 5 and 10-11. The extended time frame for resolving DPOs is a continuing problem for the
program. See also NRC OIG, Review of NRC’s Differing Professional View/Differing Professional
Opinion Program, OIG-00-A-07, Sept. 20, 2000, at 15-17 [hereinafter OIG Report on DPO]. Appeals then
can take another six months, more or less. Id.
422
A 2007 assessment by NRC reports that no DPOs were filed in that calendar year, although a handful
worked their way through the resolution and appeal process. DPO 2007 Review, supra note 409, at v. [If
there were still no DPOs filed after the change in the program, it seems the change wasn’t very effective. Is
there any evidence of an increase in DPO filings more recently?]

100

relatively critical evaluation of the program. 423 The OIG concluded that there was
underutilization of the program due to fears of retaliation and a perceived lack of
effectiveness of the program. 424 Multiple subsequent assessments of the DPO
program, 425 including the NRC OIG report published in 2000, have resulted in a number
of changes to the program following the OIG’s recommendations. 426
Non-concurrence Process
Finally, in 2006 the NRC developed a Non-concurrence Process (NCP) that
provides a formal mechanism for those in the line of concurrence (and even those
employees not in the line but well-versed in the relevant issues) to formally lodge their
non-concurrence with draft policies or draft documents. 427 The NCP is a complement to
DPO since it applies to decisions that are still in draft and acknowledges that employees
will not always concur and thus should be provided with a process to either opt out of
concurrence or to file a statement of non-concurrence. 428
The NRC’s NCP process is not as process-intensive as the DPO process. The
submitter first drafts a non-concurrence, the non-concurrence is shared with the document

423

The Report was initiated in part based on general reports by NRC employees that they did not in fact
feel free to voice dissenting opinions despite the DPO policy. OIG Report on DPO, supra note 421, at 3-4.
424
Id. at 9. The OIG found that more than fifty percent of the (27) DPO submitters that OIG interviewed
believed that some form of retaliation had occurred after filing a DPO. Id. at 18. Additionally, the
guidelines for filing DPOs retained the submitter’s supervisors in the chain of command, which was
problematic in terms of chilling engagement, increasing the risk of retaliation, and impairing the fair
resolution of the disagreement. Id. at 10.
425
The agency has conducted its own internal evaluation of the program over the years, including annual
reviews. The DPO program requires that NRC conduct an in-depth annual program review, including
audits of office and regional performance records. DPO 2007 Review, supra note 409, at 3. For a
summary of the more extensive reviews of the program, see id. at 1. There are some boilerplate similarities
between these annual reports, but the NRC does collate the data from the prior year and summarize at least
the nature of the DPOs and their disposition, as well as identify areas for ongoing improvements and
reform.
426
These changes include improving employee understanding of and comfort with the program, broadening
the scope and coverage of the protections (e.g., to contractors), and building in more sources of feedback
and tracking of the programs. See DPO 2007 Report, supra note 409; NRC, Differing Professionals
Opinions Program 2006 Program Review, ML071160295 [hereinafter DPO 2006 Report]. Employee
outreach and training appear to be the highest priorities. DPO 2007 Report, supra note 409, at v; DPO
2006 Report, id., at v. To that end, NRC has established DPO liaisons in each office to facilitate use of the
program. DPO 2007 Report, supra note 409, at 22. In the annual review process, NRC also surveys DPO
submitters, panel members, and office members for feedback on the process and includes this information
in the annual assessment, see id. at 4 and Appendix C, although the results of this information were difficult
to trace in the written annual report. NRC annually also provides a brief summary of each of the DPOs
that have been filed and their current status in its annual, public report. This information provides a
valuable window into the nature of the disagreements that are emerging through this program. See
Appendix D and E of the 2007 and 2006 Review reports, supra note 409 and id., for example.
427
NPO, supra note 417.
428
Id. at 1 of Handbook. A sample of a Non-concurrence statement is provided at Appendix D.

101

sponsor who then describes any action taken, and both forms are then shared with the
document signer for his/her consideration. 429

Retaliation is explicitly discouraged in this policy by threats of disciplinary action
in accordance with the NRC’s policy. Employees are also provided a mechanism for
submitting grievances if they perceive they have been retaliated against. 430

429

The Figure is from id. at Exhibit 1. Id. at 2-3 of Handbook; see also id. at 6-11 (providing more
complete guidelines on the process).
430
Id. at 13. A 2010 OIG report on the non-concurrence process identified several ways that the process
could be improved. As an introductory matter, OIG noted that the NCP program was still in an interim
status, despite the initial intention of NRC to finalize it within a year after adoption (i.e., 2007). OIG Audit
of NRC’s Non-Concurrence Process, OIG-11-A-02, Oct. 7, 2010, at 1-2, 10, and 28. The OIG also noted
that NRC had dedicated very little in the way of staff or resources to either the DPO or NCP programs.
“[T]he fiscal year 2010 budget is $3000 for the entire Differing Views Program [Open Door, DPO, and
NCP programs], and 1.5 full-time equivalents are designated to collectively support the Differing
Professional Opinions Program and the nonconcurrence process.” Id. at 1. The crux of its review ,
however, highlighted the need for clearer guidelines for how employees could use the process. OIG
reported, for example, that “[i]nterview results revealed that 70 percent of filers, document sponsors, and
document signers did not understand their respective rights, roles, and responsibilities in relation to the
non-concurrence process as compared to that described in [the NCP] policy.” Id. at 7. Clearer guidelines
and more effective employee training constituted the primary recommendations for remedying the

102

b. Continuing Challenges
The most substantial impediment to the NRC’s efforts to provide an open
workplace stems from employees’ concerns about possible retaliation431 and perhaps a
natural inclination of employees to seek to “get along” and not anger colleagues or (even
more so) supervisors.432
The NRC has taken a number of actions to address these concerns, including
issuing awards to DPO submitters,433 publicizing success stories, 434 buttressing its antiretaliation oversight processes and policies, 435 and attempting to publicize the successes
of DPO submitters.436 The NRC also has policies that deter retaliation by levying
sanctions against those who retaliate against another employee for expressing his/her
concerns.437 And the NRC offers redress to employees who have been retaliated
against.438
There is also evidence that the processes are being used. For example, a
nonconcurrence filing served as one of the triggers for the recent controversy over the
NRC Commissioner Chairman Jackzo’s management style. 439 Indeed, in his statements
shortcomings in the program. Id. at 10-12. OIG noted that the fact that the NCP was suspended in interim
status for years may have aggravated these problems since NRC did not develop a tracking system to
identify problems during this transitional period. Id. at 13-16. This extended interim status may also send a
signal to employees that NRC does not take the program seriously.
NRC’s Office of Enforcement appears to be addressing each of the recommendations, although the
NCP program is still not finalized. See, e.g., Letter from Stephen D. Dingbaum to R. William Borchardt re
Status of Recommendations, Sept. 9, 2011; see also Letter from Roy P. Zimmerman, Director of Office of
Enforcement to Stephen Dingbaum, July 27, 2011, reporting on status of OE’s response to the OIG’s
recommendations. It appears that NRC also has not developed a tracking system for the non-concurrences
and DPOs, and thus the availability of a public tracking system seems even more remote. DPO Review
2007, supra note 409, at 11-12. Some of these NCP documents are nevertheless available through
searching the ADAMs document retrieval system on the internet. Filers can indicate that they wish their
filings to be made public, and it appears that by checking this box, the non-concurrence is then shared more
generally through ADAMs (with the appropriate redactions for confidential information). See page three of
the Non-Concurrence Process Form, NRC Form 757.
431
In its review, NRC reports that “[t]he OIG 2005 NRC Safety Culture and Climate Survey (referred to as
the safety culture survey) found that approximately one-third of employees believe submitting a DPO has a
negative effect on career development at the NRC.” DPO 2007 Review, supra note 409, at 15.
432
In the controversial decision to over-ride a technical decision that Davis-Besse should be shut down, for
example, several employees expressed their support for a shut-down, but indicated that they did not feel
strongly enough to file a DPO. NRC OIG, NRC’s Regulation of Davis-Besse Regarding Damage to the
Reactor Vessel Head, Dec. 30, 2002, at 13. It is not clear from the OIG report whether they were pointedly
asked whether they would file a DPO. It is also not clear what these employees would consider significant
enough to warrant filing a DPO.
433
DPO 2007 Review, supra note 409, at 14.
434
Id. at 23.
435
Id. at 14-15.
436
Id.
437
NRC Management Directive 10.99, cited at p.2 of the Open Door Policy Handbook, supra 410.
438
These employees have remedies through either a negotiated or a formal grievance procedure.
Management Directive 10.101, cited at id.
439
See Appendix D for a copy of this nonconcurrence filing.

103

in response to a recent congressional investigation, Chairman Jackzo reiterated the
usefulness of these submissions, reporting that two DPOs and twelve formal
nonconcurrences had been filed within NRC during 2011.440 While this highly
publicized nonconcurrence statement may not be typical, and in any event did not involve
disagreements over technical issues, it provides a formal mechanism for staff to publicly
question management that appears to be unusual in the federal government.
However, in at least some other settings where these open processes should, in
theory, have been useful to elevate staff concern about technical issues embroiled in the
NRC’s decisionmaking, they were not used.441 A focused study of the underutilization of
these processes at past, critical points in NRC decision-making might provide added
valuable information about how to reinforce the effectiveness and use of these
programs.442
C. Summary of the Findings
Rather than attempting to summarize this section through text, a summary table is
provided below that provides a thumb nail sketch of the findings with respect to a number
of process-related factors. The analysis section then attempts to draw lessons from these
various agency processes.
EPA NAAQS

EPA Pesticide

EPA IRIS

FWS listing

NRC informal
rules

# of
chemicals/species/
rules
Nature of the
scientific evidence

6 chemicals

1000 chemicals
(50/year)

500 chemicals

Dozens species/year

Unclear how
many rules/year

Large and robust
body of relevant
research; mostly
academic and
published; publicly
available in
journals

Often industrybased; sometimes
quite limited

Usually
considerably more
limited than the
evidence available
in EPA’s program
and some of it is not
published

Engineering and
technical;
Operational
information is
often critical

Statutory
Constraints on
Decision

5 year review
period; must use
FACA science
advisory board for
review

Generally heavily
industry-based;
sometimes quite
limited, but meets
minimum
standards of
information
necessary for an
assessment
All pesticide
registrations must
be reviewed every
15 years

None

Response to petition
in 90 days;
proposed decision
within 1 year

PreNPRM
comment
opportunities

Generally 7
separate public
comment periods

Typically 2
documents, each
subjected to public

One public
comment period; the
recommended dose

None

Mandatory review
by science
advisory boards
for some licensing
and informal
rulemakings
In some informal
rulemakings,
there is an

440

Jackzo letter to Issa, dated Dec. 12, 2011, at p.1.
See supra note 432.
442
From the research conducted for this report, there was no specific lessons learned study of why these
open workplace programs were not used in some of the more publicized incidents that question the
technical veracity of NRC decision-making. A lessons learned report that specifically examines why DPOs
were not filed in Davis-Besse, etc. might shed light on these fundamental questions.
441

104

on 4 documents

comment before
the proposed
decision

is not a rulemaking,
however

Yes. Multiple
drafts of 4
consecutive
reports; external
peer review
comments; EPA
considers all
comments and
responds to
CASAC comments
Yes

Yes. Planning and
risk assessment
drafts; summary of
comments; EPA’s
response

Draft assessment
and revised
assessment; all
comments,
including
interagency
comments; EPA
response to all
comments

Yes, but only after
the proposed rule is
published

Yes

Yes

External Peer
review

Multiple reviews
by mandatory
advisory body

Generally no

Yes, but level of
peer review depends
on size of
assessment

Staff and
management
collaborate on final
product
Yes, individual peer
reviewers

Role of OMB

Yes, it clears the
proposed and final
rules

Yes, 2 rounds of
comments on
assessment

Yes, but OMB
clears the proposed
and final rules for
critical habitat only

Role of
interagency
review

Included during
public comment

Two dedicated
stages to
interagency review

Included during
public comment

Stopping rules for
emerging science

Yes, formal policy

OMB only
involved to the
extent the
Paperwork
Reduction Act is
triggered
Included during
public comment
and informal
contacts
No

Yes, generally
closed after draft
assessment

Unclear, but short
timeframe likely
makes stopping
rules unnecessary

Stopping rules for
debate

Informal closure
by advisory board

EPA determines as
needed

Complicated by
interagency review

Short timeframe
makes it nonproblematic

Authorship or
attribution of
staff-authored
reports

Yes - dedicated
acknowledgements
on reports;
authorship types of
rights
Yes

Yes - team
authorship on
reports

Limited - attribution
to field office

Yes

Yes - dedicated
acknowledgements
on reports;
authorship types of
rights
Yes

Studies available
through an

Limited
availability:

Generally available,
although availability

Are scientific
analyses and
related technical
documents that
support proposed
decision publicly
available?

Internal peer
review

Reference list
provided to peer
and public
reviewers
Availability of
supporting

Yes, although
reference list may
need to be requested
from agency contact
in Fed. Reg.
Generally available,
although availability

extensive
workshop. There
may be other
outreach activities
as well,
depending on the
regulatory
decision.
Yes. Commission
papers prepared
by staff and other
supporting
documents are
almost always
publicly available

Appears yes;
supervisory chain
reviews and signs
staff paper
Often yes,
standing advisory
bodies required
by statute; One
standing body has
broad authority to
recommend
projects
OMB only
involved to the
extent the
Paperwork
Reduction Act is
triggered
No evidence;
presumably
included during
public comment
Unclear;
Commissioner
structure likely
facilitates
authoritative
decisions
Unclear; See
above re
Commissioner
structure
Minimal Manager signs
staff papers

Yes, included in
staff papers,
technical
documents, and
proposed rules
Mixed
availability: 1)

105

literature to public

electronic database

Ability to
compare changes
in proposed
decision against
underlying
scientific analysis
Dissent policies
for staff scientists

OMB review can
obscure this to
some extent

Misconduct
policies

IV.

of the unpublished
studies was not
investigated

of the unpublished
studies was not
investigated

Yes, in general
for informal
rulemakings; 2)
No (or limited)
for license
decisions, at least
with respect to
operator
information

Generally yes;
although no to the
extent that
interagency review
impacts the initial
scientific analysis
Informal, in part
built into authorship

It depends on the
analyses included in
the administrative
record

Yes, for informal
rulemakings

Informal, in part
built into
authorship

Unpublished
studies by
manufacturers
require preclearance and
cannot be viewed
until after the
registration
decision has been
made
Yes, although it is
not clear that all
policy-relevant
choices are
accessible to the
nonscientist
Informal, in part
built into
authorship

Yes, formal
program

Yes

Yes

Yes

Informal, but
dissents have been
placed in the record
in one case
Yes

Does not appear
there are
misconduct
policies, although
they would seem
to be almost
completely
covered by
broader dissent
and
nonconcurrence
policies

Analysis and Recommendations

Despite a wealth of bad publicity, the evidence in this study reveals that agencies
have made considerable strides in ensuring the rigor and transparency of their integration
of science into regulation. Some agencies are explaining how policy and science
intersect in their regulatory projects in sophisticated, yet accessible ways. 443 Likewise,
some agencies are establishing processes that ensure both expert and internal review of
their work, and, in connection with these processes, are providing a public record of the
changes they make in response to peer review and public comment.444 Agencies are also
establishing integrity policies that allow their staff to raise scientific differences with
supervisors through various informal and formal mechanisms. 445 Finally, agencies
increasingly use the Internet to post the reference list used in their decision-making and

443

This is exemplified by the policy assessment in the NAAQS process, see Section III.A.1.a., supra.
In the IRIS assessment, interagency comments as well as peer review and public comments are all
placed in the record and EPA provides a response to all of these comments in appendix A of its assessment.
See Section III.A.1.b., supra.
445
The NRC’s collaborative workplace program provides a model example. See Section III.B.2, supra.
444

106

even to make copies of documents consulted during that process readily available through
databases and hyperlinks.446
This study also reveals features of agency decision-making processes that would
benefit from further improvements. First, while some agencies are innovating in their use
of science, little of this innovation is recorded or shared across the government. One set
of recommendations advocates for more systematic collation of innovative approaches to
regulatory science and then catalogs the particularly impressive procedures that emerged
from this particular study as either best practices or innovations. While scientific and
policy circumstances vary from program to program, thereby limiting the ability to apply
one agency’s innovations to other agencies in a mechanical way, certain minimal best
practices can and should be adopted by all agencies engaging in scientific decisionmaking.
Second, the study exposes a number of external constraints on agency decisionmaking processes that limit the ability of the agencies to improve their decision processes
in keeping with the President’s scientific integrity initiative. A second set of
recommendations advocate various forms of action with respect to these external barriers
to good regulatory science. In this study alone, for example, a Clinton Executive Order
caps the number of discretionary advisory committees that agencies can form; statutory
barriers impede the public’s access to studies that informed the agencies’ scientific
analysis; presidential review processes can alter the science underlying a rule but are
often nontransparent; and abbreviated statutory deadlines for rulemakings impede the
ability of the agencies to develop rigorous and transparent processes for integrating
science into regulation.
Third, in a number of health and environmental programs agencies necessarily
must rely on scientific research produced by regulated parties, yet the agencies do not
subject it to vigorous oversight processes. At the very least, this privately produced
research should be subjected to the same types of requirements imposed on research
published in scientific journals and research produced with federal monies.
Fourth, the findings clearly reveal the need for additional study of regulatory
science. The agencies’ use of external peer reviewers, for example, is vital to ensure both
the rigor and transparency of the integration of science into policy, yet very little is
understood about this feature of agency decision-making. This and other research topics
are integral to both an understanding and reform of the agency’s use of science for policy.
A few future topics emerging from this study that are highly recommended for future
study are listed in the final section.
446

EPA has established a large database of all of the studies used to conduct the NAAQS review. See
Section III.A.1.a., supra.

107

A. Capitalizing on Agency Successes
This section highlights various procedures and other innovations utilized wholly
within agencies that significantly advance the “transparency in the [agency’s]
preparation, identification, and use of scientific and technological information in
policymaking.”447 These approaches, by and large, have neither been cataloged nor
publicized; the report thus serves a valuable service simply in publicizing these advances.
The section begins by noting this dearth of investigations of agency successes in sciencebased regulation and the resultant lack of models of particularly good integration of
science into decision-making in the general literature. Several recommendations seek to
address this oversight, at least preliminarily.
The second set of recommendations then identify widespread and seemingly
fundamental features of nearly all agency regulatory decisions that seem to rise to the
equivalent of best practices for science-intensive regulation. Ideally, these practices
operate almost as presumptive prerequisite for science regulation that agencies should be
expected to explain when they are not followed.
The third and final set of recommendations identifies innovative approaches
found within the agencies that seem to significantly strengthen agency regulatory
programs involving scientific or technical information. While each innovation offers an
approach to regulatory science that generally advances both the transparent and robust
incorporation of technical information into regulation, this final section refrains from
hailing these innovations as clear “best practices” since the contexts and constraints on
agencies are so variable that there will be situations in which one or more of the
innovations is not practical or even advisable. Further experimentation and study of these
practices will help fine tune and further develop these approaches for their more
extensive use in the future. For the time being, however, these approaches are offered
simply as innovations worthy of agency consideration.
1. Sharing Information on Agency Approaches
a. Providing a clear explanation of agency decision processes
In all programs studied, the flow charts and processes uses by agencies to
incorporate science into their decisions had to be recreated from scattered documents and
interviews. The processes remain particularly obscure for the FWS’s listing and critical
habitat designation decisions and to some extent for the NRC’s regulatory process. Even
for heavily revised programs like the IRIS process, which have been diagramed multiple
times in the course of congressional investigations, GAO reviews, and critical

108

commentary,448 the agency’s incorporation of science into the assessment process is
generally not explicated at the level of detail that is useful for understanding whether all
of the best practices and innovations detailed in this study have been followed.
When practicable, agencies should provide an accessible summary of their
processes for incorporating science into specific regulatory programs. Such an
articulation of these basic processes is consistent with the President’s directive that “[t]he
public must be able to trust the science and scientific process informing public policy
decisions.”449 An explication of agency processes should also guide staff and provide
those outside the agency with an understanding of how science is being integrated into
regulatory decisions.
An agency’s explanations should include a description of how the agency
conducts its literature search and the availability of those results, the analytical process
the agency used to integrate scientific information into its decision-making, the respective
role of scientific staff and political management in producing technical documents, the
opportunity for robust internal discussions that encourage diverse viewpoints and dissent,
the assurance of adequate peer review of agency technical documents, and the
articulation of principles at the beginning of the analysis that guide the agency’s
decisions about how to incorporate the appropriate scientific literature at key points in the
process, etc. In situations where the processes are particularly divergent, 450 the agency
should explain why the existing process is on net beneficial to advance the agency’s
goals.
Agencies should provide the public with an accessible description of
the process that they utilize for integrating science into their decisions for
each of their science-intensive programs. This includes a statement of how
an agency evaluates the scientific information used in its analysis; how the
agency makes that information available to reviewers and the public; how
the analysis is reviewed by experts and interested parties; and how the
agency ensures that the final decision can be compared against the scientific
record.
b. Highlighting Agency Innovations

448

See, e.g., Center for Progressive Reform, Corrective Lenses for Iris, Sept.2010, available at
http://www.progressivereform.org/articles/IRIS_1009.pdf.
449
Obama Memorandum, supra note 6, at 1.
450
Particularly in the case of IRIS, EPA should be required to explain why interagency review occurs twice
during the preparation of a risk assessment, for example, while public and expert review occurs only once.
Indeed, given the fact that most of the participating agencies are stakeholders and engage out of a concern
about compliance costs or potential liabilities, it seems fully appropriate for their engagement to be
reserved to the single public comment period as is the case for interagency review in the other agency
processes studied here. See data summary in Section III.C., supra.

109

In this study, the agencies did not publicize their innovations or successes. Yet as
this study has revealed, many agencies have developed innovative mechanisms for
ensuring transparency in their use of science, and they should advertise and disseminate
these best practices for consideration by other agencies.
As part of its scientific integrity initiative, some centralized agency, like OSTP or
the standing committee of the NAS Science, Technology, and Law should play a
leadership role in publicizing what the agencies are doing especially well with respect to
the integration of science in regulation, particularly when their approaches might be
useful to other agencies. In conducting this work, this central analyzing body could
borrow from ACUS’s other best practices project, which spotlights agency innovations,
provides short summaries of these innovations, and offers a forum wherein agencies can
describe their innovations. 451 Since these agency successes in the use of science are
ground-tested and have been improved over time, they should be particularly instructive
to other agencies.
This central analytical body or even the EPA Air Office should also produce a
short, lessons-learned report based on the development of the NAAQS process. Since
Section 109 of the Clean Air Act provides little room for discretion and policy judgment,
the EPA’s effort to identify uncertainties, assumptions, and choices in these scientific
analyses is particularly instructive for other agency programs. It is also worth noting that
every agency employee interviewed in the study who was not affiliated with the NAAQS
process, with one exception, was unaware of the details of the NAAQS process, much
less the content of these reports. Stakeholders who were familiar with the strong
reputation of the NAAQS program were similarly unfamiliar with its details. The
obscurity of the NAAQS process to those working on science-intensive regulations
outside of the air pollution control arena is unfortunate and can be easily remedied with
this type of publication.
Without an effort to learn from the success stories, other agencies may needlessly
develop regulatory processes from scratch when there is a ready-made, time-worn
approach in use that provides a ready template for their efforts. Focusing on agency
innovations also publicizes the agencies’ successes and helps to counterbalance OIG,
GAO reports, and congressional hearings that, by design, tend to focus almost
exclusively on problems.
OSTP, a standing NAS Committee, or some other body should take
responsibility for identifying and publicizing the innovations developed by
agencies for transparently incorporating science into their regulatory
decisions.

451

See ACUS best practices at http://www.acus.gov/best-practices/

110

2. Developing a Set of Best Practices for Science-Intensive Regulation
Through an examination of agency processes, several features continually stand
out as a basic staple in the agency’s use of science for regulation; these process features
should be considered presumptively required unless there is a reason against their use in a
particular regulatory setting. In this subsection, a few process features are identified as
“best practices” for science-intensive regulatory decisions. Since most agencies currently
lack clear statements of their existing processes for incorporating science into regulation
(see infra Section IV.B.8 for a recommendation addressing this omission), identifying
several of these best practices helps draw out and might even fine tune some of these
unrecorded procedural details, while attempting to preserve flexibility in how they are
followed. Since these best practice principles impose few, if any new requirements,
moreover, they should involve very little added time, resources, or paperwork for the
agencies. Indeed, following the best practices might ultimately reduce the time and
resources expended on science-intensive regulations since these steps are intended to
make the agency’s decision processes more streamlined and transparent.
Each of these best practices draws its authority from three important, overlapping
sources. First, each best practice is linked to the directives in President Obama’s and/or
the OSTP’s memoranda on scientific integrity. These White House goals are considered
fundamental, at least in abstract terms, to good agency practice with respect to the
incorporation of science into policy. Moreover, each of these best practices is being
implemented in all of the agencies under study; thus they are capable of being put into
practice and deemed important enough to be utilized in at least one science-intensive
regulatory program.
a. Availability of a references list and the underlying references
One of the basic expectations for regulatory science is that the agency should
identify all of the literature it consulted (including, when scientifically appropriate, the
relevant literature it ultimately rejected) in its scientific analysis. The agency should also
ensure that this literature is, if at all possible, available to the public and peer
reviewers. 452 Agencies cannot always satisfy this expectation; but because it is a

452

See, e.g., id. at 1 (“Except for information that is properly restricted from disclosure . . . each agency
should make available to the public the scientific or technological findings or conclusions considered or
relied on it policy decisions”); Holdren Memorandum, supra note 7, at 2 (principle I.3. stressing “[o]pen
communication among scientists”, “the free flow of scientific and technological information” and directing
agencies to “expand and promote access to scientific and technological information by making it available
online in open formats.”); see also Bipartisan Center report, supra note 2, at 41 (recommending, among
other things, that “once an agency has opened a docket on a rule or guidance that will draw on scientific
studies, it should make available on the Web a list of the studies it is reviewing and should regularly update
the list”).

111

fundamental obligation, agencies should explain when they are unable to provide a list of
references used in the analysis or make unpublished studies publicly available.
Accordingly, where agency time and resources permit, agencies should attempt to
post not only the bibliography of the literature it utilized in conducting its analysis, but
also the articles themselves, particularly the unpublished studies. To the extent that the
agency is not able to post unpublished studies – due to statutory prohibitions like the one
in FIFRA mentioned above – they should articulate these limitations. Moreover, to the
extent that copyright protections prevent this posting, then agencies should seek
permission to disseminate the work, and if this fails then the agency should where
possible provide the public with information on how to request the information directly
from the author. Ensuring that the underlying research is available is a basic component
of scientific transparency and this applies with equal force to regulatory science. 453
For the agency programs examined in this study, it appears that the agencies
generally provide bibliographies and reference lists that support their regulatory projects.
EPA’s NAAQS program again provides the high water mark, reinforced through its
extraordinary HERO literature base which provides public access to the underlying
studies, or at least the abstracts as permitted by copyright law. 454 EPA, DOI, and the
NRC generally provide bibliographies for their rulemakings (this feature was not
examined in detail at the NRC), although in some cases the public or reviewers will have
to go through extra steps to gain access. 455 Like EPA, the NRC has produced a publicly
accessible online library that provides the public with instant access to many of the NRC
documents that support its science-intensive rules, including many unpublished studies
and staff reports cited in its decision documents. 456
Despite these positive developments, there do not appear to be formal policies or
a universal commitment, even in the programs studied, to ensure that the supporting
literature is identified and reasonably accessible to the public in all cases. A review of
several proposed rules at the FWS, for example, revealed that not only the references, but
the reference list itself had to be requested from (or perhaps viewed at) the field office.
Since these studies are apparently already compiled for purposes of internal review, 457 it
would seem relatively easy to make them publicly available through the internet. Indeed,
one staff indicated that FWS is currently attempting to provide this kind of access. 458

453

See supra notes 29-30 and accompanying text (discussing these values in basic science).
See Section III.A.1.a.
455
FWS does not include the bibliography in its regulatory analysis published in the Federal Register and it
is only sometimes posted online in the regulatory docket. As a result, the public must request the reference
list from the field office. See Section III.A.2, supra.
456
See Section III.A.3.
457
See Hall email, supra note 220, at 1 (describing this arrangement).
458
Interview with FWS Staff Member, Field Office, Feb. 15, 2012.
454

112

It could well be that all of the agencies will soon be developing extensive
databases of the studies they use for regulation, such as posting bibliographies online
with hyperlinks. Indeed, some of this is already being done.459 Yet given the explicit
prominence that the availability of underlying research plays both in basic science and in
the President’s directive, this type of literature accessibility should be considered an
exected feature of any rigorous regulatory program. The absence of a reference list or the
inaccessibility of references generally will deserve explanation.
In supporting its science-based regulatory decision, an agency should
identify and make publicly available a list of the scientific literature it
consulted, including even the literature it rejected when it is material to the
scientific analysis, as well as the literature it relied upon. This reference list
should be posted online whenever possible.
When an agency relies on studies that are not published, it should post
the studies on its website as soon as is practicable, subject to copyright and
other legal restrictions. When this public transparency is not possible, these
restrictions should be explained in the agency’s individual analyses and
possibly more generally in describing its regulatory program for the public.
b. Ensuring Some form of Expert Peer Review of Science-Intensive Regulatory
Products
Peer review is a cornerstone of the scientific process, and some level of expert
review is similarly integral to ensure the vigorous scrutiny of agency work. 460 Both the
President’s directive and the Holdren memorandum on scientific integrity explicitly
reference this critical role of expert peer review. 461 OMB’s Peer Review Bulletin also
underscores the value of expert review to the agency’s use of science. 462
In the federal government, agencies have at their disposal multiple tools for
engaging this expert peer review, both within the agency and using external reviewers.
These tools include: a) developing quality control processes, such as technical audits, for
459

See, e.g., EPA, IRIS Toxicological Review for Hexachloroethane (Sept. 2011), available at
http://www.epa.gov/iris/toxreviews/0167tr.pdf (providing a hyperlink in the reference list that allows the
reader to click their way to each of the references cited).
460
See supra notes 29and 43 accompanying text.
461
See Obama Memorandum, supra note 6, at 1 (directing that “[w]hen scientific or technological
information is considered in policy decisions, the information should be subject to well-established
scientific processes, including peer review where appropriate, and each agency should appropriately and
accurately reflect that information in complying with and applying relevant statutory standards”); Holdren
Memorandum, supra note 7, at 1-2 (agencies should develop policies that ensure “that data and research
used to support policy decisions undergo independent peer review by qualified experts, where feasible and
appropriate, and consistent with law”); see also BiPartisan Policy Center, supra note 2, at 17
(recommending that “[f]ederal agencies should use scientific advisory committees to the maximum extent
possible to review the science behind regulatory policies”); National Performance Review, Sept. 1993,
Improve Regulatory Science, at 61 (recommending the use of external advisory boards).
462
See OMB Final Information quality Peer Review Bulletin at 2, 3 (Dec. 2004),

113

more routine agency decisions; b) developing formal or informal processes for intraagency review; and c) utilizing external peer review by soliciting individualized reviews
or assembling expert review panels. In some settings, peer review panels are even
required by statute.463
This study reveals that some agencies utilize these peer review tools extensively
and in ways that go well beyond the bare minimum requirements set forth by statute and
OMB’s guidelines. For example, while EPA is required to engage CASAC in the review
of its proposed revision to an air quality standard, EPA is not required to have each of its
staff reports reviewed multiple times by this science advisory body. 464 Similarly, EPA
relies on external peer review as a critical step in its IRIS assessments, but there is no
legal requirement that it solicit this external review for most of its assessments. 465 Even
when pressed by very short statutory deadlines for decisions, the FWS incorporates
external peer review into its listing and habitat designation decisions. 466 And it has
persisted with this commitment to external peer review in the face of both congressional
and stakeholder opposition.467 Interviews with agency officials in all three agencies
reinforce the high value they place on expert peer review for their regulatory work,
particularly with respect to the value of external peer review. 468
The agencies studied here fared very well in their commitment to expert peer
review, but their records were not perfect. At least the NRC was criticized by its
Inspector General for failing to develop validation and quality control processes for the
information collected by agency staff in their review of permit renewals and
amendments.469 Indeed, it seems likely that if there are lapses with regard to peer
review, they may occur in programs that involve more routinized licensing decisions.
EPA’s pesticide program, which does engage internal expert review of the agency’s
scientific projects, was also criticized by one anonymous scientist for not engaging
external expert peer reviewers in the review of its risk assessments. 470 The merits of the
agencies’ choices will only become clearer once they are identified and explained.

463

See Section III.A.1.a. and A.3, supra.
See 42 U.S.C. § 7409(d)(2).
465
While some IRIS risk assessments are likely “influential” under OMB’s peer review bulletin, it seems
unlikely that all of them are. See OMB Final Information quality Peer Review Bulletin at 2, 3 (Dec. 2004),
available at http://www.whitehouse.gov/sites/default/files/omb/memoranda/fy2005/m05-03.pdf. Yet EPA
engages external peer review in all the IRIS assessments. See Section III.A.1.b., supra.
466
See Section III.A.2., supra.
467
Id.
468
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012; Interview with
EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012; Interview with EPA Staff,
National Center for Environmental Assessment, Feb. 3, 2012; Interview with FWS Staff, Endangered
Species Program, Jan. 9, 2012; Interview with NRC Staff, Oct. 26, 2011.
469
See Section III.A.3., supra.
470
See Section III.A.1.b., supra.
464

114

Although some form of expert peer review, whether conducted wholly internally
or externally, is considered fundamental in the agencies’ use of science for important
regulatory projects, the actual tools that the agencies utilize for any given science-based
project defy generalization. With the single exception of OMB-prescribed expert peer
review for “influential” scientific assessments471 and scattered congressional mandates
that require science advisory board review in narrow regulatory settings,472 there also are
few minimum requirements placed on agencies that limit or frame their ultimate choices
in engaging this expert review. This flexibility seems wholly warranted given the many
factors at play. 473
Given the diverse regulatory settings that benefit from peer review, prescriptive
requirements for expert peer review should be avoided, at least based on the current
knowledge of these processes. Agencies should explain why they select a particular
approach to expert review in their decision-making process. But beyond this explanation
requirement, at this point the range of practices, uses, and needs for peer review simply
defy generalization and hence more specific requirements.
Conversely, however, restrictions that limit the agencies’ use of external peer
review should be eliminated. A prior recommendation discusses how caps on FACA
committees and perhaps other features of the implementation of FACA may impede the
ability of agencies to enlist external peer review. 474 These and any other barriers
identified by the agencies and OSTP with respect to limiting agency peer review options
should be removed or at least reduced to the extent possible.
Beyond removing barriers, a second recommendation is to establish as a best
practice a simple expectation that agencies should, whenever practicable, explain their
peer review choices in individual programs when doing so will not impose a significant
burden on the agency This expert peer review need not involve external review, but at a
minimum should include some quality control by agency technical experts.
Consistent with President Obama’s directive, agencies should be
encouraged, not impeded, from having their scientific analyses reviewed by
other experts, even if this oversight occurs wholly inside the agency. Any
limitations on an agency’s ability to have scientific work reviewed by
scientific experts should be actively discouraged. Additionally and when
possible, agencies should endeavor to explain how they ensured the rigorous
review of their scientific products for each regulatory project.

471

See OMB, Peer Review Bulletin, supra note 465, at 37-40.
See supra note 463.
473
See National Performance Review, supra note 461, at 59 (recommending flexibility in agencies ability
to use a variety of different peer review methods for regulatory science).
474
See Section IV.A.3., supra.
472

115

c. Transparent records that apply Deliberative Process protections sparingly
Both the Obama and Holdren memoranda direct agencies to make their
underlying analyses and reasoning as transparent as possible, and this presumably
includes an expectation that the agencies will share all stages of their work through
comprehensive administrative records. As the President states: “If scientific and
technological information is developed and used by the Federal Government, it should
ordinarily be made available to the public.” 475 As just discussed, because there is a range
of interpretations of the scientific literature, alternative models, and often dozens of
important judgments with regard to how to address remaining uncertainties, it is
important for agencies to show their work to the greatest extent possible. Rulemaking
records that are incomplete and neglect to show the agency’s analytical process can
obscure these important decisions. Indeed, if the decision-making processes studied here
reveal anything – and this is exemplified most dramatically by the NAAQS process – it is
that the series of documents and records that lead up to a final decision tell the story of
the analysis. Recall that in both NAAQS and pesticide reviews, the pre-NPRM period
involves multiple different analytical documents, notice and comment processes, and
approximately four years of scientific discourse over the scientific information. 476 In
such a decision process, the final decision document is simply the conclusion or ending.
Without these earlier chapters, one is unlikely to be able to reconstruct the analysis that
preceded it.
In meeting this goal of comprehensive administrative records, the agencies’
practices were mixed. EPA’s NAAQS process and NRC’s informal rulemakings appear
to involve the creation of relatively complete and open records, and at least the NRC
indicated that it utilizes deliberative process privileges only in exceptional cases
involving privacy or security.477 The administrative records in the other programs
appeared to involve a less consistently open approach towards deliberative process and
record creation. Despite its command that all interagency written comments be included
in the record,478 for example, EPA’s IRIS risk assessments still involve lengthy phone
calls and meetings with other affected agencies that do not appear to be recorded in the
public record.479 Particularly given the early point in the process when these
communications occur, this deliberative input may lead to changes in the framing of the
basic scientific assessment that are never explicated and remain invisible. EPA’s
475

Obama Memorandum, supra note 6, at 1; see also Holdren Memorandum, supra note 7, at 2 (directing
agencies to ensure the “free flow of scientific and technological information” which includes “[o]pen
communication among scientists . . . and the public”).
476
See Sections III.A.1.a. and b., supra.
477
Statement of NRC Staff, Feb. 1, 2012.
478
See Section III.A.1.c., supra.
479
Interview with EPA Staff, National Center for Environmental Assessment, Feb. 3, 2012. It is not clear
whether this decision to rely extensively on these deliberative discussions is a choice of OMB or EPA,
although given the OMB’s more vigorous application of the privilege to its communications, it seems more
likely to be the former.

116

pesticide staff similarly concedes that phone calls originating from USDA to EPA are not
included as part of the record.480 The FWS also uses the deliberative process privilege to
protect its record until the proposed rule is published, but even after that point one FWS
staff member suggested that the FWS might still use the privilege in some cases to
withhold some documents.481
Additionally, in the programs studied, the agencies’ administrative records were
not always compiled in ways that ensured that each important analytical step in the
agency’s decisionmaking was documented. FWS staff, for example, concedes that the
record-compiling policies and practices at the Service are both dated and have become
somewhat ad hoc.482 As a result, each field office may create the record supporting a
listing differently. One office might prepare the record diligently by compiling all drafts,
carefully documenting the changes made at each step of the process and the reason for
the change, and identifying substantive input between staff and management. In other
offices the documents may be gathered late in the process, with the possibility that in this
more haphazard creation of the administrative record, key documents will be missed,
overlooked, or lost.483
An additional best practice principle for the agencies’ integration of science into
policy is thus to expect them to produce a comprehensive record that tracks their sciencebased decision-making for a given regulatory decision. This involves assembling draft
reports, memoranda, and comments that document each significant step in the agency’s
analytical process. Providing this kind of open accounting of the agency’s analytical
process parallels a similar commitment to openness in science (i.e., that the underlying
data should be made available) and indeed is even more critical in science-policy where
there is more opportunity for hidden policy choices to influence an agency’s analysis.
Any gaps in this public documentation constitute weak links in the analytic chain and
present the possibility that key decisions will be obscured from public view.
In the preparation of this comprehensive administrative record for scienceintensive regulations, there should also be a strong presumption against deliberative
process claims. To the extent that claiming deliberative process for the substance of
interagency meetings is routine, for example, this tradition should be reversed and the
privilege should be justified for each document before removing it from the public
record. As Judge Patel recently reminded OMB in a case in which OMB asserted a
similarly expansive deliberative process privilege claim: “It is the government’s burden
to set forth the exemption and justify withholding a document pursuant to that particular
exemption. This is accomplished on a document by document basis by use of a ‘Vaughn
480

Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011.
Interview with FWS Staff Member, Field Office, Feb. 15, 2012.
482
Interview with FWS Staff Member, Field Office, Feb. 15, 2012.
483
Id.
481

117

index.’”484 If instead OMB is allowed to claim deliberative process for all if its
Presidential review duties, it “would effectively remove the OMB from the FOIA’s reach.
Without express statutory authority in this respect, this court is unwilling to provide the
OMB with such a blanket waiver.”485
To effectuate this best practice, agencies should develop written policies and
regularized guidelines that instruct agency staff to place all important scientific and
technical discussions on the record unless a very narrow set of circumstances is present
that justifies an Executive Privilege.486 Each significant analytical step, draft, and change
in scientific interpretation and analysis should be included in this record. Every outside
and interagency communication should be documented and described. Internal review
comments should be made available, where possible. Again, while these complete
record-keeping practices may not be necessary for all informal rules, this documentation
is particularly important for science-intensive regulations.
Finally and ideally agencies should place their administrative records on the
internet to permit easy public access. The FWS, as noted, does make its supporting
record available at the time the proposed rule is published, but the record is apparently
available only at one field office and is not available online. 487 In practice, then, public
access to the FWS’s administrative records supporting its listing and habitat designation
decisions is limited. The other agencies studied generally posted their entire records and
supporting documents on the web, either through regulations.gov or their own program’s
electronic library.
Agencies should resist applying deliberative process protections to
documents and communications that influenced the development of sciencebased regulatory projects. To the extent agencies do invoke the deliberative
process privilege, they should justify so doing with respect to each document
that is withheld from the public. Draft science-policy analyses, such as draft
papers, can be made public with the disclaimer that they do not necessarily
484

Center for Biological Diversity v. Office of Management & Budget, 546 F. Supp.2d 722, 729 (N.D. Cal.
2008) (emphasis added).
485
Id.; see also NRDC v. U.S. DOD, 442 F.Supp.2 857, 861 (C.D. Cal. 2006) (requiring OMB and other
agencies to provide “original justifications” for withholding documents under FOIA on a document-bydocument basis).
486
In 1980, the Administrative Conference recommended that advice an agency receives from other
agencies or from the President and his/her advisers that pertains to matters of policy need not be placed in
the public record, whereas advice related to factual matters should be memorialized therein.
Administrative Conference Recommendation 80-6, Intragovernmental Communications in Informal
Rulemaking Proceedings, 45 Fed. Reg. 86,407 (1980). The current report concludes that the distinction
between fact-based and policy-based decisions is rarely clear in science-intensive rulemakings, see Section
I, supra, and it therefore declines to use that distinction to determine whether agencies are justified in
invoking the deliberative process privilege. Rather, it proposes that agencies assess each claim of
deliberative process individually and invoke that privilege only in exceptional cases, such as those
involving privacy or security.
487
See Section III.A.2., supra.

118

represent the policy or scientific position of the agency. Agencies should
prepare an administrative record that advances this transparency goal by
ensuring that the documents, meetings, and other deliberations that resulted
in potentially significant changes to scientific assumptions or interpretations
are made part of the administrative record. These administrative records
should be posted on the internet when possible.
3. Agency Innovations that may develop into Best Practices with Experience
A number of other process innovations were also discovered in the course of the
study for how agencies might use science in their regulatory decisions. These
innovations are not presented as “best practices” in the report since there is far too much
variation among agencies and it is not clear whether these innovations are feasible or
even advisable beyond the particular regulatory programs in which they were first
discovered. Yet the innovations do provide ideas that agencies may wish to consider in
evaluating and reforming their own programs. The innovations may also seed other ideas
and approaches that prove even more effective and resilient in the integration of science
for policy. Thus the innovations should be considered carefully by agencies in the
development of their own programs, but they are not established enough to constitute best
practices.
a. Four Analytical Steps that Enhance the Transparency of the Agency’s Policy
Choices
One of the core problems in science-policy is the difficulty of identifying how the
agency’s policy decision has been informed by a robust assessment of the best available
scientific evidence. 488 Administrators can claim that “the science made me do it” when
the opposite is in fact the case. Moreover, in some cases agencies have failed to develop
robust scientific records to support decisions, and in rarer cases those records have been
skillfully manipulated in ways that are invisible to non-experts and in some cases even to
expert reviewers. The literature is filled with examples of this lack of transparency of the
respective roles of science and policy in the agency’s regulatory work.489
The antidote to these problems is for the agency to identify and explain its various
assumptions, judgments, and other choices clearly and accessibly. Even though science
and policy are often imperceptibly mixed in how the agency develops a science-based
rule, an elaborated analytical process can expose these often important and/or contestable
488

See Section I, supra (discussing this problem).
See, e.g., Doremus, supra note 5; Holly Doremus, Scientific and Political Integrity in Environmental
Policy, 86 TEXAS LAW REVIEW 1601 (2008); Peter L. Strauss, Possible Controls over the Bending of
Regulatory Science, in Anthony, et. al eds., VALUES IN GLOBAL ADMINISTRATIVE LAW 126-27 (2011);
Wendy Wagner, The Science Charade in Toxic Risk Regulation, 95 COLUMBIA L. REV. 1613 (1995); see
also OIG MacDonald Report, supra note 11.
489

119

decisions at each step in the analysis for more vigorous internal and external scrutiny.
While none of the analytical steps involve just “science” or just “policy”, by explaining
the choices at each step of its process – how the agency articulated the problem;
evaluated the literature; applied the literature to the problem at hand with models, etc;
and identified the range of options – the messy merging of science and policy will be
considerably more transparent and accessible. Indeed, this is precisely what is required
by the President’s scientific integrity memorandum. 490
While the NAAQS process may not have the ultimate solution to this
implementation challenge of unpacking the inextricable science-policy weaving through
an agency’s decision, the NAAQS process appears to have the best answer presently
available within agency decision-making. 491 Key features of the NAAQS process that
help draw out the relationship between policy and the underlying scientific information
are: 1) the initial articulation of specific policy questions arising in a regulatory project
that might be informed by science; 2) an assessment of the available evidence bearing on
these questions; 3) application of the evidence to the policy questions at issue, with
robust statements of all material uncertainties and assumptions; and 4) a report that
identifies the various plausible policy alternatives based on the scientific record that is
accessible to policymakers.
These sequential analysis steps are repeated in other areas of health policy. The
strongest parallel occurs in clinical medicine, in which health care providers synthesize
limited health research, some of which may be affected by significant conflicts of
interest.492 Like the NAAQS process, clinical researchers first identify specific policy
questions to be resolved by the evidence.493 Clinical researchers then assemble the full
research bearing on the question and weight it according to basic principles established in
advance based on various quality and relevance related factors.494 For example, one a
priori principle common in literature syntheses in clinical medicine downgrades or even
490

The Holdren Memorandum requires agencies to “communicate scientific and technological findings by
including a clear explication of underlying [significant] assumptions; accurate contextualization of
[significant] uncertainties; and a description of the probabilities associated with both optimistic and
pessimistic projections, including best-case and worst-case scenarios where appropriate.” Holdren
Memorandum, supra note 7, at 2; see also Obama Memorandum, supra note 6, at 1 (directing agencies to
“make available to the public the scientific or technological findings or conclusions considered or relied on
in policy decisions.”).
491
See Section III.A.1.a, supra.
492
See, e.g., DANIEL M. FOX, THE CONVERGENCE OF SCIENCE AND GOVERNANCE 84 (2010) (noting
conflicts of interest problems in clinical medicine); Tracey J. Woodruff & Patrice Sutton, Pulling Back the
Curtain: Improving Reviews in Environmental Health, 118 ENVTL. HEALTH PERSPECTIVES 326 (2010)
(discussing the pervasive conflicts of financial interests in clinical medicine and the need for principles for
rating that evidence as part of the analysis in advance).
493
See Tracey J. Woodruff, Patrice Sutton & The Navigation Guide Work Group, An Evidence-Based
Medicine Methodology To Bridge the Gap Between Clinical and Environmental Health Sciences, 30
HEALTH AFFAIRS 931, 934-35 (2011) (setting out almost identical analytical steps for clinical medicine).
494
For a discussion of these advanced principles, see FOX, supra note 492, at 88-89; Woodruff, et al., supra
note 493, Appendix at 5-8.

120

excludes research produced with a financial conflict of interest. Conversely, another
common a priori principle upgrades research when the magnitude of the effect on the test
subjects is large or serious.495
National Academies of Science reports, as well as a recent consensus report from
the Keystone Center, further reinforce the wisdom of this sequential, methods-based
approach to synthesizing diverse evidence in a systematic way. In Science and
Judgment, for example, the Committee emphasizes the importance of these same steps,
beginning with a clear statement of the problem and data needs, 496 establishing a clear
statement of default rules and other a priori principles in advance, 497 and providing a
rigorous comparison of alternative models. 498 The Keystone Center report similarly
underscores – even more explicitly -- the importance of conducting a systematic review
of the literature in framing the science-policy decision-making effort. The Keystone
Center report also catalogues the steps entailed in this systematic review that parallel the
first three steps of the NAAQS process. 499
Clearly each of these steps cannot be done in an elaborate way for every scienceintensive regulatory project, but most of these steps are already implicit in agency
analyses. Thus, even if these discrete steps are collapsed into a single document, they
will nevertheless offer an analytically useful way to begin to identify overarching policy
questions and the important role that science plays in resolving them.
1. Policy Questions. Agencies should first identify the policy questions that can
be informed by the scientific evidence. The NAAQS process dedicates an
entire report to this stage of the process,500 yet for most rules that must be
promulgated on a tight timeframe and small budget, articulating the sciencepolicy questions could be done with relatively little effort in an initial
paragraph of an agency’s report.501 Both the BiPartisan Policy Center and
Keystone Center reports also underscore the importance of identifying the
basic questions that will be addressed with scientific research as a first step in
the integration of science.502 Ideally, agencies will also include these key
495

Id., Appendix at 6-7.
NRC, SCIENCE AND JUDGMENT, supra note 1, at 144.
497
Id. at 254-55.
498
Id. at 186-87. See also id. at 257-58 (emphasizing the need for structured steps in an assessment and
providing a series of steps that roughly parallel the NAAQS steps).
499
See Keystone Center, “Improving the Use of Science in Regulatory Decision-Making:
Dealing with Conflict of Interest and Bias in Scientific Advisory Panels, and Improving Systematic
Scientific Reviews” 18-25 (September 18, 2012), http://www.keystone.org/images/keystone-center/sppdocuments/Health/Research%20Integrity%20Rountable%20Report.pdf.
500
Id.
501
The FIFRA process, albeit inconsistently, also involves a planning stage that identifies the policyrelevant questions. Although they are presented in only a few paragraphs, these more focused questions
nevertheless go a long way to help frame the resulting scientific analysis. See Section III.A.1.b., supra.
502
See Keystone Center, supra note 499, at 20.
496

121

questions in their requests soliciting relevant information from the general
public, a request that was made in every regulatory program under study.
2. Assessment of the Evidence. Agencies should then identify and evaluate the
existing scientific evidence – mostly the literature – that bears on these policy
questions.
a) The first sub-step involves establishing the equivalent of a study design
for the literature review, which includes identifying principles in advance
for how different types of studies will be weighted.503 The Keystone
Center report recommends, for example, that “[c]riteria established for
relevance and credibility [and perhaps also for other features] should be
developed and made public before the selection of studies to be
included.”504 All studies should not be weighted equally but assessed
against these ex ante criteria. 505 Research produced under conflicts, for
example, may deserve less weight than research produced independent of
the sponsor’s influence.506 Beyond conflicts, however, a number of ex
ante principles could be established, such as how to address gaps in
research, whether to upgrade studies if they reveal significant risks, the
development of stopping rules, etc.507
b) The second substep then involves summarizing and interpreting the
literature within this study design. This literature assessment lays the
groundwork for the agency’s next step – its choice and application of
models and other assessment tools to inform the policy questions, often in
a more focused and quantitative way. Since an evaluation of the literature
is implicit in science-intensive regulatory work, separating out this phase
will not only be a natural step in most cases, but will make the resulting
analysis more transparent.508
3. Application of the Evidence. Agencies should then identify and justify their
choice of models and other analytical tools (in some programs this is done
generically at a program-wide level) 509 and then apply these application tools
to the scientific evidence described in step 2. An application of the model(s)
to the evidence should highlight the policy-relevant choices involved in
503

Lynn Goldman statement in NAS, Improving the Use of Science in the Administrative Process
Transcript, Sept. 10, 2012, at 44-46; Tracey Woodruff statement in id. at 141-42.
504
Keystone Center, supra note 499, at 20; Bipartisan Policy Center, supra note 2, at 15-16.
505
Id. at 20 and 22.
506
See supra note 492 and infra Section IV.C.
507
See, e.g., Keystone Center, supra note 499, at 22-24 (providing examples and elaboration on the criteria
to guide the relevance and the credibility assessments. See also Comments of Dr. Lynn Goldman, NAS
Meeting, Transcript, supra note 503, at 44-45; Woodruff, NAS Meeting, Transcript, id., at 141-42; see also
NRC, SCIENCE AND JUDGMENT, supra note 1, at 156 (providing examples of how these a priori principles
might work in data prioritization and synthesis).
508
Interview with EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012; Interview with
FWS Staff Member, Field Office, Feb. 15, 2012.
509
See Section III.A.1.b., supra (describing this feature of FIFRA risk assessment models, which are
reviewed by EPA’s SAP).

122

selecting among plausible models or using multiple models, many of which
will be the same as the a priori principles established in the design for the
literature synthesis. 510 This application of the evidence to the policy question
should also identify significant uncertainties that result from using these
models, assessments, etc. EPA’s NAAQS report again is considered to be
particularly good at providing this kind of explication.511
4. Bridge the Evidence to the Policy Questions. The agency should then explain
in ways that nonscientists can understand how this scientific analysis informs
the core policy questions. This discussion should identify the significant
choices that need to be resolved by policy considerations, as well as the range
of plausible choices, before recommending a preferred course.512
In most cases, the agencies under study generally did not disaggregate their
analyses into these separate steps. For example, only in the case of NAAQS and to a
lesser extent in FIFRA is an evaluation of the evidence and an assessment of the evidence
separated in the agency’s decision-making.513 The FIFRA and NAAQS programs are
also the only programs that consistently articulate specific policy questions that serve to
focus the resulting scientific analysis. 514 Finally, only the NAAQS process provided a
frank discussion of how the scientific information informs the pressing policy questions,
at least in a way that is accessible to nonscientists.
Since each of these analytical steps is necessarily implicit in much of the work the
agencies do, the recommendation below encourages the agencies to make these analytical
steps explicit, particularly the initial steps since they should be easy to separate out for
most agency analyses. The fourth step is arguably the most important, but it will likely
require more agency resources since it involves an additional discussion that attempts to
bridge policy and science. It also requires the agency to communicate this analysis to
nonscientists, which is not a simple matter. In the meantime, the agencies are encouraged
to study the NAAQS policy assessments, models from clinical medicine, and the more
elaborate suggestions in the Keystone Center report for ideas on how this
recommendation for a more regularized analytical process might be accomplished within
their unique programs. 515

510

Goldman, NAS Meeting, Transcript, supra note 503, at 45.
See NAS FORMALDEHYDE REPORT, supra note 1, at chapter 7.
512
In EPA’s policy assessment, the staff explains the different scientifically plausible options available to
the decision-maker and outlines the details of the scientific support (pro and con) for each option. CASAC
is intimately involved in reviewing this science-policy synthesis to ensure that it is accurate in its
representations of the underlying scientific assessments. See Section III.A.1.a., supra.
513
See Sections III.A.1.a. and b., supra.
514
Id.
515
See Section III.A.1.a., supra (discussing these policy assessments and giving links to some recent policy
assessments).
511

123

All significant science-policy choices made by an agency in reaching a
decision should be identified and explained in clear and understandable
terms. In order to provide this heightened level of transparency, agencies
should consider following an analytical process that: a) identifies the policyrelevant questions that can be informed by science; b) identifies in advance a
study design, such as criteria for weighting individual studies, as well as
identifying other a priori analytical choices, like stopping rules; c) provides a
synthesis of the available evidence and relevant literature guided by this
study design; and d) identify other significant assumptions, choices of
analytical techniques, and remaining uncertainties and how different
plausible choices might change the resulting policy decision. If possible, the
agency should also follow the model of the NAAQS policy assessment in
bridging science and policy in a final report, although this final step will
likely involve more effort and experimentation.
Making these analytical steps explicit may not be practicable in some
science-policy decisions and may not be practicable in other regulatory
settings. This recommendation simply encourages agencies to consider this
staged approach in their processes. Ultimately, with experience, this
analytical approach may develop into a best practice. Until then, agencies
are strongly encouraged to consider this analytical approach in conducting
their work.
b. Stopping Rules are Useful to Clarify the Point at which a Scientific Record and
Debate are Closed
The notion that a scientific record is closed on new science or debate is
antithetical to the scientific process, yet in policy settings this closure is essential if
decisions are to be reached. In this study, these points of closure are loosely referred to
as “stopping rules.”516 In the case of regulatory science, stopping rules are not based on
science but instead are based on the agency’s available resources, time and other policy
considerations. A “stopping rule” does not put an end to research or to the accumulation
of new information by regulators; it merely sets the point at which a policy decision will
be reached in the interim. The resulting regulatory decision will be revisited with new
evidence at some future date, often established in the stopping rule itself.
The need for “stopping rules” in regulatory policy arises in at least two different
contexts. First, the agency must develop an explicit point at which it closes its
consideration of new evidence. Without this type of stopping rule, each new study or
discovery that arises in the course of the regulatory process could arguably throw the
516

The concept of stopping rules were introduced into the social studies of science literature by Sheila
Jasanoff. See supra notes 45-46 and accompanying text for a discussion of stopping rules.

124

project back to the starting point, requiring the agency to conduct a re-analysis, reopen
the comment period, etc. Second, the agency may occasionally find it necessary to close
debate, even when important scientific issues have not been resolved. In this second
setting, it is unresolvable disagreements between experts rather than the emergence of
new evidence that prevents closure on an issue. Rather than suspending the regulatory
project in the hope that the experts will reach some agreement, a decision that the agency
must proceed despite these disagreements – the equivalent of a stopping rule – may be
necessary to move the project forward.
Stopping rules are not only important to ensure that the agency stays on track to
produce final decisions for science-intensive regulatory programs, but these rules are
important to enhance the transparency of the decision-making process. Explicit stopping
rules provide all participants, as well as agency staff, with clear policy direction on when
scientific disagreements have come to a close. Without clear stopping rules, points of
debate and new studies can even be used manipulatively, as well as in good faith, to
throw the science-policy decision-making off course or run it through repeat circles of
analysis and peer review, without any credible way of cutting off these iterative analyses
and discussions.
With respect to emerging science, several agency programs – particularly the
NAAQS and FWS’s listing programs – have built-in stopping rules; both agencies face
strict statutory deadlines that can and have been enforced by stakeholders in court. 517
Even in programs that lack these statutorily set stopping rules, some agencies have
developed accommodations to meet the challenges posed by emerging science. In the
NAAQS program, EPA has reinforced its statutory deadline with an agency policy that
once the integrated science assessment (EPA’s analysis of the existing literature) has
been peer reviewed, new science will not be considered until the next five year revision
of a standard.518 In IRIS assessments, EPA also attempts to limit itself to the scientific
record created at the time a risk assessment is peer reviewed, although this appears to be
a more informal convention.519 In pesticide registration reviews, by contrast, new science
is considered up to the point of the proposed decision.520 The pesticide registration
review process is new enough that there is not a lot of experience to determine whether
517

See Sections III.A.1.a. and 2., supra. For example, the one year deadline for a decision under the ESA is
so abbreviated that little new science or methodological breakthroughs are likely to emerge in the interim.
518
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012. In some cases
the agency may conduct a provisional scientific assessment near the end of the NAAQS review process that
considers whether the new science would have materially changed its decision. EPA staff reports that these
provisional science reports have never led to a finding that the new science would cause a material change
in the standard. They credit this fact largely to the enormous scientific record that exists at the time of
standard-setting and the resultant unlikelihood (but not impossibility) that a new study in fact will support a
material change in the standard. Interview with EPA Staff, Office of Air Quality Planning and Standards,
Jan. 17, 2012.
519
Interview with EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012.
520
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012;

125

earlier stopping rules are needed and whether this more open-ended approach could
create the possibility of extended delays. 521 A best practice would suggest that agencies
establish their stopping rules for emerging science ex ante – for example determining that
evidence arising after a critical analysis point will not be considered.
Setting stopping rules for when debate is closed is a much more difficult task for
programs that do not involve rigid statutory deadlines. Critical comments by peer
reviewers and the public can identify multiple differences with the agencies’ technical
analysis. Each revision of the draft in response to comments may shrink the contested
issues in need of elaboration or development, but the revisions can also open up new
issues for yet another round of critical scrutiny and debate.522 As a result, there is no
clear point for when the rule is effectively good enough, and there is also no clear point
for when an agency’s response to a set of criticisms can be considered complete,
particularly for heated scientific disagreements. 523
Not surprisingly, most agency programs lack clear stopping rules that allow them
to put an end to ongoing scientific debate, and this resulting ambiguity creates problems
for the agencies. Without clear stopping rules, it is difficult for stakeholders to
distinguish those situations in which the agency’s decision to close debate is based on
science or instead on practical realities. The lack of transparency in an agency’s stopping
rules may also explain some of the most vigorous criticisms of the quality and
transparency of the agencies’ science. For example, stakeholders from both the industry
and public interest community interviewed for the study considered the most significant
transparency problem in EPA’s pesticide program to be the agency’s insufficient and
unjustified dismissal of some of their comments. 524 Yet this could easily be the result of
confusion over the basis (stopping rules vs. science) for EPA’s decision to close debate as
it moves towards a decision. In IRIS, voluminous peer review comments from external
reviewers, combined with a high level of intense stakeholder criticism, including from
sister agencies, appear to contribute substantially both to the agency’s excessive delays in
preparing IRIS assessments and to the unwieldy nature of the resultant assessments that
endeavor to respond to these numerous comments. 525 Again, part of the problem may be
the lack of explicit stopping rules to end debate. Indeed, in IRIS it is possible that the
EPA may not have the final say on setting stopping rules; instead closure is determined
by OMB through intergovernmental arrangements that are not in the public domain. 526
521

Id.
See also Sheila Jasanoff, Research Subpoenas and the Sociology of Knowledge, 59 Law & Contemp.
Probs. 95, 99-100 (Summer 1996) (discussing the “infinite regress” problem; basic methodological features
of research are established on consensus but are susceptible to unraveling when under fierce attack).
523
Id.
524
See Section III.A.1.b, supra.
525
Interview with EPA Staff, National Center for Environmental Assessment, Feb. 3, 2012.
526
One of the major sources of frustration by EPA during OMB review is that OMB’s “stopping rule” is an
effective requirement that the agency make all of the requested corrections; until that point, the rule is not
cleared for publication. One EPA staff relayed that “[b]oth OMB’s minor and major comments are viewed
522

126

In an effort to significantly increase the transparency and productivity of their
regulatory work, agencies should endeavor to establish clear stopping rules for
technically complicated regulatory decisions at the beginning of the regulatory project
that guides it. This exercise is particularly when the agency operates without judicially
enforceable statutory deadlines.527 While agencies may deviate from these ex ante goals
as the regulatory process unfolds, they will be expected to explain their deviations as
against this initial goal.
With some explication of stopping rules, agencies should thus be clear in their
decision-making when they are closing debate simply because there is no clear resolution
(e.g., setting a stopping rule on ongoing debate) versus when the agency instead believes
there is a best scientific answer to a particular contested issue. This distinction will
provide enhanced transparency on whether the agency is deciding based on policy or
instead believes that as a scientific matter extended debate is not warranted.
In the NAAQS reviews, EPA has again been a pioneer in developing a different
and potentially easier way to develop credible stopping rules for closing debate over
unresolved scientific issues. Specifically, EPA relies on its science advisory board,
CASAC, to effectively declare “closure” when it considers the EPA’s responses to
criticisms adequate.528 This respected opinion of CASAC effectively puts a stop to
disagreements and overrides criticisms and comments filed by stakeholders and the
general public. Interestingly, moreover, while there were discussions of limiting
CASAC’s power to declare “closure” during the reform of the NAAQS process, 529 it
appears that the CASAC’s closure role continues today. 530 EPA may have realized that it
is in the agency’s best interest to have an independent and trusted scientific arbitrator
determine when the agency’s response to criticisms and comments is sufficient.
In particularly controversial science-based projects, agencies should thus consider
convening expert panels to assist them in setting stopping rules. Indeed, in weighing the
advantages of external peer review, this attribute should be factored in as a benefit.
External peer review likely works to facilitate closure only when the reviewers are
as equal; all the changes need to be made.” Interview with EPA Staff, National Center for Environmental
Assessment, Jan. 18, 2012. See also GAO, Chemical Assessments, supra note 152, at Appendix III
(identifying the important role that OMB has played in some IRIS assessments).
527
Agencies are likely not to voluntarily promulgate binding rules that set reasonable deadlines for their
projects and thus simulate statutory deadlines, but if they did set judicially enforceable deadlines on their
regulatory projects through regulation this would also address the stopping rule challenge for agencies that
lack statutorily required deadlines.
528
See Section III.A.1.a., supra.
529
This was based on a concern that this type of authority provided CASAC too much power and further
delayed the proceedings, over time. Interview with EPA Staff, Office of Air Quality Planning and
Standards, Jan. 17, 2012.
530
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012. See Section
III.A.1.a., supra.

127

convened as a panel, rather than individually, however. In the IRIS process, for example,
the use of individual external peer review provides valuable scientific feedback, but
because the reviewers don’t confer, these individual peer review comments are as likely
to expand the issues at play as narrow them. It is not unusual for individual reviewers to
place emphasis on different issues or even conflict in their comments, for example. 531 In
cases when advisory bodies are used, moreover, this external scientific review role should
extend until the public comment period ends to ensure that the panel’s deliberations
consider the issues raised by stakeholders and facilitate closure. The importance and
value of external peer review advisory boards with respect to stopping rules again
reinforces one of the previous recommendations regarding eliminating unnecessary
barriers to the agency’s use of FACA for purposes of soliciting external peer review.
Like the right to dissent and the importance of authorship, the concept of stopping
rules is new enough that agencies should be urged to set these rules, but they do not yet
rise to the level of a best practice. Over time, the use of stopping rules may be capable of
being routinized across agencies and regulatory missions, but we are not at that point yet.
In regulatory settings, particularly in cases when agencies are not
bound by judicially enforceable deadlines, the agencies should be encouraged
to establish explicit stopping rules on regulatory projects, both with regard to
when they will close their consideration of emerging research and when they
will close otherwise unresolvable scientific debate in order to reach a
decision. External peer review bodies may be particularly useful to agencies
in establishing scientifically credible points at which debate should cease.
c. Checkpoints that Identify Future Research Needs
Working in tandem with “stopping rules” are research “checkpoints,” which consist
of a list of specific research projects that should be conducted in the near term to advance
regulatory understanding on a particular issue.532 The NAAQS Policy Assessment, for
example, provides this type of explication of future research questions in need of
study. 533 These checkpoints would ideally be identified by an agency when the record
closes (as set by the stopping rules). Without adding this explicit “research checkpoint,”
there is a risk that the most important gaps in regulatory understanding will be lost or
forgotten after the rule is promulgated and the agency turns to other issues and problems.

531

Interview with EPA Staff, National Center for Environmental Assessment, Feb. 3, 2012. See also
Section III.A.1.b., supra.
532
Dr. Louis coined this term, See NAS Meeting Transcript, supra note 503, at 148-49, but as it developed
through discussions, the idea of checkpoints became primarily focused on future research needs.
533
See supra note 71and accompanying text.

128

For science-intensive rules, an agency should identify specific types of
future research projects that will advance understanding on the regulatory issue.
This identification of research questions and priorities should influence the
agency’s research agenda as well as provide a benchmark for measuring
regulatory progress. While agencies may need to experiment with when and
how they incorporate “research checkpoints” into their regulatory analysis, the
most fruitful time may occur at the point the record is closed, as determined by
stopping rules.
d. Staff Authorship or at least Attribution is important for Agency Analyses
Agency managers should consider and, where possible, capitalize on the scientific
and organizational benefits of affording authorship credits or at least attribution to staff
scientists, particularly for assessments of the relevant scientific literature that precede rule
proposals. Providing authorship or at least attribution to the agency staff who prepare a
technical analysis provides a means of accountability and well-deserved credit to
government scientists.534 Identification of the authors of an analysis also helps provide
information regarding potential conflicts or biases of the scientists; identifies their
disciplinary affiliations; and permits others to assess the extent to which the authors are
both expert in the subject matter and retain some distance from having a personal or
professional interest in the outcome of the analysis.535 In some regulatory settings, these
authorship credits are not beneficial to the agency’s work or to the staff careers. For this
reason, managers should seriously consider providing some form of authorship credit, but
are not expected to justify when authorship proves undesirable or impracticable.
Authorship provides important benefits to regulatory science. In interviews,
several management-level scientists at EPA confirmed the important role of authorship in
their programs, particularly with respect to providing credit to talented agency staff. 536
Authorship also serves to afford agency staff a stake in the final product, thus sharpening

534

Within science, authorship is fundamental to ensuring transparency, integrity, and accountability.
Journals place significant weight on this authorship. See supra note 32 and accompanying text (discussing
the critical role of authorship in science). While the development of scientific analyses used for regulation
is different and may even include nonscientists on the analytical team, the underlying value of some basic
authorship and attribution is essentially the same. Moreover, as discussed in the text, this basic credit is not
only owed to the staff preparing the analysis but helps retain talented scientists in government. Both of
these values are critically important in the President’s scientific integrity initiative. See Holdren
Memorandum, supra note 7, at 3 (principle IV noting that “[a]gencies should establish policies that
promote and facilitate, as permitted by law, the professional development of Government scientists and
engineers”); see also Obama Memorandum, supra note 6, at 1 (directing agencies to develop retention
policies that attract talented scientists).
535
Id.
536
Interview with EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012; Interview with
EPA Staff, National Center for Environmental Assessment, Feb. 3, 2012; Interview with EPA Staff, Office
of Pesticide Programs, August 5, 2011

129

debate, internal scrutiny, and the quality of the final product within the agency. 537 At the
same time, these agency managers cautioned that since the reports were co-authored by
interdisciplinary teams and subject to multiple levels of peer review and intra-agency
review, the staff did not have the power to unilaterally veto revisions (much as an
individual NAS or SAB panel member cannot control the content of each feature of the
report).538 Nor were individual authors within EPA actively encouraged to dissent or
withdraw as authors when they did not agree with every feature of the resulting
analysis. 539 Instead, authorship in these teams is consensual and parallels the approach to
authorship taken by science advisory boards or NAS panels where authors are
encouraged to agree, but remain free to dissent. Like a consensus report, the agency
authors are intimately involved in producing the product; they agree in general terms with
the substance of the report; and by contributing their name to the report they signal that
they are comfortable with the process for producing the report and its general content. 540
In the unusual case where this consensus cannot be reached, however, agency staff can be
removed from the acknowledgements or accreditations or can be changed from authors to
contributors.541 They can also, at least informally (see below), prepare a dissent for the
report and the public record.542
To the extent that management – either political, career, or both – are involved in
authorship or in the review of the report and afforded an opportunity to make changes,
this fact should be noted in the document as well. Under this approach, technical reports
would contain indicia of authorship, a list of reviewers and their roles, and would thus
provide full disclosure of the role of all contributors, much as a disclosure of conflicts
and sponsorship provides in published articles in scientific journals. This disclosure of
authorship is a standard prerequisite for publication in most scientific journals;543 in the
abstract, then, it seems like an appropriate expectation for scientific analyses conducted
by agencies.
In contrast to the EPA, at the FWS and NRC, there is no similar form of
authorship or even meaningful attribution for scientific staff, and there is no reason to
believe that these agencies are especially unique with regard to this issue. In the case of
the FWS, one agency official suggested that there is generally insufficient time or
537

Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012; Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011;
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Jan. 18, 2012; Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan.
17, 2012.
538
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012
539
Id.
540
Id.
541
Id.
542
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011.
543
See, e.g., Authorship Standards for ICJME (biomedical journals), at
http://www.icmje.org/ethical_1author.html

130

resources to ensure that the staff reports are adequately reviewed, and thus the reports
remain “works in progress” as they move through collaborative intra-agency review
processes where they are constantly revised. 544 Making these reports public, agency
officials suggest, could increase the resource and time drain on the agency and also place
staff biologists in the bulls-eye for political attack and even interference. 545 At NRC, the
justifications are less clear but could arise from a concern that authorship and attribution
may actually increase incidents of retaliation and chill rigorous staff analysis; a
management authored analysis protects staff in this way. More research is warranted on
the agencies’ reasons for not crediting staff with the scientific analyses, at least in an
acknowledgements section. For that reason, authorship and attribution is presented in
this report as an innovation that deserves consideration by agencies but does not rise to
the level of a standard, best practice at this point in its development.
As an important and related point, the programs under study provide at least
preliminary support for the observation that staff-authored analyses tend to be more
evidence-focused, nuanced, and likely to concede limitations, gaps, and assumptions in
the available evidence as compared to analyses that are heavily influenced, if not written
by managers. 546 In at least one very well-documented occasion in EPA’s NAAQS
program, the staff-authored policy assessment provided a much more complete and
sophisticated discussion of the relevant scientific literature as compared with the more
superficial, incomplete, and arguably ends-oriented, parallel analysis prepared by
political management.547 Both EPA’s Office of Research and Development and the
CASAC noted the significant differences between the staff versus the management
authored reports with regard to the robustness, quality, and transparency of the sciencebased analysis. 548 Though it is merely one example, this experience suggests that science544

Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
Id.; Interview with FWS Staff Member, Field Office, Feb. 15, 2012. In the abstract, concerns about
resources could be addressed simply by ensuring that draft reports, when made public, are accompanied by
internal review comments and marked as a draft. Alternatively, staff-authored drafts could be accompanied
by a disclaimer – as they are for draft NAAQS reports – that the staff’s analysis is a draft and does not
necessarily represent the views of the agency, even after it has been internally peer reviewed by other
agency staff. See Section III.A.1.a., supra (discussing how the NAAQS analyses contain this disclaimer).
546
At NRC and the FWS, the agencies produce more collaborative science-policy analyses that may
originate with staff but are heavily layered with supervisors and manager comments such that the final draft
scientific assessment effectively has no clear author but is a product of the agency program working
together. There is apparently ongoing discussion in at least the FWS that a listing analysis might also
benefit from greater development at the staff level, before extensive review occurs from management.
Interview with FWS Staff Member, Field Office, Feb. 15, 2012. In this case the concerns are less about
conflicts of interest emerging from the reviewing managers and arise more with respect to the fact that the
time allotted for conducting the actual analysis is cut short by the great amount time allotted for multiple
reviews by supervisors, all of which must be completed within a year. Yet placing more responsibility on
staff might have additional benefits in this situation – such as accountability and providing scientific credit
-- that make it even more worthwhile for the FWS to consider processes that place greater emphasis on
staff-authored analyses, even if they consist of only the initial literature review.
547
See Section III.A.1.a. (discussing this example in more detail).
548
Id.
545

131

integration may benefit from a two-step process that begins with a staff-authored
scientific assessment followed by a management-drafted decision based on the
evidence.549
To the extent possible, agency analyses should track the principles and norms of
science and scientific and technical assessments conducted by the agency should confer
consensus-type authorship rights on the staff who prepares them. At the same time, the
agency should identify as authors all of those who contributed in a significant way to a
scientific report, including staff attorneys and economists if they in fact contributed to the
final product. Although the concept of explicit authorship and acknowledgement is still
too preliminary to rise to the level of a best practice, it is an innovation that is grounded
in the norms of science and should be carefully considered by all of the agencies engaged
in science-intensive regulation.
Agency staff plays an important role in producing the agency’s
analyses. When practicable and appropriate, agency managers should
consider providing staff with some form of consensual authorship right or
attribution for reports or analyses to which they contribute in a significant
way. All staff authors who contributed in a significant way to a technical or
scientific report should be acknowledged, including economists, lawyers, and
other nonscientists. In a similar vein, reviewers and other contributors
should also be identified by name and general contribution.
e. The Right to Dissent
Agencies should also develop policies to ensure that they encourage vigorous and
diverse debate among agency scientists. 550 This goal of promoting both vigorous and
diverse debate is made difficult by the realities of government service (e.g., shared
mission and long tenure). Open debate may also be discouraged to the extent there are
perceptions by staff that management wants a particular finding and will continue to
request information until they “get the answer they are looking for.”551 Yet scientific
integrity requires not only protecting skeptics within the agency’s scientific staff, but
nurturing an environment of robust and open debate and vigorous scrutiny of agency
work products.552
549

Id.; Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
See, e.g., Obama Memorandum, supra note 6, at 1 (directing that the agencies ensure “the highest level
of integrity in all aspects of the executive branch’s involvement with scientific and technological
processes”); Holdren Memorandum, supra note 7, at 1 (principle I.1. directing agencies to develop policies
that “[e]nsure a culture of scientific integrity. Scientific progress depends upon honest investigation, open
discussion, refined understanding, and a firm commitment to evidence.”).
551
Cf. Patricia M. Wald, Analysts and Policymakers: A Confusion of Roles?, 17 STANFORD LAW & POLICY
REVIEW 241 (2006) (describing this problem in the security arena).
552
See supra notes 29 and accompanying text (underscoring the central role that vigorous scrutiny by
diverse peers plays in science).
550

132

The NRC’s open and collaborative workplace policy offers the most
comprehensive approach to address this challenge. 553 This program includes the open
door policy, non-concurrence process, and differing professionals program. While these
programs have not been given the resources that some might wish or that they necessarily
deserve, they are now well-established within the agency. Through these programs,
employees are encouraged to question and voice objections and are, as mentioned, given
formal avenues to pursue their disagreements and are sometimes even awarded for doing
so.554 Whether these written, formal policies are internalized within the NRC, however,
is difficult to determine. Some staff still report they fear retaliation for disagreeing with
superiors.555 On the other hand, surveys of job satisfaction at the NRC are generally
quite high. 556
Both FWS and EPA interviewees stated that in practice they utilize an approach
that parallels these formal processes in the NRC for permitting and even encouraging
dissent and internal debate. Specifically, both agencies report that it is traditionally
accepted that staff can file public memoranda dissenting on scientific findings. 557 There
was at least one example of this practice in the FWS and in the EPA pesticide program. 558
The EPA and FWS also purport to nurture diverse, open, robust debate among
scientists.559 The FWS in particular suggests that this open and vigorous give-and-take
between management and staff defines the scientific culture of the Service. 560 Moreover,
the geographic dispersion of FWS’s scientists across more than 50 staff and regional
offices creates a spatial distance among the employees, particularly in relation to
headquarters, which in the case of the FWS only seems to further ensure the diversity of
views and the perception of a level playing field with headquarters in terms of scientific
expertise and knowledge.561
The informal policies in the FWS and EPA do not appear to be documented,
however, and may not be well known by employees. Given the informality of these
dissent policies, moreover, it seems doubtful that employees will feel they can be assured
553

See Section III.B.2., supra.
See id.
555
See id.
556
See NRC News, “NRC Receives Top Rankings from Annual Survey,” Sept. 22, 2011, available at
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CDUQFjAB&url=http%3
A%2F%2Fpbadupws.nrc.gov%2Fdocs%2FML1126%2FML11265A309.pdf&ei=8lInTuJG8PO2wXZzoXmAg&usg=AFQjCNFvL50u5xPjLXdopYqMhYwMc16WEw&sig2=JGIpo7HZhdEmEJ
uEdH3BEA (reporting on this high job satisfaction report).
557
Interview with FWS Staff, Endangered Species Program, Jan. 26, 2012; Interview with EPA Staff,
Office of Pesticide Programs, August 5, 2011.
558
Id.
559
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012; Interview with FWS Staff,
Endangered Species Program, Jan. 26, 2012; Interview with EPA Staff, Office of Pesticide Programs,
August 5, 2011.
560
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012; see also Hall memo, supra note
204.
561
Interview with FWS Staff, Endangered Species Program, Jan. 26, 2012.
554

133

that supervisors will not retaliate against them in some settings when they chose to
dissent. It is also not clear whether an employee’s dissent will always be placed in the
public record, since supervisors could determine that these statements should be withheld
as deliberative process. Additionally, incomplete guidelines and oversight with respect to
how to create an administrative record supporting a decision, at least at the FWS, raise
the possibility that some important documents, like staff dissents, could be dropped out of
the record due to sheer inadvertence. 562
However it is accomplished, agency managers should ensure that agency staff can
engage in vigorous debates that include diverse points of view, and where possible
provide for the opportunity for dissent from a technical analysis or finding. While formal
dissent policies, like those developed by NRC, may prove too cumbersome for all
agencies, a simple written policy that explains the freedom of staff to publicly dissent and
that prohibits retaliation against dissenters would seem to be helpful to encourage
vigorous debate and provide enhanced scientific autonomy for agency staff. 563 Managers
should defer to dissenters’ requests to place dissents in the public record whenever
possible. To ensure some peer review of these dissenting positions, dissenting employees
should also be allowed and encouraged to publish their views in the peer reviewed
literature. This right to dissent is particularly important when agencies do not provide
authorship or contribution rights to scientists, and thus scientists cannot even withdraw
their name as a contributor in order to express dissent. A formal dissent policy also
ensures that expressions of disagreement cannot be hidden under deliberative protections
claimed by a supervisor or other agency official. 564
Much like attribution and authorship, the role of dissent and the approaches to
encouraging diverse, rigorous debate among agency scientists are only beginning to be
explored. It is thus premature to suggest that the right to dissent from a technical analysis
should be regarded as a best practice that deserves some explanation when it is not
followed. On the other hand, the importance of scientific autonomy and inviting diverse
scientific views is vital to rigorous scientific work;565 thus agency mangers should feel
obliged to ensure that their processes provide for this type of debate and free exchange.
Agencies should encourage vigorous debate among agency scientists
that may include developing written policies that allow agency staff to dissent
562

See Section IV.B.5., infra (discussing best practices for the development of administrative records).
Agencies’ scientific misconduct policies focus on intentional and reckless fabrication and thus do not
even reach these good faith disagreements. See Section III.B.1., supra, and accompanying text.
564
Protecting employees from retaliation is an extremely important feature of such a policy. Given the size
and scope of these various whistleblower issues, however, the details are left for a separate ACUS report or
similar investigation. NRC’s more formal program, by contrast, may offer a particularly helpful model for
agencies that utilize a hierarchical management structure; in these management-heavy structures, scientific
information may be at even greater risk of being dropped out or adjusted as it moves up the chain of
command. In other settings, however, the formality of the program may be unnecessarily costly and could
even undermine a culture of collegiality for agencies that purport to be more collaborative in their
preparation of analyses, like FWS.
565
See supra Section I.B.4.
563

134

or express their non-concurrence on a technical analysis to which they
contributed. In cases where written dissents are permitted, agency managers
should take seriously a staff member’s request to place a dissent or nonconcurrence into the public record. Dissenting employees should also be
allowed and encouraged to publish these dissenting positions in the peer
reviewed literature, provided that confidential governmental deliberations
are not compromised. In all cases and regardless of the public availability of
these discussions, dissenting staff members should be protected from
reprisals.
B. Interagency and External Constraints on Agencies that might impair the
Rigorous and Transparent Incorporation of Science into Regulation
Various external constraints impede the ability of the agencies to “be transparen[t] in
the preparation, identification, and use of scientific and technological information in
policymaking.”566 Although this study focused only on five programs in three agencies,
there are reasons to believe that the impediments identified here are not the only ones that
limit agencies in their effort to develop rigorous decision-making processes for the
integration of science into regulation.
The first set of constraints arises from a variety of congressional and executive branch
limits to agency decision-making processes themselves, such as precluding disclosures or
capping the number of science advisory boards an agency can empanel. The second set
of constraints involves White House oversight of science-intensive rules that pose
challenges for ensuring both the transparency and integrity of the use of science. This
section discusses these challenges and suggests potential reforms.
1. Identifying and Redressing External Constraints that Impede an Agency’s
Efforts to Improve the Rigor and Transparency of its Use of Science
Within the five programs studied, significant sources of slippage in the agencies’
development of robust decision-making processes occurred through external impediments
that arise from within the Executive Branch or Congress. It would be bad form indeed to
suggest that the agencies’ use of science should be improved when the agencies have
been placed in decision-making structures that do not allow these improvements to be
made. Yet without some formal, government-wide procedures for identifying and
addressing these external constraints, agencies may find themselves blocked from making
useful and reasonable improvements to their decision-making processes.
Several hard constraints on the scientific transparency and rigor of agency
decision-making processes arise from laws themselves. In the FWS, unreasonably short
566

Obama Memorandum, supra note 6.

135

congressional deadlines for a decision appear to be the primary cause of the FWS’s
truncated analysis process for species listing and critical habitat designations. 567 Recall
that the FWS is required by statute to make a listing decision within a little over a year
after receiving a petition.568 The FWS staff reiterated that these timelines are so tight that
they do not provide the FWS with sufficient time to utilize external peer review panels or
engage experts more actively throughout their analytical process. 569 This short timeframe
has also led the FWS to resort to collaborative staff-management authored proposed rules
(without an initial scientific analysis) that are less than ideal for important science-policy
decisions, as detailed infra. 570 Even the development of complete administrative records
supporting listing decisions may be compromised as a result of the whirlwind timeframe
that governs these decisions.571
Statutory constraints also undermine the ability of the agencies to share with the
public the underlying research the agencies use to make a decision. In EPA’s pesticide
program, for example, Section 10(g) of FIFRA prevents public access to studies
conducted by most pesticide manufacturers unless a person is granted clearance through a
certification process.572 This section was apparently passed by Congress to prevent
competitors in other countries from benefiting from safety data produced by U.S.
manufacturers as required by FIFRA. 573 Yet as detailed earlier, Section 10(g) impedes
public access to the bulk of the scientific research the agency considers in its registration
decisions. Specifically, section 10(g) requires that a person certify that he/she will not
share the information with manufacturers in other countries in order to gain access to
manufacturer-provided data. But even then, access to this critical information is limited
insofar as: it is allowed only after a pesticide registration decision is made; pesticide
manufacturers are notified of each person who views their information; and the
information must be viewed at the agency’s office.574 EPA summarizes each of these
studies in data tables, but there is effectively no external oversight over this internal
review of manufacturer data.575
In the programs examined in this study, the Paperwork Reduction Act (PRA) and
Data Quality Act (DQA) were also implicated in imposing additional statutory
567

See Section III.A.2.
See id.
569
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012. The FWS engages external peer
review on the proposed rule simultaneously with public comment. This has caused it to occasionally revise
its proposal and reopen comment. See Section III.A.2. A much more streamlined and efficient process
would involve expert review followed by public comment, or even more ideally two stages of peer review
that occur before and after public comment.
570
See id.
571
See id.
572
7 U.S.C. § 136h(g)(1). See Section III.A.1.b.
573
See, e.g., 95 vol3 1978 U.S.S.C.A.N. 1966 (discussing US manufacturers concerns that data would be
used by foreign competitors who would be afforded a competitive advantage as a result).
574
7 U.S.C. § 136a(c)(2)(A). There are some exceptions, id. at 136h(d), but they appear to be unusual.
575
See Section III.A.1.b.
568

136

impediments on the agencies’ rigorous and transparent use of science for regulation. In
the case of the PRA, EPA is currently required by OMB to clear each of its requests for
data from manufacturers in its pesticide registration program through OMB. 576 This
process can involve an additional six months of delay on each pesticide registration
decision. The PRA clearance also raises a risk (as yet unrealized) that some of EPA’s
requests for data might ultimately be blocked by OMB. 577 If this occurs, EPA may lack
data it believes is essential to conduct its pesticide assessment, and yet again the reasons
for this data shortfall would be protected by deliberative process. Somewhat similarly,
OMB enjoys general oversight of the agencies’ compliance with the Data Quality Act, 578
and there was some indication that at least in the IRIS program, OMB may be using DQA
complaints as a way to regain control as the primary clearance authority over select IRIS
assessments.579
Finally, although information was difficult to locate for purposes of comparison,
insufficient legislative appropriations to support an agency’s science-based regulatory
program can also impact the agency’s ability to do its analyses rigorously and in a timely
way. Resource limitations appear to be a significant external constraint on the FWS’s
listing process for example. 580 The FWS’s small budget allows it to allocate roughly
$250,000 to its analysis and deliberations dedicated to listing a single species and a total
of $300,000 to $500,000 for a complete habitat designation and listing rule. 581 For
purposes of comparison, the NRC allocates on average $4 million to its analysis of the
renewal of a license for one nuclear power plant and this licensing process occurs over
the course of nearly two years, in comparison to the one year afforded the FWS. 582
Although estimates of the costs of EPA’s programs were not available, 583 informal
speculation by staff in the NAAQS program placed the cost of conducting one revision of
an air quality standard closer to the NRC’s budget than the FWS’s allowance. 584
Whatever the case, it would be disingenuous to demand that an agency improve the
integrity and transparency of its use of science if it is not provided funding adequate to do
the work, at least in comparison to other agencies. Thus, the under-funding of agency
scientific work is also an important external constraint that deserves exploration and
potentially supportive advocacy by the OSTP.

576

See id.
Interview with EPA Staff, Office of Pesticide Programs, August 5, 2011.
578
See Curtis Copeland and Michael Simpson, The Information Quality Act: OMB’s Guidance and Initial
Implementation (Aug. 19, 2004), available at http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA445489
(describing OMB’s role in the DQA).
579
See, e.g., GAO, CHEMICAL ASSESSMENTS, supra note 152, at 24-26.
580
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012.
581
Id.
582
See OIG, License Renewal Report, supra note 273, at 4.
583
Estimates of the costs for the review of pesticide registrations may be available and the author is still
tracking that figure down.
584
Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012.
577

137

Constraints arising purely within the Executive Branch further obstruct the
agencies’ effort to develop robust processes for their use of science. For example, to the
extent that important non-written conversations between OMB and the agency lead to
changes in science-based rules during OMB review under Executive Order 12866, these
communications are non-divulged or explained as they are at other stages in the process;
at best one must speculate at why various changes were made to an agency rule before
publication. Caps on the number of discretionary FACA committees within an agency,
established by President Clinton under Executive Order 12838, also serve as a barrier to
the agencies’ effort to enlist external peer review of their regulatory work. 585 Both the
EPA’s IRIS and the FWS’s programs generally use contractors to solicit external peer
review in order to avoid these and other restrictions imposed on the agencies’ use of
FACA. 586 EPA officials in particular lament that the FACA limitations, such as the
arbitrary cap on the total number of discretionary advisory committees an agency may
host, render it difficult to use advisory committees for IRIS risk assessments. Since a
FACA science advisory panel is more likely to provide the agency with a cohesive and
comprehensive set of unified comments as compared with comment letters from
individual expert reviewers, IRIS officials believe a FACA panel is the preferred
option.587 Caps on discretionary FACA committees, however, cause agencies to abandon
efforts to convene advisory committees in favor of employing independent reviewers.
The use of individual peer reviewers who are not able to confer, in turn, may serve
exacerbate the already unwieldy and even conflicting comments that the agency must
address, yet this time coming from external experts. 588
There are statutory and regulatory constraints that limit the ability of
the agencies to ensure that their decisions are scientifically robust and
transparent in keeping with the President’s Directive. The agencies should
identify these legal barriers that impede public access to the scientific
information underlying agency analyses or otherwise block the agencies’
development of scientifically robust decision-making processes and OSTP or
other centralized agencies can take responsibility for collecting information
and proposing government-wide recommendations.

585

See Executive Order 12838 and OMB Circular A-135 (requiring agencies to reduce the number of
discretionary advisory committees by one-third). ACUS has recommended rescinding this cap on FACA
committees. See ACUS Recommendations 2011-7 at 10, available at http://www.acus.gov/wpcontent/uploads/downloads/2011/12/Recommendation-2011-7-Federal-Advisory-Committee-Act.pdf.
586
Interview with FWS Staff, Endangered Species Program, Jan. 9, 2012; Interview with EPA Staff,
National Center for Environmental Assessment, Feb. 3, 2012. This reluctance by agency to use FACA
committees due to the approval and paperwork requirements is explored in Reeve Bull, The Federal
Advisory Committee Act: Issues and Proposed Reforms, Sept. 12, 2011, available at
http://www.acus.gov/wp-content/uploads/downloads/2011/09/COCG-Reeve-Bull-Draft-FACA-Report-912-11.pdf, at 47-49.
587
Interview with EPA Staff, National Center for Environmental Assessment, Office of Research and
Development, Feb. 3, 2012.
588
Id.

138

2. Review by OIRA and Other White House Offices and Agencies
Evidence reveals that OIRA review can lead to material changes to science-based
rules that are not always disclosed consistent with the directives of Section 6(a)(3)(E) of
Executive Order 12866. A process that allows potentially significant changes to be made
to science-based rules without public or expert review heightens the risk that some of
these changes will be made in ways that conflict with public understanding and the
scientific record.589 At the very least, substantive changes that are made to important
science-based rules at the suggestion of OIRA should be clearly identified in the public
record and some justification should be provided for each change for the public,
consistent with the letter and spirit of Executive Order 12866 and President Obama’s
memorandum on transparency. This section offers recommendations to improve the rigor
and transparency of the decision-making process for science-intensive rules, from the
beginning to the end of the regulatory action.
Before proceeding, it is important to remind the reader that the arguments in this
report are concerned only with changes made by White House offices to technical or
scientific features of a regulation. Deliberations between agencies and White House
offices over the appropriate choice of political values, issues of statutory or other legal
interpretations, and economic and social considerations are beyond the scope of this
analysis except in cases that they impact on the agencies’ use or characterization of
technical or scientific information.
a. Improving the Transparency of OIRA Review
There is a solid body of evidence that OIRA review can lead to material changes
to scientific or highly technical features of some proposed and final rules, but these
changes are neither explained nor even disclosed. 590 Indeed, this lack of transparency of
OMB review seems particularly inappropriate for standards like NAAQS that are
subjected to repeated rigorous peer and public review during their development, but then
are negotiated out of the public eye once the rules undergo review by OMB. The
589

See, e.g., House Report.
See, e.g., Section III.A.1.a., supra; see generally Mendelson, supra note 76, at 1146-59 (2010)
(discussing the lack of transparency of OMB review); Heidi Kitrosser, Scientific Integrity: The Perils and
Promise of White House Administration, 79 FORDHAM L. REV. 101, 117 (2011) (discussing secret role of
OMB in agency oversight); Majority Staff, Nipping IRIS In the Bud, supra note 359, at 1 (detailing the
extensive secrecy that surrounds OIRA review, even with respect to Congress’ effort to understand OIRA’s
role in programs like IRIS); Rena Steinzor, Michael Patoka, and James Goodwin, Behind Closed Doors at
the White House: How Politics Trumps Protection of Public Health, Worker Safety, and the Environment
(Nov. 2011), available at Rena Steinzor, Michael Patoka, and James Goodwin, Behind Closed Doors at the
White House: How Politics Trumps Protection of Public Health, Worker Safety, and the Environment at
34-37 (Nov. 2011); see also GAO 2003, supra note 307, at 100 (“Overall, we often found it difficult to
identify the changes that had been made to agencies’ rules during OIRA’s review and/or at the suggestion
or recommendation of OIRA by reviewing material in the agencies’ rulemaking dockets”).
590

139

NAAQS rules examined here are not the only example of science-intensive rules that
have been subjected to undisclosed changes at the suggestion of OIRA, however. 591 In
the IRIS program, which actually falls outside the reach of Executive Order 12866 review
since it does not produce a final regulation,592 there is still no accounting for all of the
changes that have been made to IRIS profiles as a result of OMB’s and other interagency
suggestions since significant discussions still occur by phone or through meetings which
are withheld as deliberative process.593

591

Two former OMB employees and a number of EPA staff in the NAAQS program, for example,
recounted how OMB can get “deep” into the science (and associated policy) of EPA’s regulatory projects.
Interview with former OMB Staff Member, Jan. 9. 2012; Interview with former OMB Staff Member, Feb.
3. 2012; Interview with EPA Staff, Office of Air Quality Planning and Standards, Jan. 17, 2012; Interview
with EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012. One EPA staff member
suggested that many of OMB’s comments were actually useful in tightening the scientific analysis for IRIS
assessments, but they were often not material enough to justify the extended delay and added transaction
costs. The EPA scientist noted that because OMB formerly had final clearance authority over the draft
assessment, the agency was at its mercy; OMB typically requires the agency to make every change it
suggested, whether major or minor before it will clear the draft or final IRIS assessment for publication.
See Interview with EPA Staff, National Center for Environmental Assessment, Jan. 18, 2012. Moreover, in
some rulemakings OMB solicited and collated comments from other agencies, yet it does not attribute the
comments to the agency making the comments. Id.
A number of published accounts in the literature also reveal evidence of OMB’s willingness to
delve into all features of an agency’s science-based decision in carrying out its review of rules and other
regulatory projects, but in ways that are not subjected to rigorous public or expert oversight. See, e.g.,
GAO, Chemical Assessments, supra note 152, at Appendix III (discussing OMB’s role in several EPA
IRIS assessments); Kitrosser, supra note 590, at 2407 (discussing OMB’s potentially negative role in
compromising scientific integrity of publicly provided information in the BP oil spill); Mendelson, supra
note 76, at 1152-57; Rena Steinzor, The Case for Abolishing Centralized White House Regulatory Review,
1 MICH. J. OF ENVTL. & ADMIN. LAW 247-68 (forthcoming 2012); Rena Steinzor, Michael Patoka, and
James Goodwin, Behind Closed Doors at the White House: How Politics Trumps Protection of Public
Health, Worker Safety, and the Environment (Nov. 2011); Peter L. Strauss, Possible Controls over the
Bending of Regulatory Science, in Anthony, et. al eds., VALUES IN GLOBAL ADMINISTRATIVE LAW 126-27
(2011) (discussing OIRA’s role in the NAAQS ozone standard and referring a similar episode with respect
to a proposed regulation to protect an endangered whale from collisions with larger boats); Tatham, supra
note 76.
Surveys of EPA employees conducted by the Union of Concerned Scientists provide still more
evidence of “OMB’s meddling in EPA decision making [in a way that constitutes] . . . a major hindrance to
the agency’s scientific integrity.” Union of Concerned Scientists, Interference at the EPA at 28 (2008)
(reporting this finding in nearly 100 surveys conducted of employees in 2007, out of a total of 1586
surveys that were returned).
592
The IRIS program does not result in published binding standards and thus is not a “regulatory action” as
defined by Executive Order 12866, yet OMB has appeared to play a significant role in the development of
these standards over the last few decades. See supra Section III.A.2.; Majority Staff, Nipping IRIS In the
Bud, supra note 359, at 1.
593
It is possible to access the interagency discussions for some IRIS chemicals by visiting
http://cfpub.epa.gov/ncea/iris_drafts/archiveDrafts.cfm#C_form and accessing interagency comments (the
last column). For a sample of these OMB comments, see, e.g., OMB Staff Comments on Acrylamide,
available at http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=22440; OMB Comments on
Carbon Tetrachloride, available at http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=56725. See
Appendix F for a sample OMB comment letter. For a few profiles, EPA’s responses to
“selected major interagency” are available for at least the first round of interagency review. See EPA’s
Response to Selected Major Interagency Comments on the Interagency Science Consultation

140

Fortunately, existing requirements already anticipate and seek to prevent at least
some of these lapses in transparency; the recommendations offered here simply call upon
the agencies to come into full compliance with these existing requirements. Recall that
Executive Order 12866 requires agencies to provide a record of the substantive changes
made during OIRA’s review,594 and OMB’s own peer review bulletin requires significant
science-based decisions to undergo peer review.595 Yet currently the agencies and OMB
are not in full compliance with many and in some cases all of these requirements of
Section 6 of the Executive Order. As discussed in the next section, OMB may also not be
in compliance with its own Peer Review Bulletin.
The most important noncompliance with Section 6 occurs within the agencies.
For at least the rules included in this study, there was no meaningful compliance by the
agencies with the requirements of Section 6(a)(3)(E)(iii). 596 Specifically, there was no
documentation or attribution of the “changes made at the suggestion or recommendation
of OMB”, or at least the documentation was not discovered after a relatively timeintensive search of the rulemaking dockets. Even agency compliance with Section
6(a)(3)(E)(ii), which requires “complete, clear, and simple” documentation of “the
substantive changes between the draft submitted to OIRA for review and the action
subsequently announced” was incomplete; red-lined documents that highlight changes
made during the formal review were available only for two-thirds of the rules and a 2009
GAO study reports even lower compliance rates across a larger number of agencies. 597
Moreover, in no case was any of this documentation required by these subsections easy to
locate or provided in “plain understandable language” as required by Section 6(a)(3)(F)
and 6(b)(5).
To ensure agency compliance with Executive Order Section 6(a) in the future, the
agencies should produce a memorandum or similar document that outlines, in plain,
understandable language, all of the major, substantive changes made as a result of
OIRA’s review.598 While agencies tend to produce red-lined documents to satisfy this

Draft IRIS Toxicological Review of 1,4-Dioxane (inhalation exposure route only), September 15, 2011,
available at http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=237251 . A similar document does
not appear to be available for the second round of interagency review. See, e.g., the list of documents
posted for the interagency discussion for the same profile, available at
http://cfpub.epa.gov/ncea/iris_drafts/recordisplay.cfm?deid=122848
594
Section 6(a)(3)(E)(ii).
595
OMB, Peer Review Bulletin, supra note 465.
596
See supra Section III.A.4.a.; see also GAO 2003, supra note 307, at 101 (reaching a similar conclusion).
597
See id. at 13
598
See also GAO 2009, supra note 316, at 35 (recommending that the transparency requirements should be
applied to the “informal review period when OIRA says it can have its most important impact on agencies’
rules”); Mendelson, supra note 76, at 1164 (recommending that OMB review should be made transparent
and that to accomplish this the agency would “need to summarize the critical details of, say, executive
review positions and explain the extent to which those positions are connected to the agency’s ultimate

141

requirement, additional, more accessible identification of all major, substantive changes
is needed as well. Although there were no “best practice” models of this type of
documentation arising in this study, there are possible best practices models for (E)(ii)
disclosure in other agency programs. 599 Samples of more complete documentation are
provided in Appendix I. In these samples, the agencies provide a red-lined version of all
changes made during OIRA review, but supplement it with a cover memorandum that
identifies all of the major changes to the preamble and rule in a clear, understandable
form. While this disclosure appears to provide an identification of only the changes
occurring during OIRA’s formal review, the same memorandum format could be used for
changes made outside of the formal window as well.
Moreover, to make both documents practically accessible to the public (rather
than theoretically available), the agency should post them in a centralized website
dedicated to documenting all major changes made at the suggestion of OIRA throughout
the rulemaking and documenting all substantive changes occurring during formal OIRA
review, regardless of the source of the change. 600 Where possible and in keeping with
some current agency practices, the agency’s docket should also contain all significant,
substantive documents exchanged between the agency and OMB. This documentation
ideally will provide the public with a means to ensure that the agency has in fact
complied with the requirements of Section 6(a)(3)(E) in its identification of changes and
other disclosures. This document repository should also be cataloged as its own, separate
category of documents in the regulations.gov classification of documents. That category
would separate out those documents relevant to OMB review and allow, with one click,
for OMB-agency documents to be searched and extracted from a larger public docket.
The agencies should also extend these transparency requirements to other
significant, science-intensive agency support projects, such as IRIS profiles and scienceintensive guidances and policy documents, as well as the reports and studies that inform
these projects and other rulemakings. The literature suggests that agencies may already
be inclined to avoid informal rulemakings to avoid formal notice and comment
requirements and ultimately the threat of judicial review. 601 To redress the substantially
decision”); id. at 1167 (recommending that “at the time the agency publishes its proposed or final rule, the
agency would have to summarize the final position reached in the executive review process.”).
599
See, e.g., GAO, 2003, supra note 307, at 101 (discussing EPA’s Office of Water documentation of the
changes arising in OIRA review, which appear much more extensive than the documentation encountered
in this study) ; see also GAO 2009, at 36 (providing a similar recommendation – see recommendation #7).
EPA initiated such a site, although it is not clear if it is regularly updated and it does not provide all of the
information suggested in this report, even for the rules that appear relatively up-to-date. See
http://yosemite.epa.gov/opei/RuleGate.nsf/.
600
GAO has made a similar recommendation. See GAO 2009, supra note 316, at 36 (recommendation #6).
601
See, e.g., Jerry L. Mashaw & David L. Harfst, Regulation and Legal Culture: The Case of Motor
Vehicle Safety, 4 YALE J. ON REG. 257, 273-74 (1987) (discussing how judicial review has caused NHTSA
to avoid rulemaking in favor of recalls); Peter H. Schuck, When the Exception Becomes the Rule:
Regulatory Equity and the Formulation of Energy Policy Through an Exceptions Process, 1984 DUKE L.J.
163, 165-68, 194-98 (discussing how the procedural obstacles posed primarily by judicial review and

142

reduced transparency requirements governing agency decision-making in these
alternative forms of regulatory requirements, the default rule should be that these same
transparency requirements of Section 6 – namely at least the identification of all major
changes made at the suggestion of OIRA -- apply to all significant regulatory projects,
regardless of whether the agency product is an informal rule or some other type of
regulatory product, like an IRIS profile, or supporting study or analysis. Agencies should
also comply with the Section 6 requirements for rules that have been withdrawn from
OIRA review or where OIRA encouraged agencies not to submit rules in the first
place. 602
Finally, since there is evidence of noncompliance with the requirements of
Executive Order 6 by the agencies, some centralized mechanism of tracking compliance
and sanctioning noncompliance should be instituted. Ideally, agencies should maintain a
separate webpage dedicated to Section 6(a) compliance for their rules; the webpage
would identify for each proposed and final rule the “substantive changes” made during
OIRA review as well as the major “changes made at the suggestion or recommendation
of OIRA” at any point in the regulatory process in publicly accessible ways.
Alternatively, agencies should at least routinize their documentation so that these
documents are easier to locate and understand within individual rulemaking dockets.
It is important to note that the GAO identified these same lapses in compliance
with the Executive Order in two separate reports, and yet its detailed recommendations –
directed exclusively at OMB – have been largely ignored.603 Although the
recommendations offered for enhancing Section 6(a)(3)(E) compliance in this report
largely parallel these GAO recommendations, they are different in one important respect;
the recommendations here are directed primarily at the agencies, rather than OMB. It is
the agencies that are the responsible parties under Executive Order 12866 for
documenting the changes made at the suggestion of OIRA and occurring during OIRA
review. 604 This structure, by design, places the agencies in the central role of disclosing
the effects of OIRA review rather than placing that responsibility on OIRA, which is
presumably more conflicted in documenting its own influence, although the agencies may
be somewhat conflicted themselves. 605 OMB’s failure to coordinate, oversee, or counsel
executive order led the Department of Energy to abandon policymaking through rulemaking in favor of
policymaking through individual adjudications by an office authorized to make special exceptions to
existing rules).
602
GAO makes a nearly identical suggestion. See GAO 2009, supra note 316, at 36 (recommendation #4).
603
In its first report in 2003, GAO made eight separate recommendations to improve compliance, all of
which were directed at OMB as the coordinator of the Executive Order. As of 2009, only one of the
recommendations had been adopted by OMB and in 2013, it appears that the seven (most of which parallel
the recommendations here) were still unaddressed. Id. at 35 and 39.
604
See Sections 2(a) and 6(a) of Executive Order 12866.
605
OMB not only reviews and coordinates the agencies’ budget requests, but it is a direct arm of the White
House and thus exerts an authoritative institutional presence. OMB is thus likely to command considerable
respect from the agencies.

143

agencies on their compliance obligations under Section 6(a) – particularly when
repeatedly confronted with a series of specific recommendations that it provide this
leadership by GAO – reinforces the import of ensuring that agencies follow through on
their Section 6 responsibilities.
Under Section 2(a) and 6(a) of Executive Order 12866, the agencies are responsible
for interpreting and complying with Section 6(a). The agencies’ compliance under
Section 6(a) should include at the very least:
1) documentation of the major changes made at the suggestion or
recommendation of OIRA at any point in the lifecycle of the regulation as required
by Section 6(a)(3)(E)(iii) and 6(a)(3)(F). If there are no major changes, then the
agency should provide a statement to that effect;
2) an identification of all substantive changes made between the draft
submitted to OIRA for review and the action subsequently announced in
compliance with Section 6(a)(3)(F)(ii). This includes but is not limited to a red-lined
version of the document undergoing OIRA review;
3) for both #1 and 2, the agencies should provide a “complete, clear, and
simple” identification and explanation of each major change in “plain,
understandable language” for the public. Explication of these major changes
should be accessible to the public – through for example a cover memorandum --and not buried in hundreds of pages of red-lined documents. Although the
Executive Order technically requires this accessible explication of all changes (and
not simply the major changes) made at the suggestion of OIRA, a disclosure of the
major changes is considerably less burdensome and appears consistent with the
thrust of the Executive Order;
4) a library of all significant, substantive documents exchanged between
OIRA and the agency throughout the life cycle of the regulatory action to ensure
that the agency is in full compliance with Section 6(a)(3)(E)(ii) and (iii).
5) centralized public access to the information specified above to ensure
practical, as opposed to merely theoretical, compliance with the general
requirements of Section 6(a)(3)(E) and (F). Both reginfo.gov and regulations.gov
should link to or provide the public with document libraries that enable simple
access to and searching of documents required under Section 6.
Agencies should apply these same requirements of Section 6(a)(3)(E), as interpreted
above, to all significant science-intensive regulatory actions, including agency
guidances and other standards and policies, whether or not they are published in
the Federal Register, as well as to all significant, supporting studies and projects
that inform science-intensive agency rules, guidances, policies and related products.
The requirements should also apply to all rules that are withdrawn, whether ORIA
has reviewed them or not.

144

b. Ensuring the Transparency and Scientific Integrity of Influence in Scienceintensive rules from White House Offices, beyond OMB, and Other Agencies
Executive Order 12866 and the OMB Peer Review Bulletin ensure increased
transparency and scientific integrity for OIRA review, but they do not apply to the other
White House offices or to suggestions made by other agencies. To the extent that some
changes to science-intensive rules are made at the behest of these other offices, such as
the Office of the President, CEQ, or OSTP or other agencies, there appears to be no
requirements to ensure that the changes are disclosed, justified or reviewed. 606
Additionally, as OMB’s influence in science-intensive rules is documented more
completely under Section 6 of Executive Order 12866, the resulting heightened
transparency surrounding OMB review may simply cause these other offices and
agencies to play an even greater behind-the-scenes role in editing high stakes agency
rulemakings. When rules are highly technical and complex, it is important to ensure the
rules are subjected to vigorous public and scientific oversight, up until their final
promulgation. This is not possible if changes can be inserted without disclosure and at
the eleventh hour.
While further analysis and research is needed to determine how best to strike the
balance between the President’s need for deliberative processes versus the benefits of
transparency of material changes to science-intensive rules, in the interim the agencies
should disclose changes made at the suggestion of White House offices and other
agencies if doing so does not undermine the deliberative process. In some and perhaps
many cases substantive changes are made and whose public recognition would not
threaten deliberative processes and that would benefit from disclosure and heightened
public and scientific review. The agencies should work closely with White House offices
and other agencies to ensure that when changes are made that can be disclosed, they are
cataloged as well in keeping with the direction and spirit of Section 6.
Beyond extending Section 6(a)(3)(E)(iii) disclosures to some of the other changes
made at the suggestion or recommendation of White House offices and other agencies,
the other recommendations produced in this report may help mitigate some risks of
nontransparent changes of science-intensive rules. For example, adopting the NAAQS
analytic process discussed earlier will at least draw out the ways that science informed
the rule through the course of its development, which in turn make the identification of
last minute changes easier to locate in a voluminous record. Additionally, ensuring that
the agency’s assessments of the literature provide attribution to agency staff provides
these authors with some stake in ensuring the reports and accompanying literature is not
misrepresented later in the process. And a dissent policy, if framed broadly enough, can
606

This may be problematic to the extent there are allegations of “politicization of science” stemming from
the White House. See generally Percival, supra note 355 (describing more recent incidences, which include
some politicization of science); OIG MacDonald Report, supra note 11.

145

allow staff to take issue with changes to rules that diverge from the scientific evidence
developed during rulemaking process. While none of these reforms are magic bullets for
this broader problem of the politicization of science, they are at least steps in the right
direction.
The agency should disclose material changes made at the suggestion or
recommendation of White House offices or other agencies, consistent with Section
6(a)(3)(E)(iii), when doing so does not impair the deliberative processes.
C. Enhancing the Transparency of Research produced by Regulated Parties
Agencies often rely on research produced by private parties in reaching scienceintensive regulatory decisions, yet there is considerable evidence that this sponsorproduced research can be permeated with various biases – both overt and covert – that
affect the ultimate outcomes.607 Meta analyses of studies in biomedical medicine, for
example, reveal strong statistical evidence of a “funding effect”, in which research
funded by a sponsor produces research outcomes that are much more favorable to that
sponsor than research produced independently, without any financial conflicts. 608
While agencies review this applicant-submitted information carefully, 609 in a
number of cases this sponsored research is not accessible to outside parties for additional
verification and oversight. Indeed, in some cases the agency scientists themselves may
not be fully apprised of the circumstances surrounding the sponsor’s role in influencing
the research design, implementation, or reporting. Trade secret and related
classifications, as well barriers to public access to manufacturer research under FIFRA,
only serve to drive a further wedge between the parties producing this research and the
ability of others, even those within government, to review it.
Despite evidence of bias in sponsored research, existing federal laws and
regulation treat private research gingerly with respect to disclosure requirements; indeed,
private research is exempted from virtually all of the transparency requirements imposed
on federally funded research. 610 The laws and regulations, in other words, do precisely
the opposite from what the underlying quality of the research would demand. They tend
to insulate private research from scrutiny and focus attention on public research.

607

See, e,g., Justin E. Bekelman, Yan Li, & Cary P. Gross, Scope and Impact of Financial Conflicts of
Interest in Biomedical Research, 289 JAMA 454 (2003).
608
Wendy Wagner and David Michaels, Equal Treatment for Regulatory Science: Extending the Controls
governing the Quality of Public Research to Private Research, 30 AMERICAN JOURNAL OF LAW &
MEDICINE 119, 122 (2004); see also infra note 616 (citing numerous sources).
609
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
610
Wagner & Michaels, supra note 608, at 129 (documenting this).

146

A first, simple reform to address this nontransparency of private research used for
regulation is a set of rules that treat private and public research similarly with regard to
the public accessibility of the data and studies and that require disclosures regarding the
provenance of the research used for regulation. While additional interventions may
ultimately be needed to address more entrenched features of sponsor bias (like rejecting
all research in which sponsors reserve the right to influence or control research), 611 the
recommendations offered here take only the first step and are thus extremely modest in
their reach. These recommendations are not only anchored in an extensive literature, but
are excerpted generously from the Bipartisan Policy Center 612 and the Keystone Center
reports.613
It should be noted that this short but dense set of recommendations owes its
origins to the NAS meeting and to OSHA Administrator David Michaels in particular,
who suggested that the report also address of the most egregious transparency issues
associated with privately produced research used for regulation. 614 In sharp contrast to
some of the issues identified in the final section of this report that are in need of further
study – especially external expert review 615 – the disparate treatment of private versus
public research has been studied extensively. To avoid adding still more volume to the
final report, the reader is referred to some of the extensive documentation of these
problems that serve as the basis for reform. 616

611

See, e.g., Woodruff et al., supra note 492 (advocating this exclusionary rule for literature analyses in
clinical medicine).
612
See Bipartisan Policy Center, supra note 2.
613
Since both projects were bipartisan, they provide the primary source of recommendations even though
there are other excellent reports (to which I contributed) that provide still more fodder for recommendation,
most notably the Center for Progressive Reform Report Perspective: Clean Science, available at
http://www.progressivereform.org/perspscience.cfm, and CPR, Saving Science from Politics: Nine
Essential Reforms of the Legal System, available at
http://www.progressiveregulation.org/articles/SavingScience805.pdf.
614
See NAS Transcript, supra note 503, at 146-47 (statement of Dr. Michaels).
615
The last rigorous study of external peer review of regulatory science was done by Sheila Jasanoff in
1990. SHEILA JASANOFF, THE FIFTH BRANCH: SCIENCE ADVISERS AS P OLICYMAKERS (1990). In this
outstanding study, Prof. Jasanoff identifies a number of critical issues that deserve further study – few if
any of which have been addressed since. Moreover, as expert review has become increasingly popular,
more issues and concerns have arisen regarding its proper implementation. For example, Prof. Jasanoff
pays only limited attention to the problems of conflicts in expert reviewers in her 1990 study, but this issue
currently appears to rank as the most difficult and important problem with expert advisory review, at least
according to the emphasis placed on the issue by both the Bipartisan Center and the Keystone Center
reports.
616
See, e.g., SHELDON KRIMSKY, SCIENCE IN THE PRIVATE INTEREST (2003); DAVID MICHAELS, DOUBT IS
OUR PRODUCT (2008); NAOMI ORESKES & ERIK CONWAY, MERCHANTS OF DOUBT (2009); THOMAS O.
MCGARITY & WENDY E. WAGNER, BENDING SCIENCE: HOW SPECIAL INTERESTS CORRUPT PUBLIC HEALTH
RESEARCH 233-39 (Harvard University Press 2008); RESCUING SCIENCE FROM POLITICS (Wendy Wagner &
Rena Steinzor eds. 2006); Wagner & Michaels, Equal Treatment, supra note 608.

147

1. Disclosure of Conflicts of Interest 617
Studies reveal that sponsorship can bias results; yet despite the value of
transparency about the provenance of regulatory resesarch, the disclosure of sponsor
influence is generally not required or even requested by federal regulatory agencies. The
EPA, the Occupational Safety and Health Administration, the Mine Safety and Health
Administration, the Consumer Product Safety Commission, and the National Highway
Traffic Safety Administration have no formal mechanisms to identify potential conflicts
of interest, nor do they provide any incentive to encourage the conduct of research that is
free of sponsor control. The Food and Drug Administration (FDA) has instituted a
conflict policy requiring financial disclosures for safety research conducted by private
parties in support of a license to market a drug or food additive. 618 These disclosures do
not, however, distinguish between research where the sponsor controls the design or
reporting of the research and research where sponsors have no control.
Regulatory agencies should adopt, at a minimum, requirements for disclosure
comparable to those of biomedical journals. 619 The Keystone report reinforces this
conclusion: “information on what entities or individuals funded a study should be readily
available.”620 “[E]stablishing and documenting the funder’s involvement with any given
study, and any restrictions placed on the study’s release” is also central to a
comprehensive literature review. 621 Disclosure of conflicts of interest should be required
for all research, regardless of whether it is federally or privately funded. Scientists should
disclose whether they have a contractual right to publish their findings free of sponsor
control and should identify the extent to which their work was reviewed by an affected
party before publication or submission to the agency. Sponsors who submit data should
similarly disclose if their investigators had the contractual right to publish without
sponsor consent or influence. Finally, other parties (i.e., trade associations, unions, or
public interest groups) who submit scientific results should disclose all known conflicts
of interests of the scientists conducting the studies.
Regulators should not use conflict disclosures to exclude research; they have the
obligation to consider all evidence, according greater importance to studies of higher
quality and relevance. Federal agencies should, however, develop policies that strongly
encourage clear disclosures that counteract the strong incentives for sponsors to influence
research. “[Without these basic disclosures,] agencies and scientific advisory committees
617

This section is taken almost verbatim from David Michaels and Wendy Wagner, Disclosure in
Regulatory Science, 302 SCIENCE 2073 (2003).
618
21 C.F.R. Part 54.
619
See, e.g., ICJME Uniform Requirements for Conflicts of Interest, available at
http://www.icmje.org/ethical_4conflicts.html ; ICJME Uniform Requirements for Authorship, available at
http://www.icmje.org/ethical_1author.html
620
Keystone, supra note 499, at 20.
621
Id. at 24.

148

should be extremely skeptical of a scientific study unless they are sure that the principal
investigator(s) (as opposed to the sponsor or funder) had ultimate control over the design
and publication of the study.” 622 Only then can agencies accurately weight studies and
encourage research independence.
Due to the difficulties of applying a conflict disclosure to past research, agencies
should require conflict disclosures for future research as a first step. These disclosures
will necessarily involve varying levels of detail depending on the import of the
information. The disclosure should be required at least when the information is appended
to comments or submitted as part of an application to obtain a license or permit.
Agencies should require conflict disclosures on all future research submitted
to inform an agency’s licensing, regulatory, or other decision-making process
-- whether published and unpublished. This conflict of interest disclosure
should be similar to the conflict of interest disclosure required by the
biomedical journals. See See ICJME Standards. The regulatory conflict of
interest disclosure should also, where possible, identify whether a sponsor
reserved the right to participate in the design of the research; the collection
of data; the interpretation of data; the writing or disseminating the report, or
any other material features of the research. Finally, “[a]gencies and
scientific advisory committees should be extremely skeptical of a scientific
study unless they are sure that the principal investigator(s) (as opposed to the
sponsor or funder) had ultimate control over the design and publication of
the study.”623
2. Data Disclosures in line with OMB Circular A-110 for All Regulatory
Research624
Although some of the private data underlying regulation are already available in
theory, in practice, public access to these data can be quite limited. In licensing
programs for drugs, pesticides, and toxic substances, for example, the FDA and the
EPA do demand access to data in advance as a condition to accepting studies for
regulatory purposes, but the data often remain buried in agency files, or in some and
perhaps many cases is stored in company files rather than in agency files, where it is
insulated from FOIA requests.625

622

Bipartisan Policy Center Report, supra note 2, at 42.
Id.
624
This section is drawn almost verbatim from MCGARITY & WAGNER, BENDING SCIENCE, supra note 616,
at 244.
625
Thomas O. McGarity, Federal Regulation of Mad Cow Disease Risks, 57 ADMINISTRATIVE LAW
REVIEW 292, 318–319 (2005).
623

149

Under the Data Access Act, all publicly grant-funded research that informs
regulation must be publicly accessible with very limited exceptions. 626 “[I]n general, the
data on which conclusions are based should be available, regardless of who funded or
conducted the study.”627 This second recommendation simply extends the Data Access
Act to privately submitted research. In extending the Data Access Act to all research, it
might also be appropriate to take the additional step of requiring the privately produced
data submitted in at least an application for a license or permit to be posted on the
internet to ensure ready public access. While legislation or an OMB directive would
provide the most centralized means for extending the Data Access Act in this fashion,
agencies can extend the reach of the Data Access Act in their implementation of
individual programs or through their inherent authorities.
“Studies used in the formulation of regulation should be subject to data
access requirements equivalent to those under the Data Access Act (Shelby
Amendment) and its implementing circular (OMB Circular A-110)
regardless of who funded the study. If a study is used by an agency to inform
the development of a regulation, then the same kinds of information about
that study should be available upon request, regardless of whether the study
was funded by the federal government, industry, or some other entity.”628

626

Wagner & Michaels, Equal Treatment, supra note 608, at 137-38.
Keystone Center Report, supra note 499, at 20.
628
Recommendation copied verbatim from Bipartisan Policy Center Report, supra note 2, at 43.
627

150

3. Minimizing Unsupported and Outdated Confidential Business Information
Claims629
Considerable evidence reveals that companies use the nebulous “trade secret”
claim to unjustifiably insulate from public view some of the information they must
nevertheless share with regulators, like basic information on chemical structure or even
some toxicity information. 630 In most EPA programs, classifying information as trade
secret protected is easy to do: a company need only stamp the information “confidential
business information” and the claim automatically applies, unless an outside party seeks
the information under the FOIA. 631 In fact, even if the agency discovers that a claim is
wholly without basis— usually in the course of responding to an FOIA request—there is
no penalty. 632 As a result, from a company’s perspective, there is little to lose and a
great deal to be gained from routinely claiming trade secret protection, since the
classified information is at least temporarily removed from public view. A number of
companies in fact appear to be taking full advantage of this permissive program.
Investigations conducted by the EPA, the General Accounting Office, and an outside
consulting group all found a significant amount of unjustified overclaiming. 633
Some regulated parties have candidly conceded that they sometimes illegitimately
claim trade secret protection.634 Furthermore, they defend the EPA’s overly protective
approach on the ground that any agency-imposed requirement to justify the numerous
trade secret claims already in existence would violate the Regulatory Flexibility Act

629

The text is borrowed almost verbatim from MCGARITY & WAGNER, BENDING SCIENCE, supra note 616,
at 116
630
See, e.g., Wagner & Michaels, Equal Treatment, supra note 608, at 129-35, 151-54.
631
See Environmental Protection Agency, “Public Information and Confidentiality: Advance Notice
of Proposed Rulemaking; Withdrawal of 1994 Proposed Rule,” 65 Fed. Reg. 80394, 80395
(December 21, 2000).
632
See generally 40 C.F.R. §§ 2.201–2.310. See also Christopher J. Lewis, Comment, When Is a Trade
Secret Not So Secret?: The Deficiencies of 40 C.F.R. Part 2, Subpart B, 30 ENVIRONMENTAL LAWYER
143, 171–172 (2000).
633
Environmental Protection Agency, “Public Information and Confidentiality Regulations,” 59 Fed.
Reg. 60446, 60446–60447 (November 23, 1994); U.S. General Accounting Office, Toxic
Substances Control Act: Legislative Changes Could Make the Act More Effective 56-58 (Washington,
D.C.: GAO, 1994); Hampshire Research Associates, Inc., Influence of CBI Requirements on TSCA
Implementation 4 1 (March 1992) (report obtained from EPA through FOIA and on file with
authors); see also Environmental Protection Agency, Confidential Business Information (CBI)
Review, Pesticides: Freedom of Information Act, www.epa.gov/pesticides/foia/cbi.htm (listing
environment-related information that is commonly claimed as confidential).
634
Both the Keystone and the bipartisan reports acknowledge that CBI is overused today. See Keystone
Center Report, supra note 499, at 21; Bipartisan Center Report, supra note 2, at 43. Underlying these
bipartisan acknowledgements is extensive documentation of the overuse of CBI to classify information that
has no conceivable trade secret value, much less trade secret value that overrides the public’s interest in
learning more about the toxicity of various products or drugs.

151

because of the enormous work burden it would create for small manufacturers to sort
out which of their trade secret claims are actually valid and which are not. 635
To reduce this problem, “[a]gencies should review their laws, regulations,
policies, and practices to ensure that CBI and other proprietary claims are not being used
to protect information that need not be confidential. . . . Agencies have a responsibility to
inform all stake-holders of the methods and manner they employ to determine what
material is granted CBI status, and how it is reviewed and analyzed” 636 in as clear and
publicly understandable methods as possible.
“Confidential Business Information (CBI) claims can . . . make it difficult for
the interested public to evaluate studies that contribute to regulatory
policy.”637 Agencies that provide CBI protections for studies or data that
inform regulation should ensure that the CBI claims are justified. Given the
strong incentives to regulated parties for overclaiming CBI protection and
the resultant costs from this overclaiming to public health protection and
research, it is important that the agencies’ CBI programs not provide a safe
haven for unjustified suppression of relevant regulatory research. 638 To that
end and as a first step, the agencies should review their CBI programs to
ensure that there is rigorous oversight of CBI and related trade secret claims
on health and environmental research. Agencies should, where possible,
penalize those CBI claims that, upon review, appear unjustified.
D. Future Questions for the Study of Regulatory Science
There are many issues regarding the agencies’ use of science that will benefit
from further research. Some of the issues that emerged from this study that would benefit
significantly from further study and attention are described below:
The role of advisory groups and external peer reviewers
For some regulatory projects, Congress requires the agencies to empanel a science
advisory board to review their work.639 In a far larger number of regulatory projects, the
agencies themselves determine that some form of expert peer review will be helpful, and
635

See, e.g., Warren E. Stickle, president, Chemical Producers and Distributors Association, & Bill
Balek, president, International Sanitary Supply Association, letter to EPA,
www.cpda.com/TeamPublish/uploads/ CPDAISSA Comments.pdf. [Note; This letter has apparently
been removed from the Internet and was not located on Regulations.gov. To access the letter, one
may need to file a FOIA request with EPA].
636
Keystone Center Report, supra note 499, at 21.
637
Bipartisan Policy Center Report, supra note 2, at 43.
638
Id.
639
See Sections III.A.1.a. and 3., supra.

152

they decide the form that peer review should take. In cases where agencies elect to use
some form of external peer review, however, their decision-making processes are often
unexplained. It is not clear, for example, when or why an agency decides that it should
utilize external peer review. For instance, EPA and the FWS use external peer review for
IRIS and listing decisions respectively, but EPA does not use external peer review for
pesticide registration reviews. It is also not clear whether agencies always have the
ability to empanel FACA panels in settings when those panels might be their preferred
process for engaging external expert peer review, as compared with individual expert
reviewers. 640
Regardless of the analysis undergirding these choices, once the agency settles on
the need for external peer review (or is required to use this expert review), the first and
major challenge faced by the agency is the development of a credible process for
selecting reviewers. 641 Determining when scientists have conflicts that exclude them
from serving as an expert advisor is the most visible and perhaps the most difficult
decision that an agency faces.642 Yet there are also challenges associated with ensuring
that the agency has the proper disciplinary mix of experts and that these experts are not
skewed in favor of an interested party or otherwise cherry picked by the agency to
support its decision. Existing federal guidance on these various challenges in assembling
panels, particularly under FACA, is slim. 643
There are other challenges as well. For example, in regulatory areas that are
highly specialized, the pool of experts may not be as large as the requisite size of the
expert panel need to conduct the review.644 Agency programs also appear to vary in the
point(s) in the process they decide to use expert review 645 and in how these advisors will
be used, such as: commenting on agency products (e.g., EPA’s IRIS) 646 versus actually
advising the agency in a more iterative way as a collaborator (e.g., EPA’s CASAC in the
NAAQS) 647 versus actually suggesting projects and priorities (e.g., NRC’s ACRS)). 648
There also appear to be significant differences in the amount of work expected of these

640

See Section IV.A.3., supra.
For statutorily created advisory bodies, Congress may place additional constraints on the agencies’
selection process. See Sections III.A.1.a. and 3., supra (discussing these features in the creation of CASAC
and NRC Advisory boards).
642
See Bipartisan Policy Report, supra note 2, at chapter 2 (devoting much of its coverage of science
advisory boards to these conflict challenges).
643
See, e.g., GAO, Federal Advisory Committees, Additional Guidance Could Help Agencies Better
Ensure Independence and Balance (April 2004).
644
Interview with EPA Staff, Office of General Counsel, Jan. 13, 2012.
645
Compare CASAC w/ NRC’s ACRS, for example. See Sections III.A.1.a. and 3., supra.
646
See Section III.A.1.c., supra.
647
See Section III.A.1.a., supra.
648
See Section III.A.3., supra.
641

153

advisors,649 as well as the duration of the advisory committees, which range from
standing committees to one-shot reviews. 650
It is clear that there is no “one size fits all” to engaging external expert reviewers in
the oversight of agency regulations, yet the reasons for much of this variation remain
unclear. A study exploring these sources of variation would provide a solid foundation
for a second study that then considers the ideal roles that advisory boards might offer to
science-based regulation and how agencies might use external peer review more
effectively. This topic is extremely important and should be considered high priority for
further research.
Challenges to evaluating the reliability of data and studies submitted by
applicants, particularly studies that are non-transparent (e.g.,
unpublished and/or publicly inaccessible)
A recurring theme in the study of the different agency programs, as well as in
more general interviews, was a concern about how to ensure the reliability of scientific
data and studies that are not published and/or publicly accessible. Coincidentally, much
of this scientific information is submitted by applicants or regulated parties that generally
have vested interests in the outcome of the research. In the case of pesticides, for
example, FIFRA requires a member of the public to obtain Section 10(g) clearance before
he/she can view the data or research submitted by pesticide manufacturers in support of
their pesticide registration.651 While a member of the public can typically gain this 10(g)
access, it requires effort and is generally available only after a pesticide registration
decision has been made. 652 Since the research and data submitted by pesticide
manufacturers often comprise the majority of the studies available on a particular
chemical, and since those studies can sometimes number in the hundreds for a single
pesticide,653 these barriers to public access to the underlying research that informs
pesticide decisions are not a trivial problem.
Analogous problems arise in other agency programs. In natural resources law, for
example, FWS consultations under section 7 of the ESA are required when an
endangered species may be adversely affected by a federal action. 654 Yet the research
that the FWS uses to make a determination of whether the federal project will put the
species in jeopardy is a biological assessment conducted by the federal agency seeking to
649

See id (identifying the commitment required of scientists to serve on advisory boards for NRC).
Compare IRIS’s use of expert peer reviewers with the role of CASAC in the NAAQS process. See
Sections III.A.1.a. and c., supra.
651
7 U.S.C. § 136h(g)(1).
652
See Section III.A.1.b., supra.
653
Interview with FWS Staff, Endangered Species Program, Jan. 26, 2012. This could be empirically
verified by counting the number of studies in the data tables for each pesticide listed in EPA’s risk
assessments, but it was beyond the scope of this study.
654
16 U.S.C. § 1536(a).
650

154

undertake the project.655 Stakeholders and the FWS concede that the quality and
comprehensiveness of these assessments are not always up to the FWS’s standards. 656
Beyond the inherent conflicts involved in preparing the assessment, the applicant agency
may not consider the assessments a high priority and may allocate resources
accordingly. 657 Many of these preliminary assessments are not subject to notice and
comment, moreover, and the public may be only dimly aware that the processes are
occurring, much less have the ability and knowledge to request these documents in order
to review them. 658
Similarly, in the re-licensing of nuclear reactors, the NRC staff has been criticized
for not verifying technical statements and assessments contained in an operator’s
application. 659 The NRC’s OIG found that this verification work could have made a
material difference in the safety evaluations of applications, at least in some cases. 660
Moreover, none of this underlying technical information was available to the NRC
personnel located off-site. 661 Stakeholders could also not independently verify whether
the applicant-provided information was reliable.
While agencies acting on applicant-submitted information suggest that they
review this information carefully, 662 the process remains nontransparent to the extent that
outside parties cannot verify the reliability of this information independently. In most
cases there is also no independent peer review of the information. Stakeholders are
concerned that in these cases the agencies’ decision-making processes will not provide
sufficient supporting information to the public to allow for rigorous review. 663

655

Interview with former staff of Environmental Defense, Jan. 19. 2012; Interview with FWS Staff,
Endangered Species Program, Jan. 26, 2012.
656
Interview with FWS Staff, Endangered Species Program, Jan. 26, 2012.
657
Similar quality problems can afflict Clean Water Act Section 404 applications that permit developers to
place fill material in wetlands (technically to discharge a pollutant into waters of the United States). 33
U.S.C. § 1344. These applications are prepared by the developers, yet they provide much of the
information the Corps of Engineers use to determine whether the application should be granted. Similar
quality problems can afflict Clean Water Act Section 404 applications that permit developers to place fill
material in wetlands (technically to discharge a pollutant into waters of the United States. Interview with
staff, PEER, Aug. 5, 2012.
658
Interview with former staff of Environmental Defense, Jan. 19. 2012. The MMS’ assessment of
environmental impacts in the North Aleutian Basin was criticized by GAO fr classifying industry data as
proprietary in ways that did not allow it to be shared more widely within the agency for purposes of
analysis. GAO, Offshore Oil and Gas Development: Additional Guidance Would Help Strengthen the
Minerals Management Service’s Assessment of Environmental Impacts in the North Aleutian Basin, GAO10-276, March 2010 (concluding that staff assessments are incomplete and vulnerable in part because of
the lack of clear guidelines for preparing NEPA-required assessments of oil and gas development).
659
See Section III.A.3., supra.
660
Id.
661
Id.
662
Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
663
Interview with staff, PEER, Aug. 5, 2012; Interview with former Environmental Defense employee, Jan.
19, 2012.

155

The ability of resource-strapped agencies to provide rigorous and transparent
oversight of data submitted by applicants or regulated parties – without the benefit of
external review by experts or even stakeholders or other members of the public –
deserves further study. Some agency officials suggest that the agency dedicates
considerable staff resources to ensuring the rigorous oversight of this incoming
information, such as in pesticide registration reviews. 664 Yet the pressure to finalize
decisions, even in the licensing context and particularly under statutory deadlines, may
impair the ability of the agencies to undertake a number of iterative review steps with
applicants in order to ensure that their studies are complete. 665 To the extent that there is
no external oversight of these internal assessments of applicant-provided data and
research, added procedural protections may be needed.666
Expert Elicitation
This study’s foray into expert elicitation, thanks to the work of Roland Frye,
highlights a developing and largely obscure area of science-policy that may become
increasingly important over the next decade.667 Expert elicitation is an approach that is
even more complex than science advisory boards and even less well understood.
Moreover, with the exception of Mr. Frye’s research, there is little analysis of the existing
use of expert elicitation in the agencies. Clearly, this area deserves further study.
Just as with science advisory boards, it is useful to understand the circumstances
under which agencies decide to use expert elicitation for regulation. Researchers should
also explore the primary challenges associated with the agency’s use of expert elicitation,
such as selection of the experts; transparency of the deliberations; and the agency’s
charge to the group.668 Unlike advisory boards, moreover, there may be benefits to
agencies, at least in the beginning of their use of expert elicitation, in employing several
expert elicitation panels simultaneously or over time to obtain multiple predictions for the
same question, as well as to develop additional mechanisms to help the agency ensure
that its processes are credible.
Agency Outreach and Decision-making during the preNPRM stage
This study highlights the efforts of certain agencies to provide transparency for
their science-based decisions. These elaborate, iterative processes occur wholly before
the proposed rule and notice and comment process. While formal notice and comment is
664

Interview with EPA Staff, Office of Pesticide Programs, Jan. 27, 2012.
Id.
666
For example, in situations like trade secret when the public and independent reviewers cannot access
underlying studies or data, prophelactic processes – like mandating a classified peer review panel that
audits the quality of the information or even periodic replication of the data or studies – seems justified, at
least until it can be determined whether the information is of high quality.
667
See Frye Paper at Appendix B
668
See id.
665

156

not made superfluous by this extensive front-end work (particularly given the role of
OMB review on the proposed and final rules), it seems probable that the 7+ iterative
opportunities for public comment and peer review in the NAAQS process have a
substantial impact on the content of EPA’s draft proposed rule.
To the extent that administrative process remains focused on the notice and
comment process as the major vehicle for soliciting public and stakeholder input, this
study suggests that in at least some areas of science-based regulation, such a focus is
missing most of the action. The processes employed by EPA and to some extent the
NRC indicate that a great deal of interaction with stakeholders occurs before the
proposed rule stage. The differences between agencies in their approach to science-based
decision-making during the preNPRM period also raise normative questions about
regulatory process. It is possible, for example, that in more complex regulatory projects,
additional formal stages of public and scientific input and dialog are needed on the
agency’s scientific assessments to provide a truly meaningful opportunity for public
comment. The transparency afforded by the notice and comment period following a
proposed rule, at least in some rulemaking settings, may not be sufficient. Understanding
when these additional, preNPRM processes might be necessary for meaningful public
review and exploring the form(s) that those processes could take thus deserves further
study.
The privileged role of science
A theme that surfaced from a few interviewees 669 and was reinforced by critical
commentary,670 is a concern that the agencies may in some cases be almost too focused
on scientific information developed within the natural sciences, and that this type of
scientific information becomes “privileged” at the expense of other vital information,
including social science research, that should inform the agency’s mission. For example,
when asked what he thought the biggest problem was with EPA’s science, one prominent
interviewee stated that it was EPA’s failure to take into account social science
information, such as behavioral features that affect consumer behavior and hence cut to
the core of EPA’s mission of protecting public health. 671
Thus, another area for research is an examination of whether the agencies’ high
standards for oversight, review, and the creation of elaborate scientific databases might
669

Interview with Academic that was involved with EPA’s Science Advisory Board, July 13, 2012;
Interview with Staff, Pesticide Research Institute (a public interest organization), Aug. 1, 2011.
670
See, e.g., WILLIAM ASCHER, TODDI STEELMAN, AND ROBERT HEALY, KNOWLEDGE AND
ENVIRONMENTAL POLICY: RE-IMAGINING THE BOUNDARIES OF SCIENCE AND POLITICS (2010) (arguing that
science is unduly privileged in environmental law in the U.S. to the exclusion of other relevant and useful
information).
671
Interview with Academic that was involved with EPA’s Science Advisory Board, July 13, 2012; see
also Letter to Lisa Jackson, EPA from Deborah Swackhamer, Chair of EPA, Science Advisory Board, July,
8, 2012, 5-6 (discussing the important but neglected role of the social sciences in EPA’s work).

157

crowd out the development and use of other valuable types of information in certain
regulatory settings. This exclusion of or inattention to other types of useful information
could be due to an overly narrow framing of the kind of information that is relevant to the
decision or an unnecessarily narrow framing of the agency’s goal or mission.
Conclusion
A long-standing challenge for regulatory science is to ensure that policy decisions
can be assessed and set against the scientific record. Interested observers who are not
intimately involved in the rule should be able to trace the use of science into the final
regulation and, through this step-by-step explication judge the veracity of the use of
science at every turn. This scientific transparency should not be theoretical, either. Six
feet of NAAQS criteria documents, standing alone, do not serve as a meaningful
backdrop against which to assess the scientific rigor of a final policy decision. Scientific
transparency means that agencies have provided an accessible means for sophisticated
onlookers to identify discrepancies between the scientific record and the policy decisions
being made on that record and to evaluate why those changes were made.
The recommendations advanced here are intended to advance this goal – namely
to develop processes that ensure that agencies create scientific records that not only
inform decision-making, but can be used to check or judge the quality of those decisions.
These recommendations first attempt to remove the external barriers that agencies face in
developing more rigorous and transparent processes. A review of five different
regulatory programs in three agencies reveals that there are some significant barriers,
particularly with respect to the lack of transparency associated with OMB review.
The recommendations also suggest a list of best practices that agencies should
consider in developing their decision processes that incorporate science into regulatory
policy. These best practices not only identify key steps to ensuring rigor and
transparency in the agencies’ use of science, but identify innovations employed by other
agencies that can be adapted to agency processes without a significant investment of
resources or the need to develop new programs or hire more officials.
Finally, the recommendations call upon OSTP to help collect and generate
important information on the agencies’ use of science. This includes requiring agencies
to describe their processes involving the use of science in more detail. OSTP should also
create a forum for advertising agency innovations that can serve as best practices across
government.
Regulatory science has been in the dark long enough. President Obama’s
integrity initiative provides the impetus not only for encouraging regulatory agencies to
develop their own, improved integrity policies, but also for spurring oversight agencies
158

like OSTP to seize the opportunity to collect critical information on agency processes that
can be used more broadly to share successes and enlist outsiders in developing new and
improved regulatory policies.

159