Independent Evaluators of Federal Programs: Approaches, Devices, and Examples
Released: 2010-08-16T00:00:00

Introduction

An evaluation may provide information at any stage of the policy process about how a federal government policy, program, activity, or agency is working. The resulting information may be used to better understand policy problems and to help inform the design, implementation, and oversight of policies. A variety of stakeholders may seek evaluation information and data to help them exercise their responsibilities, obligations, and rights in a representative democracy, including the establishment, implementation, oversight, and other requirements of a federal program or project. These stakeholders and interested parties include, among others, Congress, the President, federal agency officers and employees, state and local governments, interest groups, private sector contractors, the news media, and the general public.

For a number of reasons, there is not necessarily a best way to design and carry out an evaluation of a particular program, activity, or operation. A program's context, including its longevity and the manner in which it is implemented, typically influences choices about evaluation design. In addition, stakeholders often have different needs and research questions about which they would like more information. One aspect of choosing how to carry out an evaluation involves deciding when some kind of "independence" would be a desirable attribute.

In the context of evaluation, independence may apply to an evaluation or to an evaluator. On one hand, for example, the term may relate to independence of an evaluation from the policy preferences of an individual or group ("independent evaluation"), perhaps by prohibiting political appointees from revising an evaluation. Independence also may refer to an entity that conducts evaluations that also is located outside the immediate organization responsible for policy implementation ("independent evaluator").

There is some diversity of opinion regarding how to define independence and what makes an evaluator (or evaluation) independent. For example, an evaluator's "external" status, outside the organization that is implementing a program, does not necessarily equate with independence. Nor would an evaluator's "internal" status, inside the implementing organization, necessarily equate with a lack of independence for an evaluation (e.g., if an expert panel reviewed the internally produced evaluation for bias). There is also varying opinion concerning when independence is necessary, or possibly counterproductive, and what value it may bring. In some situations, there may be little need for independence in evaluations (e.g., when equipping a program with capacity to improve its own operations iteratively, or when "adaptively" managing a project to help restore an ecosystem). Nonetheless, instances of independent evaluators appear to be growing in number and variety.

This report focuses on the characteristics of independent evaluators (IEs)—examples of which are described in the Appendix—when an evaluation is to be conducted by an entity outside the immediate organization that is responsible for policy implementation, and the IE is also intended to have one or more dimensions of independence. IEs and similar constructs that conduct independent evaluations, however, vary across a number of attributes: structure, jurisdiction, authority, resources, length of tenure, and specific duties and responsibilities. These differences, in turn, could affect their capabilities, effectiveness, and assistance to others, including their contributions to the oversight of a program or project by the executive or legislature.

After an overview of such entities—which include newly and specially created units as well as existing ones, such as the Government Accountability Office (GAO) and offices of inspector general (OIGs)—this report suggests possible broad characteristics and criteria of independent evaluators or similar units. In situations when some extent of independence in an existing or proposed evaluator were deemed desirable, Congress might consider the different kinds and degrees of independence that could be pursued. The final section describes a number of such offices, along with citations to relevant materials for each example (public laws, legislative proposals, executive branch documents, and secondary analyses).

Overview of Independent Evaluators and Similar Units

Evaluation in the Federal Government

Evaluation's roots have been characterized as reaching as far back as hundreds of years or, even more remarkably, thousands of years. This lengthy heritage notwithstanding, some observers describe the "modern era" of evaluation, and particularly "program evaluation" (focusing on government social programs), as emerging in the 1960s and growing appreciably since then. Expansion of social, economic, and environmental programs, among others, in many cases has been accompanied by legislation mandating and funding evaluation or by executive directives ordering such studies. These changes facilitated and increased the diversification of program evaluation, based on a perceived need to better inform the understanding of policy problems, formulate responses, and strengthen oversight by the executive and legislature over a growing expanse and complexity of federal programs. Accompanying this development, program evaluation has also become more sophisticated and selective in terms of its methods, methodology, type and level of operation, and subject or policy area.

The 1990s gave added impetus to assessments, evaluations, and reviews of programs, activities, and operations through a number of laws, which remain on the books. Among these supporting evaluation efforts, directly or indirectly, are the Chief Financial Officers Act of 1990 (CFO Act; P.L. 101-576 , 104 Stat. 2838); Government Performance and Results Act of 1993 (GPRA; P.L. 103-62 , 107 Stat. 285); Information Technology Management Reform Act of 1996, later renamed the Clinger-Cohen Act (CCA; Division E of P.L. 104-106 , 110 Stat. 679, and P.L. 104-208 , 110 Stat. 3009-393); and Federal Financial Assistance Improvement Act of 1999 ( P.L. 106-107 , 113 Stat. 1486). These and other earlier relevant statutes, including the Inspector General Act of 1978 (IG Act), as amended, allow for flexibility in determining various evaluation requirements, particularly concerning individual projects and programs. GPRA, for example, defines "program evaluation" quite generally as "an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives."

Independent Evaluators and Similar Units

As noted earlier, independent evaluation oftentimes refers to the review and assessment of how well programs and projects are working that is conducted by a unit outside the program office itself. Such units have been expressly authorized to perform independent evaluations or have implied authority to do so under a broad mandate to oversee or review a program, project, activity, or operation.

A Variety of Names and Titles

Independent evaluation can be, and has been, carried out by positions under a number of different names and titles. Specific ones identified in this study include independent evaluator, independent auditor, interagency coordinating research institute, accountability board, peer reviewer, independent or peer review panel, program evaluator, program evaluation unit, research and evaluation institute, inspection and evaluation (I&E) unit in an office of inspector general (OIG), and various organizational groupings in the Government Accountability Office. These have been established within the parent agencies (but separate from the program office) or in outside organizations, public or private. In addition, such entities, whatever their name and wherever located, appear to be growing in number and in a variety of policy domains.

Lack of a Precise, Agreed-Upon Definition

The concepts of independent evaluator and independent evaluation, however, have led to generalized understandings, rather than a precise, detailed, agreed-upon definition. The broad notions lack specification, standardization, and uniformity. There appears to be no express, across-the-board definition of an "independent evaluation" or "independent evaluator" in federal statute or regulation. (Further discussion of these concepts appears below.) As a result, there are significant variations among offices which may fall under the same rubric or which conduct the same function. In other words, "independent," "evaluator," and "evaluation" are subject to different meanings among offices that use similar terminology.

Considerable Diversity in Attributes and Characteristics

The research for this analysis—which identifies a number of independent evaluators or similar entities and functions—found that none of the entities were identical. Although there were similarities governing several key characteristics in some cases, there were substantial differences of structure, organization, authority, jurisdiction, funding, staffing, length of tenure, or duties and responsibilities.

Dissimilarities among these characteristics in IE-like positions and functions, in fact, appear to be more common than their similarities. This arises, in part, because of different rationales, expectations, research questions, and conditions surrounding the establishment of the positions. Because of this, the resulting independent evaluators lack standardization and uniformity across-the-board, and a number have been given substantial flexibility in organizing their own operation. Structural differences might also arise, at least in some cases, because the creator of the entity (the President, agency head, or Congress) intends to allow for discretion and flexibility at the implementing level (for the agency or IE, reflecting his or her expertise and experience). Along with this, the establishing authority for many positions—public law, executive order, or administrative fiat—often do not specify certain characteristics of the office, such as its funding, reporting requirements, tenure, or evaluation standards.

Disagreements, moreover, have arisen over the value, importance, and means of supporting independence for evaluators. The Government Accountability Office (GAO), when it examined the Office of Management and Budget's (OMB's) guidance for the Program Assessment Rating Tool (PART) in 2005, found that a

source of tension between OMB and agency evaluation interests was the evaluation's independence. PART guidance stressed that for evaluations to be independent, nonbiased parties with no conflict of interest, for example, GAO or an Inspector General should conduct them .... OMB subsequently revised the guidance to allow evaluations to be considered independent if the program contracted them out to a third party or they were carried out by an agency's program evaluation office. However, disagreements continued on the value and importance of this criterion.

Furthermore, an evaluation scholar noted that independence may not be an important dimension of an evaluation if the objectives of the evaluation primarily focus on strengthening institutions or building "agency or organizational capacity in some evaluative area."

Other considerations tie into program evaluation in general, no matter where it is carried out. These could extend to whether the subject of the evaluation is either (1) too large and broad an undertaking for the evaluator, thereby overwhelming him or her; or (2) to the contrary, too small and narrowly focused to be of utility apart from the immediate project (in developing, for instance, recommendations for best practices or the entire program). Along the same lines are concerns about whether the evaluation is integrated into the overall operation of a program and whether an evaluator might be perceived as "usurping" or competing with the prerogatives of a parallel office. Others, discussed further below, deal with selecting the most appropriate evaluation technique or method, especially in light of the diversity among them, as well as the right evaluator.

Independent evaluation entities do not arise or exist in a vacuum. Their establishment, powers, performance, effectiveness, and impact are subject to a variety of influences. And because they are usually ad hoc and idiosyncratic, they differ in their structure and operation. IE-like offices range from modest efforts—beginning with a position without separate funding or staffing, evaluating only a single, short-term project—to major undertakings, extending to a separate unit composed of a number of individuals, with its own budget and resources, operating continuously throughout the life of a long-term, interagency, and sometimes intergovernmental program. In between these ends of the spectrum are a number of possible combinations. Variations exist even among units evaluating programs in the same broad subject area. Six of the examples below, for instance, deal with interagency, intergovernmental, and, in some cases, interstate waterway programs; yet the evaluation units differ from one another, sometimes significantly, in their main characteristics or in the specifications in their establishing authority.

In sum, the characteristics of independent evaluators and similar constructs are determined by a number of formal and informal factors. Differences among evaluation constructs are reflected in the powers and protections in the authorities that established the IEs—public laws, executive orders, or administrative directives—as well as in their relationships with executive officials. Besides the variables already identified, other influences, some informal or intangible, add to this mixture. These include the expertise of the staff, competency and impartiality of the evaluator, and trust and confidence between an evaluator and the program office. These, in turn, affect the IEs' actual and potential independence, capacity, capability, and effectiveness—including their contributions to oversight by Congress and the executive—extending from the immediate program office to the parent agency, OMB, and the President.

Possible Characteristics of Independent Evaluation Units

As emphasized above, the terms "independent evaluator" and "independent evaluation" lack precise, standardized, agreed-upon definitions in public law or executive directive, and they vary considerably across a number of dimensions. Nonetheless, many possible criteria and attributes of an independent evaluator or similar construct can be identified, based on the research for this examination and on other sources, both public and private. These sources include the American Evaluation Association (AEA); evaluation reference works, such as the Encyclopedia of Evaluation ; the Joint Committee on Standards for Educational Evaluation (Joint Committee); and federal agencies that conduct and direct evaluations of programs, projects, activities, and operations. Illustrative agencies include the Government Accountability Office (GAO), the Council of the Inspectors General on Integrity and Efficiency (CIGIE) and its predecessors, and the Office of Management and Budget (OMB).

Concepts, Understandings, and Specifications of "Evaluation" and "Independence"

These different sources cover common ground, including the concepts of evaluation and independence, ways in which the terms may be understood, and how the concepts may be specified. Differences exist, especially in how detailed, specific, and elaborate these descriptions are, ranging from a short statement on a concept to a lengthy listing and interpretation of relevant standards.

American Evaluation Association

AEA, which promotes evaluation for public programs, adopts a broad understanding of the concept of evaluation, which can occur throughout the life of a program: "Evaluation is a field that applies systematic inquiry to help improve programs, products, and personnel, as well as the human actions associated with them." While recognizing that evaluators' work can vary greatly, AEA holds that

the common ground for all evaluators is that they aspire to achieve accountability and learning by providing the best possible information that might bear on the value of whatever is being evaluated.... Evaluations prepared by professional, independent evaluators help prevent information gaps by: improving knowledge and understanding of how programs work, strengthening public accountability, assessing program effectiveness and efficiency, and identifying opportunities and pathways to achieving objectives, outcomes, and efficiencies.

To support these goals, AEA specifies "several of the key elements of a national framework for evaluation practices." These benchmarks include

using appropriate professional standards in conducting the work; stating program goals and objectives as specifically as possible; issuing performance measures when the program is being developed and modifying them as appropriate to reflect what has been learned; specifying necessary requirements and resources, which "should be embedded in the authorizing legislation and regulations"; supporting department-wide or government-wide (i.e., GAO) evaluators "with the resources, organizational independence, competencies, and authorities necessary for the effective evaluation and oversight of public programs"; using private evaluators "with a broad range of viewpoints and capabilities that can provide effective independent evaluation as well as input and feedback to internal evaluation efforts"; producing a wide range of studies, recognizing the advantages and limitations of various methodological approaches; and collaborating with stakeholders.

In July 2004, AEA approved a set of guiding principles for evaluators, which interrelate with one another as well as with the foregoing standards. The five, each of which is detailed in its brochure, are:

A. Systematic Inquiry: Evaluators conduct systemic, data-based inquiries. B. Competence: Evaluators provide competent performance to stakeholders. C. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire process. D. Respect for People: Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other stakeholders. E. Responsibilities for General and Public Welfare: Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation.

Encyclopedia of Evaluation

The Encyclopedia of Evaluation offers a working definition of "independent evaluation":

For an evaluation to be considered independent, the evaluator must be impartial, objective, unencumbered, and balanced. Further, because perceived independence is as important as independence itself, the evaluator must be accountable for every step in the research process and able to document all key decisions and actions for the client organization, or other evaluators, and the community at large. Overall, external evaluations tend to hold more credibility than internal ones because the external evaluator appears to have less to gain or lose from the evaluation findings and is less likely to experience a conflict of interest.

Notably, this definition does not appear to equate automatically an evaluator's "external" status with independence, or an evaluator's "internal" status with lack of independence.

Joint Committee on Standards for Educational Evaluation

The Joint Committee's standards for evaluating educational programs might be adapted to other fields. It posits that "sound evaluations of educational programs, projects, and materials in a variety of settings should have four basic attributes." The associated standards govern:

Utility , which is intended "to ensure that an evaluation will serve the information needs of the intended user." These include stakeholder identification, information scope and selection, evaluator credibility, values identification, report timeliness and dissemination, report clarity, and evaluation impact. Feasibility, which is intended "to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal." Specifics here cover practical procedures, political viability, and cost effectiveness. Propriety , which is intended "to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation as well as those affected by its results." These concerns involve formal agreements, rights of subjects, complete and fair assessment, disclosure of findings, dealing with conflicts of interest (actual, perceived, potential), and fiscal responsibility. Accuracy , which is intended "to ensure that an evaluation will reveal and convey technically adequate information about features that determine the worth of merit of a program." The specifics here deal with: program documentation; context analysis; described procedures and purposes; defensible information sources; valid, reliable, and systematic information; analysis of quantitative information; justified conclusions; impartial reporting; and metaevaluation (that is, a means of comparing a particular evaluation against the standards developed for its field along with other pertinent standards, in order to examine its strengths and weaknesses).

Government Accountability Office

GAO explains the concept of program evaluation by including within its scope evaluations conducted by external entities, experts inside the agency that contains a program, and the employees responsible for implementing programs and policies:

Program evaluations are individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working. They are often conducted by experts external to the program, either inside or outside the agency, as well as by program managers. A program evaluation typically examines achievement of program objectives in the context of other aspects of program performance or in the context in which it occurs. Four main types can be identified, all of which use measures of program performance, along with other information, to learn the benefits of a program or how to improve it.

Inspector General Community

The 2008 Inspector General Reform Act has directed the new Council of the Inspectors General on Integrity and Efficiency to "develop plans for coordinated, Governmentwide activities that address these problems and promote economy and efficiency in Federal programs and operations, including interagency and interentity audit, investigation, inspection, and evaluation programs and projects to deal efficiently and effectively with those problems concerning fraud and waste that exceed the capability or jurisdiction of an individual agency or entity." This call to action to all IGs builds on a growing movement among individual offices to increase and enhance program evaluation.

The IG community generally has considered evaluation as part of a more encompassing function of inspection: "An inspection is defined as a process that evaluates, reviews, studies, and/or analyzes the programs and activities of a Department/Agency" for a number of purposes. The CIGIE Inspection and Evaluation Committee, which drafted the standards, included among these purposes providing factual information to managers for decisionmaking; monitoring compliance; measuring performance; assessing the efficiency and effectiveness of programs and operations; sharing best practices; identifying where administrative action may be necessary; and making recommendations for improvements to programs, policies, or procedures. To accomplish this, the I&E Committee developed standards dealing with 14 separate matters: competency; independence; professional judgment; quality control; planning; data collection and analysis; evidence; records maintenance; timeliness; fraud, other illegal acts, and abuses; reporting; follow-up; performance measurement; and working relationships and communications.

The Inspection and Evaluation Committee, however, qualifies the adoption of the standards: "While these standards are advisory, and compliance is voluntary, their consistent application is encouraged." The panel adds that "the inspection function at each Department/Agency is tailored to the unique mission of the respective Department/Agency." Although the inspection function did not originate when the IG Act was approved in 1978, it has grown over time in terms of complexity and diversity as well as size; that is, the number of OIG units, budget, and staff dedicated to it. The result is that most—but not all—IG offices conduct inspections; and these I&E units vary in expertise, location, type of work, longevity, staffing, and budget.

Office of Management and Budget

Over the past decade, OMB has also offered guidance on how to achieve independence and expertise in evaluations. Its directions and instructions, however, exhibit some differences in specifications and orientation between the Bush and Obama Administrations.

George W. Bush Administration and PART

Considerations about the independence and orientation of program evaluation were included in the operation of the Program Assessment Rating Tool (PART), which was developed during the Administration of George W. Bush but is no longer in effect. Although the guidance did not define independent evaluation or evaluator expressly, OMB referred to its role in several instances. One instance, about the scope and quality of "independent evaluations," stated,

Purpose: To ensure that the program or agency conducts non-biased evaluations on a regular or as-needed basis to fill gaps in performance information. These evaluations should be of sufficient scope and quality to improve planning with respect to the effectiveness of the program.

OMB added that such independent evaluations were to be of "high quality, sufficient scope, unbiased, independent, and conducted on a regular basis to support program improvements."

As noted previously in this report, however, GAO discussed how disagreements arose in this context over how to define independence for an evaluator and whether independence was necessary or of value in some circumstances. OMB initially advocated for a separate, outside entity—in particular, statutory IGs or GAO—to conduct the evaluation. Agencies balked at this, insisting that experienced program evaluation offices within an agency, for instance, could conduct such evaluations with independence. OMB reconsidered its stand and allowed that "evaluations could be considered independent if the program contracted them out to a third party or they were carried out by an agency's program evaluation office." OMB also recognized a possible conflict between competing analyses: "The program should defend differences if an independent entity's analysis differs from the program's analysis."

Barack Obama Administration

After Barack Obama became President, the Office of Management and Budget issued several memoranda and a guidance dealing with independent program evaluation. OMB Director Peter R. Orszag issued two memoranda on the subject—one in 2009 and another the next year—which began with a nearly identical premise: "Rigorous, independent program evaluations can be a key resource in determining whether government programs are achieving their intended outcomes as well [effectively] as possible and at the lowest possible cost."

By comparison to the PART pronouncements, the Obama OMB assumed a somewhat different stance on how to define independence. In the October 2009 memorandum, OMB concentrated more on the concept of independent evaluation instead of an independent evaluator. Notably, this memorandum made repeated reference to conducting "rigorous, independent studies that are free from political interference." The memorandum did not provide more detail on what constitutes political interference or what this sort of independence might look like. When an agency applied for Administration support in the President's budget for capacity-building funds, however, OMB required agencies to address explicitly their ability to conduct studies that are free from political interference.

In its follow-up memorandum, OMB added to this in its "evaluation initiative for FY 2012," allocating, in the President's upcoming budget proposal, "approximately $100 million to support 35 rigorous program evaluations and evaluation capacity-building proposals across the Federal government." In so doing, the evaluation initiative repeated the earlier call for an "assessment of agency capacity to conduct rigorous, independent evaluations ... that are free from political interference." A complementary goal—which also suggested several ways to institutionalize independent evaluation within an agency—is to

attract and retain talented researchers in an office with standing within the agency. Agencies are encouraged to propose changes or reforms that are needed to meet these objectives and may request funds to strengthen their internal evaluation expertise and processes. Several existing models may be worthy of consideration, including a congressionally chartered institute within an agency, or an office headed by a senior official reporting directly to the Secretary or Deputy Secretary.

The 2009 memorandum had also announced an "increased emphasis" on program evaluations and, more specifically, "impact evaluations." The 2010 memorandum followed suit, recognizing the need to overcome the absence of completed or timely program evaluations for many federal programs. As a result, the Office called on agencies to include funding for FY2011 to conduct evaluations or build capacity, to correct these defects and strengthen program evaluation. Extending this orientation, another OMB memorandum emphasized a change in the objective of evaluations and their measurements. This OMB memorandum, also issued in mid-2010, states that

the Administration is transitioning from a planning and reporting approach focused primarily on the supply of performance information to .... performance improvement strategies .... [with] Unrelenting attention to achieve the ambitious, near-term performance goals that agency leaders identify as High Priority Performance Goals ("Priority Goals") in the President's FY 2011 Budget; Establishing constructive performance review processes in agencies that are sustained over time; and Making Government Performance and Results Act documents useful.

Government Accountability Office Review in 2010

A 2010 report by the Government Accountability Office, which was released before OMB's initiatives in the same year, examined attempts by both the Bush and Obama Administrations to improve government efficiency, including ways it connects to PART and its successor. GAO found that although "most programs developed an efficiency measure," most of these failed to adopt both input and output or outcome measures; and agency officials described "challenges to developing and using program-level efficiency measures and performance measures in general." Nonetheless, "officials for some programs stated that the efficiency measures reported for PART were useful" in a variety of ways, including "to evaluate proposals from field units." The GAO report also recognized improvements coming from both the Bush and Obama Administrations along with the lengthy history of legislative reform efforts that tie efficiency into program effectiveness, particularly in the Government Performance and Results Act.

The two subsequent 2010 OMB memoranda recognized some of the same goals. Its memorandum on Evaluating Programs for Efficacy and Cost-Efficiency (2010), as the title suggests, dealt with the relationship between efficacy and efficiency and the need to measure both. And OMB's Performance Improvement Guidance added to this, specifying that evaluation should focus on "performance improvement strategies" in general and on "making GPRA documents useful" in particular.

Possible Criteria and Attributes for an Independent Evaluation Unit or Function

The aforementioned understandings of terms like "evaluation" and "independence" are not hard-and-fast rules. Rather, they might be considered as perspectives, some of which are similar. Using the perspectives and some of the examples cited later in this report, a number of possible criteria for an independent evaluator can be identified. These characteristics, which are not exhaustive, are intended only to identify a range of possibilities and options to consider for establishing an independent evaluator, modifying evaluative operations and organizations, or reviewing an evaluator's methodology and findings. CRS takes no position on the advisability of independence or the adoption of any particular characteristic.

Possible Criteria Addressing the Independence of an Evaluator

Based on the foregoing analysis and the examples cited later in this report, the following criteria illustrate a range of factors that could affect the independence of an IE-like entity, which, in turn, could help to determine the IE's capacity, capability, credibility, impact, and effectiveness.

Well-defined jurisdiction . Specifying an independent evaluator's jurisdiction—over the subject matter, specific projects or programs, and the agencies to be covered—is especially important with regard to interagency, intergovernmental, and interstate programs. Clear separation from the program or project office . A range of options exist here. For instance, this might occur through an evaluation unit which is external to the office, its parent agency, or both. An outside unit could be established in a number of ways: setting up a new office, operating under standards and guidelines that protect its independence (e.g., prohibiting revision by political appointees or OMB); contracting with a private organization; making arrangements with a relevant government-chartered organization; or relying on an appropriate federal agency, such as an IG, an executive organization with relevant expertise and experience, or GAO. Another possibility involves an office within the agency. The independence of such an evaluation might also be protected if it were assigned to an in-house program evaluation office unconnected with the program or project implementer. Avoidance of a (potential) conflict of interest, actual or perceived . For an evaluator, this might mean, among other considerations, no direct ties to the program or project office, its implementing agents, or affected parties. Neutrality, objectivity, and impartiality in conducting an evaluation as well as in reporting and disseminating its findings, conclusions, and recommendations . Even if an evaluation entity were not housed separately from the program being implemented, third-party review of the evaluation, for example, might detect or deter bias, providing some assurance of independence. In addition, evaluations might be required to focus on several competing definitions of programmatic "success," if the proper goals of a government activity were contested. Appropriate criteria and standards for conducting an evaluation and for basing its conclusions . Such criteria and standards would be determined by the subject matter being assessed as well as by the extent and detail of the coverage called for. This attribute for an independent evaluation would serve to justify the validity and reliability of its findings and conclusions. Specified type(s) of evaluation . GAO identifies these as cost-effectiveness, implementation, impact, and outcome. Refinements of these and other possibilities might be considered as well. Designated responsibilities and duties . These could extend beyond assessing program performance, to developing appropriate measures, providing information and data to operational offices, monitoring compliance with program directives, issuing recommendations for corrective action, sharing best-practices, and responding to and reconciling concerns or criticisms from report subjects. Competency and expertise (in the subject area, in evaluation, or in both) . This applies to both the principal evaluator and staff. Sufficient resources . Resources for the IE or similar posts include funding, staff, and, if necessary, contracting authority, appropriate equipment and facilities, among other items. Access to and use of relevant information and data which are reliable and valid. Program evaluations—and their credibility—rely upon the reliability and validity of the data and information that they use and that are appropriate for the subject. Accounting for and documenting steps in the evaluation process . These extend to the measurements used as well as the selection of information and data relied upon, to defend decisions if necessary. R eport contents and timetables . These include what the reports should contain (findings, conclusions, and/or recommendations) and when they should be submitted (on a fixed time schedule, after certain conditions are met, at different stages of an evaluation, or at the end of the project or program). Report recipients, availability, and dissemination . This involves who is to receive the IE reports (Congress, the President, agency officials, or the general public) and how an IE report is to be made available (in what medium and at what cost, if any). Evaluation re sponses and report revisions . This specifies whether a report may be revised by the head of an agency, OMB, or the President, or whether responses from the evaluation's subjects are required. Responses to a report's findings, conclusions, and recommendations are particularly important from the program or project office. Means to resolve differences between an evaluator and the operational office. Various means might be considered to resolve or, alternatively, transparently report, any differences in the event that the program office disagrees with an evaluator's findings, conclusions, and recommendations.

Possible Attributes When Designing or Reviewing an Independent Evaluation Unit

These criteria also suggest a range of considerations in establishing a new IE entity, or reviewing and modifying an existing one. A number of choices about organizational design and operations might be considered that could support the independence, quality, reliability, and meaningfulness of an IE's studies and reports to a variety of stakeholders, including Congress, the President, agency employees, state and local governments, interest groups, private contractors, and the public. These outcomes are likely to be enhanced by clear and specific requirements regarding the purposes, responsibilities, and duties of the IE entities as well as protections for their officers. Posed as questions, these considerations, which closely relate to one another and in some circumstances may overlap, include the following:

Selection Requirements: Are any requirements or criteria for an IE's selection specified in the authority creating the entity or position? If so, is he or she to be selected for relevant expertise or experience in the subject area or in evaluation? Appointing Authority and Requirements: Who appoints the individual who will lead the IE entity? If the independent evaluator is an individual, who will appoint the individual? Is the appointment made by a certain agency official (head of the agency or of a program office)? Is it an individual officer or a committee (in the case of an intra-agency, interagency, or intergovernmental unit)? If the appointment is made by the chair of a committee, is he or she directed to consult with other panel members? Is the appointing authority advised to or required to consult with inspectors general or with outside organizations in determining prospective candidates? Tenure : How long is an evaluation officer's tenure? Is it confined to a short-term project or a fixed deadline, such as six months or one year? Or does it extend indefinitely, throughout the life of an entire program? Funding: How is the entity funded? Are the funds from an administrative account at the discretion of an agency or program official? Or is the funding a specified amount or percentage of the overall budget of a program or project? Supervision: Who, if anyone, supervises the IE's operations? Purposes: Why exactly has an office been established, and what is it expected to accomplish? Is it to conduct an evaluation over a long-term broad program, a short-term narrow project, or a series of these? Is the unit to offer recommendations, as well as findings and conclusions? What roles is an evaluation unit expected to play in the ongoing operation of the program or project, in advising an overseer or program manager, or in the final assessment of the program? Are the IE's findings, conclusions, and recommendations expected to contribute to management and best-practices approaches, for instance, or to examine the effectiveness of the program itself, separate from its implementation? Substantive Considerations and Type of Evaluation : What substantive research questions is it to address? What type or types of evaluation is the entity to perform? How much discretion will be necessary to leave to the IE to make these decisions? Mandated Studies: What particular studies, if any, are mandated? Is an IE required to conduct certain studies (e.g., interims as well as a final) or does he or she have some discretion over the range and frequency of studies? Study Standards and Procedures : What standards, procedures, and guidelines, if any, is an IE expected to follow in conducting evaluations? Are these detailed in the unit's establishing authority? Consultation with Outside Organizations: Is an IE entity required to consult with other governmental or nongovernmental organizations in carrying out its responsibilities? In general, should the IE consult with private sector specialists in evaluation or with inspectors general; in scientific matters, with one of the relevant National Academies or private organizations; and in public administration matters, as an example, with the National Academy of Public Administration? Types of Reports: What types of reports are ordered? Are the reports limited to particular projects or stages thereof; are their contents outlined; and are the reports to contain recommendations for change? Reporting Schedules: When are the reports to be issued?—according to a fixed time schedule?—at the completion of a particular stage of a project or program?—at the end of the entire program?—or at the discretion of the IE? Report Recipients: Who is to receive the reports? Is it: the operational office?—the agency?—the President?—Congress?—another government entity, such as GAO or an inspector general?—the general public? Report Availability and Dissemination : How is the report to be publicized—by way of the IE's office, the affected agency or program office, a congressional panel which is a recipient, or the news media? How is it to be disseminated? Is it in hard-copy (from the IE or the Government Printing Office, for instance) or on the website of the IE, the affected agency, or another official recipient? Response Obligations : What obligations, if any, does an affected agency or program office have to respond to the IE's findings, conclusions, and possible recommendations? Considering or Resolving Differences: Is there a means to consider and possibly resolve differences on such matters, if disagreements arise between the IE and the program or project implementer? Jurisdictions: What is an IE's jurisdiction? Is it over a narrow, short-term project; a broad, long-range program; or some combination between these ends of the spectrum? Does the jurisdiction cover a single federal office or agency, a number of federal agencies, or a collection of federal, state, and local government entities? Does the jurisdiction cover a domain of public policy? Modes of Operation: What is the mode of operation of an independent evaluation unit? Is it in continuous session, monitoring and assessing a particular program on an on-going basis? Or does it operate sporadically, evaluating, for instance, several distinct projects or a program when it reaches certain stages? Or is it only active when certain conditions or a time schedule calls for its involvement?

Concluding Observations

In the understanding adopted here, truly independent evaluators are not found within the program office itself, because that location would (appear to) compromise their independence. Even with this qualification, however, independent evaluators of federal programs appear to have increased since the 1960s and continue to do so. IEs exist both inside the agency proper—in auditing offices, for instance, or, as a "model" raised by OMB, in a "congressionally chartered institute"—as well as outside it. Independent evaluators and comparable entities have been created as new, separate units, operating under a variety of names and titles. And evaluations have been undertaken by appropriate existing agencies, including the Government Accountability Office, offices of inspector general, or various other government constructs, such as NIST and the National Academy of Sciences.

IEs and similar constructs, moreover, are multi-faceted and diverse, differing in their establishing authority, structure, organization, responsibilities, and requirements, among other characteristics. Such differences have come about for a number of reasons, including the following:

No agreed-upon, precise, and detailed definition of what constitutes "independence" and "evaluation" exists. Suggested or recommended standards and criteria, although having commonalities, are not duplicative in all respects, with some being more detailed and expansive than others. Nonetheless, a number of criteria and attributes could be considered in creating or modifying an independent evaluator. A perceived need for flexibility in IE structure, organization, operations, and activities appears in some establishments, which allows the IE (or other officials) to determine the entity's features. By comparison, others are given detailed duties and directions upon their creation. Demands and expectations differ among IEs. The political context in which IEs are established differs over time and within and among policy arenas. Because of these and other factors, independent evaluators follow no single path or set of directions. Instead, they reveal numerous ways and directives for possible approaches to assess federal programs; provide relevant information and data to the executive, legislature, stakeholders, and the general public; enhance oversight of affected programs; and aid in the development of new legislation or executive directives.

Appendix. Examples of Independent Evaluators and Similar Positions

Following are descriptions of a number of independent evaluation entities, as set up by their establishing authority or proposed. These were selected through a search of public laws, executive orders, other administrative directives, and legislative proposals that specifically call for an "independent evaluation," "independent evaluator," or similar devices, including "peer review" or "independent study." The listing is not comprehensive. For instance, it does not include relevant reports from the Government Accountability Office, because of their substantial number. And it does not include relevant reports from inspectors general, unless these are specifically called for in legislation. Nonetheless, the descriptions of the selections illustrate the diversity of independent evaluation units and their adoption over time, particularly in the contemporary era. (For each entry, the title of the entity uses the lower or upper case as it appears in the authorization or proposal and is in bold type for emphasis.)

Based on these examples, no single entity meets all the characteristics that could be addressed in its establishing authority, although some come close. The selections also demonstrate a wide range and diversity among entities which have been empowered to conduct evaluations. As a corollary, none of the evaluation offices are identical to any other in all respects. Differences are observable in the specifics associated with: appointment, tenure, jurisdiction, evaluation criteria and standards, consultation with outside organizations, reporting mandates, reports' recipients and responses, and funding and staffing, among other characteristics.

Chesapeake Bay Protection and Restoration Independent Evaluator (Executive Order 13508)

The independent evaluator called for in Executive Order 13508, "Chesapeake Bay Protection and Restoration," issued by President Obama on May 12, 2009, is to assist a new Federal Leadership Committee for the Chesapeake Bay in its oversight of the program. The committee, established in the order, comprises representatives from a number of federal departments and agencies; it was established "to begin a new era of shared Federal leadership with respect to the protection and restoration of the Chesapeake Bay." The committee's responsibilities are "to oversee the development and coordination of programs and activities," along with managing "the development of strategies and program plans for the watershed and ecosystem of the Chesapeake Bay and oversee their implementation." Among other things, the committee is to prepare and publish a strategy, following submission of relevant draft reports from lead federal agencies. The overarching strategy is to include, among other things: defining environmental goals; identifying key measurable indicators or environmental condition and changes; describing specific programs and strategies to be implemented; identifying the mechanism to assure that governmental and other activities, including data collection and distribution, are coordinated; and "describing a process for the implementation of adaptive management principles, including a periodic evaluation of protection and restoration activities."

These requirements, directly or indirectly, tie into the functions of an independent evaluator called for in the executive order. In order to "strengthen accountability" over the program,

[t]he Committee, in collaboration with State agencies, shall ensure that an independent evaluator periodically reports to the Committee on progress toward meeting the goals of this order. The Committee shall ensure that all program evaluation reports, including data on practice or system implementation and maintenance funded through agency programs, as appropriate, are made available to the public by posting on a website maintained by the Chair of the Committee.

Key characteristics of the Chesapeake Bay Restoration independent evaluator position—based on this provision and portions of the Federal Leadership Committee's mandate—are:

an interagency and inter-governmental jurisdiction, commensurate with the committee's jurisdiction; an indefinite tenure based on the life of the committee; reporting requirements that authorize the IE to report "periodically ... on progress in meeting the goals of this order" to the committee; and a directive to the committee to make these reports and relevant data available to the public.

Proposed Chesapeake Clean Water and Ecosystem Restoration Act of 2009 Inspector General and Chesapeake Bay Program Scientific and Technical Advisory Committee

This bill ( S. 1816 , 111 th Congress), which focuses on restoration of Chesapeake Bay, provides for the Inspector General of the Environmental Protection Agency to evaluate implementation of the enactment and for a special Scientific and Technical Advisory Committee . The legislation is intended to expedite and enhance the restoration of the Chesapeake water and ecosystem. It would reauthorize the Chesapeake Bay Program, expand state and local government authority, provide new grant authorizations, and strengthen enforcement tools. In so doing, it would augment existing legislation (the Federal Water Pollution Control Act); President Obama's Executive Order 13508 (described above); and the on-going Chesapeake Bay Program, which has been a federal, state, and local government charge since the mid-1980s. The Chesapeake Executive Council would direct the Program, in accordance with a Chesapeake Bay Agreement. The Council would develop and implement management strategies and plans for the Program. The operational Office of the Council would remain in EPA, providing support for it in a number of ways. These include implementing and coordinating science, research, monitoring, and data collection; developing and maintaining pertinent information; assisting the signatories of the Agreement in developing and implementing action plans; coordinating the activities of EPA and other entities in developing strategies to improve water quality and living resources of the Bay; and implementing outreach programs for public information, education, and participation to foster stewardship of the bay. Along with this, the EPA Administrator would be authorized to enter into agreements with other federal agencies to carry out these matters. If enacted, S. 1816 would provide for various grants to public and private entities, for technical and other assistance as well as implementation and monitoring.

In accordance with E.O. 13508, S. 1816 would also require the EPA Administrator to issue an annual action plan—describing how federal funding in the President's budget submission to Congress would be used to protect and restore the Chesapeake Bay—and biennial progress reports assessing relevant developments and indicators.

The bill also provides that the " Inspector General of the Environmental Protection Agency shall evaluate the implementation of this section on a periodic basis of not less than once every three years." No other particulars about such an evaluation are specified in the proposal. The legislation would also require the EPA Administrator to consult with several different entities about particular matters. One of these directives is "to consult with the Chesapeake Bay Program Scientific and Technical Advisory Committee regarding independent review of [certain] monitoring designs."

Proposed Chesapeake Bay Accountability and Recovery Act of 2009 Independent Evaluator

This proposed legislation ( H.R. 1053 , 111 th Congress) parallels some of the main provisions in E.O. 13508, both of which deal with the Chesapeake Bay restoration and include an Independent Evaluator . The House bill, as approved by the House Committee on Natural Resources, calls for an "interagency crosscut budget" for restoration activities in the Chesapeake Bay watershed, with funding, expenditures, and accounting requirements detailed (Sec. 1). This crosscut budget, composed by the various agencies involved in the restoration program, is to be submitted to Congress by the Director of the Office of Management and Budget, after consultation with the Chesapeake Bay Executive Council, the chief executive of each Chesapeake Bay State (the definition of which includes the District of Columbia), and the Chesapeake Bay Commission. In addition, H.R. 1053 (Sec. 3) requires that the Administrator of the Environmental Protection Agency (EPA), "in consultation with other Federal and State agencies, shall develop an adaptive management plan for the Chesapeake Bay Program and restoration activities that includes," among other matters: a definition of specific and measurable objectives to improve water quality, habitat, and fisheries; a process for stakeholder participation; monitoring, modeling, experimentation, and other research and evaluation practices; a process for modifying restoration priorities that have not attained or will not attain the aforementioned specific and measureable objectives; and a process for prioritizing restoration activities and programs to which adaptive management is to be applied. The EPA Administrator is also required to submit annual reports on the implementation of the adaptive management plan.

The proposal adds an Independent Evaluator , appointed by the Administrator from among nominees submitted by the Chesapeake Executive Council. The IE is to "review and report on restoration activities and the uses of adaptive management, including on such related topics as are suggested" by the Council (Sec. 4). The IE "shall submit a report to the Congress every 3 years in the findings and recommendations of reviews" with regard to the adaptive management plans and their implementation (Sec. 4). As noted above, the plans' inclusions are: developing measurable objectives; monitoring, modeling, experimentation, and other research and evaluation practices; and processes for modifying restoration activities as well as prioritizing restoration activities and programs.

Comprehensive Everglades Restoration Plan Independent Scientific Review Panel and Independent Peer Review

The Independent Scientific Review Panel , established by a collective of interdepartmental and inter-governmental offices, is to be "convened by a body, such as the National Academy of Sciences, to review the plan's progress toward achieving the natural system restoration goals of the Plan." The panel is to produce a biennial report to Congress and members of the establishing body, that includes an assessment of the ecological indicators and other measures of progress in restoring the ecology of the natural system, based on the plan.

In addition, the enactment calls for an independent peer review of methods for project analysis . The statute charges the Secretary of the Army to contract with the National Academy of Sciences (with a specified authorization of appropriations), to conduct a study that includes reviews of various methods used and a comparative evaluation of the basis and validity of relevant state-of-the-art methods. The report—to be issued not later than one year after the date of the contract and submitted to the Secretary and to specified congressional committees—is to include the results of the study and specific recommendations for modifying any methods being used in the project for conducting economic and environmental analyses of water resource projects.

Prince William Sound (Alaska) Oil Spill Recovery Institute and Advisory Board

The Oil Spill Recovery I nstitute , established by the Secretary of Commerce, is to conduct research and carry out educational and demonstration projects designed to develop the best available techniques, equipment, and materials dealing with oil spills. In addition, the institute is to complement federal and state damage assessment efforts as well as determine, document, assess, and understand the long-range effects of relevant oil spills. The institute is to publish and make available to any person the results of its research, educational, and demonstration projects; copies of all such materials are to be provided to the National Oceanic and Atmospheric Administration.

The policies of the institute are determined by an Advisory Board, which also appoints the institute's chair. The Advisory Board, itself chaired by the Secretary of Commerce, consists of federal and state officials, representatives of relevant industries, representatives of Alaska Natives, and residents of local communities. The board may also request a scientific review of a research program it authorized every five years, to be conducted by the National Academy of Sciences. The board shall establish a Scientific and Technical Committee, composed of relevant specialists, to advise and make recommendations to the board regarding the conduct and support of research, projects, and studies.

Oil Pollution Research Interagency Coordinating Committee

The Oil Pollution Research Interagency Coordinating Committee includes representatives from a number of federal agencies. It is chaired by the representative of the United States Coast Guard and operates under a specified budget. The committee is to report to Congress on a plan dealing with oil pollution research, development, and demonstration program. At this initial stage, the enactment requires consultation with affected states on certain matters and contracting with the National Academy of Sciences for advice and guidance on preparing the plan and assessing its adequacy. Subsequently, the committee is to coordinate the establishment of a program for conducting oil pollution research and development, which includes requirements for monitoring and evaluating relevant aspects of the plan. The committee is also authorized to make recommendations for grants to the private sector to an appropriate granting agency represented on the committee; its recommendations are to ensure an appropriate balance within a region among various aspects of oil pollution research. The chairperson of the committee is to report to Congress every two years on its activities and on activities proposed and carried out under the act.

Water Resources Development Independent Peer Review Panels

Various project studies authorized by the act are subject to peer review by independent panels of experts . Some reviews are mandatory, while others are discretionary; the latter include ones requested by a governor of an affected state or certain federal agencies. For each project subject to a peer review, the Chief of Engineers of the Army Corps of Engineers is to contract with the National Academy of Sciences or a similar independent scientific and technical advisory organization to establish a panel of experts, who are to represent a balance of areas of expertise suitable for the review and who have no conflict or direct involvement with the project being reviewed. Specifications for the peer review include assessing the adequacy and acceptability of economic, engineering, and environmental methods, models, and analyses.

Certain committees of Congress are to be notified of a project study for peer review prior to its initiation. In addition, the review panels are to submit a report to the Chief of Engineers not more than 60 days after the last day of public comment for a draft project study or later, if the chief determines an extension is necessary. The panels are also directed to submit to the chief a final report containing certain analyses. They may make recommendations, to which the chief shall issue a written response for any recommendation adopted or not adopted. Copies of the reports and the chief's responses are to be made available to the public by electronic means and transmitted to certain congressional committees. The enactment authorizes federal funds to be used for each panel; it imposes a ceiling of $500,000, which can be waived by the chief if he or she determines the waiver to be appropriate.

American Recovery and Reinvestment Act (ARRA) of 2009 Recovery Accountability and Transparency Board and Independent Advisory Panel

ARRA, sometimes referred to as the Recovery Act, provides for a number of oversight mechanisms. One is the Recovery Accountability and Transparency Board , established "to coordinate and conduct oversight of covered funds to prevent fraud, waste, and abuse." The board's membership includes ten specified inspectors general and "any other Inspector General designated by the President from any agency that expends or obligates covered funds." The President is authorized to appoint the chairperson of the board, to be selected from among three officers. The three are: the OMB deputy director for management, a federal officer who already has been confirmed by the Senate for another position, or an individual who would have to be confirmed by the Senate. (The President selected the sitting Inspector General of the Department of the Interior (a confirmed position) to be chairperson.)

The B oard may request information and assistance from any agency or entity of the federal government. Supplementing this, the Board is given the same powers of inspectors general operating under the IG Act of 1978, as amended. It may conduct public hearings; enter into contracts as may be necessary to carry out its duties; and transfer funds for expenses to support administrative services and audits, reviews, or other oversight activities.

The B oard is further empowered to "conduct its own independent audits and reviews relating to covered funds and collaborate on audits and reviews" with any inspector general. In addition, the board's detailed audit and review functions pertain to the following: covered funds for relevant purposes; the reporting and competition requirements associated with contracts and grants to determine whether these meet applicable standards; the adequacy and training of personnel to oversee the covered funds; and the extent of appropriate mechanisms for interagency collaboration, including coordinating and collaborating with the Council of the Inspectors General on Integrity and Efficiency. ARRA also directs the board to make recommendations to agencies with regard to preventing waste, fraud, and abuse of the covered funds. An agency must respond within 30 days of the receipt of the recommendations, stating whether it agrees or disagrees and whether any actions will be taken to implement them.

The B oard is also authorized to issue a variety of reports to the President and Congress. These include "flash reports" on potential management and funding problems that require immediate attention; quarterly reports summarizing the findings of the Board and inspectors general of agencies; annual reports consolidating the quarterly reports; and such other reports that the Board deems appropriate. In general, all of these reports are to be made available to the public on the Board's website. The website, which is to be "user-friendly" and "public-facing," is to provide links to related websites as well as materials and information regarding the act itself; accountability matters, including findings of the Board's and IGs' audits; data on contracts and grants relating to covered funds; the use and allocation of covered funds by each federal agency; data on relevant economic, financial, grant, and contract information to enhance public awareness of the use of the covered funds; and to the extent practical, job opportunities afforded by the program and how to access these on appropriate federal, state, and local websites and locations.

Supplementing the Board is a Recovery Independent Advisory Panel . Also established by ARRA, it is composed of five members appointed by the President, based on relevant expertise. The panel is to "make recommendations to the Board on actions the Board could take to prevent waste, fraud, and abuse relating to covered funds." The panel is authorized to hold hearings and to "secure directly from any agency such information as the Panel considers necessary to carry out" its duties.

Federal Information Security Management Act (FISMA) Independent Evaluations

FISMA directs that "each year, each agency shall have performed an independent evaluation of the information security program and practices of that agency to determine the effectiveness of such program and practices." The Director of OMB—except for "national security systems"—is to oversee the development of the relevant information security policies, principles, standards, and guidelines; review and (dis)approve agency security programs; and take certain actions to ensure compliance with relevant policies, standards, and principles. Under the statute, the Director—based on guidelines and standards developed by the National Institute of Standards and Technology (NIST)—is specifically authorized to issue information security guidelines and standards, including minimum standards, which "shall be compulsory and binding."

The statute further stipulates that the "annual evaluation required by this section shall be performed by the Inspector General or an external independent auditor ," as determined by the Inspector General of the agency" or by the head of the agency if an IG has not been established by law. Such an independent evaluation is to include (1) a testing of the effectiveness of information security control techniques and (2) an assessment (made on the basis of the results of the testing) of compliance with the requirements. These annual reviews are to include the evaluator's findings and recommendations, which are communicated to the agency head; he or she then submits the results of the reviews to the Director of OMB, who, in turn, "shall summarize the results of the evaluations" and report these to Congress.

Department of Defense Independent Study on Post-Traumatic Stress Disorder Efforts

Under the National Defense Authorization Act for FY2010, the Secretary of Defense, in consultation with the Secretary of Veterans Affairs, is to provide for an independent study on the treatment of post-traumatic stress disorder. The study is to be "conducted by the Institute of Medicine of the National Academy of Sciences or such other independent entity as the Secretary shall select for purposes of the study."

The legislation also spells out certain requirements of the study. These include a listing of each operative program and method available for the prevention, screening, diagnosis, treatment, or rehabilitation of the disorder; the status of studies and clinical trials involving innovative treatments; a description of each treatment program and a comparison of methods among them, at specified locations; current and projected future annual expenditures by DOD and VA in this matter; and a description of gender-specific and racial and ethic group-specific mental health treatment and services available for members of the Armed Forces. The entity conducting the study is to submit an initial report, due on July 1, 2012, to the Secretaries of Defense and of Veterans Affairs and appropriate congressional committees (specifically, the House and Senate Committees on Armed Services, Appropriations, and Veterans' Affairs; the House Committee on Energy and Commerce, and the Senate Committee on Health, Education, and Welfare). Responses by the Secretaries to the report—including any recommendations for on the treatment of the disorder based on the report—are required six months later. An updated report and responses are to be submitted by July 1, 2014, and January 1, 2015, respectively.

Defense Science Board Independent Assessment of Improvements in Service Contracting

The National Defense Authorization Act for FY2010 provided that "the Under Secretary of Defense for Acquisition, Technology, and Logistics shall direct the Defense Science Board [DSB] to conduct an independent assessment of improvements in the procurement and oversight of services by the Department of Defense." The assessment is to cover the quality and completeness of guidance relating to the procurement of services, the extent to which best practices are being developed for setting requirements, the contracting approaches and types used for the procurement of services, and whether effective standards to measure performance have been developed. The Under Secretary, who receives the DSB study, is "to submit to the congressional defense committees a report on the results of the assessment, including such comments and recommendations as the Under Secretary considers appropriate," by a specified date (March 10, 2010, following the law's enactment on October 28, 2009).

National Academy of Sciences Review of National Security Laboratories

The National Defense Authorization Act for FY2010 also directs the Secretary of Energy to "enter into an agreement with the National Academy of Sciences to conduct a study of" three specified laboratories. The study is to include for each laboratory an evaluation: of the quality of the scientific research and the engineering being conducted there, of the criteria used to assess the scientific research and engineering, of the relationship between the quality of the science and engineering and the contract for managing and operating the laboratory, and of the management of work conducted by the laboratory for entities other than the Department of Energy. The NAS is "to submit to the Secretary of Energy a report containing the results of the study and any recommendations resulting from the study." By a specified date (January 1, 2011, following the October 28, 2009, enactment), the Secretary of Energy is to submit to the appropriate committees of Congress the NAS report and "any comments or recommendations of the Secretary with respect to that report."

Federal Transit Administration State Safety Oversight Program for Rail Transit

The Federal Transit Administration (FTA), a part of the Department of Transportation (DOT), oversees the safety and security of rail transit agencies which operate "rail fixed guideway mass transportation systems" (e.g., metrorail and subways) and which receive federal funds. Part of the process includes an oversight body for each jurisdiction. The program is designed, according to a GAO summary, "as one in which FTA, other federal agencies, states, and rail transit agencies collaborate to ensure the safety and security of rail transit systems." In most cases, each rail transit agency exists within a state, although a few extend to multi-state and other jurisdictions in such locales as the Washington, DC metropolitan area (which includes the District of Columbia, Maryland, and Virginia). Under federal law, each state or covered area

designates a State authority as having responsibility—(A) to require, review, approve, and monitor the carrying out of each safety plan; (B) to investigate hazardous conditions and accidents on the systems; and (C) to require corrective action to correct or eliminate those conditions (49 U.S.C. 5330(c)).

As an enforcement mechanism, the DOT Secretary is authorized to withhold funds if a state does not comply with these responsibilities.

The FTA program requires each state or other appropriate jurisdictions to establish an oversight body to oversee the safety and security of its system. These oversight units are responsible for developing a program standard that transit agencies must meet and for reviewing performance of the transit agencies against that standard. The oversight agency safety and security review lays out the process and criteria to be used, at least every three years, in conducting a complete review of each affected rail transit agency's implementation of the plan.

The FTA rail safety oversight program, operating through state and local oversight entities, involves

several federal agencies, including components in the Department of Homeland Security and others in DOT; jurisdiction over states and metropolitan areas and agencies; oversight units that are separate from the transit operating authorities; program implementation reviews and assessments; and time schedules for reporting.

A 2006 GAO report, however, found deficiencies in the system, including a failure to meet the three-year review schedule, to develop performance goals for the program, and to be able to track performance. In addition, GAO reported that expertise varied across the oversight entities, which, in some cases, suffered from inadequacies in funding and qualified staff.

Reinforcement of some of these and other findings came about after a serious accident on the Washington Metropolitan Area Transit Authority (WMATA) Metrorail system in mid-2009, which revealed short-comings of its Tri-State Oversight Committee (TOC). The Committee, for instance, lacked its own e-mail address, mailing address, telephone number; more importantly, it had limited resources, staff skills, and expertise. Additional problems included ineffective and inadequate communications with the WMATA and the absence of a process to evaluate corrective action plans and alternatives to TOC proposals. These defects and deficiencies, in turn, generated plans from the administration and proposals from legislators to correct them, not only in the TOC but throughout the federal transit safety system.

Forensic Science Study Conducted by an Independent Forensic Science Committee Established by the National Academy of Sciences

The Science, State, Justice, Commerce, and Related Agencies Appropriations Act of 2006 authorized the "National Academy of Sciences [NAS] to conduct a study of forensic science, as described in the Senate report," accompanying the legislation. The Senate report, which authorized NAS to establish an "independent Forensic Science Committee," recognized that

there exists little to no analysis of the remaining needs of the [criminal justice and law enforcement] community outside of the area of DNA. Therefore .... the Committee directs the Attorney General to provide [funds] to the National Academy of Sciences to create an independent Forensic Science Committee. This Committee shall include members of the forensics community representing operational crime laboratories, medical examiners, and coroners; legal experts; and other scientists deemed appropriate.

The Senate report then set forth a series of charges to the Committee. These included assess the present and future resource needs of the forensic science community, make recommendations for maximizing the use of forensic science techniques, identify potential scientific advances, make recommendations for programs that will increase the number of qualified forensic scientists and others related professions, disseminate best practices and guidelines concerning the collection and analysis of forensic evidence, and examine the role of forensic science in homeland security.

The NAS follow-up study reported on the members of the Committee; participants in hearings and meetings; literature reviewed; issues covered; and findings, conclusions, and recommendations.

Improving America's Schools Act of 1994 Independent Evaluation

This enactment establishes or expands on a number of educational programs, some of which are required to undergo an independent evaluation . The portion of the act dealing with the "Even Start Family Literacy Programs," for instance, directs the Secretary of Education to

provide for an independent evaluation of programs assisted under this part—(1) to determine the performance and effectiveness of programs assisted under this part; and (2) identify effective Even Start programs assisted under this part that can be duplicated and used in providing technical assistance to Federal, State, and local programs (Sec. 1209).

No Child Left Behind (NCLB) Act of 2001 Independent Evaluations, Reviews, and Studies, as well as Peer Reviews

The NCLB Act includes numerous provisions for program evaluations, often done by state agencies involved in the programs, as well as for various independent evaluations, reviews, and studies. In some of these, the Secretary of Education:

is allocated a specified funding amount, to "conduct an independent evaluation of the effectiveness" of the Early Reading Program, which specifies the contents of an interim and a final report (Sec. 1226). is allocated certain funds to provide for an independent evaluation of Even Start programs for several designated purposes: to determine the performance and effectiveness of them; to identify effective programs that can be duplicated and used in providing technical assistance to federal, state, and local programs; and to provide state educational agencies and relevant entities with technical assistance to ensure that local evaluations provide accurate information on the effectiveness of programs (Sec. 1239). is to conduct a national assessment of Title I (Improving the Academic Achievement of the Disadvantaged), adhering to an extensive set of criteria and standards, with the assistance of an independent review panel . It is composed of various specialists, education practitioners, parents and school board members, and technical experts, with a requirement to ensure diversity among the groups; the panel is to consult and advise the Secretary on methodological and other issues, including adherence to the highest standards of quality with respect to research design, statistical analysis, and the dissemination of the findings, and on the use of valid and reliable measures to document program implementation. A final report is to be reviewed by two independent experts in program evaluation, who may come from the review panel; they are to evaluate and comment on the degree to which the report meets the criteria and standards set forth and their comments are to be transmitted with the report (Sec. 1501(d)). is to conduct an independent study of assessments used for state accountability purposes and for making decisions about the promotion and graduation of students. Components of the study, not to last more than five years, are identified; and its purposes are spelled out: to synthesize and analyze existing research, evaluate academic assessment and accountability systems in State and local educational agencies and schools; and make recommendations to the Department of Education and specified congressional committees. The Secretary is authorized to award a contract for the study, through a peer review process , to an organization or entity capable of conducting rigorous, independent research (Sec. 1503). And is directed to use a peer review process to review whether a state has failed to make adequate yearly progress for a specified program (Sec. 6162), and may do so to review certain applications (Sec. 7142).

Agricultural Research, Extension, and Education Reform Act of 1998 Cooperative State Research, Education, and Extension Service Peer and Merit Reviews and Advisory Board Review

This 1998 act establishes several review mechanisms connected with various grant programs. One calls for the Secretary of the Department of Agriculture (USDA) to "establish procedures that provide for scientific peer review of each agricultural research grant administered on a competitive basis," by the USDA Cooperative Extension Service. These procedures are to include "a review panel [which] shall verify, at least every 5 years, that each research activity of the Department and research conducted under each research program of the Department has scientific merit." The enactment directs the review panel to consider "the scientific merit and relevance of the activity or research .... and the multistate significance of the activity or research." To help ensure its expertise and independence, the panel is to be "composed of individuals with scientific expertise, a majority of whom are not employees of the agency whose research is being reviewed," and who, to the maximum extent practicable, are to be selected from colleges and universities.

As part of the consideration of the scientific merit for each research activity, the enactment sets up procedures for a "merit review of each agricultural extension or education grant administered on a competitive basis," by the Cooperative Extension Service. To be eligible for a grant, relevant institutions are to "establish a process for merit review of the activity and review the activity in accordance with the process."

The results of the panel reviews are to be submitted to an Advisory Board , which is to review annually the relevance of priorities for such programs and adequacy of funding for them. The Secretary is required to consider the results of the Advisory Board's review "when formulating each request for proposals, and evaluating proposals" involving such programs.

Fund for the Improvement of Post-Secondary Education (FIPSE) Independent Outside Evaluator

FIPSE, operating under a project director, provides grants connected with relevant projects. An independent outside evaluator (IOE) is called for in the legislation. Hired by the project director, the IOE "must be someone who does not stand to gain personally or professionally from the project results." According to the department's guidance, the IOE is to attend the project director's meeting on evaluating a FIPSE grant and "to assist the Director in completing the initial evaluation plan/chart, due three months after the start of the grant." The IOE is also to assist the director in a number of matters, including offering advice about: which project objectives would lend themselves to measurement and evaluation, which baseline data should be collected, which measurement instruments could be used, what data might be collected from a possible comparison or control group, and how the director might disseminate evaluation results to interested parties. The independent outside evaluator is also to assist the project director in designing the evaluation instruments and to write the evaluation reports.

The Carl D. Perkins Vocational and Technical Education Assistance Act Independent Advisory Panel and Independent Evaluation

This legislation creates an Independent Advisory Panel (IAP) —appointed by the Secretary of Education and consisting of representatives from a broad range of experts and affected parties—to advise him or her on the implementation and assessment of the programs authorized by the act. The Secretary is also authorized to collect relevant information from states and localities which can be used in an evaluation. To assist in this, the Secretary "shall provide for the conduct of an independent evaluation and assessment of vocational and technical programs under this Act through studies and analyses conducted independently through grants, contracts, and cooperative agreements that are awarded on a competitive basis." The enactment also details the contents of such an assessment. These extend to: the efforts and effects of state, local, and tribal entities on such programs; impact of federal expenditures that address program improvements in relevant educational programs; preparation and qualifications of teachers in the fields; academic and employment outcomes of the education; employer involvement and satisfaction with such educational programs; use and impact of educational technology and "distance learning" in the field; and effect of state adjusted levels of performance and state levels of performance on the delivery of relevant services.

The Secretary is also to submit to specified congressional committees an interim report on the assessment and a final report summarizing all studies and analyses related to the assessment. These reports "shall not be subject to any review outside the Department of Education" prior to their submission to Congress. But "the President, Secretary, and advisory panel .... may make such additional recommendations to Congress with respect to the assessment" as each determines appropriate.

Occupational Information To Be Collected by the Social Security Administration (SSA) Independent Evaluation

Under this highly specialized, narrowly focused request, the SSA sought the services of an "Independent Evaluator (IE)" regarding "pre-award and post-award evaluations of occupational information and methodology employed by private sector entities." The IE was expected to help revise and update existing coverage published by the Department of Labor. Both pre-award and post-award evaluation services and requirements are detailed in the SSA request itself and in its attachments.