The Bush Administration's Program Assessment Rating Tool (PART)
Released: 2004-11-05T00:00:00

Introduction

Federal government agencies and programs work to accomplish widely varying missions. These agencies and programs use a number of public policy approaches, including federal spending,tax laws, tax expenditures, (1) and regulation. (2) In FY2004, estimated federal spending was $2.3 trillion,and taxexpenditures totaled approximately $1 trillion. Estimates of the off-budget costs of federalregulations have ranged in the hundreds of billions of dollars, and corresponding estimates ofbenefits of federal regulations have ranged from the hundreds of billions to trillions of dollars. (3)

Given the scope and complexity of these various efforts, it is understandable that citizens,their elected representatives, civil servants, and the public at large would have an interest in theperformance and results of government activities. Evaluating the performance of governmentagencies and programs, however, has proven difficult and often controversial:

Actors in the U.S. political system (e.g., Members of Congress, the President,citizens, interest groups) often disagree about the appropriate uses of public funds; missions, goals,and objectives for public programs; and criteria for evaluating success. One person's key programmay be another person's key example of waste and abuse, and different people have differentconceptions of what "good performance" means.

Even when consensus is reached on a program's appropriate goals andevaluation criteria, it is often difficult and sometimes almost impossible to separate the discreteinfluence that a federal program had on key outcomes from the influence of other actors (e.g., stateand local governments), trends (e.g., globalization, demographic changes), and events (e.g., naturaldisasters).

Federal agencies and programs often have multiple purposes, and sometimesthese purposes may conflict or be in tension with one another. Finding and assessing a balanceamong priorities can be controversial and difficult.

The outcomes of some agencies and programs are viewed by many observersas inherently difficult to measure. Foreign policy and research and development programs have beencited as examples.

There is frequently a time lag between an agency's or program's actions andeventual results (or lack thereof). In the absence of this eventual outcome data, it is often difficultto know how to assess if a program is succeeding.

Many observers have asserted that agencies do not adequately evaluate theperformance or results of their programs -- or integrate evaluation efforts across agency boundaries-- possibly due to lack of capacity, management attention and commitment, or resources. (4)

In spite of these and other challenges, (5) in the last 50 years both Congress and the President haveundertaken numerous efforts -- sometimes referred to as performance management, performancebudgeting, strategic planning, or program evaluation -- to analyze and manage the federalgovernment's performance. Many of those initiatives attempted in varying ways to use performanceinformation to influence budget and management decisions for agencies and programs. (6) The Bush Administration'srelease of PART ratings along with the President's FY2004 and FY2005 budget proposals, and itsplans to continue doing so for FY2006 and subsequent years, represent the latest of these efforts.

Development and Use of the PART

OMB on the PART's Purpose

The PART was created by OMB within the context of the Bush Administration's broaderBudget and Performance Integration (BPI) initiative, one of five government-wide initiatives underthe President's Management Agenda (PMA). (7) According to the President's proposed FY2005 budget, the goal ofthe BPI initiative is to "have the Congress and the Executive Branch routinely consider performanceinformation, among other factors, when making management and funding decisions." (8) In turn,

[the PART] is designed to help assess the managementand performance of individual programs. The PART helps evaluate a program's purpose, design,planning, management, results, and accountability to determine its ultimate effectiveness. (9)

The PART evaluates executive branch programs that have funding associated with them. (10) The Bush Administrationsubmitted approximately 400 PART scores and analyses along with the President's FY2004 andFY2005 budget proposals, (11) with the intent to assess programs amounting to approximately20% of the federal budget each fiscal year for five years, from FY2004 to FY2008. For FY2004,OMB assessed 234 programs. For FY2005, a further 173 programs were assessed. (12) For these two yearscombined, OMB said that about 40% of the federal budget, or nearly $1.1 trillion, had been"PARTed."

In releasing the PART, the Bush Administration asserted that Congress's current statutoryframework for executive branch strategic planning and performance reporting, the GovernmentPerformance and Results Act of 1993 (GPRA),

[w]hile well-intentioned ... did not meet its objectives. Through the President's Budget and Performance Integration initiative, augmented by the PART, theAdministration will strive to implement the objectives of GPRA. (13)

As discussed later in this report, this move and the PART's perceived lack of integration with GPRAwas controversial among some observers, in part because OMB, and by extension the BushAdministration, were seen as "substituting [their] judgment" about agency strategic planning andprogram evaluations "for a wide range of stakeholder interests" under the framework established byCongress under GPRA. (14) Under GPRA, 5 U.S.C. Â§ 306 requires an agency whendeveloping its strategic plan (15) to consult with Congress and "solicit and consider the views andsuggestions of those entities potentially affected by or interested in such a plan." Some observershave recommended a stronger integration between PART and GPRA, thereby more stronglyintegrating executive and congressional management reform efforts. (16)

PART Components and Process

OMB developed seven versions of the PART questionnaire for different types ofprograms. (17) Structurally, each version of the PART has approximately 30 questions that are divided into foursections. Depending on how the questionnaire is filled in and evaluated, each section provides apercentage "effectiveness" rating (e.g., 85%). The four sections are then averaged to create a singlePART score according to the following weights: (1) program purpose and design, 20%; (2) strategicplanning, 10%; (3) program management, 20%; and (4) program results/accountability, 50%.

Under the overall supervision of OMB and agency political appointees, OMB's programexaminers and agency staff negotiate and complete the questionnaire for each "program" -- therebydetermining a program's section and overall PART scores. In the event of disagreements betweenOMB and agencies regarding PART assessments, OMB's PART instructions for FY2005 stated that"[a]greements on PART scoring should be reached in a manner consistent with settling appeals onbudget matters." (18) Under that process, scores are ultimately decided or approved by OMB political appointees and theWhite House. When the PART questionnaire responses are completed, agency and OMB staffprepare materials for inclusion in the President's annual budget proposal to Congress.

According to OMB's most recent guidance to agencies for the PART, the definition of program will most often be determined by a budgetary perspective. That is, the "program" thatOMB assesses with the PART will most often be what OMB calls a program activity , or aggregationof program activities , as listed in the President's budget proposal:

One feature of the PART process is flexibility for OMBand agencies to determine the unit of analysis -- "program" -- for PART review. The structure thatis readily available for this purpose is the formal budget structure of accounts and activitiessupporting budget development for the Executive Branch and the Congress and, in particular,Congressional appropriations committees.... Although the budget structure is not perfect for programreview in every instance -- for example, "program activities" in the budget are not always theactivities that are managed as a program in practice -- the budget structure is the most readilyavailable and comprehensive system for conveying PART results transparently to interested partiesthroughout the Executive and Legislative Branches, as well as to the public at large. (19)

The term program activity is essentially defined by OMB's Circular No. A-11 as the activities andprojects financed by a budget account (or a distinct subset of the activities and projects financed bya budget account), as those activities are outlined in the President's annual budget proposal. (20) As noted later, thisbudget-centered approach has been criticized by some observers, because this budget perspective didnot necessarily match an agency's organization or strategic planning.

Publication and Presentation

For each program that has been assessed, OMB develops a one-page "Program Summary"that is publicly available in electronic PDF format. (21) Each summary displays four separate scores, as determined byOMB, for the PART's four sections. OMB also made available for each program a detailed PART"worksheet" to briefly show how each question and section of the questionnaire was filled in,evaluated, and scored. (22)

OMB states that the numeric scores for each section are used to generate an overalleffectiveness rating for each "program":

[The section scores] are then combined to achieve anoverall qualitative rating of either Effective, Moderately Effective, Adequate, or Ineffective. Programs that do not have acceptable performance measures or have not yet collected performancedata generally receive a rating of Results Not Demonstrated. (23)

The PART's overall "qualitative" rating is ultimately driven by a single numerical score. However,none of OMB's FY2005 budget materials, one-page program summaries, or detailed worksheetsdisplays a program's overall numeric score according to OMB's PART assessment. OMB stated thatit does not publish these single numerical scores, because "numerical scores are not so precise as tobe able to reliably compare differences of a few points among different programs.... [Overall scores]are rather used as a guide to determine qualitative ratings that are more generally comparable acrossprograms." (24) However,these composite weighted scores can be computed manually using OMB's weighting formula. (25)

The only PART effectiveness rating that OMB defines explicitly is "Results NotDemonstrated," as shown by the excerpt above. (26) The Government Accountability Office (GAO, formerly theGeneral Accounting Office) has stated that "[i]t is important for users of the PART information tointerpret the 'results not demonstrated' designation as 'unknown effectiveness' rather than as meaningthe program is 'ineffective.'" (27) The other four ratings, which are graduated from best to worst,are driven directly by each program's overall quantitative score, as outlined in the following table.

Table 1. OMB Rating Categories for thePART

Source : OMB's website, at http://www.whitehouse.gov/omb/part/2004_faq.html , "PARTFrequently Asked Question" #29. This website appears to be the only publicly available locationwhere OMB indicates how OMB translated numerical scores into overall "qualitative" ratings.

OMB Statements About the PART

Transparency and Objectivity

OMB has stated that it wants to make the PART process and scores transparent, consistent,systematic, and objective. To that end, OMB solicited and received feedback and informalcomments from agencies, congressional staff, GAO, and "outside experts" on ways to change theinstrument before it was published with the President's FY2004 budget proposal in February2003. (28) In an effortto increase transparency, for example, OMB made the detailed PART worksheets available for eachprogram. To make PART assessments more consistent, OMB subjected its assessments to aconsistency check. (29) That review was "examined," in turn, by the National Academy of Public Administration(NAPA). (30) To makethe PART more systematic, OMB established formal criteria for assessing programs and created aninstrument that differentiated among the seven types of programs (e.g., credit programs, research anddevelopment programs).

With regard to the goal of achieving objectivity, OMB made changes to the draft PARTbefore its release in February 2003 with the President's budget. For example, OMB eliminated adraft PART question on whether a program was appropriate at the federal level, because OMB foundthat question "was too subjective and [assessments] could vary depending on philosophical orpolitical viewpoints." (31) However, OMB went further to state:

While subjectivity can be minimized, it can never becompletely eliminated regardless of the method or tool. In providing advice to OMB Directors,OMB staff have always exercised professional judgment with some degree of subjectivity. That willnot change.... [T]he PART makes public and transparent the questions OMB asks in advance ofmaking judgments, and opens up any subjectivity in that process for discussion and debate. (32)

OMB career staff are not necessarily the only potential sources for subjectivity in completing PARTassessments. Subjectivity in completing the PART questionnaire and determining PART scorescould potentially also be introduced by White House, OMB, and other political appointees.

Furthermore, in a guidance document for the FY2005 and FY2006 PARTs, OMB has notedthat performance measurement in the public sector, and by extension the PART, have limitations,because:

information provided by performance measurement isjust part of the information that managers and policy officials need to make decisions. Performancemeasurement must often be coupled with evaluation data to increase our understanding of whyresults occur and what value a program adds. Performance information cannot replace data onprogram costs, political judgments about priorities, creativity about solutions, or common sense. Amajor purpose of performance measurement is to raise fundamental questions; the measures seldom,by themselves, provide definitive answers. (33)

In OMB's guidance for the FY2006 PART, OMB stated that "[t]he PART rel[ies] on objective datato assess programs." (34) Former OMB Director Mitchell Daniels Jr. also reportedly stated, with release of the President'sFY2004 budget proposal, that "[t]his is the first year in which ... a serious attempt has been made toevaluate, impartially on an ideology-free basis, what works and what doesn't." (35) Other points of viewregarding how the PART was used are discussed later in this report, in the section titled "Third PartyAssessments of the PART."

OMB Use of PART Ratings

In the President's FY2005 budget proposal, OMB stated that PART ratings are intended to"affect" and "inform" budget decisions, but that "PART ratings do not result in automatic decisionsabout funding." (36) InOMB's guidance for the FY2004 PART, for example, OMB said:

FY 2004 decisions will be fundamentally grounded inprogram performance, but will also continue to be based on a variety of other factors, includingpolicy objectives and priorities of the Administration, and economic and programmatic trends. (37)

In addition, OMB's FY2006 PART guidance states that

[t]he PART is a diagnostic tool; the main objective ofthe PART review is to improve program performance. The PART assessments help linkperformance to budget decisions and provide a basis for making recommendations to improveresults. (38)

The President's budget proposals for FY2004 and FY2005 both indicated that the PART processinfluenced the President's recommendations to Congress. (39)

Third-Party Assessments of the PART

Performance Institute

An analysis of the Bush Administration's FY2005 PART assessments by the PerformanceInstitute, a for-profit corporation that has broadly supported the President's Management Agenda,stated that "PART scores correlated to funding changes demonstrates an undeniable link betweenbudget and performance in FY '05." (40) The Performance Institute noted that the President made thefollowing budget proposals for FY2005:

Programs that OMB judged "Effective" were proposed with average increasesof 7.18%;

"Moderately Effective" programs were proposed with average increases of8.27%;

"Adequate" programs were proposed with decreases of1.64%;

"Ineffective" programs were proposed with average decreases of 37.68%;and

"Results Not Demonstrated" programs were proposed with average decreasesof 3.69%.

The Performance Institute further asserted that the PART had captured the attention of federalmanagers, resulted in improved performance management, resulted in better outcome measures forprograms, and served as a "quality control" tool for GPRA. (41) The company also assertedthat Congress, which had not yet engaged in the PART process, should do so.

Scholarly Assessments

The PART. According to a news report, oneprominent scholar in the area of program evaluation offered a mixed assessment of the PART:

Some critics call PART a blunt instrument. But HarryR. Hatry, the director of the public-management program at the Urban Institute, a Washington thinktank, said the administration appears to be making a genuine effort to evaluate programs. He serveson an advisory panel for the PART initiative. "All of this is pretty groundbreaking," he said. Mr.Hatry argues that it's important to examine outcomes for programs, and that spending decisionsought to be more closely tied to such information. That said, he did caution about how far PARTcan go. "The term 'effective' is probably pushing the word a little bit," he said. "It's almostimpossible to extract in many of these programs ... the effect of the federal expenditures." Ultimately, while Mr. Hatry is enthusiastic about adding information to the budget-making process,he holds no illusions that this will suddenly transform spending decisions in Washington. "Politicalpurpose," he said, "is all over the place." (42)

Scholars have also begun to analyze the PART using sophisticated statistical techniques,including regression analysis. (43) One team investigated "the role of merit and politicalconsiderations" in how PART scores might have influenced the President's budget recommendationsto Congress for FY2004 and FY2005 for individual programs. (44) In summary, they foundthat PART scores were positively correlated with the President's recommendations for budgetincreases and decreases (i.e., a higher PART score was associated with a higher proposed budgetincrease, after controlling for other variables). The team also found what they believed to be someevidence (i.e., statistically significant regression coefficients) that politics may have influenced thebudget recommendations that were made, and how the PART was used, for FY2004, but not forFY2005. They also found what they believed to be evidence that PART scores appeared to haveinfluence for "small-sized" programs (less than $75 million) and "medium-sized" (between $75 and$500 million) programs, but not for large programs. (45)

PART and Performance Budgeting. Observershave generally considered the PART to be a form of "performance budgeting," a term that does nothave a standard definition. (46) In general, however, most definitions of performance budgetinginvolve the use of performance information and program evaluations during a government's budgetprocess. Scholars have generally supported the use of performance information in the budgetprocess, but have also noted a lack of consensus on how the information should be used and thatperformance budgeting has not been a panacea. In state governments, for example:

Practitioners frequently acknowledge that the processof developing measures can be useful from a management and decision-making perspective. Budgetofficers were asked to indicate how effective the development and use of performance measures hasbeen in effecting certain changes in their state across a range of items, from resource allocationissues, to programmatic changes, to cultural factors such as changing communication patterns amongkey players.... Many respondents were willing to describe performance measurement as "somewhateffective," but few were more enthusiastic.... Most markedly, few were willing to attach performancemeasures to changes in appropriation levels.... Legislative budget officers ranked the use ofperformance measures especially low in [effecting] cost savings and reducing duplicative services....Slightly more than half the respondents "strongly agreed" or "agreed" when asked whether theimplementation of performance measures had improved communication between agency personneland the budget office and between agency personnel and legislators. (47)

Another scholar asserted that, among other things, "[p]erformance budgeting is an old idea with adisappointing past and an uncertain future," and that "it is futile to reform budgeting without firstreforming the overall [government] managerial framework." (48)

Government Accountability Office

GAO recently undertook a study of how OMB used the PART for the FY2004 budget. (49) Specifically, GAOexamined: (1) how the PART changed OMB's decision-making process in developing thePresident's FY2004 budget request; (2) the PART's relationship to the [Government Performanceand Results Act] planning process and reporting requirements; and (3) the PART's strengths andweaknesses as an evaluation tool, including how OMB ensured that the PART was appliedconsistently. (50)

GAO asserted that the PART helped to "structure and discipline" how OMB usedperformance information for program analysis and the executive branch budget developmentprocess, (51) made OMB'suse of performance information more transparent, and "stimulated agency interest in budget andperformance integration." (52) However, GAO noted that "only 18 percent of the [FY2004PART] recommendations had a direct link to funding matters." (53) GAO also concluded "themore important role of the PART was not in making resource decisions but in its support forrecommendations to improve program design, assessment, and management." (54)

More fundamentally, GAO contended that the PART is "not well integrated with GPRA --the current statutory framework for strategic planning and reporting." Specifically, GAO said:

OMB has stated its intention to modify GPRA goals andmeasures with those developed under the PART. As a result, OMB's judgment about appropriategoals and measures is substituted for GPRA judgments based on a community of stakeholderinterests.... Many [agency officials] view PART's program-by-program focus and the substitutionof program measures as detrimental to their GPRA planning and reporting processes. OMB's effortto influence program goals is further evident in recent OMB Circular A-11 guidance that clearlyrequires each agency to submit a performance budget for fiscal year 2005, which will replace theannual GPRA performance plan. (55)

Notably, GPRA's framework of strategic planning, performance reporting, and stakeholderconsultation prominently includes consultation with Congress. Furthermore, GAO said:

Although PART can stimulate discussion onprogram-specific performance measurement issues, it is not a substitute for GPRA's strategic,longer-term focus on thematic goals, and on department- and governmentwide crosscuttingcomparisons. Although PART and GPRA serve different needs, a strategy for integrating the twocould help strengthen both. (56)

GAO performed regression analysis on the Bush Administration's PART scores and fundingrecommendations. (57) In particular, GAO estimated the relationship of overall PART scores on the President'srecommended budget changes for FY2004 (measured by percentage change from FY2003) for twoseparate subsets of the programs that OMB assessed with the PART for FY2004. For mandatory programs, GAO found no statistically significant relationship between PART scores and proposedbudget changes. (58) For discretionary programs as an overall group, GAO found a statistically significant, positiverelationship between PART scores and proposed budget changes. (59) However, when GAO ranseparate regressions on small, medium, and large discretionary programs, GAO found a statisticallysignificant, positive relationship only for small programs.

GAO also came to the following determinations:

OMB made sustained efforts to ensure consistency in how programs wereassessed for the PART, but OMB staff nevertheless needed to exercise "interpretation and judgment"and were not fully consistent in interpreting the PART questionnaire (pp. 17-19).

Many PART questions contained subjective terms that contributed tosubjective and inconsistent responses to the questionnaire (pp. 20-21). (60)

Disagreements between OMB and agencies on appropriate performancemeasures helped lead to the designation of certain programs as "Results Not Demonstrated" (p.25). (61)

A lack of performance information and program evaluations inhibitedassessments of programs (pp. 23-24).

The way that OMB defined program may have been useful for a PARTassessment, but "did not necessarily match agency organization or planning elements" andcontributed to the lack of performance information (pp. 29-30). (62)

In response to these issues, GAO recommended that OMB take several actions, includingcentrally monitoring agency implementation and progress on PART recommendations and reportingsuch progress in OMB's budget submission to Congress; continuing to improve the PART guidance;clarifying expectations to agencies on how to allocate scarce evaluation resources; attempting togenerate early in the PART process an ongoing, meaningful dialogue with congressionalappropriations, authorization, and oversight committees about what OMB considers the mostimportant performance issues and program areas; and articulating and implementing an integrated,complementary relationship between GPRA and the PART. (63) In OMB's response, OMBDeputy Director for Management Clay Johnson III stated "We will continually strive to make thePART as credible, objective, and useful as it can be and believe that your recommendations will helpus to that. As you know, OMB is already taking actions to address many of them." (64)

In addition, GAO suggested that while Congress has several opportunities to provide itsperspective on performance issues and performance goals (e.g., when establishing or reauthorizinga program, appropriating funds, or exercising oversight), "a more systematic approach could allowCongress to better articulate performance goals and outcomes for key programs of major concern"and "facilitate OMB's understanding of congressional priorities and concerns and, as a result,increase the usefulness of the PART in budget deliberations." Specifically, GAO suggested

that Congress consider the need for a strategy that couldinclude (1) establishing a vehicle for communicating performance goals and measures for keycongressional priorities and concerns; (2) developing a more structured oversight agenda to permita more coordinated congressional perspective on crosscutting programs and policies; and (3) usingsuch an agenda to inform its authorization, oversight, and appropriations processes. (65)

Potential Criteria for Evaluating the PART or Other Program Evaluations

Concepts for Evaluating a Program Evaluation

Previous sections of this report discussed how the PART is structured, how it has been used,and how various actors have assessed its design and implementation. This section discusses potentialcriteria for evaluating the PART or other program evaluations, which might be considered byCongress during the budget process, in oversight of federal agencies and programs, and regardinglegislation that relates to program evaluation. (66) Should Congress focus on the question of criteria, the programevaluation and social science literature suggests that three standards or criteria may be helpful: theconcepts of validity , reliability , and objectivity .

Validity has been defined as "the extent to which any measuring instrumentmeasures what it is intended to measure." (67) For example, because the PART is supposed to measure theeffectiveness of federal programs, its validity turns on the extent to which PART scores reflect theactual "effectiveness" of those programs. (68)

Reliability has been described as "the relative amount of random inconsistencyor unsystematic fluctuation of individual responses on a measure"; that is, the extent to which severalattempts at measuring something are consistent (e.g., by several human judges or several uses of thesame instrument). (69) Therefore, the degree to which the PART is reliable can be illustrated by the extent to which separateapplications of the instrument to the same program yield the same, or very similar,assessments.

Objectivity has been defined as "whether [an] inquiry is pursued in a way thatmaximizes the chances that the conclusions reached will be true." (70) Definitions of the wordalso frequently suggest concepts of fairness and absence of bias. The opposite concept is subjectivity , suggesting, in turn, concepts of bias, prejudice, or unfairness. Therefore, making ajudgment about the objectivity of the PART or its implementation "involves judging a course ofinquiry, or an inquirer, against some rational standard of how an inquiry ought to have been pursuedin order to maximize the chances of producing true findings " (emphasis in original). (71)

Although these three criteria can each be considered individually, in application they may prove tobe highly interrelated. For example, a measurement tool that is subjectively applied may yield resultsthat, if repeated, are not consistent or do not seem reliable. Conversely, a lack of reliable results maysuggest that the instrument being used may not be valid, or that it is not being applied in an objectivemanner. In these situations, further analysis is typically necessary to determine whether problemsexist and what their nature may be.

Evaluating the PART

With regard to the PART, the Administration has made numerous assessments regardingprogram effectiveness. But how should one validly , reliably , and objectively determine a programis effective ? Should Congress wish to explore these issues regarding the PART or other evaluations,Congress might assess the extent to which the assessments have been, or will be, completed validly,reliably, and objectively.

Different observers will likely have different views about the validity, reliability, andobjectivity of OMB's PART instrument, usage, and determinations. Nonetheless, some previousassessments of the PART suggest areas of particular concern. For example, in its study of the PART,GAO reported that one of the two reasons why programs were designated by the Administration as"results not demonstrated" (nearly 50% of the 234 programs assessed for FY2004) was that OMBand agencies disagreed on how to assess agency program performance, as represented by "long-termand annual performance measures." (72) Different officials in the executive branch appeared to havedifferent conceptions of what the appropriate goals of programs, and measures to assess programs,should be -- raising questions about the validity of the instrument. It is reasonable to conclude thatactors outside the executive branch, including Members of Congress, citizens, and interest groups,may have different perspectives and judgments on appropriate program goals and measures. UnderGPRA, stakeholder views such as these are required to be solicited by statute. Under the PART,however, the role and process for stakeholder participation appears less certain.

Other issues that GAO identified could be interpreted as relating to the PART instrument's validity in assessing program effectiveness (e.g., OMB definitions of specific programs inconsistentwith agency organization and planning); its reliability in making consistent assessments anddeterminations (e.g., inconsistent application of the instrument across multiple programs); and its objectivity in design and usage. To illustrate with some potential examples of objectivity issues,subjectivity could arguably be resident in a number of PART questions, including, among others,when OMB conducted its assessment for FY2005: (73)

whether a program is "excessively" or "unnecessarily" ... "redundant orduplicative of any other Federal, State, local, or private effort" [question 1.3, p. 22]; (74)

whether a program's design is free of "major flaws" [question 1.4, p. 23]; (75)

whether a program's performance measures "meaningfully" reflect theprogram's purpose [question 2.1, p. 25]; (76) and

whether a program has demonstrated "adequate" progress in achievinglong-term performance goals [question 4.1, p. 47].

Use of such terms that, in the absence of clear definitions, are subject to a variety of interpretationscan raise questions about the objectivity of the instrument and its ratings.

In one of its earliest publications on the PART, OMB said that "[w]hile subjectivity can beminimized, it can never be completely eliminated regardless of the method or tool. (77) OMB went on to say,though, that the PART "makes public and transparent the questions OMB asks in advance of makingjudgments, and opens up any subjectivity in that process for discussion and debate." That said, thePART and its implementation to date nevertheless appear to place much of the process for debatingand determining program goals and measures squarely within the executive branch.