Federal Programs: Information Architecture Offers a Potential Approach for Development of an Inventory
Released: 2017-09-28T00:00:00

Letter

Background

GPRAMA requires OMB to make publicly available, on a central government-wide website, a list of all federal programs identified by agencies. For each program, each agency is to provide to OMB for publication an identification of how the agency defines the term “program,” consistent with OMB guidance, including program activities that were aggregated, disaggregated, or consolidated to be considered a program by the agency; a description of the purposes of the program and how the program contributes to the agency’s mission and goals; and an identification of funding for the current fiscal year and the previous 2 fiscal years.

In addition, GPRAMA requires OMB to issue guidance to ensure that the information provided on the website presents a coherent picture of all federal programs.

In August 2012, OMB issued guidance for implementation of the inventory requirements through a phased approach for the 24 agencies subject to the CFO Act. OMB subsequently published 24 separate inventory documents on Performance.gov in May 2013, wherein agencies were to select an approach for identifying programs and provide funding and performance information for the programs identified. For the second phase, originally planned for publication in May 2014, the 24 agencies were to update their inventories based on any stakeholder feedback they received and provide additional program-level funding and performance information. OMB’s guidance also stated that, at that time, the inventory information was to be presented in a more dynamic, web-based approach. However, agencies did not publish updated inventories in May 2014.

In October 2014, we reported that according to OMB officials plans for updating the inventories were on indefinite hold as OMB re-evaluated next steps for what type of information would be presented in the inventories and how it would be presented. OMB staff were considering how implementation of the expanded reporting requirements for federal spending information under the DATA Act could be tied to the program inventories. As of July 2017, OMB had not provided a timeline or plan for the next iteration of the federal program inventory.

In our 2014 assessment of the executive branch’s initial effort to develop a program inventory, we found that the usefulness of the 24 agency inventories was limited. Agencies had the flexibility to identify their programs using different approaches within the broad definition of what constitutes a program, which—while potentially appropriate for individual agencies—limited the comparability of information across the inventory. Further, we found that the agencies did not work together or consult with stakeholders. We also found that none of the agencies provided the necessary budget and performance information. Without performance information, it was unclear how programs supported various agency goals. We also determined that for the federal program inventory to be useful it must be accurate, complete, consistent, reliable, and valid, among other factors. We recommended a number of specific steps OMB and agencies could take to ensure the inventories are more useful to decisions makers, including providing complete performance information (including performance goals), consulting with stakeholders, and ensuring that information in the inventory is comparable within and across agencies. As mentioned above, OMB staff generally agreed with these recommendations, although they did not comment on three of our recommendations related to including tax expenditures and additional performance information.

Information Architecture Provides a Potential Approach for Developing a Federal Program Inventory

The principles and practices of information architecture—a discipline focused on how information is organized, structured, and presented to users—may offer an overarching approach for developing a useful federal program inventory. There are three key concepts in information architecture that are relevant to the development of a federal program inventory—facet, controlled vocabulary, and taxonomy. Table 1 defines these terms and provides examples of what they mean within the context of a federal program inventory. Decision rules provide consistency in how programs are included, in the application of the controlled vocabulary, and in the collection of program information in facets.

Information architecture can be visualized as a process to identify and define needed information, develop a structure for organizing and presenting it, and ensure that standards are met and maintained. These steps may not be purely sequential, but may be iterative as the inventory is developed, evaluated, and maintained. Based on the principles of information architecture, figure 1 provides a conceptual overview of this potential process for developing a federal program inventory. Each of these steps is described more fully in the sections following.

As program information in the inventory is collected and organized into facets, it can be aggregated or disaggregated to facilitate various uses. Facets—and the information or data collected within them—can be structured to allow for searching, grouping, or other functions. Individual facets could describe program characteristics or operations or could relate to budgeting or performance information, among other things. Within these facets, specific information or data would be reported such as program type, specific agency or office names, or budget data. By organizing information according to facets, programs can be identified, grouped, or organized based on certain characteristics, such as the information or data collected within the facet. For example, if a program facet on beneficiaries existed, then potentially all programs that serve the same types of beneficiaries could be identified within the inventory.

In the following sections we provide examples of how these principles and practices can be applied to federal programs. However, since we developed these examples for illustrative purposes only, this does not necessarily mean that they should be included in the inventory.

Step 1: Establish Purpose and Uses

The federal program inventory is intended to improve transparency over federal programs. There can be many specific uses for the inventory to support this purpose, and input from stakeholders—such as Congress, agency officials, state and local governments, third-party service providers, and the public—can assist in establishing these potential uses. Early stakeholder involvement can also guide efforts to determine what programs and program information should be included so that the inventory is more likely to meet stakeholder needs over time.

We have reported that a federal program inventory including performance information could be used by congressional decision makers to inform decision making to identify issues that the federal government should address, to measure progress, and to identify better strategies, if necessary, among other uses. State officials that we interviewed from three states that have developed or are developing program inventories pointed to similar potential uses. For example, in Vermont a senior performance management official said the state’s inventory could be used to facilitate cross-agency coordination, aid government transitions (such as for newly elected legislators), and link program performance with funding. Likewise, an Arizona official told us the state’s program inventory has the potential to show how agency programs align with performance goals. Michigan officials anticipate that the program inventory currently being developed in that state will have the potential to identify duplication and overlap among state programs. As a result, decision makers in that state will be better equipped to oversee the budget process.

Our prior work highlights potential uses for different types of information that could be included in a program inventory. Performance and budgeting information—including, among other types of information, performance goals, targets, and time frames; measures of efficiency; operations, such as activities and services; and costs—could facilitate a variety of potential uses, such as helping decision makers prioritize resources among programs or identifying pressing issues for the government to address; informing congressional decisions about authorizing or reauthorizing federal programs, provisions in the tax code, and other activities; and determining the scope of the federal government’s involvement, investment, and performance in a particular area.

Prioritization of some uses may be important to consider to make the inventory more effective. As we previously reported, consulting with stakeholders to understand their needs would better ensure that the information provided in the inventories is useful for stakeholder decision making. Such prioritization, for example, could also involve examining costs that agencies might face in collecting information for certain facets. Then decisions could be made to select only a subset of all potential facets for inclusion in early iterations of the inventory.

Tax expenditures are one program type that would need to be included in the program inventory to fully implement GPRAMA. Tax expenditures represent a substantial federal commitment. If the Department of the Treasury’s estimates are summed, an estimated $1.23 trillion in federal revenue was forgone from the 169 tax expenditures reported for fiscal year 2015, an amount comparable to discretionary spending. Tax expenditures are often aimed at policy goals similar to those of federal spending programs. Increased transparency over tax expenditures could help determine how well specific tax expenditures work to achieve their goals and how their benefits and costs compare to those of spending programs with similar goals. In our 2014 review of the executive branch’s initial effort to develop a program inventory, we recommended OMB include tax expenditures as a program type in the federal program inventory and work with the Department of the Treasury to produce an inventory of tax expenditures. As stated previously, OMB neither agreed nor disagreed with those recommendations.

Likewise, to enhance usefulness at the federal level, a program inventory can include program operations information, in addition to the budget and performance information required by GPRAMA. Program operations information can include descriptions of what programs do, whom they serve, and the specific activities they conduct. Including this type of information provides a more comprehensive picture of a program within the inventory. There are many potential benefits, including improved ability to identify, assess, and address fragmentation, overlap, and duplication within the federal government. Likewise, program operations information can provide opportunities to enhance service delivery among programs offering similar services or serving related populations. For example, programs serving low income or transportation disadvantaged populations could look for opportunities to facilitate access to related services by coordinating to provide transportation for these beneficiaries.

Step 2: Identify Concepts and Terms

One of the central tasks in creating an inventory of federal programs is to identify the programs to be included and the information to be collected about them. Information architecture practices suggest selecting information sources to compile a list of concepts and terms as part of a controlled vocabulary. For example, stakeholders may frequently use certain terms and concepts to describe programs and make distinctions between different types of programs that can affect the content of the inventory or the information included within it. Thus, grant programs may describe eligible beneficiaries using similar terms, such as rural and urban or youth and elderly. Once the list of concepts and terms has been compiled using a structured process for identifying key terms and concepts, preferences can be selected that best align with meeting user needs to create facets for the inventory. Potential information sources include agency budgets, budget justifications, performance reports, organizational structures, websites, and other internal documentation.

Additionally, the facets that will frame information about those programs would need to be identified and defined, with OMB deciding which facets warrant the cost of collection in the short and long term with input from agencies and stakeholders, if the information architecture approach is used. According to the National Information Standards Organization (NISO), the design and development of a controlled vocabulary can help to ensure that concepts are described distinctly by eliminating ambiguity and controlling for synonyms. As a result, the use of a controlled vocabulary can help agencies identify programs and collect associated program information in facets more consistently.

Programs and Other Organizing Concepts Vary, Presenting Challenges

Differences in how agencies use terms and concepts—especially those related to “program,” “program area,” and “activity”—create challenges for an inventory, which requires consistent information to be useful. As mentioned earlier, the Glossary of Terms Used in the Federal Budget Process defines “program” generally as an organized set of activities directed toward a common purpose. However, variations in agency organizational structures, missions, history, and funding authorities—as well as in the purposes for which agencies create or use program information, such as budgets or performance reports—can result in differences in how agencies organize and group activities using different terms. To illustrate these differences, table 2 provides our observations on how Education, USAID, and DHS used the concepts and terms of program area (for collections of related programs), program, and activity (for more specific activities within a given program) in budget documentation. Each of these three agencies includes these concepts in their documents, but how they are organized and what they contain differs.

The varied uses of these terms within and across agencies—all from agency budget documentation—illustrate one challenge of consistently using words such as “program” and the benefits of creating a controlled vocabulary that could move agencies toward a common understanding and more consistent application of these terms for an inventory. Because agencies have flexibility in deciding what activities constitute a program, an information architecture approach that would focus attention at the facet level would help make the inventory information more consistent. If consistent information is collected, then it can be more easily compared, whether or not the identification of programs is similar across agencies.

Identifying, Selecting, and Defining Program Facets Aids Consistency

Facets and the information within them can provide the structure that will allow the inventory to contain consistent information within and across agencies, aiding comparability of information. Existing guidance points to potential facets and definitions for them, including controlled vocabularies. OMB guidance, as well as requirements for the DATA Act, for example, identify and define facets related to program budget and performance, including performance goals. Program operations information was not included in OMB’s guidance for the initial inventory; however, existing taxonomies in use by federal agencies provide examples of facets and controlled vocabularies for program operations, including how programs operate and whom they serve. For example, the CFDA provides established lists to define eligible grantees and beneficiaries and includes questions to guide agency officials in collecting this information.

With the use of controlled terms, comparisons can be made across programs that serve similar populations or share program eligibility. Table 3 shows potential program operations facets and comparable information collected in the CFDA. To be included in the federal program inventory, the controlled vocabulary and corresponding definitions for facets related to program operations would be assessed against standards, as described below. With many different possible facets and the associated costs of collecting specific program information within them, OMB would need to determine priorities and time frames for required facets in consultation with agencies and stakeholders, if an information architecture approach is used.

Step 3: Determine Relationships

Determining relationships among selected concepts and terms can add to the usefulness of the information in the inventory. The controlled vocabulary can help to show relationships, such as if definitions of some terms refer to other terms or if programs are related. Taxonomies can bring additional structure by linking program facets with one another, promoting functionality and usefulness. Taxonomies tend to be hierarchical, but sometimes are organized in other ways. For example, a hierarchical structure might apply to an agency’s organizational structure in which each related facet is a subset (e.g., agency, bureau, office), and a network structure might be more appropriate for associating categories of information for which there are not specific subcategories, such as facets containing budget and performance information. Finally, decision rules that specify what collections of activities constitute a program for the purposes of the inventory help ensure consistency and comparability of information within the inventory. Program information and data could then be collected for each individual program facet.

Using Decision Rules to Identify Programs and Relationships between Programs

How broadly or narrowly agencies identify programs for the inventory will affect its usefulness. For example, an approach to inventory development that groups many activities under a relatively small number of program names could have limited usefulness, if it results in a low level of transparency over the full range of activities, functions, and costs that occur within that area. Conversely, an approach to an inventory that groups activities narrowly and includes a comparatively large number of programs could result in greater transparency and usefulness, but would likely create significant costs for agencies to identify, create, and maintain. Decision rules for determining what should be identified as a program for purposes of the inventory will need to balance usefulness and costs, if this approach is implemented.

Further, agencies will need to consider how best to organize their activities for inclusion as programs in the inventory, which could present a challenge. The three agencies we reviewed have different organizational structures, such as strategic, programmatic, or budget structures that could be used to organize inventory programs. For example, Education generally has consistent, program-focused alignment across its organizational structures. DHS has historically not had as consistent a program focus across its structures—given its origins from many different agencies—but has recently more closely aligned its budget and program structures. USAID’s different structures have presented agency efforts in multiple ways, including at a country level and also at a broader, mission- focused level such as combating malaria or providing basic education. (See appendix III for more information on how these differences can affect program identification.) Decision rules will need to be established to help agencies present programs in the inventory in a way that is as consistent as possible, given these differences, which could pose different challenges across agencies.

Because agency activities and structures differ, as do user needs, agencies implementing an information architecture approach would need to clearly illustrate the relationships among individual—or groupings of— activities and what is included under a designated program name. This will provide transparency over how the agency applied decision rules and what an agency included under that program, though some agencies may have greater challenges doing so. At the three agencies, we found that variations in the ease of identifying programs often reflected agency organizational structures. For example, Education’s internal organization allows for the relatively easy identification of a consistent list of programs when using appropriation accounts and program names, in part because its appropriations are set up similar to its programs, according to agency officials. By contrast, other agency officials—including at USAID— expressed concern about linking programs to appropriations, because their programs and appropriations are not similarly structured. Appendix III provides information on how Education’s, USAID’s, and DHS’s organizational structures might affect program identification—specifically in our case study context of identifying programs using budget documentation—including a recent DHS effort to better align its budget structure with its discretionary programs.

OMB and agencies could also establish decision rules on how to treat activities and funding streams that may not be clearly linked to specific programs or provide overall administrative or mission support, in order to ensure these items are treated consistently. This can include, for example, general administration, information technology-related maintenance, and general construction. Each of the three agencies selected for our illustrative case studies had these categories of funding and expenses. For example, Education had a “Program Administration” program that accounted for over $400 million in fiscal year 2015 obligations and funded close to half of the agency’s almost 4,100 full-time employees. Education used “Program Administration” to provide administrative support across most programs and offices in the department. If an information architecture approach were used, OMB and agencies would need to determine whether and to what extent these kinds of expense categories should be identified as distinct programs for purposes of the inventory or whether they should be allocated across programs.

Organizing Needed Information for the Inventory

Taxonomies can bring additional structure to an inventory by linking program facets with one another, promoting functionality and usefulness. Figure 2 shows how this can be applied to an individual program. In this example, program information for Education’s Promise Neighborhoods program is collected into potential facets related to the program’s organization, budget, performance, and operations.

Once program information has been collected into facets for multiple programs, a taxonomy allows for the comparison of information across programs, as well as the potential to aggregate—or disaggregate— program information at an appropriate level to facilitate a variety of uses. Table 4 provides an illustration of selected programs in three federal agencies providing early learning or child care services for different age groups. In the federal program inventory, comparisons could be made across or between multiple facets. In this case, the information included within the activities/services and beneficiary facets are compared to identify programs with similar characteristics. For example, sorting programs by information included in the two facets in table 4 would reveal that the Promise Neighborhoods and the Comprehensive Literacy Development Grants (formerly Striving Readers) programs both provide early learning services and have a larger age range of children as intended beneficiaries.

However, collecting program information for each facet may pose challenges for agencies. As we developed our hypothetical inventory, we found that a greater range of program information was readily available for some of the selected programs than for others—often depending on the extent to which programs were included by name in the documents we reviewed (e.g., budget documents, performance and strategic plans, and agency websites). For those programs that were included in the CFDA, for example, we were able to collect information for a number of our facets, such as functional codes that reflect program operations and coded entries for eligibility. Performance goals, including measures and targets, however, are not required in the CFDA.

Likewise, for programs that corresponded to individual program activities in the federal budget, we were able to readily identify budget and financing information, such as obligations, appropriations accounts, and related program activities, although identifying this type of budget information was sometimes more difficult. We have faced similar obstacles to collecting program information in other work. For example, we were unable to identify 39 (of 58) efforts or programs in the President’s budget by name or by funding for a recent report looking at federal efforts supporting U.S. manufacturing, although we were able to report program obligations for many but not all of them after we conducted a survey of agency officials.

Step 4: Evaluate Structure and Content

After identifying programs and facets and determining relevant relationships, information architecture principles suggest evaluating the taxonomy of the program inventory in several ways. For instance, an evaluation could ask how well the inventory’s structure and controlled vocabulary organize and present needed information. Further, the evaluation could involve consulting with experts or comparing with existing taxonomies and standards to ensure that all the needed terms and facets are included. Evaluation of the structure and content of the inventory can involve different methodologies, including reviewing existing standards and interviewing subject-matter experts. There are a number of sources available for standards related to the organization of information using the principles and practices of information architecture, including NISO standards for controlled vocabularies.

In addition to evaluating the inventory’s taxonomy and facets, it will also be necessary to evaluate the quality of the specific program information content. This includes examining the consistency and completeness of the information that agencies report. Consistently identifying program information in facets related to outcomes could help agencies identify where they have programs that have similar purposes or activities, and therefore opportunities to collaborate. Likewise, a complete inventory— including all federal efforts within each definition—could be a useful tool for addressing crosscutting issues. Specific aspects could include examining whether the information is accurate, consistent with the controlled vocabulary, properly formatted, and current. Including consistent and complete program information will help the inventory be more useful and allow users to better compare and contrast programs across broad areas with federal involvement, as we noted in our 2014 report.

More generally, while the inventory is developed iteratively, continuously evaluating the extent to which it is delivering the usefulness desired by stakeholders will enhance its continued usefulness. As part of this evaluation, OMB and agencies can assess the decision rules for the identification of individual programs to determine the extent to which the resulting set of programs are identified at a level that facilitates comparisons across and within agencies. This type of evaluation could lead agencies to determine that activities should be grouped together more broadly (including more activities) or more narrowly (including fewer activities) to allow for better comparisons and increased usefulness. In some cases, activities may need to be allocated differently among programs. Likewise, an evaluation could test program identification by determining if the inventory includes sufficient breadth (in terms of an agency’s total funding) and depth (at a level that is useful for decision makers). We reviewed one narrow way agencies could identify programs—using budget documentation—and found challenges to the consistency and completeness of program information (see appendix III). These examinations can lead to improvements in the inventory over time.

Step 5: Design the User Interface

A well-designed inventory interface can include features to enhance the usefulness of the program information by enabling users to navigate through the content of the inventory to meet their needs. The taxonomy structure serves as the backbone and allows for the presentation of concepts, terms, and relationships dynamically. Specifically, individual facets can be used to identify potential relationships between programs and to organize information in new ways within the inventory. For example, in our hypothetical inventory the facets containing budget information for the Department of Agriculture’s Child and Adult Care Food Program can be linked to both the School Lunch and the School Breakfast programs through a common account. The ability to view these related programs gives the user more tools and information to understand how programs fit within the whole of government and relative to one another. In addition, tagging program information (e.g., attributes or characteristics) within facets in a taxonomy can help create new relationships and allow for the grouping and linking of content in new ways. For example, HHS’s Child Care and Development Fund provides child care services to children ages 5 and under through grants that also support low-income families and children with disabilities. This program could be tagged to highlight these and other attributes of the program collected in facets related to activities and services and to eligible beneficiaries. Then, a user interested in similar programs could click on a tag (e.g., early learning services) that could generate a list of programs that also have that tag. Figure 3 depicts how such an interactive tool could allow a user to identify programs with the same tagged activities.

The interface could also include predesigned output formats for program information. This feature could allow for the creation of program summaries for individual programs or fact sheets cross-walking certain predetermined facets, such as budget and performance information, in a user-friendly format. Figure 4 provides an illustrative example of this concept.

Step 6: Validate Usability and Usefulness

The federal program inventory will exhibit its usability—and thus usefulness—during validation. This step tests the union of the user interface and the taxonomy to improve the usefulness of the inventory and test the inventory’s organization, structure, and general functionality. Using a variety of methods to test the inventory with the intended audience can validate design and content decisions, including any assumptions made about how users interact with data. One method is to conduct usability testing by asking users to complete a series of clearly defined tasks and monitoring how they navigate the inventory. For example, users could be asked to find a term that is grouped with other terms or to find everything they can about a particular topic. Analysis of resulting data from browsing or searching the user interface—such as the number of clicks or completion times of tasks—can reveal how the presentation and grouping of terms affects the completion of user tasks.

A complementary method is to interview or conduct focus groups to obtain qualitative feedback on the usability and usefulness of the inventory. To illustrate this validation methodology, we asked congressional staff to offer their perspectives on how they might use an interactive website containing a federal program inventory with search, filtering, and other navigation capabilities. Overall, congressional staff affirmed that a searchable and sortable design with the ability to provide different levels of aggregation and disaggregation of program information would be useful for a number of tasks, including the following: informing staff quickly about programs as part of background research for various tasks; developing briefing materials for members using program information, informing congressional decisions about authorizing or reauthorizing federal programs and provisions in the tax code; answering constituent questions; identifying information related to program performance; and drawing attention to information gaps, such as if program goals or targets were not developed.

The congressional staff we interviewed also stated that having links to program evaluations, especially GAO, inspector general, and Congressional Research Service reports, would be helpful for learning about program performance, as would information tags identifying direct and indirect program activities and services.

Further, we shared with congressional staff a series of illustrative examples of summary sheets containing information on a number of potential program facets, including budget, performance, and operations information. (Figure 5 presented in the previous section above is one of these examples.) These staff said the types of information and organization matched what they would expect in an inventory and would want to inform their work, although they stated the inventory could be more useful if historical information were more robust than what we included. For example, they expressed a preference for at least 5 years of budget information rather than the 3 years our hypothetical inventory provided. Some staff also emphasized the importance of having strictly defined fields, such as the program history field, in order to avoid confusion and reduce subjectivity in program information. Such feedback can provide valuable insights into the design and content users find most important, the limitations they identify, and their satisfaction with the overall interface and program inventory.

Incorporating the validation results into the design and content of the inventory would enhance usefulness by ultimately enabling users to better find the information they need. Validating the inventory and incorporating prioritized results can mitigate risks related to the opinions and assumptions that were necessary to create an initial inventory framework. Thus, the validation results can also serve as a roadmap for subsequent iterations of the inventory.

Step 7: Govern Structure and Content

Establishing and implementing a governance structure will help ensure the program inventory is continually maintained and useful. Governance specifically involves establishing the policies and procedures—including roles, accountabilities, standards, process methodologies—for maintaining and improving the inventory. Governance policies can also set a schedule for regular assessment of the inventory to monitor how it will meet user needs over the long term. Finally, governance can ensure that the inventory continues to meet factors related to usefulness, including accuracy, completeness, consistency, relatability, and validity.

Good governance requires policies that define the process for managing inventory content, maintaining and changing the taxonomy, and establishing roles and responsibilities. Policies for managing content define the conditions under which programs and program information are added, updated, and archived or deleted. These policies also define the conditions under which the information architectures—taxonomy structure, facets, and controlled vocabulary—are revisited and updated. As such, the governance policies can establish how prospective changes are evaluated and prioritized and when to make changes.

Governance of the inventory will also benefit from well-defined roles and responsibilities. This includes defining the individuals responsible for proposing and making changes to the inventory and taxonomy—both to reflect higher level changes to the purpose of the inventory and the day- to-day management of the taxonomy. In addition, implementation guidelines for each role will further clarify the expected steps by which changes to the information architecture and inventory are made. Further, governance can establish when and how processes are reviewed and updated. Documenting these roles and responsibilities will create accountability and provide a transparent process that will withstand changes in staff.

Governance also includes decision rules that determine how programs are identified and information is included. In our work to develop illustrative examples of programs an inventory might include, we encountered programs with changing names and authorizations, which would require policies to ensure consistent program information be included in the inventory and kept up to date. For example, one potential Education program has had three names since it was originally authorized: (1) Striving Readers, then (2) Striving Readers Comprehensive Literacy, and most recently (3) Comprehensive Literacy Development Grants. Each version of the program has been authorized by different statutory provisions, creating complexities in tracking program information across time, and raising questions about whether it is the same program, successor programs, or three individual programs. Governance policies that include decision rules regarding whether and how to include the evolution of programs can aid the consistency and usefulness of program information over time.

Leveraging existing governance policies, roles, and procedures can help to ensure that the inventory’s usefulness persists. We have previously reported that establishing a formal framework for providing data governance throughout the lifecycle of developing and implementing standards is key for ensuring that the integrity of data standards is maintained over time. There are a number of governance models, and many of them promote a set of common principles that includes clear policies and procedures for broad-based participation from a cross- section of stakeholders for managing the standard-setting process and for controlling the integrity of established standards. Ideally, a governance structure could include processes for evaluating, coordinating, approving, and implementing changes in standards from the initial concept through design, implementation, testing, and release. It would also address how established standards are maintained and ensure that a reasonable degree of agreement from stakeholders is gained.

Agency Comments

We provided a draft of this report for review and comment to the Director of the Office of Management and Budget (OMB), the Departments of Education and Homeland Security, the U.S. Agency for International Development (USAID), and the General Services Administration. USAID provided technical corrections, which GAO incorporated as appropriate. OMB agreed to consider this information architecture approach as it develops plans for the next iteration of the federal program inventory.

We are sending copies of this report to the Director of OMB, the Secretaries of the Departments of Education and Homeland Security, and the Administrators of USAID and the General Services Administration, as well as interested congressional committees and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or CurdaE@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The GPRA Modernization Act of 2010 (GPRAMA) includes a statutory provision for us to periodically evaluate and report on (1) how implementation of the act is affecting performance management at the 24 major departments and agencies subject to the Chief Financial Officers (CFO) Act of 1990, including whether performance management is being used to improve efficiency and effectiveness of agency programs; and (2) crosscutting goal implementation. This report is part of our response to that mandate. GPRAMA requires the Office of Management and Budget (OMB) to present a coherent picture of all federal programs by making information about each program available on a website. For this report, we examined how the principles and practices of information architecture can be applied for the development of a useful federal program inventory.

Programs are defined in our September 2005 Glossary of Terms Used in the Federal Budget Process as “generally, an organized set of activities directed toward a common purpose or goal that an agency undertakes or proposes to carry out its responsibilities.” A federal program inventory consists of the individual programs identified and information collected about each of them. As we have reported, the usefulness of a federal program inventory depends on factors such as accuracy, completeness, consistency, reliability, and validity, among others.

Our methodology involved describing the general steps that could guide the development of a useful federal program inventory using an information architecture approach, as well as assessing how the principles and practices of information architecture could be used both to identify programs and to identify, compile, and organize information within an inventory. This report is not meant to suggest requirements or best practices for developing the federal program inventory, but rather to illustrate how a particular approach could be applied to develop a useful federal program inventory. Other approaches might also be used—or could be incorporated into this framework—to develop an inventory that best addresses limitations identified in the past.

Principles and Practices of Information Architecture

To understand information architecture, we reviewed industry standards, website standards, conference and training materials, books, and leading practices. We then examined how information architecture principles can be used to create a useful federal program inventory that aligns with GPRAMA’s requirements that a website present a coherent picture of all federal programs, as well as with federal website guidelines related to providing usable information (i.e., usability). This analysis included the following steps: qualitatively analyzing the information architecture literature and interviewing information architecture practitioners to identify overarching principles; reviewing federal requirements for the usability of websites and digital services as summarized at digitalgov.gov to identify those guidelines that relate directly to one or more characteristics of a useful federal program inventory (i.e., information that is accurate, complete, consistent, reliable, and valid for its intended use); and comparing the federal policy topics we identified to the overarching principles in information architecture, and aligning information architecture principles with digitalgov.gov guidelines.

Identifying Programs and Identifying, Compiling, and Organizing Information

To gain an understanding of the intended purpose of and potential uses for a federal program inventory, we reviewed requirements in GPRAMA, as well as OMB’s guidance for the executive branch’s initial program inventory effort and our assessment of that effort. We also interviewed current and former federal officials who were knowledgeable about prior efforts to inventory or otherwise consolidate and make publicly available information about federal programs. We reviewed state websites describing state experiences in developing program inventories to understand practices for inventorying program information at the state level, and we interviewed budget and performance officials in three states that have or are developing program inventories to understand the information contained in these inventories and its potential uses.

To understand how programs could be identified and how information within the inventory could be identified, compiled, and organized, we selected individual agencies and programs to examine as case studies.

Program Identification

As part of our effort to apply relevant principles and practices of information architecture to program identification, we developed a set of observations on using budget-related resources to identify programs that could be included in a federal program inventory. See appendix II for a summary of these observations. Specifically, we examined budget-related information resources, including agency budget justification documents and program activity data. We used budget information because most agencies followed a similar approach for their initial inventories and because GPRAMA requires inventories to include budget information for programs included in the inventory.

We selected three agencies to develop these observations: the Departments of Education (Education) and Homeland Security (DHS) and the U.S. Agency for International Development (USAID). These agencies were drawn from the 24 agencies included in the initial executive branch effort to develop a federal program inventory and were selected based on a number of factors, including differences in their overall organizational structure, approach to the prior effort to develop an inventory, and the extent of the connection between their programs and the Catalog of Federal Domestic Assistance (CFDA). This allowed us to compare and contrast the agencies and the usefulness of budget information in those agencies to identify programs. Our analysis included the following: the extent to which consistent and complete lists of agency programs could be identified using budget-related information and the impact of the agency’s organizational structures on these lists; a review of the types of activities that could be characterized as programs within each agency and how activities are grouped into programs or overarching program areas with underlying programs; and a general review of the alignment of possible programs identified by budget documents with other ways agencies organize their efforts, such as performance reports and CFDA programs.

We also reviewed the relationship between the budget’s program activity data for the 24 agencies included in the initial executive branch effort and programs listed in the CFDA to obtain more insights into the different contexts in which agencies identify agency programs and present program information. We attempted to determine the extent to which CFDA programs were aligned with budget program activities by identifying as a possible match any specific CFDA program that was similar in title or funding amount with specific program activities that shared an appropriations account number.

The CFDA is a key resource to identify domestic assistance programs. While not all agency programs would be included in the CFDA, agencies submit programs for inclusion, so they have in essence identified those as programs. For this analysis, we identified the total number of CFDA programs for selected agencies and the number matched to a program activity listed in the federal budget. In a more in-depth analysis of Education’s program activities, we also identified scenarios where (1) program activities were a one-to-one match in name and dollars with information in the CFDA, (2) one program activity matched with a number of CFDA programs, (3) multiple program activities funded what the agency called a single program in the CFDA, and (4) there was no match.

Identifying, Compiling, and Organizing Program Information

As part of our effort to apply relevant principles and practices from information architecture to identify, compile, and organize information about federal programs, we identified information that can be included in a useful federal program inventory, tested the collection and organization of that information by developing a hypothetical inventory with selected programs, and used the hypothetical inventory to illustrate aspects of how a federal program inventory could be validated using an information architecture approach. Finally, we looked to our prior work to identify relevant practices in information and data governance. More specifically, our analysis included the following: Identifying needed information: We identified the types of information about programs that could be included in an inventory to make it useful (e.g., budget, performance, and operations information) by examining OMB and GAO guidance for developing program lists, including OMB’s guidance for the first inventory effort; examining state efforts to develop and use inventories; interviewing potential users; and summarizing examples of the types of program information that have been identified in our past work as being useful.

Developing definitions and a controlled vocabulary: In order to develop our hypothetical inventory, we identified needed information (i.e., facets) to include, as well as definitions for these facets (the “controlled vocabulary”). To identify and define terms that were not included in the initial executive branch’s inventory, we looked at other taxonomies or guidance, including controlled vocabularies used by the Congressional Research Service and by the Education Resources Information Center (ERIC), as well as CFDA guidance for agency officials, OMB guidance on the reporting of performance goals, our Glossary of Terms Used in the Federal Budget Process, and the Digital Accountability and Transparency Act and related OMB guidance.

Selecting individual programs for our hypothetical inventory: We collected program information for a number of individual programs from a variety of sources, including: (1) lists of budget program activities for Education, DHS, and USAID; (2) programs or efforts identified as part of our recent efforts to look at programs with common activities, services, beneficiaries, or outcomes; and (3) our efforts to examine tax expenditure programs. To ensure our set of illustrative programs included a range of programs, we selected programs with certain characteristics (e.g., program size in terms of budget, from different agencies, based on availability of information, etc.). When multiple programs were available based on certain attributes, we used a simple random selection to choose specific programs to include in our set of illustrative programs.

Collecting program information: For selected individual programs, we collected program information from budget and performance documents, as well as from agency websites and the CFDA. For programs selected from our existing work, we also leveraged reported information, and we collected information about any challenges related to identifying programs or collecting program information.

Developing a hypothetical inventory: We tested the development of a hypothetical inventory for six programs drawn from a recent report on early education and child care programs by including individual facets in an online taxonomy to demonstrate how information could be sorted by facets and presented in different ways.

Validating the form and content of the hypothetical inventory: To illustrate aspects of the validation step in the information architecture approach to developing an inventory, we developed sample materials to illustrate what the content and structure of an inventory might include, and we presented these materials to congressional staffers with committees overseeing programs providing early education and child care services (i.e., the Senate Committee on Health, Education, Labor, and Pensions and the House Committee on Education and Workforce). We solicited feedback on the form and content of our hypothetical inventory and collected information regarding potential uses and related needed information.

We conducted this performance audit from July 2016 to September 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Recommendations for the Office of Management and Budget from GAO-15-83: Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories (Oct. 31, 2014)

Appendix II: Recommendations for the Office of Management and Budget from GAO-15-83: Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories (Oct. 31, 2014)

Appendix III: Observations on Using Budget Information to Identify Federal Programs for an Inventory

We examined budget documentation to identify possible programs in three agencies: the Departments of Education (Education) and Homeland Security (DHS) and the U.S. Agency for International Development (USAID). To identify programs, we reviewed budget documentation as an illustrative starting point for several reasons. First, most agencies used budget information to help structure their initial inventories. Second, the GPRA Modernization Act of 2010 requires specific budget information to be included in an inventory. Third, agencies have significant budget related information, such as congressional budget justifications, as well as the federal budget’s appropriations and program activity accounts. We did not conduct a full evaluation of all of the ways agencies could identify programs—or all of the characteristics needed for program identification to be useful—as we did not build a full inventory and such an evaluation was outside of our scope.

Budget Documents Can Identify Possible Programs, but Variations across Agencies Illustrate the Challenge of Identifying Programs Consistently and Completely

Education, DHS, and USAID budget documents all provided information to identify possible programs. Education’s budget documentation had a specific list of programs, organized by program area, with a separate cross-walk between office, program, and statutory authority, allowing for straightforward identification. Both DHS and USAID documentation identified broader areas of effort and named dozens of programs. However, the set of programs we generated from budget documentation illustrated the challenge of identifying programs for an inventory. We evaluated the set of possible programs for consistency and completeness because we have previously reported the importance of those characteristics, among others, to make an inventory useful. We found differences across agencies in the extent to which their budget documents could generate consistent and complete lists of programs, which could result from different organizational structures. Specifically, the programs listed in budget documents did not always present agency programs consistently across other agency resources, such as performance documents or agencies’ congressional appropriations. Further, budget documentation did not always allow for the complete identification of programs for the inventory in terms of depth (at a level that would be sufficiently useful for decision making), although it did have complete breadth (as it contained each agency’s total funding).

Overall, Education’s closer alignment between budget, program, and other organizational structures generally made the identification of programs with budget documents more consistent, limiting challenges. Education’s budget, appropriation, program, and performance structures were all similar, and information about programs was presented consistently across information resources such as appropriation, performance, and Catalog of Federal Domestic Assistance (CDFA) information. Further, Education presented programs with a set of activities at a level that could be useful for decision making in an inventory. For example, Education’s Promise Neighborhoods program supported awards to local partnerships to develop and implement comprehensive, neighborhood-based plans for meeting the cradle-to-career educational, health, and social service needs of children in high-poverty communities. Although there were a number of separate activities within the program, its neighborhood focus presented information that could help decision makers evaluate the activities as a group in light of that focus. Education’s hierarchical structure generally allowed for a clear identification of relationships between agency offices, program areas, and individual programs. Also, other Education documentation presented a specific program list by administrative office and provided a cross-walk between those programs and agency goals, which made it easier to understand how specific programs contribute to the achievement of those goals.

By contrast, USAID budget documents presented greater challenges in identifying possible programs, as its documents were less aligned with a specific program structure and offered less consistent and precise identification. USAID did not have a specific, complete set of programs in budget documents. Rather, USAID identified specific funding accounts, and included some highlights—but not systematic information—about more specific efforts. For example, USAID’s budget documentation included the broad objective Peacekeeping Operations, which had a number of highlighted efforts, including the following:

Global Peace Operations Initiative ($71 million): supports U.S. contributions to international peacekeeping capacity building by providing training and equipment, as well as supporting deployment of troops and evaluations of effectiveness.

South Sudan ($36 million): supports rebuilding the military and support for the Sudan People’s Liberation Army, including training and non-lethal equipment.

Multinational Force and Observers ($28 million): supports efforts to supervise the implementation of security provisions of the Egyptian– Israeli Peace Treaty.

As part of USAID’s Performance Report within its 2015 budget justification, the agency also presented information by program area. For example, its Peace and Security objective included six program areas: (1) Counter-Terrorism; (2) Combating Weapons of Mass Destruction; (3) Stabilization Operations and Security Sector Reform; (4) Counter Narcotics; (5) Transnational Crime; and (6) Conflict Mitigation and Reconciliation. USAID provided information about its work in these areas, but had no specific list of its programs. USAID officials noted that the program areas listed in the justification broadly relate with the program areas set forth in the Department of State’s Standardized Program Structure and Definitions. USAID creates its budget justification jointly with the Department of State.

USAID had different structures across its congressional budget justification, performance structures, and program activities, though its budget justification presented a high-level funding crosswalk between its budget and performance structures. Further, USAID presented its efforts at an individual award level online, including at foreignassistance.gov, but that may be at too narrow a level to be useful for decision making when included in an inventory. The range of structures and ways to present information on the activities of USAID provides transparency and accountability on how agency funds are being used. However, it is our observation that this flexibility and range of methods for presenting information made it challenging to identify through budget documents specific programs that could be included in an inventory, if that inventory were intended to link specific programs and appropriation amounts. USAID officials noted that the agency had specific definitions of program area, program, and activity in documents other than the agency’s congressional budget justification.

Like USAID, DHS did not present a comprehensive program list in its budget documents: DHS had disparate budget, program, and other agency structures, in its case borne out of the parts of different agencies that combined to create DHS, according to agency officials. However, DHS recently aligned its discretionary programs with a standardized budget structure, and now has greater similarities across its budget justifications and appropriations structure. Specifically, DHS established four standard budget categories to be used by all of its mission components: (1) Research and Development; (2) Procurement, Construction, and Improvements, (3) Operations and Support, and (4) Federal Assistance. DHS defined each budget category and created guidance on what activities would typically be included. DHS then created six sub-categories—at the level of the budget’s program, project, and activity account—for more specific funding areas where applicable. DHS also approved other individual categories that better reflect the components’ distinct missions. For instance, in the Federal Emergency Management Agency, DHS used the standard sub-category of Mission Support along with the individual subcategory of Preparedness and Protection. Moreover, DHS added more specific information to the budget program activity for its 2018 budget justification below the program, project, and activity account level. These additions could provide more insight to what DHS considered programs and more consistently link the budget with program information, which could help provide better information at a level useful for decision making in an inventory.

Appendix IV: Observations on the Alignment between Federal Budget Program Activities and Agency Programs in the Catalog of Federal Domestic Assistance

To obtain insights into the different contexts in which agencies identify programs and present program information, we compared 24 agencies’ programs listed in the Catalog of Federal Domestic Assistance (CFDA)— a key resource to identify domestic assistance programs—and their program activity information from the agencies’ budgets. Overall, we observed that the CFDA and budget program activities listings could be helpful in supporting the development of a federal program inventory. The extent of their usefulness will vary by agency, in part, because agencies we spoke to did not view the CFDA as fully consistent with their programs. Based on our analysis, neither resource would be satisfactory for creating a definitive list of programs for any agency for purposes of an inventory. Not all agency programs would be included in the CFDA since the purpose of the catalog is to assist potential applicants to identify and obtain general information about domestic assistance programs. However, with over 2,000 programs included and information about each of those programs, the CFDA could serve as a valuable resource in efforts to develop a federal program inventory and collect program information.

Using a text analytics methodology that compared the names and funding amounts between CFDA programs and the budget’s program activities, we attempted to determine the extent to which CFDA programs were clearly aligned with budget program activities. We observed that the relationships between CFDA programs and program activities within the same appropriation account varied significantly by agency but overall was unclear for all agencies (see table 6).

We also observed that there are multiple different relationships between a CFDA program and agency program activities. Figure 6 presents illustrative examples of the different relationships that CFDA programs might have with specific program activities. These complex and uncertain relationships could affect some of the matched program numbers in table 5 (above) because multiple budget program activities or multiple CFDA programs could be matched.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Brian James (Assistant Director), Molly Laster (Analyst in Charge), Andrew Nelson, and Michelle Serfass made key contributions to this report. Leia Dickerson, Steven Flint, Hedieh Fusfield, Ellen T. Grady, Benjamin T. Licht, Drew Long, Steven Putansu, Robert Robinson, A.J. Stephens, James Sweetman, and John Yee also contributed.