Telecommunications: FCC's Performance Management Weaknesses Could Jeopardize Proposed Reforms of the Rural Health Care Program
Released: 2010-12-17T00:00:00

Letter

Background

A key goal of universal service is to ensure affordable telecommunications services to consumers living in high-cost areas, low-income consumers, eligible schools and libraries, and rural health care providers. Universal service programs are funded by statutorily mandated payments into the Universal Service Fund by companies that provide interstate and international telecommunications services. These payments are deposited into the federal Universal Service Fund, from which disbursements are made for the various federal universal service programs, including the Rural Health Care Program. Companies generally pass their universal service costs along to consumers through a universal service fee on customers’ telephone bills.

FCC’s current Rural Health Care Program is made up of three components that fund different benefits. As figure 1 illustrates, the first two components—the Telecommunications Fund and the Internet Access Fund—are commonly discussed together as the “primary Rural Health Care Program.” Both components in the primary Rural Health Care Program offer discounts on services provided to a single site. In contrast, the third component—the pilot program—encourages health care providers to form comprehensive, multisite, state and regional dedicated health care networks.

Figure 2 shows how the components in the current Rural Health Care Program may change if FCC adopts the proposed reforms described in its July 2010 NPRM. As the figure illustrates, the Health Broadband Services Program would replace the Internet Access Fund (and raise the discount percentage). A new Health Infrastructure Program would make available up to $100 million per year to support up to 85 percent of the construction costs of new regional or statewide networks for health care providers in areas of the country where broadband is unavailable or insufficient. This $100 million would be part of the overall $400 million annual spending cap that covers the Rural Health Care Program as a whole and that FCC established in 1997.

In managing the program, FCC oversees USAC—the not-for-profit corporation that administers the program. USAC uses its subcontractor, Solix, Inc., to carry out certain key aspects of the program, such as reviewing and processing funding applications. An MOU between FCC and USAC as well as FCC orders and rules set forth the roles and responsibilities of FCC and USAC in the management, oversight, and administration of the Rural Health Care Program. (See the sidebar on this page for examples of benefits provided by the Rural Health Care Program.)

USAC has highlighted the impct of the Rl Helth Cre Progrm on the Kodik Are Ntive Assocition (KANA). KANA i nonprofit corportion tht provide helth nd ociervice for the Alas Ntive of the Konig region. According to the Informtion Stem Mger of KANA, the support received from the primry Rl Helth Cre Progrm has “revoltionized telehelth ervice” t KANA. KANA ptient once hd to wit etween 6 nd 9 month to ee n epecilit, but telemedicine has redced ptient wit time 2 week nd has redced trvel co, ince mny ptient viite condcted remotely y other helth cre provider. For exmple, phyicin in Anchorge was able to ass helth ide in Kotzebue perform surgery when evere wether mde ir trvel impossle. The Informtion Stem Mger reported tht withosupport from the primry Rl Helth Cre Progrm, KANA wold e forced to go back to using dil-ervice. Withot thi support, “it wold e difficlt to fford even the llet connection etween the villge nd KANA.” satellite service charges, so that rural and urban prices are comparable within each state. Second, to support advanced telecommunications and information services, the Internet Access Fund offers most rural health care providers a 25 percent flat discount on monthly Internet access charges. Eligible rural health care providers can apply for support from both the Telecommunications Fund and the Internet Access Fund. However, FCC has stated that rural health care providers have not participated at the rate it had expected.

The steps that applicants must carry out to obtain support from one or both components of the primary Rural Health Care Program are illustrated in figure 3.

FCC created the third component of the current Rural Health Care Program, the pilot program, in September 2006 after acknowledging that the primary Rural Health Care Program was “greatly underutilized.” FCC explained that “although there are a number of factors that may explain the underutilization” of the program, it was “apparent that health care providers continue to lack access to the broadband facilities needed to support…advanced telehealth applications.” The pilot program funds 85 percent of the costs of deploying dedicated broadband networks connecting rural and urban health care providers, including the cost of designing and installing broadband networks that connect health care providers in a state or region, as well as the costs of advanced telecommunications and information services that ride over that network. This is in contrast to the primary Rural Health Care Program, which provides discounts only on monthly recurring costs for telecommunications services or Internet access to rural health care providers. The pilot program also provides funding for the cost of connecting state or regional networks to Internet2 or National LambdaRail—two national networks that connect government research institutions as well as academic, public, and private health care institutions—and the costs of connecting to the public Internet. Any eligible public and nonprofit health care provider—whether located in an urban or a rural area—was eligible to apply for funding when the pilot program was announced. However, the program rules required that applicants’ proposed networks include at least a de minimis number of public and nonprofit health care providers that serve rural areas.

FCC received 81 applications from projects seeking to participate in the pilot program. In November 2007, FCC announced that it had accepted 69 of the 81 projects into the program and capped total funding for all of the pilot projects at roughly $418 million over 3 years. Since then, a few projects have merged and 1 project has withdrawn from the program. The size and scope of the remaining projects vary widely. For example, the Illinois Rural HealthNet project is using pilot program funds to pay for the installation of approximately 1,250 miles of buried fiber. The purpose of this fiber is to create the backbone of a network that will connect rural critical access hospitals, health clinics, and community mental health centers to specialists throughout the state and nation. In contrast, a project in Wisconsin is planning to use the pilot program funds to link two existing fiber systems to establish connections between four hospitals that allow health care specialists to transmit images between facilities. (See the sidebar on this page for other examples of pilot projects.)

USAC administers the pilot program pursuant to FCC’s rules. Each pilot project must designate a project coordinator and associate project coordinator, who manage the administrative aspects of the program for the project and submit the required forms. USAC provides each project with a “coach”—that is, a designated Solix staff person who works closely with a pilot project to assist the project through the program’s administrative requirements and processes. With some exceptions, the pilot program forms and administrative processes are the same as those previously described in the primary Rural Health Care Program. However, pilot participants pay 15 percent of eligible costs (and all ineligible costs), and the pilot program funds up to 85 percent of eligible costs. In addition, pilot participants must meet additional requirements before they can receive funding: The lead entity in charge of the pilot project must obtain a letter of agency from every entity participating in its project. This letter authorizes the lead entity to act on the other entity’s behalf in all matters related to the pilot program.

Pilot participants must develop a sustainability plan describing how the project will be self-sustaining in the future, to include network ownership and membership arrangements, and describing sources of future support.

Pilot participants are required to submit quarterly progress reports describing the status of their project.

In February 2010, FCC’s Wireline Competition Bureau extended by 1 year, to June 30, 2011, the deadline for participants in the pilot program to submit to USAC requests for funding commitments.

FCC Has Not Performed the Analysis Necessary to Ensure That the Primary Rural Health Care Program Meets the Needs of Rural Health Care Providers

Participation in the Program, Although Increasing, Has Not Met FCC Projections and over Half of All Program Funds Are Used in Alaska

Annual disbursements from the primary Rural Health Care Program have increased from 1998 through 2009, yet they have never approached FCC’s original projections for participation. Figure 4 shows the total amount of funds that have been disbursed for the primary Rural Health Care Program from 1998 to 2009.

USAC disbursed just over $327 million for the primary program from 1998 through 2009. Thus, as figure 4 illustrates, total program expenditures in 12 years of disbursements have not yet reached the single year funding cap of $400 million. Also, as of September 2010, USAC has disbursed just over $26 million for the pilot program. Therefore, USAC has disbursed less than $400 million for all three components of the Rural Health Care Program since the program began in 1998. (FCC does not collect $400 million each year from telecommunications carriers for this program, but rather bases collections only on projected expenditures. FCC uses a quarterly evaluation of health care provider demand to assess how much telecommunications companies must contribute to the Universal Service Fund each quarter. This means that if FCC’s proposed reforms create more participation in the program, telecommunications companies would need to pay more in Universal Service Fund contributions. Telecommunications companies would likely pass these costs on to consumers through higher universal service fees in consumers’ telephone bills.)

According to USAC data, primary Rural Health Care Program funding was disbursed to all of the types of rural health care providers designated by statute as eligible to participate in the program. As figure 5 illustrates, over 68 percent of total applicants in 2008 were either rural health clinics or not-for-profit hospitals.

As with disbursements, the number of applicants to the primary Rural Health Care Program has generally increased since the program began. Figure 6 shows the number of rural health care providers that have applied to the primary Rural Health Care Program have increased from a low of 1,283 applicants in 1999 to 4,014 in 2009.

Similarly, the number of funding commitments issued to participants in the primary Rural Health Care Program has exhibited a slow, steady increase over time from 799 funding commitments in 1998 to 6,790 in 2008. Figure 7 shows the number of funding commitments by the type of service requested (e.g., telecommunications services or Internet access services).

Funding commitments have varied considerably among applicants within the states and territories, with almost 55 percent of the funding going to applicants in Alaska. Disbursements range from over $178 million for Alaska to none for three states (Connecticut, New Jersey, and Rhode Island). Health care providers in Wisconsin received the second-largest disbursement, approximately $18.5 million (almost 5.7 percent) of all primary Rural Health Care Program funding. For a snapshot of funding to applicants by state and territory, see appendix II, which contains the numbers of applicants and amounts committed by state for 2008. Table 1 shows the total amount of money that has been committed and disbursed to applicants, by state, over the program’s history.

Figure 8 shows the total dollar amount disbursed across the United States for funding year 2008, by ZIP code, illustrating the wide variation in geographic use and the heavy concentration of funding in Alaska.

According to FCC and USAC staff, health care providers in Alaska dominate use of the primary Rural Health Care Program because Alaska’s rural areas often require expensive satellite telecommunications services. Alaska’s vast size, harsh winter weather, and sparse population make fiber networks and other technologies either too expensive or too infeasible. Some wireless technologies also can be challenging, since Alaskan terrain often includes mountains or forests that can obstruct line-of-sight transmission. As a result, satellite is often the most feasible option for many rural communities in Alaska. Although the cost of telecommunications service in rural areas can vary considerably, satellite service can cost up to $13,000 per month, creating a significant difference in urban and rural rates in parts of Alaska, and making FCC’s Rural Health Care Program particularly attractive under such circumstances.

We also found that, according to USAC data, FCC and USAC have been successful in disbursing committed funds in the primary Rural Health Care Program. Table 1 shows that USAC generally disburses most of the funds that are committed to rural health care providers. Of the more than $380 million committed for the program, over $327 million (over 86 percent) has been disbursed, leaving just over $53 million that has been committed but not disbursed since the program began. Some of this $53 million in remaining money will eventually be disbursed as USAC closes more recent funding years.

FCC Has Not Assessed the Telecommunications Needs of Rural Health Care Providers to Guide the Evolution of the Rural Health Care Program

A needs assessment is crucial to both the effective design of new programs and the assessment of existing programs. The primary purpose of a needs assessment is to identify needed services that are lacking (in this case, telecommunications services for rural health care providers) relative to some generally accepted standard. By establishing measures of comparison, program managers can more accurately determine how well their programs are doing in meeting the needs of the targeted population of the program. We have previously recommended that needs assessments include the following characteristics: benchmarks to define when needs have increased or decreased, a plan to determine how needs assessment results will be prioritized in supporting resource allocation decisions, and integration of information on other resources available to help address the need.

However, throughout its 12 years of managing the program, FCC has not conducted a comprehensive needs assessment to learn how the program can best target the telecommunications needs of rural health care providers within the broad latitude provided by Congress in the 1996 Act.

When designing the $400 million annual spending cap for the Rural Health Care Program, FCC officials noted the scarcity of information available about the universe of eligible providers, and what it might cost to meet the providers’ telecommunications needs. As our analysis showed, the current $400 million spending cap is not based on meaningful estimates of program participation. FCC stated in its 1997 report and order that the Rural Health Care Program spending cap is “based on the maximum amount of service that we have found necessary and on generous estimates of the number of potentially eligible rural health care providers.” FCC acknowledged at the time that it expected actual program disbursements to be less than the cap for a number of reasons. Although FCC expected program disbursement to be under $400 million annually, on multiple occasions, FCC has released documents stating that the primary Rural Health Care Program is underutilized. For example, in its 2006 pilot program order, FCC states that the primary Rural Health Care Program “continues to be greatly underutilized and is not fully realizing the benefits intended by the statute and our rules. In 1997, we authorized $400 million per year for funding of this program. Yet, in each of the last 10 years, the program generally has disbursed less than 10 percent of the authorized funds.”

When we asked FCC officials what acceptable utilization of the program would mean, they said that they did not know, but that program utilization would include disbursing funds somewhere between 10 percent and 100 percent of the allowable cap. FCC’s repeated claim that the program is underutilized, without a more specific vision of what utilization would mean, is troublesome. No needs assessment has been conducted to show that the program is, in fact, underutilized. A comprehensive needs assessment could provide useful information to FCC to help officials envision acceptable program utilization—that is, how many providers actually need services, rather than just how many providers are eligible to participate under program rules.

As part of our review, we interviewed knowledgeable stakeholders to identify potential reasons for FCC’s reported underutilization. These reasons include the following: Some health care providers lack the infrastructure (e.g., the broadband facilities needed to support telemedicine) to use advanced telecommunications services.

The application process is too complex and cumbersome to justify participation.

The 25 percent Internet subsidy is not large enough to encourage participation.

The difference between urban and rural telecommunications rates is negligible or not significant enough to justify resources toward program participation.

Rural health care providers do not have enough administrative support to apply to the program annually.

Some eligible health care providers may not know about the program.

Statutory restrictions prevent support to certain providers who might benefit from the program (e.g., emergency medical technicians).

Some health care providers cannot afford expensive telemedicine equipment; therefore, they are not concerned with gaining access to the telecommunications services needed to use that equipment.

Some Medicare and Medicaid rules, including reimbursement limitations, may inhibit the use of telemedicine technologies; therefore, health care providers may not be concerned with gaining access to the telecommunications services needed to support those technologies.

Despite these and other issues, we were also told that many of the current program participants are dependent on the benefits they receive from the primary Rural Health Care Program.

Although it lacks a needs assessment, FCC has made multiple changes to the primary Rural Health Care Program over time in an attempt to address underutilization and better meet providers’ needs. For example: In a 2003 report and order, FCC provided support for rural health care providers to obtain a 25 percent discount off the cost of monthly Internet access services. The 2003 report and order states: “Because participation in the rural health care support mechanism has not met the Commission’s initial projections, we amend our rules to improve the program, increase participation by rural health care providers, and ensure that the benefits of the program continue to be distributed in a fair and equitable manner.”

In a 2004 report and order, FCC changed the definition of “rural,” revised its rules to expand funding for mobile rural health care services, and allowed a 50 percent subsidy (rather than 25 percent) for Internet access services for health care providers in entirely rural states. According to USAC, this report and order increased the number of health care providers eligible to participate in the primary Rural Health Care Program by adding new rural areas while grandfathering health care providers in areas no longer defined as rural.

In a 2006 order, FCC announced the pilot program, which will be discussed in greater detail in the next section of this report. FCC created the pilot program to address two potential reasons for the primary Rural Health Care Program’s possible underutilization: lack of infrastructure and access to dedicated broadband networks.

Without a needs assessment, however, FCC does not have key information regarding the extent to which any of these reasons actually impacted the primary Rural Health Care Program’s participation rate. FCC officials told us that the changes FCC has made to the program were based primarily on information gathered through the agency’s notice an d comment procedures and internal deliberations. FCC officials told us th this is how FCC—as a federal regulatory agency—conducts its business pursuant to the Administrative Procedure Act (APA). nothing in the APA process that would have precluded FCC from s to conducting a formal needs assessment. Using data-based assessment supplement the information gained through FCC’s regulatory procedures al would enhance FCC’s ability to fulfill its role as the manager of the Rur Health Care Program. Specifically, if FCC had obtained data through a formal needs assessment, it may have been able to more accurately ascertain why some rural health care providers are not participating, and have better ensured that programmatic changes achieved th results.

Administrative Procedure Act, 5 U.S.C. §§ 551 et seq.

FCC’s Poor Planning and Communication during the Design and Implementation of the Pilot Program Caused Delays and Difficulties

FCC’s Limited Collaboration with USAC, Federal Agencies, and Other Knowledgeable Stakeholders Affected Pilot Program Design

FCC missed multiple opportunities to collaborate with USAC, federal agencies, and other knowledgeable stakeholders when designing the pilot program. These stakeholders all could have provided useful insights into FCC’s design of the pilot program. Such consultations could have helped FCC better identify potential pitfalls in its pilot program design as well as meaningful opportunities to leverage federal resources and ensure that the pilot program targeted rural health care providers’ needs in the most efficient way.

Although USAC officials had 9 years’ experience working with the rural health care community and administering the primary Rural Health Care Program, FCC did not consult with USAC officials prior to issuing the 2006 order calling for applications to the pilot program. Our prior work has noted the importance of involving stakeholders (including third-party administrators like USAC) when designing, implementing, and evaluating programs. FCC officials stated that they did not consult with USAC because USAC does not formulate policy. However, USAC’s experience with the primary Rural Health Care Program may have provided FCC with valuable insights into how to design a pilot program, particularly regarding the administrative processes and forms. For example, FCC’s decision to use the primary program’s forms and processes for the pilot program led to a complicated administrative process, particularly since some aspects of the primary Rural Health Care Program’s forms and administrative processes were ill-suited to the pilot program. Because FCC used primary program forms rather than creating new and more tailored ones for the pilot program, the forms required complicated attachments. According to our survey of pilot project representatives, of the 57 respondents that expressed an opinion, 38 respondents rated assembling their request for proposals (RFP) package (Form 465 package) as “very difficult” or “somewhat difficult.” In addition, 27 of the 42 respondents that provided an opinion rated assembling their requests for funding (Form 466-A packages) as “very difficult” or “somewhat difficult.” Solix officials agreed that pilot participants seemed to have difficulty in completing these forms and attachments.

FCC also missed opportunities to coordinate with other federal agencies when designing the pilot program. We have noted that a lack of collaboration among federal agencies can lead to a patchwork of programs that can waste scarce funds, confuse and frustrate program customers, and limit the overall effectiveness of the federal effort. A number of federal agencies are involved in telemedicine efforts, and some provide funds to health care providers that could complement FCC’s pilot program. For example, the Health Resources and Services Administration (HRSA), the primary federal agency for improving access to health care services for people who are uninsured, isolated, or medically vulnerable, administers a Telehealth Network Grant Program that provides funds to projects to demonstrate how telehealth programs and networks can improve access to quality health care services in underserved rural and urban communities. However, with USDA being the one exception, FCC did not contact other federal agencies prior to announcing the pilot program in 2006. FCC officials told us that after announcing the creation of the pilot program in 2006, they met with representatives from various agencies within HHS in 2007, to discuss coordination. Representatives from some of these agencies reported that these meetings were primarily informational, with FCC explaining its pilot program to them, and that no strategies for collaboration or follow-up were developed. USDA officials stated that FCC officials met with them prior to announcing the pilot program to discuss USDA’s Distance Learning and Telemedicine Program, including how USDA scored applications and evaluated the program. However, it is unclear how FCC used the information that USDA provided, since similar information was not provided in FCC’s call for applications or order selecting pilot projects. According to federal and other stakeholders, officials at other agencies also could have provided FCC with an understanding of rural health care providers’ needs, potential information technology (IT) issues, and how to design a more user-friendly program and helped FCC identify additional appropriate service providers, one of which had to petition to be included.

FCC also did not request public comment on its proposed design for the pilot program. Although FCC did request comments in 2004 on providing some infrastructure support by funding upgrades to the public switched or backbone networks, FCC did not imply that it was considering a pilot program to fund the creation of private networks, or provide specific details on how such a program would operate. We have previously reported that FCC’s use of NPRMs to pose broad questions without providing actual rule text can limit stakeholders’ ability to determine either what action FCC is considering or what information would be most helpful to FCC when developing a final rule. FCC officials said that they did not issue a Notice of Inquiry (NOI) regarding the pilot program because the process would have delayed the pilot program. However, providing the public with advance notice of proposed changes and an opportunity to comment on them is desirable in that it allows agencies, according to a 2006 resource guide, to “find out earlier rather than later about views and information adverse to the agency’s proposal or bearing on its practicality.” Similarly, in comments submitted to FCC, the National Telecommunications Cooperative Association, observed that “interested or affected parties had no opportunity to explore with the Commission various aspects of the Pilot Program.” In addition, industry concerns regarding the funding of redundant networks arose after the implementation of the pilot program. If FCC had provided a more detailed explanation of the proposed pilot program and requested comment prior to establishing the program, it may have been better prepared to address these concerns.

Pilot Participants Have Experienced Delays and Difficulties, in Part, Because FCC Did Not Fully Establish Requirements Prior to Calling for Applications and Did Not Provide Effective Program Guidance

Pilot Participants Have Experienced Delays and Difficulties for Many Reasons

FCC called for applications to participate in the pilot program before it fully established pilot program requirements. This, along with the addition of requirements as the pilot program has progressed, has led to delays and difficulties for pilot participants. Most importantly, the entire pilot program itself has been delayed. Participants may issue multiple RFPs as they progress through various stages of designing and constructing their networks, but the deadline for pilot participants to submit all of their requests for funding (projects submit at least one Form 466-A for each RFP they issue) to USAC was June 30, 2010. On February 18, 2010, FCC extended this deadline by 1 year, to June 30, 2011. According to USAC data, at the time of the extension, projects had requested 11 percent of the roughly $418 million in total program funding. As of July 31, 2010, projects had requested 17 percent of the total program funding. As shown in figure 9, as of July 31, 2010, 28 projects (45 percent) have received at least one funding commitment letter, but 18 projects (29 percent) had not yet posted an RFP.

According to our survey, delayed and inconsistent guidance led to delays for many pilot projects. In addition, it appears pilot participants have struggled with requirements that were added at the same time that FCC announced the pilot participant selections, such as the need to obtain letters of agency. Figure 10 indicates the number of survey respondents reporting whether they experienced certain issues during the course of their project, and the number of respondents that reported they were delayed by that issue.

Table 2 reports the results from our survey question that asked pilot participants to rate the ease or difficulty of performing various program tasks. Four of the tasks rated as “very difficult” or “somewhat difficult” by more than half of the respondents that provided an opinion fall into one of two categories: the task is associated with program processes and forms that were carried over from the primary Rural Health Care Program (Form 465 and Form 466-A), or the task is a requirement FCC added, but that was not mentioned in the initial call for applications (developing a sustainability plan and obtaining letters of agency). In addition, when asked to list the top three things that program officials should change if FCC established a new, permanent program with goals similar to those of the pilot program, simplifying or improving the administrative process was the most frequently mentioned issue.

The Grant Accountability Project, a 2005 Domestic Working Group chaired by the former United States Comptroller General, notes that an agency’s ability to ensure that funds are used as intended is impacted when the terms, conditions, and provisions in award agreements are not well- written. The group also notes that a thorough assessment of proposed projects can reduce the risk that money may be wasted or projects may not achieve intended results. However, FCC did not fully establish the requirements of the pilot program before it requested applications, and it required projects to provide additional information after they were accepted into the program. This led to delays because (1) participants needed additional guidance on how to meet the requirements and (2) Solix staff (under USAC direction) had to retroactively review projects to determine the eligibility of the participating entities and activities. In addition, some participants faced difficulties in funding ineligible expenses. In contrast, other federal agencies generally provide extensive detail on program rules when calling for applications for competitive funding programs, including the criteria by which applications will be judged and how the criteria will be weighted. For example, USDA’s 2010 Distance Learning and Telemedicine Program Grant Application Guide provides potential applicants with specific information on eligible uses for the funds, eligible match funding, copies of the forms to be used, and information on the application scoring process.

FCC’s 2006 order establishing the pilot program and calling for applications did not provide detailed information on many essential aspects of the program, including which entities would be eligible to participate in the program (FCC provided a legal citation, but no actual text); which expenses could be paid for with program funds; and how projects could fund their 15 percent match.

After issuing its call for applications, FCC did provide some of this information on its Frequently Asked Questions Web page on the pilot program. However, this information may not have reached all interested parties, and it would have been more efficient to determine these issues in advance of requesting applications. FCC also provided some of this information in its 2007 order; however, by this time, FCC was also announcing which projects were selected. FCC did not fully screen applications to determine the extent to which their proposed activities and entities would be eligible for funding. Thus, several of the accepted projects had ineligible components. Specifically, based on survey respondents that provided a substantive answer: 25 of 59 respondents included an entity that was determined to be ineligible, 25 of 57 respondents included an expense that was determined to be 10 of 59 respondents relied on ineligible sources to fund their match.

The lack of established criteria and an in-depth screening prior to announcing pilot project awards led to a lengthy process by which Solix staff (under USAC direction) determine the eligibility of entities postaward. Pilot participants must submit documentation for every entity in their project, which Solix staff then review to determine eligibility. According to our survey, 36 of 60 respondents were delayed by difficulties in compiling and submitting the documentation needed to establish entity eligibility. In addition, although 39 survey respondents rated the current program guidance regarding entity eligibility as “very clear” or “somewhat clear”; some confusion remains, as 20 pilot participants rated the current guidance as “slightly clear” or “not at all clear.” FCC’s 2007 order also notes that program administration costs, such as personnel, travel, legal, marketing, and training costs, are ineligible for program funding. Pilot participants have indicated in written comments to FCC that they did not anticipate that administrative costs would not be eligible for funding, and some have faced challenges in funding these costs themselves. In our survey, 36 of 51 respondents indicated that funding ineligible costs, including administrative costs, has been “very difficult” or “somewhat difficult.” In addition, when asked to list the top three things that program officials should change if FCC established a new, permanent program with goals similar to those of the pilot program, providing funding for administrative costs was the second-most frequently mentioned issue.

FCC Introduced New Requirements after Its Call for Applications and Selection of Pilot Participants

FCC’s 2007 order also introduced new requirements that were not mentioned in the 2006 order. For example, while the 2006 order states that the pilot program would use the same forms and administrative processes used in the primary Rural Health Care Program, the 2007 order also requires projects to secure a letter of agency from every entity participating in a project. This letter authorizes the lead project coordinator to act on the signing agency’s behalf. Since a number of the selected pilot program participants included providers that were also participating in another participant’s proposed network, FCC noted that the letter of agency would demonstrate that the entity has agreed to participate in the network and prevent improper duplicate support for providers participating in multiple networks. Considering that FCC encouraged applicants to create statewide networks, some projects have hundreds of participating entities, creating the need for such projects to secure hundreds of letters of agency. The letter of agency requirement has proven extremely time-consuming and resource-intensive for some projects. According to our survey, 34 of 60 respondents faced difficulties in securing letters of agency, and 27 of these respondents were delayed because of these difficulties. (See fig. 10.)

Similarly, according to our survey results, FCC has not provided sufficient guidance to pilot projects on how to meet FCC’s requirement that projects comply with HHS health IT initiatives. In response to a letter from HHS, FCC outlined a number of requirements in its 2007 selection order that pilot program participants should meet to ensure that their pilot projects were consistent with HHS health IT initiatives. FCC officials stated that the explanation of the requirements in its 2007 selection order, in addition to a guidance document created in 2008, provided guidance for the pilot projects in how to meet these requirements. However, one HHS official described the language in the 2007 selection order as vague and in need of an update. In addition, the 2008 guidance document has not been revised to reflect new developments in interoperability specifications and certification programs. Currently, pilot participants are required to explain in each quarterly report how they are complying with the HHS health IT requirement. However, 34 of 47 survey respondents who provided an opinion stated the guidance provided on how to meet these requirements was “slightly sufficient” or “not at all sufficient.”

Following the release of the 2007 order, FCC created additional requirements as the program progressed and did not provide program guidance in a timely manner on how to meet these requirements. For example, FCC stated in its 2007 award order that selected pilot participants generally “provided sufficient evidence that their proposed networks will be self-sustaining by the completion of the pilot program.” However, program officials began requiring more detailed sustainability plans in the fall of 2008, after some projects had gone through the competitive bidding process and had requested funding commitment letters from USAC. Outside of an October 24, 2008, letter to USAC in which FCC noted that participants should “disclose all sources or potential sources of revenue that relate to the network” and intentions to sell or lease excess capacity in the project’s sustainability plan, FCC did not provide any other written guidance on what specific information should be included in participant’s sustainability plans until April 2009. At that time, FCC posted an item to its Frequently Asked Questions Web page that suggested more information to be included in a participant’s sustainability plan, including status of obtaining the match, projected sustainability period, network membership agreements, ownership structure, sources of future support, and management structure. USAC and Solix officials noted that some pilot participants believed that because their application was accepted by FCC, they met all of the program requirements for sustainability. In some cases, this misunderstanding led to confusion and disagreements between the pilot participants and program officials regarding the need for additional information and the amount of time that a sustainability plan should cover. Moreover, it appears some confusion remains. When asked to rate their satisfaction with any guidance they received thus far on how to develop a sustainability plan, 21 of the 60 survey respondents that provided an opinion were “very dissatisfied” or “somewhat dissatisfied.” In addition, 39 of 59 survey respondents faced difficulties in developing a sustainability plan; 33 of these respondents stated that difficulties in developing a sustainability plan have delayed their projects. (See fig. 10.)

Program Guidance Is Not Provided in an Effective Manner

In addition, because FCC is responsible for all policy decisions regarding the pilot program, unique or difficult situations are typically referred from USAC to FCC for its decision. The need for such consultations is compounded by an absence of formal written guidance for USAC and pilot participants. We have reported that information should be recorded and communicated to management and others who need it in a form and within a time frame that enables them to carry out their responsibilities. FCC staff stated that, in some cases, they have not provided written guidance because they want the pilot program to remain flexible. However, in some cases, it has taken several months for FCC to make a decision or provide guidance on issues. However, it appears that pilot participants are dissatisfied with certain elements of program guidance. As noted in figure 11, of the 59 respondents that provided an opinion, 32 respondents were “very dissatisfied” or “somewhat dissatisfied” with the clarity of program guidance, and 28 respondents were “very dissatisfied” or “somewhat dissatisfied” with the amount of formal written guidance. In addition, as we note in figure 10, 37 of 60 respondents stated that they had received inconsistent guidance. Similarly, when asked to list the top three things that program officials should change if FCC established a new, permanent program with goals similar to those of the pilot program, providing more guidance and templates was the third-most frequently mentioned issue.

In addition, although FCC recognized in its 2006 order calling for applications that ineligible entities may be participating in the networks and would need to pay their fair share of costs, FCC chose not to establish detailed guidance on how to address such issues prior to establishing the program. Instead, FCC provided guidance as questions arose from USAC (USAC’s first request about how to determine fair share was in June 2007 when it was noted that a substantial number of ineligible entities were included in applications submitted to FCC) and from pilot program participants. According to USAC officials, questions concerning payment of fair share or incremental costs for excess capacity shared with ineligible entities occurred at the pilot program training in February 2008 and continued from participants to USAC and from USAC to FCC throughout 2008. The most recent FCC guidance was a March 2009 matrix outlining nine scenarios in which excess capacity could be used. However, issues regarding excess capacity remain. FCC’s July 2010 NPRM notes that “rules governing the sharing of this subsidized infrastructure are necessary to prevent waste, fraud and abuse,” and requested comment on a number of detailed questions regarding the sharing of excess capacity with ineligible entities, different methods for allocating costs among entities, providing excess capacity for community use, and what types of guidance are needed. According to our survey, 21 of 60 respondents indicated that their project “definitely” or “probably” will include excess capacity.

Participants Reported That the Benefits Afforded by the Pilot Program Are Worth the Costs of Participating, and They Were Generally Positive about Program Officials

Although there have been some challenges, many pilot participants emphasized the importance of the pilot program in their responses to our survey as well as in comments submitted to FCC. According to our survey, if pilot participants are able to accomplish their pilot project goals: 55 of 57 respondents indicated their project “definitely” or “probably” will have entities that obtain telecommunications or Internet services that would otherwise be unaffordable; 48 of 55 respondents indicated their project “definitely” or “probably” will have entities obtain telecommunications or Internet services that would otherwise be unobtainable due to lack of infrastructure; and 58 of 59 respondents indicated that their project “definitely” or “probably” will have entities upgrade an existing telecommunications or Internet service.

In addition, when asked to consider their current understanding of the costs and administrative requirements of participating in the pilot program, 52 of 57 respondents reported that the pilot program’s benefits will outweigh the costs of participating in the program.

Pilot participants were also generally positive about the usefulness of program officials, in particular their coach. When asked to list the top three things that program officials did well in their administration of the program, respondents provided positive opinions about their communications with program officials, the effort put forth by program officials, and the coaches or the coaching concept. These responses are consistent with those provided to another question that rated program officials—be they FCC, USAC, or a project coach—as the most useful resource for pilot participants. (See fig. 12.)

Pilot participants were also satisfied with their coaches as a source of information. USAC appointed coaches to serve as a project’s direct point of contact with program officials. Of the 61 respondents who provided an opinion, some specifically noted their satisfaction with the ease with which they could contact their coach (53 respondents were “very satisfied” or “somewhat satisfied”) and the level of interaction with their coach (49 respondents were “very satisfied” or “somewhat satisfied”). Coaches were rated somewhat lower on their knowledge of the program (40 respondents were “very satisfied” or “somewhat satisfied”), although this lower rating may be related to the lack of established guidance at the beginning of the program and the need to refer difficult issues to Solix management, USAC, and FCC, depending on the complexity of the issue.

FCC Is Seeking More Input and Providing More Detail on Its Proposed New Program, but It Is Not Clear Whether Planning and Communication Have Been Fully Addressed

In its July 2010 NPRM, FCC proposed a new Health Infrastructure Program that would make available up to $100 million per year to support up to 85 percent of the construction costs of new regional or statewide networks for health care providers in areas of the country where broadband is unavailable or insufficient. In this NPRM, FCC made improvements over previous NPRMs by outlining potential program requirements and requesting comment on the proposed new program. FCC provided much more information than it did when announcing the pilot program and is allowing for stakeholder input into the program’s design. In addition, FCC recognized some of the challenges mentioned in this report and requested comment on potential improvements. In particular, FCC proposed and requested comment on requiring that applicants prove or otherwise certify that broadband at minimum connectivity speeds is unavailable or insufficient to meet their health care needs when applying to the program; requiring applicants to submit letters of agency as part of their application, rather than after they are accepted into the program; having USAC review entity eligibility; providing limited funding for administrative costs; expanding entity eligibility to include off-site administrative offices, off- site data centers, nonprofit skilled nursing facilities, and nonprofit renal dialysis facilities; and providing additional guidance regarding the funding and permitted uses of excess capacity and the allocation of related costs.

However, it remains unclear the extent to which FCC is coordinating with USAC in preparing for this program. The NPRM indicates that USAC will develop a user-friendly Web-based application for participants to use. However, during our conversations with USAC, officials noted that it would take a considerable amount of time and effort to properly develop such systems. FCC indicates in its NPRM that the new programs could be implemented by funding year 2011. If FCC does not better plan the details of the new program before it calls for applications, participants in the Health Infrastructure Program may experience the same delay and difficulties as participants have experienced in the pilot program.

FCC Has Not Followed Key Performance Management Practices, Thus It Lacks the Performance Data to Make Effective Policy Decisions and Implement Program Reforms

FCC Has Attempted to Develop Performance Goals and Measures for the Rural Health Care Program, but They Are Ineffective for Managing Program Performance

We have previously reported that results-oriented organizations commonly perform a number of key practices to effectively manage program performance. In particular, results-oriented organizations implement two key practices, among others, to lay a strong foundation for successful program management. First, these organizations set performance goals to clearly define desired outcomes. Second, these organizations develop performance measures that are clearly linked to the program goals. However, FCC has reversed these two key practices. In 2006, 8 years after FCC first implemented the primary Rural Health Care Program, the Office of Management and Budget (OMB) assessed FCC’s Rural Health Care Program and concluded that the program had no performance goals and measures. In 2007, FCC issued a report and order adopting performance measures for the Rural Health Care Program related to USAC’s processing of applications, paying invoices, and determining appeals. However, FCC stated that it did not have sufficient data to establish performance goals for the Rural Health Care Program in the report and order. Instead of specific performance-related goals, the Rural Health Care Program has operated for 12 years under broad overarching goals, including the statutory goal established by Congress in the 1996 Act, which is to ensure that rural health care providers receive telecommunications services at rates comparable for the same services in urban areas.

Furthermore, the performance measures that FCC adopted for the primary Rural Health Care Program and the pilot program in 2007 fall short when compared with the key characteristics of successful performance measures that we have identified in our past work. Following is a discussion of these characteristics and the extent to which FCC has fulfilled them in developing performance measures: Measures should be tied to goals and demonstrate the degree to which the desired results are achieved. These program goals should, in turn, be linked to overall agency goals. However, as we have previously discussed, the measures that FCC has adopted are not based on such linkage because no specific performance goals have been established. By establishing performance measures before establishing the specific performance goals that it seeks to achieve through the Rural Health Care Program, FCC may waste valuable time and resources collecting the wrong data. FCC receives the data for these performance measures on a quarterly basis from USAC, but without effective performance goals to guide its data collection, it cannot ensure that the data gained from these performance measures are an effective use of resources.

Measures should address important aspects of program performance.

For each program goal, a few performance measures should be selected that cover key performance dimensions and take different priorities into account. For example, measures should be limited to core program activities because an excess of data could obscure rather than clarify performance issues. Performance measures should also cover key governmentwide priorities, such as timeliness and customer satisfaction. FCC’s performance measures appear to address certain key performance dimensions. By selecting just three types of measures—related to USAC’s (1) processing of applications, (2) paying invoices, and (3) determining appeals—there are fewer chances of obscuring the most important performance issues. The measures also appear to take into account such priorities as timeliness and customer satisfaction. For example, the 2007 performance measures include requirements to measure the number of current and pending appeals, and the time that it takes to resolve those appeals. However, again, without first setting specific performance goals defining what the programs are specifically intended to accomplish, FCC cannot be sure that it has adopted the most appropriate performance measures.

Measures should provide useful information for decision making. Performance measures should provide managers with timely, action- oriented information in a format that helps them to make decisions that improve program performance. However, the data collected by these performance measures—such as the number of applications submitted, rejected, and granted—are output, not outcome, oriented. The FCC task force that developed the National Broadband Plan also reported that the performance measures developed for the Rural Health Care Program need to be improved to assess desired program outcomes, such as the impact of the program on patient care. The limited nature of the data obtained by current performance measures, combined with the absence of specific performance goals, raises concerns about the effectiveness of these performance measures for programmatic decision making.

FCC is attempting to improve its performance management by seeking public comment on performance goals and measures for the Rural Health Care Program in its July 2010 NPRM. For example, FCC proposed a specific measure of how program support is being used: that is, requiring beneficiaries to annually identify the speed of the connections supported by the program and the type and frequency of the use of telemedicine applications as a result of broadband access. Although this is a positive step, the NPRM does not specify whether this data collection would be linked to specific connection speed goals that participants should obtain with program funds, and it does not propose what the goal should be for type and frequency of the use of telemedicine applications. While this NPRM could lead to better goals and measures for the Rural Health Care Program, FCC has exhibited a pattern of repeatedly seeking comment on goals and measures for the Rural Health Care Program, which indicates that it does not have a clear vision for what it intends the program to accomplish within the broad statutory framework provided by Congress. FCC has sought public comment on performance goals and measures for the program on two previous occasions that did not result in effective performance goals and measures for the program: In June 2005, FCC issued a NPRM seeking comment on whether specific performance goals were needed and on ways to establish useful outcome, output, and efficiency measures for each of the universal service programs, including the Rural Health Care Program. FCC officials stated that this NPRM led to the 2007 performance measures that we have previously described.

In September 2008, FCC issued a NOI seeking comment on how to more clearly define the goals of the Universal Service Fund programs, including the Rural Health Care Program, and to identify any additional quantifiable performance measures that may be necessary or desirable. FCC officials stated that this NOI led to the July 2010 NPRM, which, again, requests comment on performance goals and measures.

Performance goals and measures are particularly important for the Rural Health Care Program, because they could help FCC to make well-informed decisions about how to address the trends that we have previously described. If FCC does use information from the latest NPRM to develop specific performance goals and measures, it should focus on the results that it expects its programs to achieve. We have identified the following practices for developing successful performance goals and measures: create a set of performance goals and measures that addresses important dimensions of a program’s performance and balance competing priorities, use intermediate goals and measures to show progress or contribution to include explanatory information on the goals and measures, develop performance goals to address mission-critical management show baseline and trend data for past performance, identify projected target levels of performance for multiyear goals, and link the goals of component organizations to departmental strategic goals.

Clearly articulated, outcome-based performance goals and measures are important to help ensure that the Rural Health Care Program meets the guiding principles that Congress has set forth.

Without Effective Performance Goals and Measures, FCC Cannot Reliably Evaluate Program Performance, Which Could Lead to a Repeat of Its Past Management Weaknesses

After implementing the key performance management practices that we have previously discussed—establishing effective performance goals and measures—results-oriented organizations implement a third, key practice: that is, evaluating the performance of their programs. Measuring performance allows these organizations to track progress toward goals and provides managers with the crucial performance data needed to make management decisions. We have previously reported that performance data can have real value only when used to identify the gap between a program’s actual performance level and the performance level identified as its goal. Again, without specific performance goals and effective performance measures, FCC cannot identify program performance gaps and is unlikely to conduct evaluations that are useful for formulating policy decisions.

FCC has not formally evaluated the performance of the primary Rural Health Care Program to determine whether it is meeting the needs of rural health care providers, and it may lack the tools to evaluate the pilot program—such as an effective progress reporting mechanism and an evaluation plan. To its credit, FCC has stated that it intends to evaluate the pilot program after its completion. However, it is unclear whether FCC has effective evaluation tools for conducting a pilot program evaluation that will be useful for making policy decisions about the future of the Rural Health Care Program. To track the progress of pilot projects, FCC requires pilot program participants to complete quarterly reports that are filed with FCC and USAC, but it is unclear whether these reports are effective tools for evaluating pilot program performance for the following reasons: Quarterly report data are not quantitative. Quarterly reports collect data that are mostly qualitative (e.g., a narrative description of a project’s network and how the network will be sustained) instead of quantitative. While qualitative data can help officials understand project progress on an individual basis, the information is not objective or easily measured.

FCC has not involved key stakeholders. We have previously reported that stakeholder and customer involvement helps agencies to ensure that efforts and resources are targeted at the highest priorities. However, key stakeholders and pilot participants are not involved in ensuring that quarterly reports are providing the most useful information possible. Pilot program coaches, who guide pilot participants through the program’s administrative processes, and USAC officials said that FCC has not told them how the reports will be used to evaluate pilot program progress. USAC and the pilot program coaches work directly with participants and without understanding how these reports will be used, they are unable to effectively guide participants into providing the most useful evaluation information possible. Additionally, of the 45 pilot program survey respondents that provided an opinion, 26 said that they receive too little feedback on their quarterly reports.

Quarterly reports may require too much information. Of the 58 pilot program survey respondents that provided an opinion, 28 said that too much information is required in quarterly reports. As we have previously discussed, an excess of data can obscure rather than clarify performance.

FCC officials told us that they have learned lessons from using these quarterly reports, and that, as part of the 2010 NPRM, FCC requested public comment on a similar reporting requirement for the proposed Health Infrastructure Program.

Furthermore, despite FCC’s intentions to evaluate the pilot program, officials have not yet developed an evaluation plan for the pilot program. FCC officials told us that this is because the pilot program is still under way, and that FCC will plan the evaluation when the pilot program is closer to completion (as we previously stated, the deadline for participants in the pilot program to select a vendor and request a funding commitment from USAC is June 30, 2011). However, we have previously reported that when conducting pilot programs, agencies should develop sound evaluation plans before program implementation—as part of the design of the pilot program itself—to increase confidence in results and facilitate decision making about broader application of the pilot program. We have previously identified the following key features of sound evaluation plans: well-defined, clear, and measurable objectives; measures that are directly linked to specific program objectives; criteria or standards for determining program performance; clearly articulated methodology and a strategy for comparing results with a clear plan that details the type and source of data necessary to evaluate the program, methods for data collection, and the timing and frequency of data collection; a detailed data-analysis plan to track the program’s performance and evaluate its final results; and a detailed plan to ensure that data collection, entry, and storage are reliable and error-free.

The lack of a documented evaluation plan for the pilot program increases the likelihood that FCC will not collect appropriate or sufficient data, which limits understanding of pilot program results. Without this understanding, FCC will be limited in its decision making about the pilot program’s potential broader application to FCC’s proposed future programs.

The National Broadband Plan states that for all four universal service fund programs, including the Rural Health Care Program, “there is a lack of adequate data to make critical policy decisions regarding how to better utilize funding to promote universal service objectives.” FCC has not effectively followed the three key performance management practices discussed in this report and has not obtained the data that it needs to make critical policy decisions and successfully manage the program. Furthermore, FCC has proposed two new programs under the Rural Health Care Program in its 2010 NPRM (the Health Broadband Services Program and the Health Infrastructure Program), even though the National Broadband Plan states that FCC does not have the data to make critical policy decisions on how to better use its funds. In our previous work, we have reported that results-oriented organizations recognize that improvement goals should flow from a fact-based performance analysis. However, the proposed improvements to the Rural Health Care Program are not based on a fact-based performance analysis because the performances of the primary Rural Health Care Program and the pilot program have not been evaluated. FCC officials told us that they believe the proposals set forth in the July 2010 NPRM are “positive first steps” toward creating improvements to performance analysis.

Because FCC has not determined what the primary Rural Health Care Program and the pilot program are specifically intended to accomplish and how well the programs are performing, it remains unclear how FCC will make informed decisions about the new programs described in the July 2010 NPRM. Moreover, as new technologies are developed, measuring the performance and effectiveness of existing programs is important so that decision makers can design future programs to effectively incorporate new technologies, if appropriate. If FCC does not institute better performance management tools—by establishing effective performance goals and measures, and planning and conducting effective program evaluations—FCC’s management weaknesses will likely continue to affect the current Rural Health Care Program, and will likely carry forward into the design and operation of proposed Rural Health Care programs.

Conclusions

Over the first 12 years of its Rural Health Care Program, FCC has distributed more than $327 million to rural health care providers to assist them in purchasing telecommunications and information services. FCC and USAC have been particularly successful in disbursing committed funds in the primary Rural Health Care Program, and FCC has generally seen slow but steady growth in both the amounts of annual disbursements and the number of annual applicants to the primary program.

However, since the Rural Health Care Program’s inception, FCC has not provided the program with a solid performance management foundation. FCC could better inform its decision making and improve its stewardship of the Rural Health Care Program by incorporating effective performance management practices into its regulatory processes. FCC has not conducted a comprehensive needs assessment to determine the needs of rural health care providers, has no specific goals and measures for the program to guide its management decisions, and has not evaluated how well the program is performing. FCC’s attempts to improve the program over time, including the 2006 pilot program, have not been informed by a documented, fact-based needs assessment; consultations with knowledgeable stakeholders, including other government agencies; and performance evaluations. Despite FCC’s efforts to improve the program, a significant number of eligible rural health care providers currently do not use the primary Rural Health Care Program, and FCC’s management of the pilot program has often led to the delays and difficulties reported by pilot participants.

We found that a number of rural health care providers depend on the support they receive from the primary Rural Health Care Program, and that most pilot program participants are seeking services that they believe would have been otherwise unaffordable. It is possible that FCC’s proposed changes to the Rural Health Care Program will increase participation by rural health care providers, thus increasing the amount of funding committed by the Rural Health Care Program and, ultimately, increasing the universal service fees paid by consumers on their telephone bills. Changes in FCC’s approach to performance management could help ensure that higher telephone bills are justified; that program resources are targeting the needs of rural health care providers; and that the program, in fact, is helping our nation to realize more widespread use of telemedicine technologies.

Recommendations for Executive Action

To improve its performance management of the Rural Health Care Program, we recommend that the Chairman of the Federal Communications Commission take the following five actions. If FCC does develop any new rural health care programs under the Universal Service Fund—such as the proposed Health Care Broadband Access Fund and the Health Care Broadband Infrastructure Fund—these steps should be taken before implementing any new programs or starting any new data collection efforts: Conduct an assessment of the current telecommunications needs of rural health care providers.

Consult with USAC, other federal agencies that serve rural health care providers (or with expertise related to telemedicine), and associations representing rural health care providers to incorporate their knowledge and experience into improving current and future programs.

Develop effective goals, and performance measures linked to those goals, for all current and future programs.

Develop and execute a sound performance evaluation plan for the current programs, and develop sound evaluation plans as part of the design of any new programs before implementation begins.

For any new program, ensure that FCC’s request for applications to the program clearly (1) articulates all criteria for participating in the program and any weighting of that criteria, (2) details the program’s rules and procedures, (3) outlines the program’s performance goals and measures, and (4) explains how participants’ progress will be evaluated.

Agency Comments and Our Evaluation

We provided a draft of this report to the Federal Communications Commission and the Universal Service Administrative Company for their review and comment.

In its written comments, FCC did not specifically agree or disagree with our recommendations but discussed planned and ongoing actions to address them. FCC agreed that it should continue to examine and work to improve the Rural Health Care Program to ensure that the program is effectively and efficiently achieving its statutory goals. In response to our first recommendation that FCC conduct an assessment of the current needs of rural health care providers, FCC stated that it is gathering information about health care needs, including needs assessments performed by other governmental agencies. FCC also stated that going forward, it is committed to developing benchmarks to define when needs have increased or decreased, applying needs assessment results to resource allocation decisions, and integrating information from other resources available to help address the need. FCC’s efforts to obtain information and assessments from other agencies and stakeholders are encouraging. We continue to believe, however, that FCC would benefit from conducting its own assessment of the telecommunications needs of the rural health care providers eligible under its Rural Health Care Program.

In response to our second recommendation that FCC consult with stakeholders and incorporate their knowledge into improving current and future programs, FCC stated that it is committed to maximizing collaboration efforts with federal and other knowledgeable stakeholders and that it will work closely with USAC to prepare for the new program. FCC included in its comments an October 2010 statement from the Office of the National Coordinator for Health IT, Department of Health and Human Services, about collaborative efforts with FCC and other federal agencies. In response to our third recommendation that FCC develop effective performance goals and measures, FCC concurred with the need to develop quantifiable performance measures. However, FCC did not specifically state whether it concurred with our recommendation to develop effective goals and to link performance measures to those goals. We continue to believe that FCC should develop program performance goals first, and then develop performance measures and link them to those goals. In response to our fourth recommendation that FCC develop and execute effective performance evaluation plans for the current and future programs, FCC stated that it intends to conduct an evaluation of the pilot program after it is concluded. While FCC did not address evaluation of the current primary program, it stated that for any future enhancements to the program, it is committed to developing and executing sound performance evaluation plans, including key features that we identified in our report. In response to our fifth recommendation that FCC identify critical program information, such as criteria for funding, and prioritization rules in its call for applications for any new programs, FCC stated that the July 2010 NPRM discusses these elements in detail. While we appreciate FCC’s efforts to better detail proposed programs in its NPRM, we continue to believe that FCC should detail the requirements for participation in the call for applications to any future programs. FCC’s full comments are reprinted in appendix III.

In its written comments, USAC stated that it will work with FCC to implement any orders or directives that FCC issues in response to our recommendations. USAC’s full comments are reprinted in appendix IV. USAC also provided technical comments that we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Chairman of the Federal Communications Commission, the Acting Chief Executive Officer of the Universal Service Administrative Company, and other interested parties. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you have any questions about this report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to address the following questions: (1) How has the Federal Communications Commission (FCC) managed the primary Rural Health Care Program to meet the needs of rural health care providers, and how well has the program addressed those needs? (2) How have FCC’s design and implementation of the pilot program affected participants? and (3) What are FCC’s performance goals and measures for the Rural Health Care Program, and how do these goals compare with the key characteristics of successful performance goals and measures?

Background Research

We conducted the following background research that helped inform all of our reporting objectives. Specifically, we reviewed: prior GAO reports on other Universal Service Fund programs; FCC’s Universal Service Monitoring Reports on the Rural Health Care documentation from FCC and the Universal Service Administrative Company (USAC) on the structure and operation of the Rural Health Care Program and pilot program; and FCC documents, including FCC orders and requests for comment on the Universal Service Fund programs, as well as written comments submitted in response to these requests.

In addition, we interviewed: officials from FCC’s Office of Managing Director and Wireline Competition Bureau to identify actions undertaken to address previously identified problems and plans to address issues of concern in the programs and officials from USAC’s Rural Health Care Division and Solix, Inc., to collect information on program operations and USAC’s actions to implement prior FCC orders on the primary Rural Health Care Program and pilot program.

Analysis of Primary Rural Health Care Program Data

To evaluate how the primary Rural Health Care Program was managed to meet the needs of rural health care providers, we examined trends in the demand for and use of primary Rural Health Care Program funding from data we obtained from USAC’s Packet Tracking System (PATS), which is used to keep track of primary Rural Health Care Program applications, and the Simplified Invoice Database System (SIDS), which is used to keep track of program disbursements. When analyzing and reporting on the data, we considered the limitations on how data can be manipulated and retrieved from both the PATS and SIDS databases since these systems were designed to keep track of applications and finances and not to be data retrieval systems. We assessed the reliability of the data by questioning officials about controls on access to the system and data back- up procedures. Additionally, we reviewed the data sets provided to us for obvious errors and inconsistencies. On the basis of this assessment, we determined that the data were sufficiently reliable to describe broad trends in the demand for and use of Rural Health Care Program funding.

We obtained the following data—including annual and cumulative figures—for funding years 1998 through 2009: the number and characteristics of applicants, including their entity type, the type of service requested, and location; the dollar amount of funding commitments and disbursements by entity type, type of service requested, and state; the number of commitments and disbursements by state; and the amount of money committed but not disbursed by entity type and type of service requested.

To provide these data, USAC performed queries on the PATS and SIDS systems and provided the resulting reports to us in July 2010. Data from both systems can change on a daily basis as USAC processes applications for funding and reimbursement, applicants request adjustments to requested or committed amounts, and other actions are taken. As a result, the data we obtained and reported on reflect the program status at the time that USAC produced the data, and thus may be somewhat different if we were to perform the same analyses with data produced at a later date.

Interviews to Assess How Well the Primary Rural Health Care Program Addressed Health Care Provider Needs

To assess how well the primary Rural Health Care Program addressed the needs of rural health care providers, we interviewed FCC and USAC officials to determine how the program was designed to address rural health care provider needs. We reviewed relevant documentation, including FCC orders, notices of proposed rulemaking, and FCC’s National Broadband Plan. We also reviewed comments and reply comments to the record to gain insight into the public perception of how the program was addressing needs. Furthermore, we interviewed representatives from stakeholder groups, including the American Telemedicine Association, the National Organization of State Offices of Rural Health, the National Rural Health Association, the Center for Telehealth and E-Health Law, and the National Telecommunications Cooperative Association, to gain their perspective on the primary Rural Health Care Program.

Survey of Pilot Program Participants

To obtain information on how FCC’s design and implementation of the pilot program affected participants, we conducted a Web-based survey of pilot projects. For a more complete tabulation of the survey results, see the e-supplement to this report. To develop the survey questionnaire, we reviewed comments submitted to FCC by representatives from the pilot projects and other stakeholders in response to FCC requests for feedback on the pilot program. We also interviewed pilot project representatives who were in various stages of the pilot program processes as well as FCC, USAC, Solix, and stakeholder groups knowledgeable about the program and issues of concern to participants. We designed draft questionnaires in close collaboration with GAO survey specialists. We conducted pretests with four pilot projects that were in various stages of the pilot program processes to help further refine our questions, develop new questions, and clarify any ambiguous portions of the survey. We conducted these pretests in person and by telephone. In addition, we had FCC and USAC review the survey prior to it being sent to the pilot participants.

We sent our survey to all 61 of the pilot projects that had recent contact information on file with USAC, as of June 2, 2010. We excluded the Puerto Rico project because at the time of our survey, it was the only project that had been withdrawn from the program for an extended period of time; thus, although we tried, locating a contact with knowledge of the program was not possible.

Our goal was to obtain responses from individuals with knowledge of and experience with the tasks related to the pilot program—such as preparing forms and responding to information requests—for each sampled entity. Our data set included the name and contact information for each project’s project coordinator and associate project coordinator. We asked USAC coaches to identify who they interacted with the most on each project (project coordinator, associate project coordinator, or someone else), and we sent the survey to that individual. If that individual was unable to complete the survey, we asked the other contact (project coordinator, associate project coordinator, or someone else) to complete the survey. One respondent was the primary point of contact for two projects, but a separate survey was completed for each project.

We notified the 61 preidentified contacts on June 2, 2010, by e-mail that the survey was about to begin and updated contact information as needed. We launched our Web-based survey on June 8, 2010, and asked for responses to be submitted by June 18. Log-in information was e-mailed to all participants. We contacted by telephone and e-mailed those who had not completed the questionnaire at multiple points during the data collection period, and we closed the survey on July 2, 2010. All 61 projects submitted a completed questionnaire with usable responses for an overall response rate of 100 percent.

We also followed up with certain projects on the basis of survey responses to gain additional information about plans for using excess capacity, as well as the extent to which the project was impacted by federal coordination with the pilot program.

While all pilot projects were selected for our survey, and, therefore, our data are not subject to sampling errors, the practical difficulties of conducting any survey may introduce nonsampling errors. For example, differences in how a particular question is interpreted, the sources of information available to respondents, or the types of people who do not respond to a question can introduce errors into the survey results. We included steps in both the data collection and data analysis stages to minimize such nonsampling errors. As we previously indicated, we collaborated with GAO survey specialists to design draft questionnaires, and versions of the questionnaire were pretested with four members of the surveyed population. In addition, we provided a draft of the questionnaire to FCC and USAC for their review and comment. From these pretests and reviews, we made revisions as necessary to reduce the likelihood of nonresponse and reporting errors on our questions. We examined the survey results and performed computer analyses to identify inconsistencies and other indications of error and addressed such issues, where possible. A second, independent analyst checked the accuracy of all computer analyses to minimize the likelihood of errors in data processing. In addition, GAO analysts answered respondent questions and resolved difficulties that respondents had in answering our questions. For certain questions that asked respondents to provide a narrative answer, we created content categories that covered more than 90 percent of the narrative responses provided, and asked two analysts to independently code each response into one of the categories. Any discrepancies in the coding of the two analysts were discussed and addressed by the analysts.

Interviews to Assess Federal Coordination

To determine the extent to which FCC coordinated with other federal agencies when designing and implementing the pilot program, we interviewed FCC officials regarding the nature of their coordination with other agencies, and followed up with representatives from other federal agencies, including the Department of Health and Human Services (Agency for Healthcare Research and Quality, Centers for Disease Control and Prevention, Centers for Medicare and Medicaid Services, Health Resources and Services Administration, Indian Health Service, National Library of Medicine, Office of the Assistant Secretary for Preparedness and Response, and Office of the National Coordinator for Health Information Technology); the U.S. Department of Agriculture’s Rural Utilities Service; and the Department of Commerce’s National Telecommunications and Information Administration. We reviewed relevant documentation and assessed the extent to which FCC coordinated with other agencies against criteria for coordination established in prior GAO reports.

Document Review and Interviews with FCC and USAC on Performance Goals and Measures

To determine the performance goals and measures of the Rural Health Care Program and how these measures compare with the key characteristics of successful performance measures, we reviewed the Telecommunications Act of 1996. We then reviewed our past products and science and evaluation literature to identify effective practices for setting performance goals and measures. We compared this information with the program goals and measures that FCC set forth in agency documentation—including FCC orders, notices of proposed rulemaking, strategic plans, and performance and accountability reports. We also reviewed the Office of Management and Budget’s Program Assessment Rating Tool 2006 report on the Rural Health Care Program’s effectiveness. In addition, we interviewed officials from FCC’s Wireline Competition Bureau and Office of Managing Director, and officials from USAC’s Rural Health Care Division to obtain their views on plans to implement Rural Health Care Program performance goals and measures.

Appendix II: 2008 Commitments to Applicants, by State and Territory

Appendix III: Comments from the Federal Communications Commission

Appendix IV: Comments from the Universal Service Administrative Company

Appendix V: GAO Contact and Staff Acknowledgments

Mark L. Goldstein, (202) 512-2834 or goldsteinm@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Faye Morrison (Assistant Director), Elizabeth Curda, Lorraine Ettaro, Cheron Green, Amy Higgins, Crystal Huggins, Catherine Hurley, Linda Kohn, Armetha Liles, Elizabeth Marchak, Valerie Melvin, John Mingus Jr., Sara Ann Moessbauer, Charlotte Moore, Joshua Ormond, Madhav Panwar, Carl Ramirez, Amy Rosewarne, Cynthia Saunders, Michael Silver, Teresa Tucker, and Mindi Weisenbloom made key contributions to this report.