Telecommunications: FCC's Performance Management Weaknesses Could Jeopardize Proposed Reforms of the Rural Health Care Program
Published: 2010-11-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Telemedicine offers a way to improve health care access for patients in rural areas. The Federal Communications Commission's (FCC) Rural Health Care Program, established in 1997, provides discounts on rural health care providers' telecommunications and information services (primary program) and funds broadband infrastructure and services (pilot program). GAO was asked to review (1) how FCC has managed the primary program to meet the needs of rural health care providers, and how well the program has addressed those needs; (2) how FCC's design and implementation of the pilot program affected participants; and (3) FCC's performance goals and measures for both the primary program and the pilot program, and how these goals compare with the key characteristics of successful performance goals and measures. GAO reviewed program documents and data, interviewed program staff and relevant stakeholders, and surveyed all 61 pilot program participants with recent participation in the program.
section_title: What GAO Found
paragraphs: FCC has not conducted an assessment of the telecommunications needs of rural health care providers as it has managed the primary Rural Health Care Program, which limits FCC's ability to determine how well the program has addressed those needs. Participation in the primary program has increased, and some rural health care providers report that they are dependent on the support received from the program. For example, a provider in Alaska has used program funds to increase the use of telemedicine, which has reduced patient wait times and travel costs. FCC has been successful in disbursing over 86 percent of all committed funds. However, FCC has disbursed only $327 million in total over the 12 years of the primary program's operation-- less than any single year's $400 million funding cap. FCC has frequently stated that the primary program is underutilized and has made a number of changes to the program, including the creation of the pilot program. Currently, FCC is proposing to replace portions of the primary program with a new broadband services program. However, without a needs assessment, FCC cannot determine how well the current program is targeting those needs--and whether the program is, in fact, underutilized--or ensure that a new program will target needs any better. FCC's poor planning and communication during the design and implementation of the pilot program caused delays and difficulties for pilot program participants. FCC did not consult with the program's administrator, other federal agencies, or relevant stakeholders prior to announcing the program, nor did it request public comment on its design. In addition, FCC called for applications to participate in the pilot program before it fully established pilot program requirements. FCC added additional program requirements after the pilot program began, and survey respondents indicated that program guidance was not provided in an effective manner. Despite these difficulties, most participants were positive about the assistance provided by program officials and reported that the benefits they anticipate receiving from the pilot program outweigh the costs of participating. However, the entire program has been delayed and projects have struggled to meet requirements that were not clearly defined at the beginning of the program. FCC has not developed specific performance goals for the Rural Health Care Program and has developed ineffective performance measures. The performance measures are limited for a number of reasons, the most important of which is that FCC has set no specific performance goals to which to link them. In addition, FCC has not evaluated the performance of the primary Rural Health Care Program and has no evaluation plan for the pilot program. Without reliable performance information, FCC does not have the data that it needs to make critical policy decisions about the overall Rural Health Care Program. If FCC does not correct these deficits in performance management, it may perpetuate the same performance management weaknesses in its stewardship of the new rural health care programs that it has proposed.
section_title: What GAO Recommends
paragraphs: GAO recommends that the FCC Chairman assess rural health care providers' needs, consult with knowledgeable stakeholders, develop performance goals and measures, and develop and execute sound performance evaluation plans. In its comments, FCC did not agree or disagree with the recommendations, but discussed planned and ongoing actions to address them.

section_title: Letter
paragraphs: Some of the most promising technologies to arise out of our nation’s  transition to broadband involve “telemedicine,” particularly for patients in  rural areas of the country. Telemedicine technologies can allow rural  patients to receive, through remote access, medical diagnosis or patient  care, often from specialists who are located in urban areas or university  hospitals. Increased use of video consultation, remote patient monitoring,  and electronic health records enabled by telemedicine technologies hold  the promise of improving health care quality, safety, and efficiency. The  Federal Communications Commission’s (FCC) Rural Health Care  Universal Service Support Mechanism—or Rural Health Care Program— was created pursuant to the Telecommunications Act of 1996 (1996 Act)  and enables rural health care providers to receive (1) telecommunications  services at rates comparable to that of their urban counterparts and   (2) access to the advanced telecommunications and information services  necessary for health care delivery. Access to reasonably priced  telecommunications services and Internet access services affords rural  health care providers the ability to provide important telemedicine  technologies that can improve the care of patients while maximizing  limited resources.
Despite these benefits, FCC has stated that its Rural Health Care Program  is underutilized, in part, because rural health care providers’ needs have  shifted away from discounted telecommunications and Internet services,  and toward the broadband networks and facilities needed to support  advanced telemedicine applications. Thus, in 2006, FCC established a  separate pilot program within the Rural Health Care Program to provide  funding for broadband infrastructure and services. Also, in March 2010, at  the direction of Congress, an FCC task force developed and released a  National Broadband Plan to provide a road map for attaining universal  access to broadband capability. As a result of recommendations in the  National Broadband Plan, FCC is currently reviewing its design of the  Rural Health Care Program and has proposed two new rural health care  programs—the Health Broadband Services Program and the Health  Infrastructure Program—in a July 2010 Notice of Proposed Rulemaking  (NPRM). FCC sought comment on these and other reforms, which could  be implemented by the beginning of the next funding year on July 1, 2011.
In response to your request that we examine the operation of the Rural  Health Care Program, this report addresses three main questions:    How has FCC managed the primary Rural Health Care Program to meet  the needs of rural health care providers, and how well has the program  addressed those needs?
How have FCC’s design and implementation of the pilot program affected  participants?
What are FCC’s performance goals and measures for the Rural Health Care  Program, and how do these goals compare with the key characteristics of  successful performance goals and measures?
For each of these questions, we reviewed FCC documents, including FCC  orders and requests for comment on the Rural Health Care Program, as  well as written comments submitted in response to these requests. We also  interviewed FCC staff and staff of the Universal Service Administrative  Company (USAC)—the not-for-profit corporation that administers the  Rural Health Care Program under a memorandum of understanding  (MOU) with FCC. To provide information on the design, operation, and  trends of the primary Rural Health Care Program, we analyzed data from  USAC on applications, funding commitments, and disbursements for the  first 12 years of the primary Rural Health Care Program (1998 to 2009). On  the basis of interviews with USAC officials to understand how these data  were handled, stored, and protected, we determined that the data were  sufficiently reliable for the purposes specified. To provide information on  the pilot program, we conducted a Web-based survey of representatives  from all 61 pilot projects that had recent contact information on file with  USAC at the time of our survey to obtain their views on program  requirements and on how to improve the program, among other things.  Our survey response rate was 100 percent. This report does not contain all  of the results from the survey; our questionnaire and a more complete  tabulation of the results can be viewed in an e-supplement to this report.  To provide information about performance goals and measures, we  reviewed FCC documentation on the agency’s performance goals and  measures for the Rural Health Care Program and compared this  information with literature on results-oriented management and effective  practices for setting performance goals and measures. Additionally, we  interviewed officials from other federal agencies, including the  Department of Health and Human Services (HHS), the U.S. Department of  Agriculture (USDA), and the Department of Commerce, to collect  information on FCC’s collaboration efforts on the Rural Health Care  Program. We also interviewed representatives from telecommunications  and rural health care stakeholder organizations to learn about the impact  of the program on their members. See appendix I for additional  information on our scope and methodology.
We conducted this performance audit from August 2009 to November  2010, in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: A key goal of universal service is to ensure affordable telecommunications  services to consumers living in high-cost areas, low-income consumers,  eligible schools and libraries, and rural health care providers. Universal  service programs are funded by statutorily mandated payments into the  Universal Service Fund by companies that provide interstate and  international telecommunications services. These payments are  deposited into the federal Universal Service Fund, from which  disbursements are made for the various federal universal service  programs, including the Rural Health Care Program. Companies generally  pass their universal service costs along to consumers through a universal  service fee on customers’ telephone bills.
FCC’s current Rural Health Care Program is made up of three components  that fund different benefits. As figure 1 illustrates, the first two  components—the Telecommunications Fund and the Internet Access  Fund—are commonly discussed together as the “primary Rural Health  Care Program.” Both components in the primary Rural Health Care  Program offer discounts on services provided to a single site. In contrast,  the third component—the pilot program—encourages health care  providers to form comprehensive, multisite, state and regional dedicated  health care networks.
Figure 2 shows how the components in the current Rural Health Care  Program may change if FCC adopts the proposed reforms described in its  July 2010 NPRM. As the figure illustrates, the Health Broadband Services  Program would replace the Internet Access Fund (and raise the discount  percentage). A new Health Infrastructure Program would make available  up to $100 million per year to support up to 85 percent of the construction  costs of new regional or statewide networks for health care providers in  areas of the country where broadband is unavailable or insufficient. This  $100 million would be part of the overall $400 million annual spending cap  that covers the Rural Health Care Program as a whole and that FCC  established in 1997.
In managing the program, FCC oversees USAC—the not-for-profit  corporation that administers the program. USAC uses its subcontractor,  Solix, Inc., to carry out certain key aspects of the program, such as  reviewing and processing funding applications. An MOU between FCC and  USAC as well as FCC orders and rules set forth the roles and  responsibilities of FCC and USAC in the management, oversight, and  administration of the Rural Health Care Program. (See the sidebar on this  page for examples of benefits provided by the Rural Health Care  Program.)
USAC has highlighted the impct of the Rl  Helth Cre Progrm on the Kodik Are  Ntive Assocition (KANA). KANA i   nonprofit corportion tht provide helth nd  ociervice for the Alas Ntive of the  Konig region. According to the Informtion  Stem Mger of KANA, the support  received from the primry Rl Helth Cre  Progrm has “revoltionized telehelth  ervice” t KANA. KANA ptient once hd to  wit etween 6 nd 9 month to ee n epecilit, but telemedicine has redced  ptient wit time 2 week nd has redced  trvel co, ince mny ptient viite  condcted remotely y other helth cre  provider. For exmple,  phyicin in  Anchorge was able to ass helth ide in  Kotzebue perform  surgery when evere  wether mde ir trvel impossle. The  Informtion Stem Mger reported tht  withosupport from the primry Rl Helth  Cre Progrm, KANA wold e forced to go  back to using dil-ervice. Withot thi  support, “it wold e difficlt to fford even the  llet connection etween the villge nd  KANA.” satellite service charges, so that rural and urban prices are comparable  within each state. Second, to support advanced telecommunications and  information services, the Internet Access Fund offers most rural health  care providers a 25 percent flat discount on monthly Internet access  charges. Eligible rural health care providers can apply for support from  both the Telecommunications Fund and the Internet Access Fund.  However, FCC has stated that rural health care providers have not  participated at the rate it had expected.
The steps that applicants must carry out to obtain support from one or  both components of the primary Rural Health Care Program are illustrated  in figure 3.
FCC created the third component of the current Rural Health Care  Program, the pilot program, in September 2006 after acknowledging that  the primary Rural Health Care Program was “greatly underutilized.” FCC  explained that “although there are a number of factors that may explain  the underutilization” of the program, it was “apparent that health care  providers continue to lack access to the broadband facilities needed to  support…advanced telehealth applications.” The pilot program funds   85 percent of the costs of deploying dedicated broadband networks  connecting rural and urban health care providers, including the cost of  designing and installing broadband networks that connect health care  providers in a state or region, as well as the costs of advanced  telecommunications and information services that ride over that network.  This is in contrast to the primary Rural Health Care Program, which  provides discounts only on monthly recurring costs for  telecommunications services or Internet access to rural health care  providers. The pilot program also provides funding for the cost of  connecting state or regional networks to Internet2 or National  LambdaRail—two national networks that connect government research  institutions as well as academic, public, and private health care  institutions—and the costs of connecting to the public Internet. Any  eligible public and nonprofit health care provider—whether located in an  urban or a rural area—was eligible to apply for funding when the pilot  program was announced. However, the program rules required that  applicants’ proposed networks include at least a de minimis number of  public and nonprofit health care providers that serve rural areas.
FCC received 81 applications from projects seeking to participate in the  pilot program. In November 2007, FCC announced that it had accepted 69  of the 81 projects into the program and capped total funding for all of the  pilot projects at roughly $418 million over 3 years. Since then, a few  projects have merged and 1 project has withdrawn from the program. The  size and scope of the remaining projects vary widely. For example, the  Illinois Rural HealthNet project is using pilot program funds to pay for the  installation of approximately 1,250 miles of buried fiber. The purpose of  this fiber is to create the backbone of a network that will connect rural  critical access hospitals, health clinics, and community mental health  centers to specialists throughout the state and nation. In contrast, a  project in Wisconsin is planning to use the pilot program funds to link two  existing fiber systems to establish connections between four hospitals that  allow health care specialists to transmit images between facilities. (See the  sidebar on this page for other examples of pilot projects.)
USAC administers the pilot program pursuant to FCC’s rules. Each pilot  project must designate a project coordinator and associate project  coordinator, who manage the administrative aspects of the program for  the project and submit the required forms. USAC provides each project  with a “coach”—that is, a designated Solix staff person who works closely  with a pilot project to assist the project through the program’s  administrative requirements and processes. With some exceptions, the  pilot program forms and administrative processes are the same as those  previously described in the primary Rural Health Care Program. However,  pilot participants pay 15 percent of eligible costs (and all ineligible costs),  and the pilot program funds up to 85 percent of eligible costs. In addition,  pilot participants must meet additional requirements before they can  receive funding:    The lead entity in charge of the pilot project must obtain a letter of agency  from every entity participating in its project. This letter authorizes the lead  entity to act on the other entity’s behalf in all matters related to the pilot  program.
Pilot participants must develop a sustainability plan describing how the  project will be self-sustaining in the future, to include network ownership  and membership arrangements, and describing sources of future support.
Pilot participants are required to submit quarterly progress reports  describing the status of their project.
In February 2010, FCC’s Wireline Competition Bureau extended by 1 year,  to June 30, 2011, the deadline for participants in the pilot program to  submit to USAC requests for funding commitments.
section_title: FCC Has Not Performed the Analysis Necessary to Ensure That the Primary Rural Health Care Program Meets the Needs of Rural Health Care Providers
subsections: section_title: Participation in the Program, Although Increasing, Has Not Met FCC Projections and over Half of All Program Funds Are Used in Alaska
paragraphs: Annual disbursements from the primary Rural Health Care Program have  increased from 1998 through 2009, yet they have never approached FCC’s  original projections for participation. Figure 4 shows the total amount of  funds that have been disbursed for the primary Rural Health Care Program  from 1998 to 2009.
USAC disbursed just over $327 million for the primary program from 1998  through 2009. Thus, as figure 4 illustrates, total program expenditures in 12  years of disbursements have not yet reached the single year funding cap of  $400 million. Also, as of September 2010, USAC has disbursed just over  $26 million for the pilot program. Therefore, USAC has disbursed less  than $400 million for all three components of the Rural Health Care  Program since the program began in 1998. (FCC does not collect $400  million each year from telecommunications carriers for this program, but  rather bases collections only on projected expenditures. FCC uses a  quarterly evaluation of health care provider demand to assess how much  telecommunications companies must contribute to the Universal Service  Fund each quarter. This means that if FCC’s proposed reforms create more  participation in the program, telecommunications companies would need  to pay more in Universal Service Fund contributions. Telecommunications  companies would likely pass these costs on to consumers through higher  universal service fees in consumers’ telephone bills.)
According to USAC data, primary Rural Health Care Program funding was  disbursed to all of the types of rural health care providers designated by  statute as eligible to participate in the program. As figure 5 illustrates, over  68 percent of total applicants in 2008 were either rural health clinics or  not-for-profit hospitals.
As with disbursements, the number of applicants to the primary Rural  Health Care Program has generally increased since the program began.  Figure 6 shows the number of rural health care providers that have applied  to the primary Rural Health Care Program have increased from a low of  1,283 applicants in 1999 to 4,014 in 2009.
Similarly, the number of funding commitments issued to participants in  the primary Rural Health Care Program has exhibited a slow, steady  increase over time from 799 funding commitments in 1998 to 6,790 in 2008.  Figure 7 shows the number of funding commitments by the type of service  requested (e.g., telecommunications services or Internet access  services).
Funding commitments have varied considerably among applicants within  the states and territories, with almost 55 percent of the funding going to  applicants in Alaska. Disbursements range from over $178 million for  Alaska to none for three states (Connecticut, New Jersey, and Rhode  Island). Health care providers in Wisconsin received the second-largest  disbursement, approximately $18.5 million (almost 5.7 percent) of all  primary Rural Health Care Program funding. For a snapshot of funding to  applicants by state and territory, see appendix II, which contains the  numbers of applicants and amounts committed by state for 2008. Table 1  shows the total amount of money that has been committed and disbursed  to applicants, by state, over the program’s history.
Figure 8 shows the total dollar amount disbursed across the United States  for funding year 2008, by ZIP code, illustrating the wide variation in  geographic use and the heavy concentration of funding in Alaska.
According to FCC and USAC staff, health care providers in Alaska  dominate use of the primary Rural Health Care Program because Alaska’s  rural areas often require expensive satellite telecommunications services.  Alaska’s vast size, harsh winter weather, and sparse population make fiber  networks and other technologies either too expensive or too infeasible.  Some wireless technologies also can be challenging, since Alaskan terrain  often includes mountains or forests that can obstruct line-of-sight  transmission. As a result, satellite is often the most feasible option for  many rural communities in Alaska. Although the cost of  telecommunications service in rural areas can vary considerably, satellite  service can cost up to $13,000 per month, creating a significant difference  in urban and rural rates in parts of Alaska, and making FCC’s Rural Health  Care Program particularly attractive under such circumstances.
We also found that, according to USAC data, FCC and USAC have been  successful in disbursing committed funds in the primary Rural Health Care  Program. Table 1 shows that USAC generally disburses most of the funds  that are committed to rural health care providers. Of the more than $380  million committed for the program, over $327 million (over 86 percent)  has been disbursed, leaving just over $53 million that has been committed  but not disbursed since the program began. Some of this $53 million in  remaining money will eventually be disbursed as USAC closes more recent  funding years.
section_title: FCC Has Not Assessed the Telecommunications Needs of Rural Health Care Providers to Guide the Evolution of the Rural Health Care Program
paragraphs: A needs assessment is crucial to both the effective design of new programs  and the assessment of existing programs. The primary purpose of a needs  assessment is to identify needed services that are lacking (in this case,  telecommunications services for rural health care providers) relative to  some generally accepted standard. By establishing measures of  comparison, program managers can more accurately determine how well  their programs are doing in meeting the needs of the targeted population  of the program. We have previously recommended that needs assessments  include the following characteristics:    benchmarks to define when needs have increased or decreased,    a plan to determine how needs assessment results will be prioritized in  supporting resource allocation decisions, and integration of information on other resources available to help address the  need.
However, throughout its 12 years of managing the program, FCC has not  conducted a comprehensive needs assessment to learn how the program  can best target the telecommunications needs of rural health care  providers within the broad latitude provided by Congress in the 1996 Act.
When designing the $400 million annual spending cap for the Rural Health  Care Program, FCC officials noted the scarcity of information available  about the universe of eligible providers, and what it might cost to meet the  providers’ telecommunications needs. As our analysis showed, the  current $400 million spending cap is not based on meaningful estimates of  program participation. FCC stated in its 1997 report and order that the  Rural Health Care Program spending cap is “based on the maximum  amount of service that we have found necessary and on generous  estimates of the number of potentially eligible rural health care  providers.” FCC acknowledged at the time that it expected actual  program disbursements to be less than the cap for a number of reasons.  Although FCC expected program disbursement to be under $400 million  annually, on multiple occasions, FCC has released documents stating that  the primary Rural Health Care Program is underutilized. For example, in  its 2006 pilot program order, FCC states that the primary Rural Health  Care Program “continues to be greatly underutilized and is not fully  realizing the benefits intended by the statute and our rules. In 1997, we  authorized $400 million per year for funding of this program. Yet, in each  of the last 10 years, the program generally has disbursed less than 10  percent of the authorized funds.”
When we asked FCC officials what acceptable utilization of the program  would mean, they said that they did not know, but that program utilization  would include disbursing funds somewhere between 10 percent and 100  percent of the allowable cap. FCC’s repeated claim that the program is  underutilized, without a more specific vision of what utilization would  mean, is troublesome. No needs assessment has been conducted to show  that the program is, in fact, underutilized. A comprehensive needs  assessment could provide useful information to FCC to help officials  envision acceptable program utilization—that is, how many providers  actually need services, rather than just how many providers are eligible to  participate under program rules.
As part of our review, we interviewed knowledgeable stakeholders to  identify potential reasons for FCC’s reported underutilization. These  reasons include the following:    Some health care providers lack the infrastructure (e.g., the broadband  facilities needed to support telemedicine) to use advanced  telecommunications services.
The application process is too complex and cumbersome to justify  participation.
The 25 percent Internet subsidy is not large enough to encourage  participation.
The difference between urban and rural telecommunications rates is  negligible or not significant enough to justify resources toward program  participation.
Rural health care providers do not have enough administrative support to  apply to the program annually.
Some eligible health care providers may not know about the program.
Statutory restrictions prevent support to certain providers who might  benefit from the program (e.g., emergency medical technicians).
Some health care providers cannot afford expensive telemedicine  equipment; therefore, they are not concerned with gaining access to the  telecommunications services needed to use that equipment.
Some Medicare and Medicaid rules, including reimbursement limitations,  may inhibit the use of telemedicine technologies; therefore, health care  providers may not be concerned with gaining access to the  telecommunications services needed to support those technologies.
Despite these and other issues, we were also told that many of the current  program participants are dependent on the benefits they receive from the  primary Rural Health Care Program.
Although it lacks a needs assessment, FCC has made multiple changes to  the primary Rural Health Care Program over time in an attempt to address  underutilization and better meet providers’ needs. For example:  In a 2003 report and order, FCC provided support for rural health care  providers to obtain a 25 percent discount off the cost of monthly Internet  access services. The 2003 report and order states: “Because participation  in the rural health care support mechanism has not met the Commission’s  initial projections, we amend our rules to improve the program, increase  participation by rural health care providers, and ensure that the benefits of  the program continue to be distributed in a fair and equitable manner.”
In a 2004 report and order, FCC changed the definition of “rural,” revised  its rules to expand funding for mobile rural health care services, and  allowed a 50 percent subsidy (rather than 25 percent) for Internet access  services for health care providers in entirely rural states. According to  USAC, this report and order increased the number of health care providers  eligible to participate in the primary Rural Health Care Program by adding  new rural areas while grandfathering health care providers in areas no  longer defined as rural.
In a 2006 order, FCC announced the pilot program, which will be  discussed in greater detail in the next section of this report. FCC created  the pilot program to address two potential reasons for the primary Rural  Health Care Program’s possible underutilization: lack of infrastructure and  access to dedicated broadband networks.
Without a needs assessment, however, FCC does not have key information  regarding the extent to which any of these reasons actually impacted the  primary Rural Health Care Program’s participation rate. FCC officials  told us that the changes FCC has made to the program were based  primarily on information gathered through the agency’s notice an d  comment procedures and internal deliberations. FCC officials told us th this is how FCC—as a federal regulatory agency—conducts its business  pursuant to the Administrative Procedure Act (APA). nothing in the APA process that would have precluded FCC from  s to  conducting a formal needs assessment. Using data-based assessment supplement the information gained through FCC’s regulatory procedures  al  would enhance FCC’s ability to fulfill its role as the manager of the Rur Health Care Program. Specifically, if FCC had obtained data through a  formal needs assessment, it may have been able to more accurately  ascertain why some rural health care providers are not participating, and  have better ensured that programmatic changes achieved th results.
Administrative Procedure Act, 5 U.S.C. §§ 551 et seq.
section_title: FCC’s Poor Planning and Communication during the Design and Implementation of the Pilot Program Caused Delays and Difficulties
subsections: section_title: FCC’s Limited Collaboration with USAC, Federal Agencies, and Other Knowledgeable Stakeholders Affected Pilot Program Design
paragraphs: FCC missed multiple opportunities to collaborate with USAC, federal  agencies, and other knowledgeable stakeholders when designing the pilot  program. These stakeholders all could have provided useful insights into  FCC’s design of the pilot program. Such consultations could have helped  FCC better identify potential pitfalls in its pilot program design as well as  meaningful opportunities to leverage federal resources and ensure that the  pilot program targeted rural health care providers’ needs in the most  efficient way.
Although USAC officials had 9 years’ experience working with the rural  health care community and administering the primary Rural Health Care  Program, FCC did not consult with USAC officials prior to issuing the 2006  order calling for applications to the pilot program. Our prior work has  noted the importance of involving stakeholders (including third-party  administrators like USAC) when designing, implementing, and evaluating  programs. FCC officials stated that they did not consult with USAC  because USAC does not formulate policy. However, USAC’s experience  with the primary Rural Health Care Program may have provided FCC with  valuable insights into how to design a pilot program, particularly regarding  the administrative processes and forms. For example, FCC’s decision to  use the primary program’s forms and processes for the pilot program led  to a complicated administrative process, particularly since some aspects  of the primary Rural Health Care Program’s forms and administrative  processes were ill-suited to the pilot program. Because FCC used primary  program forms rather than creating new and more tailored ones for the  pilot program, the forms required complicated attachments. According to  our survey of pilot project representatives, of the 57 respondents that  expressed an opinion, 38 respondents rated assembling their request for  proposals (RFP) package (Form 465 package) as “very difficult” or  “somewhat difficult.” In addition, 27 of the 42 respondents that provided  an opinion rated assembling their requests for funding (Form 466-A  packages) as “very difficult” or “somewhat difficult.” Solix officials agreed  that pilot participants seemed to have difficulty in completing these forms  and attachments.
FCC also missed opportunities to coordinate with other federal agencies  when designing the pilot program. We have noted that a lack of  collaboration among federal agencies can lead to a patchwork of programs  that can waste scarce funds, confuse and frustrate program customers,  and limit the overall effectiveness of the federal effort. A number of  federal agencies are involved in telemedicine efforts, and some provide  funds to health care providers that could complement FCC’s pilot  program. For example, the Health Resources and Services Administration  (HRSA), the primary federal agency for improving access to health care  services for people who are uninsured, isolated, or medically vulnerable,  administers a Telehealth Network Grant Program that provides funds to  projects to demonstrate how telehealth programs and networks can  improve access to quality health care services in underserved rural and  urban communities. However, with USDA being the one exception, FCC  did not contact other federal agencies prior to announcing the pilot  program in 2006. FCC officials told us that after announcing the creation  of the pilot program in 2006, they met with representatives from various  agencies within HHS in 2007, to discuss coordination. Representatives  from some of these agencies reported that these meetings were primarily  informational, with FCC explaining its pilot program to them, and that no  strategies for collaboration or follow-up were developed. USDA officials  stated that FCC officials met with them prior to announcing the pilot  program to discuss USDA’s Distance Learning and Telemedicine Program,  including how USDA scored applications and evaluated the program.  However, it is unclear how FCC used the information that USDA provided,  since similar information was not provided in FCC’s call for applications  or order selecting pilot projects. According to federal and other  stakeholders, officials at other agencies also could have    provided FCC with an understanding of rural health care providers’ needs,  potential information technology (IT) issues, and how to design a more  user-friendly program and   helped FCC identify additional appropriate service providers, one of which  had to petition to be included.
FCC also did not request public comment on its proposed design for the  pilot program. Although FCC did request comments in 2004 on providing  some infrastructure support by funding upgrades to the public switched or  backbone networks, FCC did not imply that it was considering a pilot  program to fund the creation of private networks, or provide specific  details on how such a program would operate. We have previously  reported that FCC’s use of NPRMs to pose broad questions without  providing actual rule text can limit stakeholders’ ability to determine  either what action FCC is considering or what information would be most  helpful to FCC when developing a final rule. FCC officials said that they  did not issue a Notice of Inquiry (NOI) regarding the pilot program  because the process would have delayed the pilot program. However,  providing the public with advance notice of proposed changes and an  opportunity to comment on them is desirable in that it allows agencies,  according to a 2006 resource guide, to “find out earlier rather than later  about views and information adverse to the agency’s proposal or bearing  on its practicality.” Similarly, in comments submitted to FCC, the  National Telecommunications Cooperative Association, observed that  “interested or affected parties had no opportunity to explore with the  Commission various aspects of the Pilot Program.” In addition, industry  concerns regarding the funding of redundant networks arose after the  implementation of the pilot program. If FCC had provided a more  detailed explanation of the proposed pilot program and requested  comment prior to establishing the program, it may have been better  prepared to address these concerns.
section_title: Pilot Participants Have Experienced Delays and Difficulties, in Part, Because FCC Did Not Fully Establish Requirements Prior to Calling for Applications and Did Not Provide Effective Program Guidance
subsections: section_title: Pilot Participants Have Experienced Delays and Difficulties for Many Reasons
paragraphs: FCC called for applications to participate in the pilot program before it  fully established pilot program requirements. This, along with the addition  of requirements as the pilot program has progressed, has led to delays and  difficulties for pilot participants. Most importantly, the entire pilot  program itself has been delayed. Participants may issue multiple RFPs as  they progress through various stages of designing and constructing their  networks, but the deadline for pilot participants to submit all of their  requests for funding (projects submit at least one Form 466-A for each  RFP they issue) to USAC was June 30, 2010. On February 18, 2010, FCC  extended this deadline by 1 year, to June 30, 2011. According to USAC  data, at the time of the extension, projects had requested 11 percent of the  roughly $418 million in total program funding. As of July 31, 2010, projects  had requested 17 percent of the total program funding. As shown in figure  9, as of July 31, 2010, 28 projects (45 percent) have received at least one  funding commitment letter, but 18 projects (29 percent) had not yet posted  an RFP.
According to our survey, delayed and inconsistent guidance led to delays  for many pilot projects. In addition, it appears pilot participants have  struggled with requirements that were added at the same time that FCC  announced the pilot participant selections, such as the need to obtain  letters of agency. Figure 10 indicates the number of survey respondents  reporting whether they experienced certain issues during the course of  their project, and the number of respondents that reported they were  delayed by that issue.
Table 2 reports the results from our survey question that asked pilot  participants to rate the ease or difficulty of performing various program  tasks. Four of the tasks rated as “very difficult” or “somewhat difficult” by  more than half of the respondents that provided an opinion fall into one of  two categories: the task is associated with program processes and forms  that were carried over from the primary Rural Health Care Program (Form  465 and Form 466-A), or the task is a requirement FCC added, but that was  not mentioned in the initial call for applications (developing a  sustainability plan and obtaining letters of agency). In addition, when  asked to list the top three things that program officials should change if  FCC established a new, permanent program with goals similar to those of  the pilot program, simplifying or improving the administrative process was  the most frequently mentioned issue.
The Grant Accountability Project, a 2005 Domestic Working Group chaired  by the former United States Comptroller General, notes that an agency’s  ability to ensure that funds are used as intended is impacted when the  terms, conditions, and provisions in award agreements are not well- written. The group also notes that a thorough assessment of proposed  projects can reduce the risk that money may be wasted or projects may  not achieve intended results. However, FCC did not fully establish the  requirements of the pilot program before it requested applications, and it  required projects to provide additional information after they were  accepted into the program. This led to delays because (1) participants  needed additional guidance on how to meet the requirements and (2) Solix  staff (under USAC direction) had to retroactively review projects to  determine the eligibility of the participating entities and activities. In  addition, some participants faced difficulties in funding ineligible  expenses. In contrast, other federal agencies generally provide extensive  detail on program rules when calling for applications for competitive  funding programs, including the criteria by which applications will be  judged and how the criteria will be weighted. For example, USDA’s 2010  Distance Learning and Telemedicine Program Grant Application Guide  provides potential applicants with specific information on eligible uses for  the funds, eligible match funding, copies of the forms to be used, and  information on the application scoring process.
FCC’s 2006 order establishing the pilot program and calling for  applications did not provide detailed information on many essential  aspects of the program, including    which entities would be eligible to participate in the program (FCC  provided a legal citation, but no actual text);   which expenses could be paid for with program funds; and    how projects could fund their 15 percent match.
After issuing its call for applications, FCC did provide some of this  information on its Frequently Asked Questions Web page on the pilot  program. However, this information may not have reached all interested  parties, and it would have been more efficient to determine these issues in  advance of requesting applications. FCC also provided some of this  information in its 2007 order; however, by this time, FCC was also  announcing which projects were selected. FCC did not fully screen  applications to determine the extent to which their proposed activities and  entities would be eligible for funding. Thus, several of the accepted  projects had ineligible components. Specifically, based on survey  respondents that provided a substantive answer:    25 of 59 respondents included an entity that was determined to be  ineligible,    25 of 57 respondents included an expense that was determined to be    10 of 59 respondents relied on ineligible sources to fund their match.
The lack of established criteria and an in-depth screening prior to  announcing pilot project awards led to a lengthy process by which Solix  staff (under USAC direction) determine the eligibility of entities  postaward. Pilot participants must submit documentation for every entity  in their project, which Solix staff then review to determine eligibility.  According to our survey, 36 of 60 respondents were delayed by difficulties  in compiling and submitting the documentation needed to establish entity  eligibility. In addition, although 39 survey respondents rated the current  program guidance regarding entity eligibility as “very clear” or “somewhat  clear”; some confusion remains, as 20 pilot participants rated the current  guidance as “slightly clear” or “not at all clear.” FCC’s 2007 order also  notes that program administration costs, such as personnel, travel, legal,  marketing, and training costs, are ineligible for program funding. Pilot  participants have indicated in written comments to FCC that they did not  anticipate that administrative costs would not be eligible for funding, and  some have faced challenges in funding these costs themselves. In our  survey, 36 of 51 respondents indicated that funding ineligible costs,  including administrative costs, has been “very difficult” or “somewhat  difficult.” In addition, when asked to list the top three things that program  officials should change if FCC established a new, permanent program with  goals similar to those of the pilot program, providing funding for  administrative costs was the second-most frequently mentioned issue.
section_title: FCC Introduced New Requirements after Its Call for Applications and Selection of Pilot Participants
paragraphs: FCC’s 2007 order also introduced new requirements that were not  mentioned in the 2006 order. For example, while the 2006 order states that  the pilot program would use the same forms and administrative processes  used in the primary Rural Health Care Program, the 2007 order also  requires projects to secure a letter of agency from every entity  participating in a project. This letter authorizes the lead project  coordinator to act on the signing agency’s behalf. Since a number of the  selected pilot program participants included providers that were also  participating in another participant’s proposed network, FCC noted that  the letter of agency would demonstrate that the entity has agreed to  participate in the network and prevent improper duplicate support for  providers participating in multiple networks. Considering that FCC  encouraged applicants to create statewide networks, some projects have  hundreds of participating entities, creating the need for such projects to  secure hundreds of letters of agency. The letter of agency requirement has  proven extremely time-consuming and resource-intensive for some  projects. According to our survey, 34 of 60 respondents faced difficulties  in securing letters of agency, and 27 of these respondents were delayed  because of these difficulties. (See fig. 10.)
Similarly, according to our survey results, FCC has not provided sufficient  guidance to pilot projects on how to meet FCC’s requirement that projects  comply with HHS health IT initiatives. In response to a letter from HHS,  FCC outlined a number of requirements in its 2007 selection order that  pilot program participants should meet to ensure that their pilot projects  were consistent with HHS health IT initiatives. FCC officials stated that  the explanation of the requirements in its 2007 selection order, in addition  to a guidance document created in 2008, provided guidance for the pilot  projects in how to meet these requirements. However, one HHS official  described the language in the 2007 selection order as vague and in need of  an update. In addition, the 2008 guidance document has not been revised  to reflect new developments in interoperability specifications and  certification programs. Currently, pilot participants are required to explain  in each quarterly report how they are complying with the HHS health IT  requirement. However, 34 of 47 survey respondents who provided an  opinion stated the guidance provided on how to meet these requirements  was “slightly sufficient” or “not at all sufficient.”
Following the release of the 2007 order, FCC created additional  requirements as the program progressed and did not provide program  guidance in a timely manner on how to meet these requirements. For  example, FCC stated in its 2007 award order that selected pilot  participants generally “provided sufficient evidence that their proposed  networks will be self-sustaining by the completion of the pilot program.”  However, program officials began requiring more detailed sustainability  plans in the fall of 2008, after some projects had gone through the  competitive bidding process and had requested funding commitment  letters from USAC. Outside of an October 24, 2008, letter to USAC in which  FCC noted that participants should “disclose all sources or potential  sources of revenue that relate to the network” and intentions to sell or  lease excess capacity in the project’s sustainability plan, FCC did not  provide any other written guidance on what specific information should be  included in participant’s sustainability plans until April 2009. At that time,  FCC posted an item to its Frequently Asked Questions Web page that  suggested more information to be included in a participant’s sustainability  plan, including status of obtaining the match, projected sustainability  period, network membership agreements, ownership structure, sources of  future support, and management structure. USAC and Solix officials noted  that some pilot participants believed that because their application was  accepted by FCC, they met all of the program requirements for  sustainability. In some cases, this misunderstanding led to confusion and  disagreements between the pilot participants and program officials  regarding the need for additional information and the amount of time that  a sustainability plan should cover. Moreover, it appears some confusion  remains. When asked to rate their satisfaction with any guidance they  received thus far on how to develop a sustainability plan, 21 of the 60  survey respondents that provided an opinion were “very dissatisfied” or   “somewhat dissatisfied.” In addition, 39 of 59 survey respondents faced  difficulties in developing a sustainability plan; 33 of these respondents  stated that difficulties in developing a sustainability plan have delayed  their projects. (See fig. 10.)
section_title: Program Guidance Is Not Provided in an Effective Manner
paragraphs: In addition, because FCC is responsible for all policy decisions regarding  the pilot program, unique or difficult situations are typically referred from  USAC to FCC for its decision. The need for such consultations is  compounded by an absence of formal written guidance for USAC and pilot  participants. We have reported that information should be recorded and  communicated to management and others who need it in a form and  within a time frame that enables them to carry out their responsibilities.  FCC staff stated that, in some cases, they have not provided written  guidance because they want the pilot program to remain flexible.  However, in some cases, it has taken several months for FCC to make a  decision or provide guidance on issues. However, it appears that pilot  participants are dissatisfied with certain elements of program guidance. As  noted in figure 11, of the 59 respondents that provided an opinion, 32  respondents were “very dissatisfied” or “somewhat dissatisfied” with the  clarity of program guidance, and 28 respondents were “very dissatisfied”  or “somewhat dissatisfied” with the amount of formal written guidance. In  addition, as we note in figure 10, 37 of 60 respondents stated that they had  received inconsistent guidance. Similarly, when asked to list the top three  things that program officials should change if FCC established a new,  permanent program with goals similar to those of the pilot program,  providing more guidance and templates was the third-most frequently  mentioned issue.
In addition, although FCC recognized in its 2006 order calling for  applications that ineligible entities may be participating in the networks  and would need to pay their fair share of costs, FCC chose not to establish  detailed guidance on how to address such issues prior to establishing the  program. Instead, FCC provided guidance as questions arose from USAC  (USAC’s first request about how to determine fair share was in June 2007  when it was noted that a substantial number of ineligible entities were  included in applications submitted to FCC) and from pilot program  participants. According to USAC officials, questions concerning payment  of fair share or incremental costs for excess capacity shared with  ineligible entities occurred at the pilot program training in February 2008  and continued from participants to USAC and from USAC to FCC  throughout 2008. The most recent FCC guidance was a March 2009 matrix  outlining nine scenarios in which excess capacity could be used. However,  issues regarding excess capacity remain. FCC’s July 2010 NPRM notes that  “rules governing the sharing of this subsidized infrastructure are necessary  to prevent waste, fraud and abuse,” and requested comment on a number  of detailed questions regarding the sharing of excess capacity with  ineligible entities, different methods for allocating costs among entities,  providing excess capacity for community use, and what types of guidance  are needed. According to our survey, 21 of 60 respondents indicated that  their project “definitely” or “probably” will include excess capacity.
section_title: Participants Reported That the Benefits Afforded by the Pilot Program Are Worth the Costs of Participating, and They Were Generally Positive about Program Officials
paragraphs: Although there have been some challenges, many pilot participants  emphasized the importance of the pilot program in their responses to our  survey as well as in comments submitted to FCC. According to our survey,  if pilot participants are able to accomplish their pilot project goals:    55 of 57 respondents indicated their project “definitely” or “probably” will  have entities that obtain telecommunications or Internet services that  would otherwise be unaffordable;   48 of 55 respondents indicated their project “definitely” or “probably” will  have entities obtain telecommunications or Internet services that would  otherwise be unobtainable due to lack of infrastructure; and   58 of 59 respondents indicated that their project “definitely” or “probably”  will have entities upgrade an existing telecommunications or Internet  service.
In addition, when asked to consider their current understanding of the  costs and administrative requirements of participating in the pilot  program, 52 of 57 respondents reported that the pilot program’s benefits  will outweigh the costs of participating in the program.
Pilot participants were also generally positive about the usefulness of  program officials, in particular their coach. When asked to list the top  three things that program officials did well in their administration of the  program, respondents provided positive opinions about their  communications with program officials, the effort put forth by program  officials, and the coaches or the coaching concept. These responses are  consistent with those provided to another question that rated program  officials—be they FCC, USAC, or a project coach—as the most useful  resource for pilot participants. (See fig. 12.)
Pilot participants were also satisfied with their coaches as a source of  information. USAC appointed coaches to serve as a project’s direct point  of contact with program officials. Of the 61 respondents who provided an  opinion, some specifically noted their satisfaction with the ease with  which they could contact their coach (53 respondents were “very  satisfied” or “somewhat satisfied”) and the level of interaction with their  coach (49 respondents were “very satisfied” or “somewhat satisfied”).  Coaches were rated somewhat lower on their knowledge of the program  (40 respondents were “very satisfied” or “somewhat satisfied”), although  this lower rating may be related to the lack of established guidance at the  beginning of the program and the need to refer difficult issues to Solix  management, USAC, and FCC, depending on the complexity of the issue.
section_title: FCC Is Seeking More Input and Providing More Detail on Its Proposed New Program, but It Is Not Clear Whether Planning and Communication Have Been Fully Addressed
paragraphs: In its July 2010 NPRM, FCC proposed a new Health Infrastructure Program  that would make available up to $100 million per year to support up to 85  percent of the construction costs of new regional or statewide networks  for health care providers in areas of the country where broadband is  unavailable or insufficient. In this NPRM, FCC made improvements over  previous NPRMs by outlining potential program requirements and  requesting comment on the proposed new program. FCC provided much  more information than it did when announcing the pilot program and is  allowing for stakeholder input into the program’s design. In addition, FCC  recognized some of the challenges mentioned in this report and requested  comment on potential improvements. In particular, FCC proposed and  requested comment on  requiring that applicants prove or otherwise certify that broadband at  minimum connectivity speeds is unavailable or insufficient to meet their  health care needs when applying to the program; requiring applicants to submit letters of agency as part of their application,  rather than after they are accepted into the program;   having USAC review entity eligibility;    providing limited funding for administrative costs;    expanding entity eligibility to include off-site administrative offices, off- site data centers, nonprofit skilled nursing facilities, and nonprofit renal  dialysis facilities; and   providing additional guidance regarding the funding and permitted uses of  excess capacity and the allocation of related costs.
However, it remains unclear the extent to which FCC is coordinating with  USAC in preparing for this program. The NPRM indicates that USAC will  develop a user-friendly Web-based application for participants to use.  However, during our conversations with USAC, officials noted that it  would take a considerable amount of time and effort to properly develop  such systems. FCC indicates in its NPRM that the new programs could be  implemented by funding year 2011. If FCC does not better plan the details  of the new program before it calls for applications, participants in the  Health Infrastructure Program may experience the same delay and  difficulties as participants have experienced in the pilot program.
section_title: FCC Has Not Followed Key Performance Management Practices, Thus It Lacks the Performance Data to Make Effective Policy Decisions and Implement Program Reforms
subsections: section_title: FCC Has Attempted to Develop Performance Goals and Measures for the Rural Health Care Program, but They Are Ineffective for Managing Program Performance
paragraphs: We have previously reported that results-oriented organizations commonly  perform a number of key practices to effectively manage program  performance. In particular, results-oriented organizations implement two  key practices, among others, to lay a strong foundation for successful  program management. First, these organizations set performance goals to  clearly define desired outcomes. Second, these organizations develop  performance measures that are clearly linked to the program goals.  However, FCC has reversed these two key practices. In 2006, 8 years after  FCC first implemented the primary Rural Health Care Program, the Office  of Management and Budget (OMB) assessed FCC’s Rural Health Care  Program and concluded that the program had no performance goals and  measures. In 2007, FCC issued a report and order adopting performance  measures for the Rural Health Care Program related to USAC’s processing  of applications, paying invoices, and determining appeals. However, FCC  stated that it did not have sufficient data to establish performance goals  for the Rural Health Care Program in the report and order. Instead of  specific performance-related goals, the Rural Health Care Program has  operated for 12 years under broad overarching goals, including the  statutory goal established by Congress in the 1996 Act, which is to ensure  that rural health care providers receive telecommunications services at  rates comparable for the same services in urban areas.
Furthermore, the performance measures that FCC adopted for the primary  Rural Health Care Program and the pilot program in 2007 fall short when  compared with the key characteristics of successful performance  measures that we have identified in our past work. Following is a  discussion of these characteristics and the extent to which FCC has  fulfilled them in developing performance measures:    Measures should be tied to goals and demonstrate the degree to which the  desired results are achieved. These program goals should, in turn, be  linked to overall agency goals. However, as we have previously discussed,  the measures that FCC has adopted are not based on such linkage because  no specific performance goals have been established. By establishing  performance measures before establishing the specific performance goals  that it seeks to achieve through the Rural Health Care Program, FCC may  waste valuable time and resources collecting the wrong data. FCC receives  the data for these performance measures on a quarterly basis from USAC,  but without effective performance goals to guide its data collection, it  cannot ensure that the data gained from these performance measures are  an effective use of resources.
Measures should address important aspects of program performance.
For each program goal, a few performance measures should be selected  that cover key performance dimensions and take different priorities into  account. For example, measures should be limited to core program  activities because an excess of data could obscure rather than clarify  performance issues. Performance measures should also cover key  governmentwide priorities, such as timeliness and customer satisfaction.  FCC’s performance measures appear to address certain key performance  dimensions. By selecting just three types of measures—related to USAC’s  (1) processing of applications, (2) paying invoices, and (3) determining  appeals—there are fewer chances of obscuring the most important  performance issues. The measures also appear to take into account such  priorities as timeliness and customer satisfaction. For example, the 2007  performance measures include requirements to measure the number of  current and pending appeals, and the time that it takes to resolve those  appeals. However, again, without first setting specific performance goals  defining what the programs are specifically intended to accomplish, FCC  cannot be sure that it has adopted the most appropriate performance  measures.
Measures should provide useful information for decision making.  Performance measures should provide managers with timely, action- oriented information in a format that helps them to make decisions that  improve program performance. However, the data collected by these  performance measures—such as the number of applications submitted,  rejected, and granted—are output, not outcome, oriented. The FCC task  force that developed the National Broadband Plan also reported that the  performance measures developed for the Rural Health Care Program need  to be improved to assess desired program outcomes, such as the impact of  the program on patient care. The limited nature of the data obtained by  current performance measures, combined with the absence of specific  performance goals, raises concerns about the effectiveness of these  performance measures for programmatic decision making.
FCC is attempting to improve its performance management by seeking  public comment on performance goals and measures for the Rural Health  Care Program in its July 2010 NPRM. For example, FCC proposed a  specific measure of how program support is being used: that is, requiring  beneficiaries to annually identify the speed of the connections supported  by the program and the type and frequency of the use of telemedicine  applications as a result of broadband access. Although this is a positive  step, the NPRM does not specify whether this data collection would be  linked to specific connection speed goals that participants should obtain  with program funds, and it does not propose what the goal should be for  type and frequency of the use of telemedicine applications. While this  NPRM could lead to better goals and measures for the Rural Health Care  Program, FCC has exhibited a pattern of repeatedly seeking comment on  goals and measures for the Rural Health Care Program, which indicates  that it does not have a clear vision for what it intends the program to  accomplish within the broad statutory framework provided by Congress.  FCC has sought public comment on performance goals and measures for  the program on two previous occasions that did not result in effective  performance goals and measures for the program:  In June 2005, FCC issued a NPRM seeking comment on whether specific  performance goals were needed and on ways to establish useful outcome,  output, and efficiency measures for each of the universal service  programs, including the Rural Health Care Program. FCC officials stated  that this NPRM led to the 2007 performance measures that we have  previously described.
In September 2008, FCC issued a NOI seeking comment on how to more  clearly define the goals of the Universal Service Fund programs, including  the Rural Health Care Program, and to identify any additional quantifiable  performance measures that may be necessary or desirable. FCC officials  stated that this NOI led to the July 2010 NPRM, which, again, requests  comment on performance goals and measures.
Performance goals and measures are particularly important for the Rural  Health Care Program, because they could help FCC to make well-informed  decisions about how to address the trends that we have previously  described. If FCC does use information from the latest NPRM to develop  specific performance goals and measures, it should focus on the results  that it expects its programs to achieve. We have identified the following  practices for developing successful performance goals and measures:    create a set of performance goals and measures that addresses important  dimensions of a program’s performance and balance competing priorities,   use intermediate goals and measures to show progress or contribution to  include explanatory information on the goals and measures,   develop performance goals to address mission-critical management    show baseline and trend data for past performance,  identify projected target levels of performance for multiyear goals, and link the goals of component organizations to departmental strategic  goals.
Clearly articulated, outcome-based performance goals and measures are  important to help ensure that the Rural Health Care Program meets the  guiding principles that Congress has set forth.
section_title: Without Effective Performance Goals and Measures, FCC Cannot Reliably Evaluate Program Performance, Which Could Lead to a Repeat of Its Past Management Weaknesses
paragraphs: After implementing the key performance management practices that we  have previously discussed—establishing effective performance goals and  measures—results-oriented organizations implement a third, key practice:  that is, evaluating the performance of their programs. Measuring  performance allows these organizations to track progress toward goals  and provides managers with the crucial performance data needed to make  management decisions. We have previously reported that performance  data can have real value only when used to identify the gap between a  program’s actual performance level and the performance level identified  as its goal. Again, without specific performance goals and effective  performance measures, FCC cannot identify program performance gaps  and is unlikely to conduct evaluations that are useful for formulating  policy decisions.
FCC has not formally evaluated the performance of the primary Rural  Health Care Program to determine whether it is meeting the needs of rural  health care providers, and it may lack the tools to evaluate the pilot  program—such as an effective progress reporting mechanism and an  evaluation plan. To its credit, FCC has stated that it intends to evaluate the  pilot program after its completion. However, it is unclear whether FCC has  effective evaluation tools for conducting a pilot program evaluation that  will be useful for making policy decisions about the future of the Rural  Health Care Program. To track the progress of pilot projects, FCC requires  pilot program participants to complete quarterly reports that are filed with  FCC and USAC, but it is unclear whether these reports are effective tools  for evaluating pilot program performance for the following reasons:    Quarterly report data are not quantitative. Quarterly reports collect data  that are mostly qualitative (e.g., a narrative description of a project’s  network and how the network will be sustained) instead of quantitative.  While qualitative data can help officials understand project progress on an  individual basis, the information is not objective or easily measured.
FCC has not involved key stakeholders. We have previously reported that  stakeholder and customer involvement helps agencies to ensure that  efforts and resources are targeted at the highest priorities. However, key  stakeholders and pilot participants are not involved in ensuring that  quarterly reports are providing the most useful information possible. Pilot  program coaches, who guide pilot participants through the program’s  administrative processes, and USAC officials said that FCC has not told  them how the reports will be used to evaluate pilot program progress.  USAC and the pilot program coaches work directly with participants and  without understanding how these reports will be used, they are unable to  effectively guide participants into providing the most useful evaluation  information possible. Additionally, of the 45 pilot program survey  respondents that provided an opinion, 26 said that they receive too little  feedback on their quarterly reports.
Quarterly reports may require too much information. Of the 58 pilot  program survey respondents that provided an opinion, 28 said that too  much information is required in quarterly reports. As we have previously  discussed, an excess of data can obscure rather than clarify performance.
FCC officials told us that they have learned lessons from using these  quarterly reports, and that, as part of the 2010 NPRM, FCC requested  public comment on a similar reporting requirement for the proposed  Health Infrastructure Program.
Furthermore, despite FCC’s intentions to evaluate the pilot program,  officials have not yet developed an evaluation plan for the pilot program.  FCC officials told us that this is because the pilot program is still under  way, and that FCC will plan the evaluation when the pilot program is  closer to completion (as we previously stated, the deadline for participants  in the pilot program to select a vendor and request a funding commitment  from USAC is June 30, 2011). However, we have previously reported that  when conducting pilot programs, agencies should develop sound  evaluation plans before program implementation—as part of the design of  the pilot program itself—to increase confidence in results and facilitate  decision making about broader application of the pilot program. We have  previously identified the following key features of sound evaluation plans:    well-defined, clear, and measurable objectives;    measures that are directly linked to specific program objectives;    criteria or standards for determining program performance;    clearly articulated methodology and a strategy for comparing results with    a clear plan that details the type and source of data necessary to evaluate  the program, methods for data collection, and the timing and frequency of  data collection;   a detailed data-analysis plan to track the program’s performance and  evaluate its final results; and   a detailed plan to ensure that data collection, entry, and storage are  reliable and error-free.
The lack of a documented evaluation plan for the pilot program increases  the likelihood that FCC will not collect appropriate or sufficient data,  which limits understanding of pilot program results. Without this  understanding, FCC will be limited in its decision making about the pilot  program’s potential broader application to FCC’s proposed future  programs.
The National Broadband Plan states that for all four universal service  fund programs, including the Rural Health Care Program, “there is a lack  of adequate data to make critical policy decisions regarding how to better  utilize funding to promote universal service objectives.” FCC has not  effectively followed the three key performance management practices  discussed in this report and has not obtained the data that it needs to  make critical policy decisions and successfully manage the program.  Furthermore, FCC has proposed two new programs under the Rural  Health Care Program in its 2010 NPRM (the Health Broadband Services  Program and the Health Infrastructure Program), even though the  National Broadband Plan states that FCC does not have the data to make  critical policy decisions on how to better use its funds. In our previous  work, we have reported that results-oriented organizations recognize that  improvement goals should flow from a fact-based performance analysis.  However, the proposed improvements to the Rural Health Care Program  are not based on a fact-based performance analysis because the  performances of the primary Rural Health Care Program and the pilot  program have not been evaluated. FCC officials told us that they believe  the proposals set forth in the July 2010 NPRM are “positive first steps”  toward creating improvements to performance analysis.
Because FCC has not determined what the primary Rural Health Care  Program and the pilot program are specifically intended to accomplish and  how well the programs are performing, it remains unclear how FCC will  make informed decisions about the new programs described in the July  2010 NPRM. Moreover, as new technologies are developed, measuring the  performance and effectiveness of existing programs is important so that  decision makers can design future programs to effectively incorporate  new technologies, if appropriate. If FCC does not institute better  performance management tools—by establishing effective performance  goals and measures, and planning and conducting effective program  evaluations—FCC’s management weaknesses will likely continue to affect  the current Rural Health Care Program, and will likely carry forward into  the design and operation of proposed Rural Health Care programs.
section_title: Conclusions
paragraphs: Over the first 12 years of its Rural Health Care Program, FCC has  distributed more than $327 million to rural health care providers to assist  them in purchasing telecommunications and information services. FCC  and USAC have been particularly successful in disbursing committed  funds in the primary Rural Health Care Program, and FCC has generally  seen slow but steady growth in both the amounts of annual disbursements  and the number of annual applicants to the primary program.
However, since the Rural Health Care Program’s inception, FCC has not  provided the program with a solid performance management foundation.  FCC could better inform its decision making and improve its stewardship  of the Rural Health Care Program by incorporating effective performance  management practices into its regulatory processes. FCC has not  conducted a comprehensive needs assessment to determine the needs of  rural health care providers, has no specific goals and measures for the  program to guide its management decisions, and has not evaluated how  well the program is performing. FCC’s attempts to improve the program  over time, including the 2006 pilot program, have not been informed by a  documented, fact-based needs assessment; consultations with  knowledgeable stakeholders, including other government agencies; and  performance evaluations. Despite FCC’s efforts to improve the program, a  significant number of eligible rural health care providers currently do not  use the primary Rural Health Care Program, and FCC’s management of the  pilot program has often led to the delays and difficulties reported by pilot  participants.
We found that a number of rural health care providers depend on the  support they receive from the primary Rural Health Care Program, and  that most pilot program participants are seeking services that they believe  would have been otherwise unaffordable. It is possible that FCC’s  proposed changes to the Rural Health Care Program will increase  participation by rural health care providers, thus increasing the amount of  funding committed by the Rural Health Care Program and, ultimately,  increasing the universal service fees paid by consumers on their telephone  bills. Changes in FCC’s approach to performance management could help  ensure that higher telephone bills are justified; that program resources are  targeting the needs of rural health care providers; and that the program, in  fact, is helping our nation to realize more widespread use of telemedicine  technologies.
section_title: Recommendations for Executive Action
paragraphs: To improve its performance management of the Rural Health Care  Program, we recommend that the Chairman of the Federal  Communications Commission take the following five actions. If FCC does  develop any new rural health care programs under the Universal Service  Fund—such as the proposed Health Care Broadband Access Fund and the  Health Care Broadband Infrastructure Fund—these steps should be taken  before implementing any new programs or starting any new data collection  efforts:    Conduct an assessment of the current telecommunications needs of rural  health care providers.
Consult with USAC, other federal agencies that serve rural health care  providers (or with expertise related to telemedicine), and associations representing rural health care providers to incorporate their knowledge  and experience into improving current and future programs.
Develop effective goals, and performance measures linked to those goals,  for all current and future programs.
Develop and execute a sound performance evaluation plan for the current  programs, and develop sound evaluation plans as part of the design of any  new programs before implementation begins.
For any new program, ensure that FCC’s request for applications to the  program clearly (1) articulates all criteria for participating in the program  and any weighting of that criteria, (2) details the program’s rules and  procedures, (3) outlines the program’s performance goals and measures,  and (4) explains how participants’ progress will be evaluated.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Federal Communications  Commission and the Universal Service Administrative Company for their  review and comment.
In its written comments, FCC did not specifically agree or disagree with  our recommendations but discussed planned and ongoing actions to  address them. FCC agreed that it should continue to examine and work to  improve the Rural Health Care Program to ensure that the program is  effectively and efficiently achieving its statutory goals. In response to our  first recommendation that FCC conduct an assessment of the current  needs of rural health care providers, FCC stated that it is gathering  information about health care needs, including needs assessments  performed by other governmental agencies. FCC also stated that going  forward, it is committed to developing benchmarks to define when needs  have increased or decreased, applying needs assessment results to  resource allocation decisions, and integrating information from other  resources available to help address the need. FCC’s efforts to obtain  information and assessments from other agencies and stakeholders are  encouraging. We continue to believe, however, that FCC would benefit  from conducting its own assessment of the telecommunications needs of  the rural health care providers eligible under its Rural Health Care  Program.
In response to our second recommendation that FCC consult with  stakeholders and incorporate their knowledge into improving current and  future programs, FCC stated that it is committed to maximizing  collaboration efforts with federal and other knowledgeable stakeholders  and that it will work closely with USAC to prepare for the new program.  FCC included in its comments an October 2010 statement from the Office  of the National Coordinator for Health IT, Department of Health and  Human Services, about collaborative efforts with FCC and other federal  agencies. In response to our third recommendation that FCC develop  effective performance goals and measures, FCC concurred with the need  to develop quantifiable performance measures. However, FCC did not  specifically state whether it concurred with our recommendation to  develop effective goals and to link performance measures to those goals.  We continue to believe that FCC should develop program performance  goals first, and then develop performance measures and link them to those  goals. In response to our fourth recommendation that FCC develop and  execute effective performance evaluation plans for the current and future  programs, FCC stated that it intends to conduct an evaluation of the pilot  program after it is concluded. While FCC did not address evaluation of the  current primary program, it stated that for any future enhancements to the  program, it is committed to developing and executing sound performance  evaluation plans, including key features that we identified in our report. In  response to our fifth recommendation that FCC identify critical program  information, such as criteria for funding, and prioritization rules in its call  for applications for any new programs, FCC stated that the July 2010  NPRM discusses these elements in detail. While we appreciate FCC’s  efforts to better detail proposed programs in its NPRM, we continue to  believe that FCC should detail the requirements for participation in the  call for applications to any future programs. FCC’s full comments are  reprinted in appendix III.
In its written comments, USAC stated that it will work with FCC to  implement any orders or directives that FCC issues in response to our  recommendations. USAC’s full comments are reprinted in appendix IV.  USAC also provided technical comments that we incorporated as  appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Chairman of the Federal Communications  Commission, the Acting Chief Executive Officer of the Universal Service  Administrative Company, and other interested parties. In addition, the  report will be available at no charge on GAO’s Web site at  http://www.gao.gov.
If you have any questions about this report, please contact me at   (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Major contributors to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to address the following questions: (1) How has the  Federal Communications Commission (FCC) managed the primary Rural  Health Care Program to meet the needs of rural health care providers, and  how well has the program addressed those needs? (2) How have FCC’s  design and implementation of the pilot program affected participants? and  (3) What are FCC’s performance goals and measures for the Rural Health  Care Program, and how do these goals compare with the key  characteristics of successful performance goals and measures?
subsections: section_title: Background Research
paragraphs: We conducted the following background research that helped inform all of  our reporting objectives. Specifically, we reviewed:    prior GAO reports on other Universal Service Fund programs;    FCC’s Universal Service Monitoring Reports on the Rural Health Care    documentation from FCC and the Universal Service Administrative  Company (USAC) on the structure and operation of the Rural Health Care  Program and pilot program; and   FCC documents, including FCC orders and requests for comment on the  Universal Service Fund programs, as well as written comments submitted  in response to these requests.
In addition, we interviewed:    officials from FCC’s Office of Managing Director and Wireline Competition  Bureau to identify actions undertaken to address previously identified  problems and plans to address issues of concern in the programs and    officials from USAC’s Rural Health Care Division and Solix, Inc., to collect  information on program operations and USAC’s actions to implement prior  FCC orders on the primary Rural Health Care Program and pilot program.
section_title: Analysis of Primary Rural Health Care Program Data
paragraphs: To evaluate how the primary Rural Health Care Program was managed to  meet the needs of rural health care providers, we examined trends in the  demand for and use of primary Rural Health Care Program funding from  data we obtained from USAC’s Packet Tracking System (PATS), which is  used to keep track of primary Rural Health Care Program applications,  and the Simplified Invoice Database System (SIDS), which is used to keep  track of program disbursements. When analyzing and reporting on the  data, we considered the limitations on how data can be manipulated and  retrieved from both the PATS and SIDS databases since these systems  were designed to keep track of applications and finances and not to be  data retrieval systems. We assessed the reliability of the data by  questioning officials about controls on access to the system and data back- up procedures. Additionally, we reviewed the data sets provided to us for  obvious errors and inconsistencies. On the basis of this assessment, we  determined that the data were sufficiently reliable to describe broad  trends in the demand for and use of Rural Health Care Program funding.
We obtained the following data—including annual and cumulative  figures—for funding years 1998 through 2009:  the number and characteristics of applicants, including their entity type,  the type of service requested, and location; the dollar amount of funding commitments and disbursements by entity  type, type of service requested, and state; the number of commitments and disbursements by state; and the amount of money committed but not disbursed by entity type and type  of service requested.
To provide these data, USAC performed queries on the PATS and SIDS  systems and provided the resulting reports to us in July 2010. Data from  both systems can change on a daily basis as USAC processes applications  for funding and reimbursement, applicants request adjustments to  requested or committed amounts, and other actions are taken. As a result,  the data we obtained and reported on reflect the program status at the  time that USAC produced the data, and thus may be somewhat different if  we were to perform the same analyses with data produced at a later date.
section_title: Interviews to Assess How Well the Primary Rural Health Care Program Addressed Health Care Provider Needs
paragraphs: To assess how well the primary Rural Health Care Program addressed the  needs of rural health care providers, we interviewed FCC and USAC  officials to determine how the program was designed to address rural  health care provider needs. We reviewed relevant documentation,  including FCC orders, notices of proposed rulemaking, and FCC’s  National Broadband Plan. We also reviewed comments and reply  comments to the record to gain insight into the public perception of how  the program was addressing needs. Furthermore, we interviewed  representatives from stakeholder groups, including the American  Telemedicine Association, the National Organization of State Offices of  Rural Health, the National Rural Health Association, the Center for  Telehealth and E-Health Law, and the National Telecommunications  Cooperative Association, to gain their perspective on the primary Rural  Health Care Program.
section_title: Survey of Pilot Program Participants
paragraphs: To obtain information on how FCC’s design and implementation of the  pilot program affected participants, we conducted a Web-based survey of  pilot projects. For a more complete tabulation of the survey results, see  the e-supplement to this report. To develop the survey questionnaire, we  reviewed comments submitted to FCC by representatives from the pilot  projects and other stakeholders in response to FCC requests for feedback  on the pilot program. We also interviewed pilot project representatives  who were in various stages of the pilot program processes as well as FCC,  USAC, Solix, and stakeholder groups knowledgeable about the program  and issues of concern to participants. We designed draft questionnaires in  close collaboration with GAO survey specialists. We conducted pretests  with four pilot projects that were in various stages of the pilot program  processes to help further refine our questions, develop new questions, and  clarify any ambiguous portions of the survey. We conducted these pretests  in person and by telephone. In addition, we had FCC and USAC review the  survey prior to it being sent to the pilot participants.
We sent our survey to all 61 of the pilot projects that had recent contact  information on file with USAC, as of June 2, 2010. We excluded the Puerto  Rico project because at the time of our survey, it was the only project that  had been withdrawn from the program for an extended period of time;  thus, although we tried, locating a contact with knowledge of the program  was not possible.
Our goal was to obtain responses from individuals with knowledge of and  experience with the tasks related to the pilot program—such as preparing  forms and responding to information requests—for each sampled entity.  Our data set included the name and contact information for each project’s  project coordinator and associate project coordinator. We asked USAC  coaches to identify who they interacted with the most on each project  (project coordinator, associate project coordinator, or someone else), and  we sent the survey to that individual. If that individual was unable to  complete the survey, we asked the other contact (project coordinator,  associate project coordinator, or someone else) to complete the survey.  One respondent was the primary point of contact for two projects, but a  separate survey was completed for each project.
We notified the 61 preidentified contacts on June 2, 2010, by e-mail that  the survey was about to begin and updated contact information as needed.  We launched our Web-based survey on June 8, 2010, and asked for  responses to be submitted by June 18. Log-in information was e-mailed to  all participants. We contacted by telephone and e-mailed those who had  not completed the questionnaire at multiple points during the data  collection period, and we closed the survey on July 2, 2010. All 61 projects  submitted a completed questionnaire with usable responses for an overall  response rate of 100 percent.
We also followed up with certain projects on the basis of survey responses  to gain additional information about plans for using excess capacity, as  well as the extent to which the project was impacted by federal  coordination with the pilot program.
While all pilot projects were selected for our survey, and, therefore, our  data are not subject to sampling errors, the practical difficulties of  conducting any survey may introduce nonsampling errors. For example,  differences in how a particular question is interpreted, the sources of  information available to respondents, or the types of people who do not  respond to a question can introduce errors into the survey results. We  included steps in both the data collection and data analysis stages to  minimize such nonsampling errors. As we previously indicated, we  collaborated with GAO survey specialists to design draft questionnaires,  and versions of the questionnaire were pretested with four members of the  surveyed population. In addition, we provided a draft of the questionnaire  to FCC and USAC for their review and comment. From these pretests and  reviews, we made revisions as necessary to reduce the likelihood of  nonresponse and reporting errors on our questions. We examined the  survey results and performed computer analyses to identify  inconsistencies and other indications of error and addressed such issues,  where possible. A second, independent analyst checked the accuracy of all  computer analyses to minimize the likelihood of errors in data processing.  In addition, GAO analysts answered respondent questions and resolved  difficulties that respondents had in answering our questions. For certain  questions that asked respondents to provide a narrative answer, we  created content categories that covered more than 90 percent of the  narrative responses provided, and asked two analysts to independently  code each response into one of the categories. Any discrepancies in the  coding of the two analysts were discussed and addressed by the analysts.
section_title: Interviews to Assess Federal Coordination
paragraphs: To determine the extent to which FCC coordinated with other federal  agencies when designing and implementing the pilot program, we  interviewed FCC officials regarding the nature of their coordination with  other agencies, and followed up with representatives from other federal  agencies, including the Department of Health and Human Services  (Agency for Healthcare Research and Quality, Centers for Disease Control  and Prevention, Centers for Medicare and Medicaid Services, Health  Resources and Services Administration, Indian Health Service, National  Library of Medicine, Office of the Assistant Secretary for Preparedness  and Response, and Office of the National Coordinator for Health  Information Technology); the U.S. Department of Agriculture’s Rural  Utilities Service; and the Department of Commerce’s National  Telecommunications and Information Administration. We reviewed  relevant documentation and assessed the extent to which FCC  coordinated with other agencies against criteria for coordination  established in prior GAO reports.
section_title: Document Review and Interviews with FCC and USAC on Performance Goals and Measures
paragraphs: To determine the performance goals and measures of the Rural Health  Care Program and how these measures compare with the key  characteristics of successful performance measures, we reviewed the  Telecommunications Act of 1996. We then reviewed our past products and  science and evaluation literature to identify effective practices for setting  performance goals and measures. We compared this information with the  program goals and measures that FCC set forth in agency  documentation—including FCC orders, notices of proposed rulemaking,  strategic plans, and performance and accountability reports. We also  reviewed the Office of Management and Budget’s Program Assessment  Rating Tool 2006 report on the Rural Health Care Program’s effectiveness.  In addition, we interviewed officials from FCC’s Wireline Competition  Bureau and Office of Managing Director, and officials from USAC’s Rural  Health Care Division to obtain their views on plans to implement Rural  Health Care Program performance goals and measures.
section_title: Appendix II: 2008 Commitments to Applicants, by State and Territory
section_title: Appendix III: Comments from the Federal Communications Commission
section_title: Appendix IV: Comments from the Universal Service Administrative Company
section_title: Appendix V: GAO Contact and Staff Acknowledgments
paragraphs: Mark L. Goldstein, (202) 512-2834 or goldsteinm@gao.gov.
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Faye Morrison (Assistant  Director), Elizabeth Curda, Lorraine Ettaro, Cheron Green, Amy Higgins,  Crystal Huggins, Catherine Hurley, Linda Kohn, Armetha Liles, Elizabeth  Marchak, Valerie Melvin, John Mingus Jr., Sara Ann Moessbauer, Charlotte  Moore, Joshua Ormond, Madhav Panwar, Carl Ramirez, Amy Rosewarne,  Cynthia Saunders, Michael Silver, Teresa Tucker, and Mindi Weisenbloom  made key contributions to this report.