Federal Student Aid: Progress in Integrating Pell Grant and Direct Loan Systems and Processes, but Critical Work Remains
Published: 2002-12-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To address system problems and other long-standing management weaknesses, in 1998, the Congress created a discrete unit within the Department of Education, the Office of Federal Student Aid (FSA). This office subsequently adopted a new approach to systems integration using middleware (a type of software that can allow an application to access data residing in different databases) and Extensible Markup Language (XML)--a flexible, nonproprietary set of standards that is intended to make it easier to identify, integrate, and process information widely dispersed among systems and organizations. FSA's first use of this approach is the Common Origination and Disbursement (COD) process for the Direct Loan, Pell Grant, and campus-based programs. GAO initiated a follow-up review to assess FSA's progress in implementing this process.
section_title: What GAO Found
paragraphs: FSA has made progress in implementing the COD process. Specifically, it has implemented (1) a new information technology infrastructure that uses middleware to enable data exchange among disparate systems; (2) the initial version of the basic COD system, which replaces two existing systems and is being used by schools participating in the Pell Grant and Direct Loan programs; (3) middleware into existing systems to support the COD process; and (4) a common record based on XML that schools can use to submit student financial data for the Pell Grant and Direct Loan programs. However, the implementation of the COD process is behind schedule, and its ultimate success hinges on FSA's completing critical work, including addressing serious postimplementation operational problems, and having thousands of postsecondary schools implement the common record. Further, there are important elements to managing any information technology investment that FSA has not yet completed: Determining whether expected benefits are being achieved. FSA has only some of the metrics, baseline data, and tracking processes necessary to determine whether it is achieving all expected benefits. Tracking lessons learned. FSA has relied on an ad hoc approach for gathering and disseminating lessons learned related to schools' implementation of the common record. To address this issue, FSA plans to include lessons learned as part of an update to its school testing guide. However, this does not replace the need for an ongoing mechanism to capture and disseminate lessons learned, without which schools may encounter problems that could have been avoided or mitigated.

section_title: Letter
paragraphs: Over the past decade, the Department of Education has spent millions of  dollars to modernize and integrate its disparate financial aid systems in an  effort to improve the administration of its programs and provide more  information and greater service to its customers—students, parents,  schools, and lenders. However, as we reported last year, the department’s  efforts had achieved limited success. In January 1993, for example, the  department awarded a 5-year, $39 million contract for the development  and maintenance of the National Student Loan Data System, which was to  provide information on systems across programmatic boundaries, yet it  often lacks the most recent information from other financial aid systems.
To address such systems problems and other long-standing management  weaknesses, in 1998, as you know, the Congress created the federal  government’s first performance-based organization, currently called the  Office of Federal Student Aid (FSA). Under the performance-based  organization concept, FSA is a discrete organizational unit within the  Department of Education, led by its own Chief Operating Officer. FSA  focuses on managing the operation of the student financial assistance  programs, while the Department of Education focuses on policy-making  functions.
FSA subsequently adopted a systems integration approach that uses  middleware and Extensible Markup Language (XML) technologies. In  November 2001, we reported that in selecting middleware, FSA adopted a  viable, industry-accepted mechanism for addressing its long-standing  systems integration problems. We also reported that FSA’s first use of  middleware and XML as part of a process for delivering Direct Loan and  Pell Grant aid to students was expected in early 2002. This process, called  the Common Origination and Disbursement (COD) process, was expected  to use a new system, also called COD, and be supported by a new  information technology (IT) infrastructure and various existing systems.
In the summer of 2002, we initiated a follow-up review to our November  2001 report. Our objective was to assess FSA’s progress in implementing  the COD process. In doing this work, we reviewed applicable FSA and  Accenture (the prime contractor) documentation, including FSA’s  modernization blueprint, the COD business case, the COD system’s  requirements, the COD and Enterprise Application Integration  implementation timelines, Accenture project status briefings, and test  guides and results. In addition, we reviewed reports by an independent  verification and validation contractor. We also interviewed applicable  officials from FSA IT and program offices and Accenture.
We performed our work at FSA headquarters in Washington, D.C.,  between August and November 2002, in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: The Department of Education’s FSA manages and administers student  financial assistance programs authorized under title IV of the Higher  Education Act of 1965, as amended. These postsecondary programs  include the William D. Ford Federal Direct Loan Program (often called the  Direct Loan program), the Federal Family Education Loan Program (often  called the Guaranteed Loan program), the Federal Pell Grant Program, and  campus-based programs. Annually, these programs together provide  about $50 billion in student aid to approximately 8 million students and  their families.
During the past three decades, the Department of Education has created  many disparate information systems to support these various student  financial aid programs. In many cases, these systems—run on multiple  operating platforms using different network protocols and maintained  and operated by a host of different contractors—were unable to easily  exchange the timely, accurate, and useful data needed to ensure the  proper management and oversight of the various student financial aid  programs. For example, as we reported in 1997, neither the National  Student Loan Data System nor other systems were designed for efficient  access to reliable student financial aid information, since many systems  were incompatible and lacked data standards and common identifiers. In  addition, because FSA used three separate systems to originate and/or  disburse title IV funds, access to student and school data was fragmented  and unreliable. As a result, FSA found it increasingly difficult to quickly  access data to support day-to-day operational and management decisions,  and schools could not easily access data to obtain a clear picture of the  title IV student aid that had been disbursed.
In September 1999, FSA issued its initial modernization blueprint, which  was subsequently updated in July 2000, to transform the title IV student  financial aid systems using technology. COD is one of four school service  business processes in FSA’s blueprint and is intended to implement a  simplified process for the operation of the Direct Loan and Pell Grant  programs. According to FSA’s modernization blueprint, the common  origination and disbursement process is composed of seven steps  involving students, the Department of Education, and schools: (1) obtain  applicant data, (2) determine eligibility, (3) determine award, (4) notify the  Department of Education of the intent to disburse, (5) obtain funds from  Education, (6) disburse funds to student, and (7) close out. A common  process to support origination and disbursement is considered critical to  FSA’s goal of achieving an enterprisewide solution that provides real-time  data to students, schools, and financial partners via Web portals.
To implement COD, FSA is using middleware and XML technologies.  Specifically, middleware is being used to integrate FSA systems that  support the COD process. Traditionally, systems integration would  require building separate point-to-point interfaces between every two  applications. Although this approach can be effective, it creates several  problems, such as (1) every connection between two applications requires  custom programming; (2) a lot of connections have to be developed when  there are multiple data sources; and (3) whenever the logic or data in one  application changes, the accompanying interface often also needs to be  altered. Middleware represents an alternative means to the traditional  approach, and it can provide a quicker and more robust solution to  systems integration. In essence, middleware separates the business  application from the technical details of interapplication communications.  Thus, middleware can simplify and reduce the number of interfaces for  multiple systems because it can handle differences in data formats and  record layouts.
As part of the COD process, XML is being used to consolidate multiple  legacy record formats previously used by schools to submit data on the  Pell Grant and Direct Loan programs. By using an XML-based common  record, schools can transmit one file with all of the student’s data instead  of submitting separate legacy records with redundant student and school  information.
Appendix I provides a high-level depiction of the systems and technologies  supporting the COD process as of November 2002. As depicted, the COD  system can translate or convert legacy records by using middleware. In  addition, middleware has been built into several existing systems so that  they can establish connectivity and exchange data with the COD system  through a common IT infrastructure. This IT infrastructure, called the  Enterprise Application Integration (EAI) bus, is also implemented using  middleware to route data between systems in a correct format. In addition,  as part of the COD process, some schools have begun submitting Pell  Grant and Direct Loan data using the XML-based common record.
FSA hired Accenture as its “modernization partner” to help carry out its  modernization blueprint, including the implementation of the COD  process. Accenture is the prime contractor providing leadership of  critical planning activities that are essential to the success of FSA’s  modernization. Regarding the COD system part of FSA’s modernization,  FSA also hired an independent verification and validation contractor to  review the initial release of this system, which was completed earlier this  year.
section_title: Progress in Implementing the COD Process, but Critical Work Remains, and Benefits and Lessons Learned Are Not Being Tracked
paragraphs: FSA has made progress in implementing the new COD process. In  particular, it has begun implementing (1) its middleware solution in its IT  infrastructure and various existing systems, (2) the COD system, and   (3) an XML-based common record. However, FSA’s implementation of  COD is behind schedule, and critical work remains to be completed. For  example, the basic COD system was to be completed by mid-October 2002;  however, only about three-quarters of the COD basic system requirements  had been implemented as of October 23, 2002. In addition, FSA is not  tracking whether it is achieving certain benefits because it is still in the  process of defining applicable metrics to measure progress. Without such  tracking processes, FSA lacks critical information about whether it is  achieving expected benefits. Finally, FSA lacks assurance that it has  captured and disseminated important lessons learned related to schools’  implementation of the common record because it believes that its current  ad hoc process is adequate. Accordingly, the thousands of schools that  have not yet implemented the common record may not benefit from the  experience of those that have.
subsections: section_title: FSA Has Begun Implementing COD Capabilities, but Critical Work Remains
paragraphs: FSA has made progress in implementing COD. The following are  significant elements of the COD process that have been implemented:    Deployment of the EAI bus. As a prerequisite to implementing COD, in late  October 2001, FSA deployed its middleware solution in an EAI “bus”—an  IT infrastructure that uses middleware to access data from disparate  systems, transform the data formats as necessary, and route the data to  the appropriate requesting systems, thus enabling data exchange among disparate systems. The EAI bus provides the set of technical capabilities  necessary to integrate FSA’s disparate systems.
Initial implementation of basic COD system (release 1.x). On April 29,  2002, FSA went live with version 1.0 of the basic COD system. As of mid- November 2002, FSA had released an additional five sub-versions of the  COD system (e.g., version 1.1). The COD system replaces the Direct Loan  Origination System and the Recipient Financial Management System, and  it currently processes files for all schools participating in the Pell Grant  and Direct Loan programs. According to FSA, in the first 6 months of its  operation, the COD system processed just under 16 million transactions,  representing Pell Grant and Direct Loan awards totaling almost $10 billion  to over 5 million recipients.
Implementation of middleware in selected systems. As of mid-November  2002, FSA had built middleware into seven systems so that these systems  can interact with the COD system through the EAI bus. These systems  include (1) the Central Processing System, which determines students’  eligibility and award levels, and (2) the National Student Loan Data  System, which contains loan- and grant-level information and is used by  schools to screen student aid applicants to identify ineligible borrowers.
Development and implementation of the common record. Using XML,  FSA developed and began implementing a common record that schools  can use to submit student financial aid data to the COD system. The  common record, designed with assistance from members of the National  Council of Higher Education Loan Programs and the Postsecondary  Electronic Standards Council, consolidates multiple legacy file formats  previously used by the Pell Grant and Direct Loan programs.
Although FSA has made progress in implementing the COD process,  critical work remains to be completed. First, FSA is behind schedule in  implementing the basic COD system. Although FSA had planned to  complete the basic COD system by mid-October 2002, only about three- quarters of the COD basic system requirements had been implemented as  of October 23, 2002. For example, as of early November 2002, one of the  basic business functions that remains to be implemented is to enable FSA  to make automated adjustments in batches to school current funding  levels. FSA now estimates that most of the remaining functionality will be  completed by the end of September 2003. According to FSA IT and  program officials, the implementation of the basic COD functionality was  delayed to allow adequate time for testing to ensure the quality of the  system.
Second, as of November 19, 2002, Accenture reported several operational  problems that needed to be addressed. For example, in some cases, the  COD system was incorrectly processing school batch data that contained  multiple change records for an individual student. According to COD and  contractor officials, the causes of operational problems included unclear  requirements and software design defects. An independent verification  and validation contractor also found problems with the requirements and  design aspects of release 1.0. The COD Contracting Officer’s  Representative characterized these operational problems as very serious  and stated that they could impede operations and the delivery of future  COD releases. This same official noted that FSA and Accenture are  currently undertaking efforts to address these problems. For example,  FSA has established production teams composed of agency and contractor  staff to address problems in specific areas. In addition, FSA has  established a continuous improvement process to more rigorously manage  its relationship with Accenture.
Third, fewer postsecondary schools than planned have implemented the  common record. FSA had estimated that 50 schools (out of about 5,500)  would implement the common record in fiscal year 2002. However, as of  November 26, 2002, only 22 schools had implemented and tested the  common record with FSA. FSA COD officials attributed the fewer-than- expected number of schools using the common record to schools and  vendors not being ready to implement it. FSA expects that the number of  schools using the common record will be considerably higher during the  next award year (2003–2004) because, by April 2003, it plans to implement  and test the common record with EDExpress, a software application FSA  distributes free of charge to about 3,000 schools for use in submitting data.  In addition, FSA expects that all schools will be using the common record  format by March 2004, in time for the 2004–2005 award year.
section_title: FSA Is Not Completely Tracking Actual COD Benefits or Lessons Learned Related to Schools’ Implementation of the Common Record
paragraphs: In its COD business case, FSA outlined five expected benefits: (1) reduced  cost, (2) increased customer satisfaction, (3) increased employee  satisfaction, (4) increased financial integrity, and (5) the integration and  modernization of legacy systems. An important aspect of implementing an  IT investment cited by the Office of Management and Budget and our IT  investment management guide is evaluating the results of the investment  by determining whether such expected benefits are being achieved.  However, as illustrated in table 1, at this time FSA has only some of the  data necessary to determine whether it is achieving all expected benefits.  In particular, for the increased customer satisfaction and financial  integrity benefits, FSA (1) has not fully defined the performance metrics to  be used, (2) does not have all baseline data, and/or (3) is not fully tracking  whether the benefits are being achieved. In these cases, FSA COD officials  stated that they were in the process of developing relevant metrics, which  would be tracked to measure the project’s performance against expected  benefits. However, until FSA develops these data and begins tracking  actual benefits and comparing them with expected benefits, it will lack  vital data with which to demonstrate actual investment results.
FSA IT officials also stated that they plan to have a contractor conduct a  postimplementation review of the COD basic system in fiscal year 2003,  which is expected to look at the achievement of expected benefits. While  this is an important initiative that could provide FSA with valuable  information, it does not take the place of a continuing and systematic  process of tracking actual benefits.
According to our IT investment management guide, another critical  activity is establishing a process for developing and capturing lessons  learned in a written product or knowledge base and disseminating them to  decision-makers. Lessons-learned mechanisms serve to communicate  acquired knowledge more effectively and to ensure that beneficial  information is factored into planning, work processes, and activities.  Lessons learned can be based on positive experiences that save money or  on negative experiences that result in undesirable outcomes.
FSA has recognized the importance of generating lessons learned in  certain areas. For example, it has implemented a process for developing  lessons related to managing the relationship between the agency and its  prime contractor. However, FSA lacks such a process for capturing or  disseminating lessons related to school migration to the common record.  FSA COD officials stated that lessons learned pertaining to school  migration to the common record are addressed through periodic  discussions during biweekly conference calls with schools undergoing  testing with FSA and during portions of various FSA-sponsored  conferences. FSA COD officials stated that they believed this process for  capturing and disseminating lessons learned was adequate.
However, by relying on such an ad hoc process, FSA lacks assurance that  it has captured and disseminated all key lessons learned related to  schools’ implementation of the common record and could overlook  important improvements that could be made. In addition, schools that do  not attend the conferences may not receive and benefit from the lessons  identified in the initial phase of implementation. As a result, schools may  encounter problems that could have been avoided or mitigated had they  known of other schools’ experiences. This could hamper FSA’s ability to  facilitate the transition of schools to the new common record and thus the  agency’s ability to fully implement the new COD process and achieve the  expected benefits.
In commenting on a draft of this report, FSA stated that it plans to provide  lessons learned as part of a planned update to its school testing guide.
While this is a positive step, it does not replace the need for mechanisms  to continuously capture and disseminate acquired knowledge as schools  implement the common record.
Table 2 includes examples of lessons learned provided by FSA at our  request that were drawn from schools’ initial implementation of the  common record for the 2002–2003 award year. Such information would be  important for the thousands of schools that have not yet implemented the  common record so that they can avoid problems during the common  record implementation and testing processes.
section_title: Conclusions
paragraphs: FSA has taken important steps toward achieving full implementation of  the new COD process. However, critical actions, such as completing the  basic functionality of the COD system and the implementation of the  common record at thousands of affected schools, must still be undertaken.  In addition, FSA has not yet fully established the metrics and processes to  track actual benefits related to all of its expected benefits or the lessons  that have been generated by the few schools that have implemented the  common record thus far. By not tracking actual benefits, FSA lacks  information that is critical to determining whether it is meeting all of its  goals. Further, not capturing and disseminating information to schools  regarding lessons learned could make achieving these goals more difficult.
section_title: Recommendations
paragraphs: To determine the extent to which the new COD process is achieving  expected results related to customer satisfaction and financial integrity,  we recommend that you direct FSA’s Chief Operating Officer to  expeditiously develop metrics and baseline data to measure these benefits  and develop a tracking process to assess the extent to which the expected  results are being achieved.
To ensure that the schools that have not yet implemented the common  record benefit from the experiences of those that have, we recommend  that you direct FSA’s Chief Operating Officer to establish a process for  capturing lessons learned in a written product or knowledge base and for  disseminating them to these schools.
section_title: Agency Comments and Our Evaluation
paragraphs: In providing written comments on our draft report, FSA’s Chief Operating  Officer provided technical comments and updated information, but did not  comment on our recommendations. Specifically,    The Chief Operating Officer did not believe the report adequately  portrayed the level of COD progress that had been made. In particular, she  took issue with our using the completion of 75 percent of COD  requirements as an indication of progress. Although the Chief Operating  Officer did not disagree with the accuracy of this figure, she stated that  FSA’s informal analysis indicated that between 85 to 90 percent of COD  functions had been implemented, which she believed was a better gauge of  progress. We believe that we have accurately portrayed FSA’s progress in  implementing the COD process. First, since FSA’s analysis was “informal,”  and FSA’s supporting documentation had limited detail that we could not  validate, we do not agree that this should be the primary basis for an  analysis of COD’s progress. Second, we included both the percentage of  COD’s requirements that had been implemented and FSA’s estimate in our  report. Nevertheless, we modified our report to include additional data  provided by FSA regarding the number of transactions processed by the  COD system to further indicate progress.
The Chief Operating Officer agreed that the tracking of all of the expected  benefits is not in place at this time, but stated that work is under way in  this area. FSA also provided updated information and supporting  documentation related to the tracking of some of the expected benefits.  We made changes to the report reflecting this new information, as  appropriate.
The Chief Operating Officer agreed that it is important and beneficial to  communicate lessons learned, but stated that FSA’s informal method for  communicating lessons related to school migration to the common record  worked well in the first year of COD implementation. FSA also noted that  it plans to include lessons learned in a planned update to its school testing  guide. We modified the report to reflect this initiative, but we do not agree  that FSA’s informal method or its plan to include lessons learned in its  testing guide is adequate because these approaches do not provide a  continuous process for actively capturing and disseminating lessons  learned. As a result, some important lessons may be overlooked, and all  schools may not be aware of potential problems associated with  implementing the common record.
FSA’s written comments, along with our responses, are reproduced in  appendix II.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Education, the Chief Operating Officer of  Education’s Office of Federal Student Aid, and the Director of the Office of  Management and Budget. This report will also be available at no charge on  the GAO Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please  contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director, at  (202) 512-9556. We can also be reached by E-mail at pownerd@gao.gov and  lambertl@gao.gov, respectively. Other individuals making key  contributions to this report included Jason B. Bakelar and Anh Q. Le.
section_title: Appendix I: Systems and Technologies Supporting the Common Origination and Disbursement Process, as of November 2002
subsections: section_title: Middleware
paragraphs: Legacy records (MQSeries) (MQSeries) (MQSeries)
Single common record file (XML)
18 legacy files      (secured FTP)
FTP        File Transfer Protocol LOWeb   Loan Origination Web NSLDS   National Student Loan Data System PEPS     Postsecondary Education Participants System XML      Extensible Markup Language This is a temporary interface. DLOS is targeted to be retired in fiscal year 2003.
MQSeries is IBM’s proprietary message format.
CThis is an information technology infrastructure that enables data exchange among disparate  systems.
section_title: Appendix II: Comments from the Department of Education’s Office of Federal Student Aid
paragraphs: The following are GAO’s comments on the Department of Education’s  Office of Federal Student Aid’s letter dated December 18, 2002.
subsections: section_title: GAO Comments
paragraphs: 1.  We revised the draft report title to clarify that our follow-up review  was focused on assessing FSA’s progress in implementing the COD  process.  2.  Information related to operational problems was contained in the draft  report. We asked the COD Contracting Officer’s Representative to  characterize these problems, and he stated that they were very serious.  In addition, we confirmed the seriousness of these problems at the  conclusion of our review with FSA IT and program officials.  3.  We modified this report to include data on the number of transactions  processed. We also modified our report to clarify that all schools  participating in the Pell Grant and/or Direct Loan programs currently  use the COD system.  4.  We do not agree that the report implies that FSA’s use of a phased-in  approach in implementing the common record increases risks. Instead,  this report notes that the implementation is not yet complete.  5.  We believe that we have accurately portrayed FSA’s progress in  implementing the COD process. First, since FSA’s analysis was  “informal,” and FSA’s supporting documentation had limited detail that  we could not validate, we do not agree that the COD Development  Manager’s functionality estimate should be the primary basis for an  analysis of COD’s progress. Second, we included both the percentage  of COD’s requirements that had been implemented and FSA’s estimate  in our report.  6.  We modified our report to reflect this updated information as  appropriate.  7.  We do not agree that FSA’s informal process for capturing and  disseminating lessons learned was adequate because (1) it may lead to  important lessons being overlooked and (2) all schools may not be  aware of potential problems associated with implementing the  common record.  8.  We modified this report to reflect that FSA plans to include lessons  learned in a planned update to its school testing guide. While this is a  positive step, it does not replace the need for mechanisms to  continuously capture and disseminate acquired knowledge as schools  implement the common record.