Information Technology Modernization: Corporation for National and Community Service Needs to Develop a System That Supports Grant Monitoring
Published: 2017-08-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: CNCS engages more than five million Americans yearly in national volunteer service by awarding grants to programs such as AmeriCorps and Senior Corps. In fiscal year 2016, CNCS received almost $800 million in appropriations to fund approximately 2,300 grants. The agency is taking steps to modernize its outdated IT infrastructure, including developing and delivering systems to replace its legacy grant management system and improve IT support of the agency's grant monitoring program. In a March 2017 report, GAO noted system deficiencies that resulted in usability, data quality, and other technical issues.
This report examines (1) the extent to which CNCS's IT modernization projects align with business and management needs for grant monitoring and (2) progress CNCS has made toward ensuring the successful and timely delivery of new systems to support grant monitoring. GAO assessed agency planning and IT project management documents against key practices and conducted interviews with agency officials.
section_title: What GAO Found
paragraphs: The Corporation for National and Community Service's (CNCS) information technology (IT) modernization projects are currently planned to align with the agency's business and management needs for its existing process, but are not yet defined for a future risk-based process. The projects include the development of a modernized system in two versions. The first version was planned to provide support for business needs of the agency's existing grant monitoring process. The second version is to provide additional functionality to support monitoring within a yet-to-be-defined risk-based process. CNCS officials and the development contractor responsible for delivering the first version of the system agreed to a set of requirements that address business needs for improving outcomes of the existing monitoring process. However, because business needs for a risk-based monitoring process have not been determined, OIT officials and system stakeholders have not defined requirements for the second version of the future system, as intended by CNCS's IT modernization plans.
CNCS has taken steps to help avoid continued delays, but progress toward delivering the system has been limited. In July 2015, CNCS initiated a project that was to deliver the first version of the system in April 2016. After subsequent delays, agency officials updated plans to reflect a September 2016 delivery. However, as of July 2017, this version had not been delivered, and the delivery date was changed to October 2017. Successful development and delivery of IT systems relies on adherance to key practices for managing project schedules and testing. However, weaknesses in CNCS's practices introduced risks to successful delivery of the system. In particular, the system development project schedules could not be used to track progress because they did not include actual dates when activities were started and finished. In addition, although CNCS and its contractor conducted testing according to plans, the grant monitoring system was not included in all phases of testing. Agency officials used other tools to track progress, and plans did not require them to conduct all stages of testing for the system. However, unless CNCS officials improve system development practices for managing project schedules and testing, they will continue to introduce risks to successful delivery of system functionality that supports grant monitoring.
The grant monitoring system development project experienced delays when CNCS did not initially conduct oversight needed to ensure that the contractor took corrective actions as planned. In monitoring the contractor's performance during system development and testing, agency officials enhanced oversight to avoid continued delays. In ongoing management reviews of the project, CNCS officials reported that, since the last corrective action plan was provided by the contractor in December 2016, performance has improved and the project is on track to deliver the first version of the grant monitoring system in October 2017.
However, successful delivery of system functionality that supports the agency's grant monitoring process will remain at risk unless CNCS takes steps to correct weaknesses in system development practices for managing project schedules and testing. Further, CNCS's grant monitoring officers will continue to rely on an outdated legacy system to support the processes they follow for monitoring the use of millions of dollars of grant funds awarded each year.
section_title: What GAO Recommends
paragraphs: GAO is recommending three actions to improve CNCS's practices for defining requirements and developing schedules and test plans for grant monitoring system development. In comments on a draft of this report, CNCS did not fully agree with the recommendations. Based on additional documentation that CNCS provided, GAO revised the report, as appropriate, and modified the recommendations.

section_title: Letter
paragraphs: The Corporation for National and Community Service (CNCS), as the  nation’s largest federal grant maker for service and volunteering, engages  more than five million Americans in national volunteer service initiatives to  support the American culture of citizenship, service, and responsibility. Its  mission is to improve lives, strengthen communities, and foster civic  engagement through service and volunteering. To do so, the agency  awards grants and provides technical assistance to nonprofit volunteer  programs like AmeriCorps and Senior Corps that, in turn, provide services  to address a range of community challenges, such as tutoring children  and mentoring individuals with special needs. In fiscal year 2016, CNCS  administered grants totaling about $784 million to support national  service, and the agency reported that nearly 325,000 Americans  participated in its major programs.
However, the CNCS Office of Inspector General (OIG) reported in March  2014 that the agency was experiencing problems with its grant  management processes, noting that the agency did not have a  comprehensive risk management strategy. The OIG also reported that  the agency’s grant monitoring process and the legacy system used to  support the process were outdated, inefficient, and ineffective.
In October 2014, CNCS established plans to modernize its information  technology (IT) computing environment that included projects to develop  new systems to support its operations. Among other operations, the IT  modernization plan addressed the need for improved system support for  the management of grants, including processes for monitoring the use of  funds awarded by its grant programs.
To support the committee in its oversight of CNCS’s grant management  activities, we evaluated the extent to which the agency’s IT modernization  plans address improved technical support of grant monitoring processes.  Our specific objectives were to determine (1) to what extent CNCS’s  planned IT modernization projects align with business and management  needs for grant monitoring and (2) what progress CNCS has made  toward ensuring the successful and timely delivery of new systems to  support its grant monitoring process.
To determine the extent to which CNCS’s modernization projects aligned  with business and management needs for grant monitoring, we collected  and reviewed documentation that described CNCS’s plans regarding its  IT modernization projects, including the IT Modernization Program  Management Plan and software development planning documents  related to business needs for grant monitoring system support. We  compared the system functional requirements defined in the  documentation to business and user needs described in the  modernization plan and identified consistencies and any gaps between  them. We also compared the relevant contents of the modernization and  software development plans to the agency’s business planning  documentation, such as the CNCS Strategic Plan, to determine the extent  to which the IT plans reflected business and user needs for grant  monitoring support.
To determine the progress CNCS had made to ensure the successful and  timely delivery of systems to support grant monitoring, we collected and  assessed documentation that provided evidence of planned and ongoing  system development and implementation activities, such as project  schedules and test plans. We compared reported actual dates and  milestones for completing the activities to planned dates documented in  IT modernization project management artifacts, such as reports of project  performance and testing results, to determine whether milestones had  been met and the extent to which system functionality had been delivered  to support grant monitoring. Further, we obtained and assessed  documentation describing activities of the agency’s system development  contractor and agency officials’ conduct of project requirements  management, testing, scheduling, and oversight. We compared the  actions taken to software development practices defined by the Software  Engineering Institute (SEI). The scope of our study did not include a full  assessment of CNCS’s software development practices.
We supplemented the information we obtained from our assessment of  CNCS documentation with interviews of CNCS officials who are  knowledgeable of the overall IT modernization program, including the  practices used to determine requirements and to develop new grant  monitoring system capabilities. These officials included the agency’s  Chief Information Officer (CIO), Chief Operating Officer, and Chief Risk  Officer.
To determine the reliability of the data obtained from CNCS’s information  systems used for managing requirements, conducting system testing, and  tracking system defects found during testing, we reviewed relevant  documentation, such as requirements definitions, test plans and reports  of results, and reports from a project tracking tool. We compared the data  collected from the documentation to actual results of activities, reflected in  project status reports and schedule updates, to determine the integrity  and reliability of the data. To further confirm data reliability, we  interviewed agency officials knowledgeable of the system development  projects and the agency’s IT management processes. We determined  that the data we used from these sources were sufficiently reliable for the  purposes of our audit. A full description of our objectives, scope, and  methodology can be found in appendix I.
We conducted this performance audit from January 2016 through August  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: CNCS was created in 1993 with the enactment of the National and  Community Service Trust Act of 1993. The agency is led by a Chief  Executive Officer and Board of Directors who are appointed by the  President, and with the advice and consent of the Senate. The Chief  Executive Officer oversees the agency, which includes about 600  employees operating throughout the United States and its territories.
The agency provides grants to volunteer organizations throughout the  United States to strengthen communities and foster civic engagement.  For example, the National and Community Service Trust Act of 1993  established the CNCS and authorized it to make grants such as through  the AmeriCorps state and national grant programs. The act also made the  agency responsible for administering Volunteers in Service to America  (VISTA) and National Senior Volunteer Corps. The Serve America Act  was also enacted in 2009, which gave CNCS responsibility for  administering several newly established programs, such as the Social  Innovation Funds, which provide community-level grants.
CNCS grants are typically awarded to fund programs for a 3-year period,  with funds distributed annually. In fiscal year 2016, CNCS received  appropriations totaling more than $787.9 million and used $783.9 million  to fund approximately 2,300 grants. Table 1 shows the amounts and  number of grants disbursed by each of the programs in fiscal year 2016.
subsections: section_title: CNCS’s Grant Management Process
paragraphs: Within CNCS, the Chief of Program Operations, Chief Risk Officer, and  Chief Financial Officer share responsibility for managing the grants.  Program officers under the Chief of Program Operations and grant  officers under the Chief Financial Officer manage the administration of the  grants, including monitoring the use of grant funds by grantees. The  program officers focus on issues related to grantee performance and  compliance with program objectives, while the grant officers focus on  grantees’ financial issues and performance.
The program and grant officers use IT systems to support monitoring and  related activities. These systems are developed and maintained by the  Office of Information Technology (OIT). This office is led by the agency’s  CIO, who reports to the Chief Operating Officer. Figure 1 illustrates the  position of the grant management entities within the agency as of April  2017.
To manage its grant program, CNCS conducts a four-phase process that  covers pre-award grant reviews through actions taken to close out the  grants when the period of related projects expires. According to CNCS  policy, during the pre-award phase (before CNCS makes a grant award),  grant officers evaluate the potential grantee’s financial management  capabilities and other aspects, such as whether the grantee has any open  audit findings on any current or prior grants, to determine if the grant  should be awarded.
Once grants are awarded for the upcoming fiscal year, CNCS program  and grant officers take steps to assess the grants and plan for monitoring  the use of funds provided to the grantees. During the agency’s annual  assessment phase, which is conducted between August and October of  each year, the officers establish the upcoming year’s monitoring plan.  During the monitoring phase, program and grant officers carry out the  plans made during the assessment phase.
Lastly, when the period of the project supported by a grant expires, the  grant is closed. During closeout and no later than 90 days after the  expiration date of the grant, grantees submit required documentation to  their grant officer and pay any outstanding obligations. CNCS’s program  and grant officers verify completion of these actions and reconcile funds,  then notify grantees when they have met all programmatic and fiscal  requirements of the award. At that time, the grant is officially closed. An  overview of the four phases of the monitoring process is depicted in figure  2.
Grant monitoring activities are initiated by program and grant officers  during the assessment phase, when awarded grants are assessed, rated,  and prioritized, and monitoring plans are made. The assessment phase is  made up of four steps.
In the first step of the assessment phase, officers determine which  grants are to be monitored in the upcoming year. The universe of  grants being assessed typically includes all grants that are active at  the time CNCS is ready to begin the assessment phase, usually in  mid-August of each year, and that are expected to be active during  the fiscal year beginning in October.
At the second step of the process, the officers further assess each of  the grants to be monitored to identify potential vulnerabilities based on  a set of criteria related to program compliance, financial weakness, or  other issues defined by the program and grant officers. CNCS defined  19 criteria that indicate a grant’s potential vulnerabilities to be  examined during an assessment. The officers may consider other  key concerns or challenges in their assessment on a case-by-case  basis.
Based on the outcomes of the second step, a third step is taken to  determine the priority of each grant for monitoring purposes. Each  grant is rated as high, medium, or low priority to determine the types  of monitoring activities to be conducted during the year. The three  priority categories are determined based on the presence of any of  the pre-determined 19 criteria and an automated scoring process  performed by the IT system that is used to support the agency’s grant  monitoring process. The plan typically is to be prepared by the end of  October.
Finally, during the fourth step of the annual assessment, program and  grant officers plan the monitoring activities for each grant that are to  be conducted during the year, based on the priority ratings  determined during the third step. For example, grant monitoring plans  may include on-site compliance visits to the grantees’ facilities, desk  reviews of documentation submitted by the grantees, reviews of  grantees’ financial activities, and follow-up visits based on the  outcomes of compliance visits and other reviews.
The monitoring phase is conducted each year between October and  August, when program and grant officers are to complete the activities  described in their plans. During that time, they may add or omit activities,  with supervisory approval, for a number of reasons. For example,  program and grant officers may consider adding an activity for a grantee  based on newly identified issues, such as findings of the OIG that suggest  the grantee may have demonstrated poor financial management  capabilities in the past. In such cases, the officers may add additional on- site or follow-up visits to their plans.
The officers produce reports on the progress made by grantees to  accomplish the objectives and meet performance measures associated  with a particular grant. For example, the reports could include data that  reflect the number of individuals who received assistance in disaster  preparedness, mitigation, response, and recovery efforts; the percentage  of economically disadvantaged people who received housing-related  support; or the percentage of students served by a grant program who  showed improvements in academic performance. In addition, the officers  review grantees’ financial status reports that identify funds expended  compared to funds allocated for the grantees’ awards through the CNCS  budget. The information reported is used by the program and grant  officers to help them identify areas that may need more focused attention  or targeted monitoring.
In our March 2017 report on CNCS’s efforts to improve grant monitoring,  we noted that the agency’s annual assessment process may not result in  the riskiest grants receiving a high priority for monitoring because of  limitations in its scoring model. We also noted that CNCS has not  evaluated its monitoring efforts to identify opportunities to improve its  assessment of, and response to, risks related to grant management. We  recommended that CNCS, as part of its efforts to establish enterprise- wide risk-based management, develop a risk-based approach to grant  monitoring and establish a policy to ensure that all grants expected to be  active in a fiscal year are assessed for potential risk. Such an approach  would incorporate key practices, such as identifying risks relevant to the  use of funds awarded by the grant program, analyzing and responding to  the risks, and addressing risks as part of the grant monitoring process.  The agency did not comment on this recommendation and has not yet  addressed it.
section_title: Use of IT to Support CNCS’s Grant Management Process
paragraphs: CNCS’s legacy IT system, eGrants, has been used since 1998 to provide  functionality needed to support the agency’s grant management process.  Grant management includes annual assessment and monitoring activities  such as planning, documenting related activities, generating  correspondence, and reporting financial status and progress of grantees’  actions toward meeting performance goals. CNCS officials reported that,  in fiscal year 2016, they spent about $11 million maintaining their legacy  IT environment, which includes eGrants and other systems used to  manage functions such as payroll and benefits.
To support the annual assessment process, program and grant officers  record in eGrants the results of their evaluations of each grantee. Such  evaluations are based on the 19 criteria that were defined to determine  the priority of the grants to be monitored and the monitoring activities to  be conducted. The officers indicate which of the criteria are present by  choosing “yes” or “no” from a drop-down list. Then, based on the criteria,  the system sorts and rates the priority of each grant as high, medium, or  low. Figure 3 depicts the input screen of the eGrants Monitoring and  Oversight Assessment Module, which is used by the officers to select  criteria used to indicate the priority level of grants.
Once the assessment has been completed, the program and grant  officers record activities that they plan to take during the year, such as  conducting a follow-up visit with the grantee based on any audit findings  in eGrants, along with the proposed time frames for conducting the  activities. The program and grant officers also use eGrants to generate  notification letters that are sent to grantees to schedule a site visit. After  the visit has been conducted, the grant or program officers enter  information regarding the results of the visit into the system, along with  the grantee’s responses.
After a site visit, the officers may also identify follow-up actions to be  taken and use eGrants to record those actions. The program and grant  officers also use the system to generate reports on the progress of the  grantees toward addressing any deficiencies identified as a result of  monitoring activities they conducted. Grant officers also enter information  about the results of financial reviews into the system.
However, CNCS has encountered challenges in using eGrants. For  example, according to CNCS’s February 2013 IT Modernization Plan,  data and information are not standardized and are entered into the grant  management and monitoring systems with limited checks and validation,  which results in data quality issues. In addition, according to the plan,  technical limitations of eGrants make it difficult and time-consuming to  integrate data from multiple sources and filter out inaccuracies. For  example, agency officials stated that initiatives such as reporting on state- level grants require significant intervention by technical staff to obtain,  integrate, and analyze data due to the disparate nature of the way the  data are stored.
In October 2013, the CNCS OIG also reported problems with the  agency’s legacy IT system used to support monitoring activities. The  OIG reported that eGrants was inefficient and ineffective at supporting  monitoring officers’ efforts to predict and detect improper use of funds  awarded by CNCS grants. The OIG noted that, although the agency had  collected data from its grantees to monitor their use of funds, its outdated  systems did not provide the kind of data analytics capabilities needed to  monitor early detection of fraud and mismanagement of grant funds. For  example, the OIG reported that CNCS could not readily compare  information across grants awarded or among the grantee population to  identify anomalies and outliers that indicate potentially improper use of  funds. The OIG further reported that the agency was falling behind in its  efforts to increase the use of technology to support improved risk  management practices, relying instead on inefficient processes for  conducting oversight of the use of grant funds.
Recognizing the growing gap between its business needs and its IT  infrastructure, CNCS engaged a contractor to provide guidance on  modernizing its infrastructure. In May 2014, the contractor reported the  results of its independent evaluation of the agency’s IT strategy,  modernization plan, technology adoption, and costs for supporting  CNCS’s strategic goals. The evaluation highlighted the limited system  collaboration and communication capabilities among CNCS end users,  data inconsistency, system availability issues, and data transparency and  reporting issues as areas needing improvement. In its report, the  contractor noted that CNCS had been slowly migrating its legacy grant  management system to a new, more current technology, but that its  efforts had not kept pace with mission-related programmatic demands.  Consequently, program staff had developed workaround solutions to  manage critical information accessed from multiple disparate sources,  such as spreadsheets, e-mail, local and shared files, and local databases.
Our March 2017 report on CNCS’s grant monitoring process identified  similar problems associated with eGrants. We reported that, according  to CNCS officials, eGrants does not automatically produce standard  historical reports, and, in order to produce a report from the system,  significant manual manipulation of the data is required. We also  discussed system users’ concerns regarding the usability of eGrants and  its lack of integration with other tools used to collect monitoring data. For  example, although each program officer has an electronic monitoring tool  to use when conducting on-site monitoring, the data collected using these  tools are not automatically integrated into and analyzed by eGrants. We  further noted CNCS officials’ concerns that the agency’s existing IT  environment did not provide automated mechanisms, such as trend  analyses, that could help them identify patterns of potentially improper  use of funds and risk factors to be considered when monitoring grants  within a risk-based management approach.
section_title: CNCS’s IT Modernization Program
paragraphs: In February 2013, CNCS’ CIO released its IT modernization plan that  describes a program intended to result in the development of systems to  improve support of the agency’s grant management activities. Program  documentation describes the implementation of systems that, when  executed collectively, are to provide grantees and agency staff using the  systems with new tools and capabilities to increase data quality and  transparency, staff productivity, and program effectiveness. The IT  modernization program has two overall goals.
Modernize grant management systems—Re-architect systems and  standardize databases for greater agility; improve data quality and  provide a better user experience through new data edits and  validations, pre-populated forms, optimized navigation, and context- driven screens; maximize staff productivity by consolidating core staff  functions currently in multiple systems into a common system.
Enhance mobile web and information sharing—Optimize key CNCS  information for mobile web viewing and develop capabilities to accept  grantee system information and share CNCS data with the public.
CNCS’s IT modernization program consists of three phases with projects  in each phase that are intended to result in improvements to the overall IT  computing platform, retirement of outdated legacy systems, and  replacement of the retired systems. Specific projects are to be conducted  as part of an overall initiative intended to result in the retirement of  outdated legacy systems and the implementation of a modernized  computing environment, referred to as the Grants and Member  Management (GMM) platform.
Phase 1 of the program includes projects to result in the development  and delivery of systems to support existing grant management  processes, such as planning and applying for grants, reviewing  applications, awarding funds, reporting on grantee and project status,  monitoring the use of grant funds and grantee activity, and closing out  grants and projects.
Phase 2 includes projects to develop systems to support services  such as member recruitment, training, travel, awards, and enterprise  risk management, including grant monitoring and oversight. This  phase includes a project to deliver system capabilities that support the  agency’s grant monitoring process as part of a future enterprise-wide  risk management approach. The delivery of the systems to be  developed during the first two phases is intended to allow the agency  to retire outdated legacy systems.
Phase 3, the final phase, is to retire any remaining legacy systems to  complete the transition to the new GMM platform.
CNCS defined IT modernization projects to be completed throughout the  three phases of the initiative, such as the development of modernized  systems to be used by the agency’s staff to manage grants, and the  transition of all system functionality from the legacy environment to the  platform. According to the modernization plan, the new systems are to be  deployed via a technology platform that allows users to integrate data  from external data sources in real time. Specifically, the platform is to be  based on cloud computing that, according to CNCS’s IT strategic plan, is  expected to lower IT costs, improve system access and stability, and  enable IT staff to focus on supporting the needs of agency stakeholders  who are to use the systems.
To implement the GMM cloud-based platform and develop the new  systems, CNCS acquired the services of a development contractor  through a task order that specifies software and services on a time and  materials basis. The task order for the implementation of GMM was  issued in September 2014, and the contractor began work on Phase 1,  including the first modernized grant management system development  project, in October 2014.
In January 2017, agency officials estimated the cost to complete all of the  activities associated with the projects during the three phases of the IT  modernization program to be $43.6 million—$39.3 million for labor and  $4.3 million for software and services.
section_title: CNCS’s IT Modernization Projects Are Aligned with Business and Management Needs of the Existing Grant Monitoring Process, but Are Not Defined for a Future Risk- based Approach
paragraphs: To define system requirements that meet users’ business needs, key  practices identified by SEI call for, among other things, ensuring that the  business needs are understood and that supporting system requirements  are defined and agreed upon by system stakeholders, including system  developers and users. Consistent with these practices, CNCS’s software  management processes identify steps for defining system functional and  technical requirements, including the analysis of users’ needs, reviews to  obtain stakeholders’ and management’s approval of requirements, and  the development and maintenance of documentation that describes the  requirements and any changes made throughout the development of the  system.
As currently planned, CNCS’s projects to develop and deliver the  modernized grant monitoring system align with the agency’s business and  management needs for its existing process, but are not yet defined for a  future risk-based process. When originally planning for the IT  modernization Phase 1 projects, CNCS included the development of an  initial version of the system to be used by program and grant officers to  support activities they conduct in accordance with the existing grant  monitoring process. Toward this end, initial requirements for the first  version of the grant monitoring system were defined to replicate  functionality provided by the legacy eGrants monitoring system and to  provide enhancements needed to help program and grant officers more  effectively conduct monitoring activities.
CNCS took steps to define requirements for the first version of the system  in accordance with its established processes. Specifically, OIT officials,  the development contractor, and system users held requirements  gathering sessions to determine users’ needs beginning in July 2015, as  planned. As a result of the sessions, program and grant officers who were  to use the system defined and documented 26 business requirements to  be addressed by functionality implemented in the first version.
Of these 26 requirements, 6 were defined to replicate the functionality  provided by the legacy eGrants system. For example, the new system  was to provide an input screen for users to document assessments of  grant awards, and another screen for users to enter data for planning  monitoring activities—functionality provided by the eGrants monitoring  system. The system was also to replicate other functionality of the  eGrants system, such as allowing users to record completed monitoring  activities and to add flags to organizations’ records that indicate potential  needs for additional monitoring.
The other 20 requirements that the system users defined were  enhancements to help improve the outcomes of monitoring activities in  support of CNCS’s goal to modernize grant management processes. For  example, the system was to, among other things, enable the officers to  create monitoring plans for awards and associate monitoring activities  with plans—functionality that was not provided by eGrants.
According to project planning documentation, the system users reached  agreement with OIT officials and the contractor to deliver system  functionality to meet all 26 requirements. However, as the contractor  proceeded with developing the first version of the system, CNCS changed  its plans for addressing the 26 requirements. Specifically, in a project  review held in October 2015, OIT officials, the development contractor,  and system users agreed to defer the 20 requirements for improving the  outcomes of monitoring activities and to include them in the requirements  definitions for the second version of the monitoring system.  Consequently, the requirements that were defined to replicate the  capabilities of the legacy eGrants system were addressed by functionality  planned for the first version.
OIT officials stated that the additional 20 requirements were deferred  because the agency had not defined business needs for a grant  monitoring system beyond those relevant to CNCS’s existing process.  Specifically, agency officials responsible for assuring that major IT  investments achieve business needs decided to minimize the investments  to only support existing business needs and operations until a new risk  management program is established and ongoing business process  reengineering efforts are completed. Thus, the plans for delivering the  first version of the system were modified to align with business needs of  CNCS’s existing grant monitoring process.
Project plans for the second version of the system call for enhancements  to the IT capabilities delivered by the first version. These planning  documents also indicate that, in addition to the requirements deferred  from the first version, the second version of the system is to provide any  additional functionality needed to align with business needs for monitoring  grants within a risk-based approach to enterprise-wide management.
However, in March 2017, we reported that CNCS had not evaluated its  monitoring efforts to identify opportunities to improve its assessment of,  and response to, risks related to grant management.  We recommended  that CNCS, as part of its efforts to establish enterprise-wide risk-based  management, develop a risk-based approach to grant monitoring and  establish a policy to ensure that all grants expected to be active in a fiscal  year are assessed for potential risk. In addition, according to CNCS’s  Chief Risk Officer and Chief Information Officer, as of April 2017, the  agency had not defined a risk-based management approach and,  therefore, had not defined business needs for monitoring grants based on  risks. Because business needs for a risk-based monitoring process have  not been determined, OIT officials and system stakeholders cannot begin  to define requirements for the second version of the system to align with  business and management needs for a future process, as intended by  CNCS’s IT modernization plans.
section_title: CNCS Has Made Limited Progress Toward Successfully Delivering a Modernized Grant Monitoring System, but Has Taken Steps to Help Avoid Continued Delays
paragraphs: Although the agency and its development contractor took steps to align  the first version of the system with changing business needs, CNCS has  made limited progress toward developing and delivering the modernized  system capabilities to support grant monitoring. CNCS began the  development effort for the first version of the system in July 2015 and  initially planned for the system to be delivered in April 2016. After  subsequent delays, officials and the contractor then planned to deliver the  system in September 2016. However, as of July 2017, the system still  had not been delivered to support the agency’s existing monitoring  process.
In CNCS’s and the contractor’s efforts to develop the system, incomplete  project schedules and testing practices introduced risks to the agency’s  ability to deliver the system. In addition, information documented by  agency officials in weekly and monthly reports on the status of the project  indicate that performance deficiencies on the contractor’s part contributed  to delays in the completion of testing, development, and system delivery.
subsections: section_title: Delivery of the First Version of the Grant Monitoring System Has Been Delayed
paragraphs: OIT officials’ and the development contractor’s initial project schedules  reflected plans to begin defining requirements for the first version of the  system in May 2015 and to finalize requirements in November 2015. The  development contractor was to begin testing to identify and correct any  problems with the system in February 2016. The system users were also  to begin participating in initial testing in February 2016 and complete their  final testing in March 2016, at which time any problems detected during  the earlier test phases were to be corrected and the system changes re- tested and approved. The system was to be in use by the end of April  2016.
As previously mentioned, OIT officials, the contractor, and system users  began defining requirements for the system in May 2015, and finalized  initial requirements in November 2015, as planned. The contractor also  developed a test version of the system and, along with system users,  completed the initial phases of testing in March 2016 according to plans.  However, technical and functional problems encountered during  subsequent phases of system testing caused delays in the completion of  later testing phases, in which users were to participate. Specifically, test  completion was delayed from April 2016 until October 2016—6 months  later than initially planned, and 1 month after the revised date (of  September 2016) that CNCS had said it intended to deliver the system.
According to CNCS’s CIO and project planning documentation, the  contractor’s testing of the first version of the grant monitoring system is  now expected to be completed and available for final users’ testing in  September 2017, and the system is expected to be delivered for  operations in October 2017. OIT officials reported that, as of January  2017, the agency had spent $12.7 million on conducting the Phase 1  GMM platform modernization and related system development projects— $11.2 million on labor and $1.5 million on software and services.
Further, as part of its modernization program, CNCS initially planned to  deliver the second version of the system in October 2017. However,  because of delays in the delivery of the first version and the agency’s lack  of progress in determining business needs for monitoring grants in a risk- based management environment, OIT officials, system users, and the  development contractor are not able to define system requirements or to  complete plans for developing and delivering the final system to support a  future monitoring process. Thus, a new date for delivering the second  version of the system has not been determined.
Figure 4 provides a timeline of planned and delayed delivery dates for the  first and second versions of the grant monitoring system.
section_title: Weaknesses in Software Development Practices Introduced Risks to Successful Delivery of the First Version of the Grant Monitoring System
paragraphs: In the development of the first version of the system, CNCS and the  development contractor did not follow all established software  development practices. Specifically, OIT officials and the contractor did  not develop project schedules that could be used to effectively track and  monitor the progress of the system development effort, and they did not  conduct complete testing of the grant monitoring system.
subsections: section_title: Practices for Developing and Managing Schedules Do Not Fully Support Effective Project Monitoring and Tracking of Activities
paragraphs: According to practices defined in GAO’s Schedule Assessment Guide, an  integrated master schedule should be the focal point of program  management. Such a schedule constitutes a program-level schedule  that includes the effort necessary from all agency and contractor  stakeholders to ensure successful execution from start to finish.  According to GAO’s guide, managers are to identify a baseline, or target,  schedule that is to be maintained and updated with actual dates when  activities are started and finished. The baseline is used to measure and  monitor the performance of projects and to determine when actual dates  differ from baseline dates and may affect future work. Further, it is useful  in determining when strategies are needed to allocate resources more  efficiently to meet time constraints. An agency’s program management  office is ultimately responsible for the development and management of  the schedule.
However, CNCS and its contractor did not develop schedules for the  Phase 1 projects of the GMM modernization plan that would have been  useful for managing and tracking the progress of development and testing  activities to ensure the first version of the grant monitoring system was  delivered in a timely manner. CNCS’s contractor developed, and OIT  officials approved, a schedule which they called an integrated master  schedule. However, this schedule was not constructed as an integrated  master schedule should be and, thus, did not provide a useful  management tool. Specifically, the schedule that CNCS approved  established baseline start and finish dates for activities, but did not reflect  actual dates for managing the GMM Phase 1 system development  projects, including the first version of the grant monitoring system. For  example, one version of the schedule identified a testing activity for the  grant monitoring system to take place from April 2016 through June 2016,  but the schedule did not include dates for when the activity was actually  started and completed. Another later version of the schedule included a  planned delivery date of October 2017 for the system, but did not include  the originally planned date of April 2016. Thus, the schedule could not be  used to track the status of project activities or to monitor and measure the  performance of the grant monitoring system development project to help  ensure its successful completion.
In discussing this matter, OIT officials said they did not develop and  maintain a baseline schedule that identified planned and actual start and  completion dates because they believed that they would be able to  effectively track the status of project-level activities using other tools and  techniques inherent in their approach to developing the system. For  example, the officials stated that they used a field included in the  schedules for entering notes to identify actions that needed to be taken to  complete activities and to track the status of each of the actions.  However, in many cases, OIT officials included certain program-level  actions and completion dates in the notes, but they did not record the  status of the actions. For example, they did not indicate whether or not  development and testing activities took place as planned. The officials  also noted that their current approach requires users interested in  referencing baseline changes to consult historic copies of the schedule or  go through the tedious process of comparing the notes field with project  files. As such, this approach could not be used as an effective tracking  tool to determine the status of the activities or the project as a whole.
Regardless of OIT officials’ approach to tracking the status of project  activities, until they construct a complete integrated master schedule that  includes both baseline and actual dates for developing the systems, the  agency will lack an effective management tool that could be used to help  avoid additional delays in delivering the first version of the modernized  grant monitoring system to support its existing process and retiring the  legacy eGrants system. Likewise, unless OIT officials and the  development contractor develop a schedule that can be used to monitor  and measure progress of the development and delivery of the second  version of the system against baseline and actual dates, the agency will  remain at risk that it will not be able to successfully execute all activities  needed to deliver system capabilities that could support a future risk- based monitoring process.
section_title: Testing Practices Introduced Risks to Successful Delivery of the System
paragraphs: According to SEI, testing the functionality of a new IT system is essential  to validate that the system will satisfy users’ requirements. Effective  testing facilitates early detection and correction of software and system  anomalies, provides an early assessment of software and system  performance, and provides information to key stakeholders for  determining the business risk of releasing the product in a current state.  SEI guidance advises that a test plan be used to ensure that testing is  carried out in a systematic manner. It describes the technical and  management approach to be followed for testing a system or a  component of a system, such as the projects within the CNCS IT  modernization program.
Consistent with SEI guidance, CNCS and the development contractor  made plans to test the first version of the system early on and throughout  the development phase. However, the program experienced halts and re- starts of testing activities, along with delays in system implementation, in  part because the agency and its contractor did not conduct all stages of  testing for the system.
OIT officials describe in the plans for testing all the GMM Phase 1  systems, including the grant monitoring system, a sequence of testing  events. For example, during the systems development process, the  developer was to conduct initial testing activities to determine whether  each of the systems was functioning independently as expected,  addressed users’ requirements, and met usability and security  requirements—referred to as unit testing. The contractor’s testing team  also was to conduct another stage of testing to ensure that all aspects of  the system affected by any changes made during initial unit testing  functioned as they did prior to the modifications and adjustments that  were made. This stage is referred to as regression testing.
According to the plans, once the development contractor had completed  unit and regression testing activities, the systems were to go through  three more stages of testing in a system integration test environment— the last two involving system stakeholders and users. These test stages  included:
Alpha testing—testing conducted by the system developer which was  to result in a working prototype that users and stakeholders could  evaluate to determine whether to keep developing the product as built  or to make modifications before completing additional development.
According to test plans, the version of the system delivered after  alpha testing was to include 50 percent of the desired functionality.
Beta testing—testing conducted by the agency and stakeholders to  validate a fully working prototype that the stakeholders could test  more thoroughly prior to delivery. CNCS OIT and stakeholders were  to conduct beta testing for about 8 weeks to validate that the  functionality needed to address business needs was implemented.  The beta testing phase was also intended to provide the CNCS user  community an opportunity to provide feedback on the user interface  before the system would be ready for the next phase of testing—user  acceptance testing. The system resulting from the beta testing phase  was to include more than 80 percent of the desired functionality  required for the system.
User acceptance testing—testing conducted by users to finally accept  the system, confirming that all functionality has been built and  requirements met. User acceptance testing was to be conducted for 3  weeks to validate that the system would provide full functionality, as  defined by system requirements, when deployed into a live, working  environment.
CNCS and its contractor completed and approved the results of the unit  and regression test stages for the Phase 1 GMM systems in September  2015, as planned. However, although the agency conducted alpha testing  according to plans, it did not include the grant monitoring system in that  phase of testing.
For example, according to OIT officials, the development contractor and  stakeholders tested 50 percent of the functionality of the GMM Phase I  systems, as planned; however, this approach did not guarantee that all  systems would be tested, and the grant monitoring system was not  included in the alpha testing activities.
Subsequently, OIT officials, the development contractor, and  stakeholders began a first round of beta testing of the system on April 18,  2016, and tested through April 22, 2016. Throughout the beta test period,  testers noticed significant problems related to usability and technical  deficiencies, such as awkward screen layouts and a lack of data that  were to be migrated from the legacy eGrants system—problems that  should have been detected and corrected during alpha testing. Based on  the extent of these problems, OIT officials determined that it was not  appropriate to proceed with the testing and, on May 9, 2016, called off  beta testing so that the development contractor could make the system  changes needed to correct the problems.
OIT officials resumed and completed the first round of beta testing (which  had begun in April 2016) in October 2016. They subsequently began and  completed a second round of testing in December 2016. Results of these  testing activities were approved by OIT management and system users,  who then planned to begin a final round of testing in April 2017. They also  planned to conduct user acceptance testing from July through September  2017.
OIT officials stated that the grant monitoring system was not included in  the alpha testing because system test plans required that 50 percent of  the overall GMM Phase 1 system functionality be tested during that  phase—but not 50 percent of the functionality to be provided by each  project. The officials stated that, in managing the development of the  overall GMM program, they tested more than 50 percent of other Phase I  systems’ functionality and, therefore, did not need to include the grant  monitoring system to meet the requirement. In addition, the officials  stated that, when alpha testing began for the other GMM Phase 1  systems in March 2016, they and system stakeholders still had not  agreed on final requirements and, therefore, were not ready to test the  system.
However, because they did not complete all phases of testing needed to  ensure early detection and correction of system problems throughout the  development of the first version of the grant monitoring system, the  agency was not prepared to conduct the beta and user acceptance  phases of testing. Had they started beta testing with a version of the  system that had passed alpha testing, CNCS could likely have detected  and corrected system errors earlier in the testing process and, thus,  avoided the problems experienced when conducting beta testing of the  grant monitoring system. Further, unless OIT officials and the  development contractor conduct alpha testing of grant monitoring  functionality when managing the development of the second version of  the system, they will likely experience similar challenges and risks to  plans for delivering a system to support future risk-based monitoring  processes.
section_title: CNCS Enhanced Oversight of Contractor Performance to Help Avoid Continued Delays
paragraphs: According to established practices defined by SEI, successful completion  of IT development projects calls for the delivery of quality products by any  contracting entities involved in the project and effective oversight on the  part of the contracting agency. As such, CNCS’s agreement with the IT  modernization development contractor specified that the contractor was  to provide system development, modernization, and enhancement  support including, but not limited to, program management and planning.  The contractor was required to support all phases of the agency’s system  development life cycle process, including requirements analysis,  development, testing, documentation, deployment, and post-deployment  support (e.g., training).
In addition, according to SEI, effective project oversight calls for close  monitoring of contractor performance, including regular status reporting  and progress reviews to management, and reviewing contractor  deliverables that document the level of progress toward meeting  milestones. SEI emphasizes the importance of project oversight to ensure  progress is being made according to plans for system development  projects, and calls for appropriate corrective actions to be taken when  project performance deviates significantly from plans. Effective project  oversight also includes notifying management as soon as the project  performance deviates from plans.
CNCS and the development contractor finalized initial requirements for  the monitoring system in November 2015, as planned. However, even  with agency officials’ initial efforts to oversee the contractor’s progress  toward delivering the system, the project experienced repeated delays  during the development and testing phases. For example, soon after  development of the system began, OIT officials required the contractor to  develop plans and schedules for resolving multiple technical and software  quality issues. Nevertheless, minutes from project management meetings  indicated that many of the issues remained unresolved, which called for  repeated corrective action plans. Specifically, from November 2015  through July 2016, OIT officials required the contractor to develop and  deliver seven corrective action plans to address deficiencies and defects  related to the development of the system.
As the agency took steps to monitor the status of the contractor’s actions  to address system deficiencies, the OIT staff responsible for overseeing  the contractor’s performance and other management officials held weekly  and monthly reviews of the contractor’s actions. In the reviews, the  officials assessed any progress made toward completing the actions and  planned contingencies for mitigating risks associated with any lack of  progress noted.
For example, minutes from a December 2015 IT management review  meeting documented the acceptance of the contractor’s first corrective  action plan. The minutes also described plans to hold weekly status  meetings to monitor progress and define actions that would be taken if  the contractor did not meet milestones for correcting system development  deficiencies. Later, following an April 2016 IT management meeting, OIT  officials again noted that the contractor was behind schedule in  developing data migration and training materials required by the fourth  corrective action plan. As another example, in May 2016, when the  contractor was addressing the fifth corrective action plan, OIT and other  agency management officials documented their intention to make a  decision on whether to issue a cure notice. Consequently, when the  contractor had not resolved all the problems identified in the first five  corrective action plans, CNCS issued a cure notice in June 2016, and  required the contractor to deliver two additional corrective action plans in  July 2016 and December 2016.
The first of the two additional corrective action plans—the sixth corrective  action plan—was to result in the delivery of the first version of the system  in October 2016. In a management review meeting held in August 2016,  agency officials noted that the development contractor was on track to  meet the requirements of the plan. Regardless, OIT officials reported that  they made a decision to delay delivery of the system because they did not  believe that there was enough time for users to adequately test and  receive training on the new system. The officials then required the  contractor to develop a seventh corrective action plan for delivering, by  December 2016, a version of the system that had passed beta testing to  allow the additional time needed for users to complete acceptance testing  and for the agency to conduct training. Figure 6 depicts the timeline of  CNCS’s corrective action plan requirements and the contractor’s delivery  of the plans.
According to CNCS’s CIO and documented results of status reviews held  with IT management officials, since the last corrective action plan was  delivered by the contractor in December 2016, performance has improved  and the project is on track to deliver the first version of the grant  monitoring system in October 2017. For example, the contractor delivered  a version of the system and beta testing began in December 2016, along  with plans to continue beta testing from April 2017 through September  2017. Weekly reports to management on the status of the project  indicated that, as of June 2017, all requirements defined for the first  version of the system had been implemented. The reports also indicated  that the development contractor was taking steps to prepare for the user  acceptance test phase, which was to be conducted from July through  September 2017. The reports note that the contractor was continuing to  resolve migration and integration issues identified in the April 2017 beta  test phase, prepare training materials for system users, and obtain  stakeholder input on deployment plans. In addition, OIT officials noted in  a June 2017 report to IT management officials that there were no  outstanding risks to the planned delivery of the system.
section_title: Conclusions
paragraphs: CNCS’s IT modernization plans included projects that were to result in the  delivery of system functionality to align with business and management  needs of the agency’s grant monitoring processes—its existing process  and a future process to be conducted as part of an enterprise-wide risk- based management approach. When developing the first version of the  modernized grant monitoring system, CNCS officials managed  requirements to align with changing business needs of the agency.  However, agency officials do not have the information needed to define  requirements and plan for the delivery of a second version of the system  to align with grant monitoring processes within a future risk-based  management approach. Specifically, as we reported in March 2017, the  agency has not defined risk factors to be considered when prioritizing  grants and grantees for monitoring activities.
CNCS initiated the first project and planned to develop and deliver the  first version of the system to support its existing grant monitoring process.  This version of the system was to be delivered in April 2016, but  weaknesses in schedule and testing management practices have  introduced risks to the successful delivery of a modernized system.  Unless CNCS officials make needed improvements in system  development practices, successful delivery of system functionality  necessary to support grant monitoring will remain at risk. Meanwhile,  CNCS’s grant monitoring officers continue to rely on the agency’s  outdated legacy system to support an inefficient and ineffective process  for monitoring the use of hundreds of millions of dollars in grants awarded  each year.
section_title: Recommendations for Executive Action
paragraphs: We are making the following three recommendations to CNCS:  The Chief Executive Officer should direct the Chief Information Officer to  take steps needed to ensure that system requirements are defined to  align with the business needs of CNCS’s future risk-based grants  monitoring process (Recommendation 1).
The Chief Executive Officer should direct the Chief Information Officer to  ensure that the system development project schedule identifies in the  baseline both planned and actual dates for completing all project-level  activities, and can be used to monitor and measure progress of the grant  monitoring system project (Recommendation 2).
The Chief Executive Officer should direct the Chief Information Officer to  ensure that test plans are defined and implemented to include the second  version of the grant monitoring system in all stages of testing during  development, and results of initial stages are approved before conducting  subsequent test stages (Recommendation 3).
section_title: Agency Comments and Our Evaluation
paragraphs: We received oral comments on a draft of this report during a discussion  with CNCS’s Chief Information Officer (CIO), Deputy CIO, and other  agency officials. In their comments, the officials partially agreed with three  of the five recommendations included in the draft report and disagreed  with two of the recommendations. The officials also shared additional  information regarding the agency’s actions to develop and test the  modernized grant monitoring system and subsequently provided  documentation to support their views. After reviewing the documentation,  we revised relevant sections of the report and modified the conclusions  and recommendations as appropriate. Further, as a result of incorporating  the additional information provided by the officials, our final report reflects  three, rather than the original five, recommendations.
Specifically, the officials partially agreed with our first recommendation  that CNCS ensure that system requirements are aligned with the  business needs of CNCS’s current and future risk-based grants  monitoring processes. The officials agreed that they needed to take steps  to ensure that requirements for the second version of the grant monitoring  system will align with business needs of a risk-based monitoring  approach and stated that they intend to continue to work with  stakeholders to define such requirements. However, the officials  disagreed with our initial conclusion that the first version of the system did  not meet business needs of the current grant monitoring process and,  consequently, disagreed with the part of the recommendation that they  ensure the system’s requirements aligned with business need of the  current process. We requested, and the officials provided in June 2017,  additional documentation reflecting alignment of the first version of the  system with current grant monitoring business needs. As a result of our  assessment of the supporting documentation, we revised the report to  incorporate the additional information and modified the initial conclusion  and the related recommendation.
The officials also partially agreed with the second recommendation in the  draft report regarding the need to develop a complete integrated master  schedule for managing the grant monitoring system development project.  Specifically, we recommended that the schedule include actual start and  finish dates and detailed system-level schedules that identify specific  activities, resource requirements, and interdependencies between  activities. The officials agreed with that part of the recommendation  stating a need for baseline schedules that document both planned and  actual start and finish dates for project activities. The officials told us that  they intend to include the dates in future planning efforts to allow for  easier tracking of progress and more effective project management.
However, the officials disagreed with the part of the recommendation that  stressed a need to develop schedules for the development and testing  phases of the individual Phase 1 systems, rather than the overall GMM  Phase 1 modernization program. Contrary to statements made by OIT  officials throughout our review, the officials stated that the agency had, in  fact, developed such schedules. Subsequently, in June 2017, the  agency provided us an integrated master schedule for the earlier phases  of the project through the requirements definition phase, and that  incorporated specific activities, interdependencies, and resources  required for the individual system-level projects, as we initially  recommended.
The officials added that they used other software development  management tools and metrics to track the status of activities conducted  during the system development and testing phases, and they provided  documented evidence of those activities. The officials noted that their  software development methodology and agreement with the contractor  allowed for resource requirements to be identified in other project  management artifacts and the contractor’s planning documents. After  assessing the additional documentation, we revised the report to  incorporate this information and modified the conclusion and  recommendation regarding project schedule management practices.
The agency officials also partially agreed with our third recommendation.  This recommendation called for the agency to use an integrated master  schedule for managing the system development project that included all  the elements identified in the second recommendation. Specifically, the  officials agreed with the part of the recommendation addressing the need  for CNCS to track the progress of the projects by comparing actual start  and finish dates to target dates. However, as noted for the second  recommendation, the officials disagreed with the need to use individual  schedules to monitor the status of each system. After considering the  additional evidence provided in the agency’s response to the second  recommendation, we modified the report accordingly. We also combined  the second and third recommendations contained in the draft report into  one recommendation, as reflected in this final report.
Further, CNCS’s CIO and Deputy CIO disagreed with a fourth  recommendation in the draft report which called for the agency to follow  plans for testing the grant monitoring system. The officials asserted that  the agency had followed plans when testing the first version of the  system, and provided additional information regarding system testing  requirements and processes. Based on our review of this additional  information, we acknowledged in the report that the development  contractor and system users had followed plans. Nevertheless, we  continue to emphasize the need for CNCS to, in future project  management activities, develop and implement test plans that ensure  each system included in the GMM project is adequately tested throughout  the early and all subsequent phases of testing. We modified our  recommendation to reflect our position on this matter.
Finally, the CIO and Deputy CIO disagreed with the fifth recommendation  in the draft report, which pertained to deficient contractor oversight  practices conducted by the agency. The officials stated that the actions  taken by the agency to enhance oversight, as described in the draft  report, had led to improvements in contractor performance and in the  agency’s overall ability to get the project on schedule for an October 2017  delivery. To substantiate their position, in June 2017 the officials provided  additional project management artifacts, such as weekly and monthly  management reports, that documented the recent status of project  activities. Specifically, the reports documented the contractor’s actions to  correct system defects, plan for system deployment, and develop training  materials. Based on our assessment of this documentation, we modified  the relevant discussion in the report to incorporate the newly obtained  information regarding the outcomes of CNCS’s actions. We also removed  the recommendation.
Beyond the aforementioned oral comments, CNCS’s Chief Operating  Officer also provided technical comments via email, which we  incorporated as appropriate.
We are sending copies of this report to the CNCS Chief Executive Officer,  the Director of the Office of Management and Budget, and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
Should you or your staff have questions on matters discussed in this  report, please contact me at (202) 512-9286. I can also be reached by e- mail at pownerd@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix II.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this study were to determine (1) to what extent the  Corporation for National and Community Service’s (CNCS) planned  information technology (IT) modernization projects align with business  and management needs for grant monitoring and (2) what progress  CNCS has made toward ensuring the successful and timely delivery of a  new system to support its grant monitoring process.
To address the first objective, we collected and reviewed documentation  that describes the CNCS IT modernization projects, including the IT  Modernization Program Management Plan and related system  development plans. We assessed whether the agency’s project plans  aligned with needs of grant monitoring officials by comparing their  contents against documentation that described the agency’s grant  monitoring process and requirements for providing IT support of the  current monitoring process. We identified enhancements needed to assist  program and grant officers, by examining planning documentation, such  as CNCS’s Strategic Plan, system requirements definitions, and  presentations and minutes from business stakeholder requirements  gathering sessions. We also reviewed documentation that noted changes  to requirements and interviewed agency officials, such as the Chief  Information Officer, to determine the rationale for the changes. We  compared the final requirements definition to the needs initially defined by  grant monitoring officials to determine the impact that changes had on the  agency’s plans to align its IT modernization projects with business needs  for grant monitoring.
To address the second objective, we first determined the progress CNCS  had made toward delivering a modernized system to support its existing  grant monitoring process. To do so, we collected and evaluated  documentation that provided evidence of ongoing system development  and implementation activities, including project schedules and status  tracking reports. We compared planned dates and milestones  documented in the schedules to actual dates obtained from project  artifacts, such as monthly program performance reports, to determine  whether milestones had been met and, when they had not, identified the  tasks that were not conducted or completed as planned and that caused  gaps between intended outcomes and functionality to be provided by the  system and delays in the system development and delivery schedule. We  also obtained and examined documentation describing the development  contractor’s activities, such as minutes and presentations from program  reviews, and weekly and monthly reports on the status of system  development efforts, to identify any deficiencies in system development  practices that may have affected CNCS’s and the development  contractor’s ability to deliver the system.
To assess CNCS’s efforts to develop and manage project schedules, we  compared schedules for developing, testing, and delivering the first  version of the grant monitoring system, from October 2015 to December  2016, to criteria defined in GAO’s Schedule Assessment Guide and held  discussions with GAO’s scheduling experts to determine appropriate  criteria for assessing scheduling practices with an Agile software  development methodology. We also examined data collected from  project planning and management artifacts, such as reports that  documented dates of regression and initial beta testing activities and  status reports that provided evidence of ongoing and completed system  development and implementation activities. We then compared the  planned dates and milestones contained in the schedules to actual dates  obtained from project artifacts, to determine whether the project  schedules reflected planned and actual dates for the completion of  activities, delays in meeting project milestones, and interdependencies  and conflicts between activities and resources.
To assess CNCS’s efforts to test the system, we compared  documentation that described agency practices for conducting system  testing, such as test plans and reports on testing outcomes, to testing  management criteria defined by the Software Engineering Institute (SEI)  and documented in its Capability Maturity Model® Integration (CMMI®)  framework. To determine whether CNCS had taken steps to ensure,  through its testing activities, the successful delivery of the grant  monitoring system, we assessed documentation of activities that were to  occur as part of the alpha, system integration, beta, and user acceptance  testing phases. Specifically, we examined test plans and reports of testing  outcomes and compared them to criteria for effective system testing  identified in SEI’s framework to determine whether CNCS conducted  testing as planned and the impact of any deficiencies on the agency’s  ability to deliver the system as planned.
Lastly, we determined the level of contractor oversight conducted by  CNCS officials by examining documentation such as program schedules,  documented requirements and deadlines for the contractor to address  system deficiencies identified by the Office of Information Technology  officials, corrective action plans delivered by the contractor, and  correspondence between the agency and the contractor. We compared  the activities to practices for conducting effective project oversight  established by SEI.
We supplemented the information we obtained from our assessment of  agency documentation by interviewing CNCS officials, including the Chief  Information Officer, Deputy Chief Information Officer, and Chief Risk  Officer, who were knowledgeable of efforts to develop systems and  oversee the work of contractors to support the grant monitoring process.
To determine the reliability of the data obtained from CNCS’s information  systems used for managing requirements and conducting system testing,  we compared information obtained from documentation of requirements  definitions and test plans against requirements tracking documents and  reports of test results. We also discussed with knowledgeable agency  officials within the Office of Information Technology the status of system  development processes, including actions taken to define and manage  system requirements and to conduct the various phases of testing. Based  on our efforts, we determined that the data we used from these sources  were sufficiently reliable for the purposes of our audit.
We conducted this performance audit from January 2016 through August  2017 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Teresa F. Tucker (Assistant  Director), Freda E. Paintsil (Analyst-In-Charge), Melina I. Asencio,  Alexander H. Bennett, Christopher G. Businsky, Nancy E. Glover,  Thomas E. Murphy, and Paige E. Teigen made key contributions to this  report.