Information Technology Modernization: Corporation for National and Community Service Needs to Develop a System That Supports Grant Monitoring
Released: 2017-09-18T00:00:00

Letter

Background

CNCS was created in 1993 with the enactment of the National and Community Service Trust Act of 1993. The agency is led by a Chief Executive Officer and Board of Directors who are appointed by the President, and with the advice and consent of the Senate. The Chief Executive Officer oversees the agency, which includes about 600 employees operating throughout the United States and its territories.

The agency provides grants to volunteer organizations throughout the United States to strengthen communities and foster civic engagement. For example, the National and Community Service Trust Act of 1993 established the CNCS and authorized it to make grants such as through the AmeriCorps state and national grant programs. The act also made the agency responsible for administering Volunteers in Service to America (VISTA) and National Senior Volunteer Corps. The Serve America Act was also enacted in 2009, which gave CNCS responsibility for administering several newly established programs, such as the Social Innovation Funds, which provide community-level grants.

CNCS grants are typically awarded to fund programs for a 3-year period, with funds distributed annually. In fiscal year 2016, CNCS received appropriations totaling more than $787.9 million and used $783.9 million to fund approximately 2,300 grants. Table 1 shows the amounts and number of grants disbursed by each of the programs in fiscal year 2016.

CNCS’s Grant Management Process

Within CNCS, the Chief of Program Operations, Chief Risk Officer, and Chief Financial Officer share responsibility for managing the grants. Program officers under the Chief of Program Operations and grant officers under the Chief Financial Officer manage the administration of the grants, including monitoring the use of grant funds by grantees. The program officers focus on issues related to grantee performance and compliance with program objectives, while the grant officers focus on grantees’ financial issues and performance.

The program and grant officers use IT systems to support monitoring and related activities. These systems are developed and maintained by the Office of Information Technology (OIT). This office is led by the agency’s CIO, who reports to the Chief Operating Officer. Figure 1 illustrates the position of the grant management entities within the agency as of April 2017.

To manage its grant program, CNCS conducts a four-phase process that covers pre-award grant reviews through actions taken to close out the grants when the period of related projects expires. According to CNCS policy, during the pre-award phase (before CNCS makes a grant award), grant officers evaluate the potential grantee’s financial management capabilities and other aspects, such as whether the grantee has any open audit findings on any current or prior grants, to determine if the grant should be awarded.

Once grants are awarded for the upcoming fiscal year, CNCS program and grant officers take steps to assess the grants and plan for monitoring the use of funds provided to the grantees. During the agency’s annual assessment phase, which is conducted between August and October of each year, the officers establish the upcoming year’s monitoring plan. During the monitoring phase, program and grant officers carry out the plans made during the assessment phase.

Lastly, when the period of the project supported by a grant expires, the grant is closed. During closeout and no later than 90 days after the expiration date of the grant, grantees submit required documentation to their grant officer and pay any outstanding obligations. CNCS’s program and grant officers verify completion of these actions and reconcile funds, then notify grantees when they have met all programmatic and fiscal requirements of the award. At that time, the grant is officially closed. An overview of the four phases of the monitoring process is depicted in figure 2.

Grant monitoring activities are initiated by program and grant officers during the assessment phase, when awarded grants are assessed, rated, and prioritized, and monitoring plans are made. The assessment phase is made up of four steps.

In the first step of the assessment phase, officers determine which grants are to be monitored in the upcoming year. The universe of grants being assessed typically includes all grants that are active at the time CNCS is ready to begin the assessment phase, usually in mid-August of each year, and that are expected to be active during the fiscal year beginning in October.

At the second step of the process, the officers further assess each of the grants to be monitored to identify potential vulnerabilities based on a set of criteria related to program compliance, financial weakness, or other issues defined by the program and grant officers. CNCS defined 19 criteria that indicate a grant’s potential vulnerabilities to be examined during an assessment. The officers may consider other key concerns or challenges in their assessment on a case-by-case basis.

Based on the outcomes of the second step, a third step is taken to determine the priority of each grant for monitoring purposes. Each grant is rated as high, medium, or low priority to determine the types of monitoring activities to be conducted during the year. The three priority categories are determined based on the presence of any of the pre-determined 19 criteria and an automated scoring process performed by the IT system that is used to support the agency’s grant monitoring process. The plan typically is to be prepared by the end of October.

Finally, during the fourth step of the annual assessment, program and grant officers plan the monitoring activities for each grant that are to be conducted during the year, based on the priority ratings determined during the third step. For example, grant monitoring plans may include on-site compliance visits to the grantees’ facilities, desk reviews of documentation submitted by the grantees, reviews of grantees’ financial activities, and follow-up visits based on the outcomes of compliance visits and other reviews.

The monitoring phase is conducted each year between October and August, when program and grant officers are to complete the activities described in their plans. During that time, they may add or omit activities, with supervisory approval, for a number of reasons. For example, program and grant officers may consider adding an activity for a grantee based on newly identified issues, such as findings of the OIG that suggest the grantee may have demonstrated poor financial management capabilities in the past. In such cases, the officers may add additional on- site or follow-up visits to their plans.

The officers produce reports on the progress made by grantees to accomplish the objectives and meet performance measures associated with a particular grant. For example, the reports could include data that reflect the number of individuals who received assistance in disaster preparedness, mitigation, response, and recovery efforts; the percentage of economically disadvantaged people who received housing-related support; or the percentage of students served by a grant program who showed improvements in academic performance. In addition, the officers review grantees’ financial status reports that identify funds expended compared to funds allocated for the grantees’ awards through the CNCS budget. The information reported is used by the program and grant officers to help them identify areas that may need more focused attention or targeted monitoring.

In our March 2017 report on CNCS’s efforts to improve grant monitoring, we noted that the agency’s annual assessment process may not result in the riskiest grants receiving a high priority for monitoring because of limitations in its scoring model. We also noted that CNCS has not evaluated its monitoring efforts to identify opportunities to improve its assessment of, and response to, risks related to grant management. We recommended that CNCS, as part of its efforts to establish enterprise- wide risk-based management, develop a risk-based approach to grant monitoring and establish a policy to ensure that all grants expected to be active in a fiscal year are assessed for potential risk. Such an approach would incorporate key practices, such as identifying risks relevant to the use of funds awarded by the grant program, analyzing and responding to the risks, and addressing risks as part of the grant monitoring process. The agency did not comment on this recommendation and has not yet addressed it.

Use of IT to Support CNCS’s Grant Management Process

CNCS’s legacy IT system, eGrants, has been used since 1998 to provide functionality needed to support the agency’s grant management process. Grant management includes annual assessment and monitoring activities such as planning, documenting related activities, generating correspondence, and reporting financial status and progress of grantees’ actions toward meeting performance goals. CNCS officials reported that, in fiscal year 2016, they spent about $11 million maintaining their legacy IT environment, which includes eGrants and other systems used to manage functions such as payroll and benefits.

To support the annual assessment process, program and grant officers record in eGrants the results of their evaluations of each grantee. Such evaluations are based on the 19 criteria that were defined to determine the priority of the grants to be monitored and the monitoring activities to be conducted. The officers indicate which of the criteria are present by choosing “yes” or “no” from a drop-down list. Then, based on the criteria, the system sorts and rates the priority of each grant as high, medium, or low. Figure 3 depicts the input screen of the eGrants Monitoring and Oversight Assessment Module, which is used by the officers to select criteria used to indicate the priority level of grants.

Once the assessment has been completed, the program and grant officers record activities that they plan to take during the year, such as conducting a follow-up visit with the grantee based on any audit findings in eGrants, along with the proposed time frames for conducting the activities. The program and grant officers also use eGrants to generate notification letters that are sent to grantees to schedule a site visit. After the visit has been conducted, the grant or program officers enter information regarding the results of the visit into the system, along with the grantee’s responses.

After a site visit, the officers may also identify follow-up actions to be taken and use eGrants to record those actions. The program and grant officers also use the system to generate reports on the progress of the grantees toward addressing any deficiencies identified as a result of monitoring activities they conducted. Grant officers also enter information about the results of financial reviews into the system.

However, CNCS has encountered challenges in using eGrants. For example, according to CNCS’s February 2013 IT Modernization Plan, data and information are not standardized and are entered into the grant management and monitoring systems with limited checks and validation, which results in data quality issues. In addition, according to the plan, technical limitations of eGrants make it difficult and time-consuming to integrate data from multiple sources and filter out inaccuracies. For example, agency officials stated that initiatives such as reporting on state- level grants require significant intervention by technical staff to obtain, integrate, and analyze data due to the disparate nature of the way the data are stored.

In October 2013, the CNCS OIG also reported problems with the agency’s legacy IT system used to support monitoring activities. The OIG reported that eGrants was inefficient and ineffective at supporting monitoring officers’ efforts to predict and detect improper use of funds awarded by CNCS grants. The OIG noted that, although the agency had collected data from its grantees to monitor their use of funds, its outdated systems did not provide the kind of data analytics capabilities needed to monitor early detection of fraud and mismanagement of grant funds. For example, the OIG reported that CNCS could not readily compare information across grants awarded or among the grantee population to identify anomalies and outliers that indicate potentially improper use of funds. The OIG further reported that the agency was falling behind in its efforts to increase the use of technology to support improved risk management practices, relying instead on inefficient processes for conducting oversight of the use of grant funds.

Recognizing the growing gap between its business needs and its IT infrastructure, CNCS engaged a contractor to provide guidance on modernizing its infrastructure. In May 2014, the contractor reported the results of its independent evaluation of the agency’s IT strategy, modernization plan, technology adoption, and costs for supporting CNCS’s strategic goals. The evaluation highlighted the limited system collaboration and communication capabilities among CNCS end users, data inconsistency, system availability issues, and data transparency and reporting issues as areas needing improvement. In its report, the contractor noted that CNCS had been slowly migrating its legacy grant management system to a new, more current technology, but that its efforts had not kept pace with mission-related programmatic demands. Consequently, program staff had developed workaround solutions to manage critical information accessed from multiple disparate sources, such as spreadsheets, e-mail, local and shared files, and local databases.

Our March 2017 report on CNCS’s grant monitoring process identified similar problems associated with eGrants. We reported that, according to CNCS officials, eGrants does not automatically produce standard historical reports, and, in order to produce a report from the system, significant manual manipulation of the data is required. We also discussed system users’ concerns regarding the usability of eGrants and its lack of integration with other tools used to collect monitoring data. For example, although each program officer has an electronic monitoring tool to use when conducting on-site monitoring, the data collected using these tools are not automatically integrated into and analyzed by eGrants. We further noted CNCS officials’ concerns that the agency’s existing IT environment did not provide automated mechanisms, such as trend analyses, that could help them identify patterns of potentially improper use of funds and risk factors to be considered when monitoring grants within a risk-based management approach.

CNCS’s IT Modernization Program

In February 2013, CNCS’ CIO released its IT modernization plan that describes a program intended to result in the development of systems to improve support of the agency’s grant management activities. Program documentation describes the implementation of systems that, when executed collectively, are to provide grantees and agency staff using the systems with new tools and capabilities to increase data quality and transparency, staff productivity, and program effectiveness. The IT modernization program has two overall goals.

Modernize grant management systems—Re-architect systems and standardize databases for greater agility; improve data quality and provide a better user experience through new data edits and validations, pre-populated forms, optimized navigation, and context- driven screens; maximize staff productivity by consolidating core staff functions currently in multiple systems into a common system.

Enhance mobile web and information sharing—Optimize key CNCS information for mobile web viewing and develop capabilities to accept grantee system information and share CNCS data with the public.

CNCS’s IT modernization program consists of three phases with projects in each phase that are intended to result in improvements to the overall IT computing platform, retirement of outdated legacy systems, and replacement of the retired systems. Specific projects are to be conducted as part of an overall initiative intended to result in the retirement of outdated legacy systems and the implementation of a modernized computing environment, referred to as the Grants and Member Management (GMM) platform.

Phase 1 of the program includes projects to result in the development and delivery of systems to support existing grant management processes, such as planning and applying for grants, reviewing applications, awarding funds, reporting on grantee and project status, monitoring the use of grant funds and grantee activity, and closing out grants and projects.

Phase 2 includes projects to develop systems to support services such as member recruitment, training, travel, awards, and enterprise risk management, including grant monitoring and oversight. This phase includes a project to deliver system capabilities that support the agency’s grant monitoring process as part of a future enterprise-wide risk management approach. The delivery of the systems to be developed during the first two phases is intended to allow the agency to retire outdated legacy systems.

Phase 3, the final phase, is to retire any remaining legacy systems to complete the transition to the new GMM platform.

CNCS defined IT modernization projects to be completed throughout the three phases of the initiative, such as the development of modernized systems to be used by the agency’s staff to manage grants, and the transition of all system functionality from the legacy environment to the platform. According to the modernization plan, the new systems are to be deployed via a technology platform that allows users to integrate data from external data sources in real time. Specifically, the platform is to be based on cloud computing that, according to CNCS’s IT strategic plan, is expected to lower IT costs, improve system access and stability, and enable IT staff to focus on supporting the needs of agency stakeholders who are to use the systems.

To implement the GMM cloud-based platform and develop the new systems, CNCS acquired the services of a development contractor through a task order that specifies software and services on a time and materials basis. The task order for the implementation of GMM was issued in September 2014, and the contractor began work on Phase 1, including the first modernized grant management system development project, in October 2014.

In January 2017, agency officials estimated the cost to complete all of the activities associated with the projects during the three phases of the IT modernization program to be $43.6 million—$39.3 million for labor and $4.3 million for software and services.

CNCS’s IT Modernization Projects Are Aligned with Business and Management Needs of the Existing Grant Monitoring Process, but Are Not Defined for a Future Risk- based Approach

To define system requirements that meet users’ business needs, key practices identified by SEI call for, among other things, ensuring that the business needs are understood and that supporting system requirements are defined and agreed upon by system stakeholders, including system developers and users. Consistent with these practices, CNCS’s software management processes identify steps for defining system functional and technical requirements, including the analysis of users’ needs, reviews to obtain stakeholders’ and management’s approval of requirements, and the development and maintenance of documentation that describes the requirements and any changes made throughout the development of the system.

As currently planned, CNCS’s projects to develop and deliver the modernized grant monitoring system align with the agency’s business and management needs for its existing process, but are not yet defined for a future risk-based process. When originally planning for the IT modernization Phase 1 projects, CNCS included the development of an initial version of the system to be used by program and grant officers to support activities they conduct in accordance with the existing grant monitoring process. Toward this end, initial requirements for the first version of the grant monitoring system were defined to replicate functionality provided by the legacy eGrants monitoring system and to provide enhancements needed to help program and grant officers more effectively conduct monitoring activities.

CNCS took steps to define requirements for the first version of the system in accordance with its established processes. Specifically, OIT officials, the development contractor, and system users held requirements gathering sessions to determine users’ needs beginning in July 2015, as planned. As a result of the sessions, program and grant officers who were to use the system defined and documented 26 business requirements to be addressed by functionality implemented in the first version.

Of these 26 requirements, 6 were defined to replicate the functionality provided by the legacy eGrants system. For example, the new system was to provide an input screen for users to document assessments of grant awards, and another screen for users to enter data for planning monitoring activities—functionality provided by the eGrants monitoring system. The system was also to replicate other functionality of the eGrants system, such as allowing users to record completed monitoring activities and to add flags to organizations’ records that indicate potential needs for additional monitoring.

The other 20 requirements that the system users defined were enhancements to help improve the outcomes of monitoring activities in support of CNCS’s goal to modernize grant management processes. For example, the system was to, among other things, enable the officers to create monitoring plans for awards and associate monitoring activities with plans—functionality that was not provided by eGrants.

According to project planning documentation, the system users reached agreement with OIT officials and the contractor to deliver system functionality to meet all 26 requirements. However, as the contractor proceeded with developing the first version of the system, CNCS changed its plans for addressing the 26 requirements. Specifically, in a project review held in October 2015, OIT officials, the development contractor, and system users agreed to defer the 20 requirements for improving the outcomes of monitoring activities and to include them in the requirements definitions for the second version of the monitoring system. Consequently, the requirements that were defined to replicate the capabilities of the legacy eGrants system were addressed by functionality planned for the first version.

OIT officials stated that the additional 20 requirements were deferred because the agency had not defined business needs for a grant monitoring system beyond those relevant to CNCS’s existing process. Specifically, agency officials responsible for assuring that major IT investments achieve business needs decided to minimize the investments to only support existing business needs and operations until a new risk management program is established and ongoing business process reengineering efforts are completed. Thus, the plans for delivering the first version of the system were modified to align with business needs of CNCS’s existing grant monitoring process.

Project plans for the second version of the system call for enhancements to the IT capabilities delivered by the first version. These planning documents also indicate that, in addition to the requirements deferred from the first version, the second version of the system is to provide any additional functionality needed to align with business needs for monitoring grants within a risk-based approach to enterprise-wide management.

However, in March 2017, we reported that CNCS had not evaluated its monitoring efforts to identify opportunities to improve its assessment of, and response to, risks related to grant management. We recommended that CNCS, as part of its efforts to establish enterprise-wide risk-based management, develop a risk-based approach to grant monitoring and establish a policy to ensure that all grants expected to be active in a fiscal year are assessed for potential risk. In addition, according to CNCS’s Chief Risk Officer and Chief Information Officer, as of April 2017, the agency had not defined a risk-based management approach and, therefore, had not defined business needs for monitoring grants based on risks. Because business needs for a risk-based monitoring process have not been determined, OIT officials and system stakeholders cannot begin to define requirements for the second version of the system to align with business and management needs for a future process, as intended by CNCS’s IT modernization plans.

CNCS Has Made Limited Progress Toward Successfully Delivering a Modernized Grant Monitoring System, but Has Taken Steps to Help Avoid Continued Delays

Although the agency and its development contractor took steps to align the first version of the system with changing business needs, CNCS has made limited progress toward developing and delivering the modernized system capabilities to support grant monitoring. CNCS began the development effort for the first version of the system in July 2015 and initially planned for the system to be delivered in April 2016. After subsequent delays, officials and the contractor then planned to deliver the system in September 2016. However, as of July 2017, the system still had not been delivered to support the agency’s existing monitoring process.

In CNCS’s and the contractor’s efforts to develop the system, incomplete project schedules and testing practices introduced risks to the agency’s ability to deliver the system. In addition, information documented by agency officials in weekly and monthly reports on the status of the project indicate that performance deficiencies on the contractor’s part contributed to delays in the completion of testing, development, and system delivery.

Delivery of the First Version of the Grant Monitoring System Has Been Delayed

OIT officials’ and the development contractor’s initial project schedules reflected plans to begin defining requirements for the first version of the system in May 2015 and to finalize requirements in November 2015. The development contractor was to begin testing to identify and correct any problems with the system in February 2016. The system users were also to begin participating in initial testing in February 2016 and complete their final testing in March 2016, at which time any problems detected during the earlier test phases were to be corrected and the system changes re- tested and approved. The system was to be in use by the end of April 2016.

As previously mentioned, OIT officials, the contractor, and system users began defining requirements for the system in May 2015, and finalized initial requirements in November 2015, as planned. The contractor also developed a test version of the system and, along with system users, completed the initial phases of testing in March 2016 according to plans. However, technical and functional problems encountered during subsequent phases of system testing caused delays in the completion of later testing phases, in which users were to participate. Specifically, test completion was delayed from April 2016 until October 2016—6 months later than initially planned, and 1 month after the revised date (of September 2016) that CNCS had said it intended to deliver the system.

According to CNCS’s CIO and project planning documentation, the contractor’s testing of the first version of the grant monitoring system is now expected to be completed and available for final users’ testing in September 2017, and the system is expected to be delivered for operations in October 2017. OIT officials reported that, as of January 2017, the agency had spent $12.7 million on conducting the Phase 1 GMM platform modernization and related system development projects— $11.2 million on labor and $1.5 million on software and services.

Further, as part of its modernization program, CNCS initially planned to deliver the second version of the system in October 2017. However, because of delays in the delivery of the first version and the agency’s lack of progress in determining business needs for monitoring grants in a risk- based management environment, OIT officials, system users, and the development contractor are not able to define system requirements or to complete plans for developing and delivering the final system to support a future monitoring process. Thus, a new date for delivering the second version of the system has not been determined.

Figure 4 provides a timeline of planned and delayed delivery dates for the first and second versions of the grant monitoring system.

Weaknesses in Software Development Practices Introduced Risks to Successful Delivery of the First Version of the Grant Monitoring System

In the development of the first version of the system, CNCS and the development contractor did not follow all established software development practices. Specifically, OIT officials and the contractor did not develop project schedules that could be used to effectively track and monitor the progress of the system development effort, and they did not conduct complete testing of the grant monitoring system.

Practices for Developing and Managing Schedules Do Not Fully Support Effective Project Monitoring and Tracking of Activities

According to practices defined in GAO’s Schedule Assessment Guide, an integrated master schedule should be the focal point of program management. Such a schedule constitutes a program-level schedule that includes the effort necessary from all agency and contractor stakeholders to ensure successful execution from start to finish. According to GAO’s guide, managers are to identify a baseline, or target, schedule that is to be maintained and updated with actual dates when activities are started and finished. The baseline is used to measure and monitor the performance of projects and to determine when actual dates differ from baseline dates and may affect future work. Further, it is useful in determining when strategies are needed to allocate resources more efficiently to meet time constraints. An agency’s program management office is ultimately responsible for the development and management of the schedule.

However, CNCS and its contractor did not develop schedules for the Phase 1 projects of the GMM modernization plan that would have been useful for managing and tracking the progress of development and testing activities to ensure the first version of the grant monitoring system was delivered in a timely manner. CNCS’s contractor developed, and OIT officials approved, a schedule which they called an integrated master schedule. However, this schedule was not constructed as an integrated master schedule should be and, thus, did not provide a useful management tool. Specifically, the schedule that CNCS approved established baseline start and finish dates for activities, but did not reflect actual dates for managing the GMM Phase 1 system development projects, including the first version of the grant monitoring system. For example, one version of the schedule identified a testing activity for the grant monitoring system to take place from April 2016 through June 2016, but the schedule did not include dates for when the activity was actually started and completed. Another later version of the schedule included a planned delivery date of October 2017 for the system, but did not include the originally planned date of April 2016. Thus, the schedule could not be used to track the status of project activities or to monitor and measure the performance of the grant monitoring system development project to help ensure its successful completion.

In discussing this matter, OIT officials said they did not develop and maintain a baseline schedule that identified planned and actual start and completion dates because they believed that they would be able to effectively track the status of project-level activities using other tools and techniques inherent in their approach to developing the system. For example, the officials stated that they used a field included in the schedules for entering notes to identify actions that needed to be taken to complete activities and to track the status of each of the actions. However, in many cases, OIT officials included certain program-level actions and completion dates in the notes, but they did not record the status of the actions. For example, they did not indicate whether or not development and testing activities took place as planned. The officials also noted that their current approach requires users interested in referencing baseline changes to consult historic copies of the schedule or go through the tedious process of comparing the notes field with project files. As such, this approach could not be used as an effective tracking tool to determine the status of the activities or the project as a whole.

Regardless of OIT officials’ approach to tracking the status of project activities, until they construct a complete integrated master schedule that includes both baseline and actual dates for developing the systems, the agency will lack an effective management tool that could be used to help avoid additional delays in delivering the first version of the modernized grant monitoring system to support its existing process and retiring the legacy eGrants system. Likewise, unless OIT officials and the development contractor develop a schedule that can be used to monitor and measure progress of the development and delivery of the second version of the system against baseline and actual dates, the agency will remain at risk that it will not be able to successfully execute all activities needed to deliver system capabilities that could support a future risk- based monitoring process.

Testing Practices Introduced Risks to Successful Delivery of the System

According to SEI, testing the functionality of a new IT system is essential to validate that the system will satisfy users’ requirements. Effective testing facilitates early detection and correction of software and system anomalies, provides an early assessment of software and system performance, and provides information to key stakeholders for determining the business risk of releasing the product in a current state. SEI guidance advises that a test plan be used to ensure that testing is carried out in a systematic manner. It describes the technical and management approach to be followed for testing a system or a component of a system, such as the projects within the CNCS IT modernization program.

Consistent with SEI guidance, CNCS and the development contractor made plans to test the first version of the system early on and throughout the development phase. However, the program experienced halts and re- starts of testing activities, along with delays in system implementation, in part because the agency and its contractor did not conduct all stages of testing for the system.

OIT officials describe in the plans for testing all the GMM Phase 1 systems, including the grant monitoring system, a sequence of testing events. For example, during the systems development process, the developer was to conduct initial testing activities to determine whether each of the systems was functioning independently as expected, addressed users’ requirements, and met usability and security requirements—referred to as unit testing. The contractor’s testing team also was to conduct another stage of testing to ensure that all aspects of the system affected by any changes made during initial unit testing functioned as they did prior to the modifications and adjustments that were made. This stage is referred to as regression testing.

According to the plans, once the development contractor had completed unit and regression testing activities, the systems were to go through three more stages of testing in a system integration test environment— the last two involving system stakeholders and users. These test stages included:

Alpha testing—testing conducted by the system developer which was to result in a working prototype that users and stakeholders could evaluate to determine whether to keep developing the product as built or to make modifications before completing additional development.

According to test plans, the version of the system delivered after alpha testing was to include 50 percent of the desired functionality.

Beta testing—testing conducted by the agency and stakeholders to validate a fully working prototype that the stakeholders could test more thoroughly prior to delivery. CNCS OIT and stakeholders were to conduct beta testing for about 8 weeks to validate that the functionality needed to address business needs was implemented. The beta testing phase was also intended to provide the CNCS user community an opportunity to provide feedback on the user interface before the system would be ready for the next phase of testing—user acceptance testing. The system resulting from the beta testing phase was to include more than 80 percent of the desired functionality required for the system.

User acceptance testing—testing conducted by users to finally accept the system, confirming that all functionality has been built and requirements met. User acceptance testing was to be conducted for 3 weeks to validate that the system would provide full functionality, as defined by system requirements, when deployed into a live, working environment.

CNCS and its contractor completed and approved the results of the unit and regression test stages for the Phase 1 GMM systems in September 2015, as planned. However, although the agency conducted alpha testing according to plans, it did not include the grant monitoring system in that phase of testing.

For example, according to OIT officials, the development contractor and stakeholders tested 50 percent of the functionality of the GMM Phase I systems, as planned; however, this approach did not guarantee that all systems would be tested, and the grant monitoring system was not included in the alpha testing activities.

Subsequently, OIT officials, the development contractor, and stakeholders began a first round of beta testing of the system on April 18, 2016, and tested through April 22, 2016. Throughout the beta test period, testers noticed significant problems related to usability and technical deficiencies, such as awkward screen layouts and a lack of data that were to be migrated from the legacy eGrants system—problems that should have been detected and corrected during alpha testing. Based on the extent of these problems, OIT officials determined that it was not appropriate to proceed with the testing and, on May 9, 2016, called off beta testing so that the development contractor could make the system changes needed to correct the problems.

OIT officials resumed and completed the first round of beta testing (which had begun in April 2016) in October 2016. They subsequently began and completed a second round of testing in December 2016. Results of these testing activities were approved by OIT management and system users, who then planned to begin a final round of testing in April 2017. They also planned to conduct user acceptance testing from July through September 2017.

OIT officials stated that the grant monitoring system was not included in the alpha testing because system test plans required that 50 percent of the overall GMM Phase 1 system functionality be tested during that phase—but not 50 percent of the functionality to be provided by each project. The officials stated that, in managing the development of the overall GMM program, they tested more than 50 percent of other Phase I systems’ functionality and, therefore, did not need to include the grant monitoring system to meet the requirement. In addition, the officials stated that, when alpha testing began for the other GMM Phase 1 systems in March 2016, they and system stakeholders still had not agreed on final requirements and, therefore, were not ready to test the system.

However, because they did not complete all phases of testing needed to ensure early detection and correction of system problems throughout the development of the first version of the grant monitoring system, the agency was not prepared to conduct the beta and user acceptance phases of testing. Had they started beta testing with a version of the system that had passed alpha testing, CNCS could likely have detected and corrected system errors earlier in the testing process and, thus, avoided the problems experienced when conducting beta testing of the grant monitoring system. Further, unless OIT officials and the development contractor conduct alpha testing of grant monitoring functionality when managing the development of the second version of the system, they will likely experience similar challenges and risks to plans for delivering a system to support future risk-based monitoring processes.

CNCS Enhanced Oversight of Contractor Performance to Help Avoid Continued Delays

According to established practices defined by SEI, successful completion of IT development projects calls for the delivery of quality products by any contracting entities involved in the project and effective oversight on the part of the contracting agency. As such, CNCS’s agreement with the IT modernization development contractor specified that the contractor was to provide system development, modernization, and enhancement support including, but not limited to, program management and planning. The contractor was required to support all phases of the agency’s system development life cycle process, including requirements analysis, development, testing, documentation, deployment, and post-deployment support (e.g., training).

In addition, according to SEI, effective project oversight calls for close monitoring of contractor performance, including regular status reporting and progress reviews to management, and reviewing contractor deliverables that document the level of progress toward meeting milestones. SEI emphasizes the importance of project oversight to ensure progress is being made according to plans for system development projects, and calls for appropriate corrective actions to be taken when project performance deviates significantly from plans. Effective project oversight also includes notifying management as soon as the project performance deviates from plans.

CNCS and the development contractor finalized initial requirements for the monitoring system in November 2015, as planned. However, even with agency officials’ initial efforts to oversee the contractor’s progress toward delivering the system, the project experienced repeated delays during the development and testing phases. For example, soon after development of the system began, OIT officials required the contractor to develop plans and schedules for resolving multiple technical and software quality issues. Nevertheless, minutes from project management meetings indicated that many of the issues remained unresolved, which called for repeated corrective action plans. Specifically, from November 2015 through July 2016, OIT officials required the contractor to develop and deliver seven corrective action plans to address deficiencies and defects related to the development of the system.

As the agency took steps to monitor the status of the contractor’s actions to address system deficiencies, the OIT staff responsible for overseeing the contractor’s performance and other management officials held weekly and monthly reviews of the contractor’s actions. In the reviews, the officials assessed any progress made toward completing the actions and planned contingencies for mitigating risks associated with any lack of progress noted.

For example, minutes from a December 2015 IT management review meeting documented the acceptance of the contractor’s first corrective action plan. The minutes also described plans to hold weekly status meetings to monitor progress and define actions that would be taken if the contractor did not meet milestones for correcting system development deficiencies. Later, following an April 2016 IT management meeting, OIT officials again noted that the contractor was behind schedule in developing data migration and training materials required by the fourth corrective action plan. As another example, in May 2016, when the contractor was addressing the fifth corrective action plan, OIT and other agency management officials documented their intention to make a decision on whether to issue a cure notice. Consequently, when the contractor had not resolved all the problems identified in the first five corrective action plans, CNCS issued a cure notice in June 2016, and required the contractor to deliver two additional corrective action plans in July 2016 and December 2016.

The first of the two additional corrective action plans—the sixth corrective action plan—was to result in the delivery of the first version of the system in October 2016. In a management review meeting held in August 2016, agency officials noted that the development contractor was on track to meet the requirements of the plan. Regardless, OIT officials reported that they made a decision to delay delivery of the system because they did not believe that there was enough time for users to adequately test and receive training on the new system. The officials then required the contractor to develop a seventh corrective action plan for delivering, by December 2016, a version of the system that had passed beta testing to allow the additional time needed for users to complete acceptance testing and for the agency to conduct training. Figure 6 depicts the timeline of CNCS’s corrective action plan requirements and the contractor’s delivery of the plans.

According to CNCS’s CIO and documented results of status reviews held with IT management officials, since the last corrective action plan was delivered by the contractor in December 2016, performance has improved and the project is on track to deliver the first version of the grant monitoring system in October 2017. For example, the contractor delivered a version of the system and beta testing began in December 2016, along with plans to continue beta testing from April 2017 through September 2017. Weekly reports to management on the status of the project indicated that, as of June 2017, all requirements defined for the first version of the system had been implemented. The reports also indicated that the development contractor was taking steps to prepare for the user acceptance test phase, which was to be conducted from July through September 2017. The reports note that the contractor was continuing to resolve migration and integration issues identified in the April 2017 beta test phase, prepare training materials for system users, and obtain stakeholder input on deployment plans. In addition, OIT officials noted in a June 2017 report to IT management officials that there were no outstanding risks to the planned delivery of the system.

Conclusions

CNCS’s IT modernization plans included projects that were to result in the delivery of system functionality to align with business and management needs of the agency’s grant monitoring processes—its existing process and a future process to be conducted as part of an enterprise-wide risk- based management approach. When developing the first version of the modernized grant monitoring system, CNCS officials managed requirements to align with changing business needs of the agency. However, agency officials do not have the information needed to define requirements and plan for the delivery of a second version of the system to align with grant monitoring processes within a future risk-based management approach. Specifically, as we reported in March 2017, the agency has not defined risk factors to be considered when prioritizing grants and grantees for monitoring activities.

CNCS initiated the first project and planned to develop and deliver the first version of the system to support its existing grant monitoring process. This version of the system was to be delivered in April 2016, but weaknesses in schedule and testing management practices have introduced risks to the successful delivery of a modernized system. Unless CNCS officials make needed improvements in system development practices, successful delivery of system functionality necessary to support grant monitoring will remain at risk. Meanwhile, CNCS’s grant monitoring officers continue to rely on the agency’s outdated legacy system to support an inefficient and ineffective process for monitoring the use of hundreds of millions of dollars in grants awarded each year.

Recommendations for Executive Action

We are making the following three recommendations to CNCS: The Chief Executive Officer should direct the Chief Information Officer to take steps needed to ensure that system requirements are defined to align with the business needs of CNCS’s future risk-based grants monitoring process (Recommendation 1).

The Chief Executive Officer should direct the Chief Information Officer to ensure that the system development project schedule identifies in the baseline both planned and actual dates for completing all project-level activities, and can be used to monitor and measure progress of the grant monitoring system project (Recommendation 2).

The Chief Executive Officer should direct the Chief Information Officer to ensure that test plans are defined and implemented to include the second version of the grant monitoring system in all stages of testing during development, and results of initial stages are approved before conducting subsequent test stages (Recommendation 3).

Agency Comments and Our Evaluation

We received oral comments on a draft of this report during a discussion with CNCS’s Chief Information Officer (CIO), Deputy CIO, and other agency officials. In their comments, the officials partially agreed with three of the five recommendations included in the draft report and disagreed with two of the recommendations. The officials also shared additional information regarding the agency’s actions to develop and test the modernized grant monitoring system and subsequently provided documentation to support their views. After reviewing the documentation, we revised relevant sections of the report and modified the conclusions and recommendations as appropriate. Further, as a result of incorporating the additional information provided by the officials, our final report reflects three, rather than the original five, recommendations.

Specifically, the officials partially agreed with our first recommendation that CNCS ensure that system requirements are aligned with the business needs of CNCS’s current and future risk-based grants monitoring processes. The officials agreed that they needed to take steps to ensure that requirements for the second version of the grant monitoring system will align with business needs of a risk-based monitoring approach and stated that they intend to continue to work with stakeholders to define such requirements. However, the officials disagreed with our initial conclusion that the first version of the system did not meet business needs of the current grant monitoring process and, consequently, disagreed with the part of the recommendation that they ensure the system’s requirements aligned with business need of the current process. We requested, and the officials provided in June 2017, additional documentation reflecting alignment of the first version of the system with current grant monitoring business needs. As a result of our assessment of the supporting documentation, we revised the report to incorporate the additional information and modified the initial conclusion and the related recommendation.

The officials also partially agreed with the second recommendation in the draft report regarding the need to develop a complete integrated master schedule for managing the grant monitoring system development project. Specifically, we recommended that the schedule include actual start and finish dates and detailed system-level schedules that identify specific activities, resource requirements, and interdependencies between activities. The officials agreed with that part of the recommendation stating a need for baseline schedules that document both planned and actual start and finish dates for project activities. The officials told us that they intend to include the dates in future planning efforts to allow for easier tracking of progress and more effective project management.

However, the officials disagreed with the part of the recommendation that stressed a need to develop schedules for the development and testing phases of the individual Phase 1 systems, rather than the overall GMM Phase 1 modernization program. Contrary to statements made by OIT officials throughout our review, the officials stated that the agency had, in fact, developed such schedules. Subsequently, in June 2017, the agency provided us an integrated master schedule for the earlier phases of the project through the requirements definition phase, and that incorporated specific activities, interdependencies, and resources required for the individual system-level projects, as we initially recommended.

The officials added that they used other software development management tools and metrics to track the status of activities conducted during the system development and testing phases, and they provided documented evidence of those activities. The officials noted that their software development methodology and agreement with the contractor allowed for resource requirements to be identified in other project management artifacts and the contractor’s planning documents. After assessing the additional documentation, we revised the report to incorporate this information and modified the conclusion and recommendation regarding project schedule management practices.

The agency officials also partially agreed with our third recommendation. This recommendation called for the agency to use an integrated master schedule for managing the system development project that included all the elements identified in the second recommendation. Specifically, the officials agreed with the part of the recommendation addressing the need for CNCS to track the progress of the projects by comparing actual start and finish dates to target dates. However, as noted for the second recommendation, the officials disagreed with the need to use individual schedules to monitor the status of each system. After considering the additional evidence provided in the agency’s response to the second recommendation, we modified the report accordingly. We also combined the second and third recommendations contained in the draft report into one recommendation, as reflected in this final report.

Further, CNCS’s CIO and Deputy CIO disagreed with a fourth recommendation in the draft report which called for the agency to follow plans for testing the grant monitoring system. The officials asserted that the agency had followed plans when testing the first version of the system, and provided additional information regarding system testing requirements and processes. Based on our review of this additional information, we acknowledged in the report that the development contractor and system users had followed plans. Nevertheless, we continue to emphasize the need for CNCS to, in future project management activities, develop and implement test plans that ensure each system included in the GMM project is adequately tested throughout the early and all subsequent phases of testing. We modified our recommendation to reflect our position on this matter.

Finally, the CIO and Deputy CIO disagreed with the fifth recommendation in the draft report, which pertained to deficient contractor oversight practices conducted by the agency. The officials stated that the actions taken by the agency to enhance oversight, as described in the draft report, had led to improvements in contractor performance and in the agency’s overall ability to get the project on schedule for an October 2017 delivery. To substantiate their position, in June 2017 the officials provided additional project management artifacts, such as weekly and monthly management reports, that documented the recent status of project activities. Specifically, the reports documented the contractor’s actions to correct system defects, plan for system deployment, and develop training materials. Based on our assessment of this documentation, we modified the relevant discussion in the report to incorporate the newly obtained information regarding the outcomes of CNCS’s actions. We also removed the recommendation.

Beyond the aforementioned oral comments, CNCS’s Chief Operating Officer also provided technical comments via email, which we incorporated as appropriate.

We are sending copies of this report to the CNCS Chief Executive Officer, the Director of the Office of Management and Budget, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staff have questions on matters discussed in this report, please contact me at (202) 512-9286. I can also be reached by e- mail at pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Objectives, Scope, and Methodology

The objectives of this study were to determine (1) to what extent the Corporation for National and Community Service’s (CNCS) planned information technology (IT) modernization projects align with business and management needs for grant monitoring and (2) what progress CNCS has made toward ensuring the successful and timely delivery of a new system to support its grant monitoring process.

To address the first objective, we collected and reviewed documentation that describes the CNCS IT modernization projects, including the IT Modernization Program Management Plan and related system development plans. We assessed whether the agency’s project plans aligned with needs of grant monitoring officials by comparing their contents against documentation that described the agency’s grant monitoring process and requirements for providing IT support of the current monitoring process. We identified enhancements needed to assist program and grant officers, by examining planning documentation, such as CNCS’s Strategic Plan, system requirements definitions, and presentations and minutes from business stakeholder requirements gathering sessions. We also reviewed documentation that noted changes to requirements and interviewed agency officials, such as the Chief Information Officer, to determine the rationale for the changes. We compared the final requirements definition to the needs initially defined by grant monitoring officials to determine the impact that changes had on the agency’s plans to align its IT modernization projects with business needs for grant monitoring.

To address the second objective, we first determined the progress CNCS had made toward delivering a modernized system to support its existing grant monitoring process. To do so, we collected and evaluated documentation that provided evidence of ongoing system development and implementation activities, including project schedules and status tracking reports. We compared planned dates and milestones documented in the schedules to actual dates obtained from project artifacts, such as monthly program performance reports, to determine whether milestones had been met and, when they had not, identified the tasks that were not conducted or completed as planned and that caused gaps between intended outcomes and functionality to be provided by the system and delays in the system development and delivery schedule. We also obtained and examined documentation describing the development contractor’s activities, such as minutes and presentations from program reviews, and weekly and monthly reports on the status of system development efforts, to identify any deficiencies in system development practices that may have affected CNCS’s and the development contractor’s ability to deliver the system.

To assess CNCS’s efforts to develop and manage project schedules, we compared schedules for developing, testing, and delivering the first version of the grant monitoring system, from October 2015 to December 2016, to criteria defined in GAO’s Schedule Assessment Guide and held discussions with GAO’s scheduling experts to determine appropriate criteria for assessing scheduling practices with an Agile software development methodology. We also examined data collected from project planning and management artifacts, such as reports that documented dates of regression and initial beta testing activities and status reports that provided evidence of ongoing and completed system development and implementation activities. We then compared the planned dates and milestones contained in the schedules to actual dates obtained from project artifacts, to determine whether the project schedules reflected planned and actual dates for the completion of activities, delays in meeting project milestones, and interdependencies and conflicts between activities and resources.

To assess CNCS’s efforts to test the system, we compared documentation that described agency practices for conducting system testing, such as test plans and reports on testing outcomes, to testing management criteria defined by the Software Engineering Institute (SEI) and documented in its Capability Maturity Model® Integration (CMMI®) framework. To determine whether CNCS had taken steps to ensure, through its testing activities, the successful delivery of the grant monitoring system, we assessed documentation of activities that were to occur as part of the alpha, system integration, beta, and user acceptance testing phases. Specifically, we examined test plans and reports of testing outcomes and compared them to criteria for effective system testing identified in SEI’s framework to determine whether CNCS conducted testing as planned and the impact of any deficiencies on the agency’s ability to deliver the system as planned.

Lastly, we determined the level of contractor oversight conducted by CNCS officials by examining documentation such as program schedules, documented requirements and deadlines for the contractor to address system deficiencies identified by the Office of Information Technology officials, corrective action plans delivered by the contractor, and correspondence between the agency and the contractor. We compared the activities to practices for conducting effective project oversight established by SEI.

We supplemented the information we obtained from our assessment of agency documentation by interviewing CNCS officials, including the Chief Information Officer, Deputy Chief Information Officer, and Chief Risk Officer, who were knowledgeable of efforts to develop systems and oversee the work of contractors to support the grant monitoring process.

To determine the reliability of the data obtained from CNCS’s information systems used for managing requirements and conducting system testing, we compared information obtained from documentation of requirements definitions and test plans against requirements tracking documents and reports of test results. We also discussed with knowledgeable agency officials within the Office of Information Technology the status of system development processes, including actions taken to define and manage system requirements and to conduct the various phases of testing. Based on our efforts, we determined that the data we used from these sources were sufficiently reliable for the purposes of our audit.

We conducted this performance audit from January 2016 through August 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Teresa F. Tucker (Assistant Director), Freda E. Paintsil (Analyst-In-Charge), Melina I. Asencio, Alexander H. Bennett, Christopher G. Businsky, Nancy E. Glover, Thomas E. Murphy, and Paige E. Teigen made key contributions to this report.