USDA is an equal opportunity provider, employer, and lender
PREFACE

Abt Associates CN ART Grants | Final Study Report Final ▌i 

About This Report

This study provides FNS with formative research on the Child Nutrition (CN) Administrative Review and 

Training (ART) grants. The data collected during the study aim to help FNS understand any effects of the 

ART grantees’ interventions on administrative processes, examine the potential for long-term 

sustainability of grant-funded activities, and describe challenges to ART grant implementation and 

sustainability.

Authors

René Nutter

Meaghan Glenn

Robin Koralek

Correne Saunders

Kaylyn Williams

Abt Associates | 6130 Executive Boulevard | Rockville, MD 20852
CONTENTS

Abt Associates CN ART Grants | Final Study Report Final ▌ii

CONTENTS

Introduction............................................................................................................................... 1

Study Purpose and Background ...................................................................................... 1

Study Purpose ........................................................................................................ 1

Background of Administrative Review and Training Grants..................................... 1

Study Objectives ............................................................................................................. 2

Study Methods ................................................................................................................ 2

ART Grant Projects Overview .................................................................................................. 3

Grantees and Award Amount .......................................................................................... 3

Length of Grant Period .................................................................................................... 3

Grantee Goals................................................................................................................. 5

Intervention Types........................................................................................................... 5

Organizational Structure.................................................................................................. 6

Planning and Implementation of ART Grant Projects ............................................................ 8

Successes ....................................................................................................................... 8

Improvements to the Administrative Review Process.............................................. 8

Improved Quality of Administrative Reviews ........................................................... 8

Expanded Access to Training and Technical Assistance ........................................ 9

Challenges ...................................................................................................................... 9

Staffing ................................................................................................................... 9

Project Management............................................................................................... 9

Rollout of New AR Requirements..........................................................................10

Technology ...........................................................................................................11

Timing of Challenges ............................................................................................12

Overcoming Challenges........................................................................................12

Lessons Learned...........................................................................................................12

Findings Regarding the Effects of ART Grant Projects........................................................14

Changes to AR Process (Including Data Reporting/Collecting).............................14

Administrative Error Rates ....................................................................................14

Administrative Costs .............................................................................................14

Direct Certification ................................................................................................14

Other School Nutrition Programs ..........................................................................14

Staff Burden..........................................................................................................14

Post-Grant Activities/Efforts ..................................................................................................15

Sustainability .................................................................................................................15

Expansion and Modification, Including Updates.............................................................16

Considerations for FNS..........................................................................................................17

Appendix A. Grantee Profiles ...............................................................................................18

Appendix B. Research Questions ........................................................................................88

Appendix C. Extant Data Review Protocol...........................................................................90

Appendix D. Interview Protocols..........................................................................................95

Appendix E. Data Summary Table .....................................................................................112
CONTENTS

Abt Associates CN ART Grants | Final Study Report Final ▌iii

List of Exhibits

Exhibit 1: Changes to LEA Reviews under the New Administrative Review Process.................. 1

Exhibit 2: Study Process ............................................................................................................ 2

Exhibit 3: Percentage Returned, by Grantee .............................................................................. 3

Exhibit 4: Grant Period, by Grantee............................................................................................ 4

Exhibit 5: Expressed Goals, by Grantee..................................................................................... 5

Exhibit 6: Intervention Type, by Grantee .................................................................................... 6

Exhibit 7: Distribution of Software Vendors Used by Multiple Grantees, by Vendor* .................. 7

Exhibit 8: Potential Post-ART Grant Funding ............................................................................15
INTRODUCTION

Abt Associates CN ART Grants | Final Study Report Final ▌1 

Introduction

Study Purpose and Background

Study Purpose

This study provides FNS with formative research on the Child Nutrition (CN) Administrative Review and 

Training (ART) grants. The data collected during the study aim to help FNS understand any effects of the 

ART grantees’ interventions on administrative processes, examine the potential long-term sustainability 

of grant-funded activities, and describe challenges to ART grant implementation and sustainability.

Background of Administrative Review and Training Grants

Federal legislation in 2004 and 2010 amended the Richard B. Russell National School Lunch Act (P.L. 

79-396, 60 Stat. 230) to require State agencies (SAs) to conduct additional reviews of selected Local 

Education Agencies (LEAs) and provide annual funding to SAs to use for oversight and training of LEA 

staff, with a focus on LEAs that demonstrate high levels of or a high risk for administrative errors, as 

identified by SAs. The amended legislation also required SAs to implement a more robust and unified 

accountability system.

Following changes from the 2010 legislation via the Healthy, Hunger-Free Kids Act (HHFKA) (P.L. 111-

296), the U.S. Department of Agriculture (USDA) established the Administrative Review (AR), a new 

process for LEA reviews. Changes implemented under this new process are illustrated in Exhibit 1. SAs 

must review all their LEAs’ National School Lunch Program (NSLP) at least once during the three-year 

review cycle.

Exhibit 1: Changes to LEA Reviews under the New Administrative Review Process

To assist SAs in meeting these new AR requirements and per the HHFKA legislation, FNS funded the 

ART grants, which allowed SAs to fund administrative oversight and AR training for LEAs. The ART 

grants is a competitive grant available to all State agencies administering the school meal programs. FNS 

awarded ART grants to 30 States between 2009 and the end of fiscal year in 2017. 

The focus of the ART grants is threefold: (1) oversight and training of LEA administrative personnel in 

school meal operations; (2) modifications to update processes and systems to comply with AR; and 

(3) implementing technology improvements to address administrative errors in error-prone LEAs. 

The ART Grant Summary of Best Practices provided FNS with an initial glimpse of the promising 

practices and lessons learned from ART grantees that implemented technology-related projects and 

completed or closed out their grants by May 2015. FNS’s interest in bolstering initial findings from the 

ART Grant Summary of Best Practices, which was created from FNS supported technical assistance 

provided to ART grantees, resulted in the commission of this study.
INTRODUCTION

Abt Associates CN ART Grants | Final Study Report Final ▌2 

Study Objectives

The formative research ART study focused on a comprehensive set of research questions, grouped within 

three study objectives: 

1. describe how interventions supported by ART grants intended to improve grantee administrative 

processes;

2. examine how ART grantees sustain their activities after their FNS funding has ended; 

3. describe challenges to implementation and sustainability of ART grant interventions and how 

those challenges may be overcome.

Study Methods

Of 30 ART grants awarded, 20 States completed/closed out their grants in time for inclusion in this study

(by the end of fiscal year 2017). ART grants included both planning and implementation grants, this study 

only included implementation grants. 

The study used a qualitative approach to address the three objectives. The approach included a review of 

existing documents, such as grantee monthly progress reports and final reports, telephone interviews with 

SA and LEA staff from the 20 targeted ART grantees, and additional follow-up telephone interviews with 

10 State AR staff to gather more detail about their AR processes. Exhibit 2 provides an overview of the 

study approach. 

Exhibit 2: Study Process 

See Appendix A for a profile of each grantee and its project. See Appendix B for the research questions 

the study addressed. See Appendix C for the protocol used to review extant data and Appendix D for the 

protocols used in the interviews. 
PROJECTS OVERVIEW

Abt Associates CN ART Grants | Final Study Report Final ▌3 

ART Grant Projects Overview

Grantees and Award Amount

The study examined 20 States grantees with 22 ART grant projects. Grantees included U.S. States and 

Guam. The grantees were clustered in the Midwest, Mountain Plains and Western FNS regions; however, 

grants were also awarded to some grantees in the 

Northeast and South. More than $23 million was 

awarded between 2009 and 2016, with grant 

amounts ranging from $251,655 to $1,960,526. In 

total, $2,127,274.11 (9.1%) of all money awarded 

was returned (see Exhibit 3). The amount of 

money returned ranged from 0% to up to 53% 

returned. The reasons for returning funds varied, 

but many grantees attributed it to changes in their 

implementation plans. For example New 

Hampshire originally proposed to contract an 

outside vendor to develop a new automated 

system. However, once awarded, the State 

decided to save costs it would develop the system 

in-house. 

Length of Grant Period

Initial grant periods ranged from one to three 

years as per the ART grant application. However, 

actual grant periods ranged from two to six years, 

depending on the grantee and scope of its project. 

Six grantees completed their projects during their 

intended grant period, but 15 required no-cost 

extensions. The majority of grantees received at 

least one no-cost extension; however, in some 

cases, up to four such extensions were awarded. 

Among the issues to which grantees attributed 

their need for a no-cost extension were delays in 

implementation, changes in scope of work, 

changes in partners, project management 

challenges, and staff turnover. Also cited was 

USDA’s implementation of the AR process midgrant, which affected grantees’ implementation 

plans. 

Exhibit 3: Percentage Returned
PROJECTS OVERVIEW

Abt Associates CN ART Grants | Final Study Report Final ▌4 

Exhibit 4: Grant Period, by Grantee
PROJECTS OVERVIEW

Abt Associates CN ART Grants | Final Study Report Final ▌5 

Grantee Goals

The goals of the funded projects can be summarized in 

three overarching categories: (1) reducing errors; 

(2) improving compliance with the new AR process; and 

(3) targeting error-prone LEAs. Several grantees addressed 

all three goals in their intervention; other grantees such as 

Alabama took a more targeted approach, identifying just 

one goal for their project. Exhibit 5 demonstrates the 

expressed goals, by grantee.

Intervention Types

The strategies ART grantees selected for reducing 

administrative errors varied widely within the 

requirements set by FNS in the Request for Applications. 

For example, some grantees implemented direct ART 

grant–funded services and trainings for LEAs, and these 

trainings varied in duration, scope, and intensity. Other

grantees implemented updates to technologies that affected

LEAs statewide. In some States where service-oriented or 

training interventions were implemented, specific LEAs or 

types of LEAs were sometimes targeted, whereas in States 

where technology-based interventions were implemented, 

all LEAs within the State could be affected. 

Technology. The technology interventions included 

creating new technology systems for conducting ARs, 

working with a technology solutions provider to customize 

an off-the-shelf model, and updating and integrating the 

new AR process into an existing child nutrition platform. 

Through the technological updates grantees were able to 

automate the ARs, communicate more seamlessly with 

LEAs, and provide real-time monitoring of the AR

process. 

Training. Given the changes to the AR process, many 

grantees used funds to provide training to LEAs. These 

trainings included, but were not limited to, producing 

online modules to assist in corrective action, training on 

using the new systems for the automated reviews, and 

training on compliance with meal counting or nutritional 

quality of school meals. 

Exhibit 6 demonstrates the breadth of the interventions, 

highlighting the diversity in approaches grantees took to 

achieving their stated goals.

Exhibit 5: Expressed Goals, by Grantee
PROJECTS OVERVIEW

Abt Associates CN ART Grants | Final Study Report Final ▌6 

Exhibit 6: Intervention Type, by Grantee

Organizational Structure

Grantees had full discretion over their organizational structures. Some grantees opted to manage their 

project internally with State staff, whereas others opted to contract out their project management. Many 

interviewees emphasized the role of the project manager as being critical to the success of the project. 

Some grantees had a single project manager who managed the budget, timeline, and progress; other 

grantees, such as Kansas, opted to have multiple program managers: one who focused on IT and another 

Child Nutrition project manager who oversaw tasks outside of IT-specific activities. 

The grantees also partnered with a wide variety of organizations, see Exhibit 7. These included local 

institutions, such as universities or consultants, to develop systems or curricula for trainings. Some 

grantees also worked with other agencies within their State. For example, New Hampshire, Washington 

and Michigan opted to have their in-house State IT departments develop their technology solutions. 

As a provision to receiving the ART grant, States with a technology-related grant project received 

technical assistance. FNS contracted with an organization to provide grantees with this technical 

assistance. The contractor assisted grantees on project management tasks, including providing grantees 

with templates, examples, and hosting webinars on basic project management principals. The intent of the 

technical assistance was to aid the States’ management of their grant project to stay within scope, 

schedule and budget.

Of particular note is the role of software vendors. Within the 20 grantees, 14 used software vendors to 

implement their interventions. The role of the vendors varied depending on the grantee. Three vendors 

were used by multiple grantees. Five contracted with the same company to customize an off-the-shelf AR 

module for each State. Five additional grantees also had some overlap in vendors used (three used one, 

two used another). Other outside vendors were used but were not found to be duplicative among the State 

agencies involved in this study. 
PROJECTS OVERVIEW

Abt Associates CN ART Grants | Final Study Report Final ▌7 

Exhibit 7: Vendor Types, by Grantee

 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌8 

Planning and Implementation of ART Grant Projects

The SAs, LEAs, and Administrative Reviewers identified successes and challenges in planning and 

implementing their ART grants. Perceived successes included improvements to the review process and to 

the quality of reviews. To achieve these successes, grantees overcame challenges to managing and 

implementing their grants within budget, timing and staffing constraints, and challenges related to 

designing and deploying new technology systems. Grantees also shared whether they plan to apply for 

another grant in the future, and they reported some lessons learned for future efforts.

Successes

Improvements to the Administrative Review Process

Most of the grantees reported that the efforts funded 

through their ART grant led to improvements in their AR

process. Specifically, many of the grantees described their

post-grant AR process as more streamlined, more concise,

simpler, or faster than their processes in place prior to 

implementing the new AR systems and receiving their

ART grants. One grantee estimated a time savings of 85%

when producing summaries of review results previously. 

Besides saving time, many grantees also described their 

new AR system as more accurate, cleaned up, organized, 

and of much better quality. In particular, grantees reported

the improvements to the AR process as giving them more 

flexibility, especially when it came to the ability to add dat

throughout the year and update data “on the fly” when 

corrections were needed. That is, having a streamlined 

process and more accurate data allowed grantees to monito

and track across multiple programs and to pull data that 

could identify a multitude of issues. For example, one grantee described implementing a new report that 

identifies anomalies that could indicate a data entry issue on the LEA level that needs correction, outside 

of the formal review process.

a 

r 

Improved Quality of Administrative Reviews

Besides increasing flexibility, improvements to the AR

process also yielded what grantees perceived as successes in 

improving the quality of the reviews. Grantees identified 

improvements in communication between the SA and LEAs

as a sometimes unexpected benefit of organizing and 

streamlining policies, procedures, and data collection related 

to the ARs. For example, some grantees reported the new 

system made it easier for reviewers and LEAs to understand 

what was expected of them and to document progress 

throughout the review. One grantee remarked on one 

particular aspect of its new system—visual checklists—as 

having improved communication in this way. Another 

reported that ART-funded training helped to improve 

communication between the SA and LEAs because LEAs 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌9 

felt more supported, whereas the previous process, in its focus on the audit sections, sometimes led to 

adversarial relationships.

Expanded Access to Training and Technical Assistance

Grantees also reported successes in expanding access to 

training. Some grantees reported that the time savings and 

more accurate and timely data let their reviewers devote more 

of their time to providing in-depth technical assistance and 

training on the particular issues identified in the course of a 

review. Additionally, grantees perceived that online training 

modules led to more knowledgeable Child Nutrition staff. 

Specifically, grantees cited the ability of staff to access 

training 24/7 and from remote locations as a major contributor 

to their increased professional development and mastery of 

policies and procedures.

Challenges

Grantees identified a range of challenges1 that arose during the planning and implementation of their ART 

grants. Broadly, the four most commonly cited challenges had to do with staffing, project management, 

rollout of new AR requirements, and technology.

Staffing

More than half of grantees pointed to challenges in meeting project goals and timelines due to lack of 

staff availability. Some grantees with small teams, or staff who managed several projects at once, reported 

the amount and/or complexity of the work was overwhelming. 

Grantees also reported staff turnover as an issue, at both the SA and LEA level. Employee turnover often 

made it difficult for grantees to keep up with training new staff (both in the program requirements and in 

using the new technology). One State saw a 25% staff turnover rate at the LEA level, reporting the 

turnover, and the need to train new LEA staff, interrupted the grantee’s progress on its ART project. 

Another State reported its high turnover rate at SA and LEA levels resulted in a general lack of 

institutional knowledge that made it especially difficult to test new systems to ensure they met the 

project’s needs. 

At least two grantees reported that their new systems built with grant funds are not being used to their full 

potential because their staff do not have the time to load required information and configure the systems.

Project Management

Some grantees reported high turnover in the project manager 

role—one grantee had at least four project managers over its 

ART grant period—stalling progress on the project. High staff

turnover with some grantees made it challenging to maintain 

institutional knowledge. One grantee reported its project 

manager did not know the program well enough to effectively 

manage the project. Two grantees discussed a need for a formal 

project management plan, to help ensure a smoother transition 

when staff turned over and “alleviate some of the stops and 

starts.” 

 

1

 Individual grantee challenges are highlighted in the Grantee Profiles (see Appendix A). 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌10

At least five grantees reported that their project manager lacked communication skills, which strained 

relationships among the project team members or between the project team and partners. Two grantees 

specifically mentioned the challenge of facilitating communication between program staff and technology 

staff. Both grantees expressed the need for a project manager who understood the program and could 

effectively communicate the needs of the program to partners, programmers, and IT staff.

Aside from issues with project managers, several grantees reported challenges with planning for and 

prioritizing the sometimes competing requirements of the grant project with other work. For example, 

several grantees highlighted how difficult it was to move forward with developing new systems for the 

grant when they were still awaiting or receiving guidance from USDA on new program guidelines. Such 

programmatic demands required staff and resources to be pulled away from ongoing ART grant efforts, 

and that led to some confusion with what activities had yet to be completed. Grantees expressed the need 

for a dedicated project manager with experience managing a federal grant and could handle tasks like 

building and tracking timelines and budgets, and fulfilling reporting requirements. Preferably, this would 

be someone other than a program director who is also juggling the technical implementation of the 

intervention and program office management. 

Hiring and procurement processes presented another challenge for effective project management. One 

grantee reported it took nearly seven months to finish hiring key project staff, which led to significant 

delays at the beginning of the grant period. Other grantees reported delays related to the multiple layers of 

review required for procurement (especially for contracting with software vendors) and issues drafting 

security plans for IT projects, which they had not anticipated before the ART project began. At least two 

grantees reported that a lengthy and complex procurement process (within their own State 

administrations) was why they needed grant extensions. 

Four grantees reported that FNS’ technical assistance came too late and created more work for them. At 

least two grantees suggested it might have benefited them more to have received training or technical

assistance prior to applying for the grant, or at the grant planning stage. Assistance earlier in the process

might have helped them to better understand the grant’s project management requirements and possibly 

set up a more formal, more effective project plan at the onset. At least one grantee reported that by the 

time it received assistance, the assistance felt like added burden due to micromanagement and 

requirements that overwhelmed them.

Despite inclusion in the Request for Application for the grants, several grantees reported they were not

aware that various aspects of project administration were allowable costs (e.g., travel for required FNS 

meetings and staff salaries) until after the grant had closed. 

Other grantees were unfamiliar with grant administration 

requirements such as monthly progress reporting and annual 

FNS-required reports. This made project management more 

challenging, as they experienced that learning curve 

concurrent with implementing their intervention.

Rollout of New AR Requirements

As noted above, during the CN ART grant period USDA 

switched from Coordinated Review Efforts (CREs) to 

Administrative Reviews (ARs). Many grantees reported 

challenges as a result of the new, more detailed AR

requirements from FNS. The Healthy Hunger Free Kids Act 

(HHFKA) of 2010 created a new AR process, which did not replace the CREs until School Year 2013-14, 

well after the start of many of the ART grants (ART grants begun in 2009). About one third of grantees 

reported USDA’s policy changes during the grant period delayed or otherwise adversely affected 

implementation of their grant projects, and noted staff would have benefited from additional guidance 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌11

about the new requirements. Additionally grantees reported 

challenges stemming from staff not fully understanding the 

functionality needed to address the new AR requirements. 

Technology

Most of the grantees’ ART grant projects included a 

technology component, with some projects making a major 

shift from paper-based to electronic processes. In the course 

of implementing new systems, grantees found LEAs varied 

in their degrees of computer literacy and use. Seven grantees 

reported that in addition to training on their new systems, 

they had to provide basic computer skills training, from 

turning on a computer and using a mouse, to creating a secure password, to setting up Internet access. At 

least two grantees reported a challenge setting up and maintaining access levels for LEA staff who would 

be using the computers and high rates of staff turnover demanded frequent updates and changes to the 

security framework.

Besides issues of computer literacy, several grantees reported challenges related to data systems and 

infrastructure. Several grantees had issues getting their new systems to interface and work with their 

existing systems, including their financial systems, direct certification systems, local systems, and pointof-sale systems at the LEA level. 

At least five grantees reported that one of their greatest challenges was working with software vendors to 

get the right balance of standardization and customization. That is, the grantees reported that their 

understanding at the onset of the grant project was that 

vendors would start with off-the-shelf systems and then 

provide minor tweaks to make the systems work in their 

States. Instead, grantees found that their needs and 

requirements led to vendors making many more 

modifications than anticipated. This sometimes caused a 

breakdown in grantees’ relationships with vendors. A few 

grantees also reported difficulty in getting vendors to 

respond. In particular, at least five grantees reported that 

their IT vendors were working multiple projects at once and 

were unable to provide dedicated staff or sufficient time to 

keep the grantees’ projects on schedule. For most of the 

grantees that reported this as an issue, it seemed they learned 

of their vendor’s shortcomings after the contract was signed. 

One grantee reported that customization led to a higher-thananticipated cost for updating and maintaining the system2

. 

Customization aside, even when a new USDA regulation or 

policy requires multiple grantees to make the same change 

(e.g., update a nondiscrimination notice) and those grantees 

share a single vendor, each grantee still pays the full cost of 

updating the code needed to implement the change within 

the parameters of its unique system.

 

2

 Additional funding sources used are highlighted in the Grantee Profiles (see Appendix A). 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌12

Timing of Challenges

Grantees reported challenges throughout the planning, implementation, and maintenance phases of their 

ART projects. In general, early challenges centered on project management, procurement, hiring, and 

competing demands for staff availability. Technology and vendor challenges tended to linger throughout 

the project. Later in project timelines, staff turnover became a greater challenge due to the loss of 

institutional and project knowledge. 

Overcoming Challenges

Grantees used a number of methods to overcome challenges faced during their grant projects. One grantee 

reported that developing training, funded by the ART grant, helped it overcome the challenge of 

onboarding new staff when there was turnover. Another approach to staff turnover is to build in some 

redundancy. One grantee reported it helped to have two project managers, so when one left the project, 

there was still someone who knew what was going on. 

In dealing with challenges related to project management, grantees pointed to a combination of effective 

communication and monitoring/accountability. For example, two grantees reported conducting group 

calls that involved multiple stakeholders in order to get their projects back on track after a delay or 

problem (rather than multiple one-on-one meetings among different members of the team). Three more 

grantees reported the usefulness of holding regular check-in meetings and other accountability measures 

(such as demanding satisfaction with IT system changes prior to payment) in getting or keeping their 

projects moving forward. 

In dealing with technology challenges, grantees also reported maintaining communication and holding 

people accountable as useful. Five grantees also pointed to the importance of training—both in frequency 

(holding more trainings) and in access (bringing training to those who cannot come to a central location, 

or providing it online). 

Lessons Learned

Several grantees reported they would consider applying for 

or had received another ART grant or a Child Nutrition 

Technology Innovation Grant (TIG). About half of those

considering potential grant applications, were leaning 

towards the TIG because they needed to fund improvements 

to their whole system (across multiple nutrition programs), 

and ART only funds system improvements directly tied to 

the school meal programs. Four of those grantees reported

that the next time around, they would hire a project manager 

or make a project management plan mandatory. In response 

to IT and vendor challenges, one grantee reported it would 

involve its IT team at the beginning of the project. Another 

grantee reported it would include specific system and 

business rule requirements in the procurement process, 

rather than after a vendor was selected. 

One grantee reported it wished it had factored in IT staff 

billable hours that could be charged to the grant. At least two 

grantees reported they had not realized until later in their

projects they could ask for an extension, rather than having 

to return unused funds. 
PLANNING AND IMPLEMENTATION

Abt Associates CN ART Grants | Final Study Report Final ▌13

Regarding implementing a technology project, several grantees reported they had underestimated how 

extensive, lengthy, and cost-intensive testing would be. At least two grantees reported they had not 

factored in how expensive maintenance costs would be for a vendor to continually update the system once 

it was in place. 

One grantee recommended doing research ahead of time, asking other grantees specific questions about 

their experiences with vendors, about the costs, challenges, and maintenance.
FINDINGS REGARDING THE EFFECTS

Abt Associates CN ART Grants | Final Study Report Final ▌14

Findings Regarding the Effects of ART Grant Projects

Grantees set out to use ART grant funds to make improvements in their programs in a number of areas. 

Some grantees identified effects of their interventions on specific objectives or goals, and others alluded 

to unexpected results in areas that were more difficult to measure. For example, grantees reported online 

training led to greater transparency in the AR process and provided clearer expectations and better 

communication between the State reviewers and the LEAs. Other, more measureable findings, are 

discussed below. 

Changes to AR Process (Including Data Reporting/Collecting)

Grantees noted their AR reports were more accurate, which improved the quality and speed of their AR 

processes. Those improvements led to consolidated data that provided more detail for every level of the 

review—State, LEA and Administrative Reviewer. 

Administrative Error Rates

Overall, 11 grantees reported their ART grant projects reduced error rates, whereas seven said there were 

too many variables to comment on the impact on error rates. Grantees specifically mentioned having 

better controls in place and more accurate nutrient analysis contributed to the reductions. No grantees 

reported that administrative error rates increased. 

Administrative Costs

Overall, six grantees reported their ART grant projects reduced administrative costs, by reducing 

paperwork and the time and effort required to conduct the AR. Three grantees reported their costs

increased, including costs for more training over time, added costs for maintenance and updates to new 

systems, and increased staff time spent on ARs. Six grantees reported no perceived impact on 

administrative costs.

Direct Certification

Improvements to direct certification were reported among the greatest successes of the ART grants. 

Grantees highlighted how much easier it is to identify eligible families, and how much more secure that 

process is. One grantee specifically mentioned that better direct certification allows the State to confirm 

eligibility for families more quickly.

Other School Nutrition Programs

Grantees acknowledged that funds from the ART grants were restricted to efforts targeted to the school 

meal programs, but some grantees reported other Child Nutrition Programs had benefited from systemwide improvements, increased eligibility matching, and training. One grantee mentioned the crossover

benefits for foundational skills such as leadership, communication, and team building that came about 

through the ART grant for staff who work across multiple 

programs. Another grantee mentioned the benefits of having 

built-in data entry checks, which improved the quality of 

data entry and eligibility screening for students and families 

across multiple programs.

Staff Burden

Following implementation of their ART projects, 14 

grantees reported staff spend less time on the AR process. 

Several grantees mentioned the time savings is partly due to 

a new system with centralized data entry, which reduces the 

time of both submission and review. One grantee mentioned 

data entry is more convenient and less disruptive to the 

regular work schedule for staff.
POST - GRANT

Abt Associates CN ART Grants | Final Study Report Final ▌15

Post-Grant Activities/Efforts

Sustainability

For the most part, grantees planned to continue the initiatives 

they developed under their ART grants.

Many reported they rely heavily on State Administrative 

Expense (SAE) funds, their own in-house operating budgets, 

or in-kind resources to maintain and update their IT systems, 

training modules, and other activities developed under the 

ART grant (Exhibit 8). For example, several grantees use 

SAE funds for annual maintenance agreements with software

vendors (i.e., annual software license fees). 

Grantees also relied on, or were applying for, other FNS 

funding streams to maintain and enhance their activities. As 

described in the previous section, at least half had received or were considering applying for TIGs. One 

grantee, for example, applied for a TIG and then issued a request for proposals for software vendors to 

make changes to its system that would require additional technical assistance and planning. Its contract 

award is contingent upon the State receiving TIG funds. 

Grantees also supplemented SAE funds with operating and State technology funding to support annual 

maintenance of their purchased AR system. One grantee reported relying on in-kind support from a 

university to continue hosting its web server.

Exhibit 8: Potential Post-ART Grant Funding

Note: Total does not add to 20 because grantees reported considering multiple funding sources to sustain their initiatives. 
POST - GRANT

Abt Associates CN ART Grants | Final Study Report Final ▌16

Sustaining ART grant initiatives has not been without challenges. One grantee reported not realizing it

would need to maintain the system developed under the grant and had not set aside resources for doing so. 

Another grantee reported key IT staff left the State, and nobody else knew the programming code for its

system.

At least one grantee discontinued some of the training developed under its ART grant, reporting it was too 

expensive to replicate statewide. It had been beneficial for the LEAs that participated during the grant 

period, but it was not feasible to expand the foundational skills workshops statewide. Another grantee 

reported it would be challenging for it to find resources to 

support staff time to apply for new grants. 

Expansion and Modification, Including 

Updates

Several grantees continued to enhance and expand activities 

originally funded by the ART grants. For example, one 

grantee is planning to add a function that allows LEAs to 

enter their corrective actions within the online review tool. 

Other expansions included developing additional training 

modules and system-generated reports. 

One grantee is considering expanding its new system to 

incorporate other Child Nutrition Programs administered by 

the SA. Another grantee reported investigating 

advancements made by other grantees for solutions that 

might more comprehensively address its short-term needs 

and adapt to its longer-term needs.
CONSIDERATIO NS FOR FNS

Abt Associates CN ART Grants | Final Study Report Final ▌17

Considerations for FNS

Grantees reported many challenges and lessons learned based on their experiences planning for and 

implementing initiatives funded by the CN ART grants. Based on these experiences, FNS may want to 

consider the following when funding future ART grants: 

• If possible, avoid implementing major policy changes during the grant period. Many grantees 

reported challenges related to rollout of new AR requirements during the grant period. Allowing sites 

adequate time to develop and implement new policies and regulations, including guidance to grant 

applicants, prior to grant award will eliminate later confusion among grantees and help ensure their 

interventions are fully compliant with FNS requirements.

• Provide clear grant administration guidance. FNS should ensure grantees are aware of various 

aspects of grant administration, including allowable costs and reporting requirements. FNS provides 

training to ART grantees on required reports and grant administration, however, several of the 

interviewees were new to grant administration and were unclear regarding what were allowable costs 

and how often FNS expected progress reports and in-person meetings/conferences. The trainings for 

the grantees may not be clear or may need to done more frequently due to State staff turnover. 

• Provide technical assistance related to IT procurement. Grantees may not have experience 

procuring IT services. Grantees not familiar with sourcing and scoping out specific work 

requirements for IT systems would benefit from early guidance on the following: 

− costs related to design and ongoing maintenance; 

− drafting systems specifications and other technical requirements; and

− setting realistic schedules for design, implementation, and testing prior to selecting software

vendors or working with in-house IT staff. 

• Help grantees share information with one another and leverage their collective buying power.

Designing and implementing new systems is expensive and time-consuming. FNS could facilitate 

information sharing across grantees, allowing them to benefit from one another’s developments and 

advancements in implementing similar systems. Some grantees recommended that FNS create a 

single AR tool, rather than having grantees develop their own systems. Another option would be for 

FNS to negotiate with software vendors on behalf of the 

grantees to leverage their purchasing power for systems 

development. 

• Expand the scope of ART grants beyond the school 

meal programs. Grantees and LEAs manage multiple 

Child Nutrition Programs, including the Child and 

Adult Care Food Program and the Summer Food 

Service Program, and it is not efficient nor costeffective for them to develop initiatives for single 

programs. Expanding the scope of the ART grant to 

multiple programs will allow grantees to upgrade their 

entire systems with its funds.

• Customize the role of the FNS technical assistance

contractor to grantees’ specific needs. Some grantees 

appreciated the technical assistance support, whereas

others found it diverted their efforts and implementing 

their interventions. FNS should encourage its contractor to assess each grantee’s individual needs and 

tailor its support accordingly and as soon as possible in the grant’s cycle. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌18

Appendix A. Grantee Profiles

The following appendix includes detailed profiles for each grantee. Each profile presents an overview of 

the grantee’s project(s) along with highlights related to planning and implementation, successes and 

challenges, and lessons learned. The information provided is drawn from both the extant data and 

qualitative interviews.
Abt Associates CN ART Grants | Final Study Report Final ▌19

Overview

Grant Start Year: 2009 Grant End Year: 2013

Purpose: Reducing errors and implementing trainings Extension(s): Yes

Project Name: Alabama ART Vendor: Outside, for-profit training provider

Grant Funds Awarded: $494,769 Grant Funds Returned: $217,491

Alabama received an Administrative Review and Training (ART) grant in FY 2009 to improve its

technology systems, particularly related to its administrative review (AR) and verification processes. Its 

goal was to improve the IT systems to reduce errors. With the improvements, Local Educational Agency 

(LEA) staff are able to use the verification summary tool to randomly select which households should be 

verified and to confirm the number of verifications staff need to collect and process, which was not 

possible prior to the ART grant. Additionally, parts of the AR process have been automated, requiring 

significantly fewer paper-and-pencil processes, thus reducing errors. The State director estimated that 

grant activities led to a 25-40% reduction in the error rate. 

Goals and Objectives

Grant Application Post-Grant Implementation Updates

Goals & Objectives

• Training LEA administrative staff on application certification, 

verification, meal counting, and meal claiming procedures in 

order to reduce errors and increase the maximum meals 

claimed. 

• Technology improvements to address administrative errors 

including the development of a new system to identify and 

monitor error-prone sites and computerization of verification 

submission. 

Goals & Objectives

• Though the interviewees confirmed the goals and 

objectives were not changed from those stated in 

the application, the interviews focused mostly on 

the technology improvements, including the 

verification summary report and the automation of 

the AR auditing process. 

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and LEA staff members about their experiences related to the ART grant and its 

associated activities.

Highlights

Alabama applied for an ART grant in 2009 to implement new online training modules, improve the AR 

system, and simplify the verification process. Activities funded through the ART grant achieved the 

following: 

• Nine online training modules were created to focus on meal counting and claiming, error rate 

reduction, and administrative reviews. 

• The State moved its administrative reviews from a paper-and-pencil process onto a newly 

implemented software system.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌20

• The new verification tool tells LEAs how many applications need to be reviewed for verification and 

how many have already been verified, tracks the verification progress, reduces error rates, and works 

with existing point-of-sale software.

Planning and Implementation

Management

At the State level, an education specialist who was involved with the National School Lunch Program 

(NSLP) was primarily responsible for managing the grant application. School Food Authorities were not 

directly involved in grant planning or application, though their feedback was solicited to inform what 

would make the AR process more seamless for them. After the grant was awarded, the SA involved 

administrators from each of its four sections (Child and Adult Care, NSLP, Food Distribution, and Audit) 

in grant management. Though the project manager left the team in the middle of the project, the State’s IT 

person was able to provide continuity throughout the grant period. 

Funding

The State returned a significant portion (44%) of the ART funds at the end of the grant period. This return 

was attributed to an overestimation of the budget expenditures, lack of knowledge regarding what were 

allowable grant costs, and that the majority of staff time was not charged to the project (e.g. billable hours 

for internal IT staff were not charged to the grant). 

Successes

The grant activities were successful overall, as they effectively reduced administrative costs and error 

rates. The verification tool, which randomly selects households to be verified and provides relevant 

information to LEA staff, has been particularly helpful to LEAs and is still in use.

• Administrative Costs: This intervention saved administrative costs, mostly through staff time. Once 

LEAs learned the new system, the process was faster for them. The audit manager spent 50% less 

time (estimated) on verification processing following the rollout of the new system.

• Error Rates: The grant activities successfully lowered the number of administrative errors in meal 

counting and claiming. There are fewer errors found at the LEA level during the administrative 

reviews; a 25-40% reduction in error rate was estimated.

• Data Systems: The verification system developed through the ART grant works well and is still in 

use. Additionally, an AR tool was created to select districts to participate in the reviews and collect 

data on error rates. 

Challenges

Though the goals of the project were achieved, there were challenges, including staffing and 

misunderstanding grant requirements. 

 Initial Grant Amount: $494,769.

 Activities Funded by the Grant: Creation of online trainings, an audit tool, and verification 

summary report.

 Additional Funding Sources: State Administrative Expenses were used to sustain grant 

activities past the grant end date.

 Funds Returned: $217,491.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌21

• Project Management: Project coordination strategies were inconsistent due to staff turnover at the 

SA. The lack of consistency in oversight from the beginning to the end of the project created some 

disorganization and lack of continuity/progress in grant activities. Additionally, State staff reported it 

would have been helpful to have an IT person who was dedicated to working on Child Nutrition 

projects, instead of spreading time across projects.

• Staff Turnover: During the time the new AR/verification systems were being implemented, there 

was turnover among Child Nutrition directors at the LEAs. This particularly affected the smaller 

school districts, as the Child Nutrition director was often the only person familiar with the program or 

the submitted data. 

• Buy-in from State Staff: It was difficult at times to get buy-in from State staff. Interviewed staff 

reported the lack of project leadership continuity may have contributed to the issue. 

• Understanding Grant Requirements: Staff at the State level were not fully aware of USDA’s 

policies and procedures or what costs were allowable within grant funding. For example, the initial 

budget did not account for travel expenses to attend grantee meetings in Washington, DC.

Lessons Learned

After completing this ART grant project, State staff realized having a dedicated project manager is 

imperative for organization and communication. The ART project led to a better understanding of future 

grant requirements and the flexibility that exists to change the scope of work and project deadlines.
Abt Associates CN ART Grants | Final Study Report Final ▌22

Overview

Grant Start Year: 2012 Grant End Year: 2015

Purpose: Technology improvements to address admin 

errors through the use of targeted monitoring and 

increased training

Extension(s): Yes

Project Name: Alaska ART Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $1,058,915.00 Grant Funds Returned: $3,751.98

Alaska chose an intervention strategy to address administrative errors through the use of targeted 

monitoring and increased training. Specifically, the Administrative Review and Training (ART) grant was 

used to design a software solution to improve program integrity in the administration of the National 

School Lunch Program (NSLP) and School Breakfast Program (SBP). The State purchased and 

customized software. The software solution automated systems to gather data and provide real-time 

monitoring and benchmarking. These functions were previously completed manually. The new system 

identifies errors and provides specific corrective action. The State is able to systematically and 

consistently measure the impact and success of the corrective action taken. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Reduce application errors. 

• Improve Integrity of Benefit Issuance Document. 

• Improve menu compliance. 

• Production record compliance. 

• Improve coordinated review efforts (CRE). 

Goals & Objectives

• Goals 1-4 remained the same 

• Goal 5 changed to “improve administrative review tools,” 

consistent with FNS’s change from CRE to AR.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

The SA was particularly interested in developing a system and the processes to work with schools 

experiencing significant problems, particularly the most error-prone schools. The software intervention 

was selected as a means for the SA to support smaller districts lacking the infrastructure and resources to 

afford automated systems. Additionally, the system implemented needed to allow the SA to provide 

support from its “home base” because travel in Alaska is expensive and takes significant time. The ART 

grant–funded activities achieved the following: 

• Eligibility errors have decreased significantly.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌23

• The direct certification rate has improved (94% or higher), and the direct certification process is 

completed through the new system. This is particularly noteworthy as it eliminated the time to send 

individual, encrypted emails for each student, and the SA has been able to add direct certification of 

foster children and migrant children. The State is also able to operationalize the Food Distribution 

Program for Indian Reservation (FDPIR) children with the new system.

• Staff and travel costs have decreased. Applications are entered at the local level and reviewed and 

approved at the SA level (in Juneau). 

• LEAs are able to see immediate technical assistance from SA staff.

Planning and Implementation

Management

Alaska contracted with a local IT company as the project manager (with approval from the overall State 

Administration). The contracted project manager worked through issues with negotiating IT security 

plans between the State and the new software vendor. The project manager also oversaw the project 

implementation, made decisions about the “look and feel” of the online program, and coordinated 

technical assistance as needed. 

Funding

Alaska was awarded $1,058,915 to complete its 2012 ART grant, and used 99% of the total award. State 

Administrative Expense (SAE) funding was used to cover staff time. One no-cost extension was granted 

due to delays encountered with negotiating acceptable security plans between the State and the software 

vendor; working with the internal IT department for a mutually acceptable “look and feel” to the new 

online system; and working through the State system to finalize procurements with the contracted project 

manager and the software vendor. 

Successes

The grant achieved the goals of streamlining and automating the administrative review process,

improving the application and claiming processes, and improving menu and production record 

compliance. 

• Administrative Costs: The amount of staff time and travel costs required for administrative 

processes has significantly decreased. Most of the AR process formerly done in person may now be 

monitored by SA staff off-site.

• Error Rates in Application Processing: After the initial implementation year (where error rates 

were at 45%), the State began seeing rates significantly reduced in the second and third years.

• Training: The new system allows SA staff to conduct real-time training and technical assistance with 

LEAs. For example, SA staff are able to walk through a menu planning module (on-screen) and assist 

the LEA with completing the task. 

• Data Systems: All school districts in Alaska use the new system.

 Initial Grant Amount: $1,058,915.00.

 Activities Funded by the Grant: IT software, hardware such as dedicated servers and 

scanners, and a contract for an external project manager.

 Additional Funding Sources: SAE funds were used for internal staffing costs, which were not 

included in the ART grant budget.

 Funds Returned: $3,751.98.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌24

• Direct Certification: Though not an original goal of the grant, the State has increased direct 

certification to 94-96% over the last couple of years, and 100% of the LEAs use the new online 

system for direct certification. 

• Community Eligibility Provision (CEP): The decreased error rates allowed a number of schools to 

qualify for CEP. The SA is thrilled with this unanticipated benefit of the grant project intervention. 

Challenges

A no-cost extension was required due to unanticipated delays with procurement processes, the need for 

mutually acceptable IT security plans, and the State IT department’s other requirements. 

• Administrative Costs: The SA did not account for the ongoing, annual cost of the software license.

• Data Systems: The SA found testing the new system to be the most time-consuming and frustrating 

part of implementing the new software. The lack of a dedicated IT team further complicated the 

process. 

• Adapting Implementation to AR: The State struggled to adjust its implementation plans to the new 

AR process, since the original plans were written to align with the Coordinated Review Effort. Due to 

this challenge and technical issues with data systems discussed above, as of March 2019 the software 

vendor’s module for the AR was not yet completed, but the State is able to use the software vendors 

system for other aspects of program implementation. AR staff continues to use spreadsheets to 

conduct reviews.

• Internal IT Administration: Limited IT staffing puts larger demands on SA staff to conduct their 

own testing and review. Also, the State IT department required that the “look and feel” of the new 

system resemble that of other State programs.

• Procurement Process: The mandates with the State of Alaska regarding procurement processes 

increased the time it took to purchase the software, hardware, and project management resources 

needed to successfully implement the interventions. 

Lessons Learned

Applying for an ART planning grant, before attempting to implement a new technology through an ART 

implementation grant, would have helped the SA better understand what it would need in a new system to 

resolve the challenges it faced. It also would have assisted with implementing the new system more 

efficiently and effectively.

The SA would contract out testing of future technology solutions. Relatedly, AR staff said it would have 

been helpful to draw on the experience that other states had with implementation of similar systems. 

Alaska might have been able to avoid some of the technical issues with rolling out new systems if it could 

have taken advantage of knowledge from other states.
Abt Associates CN ART Grants | Final Study Report Final ▌25

Overview

Grant Start Year: 2010 Grant End Year: 2012

Purpose: Creating and delivering training materials Extension(s): No

Project Name: School Meal Program Integrity (SMPI) 

Training

Vendor(s): University or Community College training 

developer

Grant Funds Awarded: $500,000 Grant Funds Returned: $159,942.09

In 2009, California received an Administrative Review and Training (ART) grant to train school nutrition 

professionals in error prone agencies on how to administer high-quality nutrition programs in compliance 

with federal regulations on meal claiming, menu patterns, and nutrition standard requirements. The 

Nutrition Services Division (NSD) targeted agencies using Nutrient Standard Menu Planning (NSMP) to 

help them transition to the new Food-Based Menu Planning (FBMP) requirements. 

To accomplish this goal, the State developed School Meal Program Integrity (SMPI) training materials 

for food service professionals. A training developer was contracted to implement and create SMPI 

materials including a one-day FBMP course (Successful Menu Planning: Transitioning to Current USDA 

Nutrition Standards) and a companion train-the-trainer course, which included information on effective 

teaching strategies and classroom management. The grant also funded the creation and dissemination of 

five web-based trainings, which covered topics to assist agencies in decreasing administrative errors and 

improving compliance with School Nutrition Program (SNP) regulations. Lastly, the training developer 

created self-assessment tools for menu planning and meal counting. It also conducted an evaluation of all 

the grant activities, resulting in some modifications and improvements based on feedback from 

participants and trainers. As a result, the State Agency (SA) is able to provide easily accessible and 

comprehensive training materials to Local Educational Agency (LEA) staff. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Develop School Meal Program Integrity (SMPI) training 

materials focused on ensuring that meals claimed for 

reimbursement contain meal elements (e.g. food components, 

menu items, portion sizes) and meet nutrition standards 

consistent with USDA requirements for Food-Based Menu 

Planning. 

• Utilize a cadre of mentor instructors, with experience and 

expertise in meeting the USDA meal requirements, to provide 

the delivery of the SMPI training materials to the targeted atrisk school district directors, managers, and front-line 

personnel.

• Utilize Web-based or other technology to increase number of 

at-risk sponsors receiving training and instruction. 

• Provide self-assessment tools to assist at-risk agencies in 

Goals & Objectives

• The State adjusted its goals after the grant was 

awarded (with approval from USDA) to focus on 

the FBMP course and webinars (developed in 

partnership with training developer), rather than 

produce two different courses (one for FBMP and 

another for Nutrient Standard Menu Planning), 

due to changes in USDA regulations (which 

required FBMP).

• Also the SA did not end up pursuing a 

partnership with the software developer to create

the college course Menu Management for Child 

Nutrition Programs. California met its remaining 

goals of implementing a variety of training 

materials that used web-based technology, self-
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌26

Grant Application Post-Grant Implementation

monitoring their continuous improvement in meeting all 

requirements for meal claiming and improving compliance with 

nutrition standards.

• Ensure consistency among CDE staff conducting 

administrative reviews.

• Provide ongoing oversight and evaluation of the effectiveness 

of the ART grant in decreasing administrative errors relating to 

meal claiming and nutrition standard requirements.

• Develop a college-level course on Menu Management for Child 

Nutrition Programs.

assessment tools, and an evaluation of grant 

activities. 

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.

Highlights

California’s 2010 ART grant funded the implementation of multiple trainings for food service 

professionals aimed at improving compliance with federal regulations on meal claiming, menu patterns, 

and nutrition standard requirements. The activities funded by the ART grant achieved the following: 

• Taught a two-day train-the-trainer course on March 28-29, 2012, with a total of 23 participants. 

• Offered the SMPI course Successful Menu Planning: Transitioning to Current USDA Nutrition 

Standards for at-risk agencies, and provided a companion Instructor Guide, course manual, and preand post-tests. From April 17, 2012, through September 13, 2012, some 301 participants attended the 

training. 

• Developed and presented five webinars in 2012 that targeted the areas of meal counting, offer versus 

serve, salad bars, menu production records, and requirements for transporting meals. The number of 

attendees ranged from 147 to 237, and the information was later posted online. 

• Developed self-assessment tools for menu planning, designed to assist in evaluating menus for 

compliance with meal patterns, as well as other features of menu designs. 

• Conducted evaluations of all the grant activities, all of which received positive reviews along with 

suggestions on improvement that were incorporated into the final curriculum. 

Planning and Implementation

Management

Existing staff at the SA managed the budget, timeline, and progress of the ART grant. The SA contracted 

and worked closely with one of its existing partners to develop and implement all of the SMPI materials. 

In the original contract, a second training developer was also contracted to develop and teach a threesemester unit course for on-site school nutrition managers, an additional certification program for child 

nutrition managers, and obtain college approval to adopt the class into its curriculum and teach it for one 

semester. However, these activities were not realized. Initial delays and complications in the contract 

review process led to changes to the scope of the final project.

Funding

In 2010, California was awarded a $500,000 ART grant to create training materials and presentations to 

deliver to the staff at at-risk schools throughout the State. To complete these activities, the State used 68% 

of the funds that it was awarded, returning $159,942.09. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌27

 Initial Grant Amount: $500,000.

 Activities Funded by the Grant: Creation and dissemination of a one-day FBMP course, a 

two-day train-the-trainer course, five web-based trainings or webinars, two self-assessment 

tools for menu planning and meal counting, and an evaluation of the grant activities. 

 Additional Funding Sources: State Administrative Expense Funds.

 Funds Returned: $159,942.09.

Successes

The grant achieved its goal of developing and disseminating training and technical assistance to help food 

service professionals in high-risk LEAs. With the ART grant, the State contracted with a training 

developer to create materials, conduct training, and present webinars on a variety of topics that presented 

challenges for LEAs. 

• Error Rates in Administrative Processing: LEAs expressed that the online webinars and tools 

helped staff easily access information about meal planning, procurement, compliance, and direct 

certification. The accessibility of trainings then helped decrease administrative errors because staff 

were better prepared for activities such as procurement reviews. 

• Training and Professional Standards: 

o With the assistance of this grant, at-risk LEAs received more detailed in-depth one-day 

training on the new lunch menu pattern than the California Department of Education (CDE) 

would have been able to provide otherwise. At-risk LEAs were identified as those that were 

using NSMP, those with new directors, and agencies with findings in prior Coordinated 

Review Efforts.

o All LEAs (including at-risk agencies and those not identified as at-risk) were able to attend 

the webinars. Webinar materials and handouts were then posted on the training developer’s 

university website for future use by food service professionals.

o The menu planning self-assessment tool became part of the yearly CDE-required online 

training module for menu planning. 

• Administration of School Nutrition Programs: Approximately half of the attendees at the 

Successful Menu Planning: Transitioning to Current USDA Nutrition Standards courses were using 

NSMP prior to July 1, 2012. These eight food service directors in particular had no knowledge of how 

to determine the contribution to the meal pattern for recipes and purchased products. These trainings 

were very beneficial to these agencies to implement the FBMP standards.

• Administrative Time: LEAs noted that the training helped staff gather information and be more 

aware of what is needed for the administrative review. They noted that the training materials allowed 

everyone to acquire the same knowledge base. This saved administrative time because it allowed staff 

to avoid additional research on the administrative review process. SAs also explained that the 

trainings helped alleviate the administrative burden of staff turnover at the food director level because 

it helped the review/training process run more smoothly. 

• Improved Management Relationship: SA staff reported that when they provide effective training 

such as the SMPI program, LEAs feel better supported and are therefore more open to the SA during 

reviews, rather than having an adversarial relationship. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌28

Challenges

• Administrative Costs: LEAs communicated that even though the training tools made information 

more readily available and provided a road map for the administrative review, they did not believe 

that the training reduced the amount of work required for staff. 

• IT Challenges: The SA expressed that the State does not have a department-level learning 

management system, making it difficult to shift trainings to web formats. The State also had 

challenges ensuring materials met California’s 508 Accessibility Standards (the State IT department 

disagreed that some of the materials from USDA met the standard). Also, the training developer

experienced technical difficulties during the first two webinars, but offered the remaining webinars 

live, with no additional problems. 

• Sustainability: The SA and LEAs mentioned some challenges related to keeping the training 

curriculum up-to-date. In particular, the LEAs expressed challenges in applying their training when 

their vendors don’t use the same system, or if vendors are in a different state and are not up-to-date on 

California policies. The SA mentioned it has been a challenge to keep up with changing technology, 

and to keep all of the training materials in compliance with California’s latest accessibility standards, 

which are sometimes higher than national standards. 

• Vendor Challenges: The SA continues to value its relationship with community college partners, and 

stressed the importance of allowing a lot of time and flexibility to account for some of the typical

restraints (i.e., tight budgets, sparse staffing, and limited resources) common to community colleges 

and might lead to unexpected changes in timeline or ability to fulfill grant requirements.

Lessons Learned

Through the ART grant, California learned that providing web-based and accessible training materials is 

invaluable. The online materials can now reach a large number of participants in remote areas. 

Additionally, LEA staff and the SA noted that having access to training materials at a later time was 

helpful. The webinars, webinar handouts, and participant training manual are now accessible online for 

training new food service professionals and other staff at districts who are unable to attend in-person 

training. These training materials are also available to staff in other states and are flexible to be modified 

or developed into additional materials. 

Other lessons included the importance of engaging Child Nutrition directors as trainers. They were seen 

as excellent trainers and inspirations to peers. The participants appreciated the information provided by 

current or recently retired food service directors, including examples of their experiences.
Abt Associates CN ART Grants | Final Study Report Final ▌29

Overview

Grant Start Year: 2014 Grant End Year: 2017 

Purpose: Developing a new technology system to 

improve the error rate and the direct certification rate

Extension(s): Yes

Project Name: CNP Statewide System Vendor: Outside, for-profit software vendor 

Grant Funds Awarded: $1,700,000 Grant Funds Returned: $53,482.61

In 2014, Guam received an Administrative Review and Training (ART) grant to improve its Child 

Nutrition Program (CNP) operations. Guam used the grant funding to move from a manual process to a 

technology-driven system, CNP Statewide System, which allowed the State Agency (SA) to better assess 

and monitor School Food Authorities’ (SFA) compliance with certification, verification, meal counting, 

and claiming and to allow for the establishment of performance benchmarks in financial management and 

meal planning. With these updates, the SA enhanced its ability to identify, review, oversee, and train 

high-risk SFAs in application processing and certification, verification, and meal counting. As a result, 

the SA was able to reduce error rates, improve communication with SFAs, and increase overall efficiency 

with the new CNP Statewide System. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Customize an off-the-shelf software to manage the CNP 

program. The system will improve the ability to use student, 

school, and financial data; help the SA reach the 90% direct 

certification benchmark; and improve the point of sale (POS) 

interface.

Goals & Objectives

• The goals and objectives did not change from the 

grant application to implementation phase. Guam 

met these goals by implementing the CNP 

Statewide System, decreasing error rates, and 

increasing direct certification rates. 

The following sections include information from grant applications, final grant reports, and discussions 

with SA and Local Educational Agency (LEA) staff members about their experiences related to the ART 

grant and its associated activities.

Highlights

Guam’s 2014 ART grant funded the implementation of a customized technology-driven system to 

manage its CNP. The activities supported by the ART grant achieved the following:

• Centralized all information needed for meal applications based on categorical and income eligibility.

• Facilitated better monitoring and communication between SA and LEA staff. 

• Improved efficiency by reducing calculation errors. 

• Led to more accurate and efficient matching of student data (including matching with siblings) to 

meet direct certification benchmarks. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌30

Planning and Implementation

Management

Staff from Guam’s SA and LEAs were involved with the 2014 ART grant project. The senior State

program officer and the superintendent decided to apply for the grant with the help of LEAs that helped 

gather information for the application, including information on what types of challenges they were 

experiencing. The State program officer oversaw all the grant activities and administrative reviews. 

Additionally, the State financial officer, who was also a State program officer, managed the financial 

monitoring of the grant accounts and supported grant activities. 

Funding

In 2014, Guam was awarded an ART grant for $1,700,000. It used 97% of the allocated funds. The 

majority of ART funding was spent on vendor contracts. The grant also covered additional costs including 

travel, personnel, and supplies; in particular, the SA program officer’s labor costs were funded by the 

grant. No other supplementary funding sources were used as a part of this project. 

Successes

The grant successfully achieved its goal of transitioning from a manual to an automated process by 

developing a software solution to manage the CNP operations in Guam. Additionally, the development of 

the new CNP Statewide System helped the SA achieve its objectives of decreasing verification and 

certification errors. 

• Administrative Costs: The ART grant reduced the amount of time staff spent on the administrative 

reviews. Additionally, LEA staff noted that the new system’s automatic calculations reduced staff 

time on related tasks. 

• Error Rates in Application Processing: LEA and SA staff described decreased error rates in terms 

of meal claims and calculation errors due to the CNP Statewide System’s automation of mealtime 

reports. 

• Training and Professional Standards: LEAs reported that the vendor was quick to respond to any 

issues, and the SA provided readily available support to staff. 

• Data Systems: The automation and centralization of program data helped the SA review ongoing 

applications, streamline administrative reviews, and improve the efficiency of audits. Additionally, 

the improved transfer and review of data facilitated better communication between LEAs and the SA, 

making it easier for the SA to address questions during the administrative review. 

• Improved Direct Certification Process: SA staff saw an improvement in direct certification within a 

year of hiring the vendor. They reported that the system’s improved matching algorithm led to an 

additional 2,000-3,000 matches and more potential matches including siblings. This greatly enhanced 

the previous system, which required that SA staff manually match children with similar names for 

approximately 30,000 students. 

 Initial Grant Amount: $1,700,000.

 Activities Funded by the Grant: Development and implementation of the CNP Statewide 

System, travel costs, supplies, and labor costs for State program officer.

 Additional Funding Sources: None.

 Funds Returned: $53,482.61.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌31

• Administration of School Nutrition Programs: LEAs explained that the new system helped with 

program management because the centralization of data made it easier for staff to review program 

metrics. 

Challenges

The Guam ART grant successfully improved efficiency and reduced errors in the Child Nutrition and 

Food Distribution programs; there were still challenges in implementing the CNP Statewide System. For 

example, one no-cost extension was granted due to delays with the procurement process and delays in 

finalizing data-sharing agreements for student data. Another significant challenge included a series of 

technical issues involved with developing and testing the system. 

• Administrative Delays: The procurement process took longer than the SA expected and was delayed 

by about a year, necessitating a no-cost extension. Migrating data from military schools (overseen by 

the U.S. Department of Defense) to the statewide system also required a separate Memo of 

Understanding that took time to finalize, leading to overall delays in launching the CNP Statewide 

System.

• Data Systems: Issues arose in the development and testing phases of implementation, including 

challenges with data migration as the SA pilot tested various modules. In response to the delays, the 

SA scheduled additional calls with the vendor and performed live tests to help address immediate 

issues. Other noted technical issues included challenges with connection speeds in Guam. For 

instance, uploading large documents sometimes took up to a day and a half. 

Lessons Learned

Many of the State’s challenges came from unexpected administrative and IT delays. The SA staff also 

described the need to leverage expertise from the various stakeholders (i.e., IT, LEAs, etc.) in the early 

phase of development, otherwise expectations can be unrealistic. Another lesson the SA learned is to be 

more specific about the types and sources of student data that are needed to make the system useful, and 

to allow sufficient time for MOUs to be developed before data can be shared. 
Abt Associates CN ART Grants | Final Study Report Final ▌32

Overview

Grant Start Year: 2011 Grant End Year: 2013

Purpose: Creation of a new Child Nutrition IT system Extension(s): Yes

Project Name: Hawaii Child Nutrition Program System 

(HCNP_S); Administrative Review (AR) Tool

Vendor: Outside, for-profit software vendor; Outside, forprofit project planning consultant

Grant Funds Awarded: $1,499,385 Grant Funds Returned: $46,613.71

In 2011, Hawaii received an Administrative Review and Training (ART) grant to develop and implement 

a comprehensive Child Nutrition (CN) technology system called the Hawaii Child Nutrition Program 

System (HCNP_S). The system aims to recognize at-risk schools through real-time data analysis and 

therefore ensure that corrective action as a result of administrative review (AR) of the Fresh Fruit and 

Vegetable Program (FFVP) and the Seamless Summer Option (SSO) is being implemented and 

maintained. HCNP_S allowed the State to reduce administrative errors and to improve program integrity 

and administrative accuracy.

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• The project goal is to develop and implement a 

comprehensive CN technology system to enable the SA to 

recognize “at risk” schools through real-time data analysis to 

ensure that corrective action resulting from CRE reviews, 

FFVP, and SFSP reviews are being implemented and 

maintained to reduce administrative errors, improve program 

integrity, and administrative accuracy.

• The technology system will:

o Have the ability to modify forms, reports, interfaces, 

and outputs as necessary, to improve program 

integrity and administrative accuracy especially 

targeted to schools/districts that are error prone.

o Collect benchmarking data to be used by the SA to 

establish a CN database to compare “at-risk” 

schools with peer schools.

o Provide appropriate training for sponsors/schools, 

SFAs, and organizations to prevent repeated 

administrative errors.

o Provide SA-level monitoring capability through 

proposed accountability software.

o Identify problems and take corrective action before 

they become long-term behaviors.

Goals & Objectives 

• The goals and objectives did not change from the 

grant application to implementation phase. Hawaii 

met these goals by implementing a 

comprehensive Child Nutrition technology system. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌33

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Hawaii’s 2011 ART grant funded the implementation of a State-hosted centralized Child Nutrition 

technology system and online administrative review tool. The technology system that the ART grant 

funded, HCNP_S, was able to achieve the following: 

• Modify forms, reports, interfaces, and outputs as necessary to improve program integrity and 

administrative accuracy especially targeted to schools and districts that are error prone. 

• Collect benchmarking data to be used by the SA to establish a CN database to compare at-risk schools 

with peer schools. 

• Identify problems and take corrective action before they become long-term behaviors.

• Provide appropriate training for sponsors/schools, School Food Authorities (SFAs), and organizations 

to prevent repeated administrative errors.

• Provide SA-level monitoring capability through accountability software.

• Improve the efficiency and accuracy of and reduce errors in the counting and claiming process.

Planning and Implementation

Management

A number of staff from Hawaii’s SA, a software and vendor, and a project planning consultant, were 

involved with the 2011 ART grant project. The SA director decided to apply for the grant and managed 

the application process with the help of the Hawaii Department of Education’s Office of Information 

Technology (OTIS). The initial ART grant coordinator oversaw all management of the grant project 

including monitoring tasks, schedules, and budgets for the first two years. Then there was a change in 

staffing, and a different grant coordinator monitored the project for the last year. Additionally, a project 

planning consultant was hired to help with project planning of the software vendor, which built the 

HCNP_S system and AR tool. 

Funding

In 2011, Hawaii was awarded $1,499,385 to create a new Child Nutrition technology system and online 

administrative review tool. The State used 97% of the allocated funds, and requested and received no-cost 

extensions to finish the project.

 Initial Grant Amount: $1,499,385.

 Activities Funded by the Grant: Development of a CN technology system, training for LEAs 

on the new system, an ART grant coordinator, and part-time staff from the University of 

Hawaii.

 Additional Funding Sources: State Administrative Expense funds were used to fund staff 

salaries and to expand the new system to include the Child and Adult Care Food Program 

and the Summer Food Service Program.

 Funds Returned: $46,613.71.


APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌34

Successes

• Improved Efficiency: LEA and SA staff reported that HCNP_S helped save time and improve 

internal efficiency. The system allows the SA to retrieve LEA data automatically, replacing a paperbased system and drastically reducing the duration of administrative reviews. Specifically, prior to 

HCNP_S, approvals for site-based attendance forms had to be signed in person. With HCNP_S, the 

forms are uploaded to the system and can be approved electronically. Additionally, the State and SA 

staff reported significant time saved in the claims process. HCNP_S keeps real-time data that is 

continuously available to sponsors and to the SA. As a result, fiscal SA staff reduced their time spent 

on monitoring claims by 70-75%. 

• Training and Professional Standards: The centralization of the system allows the SA to monitor 

training and to address corrective actions. The SA can update annual mandatory trainings based on 

the previous year’s common findings and new requirements. The SA also offered training on the new 

AR tool, including hands-on system-user training, technical assistance, and outreach strategies to 

educate SFAs on the effective and efficient use of HCNP_S. 

• Administration Costs: Aside from the savings realized by a significant reduction in staff time, 

administrative costs were also reduced by switching from a paper to a digital system. As an example, 

after HCNP_S implementation, claims could be corrected online rather than in person, which cut 

mailing costs for SFAs and the SA. 

• Error Rates in Application Processing: The State reported that HCNP_S has led to a reduction in 

error rates. There are some built-in checks to detect common errors. Additionally, the new system led 

to a reduction in the number of over-claims and returned claims, and it eliminated late claims. SA 

staff noted that whereas prior to HCNP_S it was easy to forget to follow up on the collection of 

certain forms, the new system streamlined the requirements and made it much easier to keep track of 

everything needed to complete the administrative review on time. The SA mentioned the process for 

addressing and correcting errors was greatly improved; whereas data entry errors prior to the 

HCNP_S required multiple rounds of mailing corrected paper forms back and forth between the SFA 

and the SA, the new system made it possible to fix the data in real time. 

• Data Systems: The automation, centralized organization, and streamlined administrative review 

system allowed Hawaii to collect timely and accurate source data. Some of the benefits include ease 

of data retrieval during audits and automatically transferring eligibility determinations into student 

records. 

• Improved Direct Certification: SA staff explained that the HCNP_S improved direct certification 

match percentages because the State-level and SFA-level files are now matched in the HCNP_S. 

Challenges

• Vendor Challenges: The SA noted that the primary issue it faced during implementation of the grant 

was working with the software vendor. It was the SA’s impression that the vendor had taken on too 

many contracts and could not keep up with the project timeline. Additionally, the vendor was working 

with a number of states and attempted to modify a common platform for each state. Some of the 

initial challenges SFAs experienced in using the AR tool had to do with questions in the system that 

were not relevant for Hawaii. 

• IT Challenges: SA staff described multiple technical issues that arose with the system. These issues 

were addressed and often identified by SFA staff who had the most contact with the system. As an 

example, during the first year of implementation, LEA staff described that the system would 

automatically log out and not save data entered. This feature was fixed during the second year, but it 

was a significant challenge for staff during initial implementation. 

• Buy-in with SFAs: SFAs initially were hesitant to switch from a paper-based to an online system, 

which created lack of buy-in. The State restated its expectation that in future the online AR tool 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌35

would exclusively be used for administrative reviews. It then focused on providing technical 

assistance to staff who needed help transitioning to the new system and AR process. 

Lessons Learned

Many of the State’s challenges stemmed from communication issues with the vendor, initial bugs in the 

system, and a steep learning curve for SFA staff. To address these issues, SA staff recommended, when 

working with vendors, maintain a positive line of communication and focus on troubleshooting important 

and larger issues. Additionally, the SA found that providing consistent training and technical assistance to 

SFAs helps increase buy-in during the early stages of implementation. This sentiment was also expressed 

by SFA staff, who thought highly of the SA’s efforts to provide training and ease the transition from a

paper to an online system. SFA staff also noted how easy it was to contact a point-person at the State 

office, and expressed satisfaction for the available technical assistance. 
Abt Associates CN ART Grants | Final Study Report Final ▌36

Overview

Grant Start Year: 2010 Grant End Year: 2012

Purpose: LEA trainings to reduce Coordinated Review 

Effort error findings

Extension(s): No

Project Name: Idaho ART Vendor: [None, developed in-house]

Grant Funds Awarded: $251,655.00 Grant Funds Returned: $0

Idaho’s 2010-2012 Administrative Review and Training (ART) grant focused on improving National 

School Lunch Program (NSLP) integrity at the Local Educational Agency (LEA) by providing training to 

school food service directors, as measured by reduced error findings from the Administrative Review 

(AR) process. The ART grant enabled the State Agency (SA) to provide targeted training and technical 

support to school districts and schools to help improve the accountability of local programs and to educate 

NSLP sponsors on updated USDA nutrition standards and regulations. The training and technical 

assistance (TA) provided by the grant to the LEAs focused on the ABCs of USDA NSLP rules and 

regulations, implementing the new NSLP standards, School Meal Initiative (SMI) nutrient analysis, and 

transitioning to a Food-Based Menu Planning method. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Improve NSLP integrity at school district level through 

regional trainings and one-on-one TA. 

• Improve NSLP integrity at charter schools through annual 

training and one-on-one TA. 

• Supplementary training to Residential Child Care Institutions 

where SFAs obtain information that they need to avoid 

becoming error prone in the future. 

Goals & Objectives

• The goals and objectives for this project did not 

change from the application to the implementation 

phase. 

The following sections include information from grant applications, final grant reports, and discussions 

with SA staff members about their experiences related to the ART grant and its associated activities.

Highlights

Idaho’s 2010 grant was focused on providing regional training on the new National School Meal Program 

menu requirements and providing additional one-on-one administrative and SMI TA and oversight to 

error-prone school districts, charter schools, and Residential Child Care Institutions (RCCI).

• The SA held webinars, regional trainings, train-the-trainer sessions, and one-on one trainings on a 

variety of topics for local school nutrition directors and supervisors to prepare them to implement the 

new USDA standards and menu planning methods. 

• Regional training and webinars improved job performance of LEA staff. According to the SA, many 

of the directors and supervisors found that staff training on USDA NSLP rules and regulations 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌37

resulted in improved proficiency and skill, reducing their administrative error rates (though error rate 

reduction was not formally evaluated). 

• RCCI training prevalence increased as a result of the ART grant funding. During the first year of the 

grant, there was only one RCCI training location and 62% of the RCCIs attended. The second year, 

regional RCCI training was offered, resulting in an 83% attendance rate. Within the two years of the 

grant, 96% of all RCCIs were trained. This resulted in an improvement during the CREs. 

• The State conducted 102 one-on-one TA visits during the life of the grant. 

• Idaho has made all of the manuals, training materials, and other resources publicly available online to 

any state. 

Planning and Implementation

Management

Idaho’s state nutrition director led the grant application process with the help of a grant coordinator. After 

the grant was awarded, the grant implementation process was managed by the coordinator with the 

support of the director and other SA staff. The Idaho SA was a small office, employing 15 people during 

the life of the grant, and all staff supported the grant activities. The grant coordinator was responsible for 

viewing USDA webinars and keeping abreast of all the guidance sent to the SA office on the Healthy, 

Hunger-Free Kids Act to ensure the dissemination of current and accurate information to the State’s 

sponsors. 

Funding

Idaho was awarded a $251,655 grant, and used 100% of the allocated funds. The majority of ART 

funding was spent on conducting LEA trainings. State Administrative Expenses (SAE) funds were used 

for SA staff time spent on ART grant activities. No other supplementary funding sources were used as a 

part of this project.

Successes

Providing trainings via webinar and regional, in-person meetings increased access for the rural, at-risk 

directors and supervisors targeted by the grant and resulted in increased attendance. 

• Training and Professional Standards: The State received positive feedback from LEAs and charter 

schools on the ART trainings. The grant provided resources that enabled the SA to create more robust 

training and TA support, in addition to mandated trainings, that Idaho would not have been able to 

offer otherwise. The SA observed that TA instills confidence in the directors and supervisors who 

need assistance. The benefits of the grant activities were highly visible in regards to RCCIs. Staff 

turnover in RCCIs is high, which can lead to high error rates because transfer of information is not 

 Initial Grant Amount: $251,655.00.

 Activities Funded by the Grant: The 2010-2012 ART grant funded regional and one-on-one 

training for local school nutrition directors and supervisors, to prepare them to implement 

the new USDA standards and menu planning methods. The major grant activities included 

preparing training materials for regional, one-on-one, and online training and TA; 

developing an Idaho School Food Service Manual; and delivering the training and TA. 

 Additional Funding Sources: SAE funds were used for SA staff time spent on ART grant 

activities.

 Funds Returned: $0.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌38

good. The SA noted that continued annual RCCI-focused training is needed to maintain the 

improvements. 

o Webinar Format: More than 50 school districts signed up for any given webinar. All 

webinars are archived online and available for LEAs as needed. Webinars were organized 

into segments by topic/issue, allowing LEA staff to easily access targeted information, using 

their time efficiently. The SA noted that monthly webinars provide the ability to very quickly 

deliver the latest information on rules, regulations, and state guidance. Prior to the webinar 

implementation, the SA relied on a newsletter that took three months to plan, write, and 

distribute. 

o Regional/Rural Trainings: Regional trainings provided the opportunity for increased 

numbers of rural at-risk directors and supervisors to attend. 

Challenges

Idaho reported few challenges in its reports and interviews. There was no measured decrease in 

administrative costs or error rates reported as a result of the training and TA provided by the State. 

• Administrative Costs: The SA noted that no cost savings were observed as a result of the grant. 

• Error Rates in Application Processing: The SA was not able to describe how the ART grant 

trainings affected error rates.

• Training and Professional Standards: The SA noted that because the 2010-2012 ART grant was 

concurrent with the USDA reauthorization, there were challenges related to changing regulations, 

which created confusion and necessitated some retraining. The State recalled an instance when the SA 

conducted a monthly webinar, and before the end of that day had received new guidance from the 

regional office.

Lessons Learned

Idaho indicated that the trainings and one-on-one TA to LEAs were especially helpful for new LEA 

supervisors and supervisors with limited experience. Additionally, the SA noted that regional trainings 

and webinars were critical for increasing attendance at trainings, especially in rural states such as Idaho.
Abt Associates CN ART Grants | Final Study Report Final ▌39

Overview

Grant Start Year: 2011 Grant End Year: 2014

Purpose: Technology improvements to address 

administrative errors through the use of targeted 

monitoring, and increased training in error-prone LEAs

Extension(s): Yes

Project Name: Administrative Review Tool Vendor: Outside, for-profit software vendor 

Grant Funds Awarded: $1,017,920.00 Grant Funds Returned: $45,901.65 

In 2011, Indiana received an Administrative Review and Training (ART) grant to develop a tool to 

automate the administrative review (AR) process in accordance with the revisions to the process detailed 

in the Healthy, Hunger-Free Kids Act of 2010. The State designed the Administrative Review Tool to 

help School Food Authorities (SFAs) with data reporting and compliance problems, to identify fraudulent 

claiming patterns, and to automate both the review and corrective action processes. The Administrative

Review Tool streamlines communication between State field specialists and SFAs by providing a 

platform through which field specialists can schedule on-site review dates and assign tasks to SFA staff 

before on-site review. It also generates automated findings reports after on-site reviews. 

Prior to the changes implemented under the ART grant, the AR process was paper based and conducted 

primarily on-site. After on-site reviews, field specialists transferred their findings to an online system, 

answered remaining questions, and wrote final letters, which included corrective and fiscal actions. The 

old process took approximately a day to complete for each field specialist. The new system allows the 

SFAs to answer questions prior to on-site reviews. Field specialists use the Administrative Review Tool 

to enter information gathered during the AR, either on-site or once they return to their office. Reports are 

automatically generated based on findings. The tool tracks progress, turning green when questions are 

completed. 

Additionally, Indiana used grant funding to train SFAs on the new Administrative Review Tool. During 

the in-person trainings, the software developers demonstrated how to use the tool to complete each step of 

the administrative review. Field specialists also provide refresher training at the beginning of the year for 

SFAs participating in ARs during the current year. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Identify SFA organizations with data reporting and compliance problems 

and potential fraudulent claiming patterns.

• Automate Coordinated Review Effort (became “Administrative Review” 

after award) and workflow tasks for the corrective action process. 

• Leverage the existing CNPweb administrative application's capabilities 

and central Sponsor and Claims data repository. 

• Improve SFA staff understanding of program operations and 

Goals & Objectives

• The goals and objectives did not change 

from the grant application to 

implementation phase. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌40

Grant Application Post-Grant Implementation

responsibilities. 

• Improve program reporting.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Indiana’s 2011 ART grant was awarded to develop an automated tool for ARs, identify fraudulent 

claiming patterns, and automate workflow tasks for the corrective action process. The activities funded by 

the ART grant achieved the following: 

• The State developed an online Administrative Review Tool, which integrated the AR process into the 

existing CNPweb administrative systems central database. The automation streamlined workflow 

tasks and corrective action processes. 

• The Administrative Review Tool allows access to SFA application data, site information, claimed 

meal counts, and claim payments. 

• The State anticipates that the Administrative Review Tool will streamline the AR process, reduce 

errors, and improve the quality and timelines of reviews. 

• The tool creates a single, organized platform for communication between State field staff and SFAs, 

simplifying the communication process and eliminating the need for continuous email communication 

between the State and SFAs. 

Planning and Implementation

Management

Over the course of this grant, Indiana’s project team included the SA director, a project specialist, a 

project coordinator, and a school monitoring coordinator. They worked with a team of developers from 

their software vendor to plan and implement this initiative. 

Funding

Indiana was awarded $1,017,920 to complete its 2011 ART grant, and it used 95% of the total award. The 

grant project was put on hold until FNS issued the final AR guidelines, allowed by a no-cost extension.

 Initial Grant Amount: $1,017,920.00.

 Activities Funded by the Grant: Funds were used to develop a new Administrative Review 

Tool that was integrated into the SA’s online database system.

 Additional Funding Sources: State Administrative Expense funds were used to fund 

additional components, such as Summer Food Service Program and the Child and Adult Care 

Food Program modules within the same platform.

 Funds Returned: $45,901.65.

Successes

The grant achieved the goals of creating an automated tool for ARs built upon an existing platform and 

improving program reporting. The tool, in conjunction with new AR requirements, resulted in more 

thorough but less time-consuming audits. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌41

• Administrative Costs: Though there has not been a significant change in State staff time spent on 

administrative review as a result of the Administrative Review Tool, it is difficult to disentangle any 

increased efficiency from the increased requirements related to FNS’s new AR requirements. State 

staff spend less time on-site, which is less disruptive for SFAs. Post-visit activities have also been 

significantly reduced for State staff. SFA staff spend less time organizing their files after on-site 

reviews. 

• Error Rates in Application Processing: Use of the Administrative Review Tool has resulted in 

more accurate reviews. Error rates have been reduced, as the tool accesses data from other systems, 

eliminating errors due to manual data entry. Embedded checks alert SFAs to correct or confirm items 

prior to the on-site review. With review data stored in the CNPweb database, the Indiana Department 

of Education can apply various analytical methods and tools to identify potential problem areas. 

• Training and Professional Standards: State field staff have expedited the review process, thereby 

freeing up time to provide technical assistance to SFAs. 

• Data Systems: The automation of the AR process using the Administrative Review Tool was 

successful in streamlining data collection, improving data quality. 

• Administration of School Nutrition Programs: Though the Administrative Review Tool is not 

intended to help with the local administration of school nutrition programs, it may help SFA staff 

understand the purpose of ongoing data collection. 

Challenges

Indiana experienced significant delays implementing the Administrative Review Tool while waiting for 

new AR guidelines from FNS. 

• Training and Professional Standards: SFA training about the new AR process also addressed the 

Administrative Review Tool. However, SFA staff would benefit from additional training on the tool, 

including a visual step-by-step guide, particularly if training occurs well in advance of the AR. 

• Data Systems: Varying degrees of technical skills among SFA staff affected comfort with the 

Administrative Review Tool. SFAs indicated additional response options are needed for some 

questions. 

• New Guidelines: Indiana experienced delays in system development and implementation to address 

new AR guidelines received from FNS during the process. 

Lessons Learned

If Indiana were to apply for another grant, it would ensure the timeline accounted for the rollout of any 

new FNS policies and procedures and would allow more time for development of any online tools. The 

State noted it could have developed sections of the Administrative Review Tool that were not dependent 

on the new FNS forms while waiting for additional information from FNS. Budgeting for a full-time 

project manager would have been beneficial in keeping the plan on schedule and avoiding delays. 
Abt Associates CN ART Grants | Final Study Report Final ▌42

Overview

Grant Start Year: 2009 Grant End Year: 2015

Purpose: Update technology systems Extension(s): Yes

Project Name: Iowa ART Vendor: Outside, for-profit training provider; Outside, forprofit software vendor

Grant Funds Awarded: $1,700,000.00 Reported Grant Funds Returned: $0

In 2009, Iowa received an Administrative Review and Training (ART) grant to develop and implement an 

integrated process for accessing and reviewing Local Educational Agency (LEA) data; a web-based 

reference library; an integrated process for the State Agency (SA) to access and use financial data 

submitted by LEAs; and a USDA Foods Ordering System. Prior to the grant, many of the systems and 

tools used by the SA were outdated and did not allow the SA to directly interact with LEA data and 

systems. The new applications and tools allow SA staff to quickly identify potential issues and errors and 

address them effectively and efficiently.

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Administrative Oversight.

o New tools/reports.

o Enhanced system functionality to monitor 

SFAs.

• Reference Library and Tutorials.

o Easily accessible and searchable repository of 

information for use by staff of SFAs 

participating in the NSLP/SBP.

o Develop 40 web-based tutorials.

• Financial Management

o Tools/reports to monitor SFAs for program 

viability.

o Support SA in providing TA for high-risk SFAs.

• Nutrient Analysis

o Monitor SMI data and SFA compliance.

Goals & Objectives

• Administrative Oversight.

o New tools/reports.

o Enhanced system functionality to monitor 

SFAs.

• Reference Library and Tutorials.

o Easily accessible and searchable repository of 

information for use by staff of SFAs 

participating in the NSLP/SBP.

o Develop 40 web-based tutorials.

• Financial Management. 

o Tools/reports to monitor SFAs for program 

viability.

o Support SA in providing TA for high-risk SFAs.

• Iowa USDA Foods System. 

o Replace Commodities application with new 

USDA Foods System.

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌43

Highlights

Iowa’s 2009 ART grant was awarded so it could update its technology systems, design online training 

tutorials, improve its financial management and oversight, and implement a USDA Foods System. The 

activities funded by the ART grant achieved the following:

• Sole-source procurement was used to implement a variety of new reports and business rules for the 

review and prevention of errors. Reports use verification collection data, which can be filtered for 

common errors and reviewed by the SA for any needed corrective action. 

• A training provider was contracted to develop 17 web-based, interactive tutorials. Each tutorial 

contains five parts: (1) online tutorial content; (2) accompanying text document; (3) content quiz; (4) 

participant survey; and (5) completion certificate. 

• The SA developed a comprehensive financial report generated from data uploaded from each 

district’s accounting report, meal claims, and the Iowa USDA Foods System.

• The antiquated Commodities application was replaced with the new Iowa USDA Foods System 

application. 

Planning and Implementation

Management

The Iowa project experienced delays early in the implementation process due to staff turnover with its SA 

bureau chief and the original project manager, resulting in an approximately two-year delay. The original 

grant manager came onboard as a full-time project manager for the effort and oversaw the project 

schedule, budget, and implementation. The meal nutrient requirements were changed, so the SA requested 

to repurpose the money for Objective #4 “Nutrient Analysis” with “Iowa USDA Foods System.” The 

project manager spearheaded all of the tasks for the grant and worked with contractors, consultants, or 

other SA staff (primarily a data analyst and a business analyst), as needed. 

Funding

Iowa was awarded $1,700,000 to complete its 2009 ART grant, and used 100% of the total award. 

 Initial Grant Amount: $1,700,000.00.

 Activities Funded by the Grant: A multitude of new reports were created, allowing SA 

consultants to quickly identify errors and contact LEAs for corrections. The web-based 

tutorials and reference library provide unlimited access to training to improve compliance 

with Special Milk Program (SMP) standards. The Comprehensive Financial Report allows 

LEAs to review and evaluate the management of SMP funds, specifically with USDA Foods

System and the Fresh Fruit and Vegetable Program. The real-time, comprehensive budget 

allows LEAs unlimited access to budget data, thereby reducing the risk of administrative 

error and increasing the use of USDA Foods System funds. 

 Additional Funding Sources: To complete the USDA Foods System application, the SA used 

State Administrative Expense funds.

 Funds Returned: $0.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌44

Successes

The grant provided the SA with the opportunity to create new reporting systems, streamline a number of 

administrative and financial processes, create a comprehensive training tutorials and reference material 

portal, and implement an efficient and accessible USDA Foods System.

• Administrative Oversight: The design and implementation of new administrative tools and 

processes resulted in 5% or less incorrectly reported direct certification students and decreased the 

time needed to identify errors by half.

• Training and Professional Standards: More than 80% of the SMP staff met professional standards 

the year following the implementation of the training tutorials and reference library portal. 

• Financial Management: The creation and implementation of the new financial report allows LEAs 

to enter data that then are integrated from the SA Certified Annual Report into the Comprehensive 

Financial Report (IowaCNP). 

• Administration of School Nutrition Programs: The Iowa USDA Foods Systems significantly 

reduces staff time. Prior to the system implementation, 586 hours of staff time were spent on tasks 

related to ordering, distribution, and tracking. The new system has reduced the staff time needed to 84 

hours. 

Challenges

Iowa achieved the goals and objectives for the ART project; however, it required two no-cost extensions 

and it experienced the following challenges with the project:

• Administration: The most critical factor contributing to the challenges of the Iowa ART project was 

hiring a full-time project manager. Essentially no progress was made until a project manager was 

onboard. 

• Training and Professional Standards: The tutorial design began without a full understanding of 

508 compliance requirements. The SA had to work with the vendor to meet compliance standards. As 

a result, the task was delayed and cost significantly more than anticipated. Fewer than half of the 

originally proposed training tutorials were produced due to the oversight. 

• Administration of School Nutrition Programs: Several components of the Iowa USDA Foods 

System were implemented as part of the ART grant, but ultimately the SA used State Administrative 

Expense funding to complete the entire transition of the system for in-house application maintenance.

Lessons Learned

Iowa chose to prioritize the tutorials and began designing those that were of highest need due to staff skill 

deficiencies. However, the highest priority topics tended to be the most complex, which increased the 

learning curve as the SA worked with the vendor on the design and tutorial content. The SA stated it 

would have been more efficient to have started with smaller, simpler tutorials so a common, mutually 

agreed upon design and approach could be identified. 

It was important for the project manager to communicate early and often with the project team and 

vendors. Frequent and open communication promoted progress and addressed risks before they became 

issues. Informal communication was helpful with day-to-day tasks and keeping work groups updated and 

productive. 

All procurement options were considered to select the most appropriate fit for the work. Tutorial 

development was a large project and required a team of experts to produce high-quality e-learning 

content. Rather than contract several individual consultants, a Request for Proposals for the project was 

prepared and the SA contracted with a media company. Doing so made oversight more efficient and 

reduced costs. 
Abt Associates CN ART Grants | Final Study Report Final ▌45

Overview

Grant Start Year: 2013 Grant End Year: 2017

Purpose: Technology improvements and LEA training to 

increase efficiency and decrease errors in AR process 

Extension(s): Yes

Project Name: KN-CLAIM v2 Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $1,263,223.00 Grant Funds Returned: $0

In 2013, the Kansas State Department of Education (KSDE) received an Administrative Review and 

Training (ART) grant to replace its obsolete systems with an integrated online solution, including a fully 

automated e-review system for administrative reviews. The State was motivated to implement this new 

system because of a major security incident with its obsolete information and claiming system. In an 

effort to improve security across all Child Nutrition & Wellness programs, the grant-funded project 

(Kansas Nutrition–Claims and Information Management Version 2, or KN-CLAIM v2) integrated all 

programs’ claiming and logging into one updated system. The new system was developed to both 

industry and KSDE security standards, which included reauthorizing users, in an effort to reduce the 

probability of future security incidents. 

Preceding the changes implemented under the ART grant, administrative review involved completing 

forms provided by USDA in spreadsheets and communicating with the Local Educational Agencies 

(LEA) via email before and after the on-site review. The new system is an e-review, and LEAs and State 

Child Nutrition consultants are able to access an online system to complete required forms. Some 

information is completed outside of the e-review and then uploaded by the LEAs for review by Child 

Nutrition consultants. The e-review generates a findings report and calculates fiscal action, reducing the 

labor burden for the Child Nutrition consultants following the on-site portion of the review. 

To reduce the risk of administrative errors and improve the administration and integrity of school food 

service programs, the State created trainings on program requirements and professional standards. These 

include video tutorials that can be accessed by LEAs 24/7 to provide technical assistance, hands-on 

technology training, and an eight-week English as a Second Language (ESL) course for LEAs. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Reduce the risk of administrative error by replacing the 

obsolete KN-CLAIM and CNP Logging systems with an 

integrated online solution.

o Implement a system that utilizes current technology 

and security protocols.

o Increase customer service and accuracy by 

simplifying data entry and reducing the number of 

times the same data is entered.

o Implement enhanced logging and reporting 

Goals & Objectives

• The goals and objectives did not change from the 

grant application to implementation phase. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌46

Grant Application Post-Grant Implementation

capabilities.

• To implement comprehensive administrative review 

capabilities to identify at-risk LEAs and comply with the new 

administrative review process.

o Enhance reporting and monitoring capabilities to 

identify error-prone LEAs in real-time.

o Develop online functionality to support new 

requirements for administrative reviews.

o Empower LEAs to identify and reduce errors before 

data, claims, and/or information is submitted.

• To improve the integrity of school food service programs and 

administration, and to reduce administrative error through 

training and technical assistance.

o Improve counting and claiming accuracy. 

o Provide training opportunities to develop basic 

computer literacy and technology skills.

o Offer a series of online training video tutorials.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Kansas’s 2013 ART grant enabled the SA to update and integrate its technology systems in a secure 

platform, streamline the administrative review process, and provide training to LEAs. The activities 

funded by the ART grant achieved the following:

• Replaced the obsolete KN-CLAIM system and its Child Nutrition Programs (CNP) Logging system 

with an integrated online solution (KN-CLAIM v2).

• Implemented enhanced logging and reporting capabilities through creation of an e-review for the 

administrative review that: 

o Includes interactive communications between State Child Nutrition consultants and LEAs, 

reducing the number of emails exchanged between reviewers and LEAs. 

o Calculates fiscal action and findings reports, reducing State staff burden.

• Created online training modules that can be accessed 24/7 to provide assistance to LEAs that would 

be using KN-CLAIM outside of normal business hours. 

• Offered an eight-week English as a Second Language class to decrease errors due to language 

proficiency. 

• Documented business rules and created procedures for updating KN-CLAIM. 

Planning and Implementation

Management

The grant was managed by two project managers, one from the Kansas Department of Education’s IT 

team and the other from the Child Nutrition & Wellness team. The Kansas Information Technology 

Office (KITO) oversaw the grant. The project managers sought input from the Child Nutrition & 

Wellness Advisory Council made up of LEAs. 

Funding

Kansas was awarded $1,263,223 to implement its 2013 ART grant, and used 100% of the total award. 

The majority of the ART grant was used to purchase a server to support the new system, to purchase 

mobile devices for Child Nutrition consultants to use while completing reviews in the field, and to fund 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌47

staff salaries. A significant portion of the grant was spent on the software vendor to support rewriting the 

claiming and logging system. Kansas supplemented the cost of grant activities with State Administrative 

Expense funds and a direct certification outstanding performance award. 

 Initial Grant Amount: $1,263,223.00.

 Activities Funded by the Grant: The majority of the funding went towards technical 

improvements to the State claiming and logging system, as well as integrating the new 

administrative review system with the existing system. Funds also supported salaries for the 

State child nutrition director, IT director and other IT support within the SA, and covered

travels cost to attend mandatory USDA meetings. The SA also invested in a new server to 

support the administrative review system, as well as Windows 8 mobile devices that 

consultants use for the automated review. Funds were also used for training materials and 

indirect costs. 

 Additional Funding Sources: State Administrative Expense funds were used to support staff 

time spent on program testing. Additionally, Kansas used a direct certification outstanding 

performance award to pay for unanticipated KITO fees. 

 Funds Returned: $0.

Successes

The grant was successful in streamlining the administrative review process and reducing the time spent by 

staff on completing the reviews. This was accomplished by updating the CNP logging system, and 

integrating it with an e-review for administrative reviews. The grant also successfully created eight online 

trainings and had more than 14,000 training registrations. 

• Administrative Costs: The integrated review process saves an estimated eight hours per review. KNCLAIM v2 also reduces burden for LEA staff, who can upload forms directly into the system and see 

the fiscal action as the review progresses. Administrative review summaries are automatically 

generated, saving the State 279 hours per program year. 

• Error Rates in Application Processing: Though difficult to measure, the State noted that there has 

been a reduction in human error as a result of integrating the administrative review process into the 

online KN-CLAIM system.

• Training and Professional Standards: The State had more than 14,000 training registrations, and 

LEAs indicated that the trainings were effective. The State has developed eight online training 

modules and continues to develop more.

• Data Systems: The ART grant integrated all of the CNP reporting in one place, allowing LEAs to 

monitor multiple programs in one system. This information is easily shared with the superintendent or 

Board of Education. 

• Streamlined Reporting: The system automates letters, corrections, and fiscal action calculations, 

reducing the time consultants spend on reviews (by eight hours per review).

• Streamlined Communication between State and LEAs: The project has resulted in improved 

communication and clarity. The dashboard, for example, allows LEAs to monitor their progress and 

communicate with State consultants throughout the process. 

• Transparency: The State can analyze and pull data from the system and post on a public site. 

• Project Management: Having two project managers, one representing Child Nutrition & Wellness 

and the other representing the Department of Education’s IT team, allowed for successful 

implementation of grant activities. Each project manager took ownership of specific activities, and 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌48

having two individuals invested in the project timeline was helpful when the IT project manager left 

and a new one was hired. 

Challenges

The State and LEAs were pleased with the grant processes and the final product and had limited 

challenges throughout the implementation of the grant. The challenges revolved around contracting with a 

third-party vendor. 

• Vendor Challenges: Because a limited number of vendors can build these systems, the State was 

competing with other States for the vendor’s time and attention. 

• Project Management: The vendor set an aggressive deliverables schedule that was difficult to 

maintain and therefore caused some delays that left LEAs waiting, in anticipation of the release. 

Lessons Learned

The challenges above highlight the need to set a realistic deliverables schedule, especially when working 

with a software vendor. Some of the delays were due to the desire to continue testing; however, the State 

noted that when implementing technology solutions, there will always be potential improvements. 

Therefore, it is important to set a release date and remain firm in going live on that date, if at all possible. 

After the grant award, the decision was made to produce the online trainings in-house. Though this 

reduced costs, the State found this process time-consuming and realized that timelines should be adjusted 

when making these decisions. Finally, the State learned that trainings should be conducted in computer 

labs with at least one Child Nutrition consultant present, to account for trainees’ varying levels of 

computer literacy. 

Kansas is interested in applying for other ART grants. However, it notes that it may be more beneficial to 

apply for a grant such as the Child Nutrition Technology Innovation Grant, which would allow it to use 

funding for all Child Nutrition Programs within the KN-CLAIM v2 system. 
Abt Associates CN ART Grants | Final Study Report Final ▌49

Overview

Grant Start Year: 2012 and 2014* Grant End Year: 2014 (2012 Grant) and 2017 (2014 Grant)

Purpose: Expand the Massachusetts School Meals 

Accountability and Responsibility Training Tools (Mass 

SMARTTs) to enhance the administration and quality of 

school meals and minimize administrative errors

Extension(s): 2012: Yes ; 2014: No

Project Name: School Meals Accountability and 

Responsibility Training Tools: Making It Count and The 

Massachusetts Training, Evaluation, and Research 

Initiative (MassTERI) 

Vendor/Partner: Outside, for-profit software vendor; 

University or Community College training developer

Grant Funds Awarded: $1,243,647.88 (2012) and

$1,495,840.00 (2014)

Grant Funds Returned: $9,894.19 (2012) and $4,220.80 

(2014)

* Massachusetts received three ART grants, in 2010, 2012, and 2014. This profile focuses on the 2012 and 2014 grants.

The primary goal of both the 2012 and 2014 Administrative Review and Training (ART) grants was to 

expand the Massachusetts School Meals Accountability and Responsibility Training Tools (SMARTTs) 

website to enhance the administration and quality of school meals by improving school nutrition 

personnel practices, thereby minimizing administrative errors. The State also developed technology and 

self-assessment tools to facilitate accurate monitoring and reporting and to improve tracking and analysis 

at site and district levels. The 2014 ART grant built and expanded the work completed during the 2012 

ART grant. 

Results from a previous Massachusetts grant project revealed the critical need to provide relevant, 

learner-directed education to school nutrition personnel at all levels to improve their skills and increase 

their confidence in using the technology needed to implement USDA requirements for meal quality and 

accountability. In response, the State used the 2012 and 2014 ART grants to enhance and expand 

SMARTTs and offer greater training opportunities to school nutrition personnel. First, Massachusetts 

enhanced collaboration among local and state agencies and organizations by expanding the SMARTTs 

advisory group membership to ensure it represented a diverse set of stakeholders and to improve the 

leadership competencies of school nutrition managers and directors. The advisory group pilot tested 

technology improvements and contributed to design and development of the SMARTTs modules. These 

Making It Count instructional modules provide interactive trainings and deliver user support to reduce 

administrative errors and improve the nutritional quality of school meals statewide. The modules focus on 

topics including nutrition standards, meal patterns, and meal benefit eligibility. 

The grants also funded the MassTERI initiative, which provided foundation skills courses to school-level 

food services staff. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌50

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives – 2012 Grant

• Enhance the integrity and quality of school meals by improving 

personnel practices at all levels of school nutrition operations.

• To substantially reduce administrative errors through the 

improvement of meal eligibility and reimbursement system 

interface to increase quality assurance of incoming LEA 

application and data reports.

• Improve the personal and professional competencies of all school 

nutrition personnel to address the quality of meals served, the 

accuracy of the counting and claiming of meals, and eradicating 

errors.

Goals & Objectives – 2014 Grant

• Improve nutritional quality of school meals through statewide 

compliance with the USDA meal patterns.

• Modify Making It Count into useful group continuing education 

trainings. 

• Improve accurate and effective delivery of training and education 

to school nutrition personnel.

• Educate school nutrition administrators on new School Food 

Agency Second Review of Applications.

• Develop tools that allow: (1) State staff to analyze AR data and 

identify district-level problem areas requiring additional support; 

and (2) District staff to support themselves in identifying potential 

claim errors.

Goals & Objectives – 2012 Grant

• The goals and objectives did not change from 

the grant application. 

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Massachusetts used the 2012 and 2014 grants to fund three major initiatives: The Making It Count 

training module and website development, MassTERI skill-building workshops for local nutrition staff, 

and improved claim analysis reporting. The major accomplishments of Massachusetts’s 2012 and 2014 

ART grants included the following:

• Development of the Making it Count videos and website to educate school nutrition personnel on a 

variety of subjects. Modules include training on meal pattern requirements, identifying reimbursable 

meals, menu planning food production records, portioning and serving meals, developing meal 

counting and point-of-sale systems, consolidation and reporting, making the FP9 count, and meal 

benefit issuance and reporting. This website is public and available to all States and districts across 

the United States.

• Development of analysis and reporting tools, which issue warnings and errors based on edit checks. 

This system allowed the SA to analyze data trends and provide better support to its LEAs. 

• Train-the-trainer workshops designed to increase the number of qualified school nutrition trainers in 

Massachusetts to provide trainings that meet USDA’s professional standards.

• Establishment of a diverse advisory group (comprising large and small districts, urban and rural 

districts, and charter schools) that provided guidance on the SMARTTs design and development. 

• Creation of MassTERI (Massachusetts Training and Evaluation Research Initiative) in collaboration 

with the University or Community College training developer. These skill-building workshops 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌51

provided staff with the context for understanding the State’s procedures and policies. They also 

provided an opportunity for staff who may have never had a formal educational experience or 

professional development to participate in workshops such as introduction to computers, kitchen 

math, English as a Second Language, and communication to district-level food service staff. The 

State received a lot of positive testimonials from participants. 

Planning and Implementation

Management

The executive director at the Department of Elementary and Secondary Education (ESE) and the School 

Meals program coordinator were directly involved in preparing the grant applications. Many SA staff 

contributed to the content and script development for the Making It Count initiative. Massachusetts 

engaged a project coordinator to lead and manage grant activities and relied on a financial management 

supervisor for contract management support. Additionally, the SA benefitted from the SMARTTs 

Advisory Board members, who met several times each year to contribute to the project. 

Massachusetts contracted with a training developer to develop and produce the Making It Count videos. 

The training developer subcontracted with a web designer and film producer to carry out the work. The 

SA explained that the State had a successful history of working with the training developer, which had 

proven itself able to execute high-quality products within budget and in a timely manner. So the State was 

able to minimize project risk by partnering with it again.

The SA also contracted with software vendor to update its administrative review software for claim 

analysis reports. 

Funding

Massachusetts was awarded a $1,243,647 grant in 2012 and used 99% of the allocated funds. It received a 

no-cost extension in 2012 to accommodate regulatory changes within these projects, to conduct an 

additional round of MassTERI Foundation Skills workshops, and to complete production of the Making It 

Count Breakfast Modules, with transition of the website into Spanish. Counting the extension, 

Massachusetts accomplished all grant objectives within the award period. In 2014, Massachusetts was 

awarded $1,495,840 and used more than 99% of the allocated funds. 

 Initial Grant Amount: $1,243,647.88 (2012); $1,495,840.00 (2014).

 Activities Funded by the Grants: The majority of the funding was used to create the Making 

It Count training modules and website, as well as the MassTERI training program through a 

subcontract with training developer. Massachusetts also subcontracted with software 

vendor for the development of claim analysis software module updates. 

Specifically, the 2012 grant funded Advisory Group meetings, promotion of the 

Making It Count modules at national and statewide conferences and seminars, claim 

analysis software updates, data analysis for trends and meal claiming anomalies, 

merged claim analysis reports into the administrative review operational plan with 

ESE school lunch and review coordinators, MassTERI workshops and evaluations, and 

Making It Count module development with video and audio components and website. 

The 2014 grant funded the expansion of the Making It Count training modules. 

Massachusetts modified the website to be more user friendly as well as mobile and 

tablet accessible. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌52

Additionally, the grant funded the development of a seven-part food production 

records/weights and measure module, the creation of self-study guides for each 

module, the addition of interactive site-based activities to complement online 

modules, integration of USDA Professional Standards requirements, and increased 

accessibility through a Spanish version of the website and closed captioning in 

Portuguese, Haitian Creole, and Cantonese. The 2014 grant was also used to fund 

train-the-trainers workshops to increase the number of qualified school nutrition 

trainers in Massachusetts.

 Additional Funding Sources: No additional funding sources were used for the projects. The 

web server used for Mass SMARTTs is provided in kind by the training developer. 

 Funds Returned: $9,894.19 (2012); $4,220.80 (2014).

Successes

• Error Rates in Application Processing: The claim analysis reports developed with the software 

vendor allow the SA to identify meal claiming anomalies, such as two schools with identical meal 

counts or large increases from one month to another for one particular meal. It notifies LEAs of errors 

and allows them to correct before submitting claims. It also allows the State to review all LEA actions 

in the system and contact a LEA regarding any irregular patterns in the data. The claim analysis 

reports help the SA identify districts that may need technical assistance regarding meal claiming and 

Community Eligibility Provision participation. The updated reports provide LEAs with more directed 

instruction on how to address the identified errors. 

o Additionally, the SA assigned the Making It Count modules as part of the corrective action 

process. Having a centralized and comprehensive set of training tools provides districts the 

resources to address and limit errors. 

• Training and Professional Standards: With the update and new additions to the Making It Count 

website, Massachusetts is now able to offer greater training opportunities to school nutrition 

personnel across the United States. Statewide, there had been an increased demand for training 

surrounding production records and weights and measures. Though Massachusetts provides a 

production records training in-person once a year, the new seven-part online module will allow school 

nutrition personnel to receive this training anytime and anywhere there is internet access. Modules 

and interactive activities were also modified to be tablet and smart phone accessible so that the 

trainings can be completed on devices that many school nutrition personnel readily have available. 

The addition of self-study and group-study guides has also provided school nutrition directors and 

managers with the confidence to conduct trainings themselves to their staff. LEA respondents 

commented on the benefits of having a vetted training tool from the SA that reduced the time spent 

preparing for district trainings and allowed them to onboard new staff efficiently. 

o Although most Massachusetts school nutrition directors and managers are fully 

knowledgeable of the National School Lunch Program, many shared concern over their 

ability to teach their skills to their staff. Especially with the onset of USDA’s professional 

standards, there was growing anxiety over providing effective and efficient training by those 

who have had no experience as a teacher. Through the development of the train-the-trainer 

workshop, Massachusetts has seen an increase in confidence and training hours provided by 

participants. Many of these participants, most of whom have never completed a presentation 

before, have since produced trainings and poster sessions within their schools and districts 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌53

and at Massachusetts and national SNA conferences to assist school nutrition personnel in 

meeting USDA’s professional standard requirements.

• Communication: The SA employed a multi-pronged approach to advertising the Making It Count 

resource. LEAs noted that the SA successfully communicated the availability of the resource via 

email, conferences, and handouts featuring QR codes. 

Challenges

• Competing Priorities: The grant activities occurred on the same timeline that the SA was developing 

new administrative review processes and software (not funded with the ART grant), so Massachusetts 

experienced some difficulty prioritizing staff time and attention, given the multiple projects. 

• Film Production Challenges: The SA experienced challenges related to the process of producing 

multiple instructional videos. Producing the scripts for each module required a lot of effort by SA 

staff to ensure that all of the information was accurate and clear. Additionally, filming in schools 

presented the challenge of obtaining student releases. 

Lessons Learned

Though the MassTERI project received positive feedback from participants, the SA learned that the 

endeavor was very expensive and not easy to replicate statewide. Once the grant period ended, the 

program was not sustainable. 

The SA benefited from partnering with a local university. Not only did Massachusetts benefit from the 

expertise of the training developer and labor management workplace education program, but it also 

received assistance with data collection and reporting. 
Abt Associates CN ART Grants | Final Study Report Final ▌54

Overview

Grant Start Year: 2013 Grant End Year: 2017

Purpose: Technology improvements to AR process Extension(s): Yes

Project Name: Michigan Administrative Review System 

(MARS)

Vendor: [None, developed in-house]

Grant Funds Awarded: $966,618 Grant Funds Returned: $0

In 2013, Michigan received an Administrative Review and Training (ART) grant to improve its 

technological infrastructure in order to increase the efficiency of data collection, reduce processing time, 

and reduce internal errors related to the administrative review (AR) process. The grant was also used to 

provide technical assistance and training to high-risk local educational agencies (LEA). The training 

included creating 36 online training modules and hosting regional trainings and targeted trainings for 

error-prone LEAs. 

Prior to this intervention, Michigan’s Administrative Reviews were conducted manually, and all 

documents were exchanged by mail, email, or fax. The on-site review was done by hand, and analysts 

brought their handwritten notes back to State managers who would enter everything in a spreadsheet. 

Corrective action and findings reports were written on a carbon copy, which was given to LEAs on-site. 

LEAs had six months to mail or email responses back to the State. With the introduction of this new 

system, the Michigan Administrative Review System (MARS), all tracking and notification is done 

electronically. Using MARS, LEA staff download blank forms and upload completed forms. State 

analysts use tablets to access information in MARS during on-site reviews. Once reviews are complete, 

MARS creates findings packages. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Improve identification of high-risk/error-prone LEAs, and target 

them for training, technical assistance, follow-up, and other 

corrective action. 

• Improve State of Michigan SNP technology to maximize 

efficiency in the areas of recordkeeping, reviews, tracking LEA

activity, and training while maintaining a secure system.

• Utilize programmatic, review, analysis, and recordkeeping 

systems that will allow for information-sharing and real-time, 

fluid assessment of various measures including (1) number and 

status of high-risk/error-prone LEAs by subcategory; (2) 

success of corrective action; (3) training efforts; (4) technical 

assistance; (5) error rates in administrative review process; (6) 

time needed to complete and process reviews. 

Goals & Objectives

• The goals and objectives did not change from 

the grant application to the implementation 

phase. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌55

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and LEA staff members about their experiences related to the ART grant and its 

associated activities.

Highlights

Michigan’s 2013 ART grant was awarded to update the State’s technology systems and decrease 

processing time for data collection, individual reviews, and summary reports. In order to accomplish this, 

the State built the Child Nutrition Program component (MARS), of the Department of Education’s Grant 

Electronic Monitoring System (GEMS), which was also being updated. Grant funds were also used to 

identify and train high-risk or error-prone LEAs. The activities funded by the ART grant achieved the 

following: 

• The State implemented an electronic, online monitoring system (MARS) allowing it to automate the

AR process. MARS also allows the State to create real-time customized reports, track the status of

corrective actions, and document technical assistance efforts.

• With the development of MARS, the State reduced time spent compiling data and completing reports,

reduced processing time, enhanced error checks, and improved “cross-talk” between data sets.

• The State provided technical assistance to high-risk/error-prone LEAs through the development of 36

online training modules, as well as in-person regional and on-site trainings.

Planning and Implementation

Management

The grant was implemented through the Office of Health and Nutrition Services. A project manager 

oversaw the effort, working with a consultant manager. A technological liaison and developers from the 

State Department of Technology Management and Budget built the system in-house. 

Funding

Michigan was awarded $966,618 to complete its 2013 ART grant, and used 100% of the total award. 

There were no other supplementary funds used during the grant period, 2013-2017, though the State did 

need to reallocate its awarded funds midway through the project from its original budget.

Successes

The grant achieved the goal of building out the Child Nutrition Program component of GEMS. The grant 

helped streamline and automate AR processes and increased efficiency of communication between the 

State and the LEAs. 

• Administrative Costs: There has been a reduction in time spent on ARs by the State staff due to

activities funded by the 2013 ART grant, allowing the State to provide more technical assistance to

LEAs.

 Initial Grant Amount: $966,618.

 Activities Funded by the Grant: Funds were used for staffing, travel for regional error-prone

trainings, technical assistance visits, supplies, attendance at the Child Nutrition Access and

Accountability through Technology (CNAAT) conference, and indirect costs.

 Additional Funding Sources: Following the completion of the grant period, State

Administrative Expense funds were used to sustain the new system.

 Funds Returned: $0.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌56

• Training and Professional Standards: The State developed 36 online training modules, which can 

be updated as policies change. SA staff refer LEAs to these modules during the corrective action 

process. From the State’s perspective, the identification of error trends helps with identifying needed 

training for the LEAs.

• Data Systems: The automation, organization, and streamlining of data collection through MARS led 

to better quality data. The new system helps LEAs prepare for AR components in advance of on-site 

visits, and it eliminates the amount of paper that the LEAs must prepare and share with the State. 

Challenges

The grant-funded interventions were effective at improving the AR process, primarily for the State. 

However, a no cost-extension was required due to staff turnover that caused delays in the implementation 

of the grant activities. Though the grant was used to develop online training modules, LEAs expressed a 

need for additional hands-on trainings on using GEMS/MARS. 

• Training and Professional Standards: Though the training modules covered topics related to AR, 

LEAs indicated a desire for more hands-on training related to using the MARS system. Additionally, 

developing content for 36 online training modules was labor intensive. 

• Staff Turnover: There was repeated project manager turnover and subsequent delays. This was 

further complicated by the lack of a project management contingency plan.

• Data Systems: Though costs were saved by using the State Department of Technology Management 

and Budget to build the system in-house, the project had to compete for the limited State resources 

(i.e., developers and other related staff). There are still components of the findings report that are not 

captured in MARS, and State staff have identified “work-arounds” for entering these findings. 

• Administration of School Nutrition Programs: Though not directly attributed to MARS, LEAs 

responsible for multiple districts indicated they spend a significant amount of time on AR-related 

activities during the review cycle. Additionally, LEAs indicated preparing and uploading materials 

into the system is time-consuming. As noted above, MARS is an effective repository for information 

at the State level. It is not, however, designed for LEA program operations or management. 

• Policy and Program Requirements: The convergence of new AR policies and requirements and a 

new system was overwhelming for staff responsible for grant-related activities. 

Lessons Learned

MARS is an effective tool for the State to conduct ARs and identify error-prone LEAs in need of 

technical assistance, but it is intended to help LEAs manage their programs. Developing a strong project 

management plan would have helped minimize delays due to staff turnover. Establishing project staff 

competencies prior to contracting staff would help avoid delays in development. Input from internal and 

external stakeholders regarding the forms, training modules, and updates and enhancements to the online 

monitoring system ensured buy-in and ownership of MARS. Stakeholder feedback was key in facilitating 

changes that streamlined and created efficiencies in the AR process. Stakeholders began believing in and 

supporting MARS when the changes they recommended were designed and implemented. 
Abt Associates CN ART Grants | Final Study Report Final ▌57

Overview

Grant Start Year: 2011 Grant End Year: 2016

Purpose: Creating a new data system that works with preexisting systems to produce reports on key performance 

indicators; developing and delivering e-learning modules

Extension(s): Yes

Project Name: Serving with Success (e-learning modules) Vendor: Outside for-profit software vendor; University or 

Community College training developer

Grant Funds Awarded: $1,276,900, reduced to $922,900 Grant Funds Returned: $143,392.56

Missouri received an Administrative Review and Training (ART) grant in 2011 to develop and implement 

a centralized data system that could provide the State Agency (SA) a comprehensive analysis of Local 

Educational Agency (LEA) key performance indicators. The grant also let the State implement an elearning system to enhance the ability to provide needed program training for LEAs and schools. Prior to 

the grant, the State did not have the ability to do benchmarking or electronically collate LEA reports. 

The new centralized reporting system allows the SA to remotely monitor program performance of LEAs 

for more efficient and effective administrative reviews. The e-learning modules provide consistent 

training designed to prevent errors and support corrective actions across a broader range of LEAs.

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Implement a technology system with a central data 

warehouse that will compile the data from different systems 

and entities in real time into comprehensive reports. 

• Implement electronic learning using a series of web-based, 

individualized learning modules. 

Goals & Objectives 

• With approval from FNS, the SA dropped “real time” 

from the description of the data reports. There were 

no other changes to project goals from the grant 

application to the implementation phase.

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.

Key Features

Missouri is a large state with limited resources. The ART grant provided a centralized reporting system 

and online training modules that enable the SA to conduct comprehensive administrative reviews of LEAs 

effectively and efficiently, even with the ongoing staffing and funding constraints. Activities funded by 

the ART grant achieved the following: 

• Developed 50 total reports, including four key performance indicators (KPIs), 26 administrative 

reports, six financial reports, and 14 compliance reports. 

• Developed the ability to produce ad hoc reporting and application and claim information. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌58

• Produced a series of 23 e-learning modules covering all topics contained within the administrative 

review.

Planning and Implementation

Management

The Missouri Department of Elementary and Secondary Education (DESE), School Food Services 

provided oversight for the project. Two different groups at the State level made decisions relevant to the 

centralized reporting system component of the grant—the Missouri Office of Administration, Information 

Technology Services Division, and the DESE Office of Data System Management. Prior to the grant, the 

State acquired an administrative review module from a software vendor, the State’s existing web-based 

application vendor. Originally DESE planned to build the system in-house with support from a software 

vendor, but after the award, a Project Assessment Quotation resulted in a partnership with another 

software vendor, which ultimately built the reporting system. The training developer created e-learning 

modules. 

Funding

FNS awarded Missouri an ART grant in 2011 for $1,276,900. A grant adjustment in 2013, described in 

the challenges below, resulted in a reduction of funds by $354,000. Of the revised amount of 

$922,900.00, DESE used $779,507.00 (84.5%) to create a centralized reporting system and e-learning 

modules, and returned the remaining funds. 

Successes

The grant achieved the goals of creating a centralized reporting system for collating and organizing data 

relevant to the administrative review process and of developing e-learning modules that cover topics 

relevant to administrative reviews. Data from the reports has provided the ability to identify LEAs that are 

at risk for administrative errors, analyze trends, and track and review administrative review compliance. 

The e-learning modules provide Missouri-specific training in a central, easily accessible location. LEAs 

use the training to onboard new staff, refresh knowledge over time, and as a corrective action to address 

administrative errors.

• Training and Professional Standards: A series of 23 e-learning modules, called Serving with

Success, is available on DESE’s website (https://dese.mo.gov/financial-admin-services/foodnutrition-services/serving-success). The training series covers a range of topics relevant to program 

performance and administrative reviews through the use of videos, photos of documents, and 

interviews with staff. There is an assessment quiz for each module.

• Data Systems: Missouri created a centralized reporting system that pulls data from existing systems 

and organizes them into key performance indicators by LEA. This allows the compliance staff at the 

SA to quickly identify error-prone LEAs and to notice trends that might require the State’s attention. 

The information provided through the reports allows the SA to prioritize more intensive training and 

support for at-risk LEAs, within the constraints of limited funding and staff. Overall, 50 reports were 

produced, along with the ability to produce ad hoc reports that pull from application and claim 

information.

 Initial Grant Amount: $1,276,900.00, reduced to $922,900 after award.

 Activities Funded by the Grant: Central data warehouse and e-learning modules.

 Additional Funding Sources: State Administrative Expense funds covered staffing.

 Funds Returned: $143,392.56 (in addition to $354,000 returned after award).
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌59

Challenges

Missouri encountered challenges with the involvement of multiple stakeholders and staffing (turnover), 

resulting in four no-cost extensions and a reduction in the original grant amount. 

• Multiple Stakeholders: The State initially planned to develop an RFP for the centralized reporting 

system, but there were two agencies at the state level involved in that process and each had a different 

perspective on how best to move forward. The process shifted from developing an RFP, to developing 

the system in-house, to producing a Project Assessment Quotation for vendors on the existing State of 

Missouri contract. The initial delays and confusion over how to move forward resulted in a grant 

reduction and a one-year no-cost extension.

• De-scoping the Grant: Missouri de-scoped its contract to eliminate the development of an 

unnecessary technological feature: real-time reporting capabilities. That is, two years into the grant, 

the State realized that because LEAs submitted reports monthly, and the reports were processed using 

LEA data, there was no need to enable real-time reporting because “real time” would only ever have 

reflected monthly updates, not daily changes. Removal of “in real time” from the description of the 

data reports resulted in a grant reduction of about 30%. Ultimately, the reports were built off of data 

reported by the LEAs and provided on a monthly basis.

• Staffing: Managing the grant in-house placed a significant burden on SA staff, who maintained their 

regular job duties while also attending to project management, procurement, and the development of 

content for the reporting and training modules. Even with contracting out both components of the 

grant, the SA still had to spend an unanticipated amount of effort and involvement in order to provide 

guidance, feedback, testing, and review. This led to the second and third no-cost extensions.

• Staff Turnover: During the second phase of the grant, when the State focused on developing the elearning modules, the project lost key staff who were providing oversight, which led to the fourth 

extension.

Lessons Learned

Missouri addressed the challenges listed above to implement its ART grant project successfully. The 

process resulted in some lessons learned for future efforts. First, it is important to involve the relevant 

decision-makers early on and (particularly for efforts that involve IT) to identify what process will be 

used for procurement. Specifically, SAs should know at the time of grant application (or soon after grant 

award) whether the work can be handled in-house or a vendor is needed (and contracted by which 

procurement process).

Second, Missouri learned that a dedicated project manager is essential to keeping things moving. Having 

a detailed timeline for the project, along with regular progress updates and weekly check-in meetings with 

important stakeholders and participants, helps keep the project on track. It is also important to have 

people on the team who understand both IT and the program needs, who can translate and mediate 

discussions during check-in meetings.

Third, it is important to add sufficient time to the project timeline to account for staff turnover, long 

procurement processes, and multiple rounds of testing and review, especially if project staff will also be 

juggling other responsibilities. 
Abt Associates CN ART Grants | Final Study Report Final ▌60

Overview

Grant Start Year: 2011 Grant End Year: 2016

Purpose: Web-based system and training Extension(s): Yes

Project Name: MyNHDOE System Vendor: [None, developed in-house]

Grant Funds Awarded: $1,409,532.85 Reported Grant Funds Returned: $753,202.42

New Hampshire received an Administrative Review and Training (ART) grant in 2011 to develop a webbased system for application and claiming and for conducting administrative reviews. The State also 

created online training modules and held a series of conferences to provide training on a variety of topics 

related to administrative review. The online training is also used as a tool for corrective action identified 

during administrative reviews. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Develop an electronic web-based system that will identify atrisk and error-prone schools.

• Provide training to schools that have been identified as atrisk or error-prone by the new system, to build consistent 

Food Service operations management outcomes. 

Goals & Objectives

• The goals and objectives did not change from the 

grant application to implementation phase.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) staff members about their experiences related to the ART grant and its associated 

activities

Highlights

New Hampshire’s 2011 ART grant was awarded so it could update its technology systems and provide 

training to schools identified as at risk. Activities funded by the ART grant achieved the following: 

• In-house development of a web-based system that includes application and claiming and 

administrative reviews.

• Development of a series of ART conferences that covered topics such as verification and free and 

reduced applications. 

• Development of online training comprising 10 modules that can be used as part of corrective action 

during administrative reviews.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌61

Planning and Implementation

Management

Project oversight was provided by the administrator of the New Hampshire Office of Nutrition. The 

administrator worked with a dedicated project manager. The New Hampshire Department of Information 

Technology built the system in-house. 

Funding

The State of New Hampshire was awarded $1,409,532.85 through an ART grant in 2011. The grant funds 

were used to develop an in-house web-based system that includes administrative reviews. The decision to 

build the system in-house as opposed to purchasing an off-the-shelf system, as originally intended, led to 

significant cost savings. The State returned 53% of the award at the end of the grant period. In addition to 

grant funds, the state used State Administrative Expense funds to purchase printers that allowed staff to 

print the forms necessary for the on-site review. 

 Initial Grant Amount: $1,409,532.85.

 Activities Funded by the Grant: Design and implementation of a new technology solution; 

training.

 Additional Funding Sources: State Administrative Expense funds.

 Funds Returned: $753,202.42.

Successes

The grant achieved the goals of creating a web-based system for conducting administrative reviews and 

developing training for at-risk schools. 

• Training and Professional Standards: New Hampshire developed 10 online training modules that 

were used by 73 people during the grant period. The State also hosted two conferences that provided 

training on a variety of topics including verification, free and reduced applications, etc. 

• Data Systems: The State developed a web-based system—MyNHDOE—that includes administrative 

review forms as well as the FNS 742 verification summary report and the Community Eligibility 

Provision, which improved efficiency in completing claims and applications. 

Challenges

Though the goals of the project were achieved, there was a need for two no-cost extensions due to delays 

related to staff turnover. 

• Constraints of State Bidding Process: The SA initially planned to use the grant to purchase an offthe-shelf web-based system. However, in order to fulfill State requirements to contract with the 

lowest bidder, it switched to an in-house version. This helped reduce the cost of the project, but it 

created a significant burden on project management staff and the New Hampshire Department of 

Information Technology. 

• Training and Professional Standards: Developing the online training modules required significant 

content knowledge. The developer of one of the 10 training modules did not have sufficient content 

knowledge to create the module and it had to be scrapped. The State had to redevelop the module 

using the correct information. 

• Data Systems: A lack of continuity in IT staff and a lack of dedicated staff led to delays in 

development of MyNHDOE. This has also made post-grant-period modifications difficult given the 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌62

lack of historical knowledge about the system. The State did not account for the time it takes to enter 

data into the system in order to test MyNHDOE. 

• Staff Turnover: Turnover in administrative and IT staff led to significant delays in the development 

and rollout of MyNHDOE, eventually resulting in two no-cost extensions. 

• Project Management: The initial project manager lacked the appropriate skill set to effectively 

manage the project. 

Lessons Learned

The project challenges detailed above highlight the need to include staff with the right skill set. When 

hiring a project staff, it is important to make sure their skills align with project needs and that they have 

adequate time to dedicate to the project. Online training development may require knowledge of 

complicated policies and regulations. It is important to ensure content developers have the appropriate 

knowledge. 

It is important to determine the system’s needs before building begins, including what is needed to 

maintain the system. The State learned that it is important to account for all steps in the project timeline, 

including the time it takes to effectively test a new system. When considering off-the-shelf products, it is 

important to account for funds to support maintenance after the grant period. 

MyNHDOE may be an effective tool to help administrators identify at-risk schools. Given the small size 

of the State, however, at-risk schools may already be known by State Agency staff, and such a tool may 

not add much value. 
Abt Associates CN ART Grants | Final Study Report Final ▌63

Overview

Grant Start Year: 2014 Grant End Year: 2017

Purpose: Technology improvements Extension(s): Yes

Project Name: North Dakota ART Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $389,973 Grant Funds Returned: $0

In 2014, North Dakota received an Administrative Review and Training (ART) grant to update its 

administrative review technology system. The State used the grant funding to purchase an Automated 

Eligibility Processing System that reduces eligibility determination errors, creates efficiencies for Local 

Educational Agencies (LEA) and the State Agency (SA), and allows oversight of and access to LEA 

determinations for completion of administrative reviews. The grant also funded a Comprehensive 

Administrative Review System that integrates and automates all review processes to increase program 

integrity. 

With these updates, the SA moved from a paper-based process to an automated process, which centralizes 

information, improves application and claiming accuracy, and ensures that administrative review 

procedures comply with federal requirements. The multi-user interface allows the SA and LEAs to 

simultaneously access the system, monitor progress, identify issues, and communicate effectively. As a 

result, the SA is able to provide more comprehensive and targeted technical assistance and training to 

LEA staff. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Improve accuracy in application and claiming processes by

providing LEAs and State staff with tools to accurately 

determine meal eligibility and increase reporting accuracy. 

• Update administrative review process to meet new 

requirements; streamline the process for efficiency, data 

completion, and accuracy.

Goals & Objectives

• The goals and objectives did not change from the 

grant application to implementation phase. North 

Dakota met these goals by implementing a Statehosted centralized meal application determination 

system and state-level administrative review 

system.

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities

Highlights

North Dakota’s 2014 ART grant funded the implementation of a State-hosted centralized meal application 

determination system and administrative review system. The activities funded by the ART grant achieved

the following:
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌64

• Partially automated a web-based administrative review process, including electronic form submission 

and automated reporting features.

• Ensured the administrative review process complies with federal requirements.

• Improved management and monitoring: State and LEAs can easily track requirements for the process:

o Allows the State to efficiently identify issues and track corrective action requests to ensure 

prompt completion. 

• Centralizes all information and facilitates better communication between State and LEA staff:

o Sends the SA’s off-site and on-site confirmation letters through the new system.

o Allows the State to better understand where additional technical assistance is needed.

Planning and Implementation

Management

A number of staff from North Dakota’s SA and the software vendor were involved with the 2014 ART 

grant project. The SA director oversaw the project and the staff who worked on it. The State technology 

coordinator worked closely with internal SA staff, but a project manager was also hired to work directly 

with the software vendor. The project manager oversaw the project task schedule and budget and worked 

with a grant manager on finances. The software vendor also assigned a vendor project manager who 

worked directly with State staff. 

Funding

North Dakota was awarded a $389,973 grant, and stated they used 100% of the allocated funds. The 

majority of ART funding was spent on vendor contracts. The SA’s technology coordinator was also 

funded through the ART grant. The SA project manager was funded by State Administrative Expenses 

funds. No other supplementary funding sources were used as a part of this project.

 Initial Grant Amount: $389,973.

 Activities Funded by the Grant: Customization and implementation of the software.

 Additional Funding Sources: State Administrative Expenses.

 Funds Returned: $0.

Successes

The grant achieved the goals of streamlining and automating the administrative review process and 

improving the application and claiming processes. Additionally, North Dakota employed a dedicated 

grant management team, including a project manager, technology coordinator, and a fiscal grant manager, 

which facilitated the successful implementation of the grant.

• Administrative Costs: The ART grant did not present significant changes in administrative costs or 

time spent on administrative review for State staff. 

• Error Rates in Application Processing: State staff noted increased error identification as a result of 

the improved organization and management of the administrative review process afforded by the new 

system. 

• Training and Professional Standards: The centralized and automated nature of the new system 

allows the SA to identify areas where LEAs need assistance. LEAs noted that State staff were easily 

accessible for technical assistance over the phone. 

• Data Systems: The automation, centralized organization, and streamlining of the new administrative 

review system created better data quality, improved monitoring and follow-up, and providing 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌65

facilitated communication between the State and the LEA staff. The new system also improved the 

workflow for LEA staff, because components of the administrative review are prepared in advance, 

reducing on-site review time. 

• Administration of School Nutrition Programs: The online free/reduced-price meal applications 

have been well received by parents. LEAs have seen an increased number of families completing the 

online application. The SA noted that more LEAs have completed verification accurately and on time. 

• Improved Direct Certification Process: The new system allows LEAs to accurately and quickly 

identify students for direct certification. The direct certification process more efficient because 

verified information is sent to LEAs directly from the State.

• Grant Management: The State’s grant management team included a technology coordinator who 

was the primary support staff for the SA’s computer systems used to manage the nutrition programs. 

Additionally, the State hired a dedicated project manager to work with the vendor and the State IT 

department on the new technology software. North Dakota also had a grant manager assigned by the 

Department of Public Instruction’s internal fiscal office. 

Challenges

Though the grant was effective at streamlining the administrative review process and providing online 

application options, there is still room for improvement and a need for additional training. Additionally,

two no-cost extensions were needed to accommodate delays related to staff turnover and vendor issues. 

• Administrative Costs: The new system is able to automatically complete direct certification 

matching using uploaded Supplemental Nutrition Assistance Program data; however, the North 

Dakota Department of Human Services would not approve the release of that data to the vended 

system. As a result, direct certification student data must be downloaded nightly from the State direct 

certification system into the new student eligibility system. This process does not delay issuance of 

free meal benefits to students, but does increase the administrative burden on the SA. 

• Training and Professional Standards: LEA staff believed the initial training videos were too broad, 

as they were not tailored to the LEAs’ use of the system and covered modules that were not required. 

LEAs initially didn’t understand which system components they would be using, resulting in 

confusion and a need for clarification and direction from the State. LEAs suggested that a more 

directive and tailored training (i.e., in person) was needed, specifically with completing the new 

spreadsheets.

• Staff Turnover: North Dakota lost many administrative reviewers due to staff turnover, resulting in 

increased training time, offsetting the anticipated efficiencies of the system. Interviewees noted that 

some State program staff were resistant to the new system and unwilling to use it properly, which 

may have contributed to staff turnover.

• Vendor Challenges: The software vendor’s original project manager was unresponsive and did not 

understand the system the SA was working with. The software vendor assigned a new project 

manager to the contract. Additionally, the SA noted the schedule was delayed as a result of errors in 

the technology, requested changes that had not been integrated by the vendor, and features that did 

not meet the SA’s expectations and required revisions. The SA noted that this is an ongoing challenge 

that requires persistent follow-up with the vendor. Unfortunately, the State no longer has funding for 

the project manager position and is not able to withhold payment to the vendor, so it has very little 

leverage to ensure prompt vendor response. 

• Lack of LEA Buy-in: North Dakota has many small LEAs, and the SA saw them as the target 

audience for the new student eligibility module. However, the LEAs were interested only initially, 

until they learned of the work required on their end. LEA buy-in was also affected when the system 

feature that auto-populated eligibility information (to save LEAs time) didn’t work when the system 

was rolled out at the beginning of the school year. The failure caused many LEAs to opt out of using 

the system and to stay uninterested even after the problem was fixed. The State expects use by LEAs 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌66

to increase as they become more familiar with the system and understand the benefits of automating 

the student eligibility system. Large LEAs that already had an electronic application determination 

system as part of their point-of-sale system use the State online household application but do not plan 

to fully use the centralized system. 

Lessons Learned

North Dakota noted that it would consider applying for an ART planning grant in the future in order to 

research system options and functionality and to better prepare the SA to design and implement new 

technology that meets its needs. Many of the State’s challenges were a result of the SA staff not fully 

understanding the functionality needed to address the new administrative review requirements. 

Additionally, the SA acknowledged that expanding pilot testing to a wider range of LEAs (beyond those 

already interested in the technology) and gathering LEA feedback on the administrative review module 

would provide valuable information regarding system functionality from the LEA perspective. 

Other lessons learned included the importance of budgeting for a dedicated project manager, immediately 

addressing vendor service-related issues to prevent project delays, and including at least three months in 

the schedule to complete the procurement process of a vended system.
Abt Associates CN ART Grants | Final Study Report Final ▌67

Overview

Grant Start Year: 2013, 2014 Grant End Year: 2016 (2013 grant), 2017 (2014 grant)

Purpose: Technology improvements (2013 grant), 

technical assistance (2014 grant)

Extension(s): No

Project Name: Pennsylvania Student Eligibility & Direct 

Certification Systems (PA-SES)

Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $1,499,977 (2013), $464,801 (2014) Grant Funds Returned: $0 

2013 Grant

Pennsylvania applied for its 2013 Administrative Review and Training (ART) grant in response to the 

changes to the administrative review process mandated by the Healthy, Hunger-Free Kids Act of 2010. 

These changes affected Pennsylvania in particular because its State Agency (SA) has oversight for more 

than 900 Local Educational Agencies (LEA). The new regulations shifted administrative reviews (AR)

from a five-year to three-year cycle, resulting in a 67% increase in the volume of ARs scheduled. 

In order to accommodate this shift, the SA used its 2013 ART grant to purchase an automated eligibilityprocessing system. The SA selected an automated, commercial off-the-shelf solution to develop the 

Pennsylvania Student Eligibility and Direct Certification Systems (PA-SES). This software was used to 

facilitate efficient AR; reduce eligibility determination errors while creating efficiencies for the LEAs and 

the SA; increase direct certification (DC) match rates; and allow remote oversight of eligibility 

determination, verification, and DC for LEAs.

The 2013 ART grant was also used to migrate the DC process from the existing system, COMPASS, to 

PA-SES. COMPASS was Pennsylvania’s online benefit application for free and reduced-price meal

benefits. Prior to implementing the new PA-SES system, LEAs would directly certify by uploading 

student files to match against the State’s Department of Human Services benefits data, which include 

Supplemental Nutrition Assistance Program, Temporary Assistance for Needy Families, and Medical 

Assistance, as well as whether the family met the income threshold guidelines. The benefits data were 

provided by the Department once a month, and LEAs were encouraged to perform the DC process once a 

month and a minimum of three times a year. 

Now PA-SES provides stronger matching algorithms during the DC matching and is more cost-effective, 

given the ability to use the full student eligibility system. The new technology includes automatic DC of 

matched students, generates lists of possible household matches, reviews and manages unmatched 

students, extends DC to siblings, and creates eligibility roster reports. 

2014 Grant

Through the 2014 ART grant, the SA identified and provided technical assistance to LEAs at high risk for 

administrative errors or those significantly out of compliance with menu planning requirements. This 

grant funded a new position dedicated to providing training and support to high-risk LEAs using the new 

PA-SES system. It enabled the Division of Food and Nutrition (DFN) to provide one-on-one, on-site 

technical assistance to 97 LEAs in the areas of certification, verification, meal counting and claiming, and 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌68

menu pattern compliance. Additionally, the State also provided on-site technical assistance visits focused 

on meal counting and claiming to 35 individual private schools in the Archdiocese of Philadelphia and 31 

schools in the School District of Philadelphia.

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

2013 Grant

• Implement a new direct certification and application system 

that will simplify work for LEAs, reduce error rates, and 

increase the number of directly certified students.

• Implement a system that creates efficiencies and streamlines 

the administrative review process for the SA respective to 

areas of validating the certification process (direct certification, 

eligibility determination and verification)

o Increase the number of students directly certified to 

meet federal thresholds. 

o Validate the LEA’s certification process during an 

administrative review remotely in order to decrease 

the amount of time needed on-site at the LEA. 

o Provide a more efficient means to monitor and 

provide technical assistance to error-prone schools 

relative to eligibility determination errors associated 

with Performance Standard 1. 

2014 Grant

• Ensure LEAs have resources in order to maximize 

understanding of administrative and menu compliance 

requirements and identify error-prone or high-risk LEAs.

• Ensure LEAs have the technology to reduce administrative 

and menu compliance errors by providing statewide access to 

the Student Eligibility System and Menu Planning System at 

no charge to a LEA. 

Goals & Objectives

• The goals and objectives for both the 2013 and 

2014 projects did not change from the application 

to the implementation phase.

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.

Highlights

Pennsylvania’s 2013 ART grant project streamlined and digitized its AR and DC processes. Over 300 

LEAs use the full student eligibility system, and all of the LEAs (more than 900) are using it for DC. The 

2013 grant allowed for the implementation of the PA-SES software, which: 

• Streamlines the SA’s administrative review process.

• Decreases administrative reviewer time on-site.

• Increases DC of students through automatic matching.

• Provides DFN access to all DC data at the LEA level and allows determination of the frequency of 

LEAs performing DC matching. 

• Collects online, user-friendly meal applications and ensures completion.

• Provides immediate and automatic notifications to parents/households. 

• Automates the verification process.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌69

The 2014 ART grant provided one-on-one training and technical assistance to high-risk LEAs, along with 

access to the new software system. The 2014 grant allowed the SA to employ dedicated technical 

assistance staff to work with high-risk LEAs to train LEA staff on the system and address other sitespecific problem areas, including meal counting and claiming, eligibility determination, verification, and 

DC.

Planning and Implementation

Management

Prior to the award of the 2013 ART grant, Pennsylvania contracted with a dedicated project manager/IT 

liaison who was heavily involved in the process of purchasing and installing the software and training 

State staff. The state nutrition director was directly responsible for writing the grant application and 

overseeing implementation. The State also benefited from having a software vendor project manager and 

software support team. The SA employed a compliance manager and school nutrition program manager to 

assist and consult during grant planning and implementation. 

Additionally, the software solution was facilitated through the State’s Dell Contract, which provided 

access to State-approved software. Use of the existing Dell Contract streamlined grant management, as 

the vendors were preapproved and the SA was not required to conduct a formal bid process. As a large 

state, Pennsylvania conducts approximately 300 administrative reviews each year. The DFN department 

responsible for administrative reviews aided in the 2013 grant implementation. This department 

comprises a manager, five supervisors, and 13 field advisors who conduct the reviews. The 2014 ART 

grant funded an individual to provide one-on-one technical assistance to LEAs and schools identified as at 

high risk. 

Funding

Pennsylvania was awarded $1,499,977.00 for its 2013 grant, and $464,801.00 for its 2014 grant. The 

State stated they used 100% of its total award for each grant. 

 Initial Grant Amount: $1,499,977.00 (2013); $464,801.00 (2014).

 Activities Funded by the Grant: The majority of 2013 funding went to the upfront cost to 

purchase, install, and pilot the software. Funds were also used to promote the new 

technology and train LEAs. The 2014 grant funding was used to fund technical assistance 

staff.

 Additional Funding Sources: State Administrative Expense (SAE) funding was used to fund 

the project manager position for the 2013 grant, as well as other staffing costs. Since the 

grant expired, the SA uses SAE funds for annual software maintenance and hosting fees for 

the new technology.

 Funds Returned: $0 (2013; 2014).

Successes

Pennsylvania’s 2013 ART grant enabled the purchase and implementation of the PA-SES system, which 

streamlines the administrative review process, increasing efficiency and reducing errors for the SA and 

the roughly 300 LEAs that opted to use the full student eligibility system. The grant also resulted in the 

migration of the DC process from a Department of Human Services web application (COMPASS) into 

PA-SES, resulting in increased DC matches, as this module was mandatory for all 900 LEAs. The SA 

noted an initial DC increase from 70% to 80%, with continued gradual increases. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌70

Regarding the 2014 grant, the SA noted that working with LEAs one-on-one to provide technical 

assistance in areas of concern resulted in corrections in areas of noncompliance identified during 

administrative reviews. LEAs benefited from the technical assistance visits and support.

• Administrative Costs: The SA noted savings related to the decreased time needed to complete the 

eligibility and verification processes and the automation of notifications. The online system also 

allows for multi-user remote access and may result in travel savings. Additionally, the SA responded 

to the LEAs’ challenge to secure IT staff time by requiring superintendents to commit IT resources 

and by conducting conference calls with LEA IT staff and large student information system providers. 

Instead of multiple schools contacting sales reps about file transfer issues, the State facilitated the 

communication with the student information software companies and provided webinars for LEAs. 

The SA noted that this resulted in overall cost reductions because issues were addressed and paid for 

once, rather than each time a different LEA contacted the software company. 

• Error Rates in Application Processing: The 2013 ART grant verification reporting improvements 

allow for improved monitoring and reporting, and they increased DC matching. The online system 

rejects incomplete meal applications and alerts the State and LEA when corrections are required. The 

2014 ART grant has made it possible for the Pennsylvania Department of Education (PDE) to provide 

additional training and technical assistance to help reduce the amount of administrative errors in highrisk LEAs. The additional trainings and technical assistance have helped LEAs to better understand 

the Child Nutrition Programs and allowed them to successfully provide these programs to children in 

Pennsylvania.

• Training and Professional Standards: DFN promoted the PA-SES to LEAs through a designated 

website, software previews, email campaigns, and memorandums. Though use of the new eligibility 

system wasn’t mandated, the SA provided training to all LEAs that opted in. Additionally, training 

was provided statewide via webinar for the mandatory use of the DC module in PA-SES. The 2014 

ART grant allowed the PDE to provide one-on-one technical assistance in areas of concern. Providing 

technical assistance in this manner allowed the LEAs to work with the SA to help correct areas of 

noncompliance found during administrative reviews. The SA received positive feedback from LEAs 

in regard to using on-site one-on-one technical assistance to help prevent future administrative errors.

• Data Systems: The PA-SES system streamlines SA management of student eligibility; increases 

LEA DC percentages, leading to increases in statewide DC percentages; and increases efficiencies of 

the off-site review of information during the administrative review.

Challenges

Major challenges for Pennsylvania included the lack of a common statewide student information system, 

lack of local IT resources, and burdensome contracting and hiring processes. Additional challenges 

include frequent LEA staff turnover and reluctance of LEAs to adopt the new technology and 

electronically share data with the SA.

• Data Systems: Pennsylvania identified the lack of a statewide student information/school enrollment 

system as the biggest challenge faced. The PS-SES student eligibility system is a food service, Child 

Nutrition Program system, but the student information system is what drives the enrollment in the 

school. In Pennsylvania, LEAs purchase their own student information system if they want to use an 

IT-based system; everything is locally driven. In states where LEAs have a common system, it is 

easier to feed information into PS-SES; in Pennsylvania, each LEA has to upload its student 

information files, to ensure changes in the student information system are integrated into PA-SES. 

Though the SA expected this issue, it complicated the process. Additionally, the State encountered 

difficulty creating an eligibility algorithm that limits the number of matches that schools must review, 

without missing any eligible students.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌71

• Limited LEA IT Resources: After a first round of onboarding the PA-SES system, the State realized 

that the commitment of local IT staff at LEAs and individual schools was critical. The SA found that 

even if the food service director was onboard, IT knowledge limitations resulted in a large LEA 

dropout rate. Also, although most Pennsylvania school districts have an electronic system, many 

charter schools and private schools may not, making it harder for them to implement and maintain the 

new system. 

• Administrative Delays: The SA noted that despite selecting an approved vendor from the established 

Dell Contract, moving the software vendor contract agreement through the state system for the 2013 

ART grant was time-consuming and challenging. Even though the 2014 ART grant’s technical 

assistance position was a carryover from the prior ART grant, rehiring for the position took a very 

long time in the State’s human resources system. 

• Staff Turnover: Regarding both the 2013 and 2014 grants, the SA noted that staff turnover at the 

LEA level presented a constant challenge, requiring frequent re-trainings. 

• Participation Rates: Though Pennsylvania was happy with the overall rate of LEA participation in 

the new system (300 LEAs out of 900), the State encountered LEAs reluctant to share and save 

information to the cloud. Pennsylvania also has robust participation in the Community Eligibility 

Provision (200 LEAs), so those LEAs aren’t incentivized to use the PA-SES for full application 

processing, because they don’t have to process individual applications.

Lessons Learned

Pennsylvania observed that establishing electronic free and reduced-price meal applications as the default 

option increased online participation. When schools gave out paper applications at the beginning of the 

school year, online participation was low. If LEAs adopted the electronic form as the primary method and 

required parents to request a paper application, online participation increased. The SA noted that LEAs 

with the online application as the default had approximately 70% online participation, as opposed to 40% 

in LEAs that did not. 

Additionally, the State noted that local IT staff availability is critical to successfully implementing new 

technologies, such as the new PA-SES system, to its LEAs. The State observed that the majority of LEAs 

dropping out during implementation did so as a result of a lack of local IT staff time, noting that the total 

time commitment is minimal overall, but it is especially critical during the initial implementation stage.

The State also underscored the benefits of the one-on-one technical assistance provided through the 2014 

ART grant, noting that it would be prioritized in future grant applications. 
Abt Associates CN ART Grants | Final Study Report Final ▌72

Overview

Grant Start Year: 2013 Grant End Year: 2015

Purpose: Technology improvements Extension(s): No

Project Name: CN Central Vendor: Outside, for-profit software vendor (2)

Grant Funds Awarded: $312,183.00 Grant Funds Returned: $5,680.15

Under the Administrative Review and Training (ART) grant, the Rhode Island Department of Education 

(RIDE) worked with a vendor to customize its commercial off-the-shelf software system, CN Central, to 

align with USDA guidelines, streamline workflow, and leverage the functionality of existing Coordinated 

Review Effort (CRE) software. With the rollout of the new system, the State Agency (SA) is able to select 

and track Local Education Authorities (LEA) for the administrative review three-year cycle. CN Central 

includes a secure log-on system as well as a communications and notification system that issues, captures, 

and stores communications among RIDE, the administrative reviewers, and LEA staff. This feature 

streamlines both the initial review and subsequent follow-up, such as the corrective action processes tied 

to regulatory requirements. 

Preceding the changes implemented under the ART grant, all administrative review forms were paper, not 

electronic, and were scanned and emailed before the on-site review. The new system was meant to reduce 

this labor burden for LEA staff, by having them upload documents through an online portal, accessible by 

both the State and the respective LEA. State reviewers can enter information collected on-site through this 

system. 

However, State reviewers and LEA staff reverted back to old processes and stopped using the system 

after school year 2017-2018. 

Additionally, RIDE used ART grant funding to create an electronic version of the Meal Benefit 

Application verification summary report (FNS 742) that was integrated into the pre-existing Child 

Nutrition Program, CNPConnect. Prior to the grant implementation, these forms were filled out on paper 

by LEA staff and very often contained incomplete or erroneous information. The new electronic version

is mandatory for the schools, and it automatically reviews and monitors submissions to identify errors. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Improve the Rhode Island Department of Education (RIDE) CNPConnect 

software to enhance the AR process.

o Implement the new USDA AR changes to the existing 

Comprehensive Review Evaluation (CRE) processes by redistributing a portion of reviewer’s preparation steps to the districts.

o Use CNPConnect to identify “error-prone” schools and districts.

• Contract with software vendor to customize CN Central system. 

Goals & Objectives

• The goals and objectives for this 

project did not change from the 

application to the implementation 

phase.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌73

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.

Highlights

Rhode Island applied for an ART grant in 2013 so that it could streamline and digitize its administrative 

review (AR) process and create an electronic Meal Benefit Application. The activities funded by its 2013 

ART grant achieved the following:

• Created a standalone AR module that allowed for electronic preparation of the off-site and on-site 

reviews and redistributed a significant portion of the State reviewers’ workload in the review 

preparation workflow steps to the LEAs. 

o Included an interactive communications and notification system that issues and stores 

customized system-based communications among the SA, administrative reviewers, and 

LEAs. 

o Included a Corrective Action Plan tool that allows the SA to select predetermined and 

customized corrective actions. 

• Created an advanced claim edit for school meals programs, an electronic validation web form, which 

decreased errors through a series of checks and made it simpler for the State to monitor submissions 

and detect errors. New site-level checks included possible block claiming, stagnant attendance 

numbers, and number of meals claimed matching the attendance for 15 consecutive days. 

• The State used the grant-funded system during school years 2016-2017 and 2017-2018, but it has 

since reverted back to old processes.

Planning and Implementation

Management

The RIDE project team consisted of six members: the state child nutrition director, a finance manager, 

and four content experts, but they did not have a dedicated project manager. The staff conducting the 

administrative reviews were not directly involved in the selection of the software. There were two

software vendors, one working on each component: one customized the AR module, and the other created 

the electronic Meal Benefit Application. For the most part, LEAs were not involved in the ART grant 

process before the new systems were rolled out, though at least one district was involved with testing the 

near-final project.

Funding

Rhode Island was awarded $312,183.00 for the 2013 ART grant and used 98% of its allocated funds. The 

ART grant funds were used almost exclusively to pay the vendors for their work on the project. No other 

supplementary funding sources were used as a part of this project.

 Initial Grant Amount: $312,183.00.

 Activities Funded by the Grant: Customization of an electronic AR module and creation of 

an electronic Meal Benefit Application.

 Additional Funding Sources: None.

 Funds Returned: $5,680.15.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌74

Successes

The new AR module was used from April 2016 through the 2017-2018 school year. It allows for multiple 

user access, increases consistency among reviewers, and streamlines dissemination of AR findings, 

allowing letters to be issued while reviewers are on-site. 

• Error Rates in Application Processing: The new electronic version of the FNS 742 form was 

successful in reducing errors, as LEA staff could not submit the form if they had made errors that 

were flagged by the system. The SA reported, anecdotally, a 75% reduction in errors.

• Data Systems: The new AR module system allows the SA to easily establish and monitor timelines 

for LEAs, keeps all materials in a centralized location, and provides a standardized process with builtin checks that reduce submission of incomplete and incorrect information. 

• Administration of School Nutrition Programs: Following the implementation of the new systems, 

there was an increase in the direct certification rates. However, the direct certification process 

underwent an improvement at this time, as well. Thus, it is not clear that the ART grant activities are 

responsible for this improvement. 

Challenges

Major challenges during the implementation of Rhode Island’s ART grant included the development and 

design of the AR module, resource limitations, and minor schedule setbacks. 

• Administrative Costs: As a result of the increased complexity of the AR process and the 

introduction of new software, more time is needed (and scheduled) for on-site reviews. Additionally, 

high staff turnover at the State and LEA levels resulted in increased time spent on administrative 

review preparation. Additional costs were incurred, as State staff time was required to provide oneon-one support to LEA staff to help them navigate the software and prepare for the on-site evaluation. 

• Data Systems: The COTS system did not provide the functionality that the State and LEA staff 

needed to efficiently complete the review process, and customization was not included in the budget. 

The system would not send notifications when actions were taken in the system, and the letter 

templates were not customizable. Additionally, the system was very slow and the interface was not 

intuitive, dis-incentivizing adoption.

• Behavior Change and Lack of Training: Especially because Rhode Island is a small state, the 

LEAs have close relationships with their State contacts and are accustomed to emailing State staff 

directly. This habit proved hard to break, as LEAs would refuse to use the new system, preferring to 

send materials via email, thus requiring State staff time to upload the LEA materials to the system. 

Additionally, the LEAs did not receive any training on the new system to facilitate the transition.

• Vendor Challenges: Following the award of the contract, discussions with the vendor uncovered an 

understanding gap caused by misinterpretation of the vendor’s capabilities and miscommunication of 

the SA’s needs because staff lacked the technical expertise (language and information) needed to 

accurately communicate project needs. Respondents noted that the process would have benefited from 

an IT project manager to bridge that gap.

• Schedule Changes: Though no extensions for this grant were taken, the internal timelines were 

delayed on a few occasions due to USDA updates to the AR process, procurement delays with the 

vendor, and revisions to the AR software. 

• Resource and Software Limitations: Because RIDE was concerned that integrating the AR module 

into the existing CNPConnect system would cause problems, it developed a standalone AR module. 

Adapting a COTS solution to RIDE’s needs enhanced the prior Comprehensive Review Evaluation 

module to meet the new SMART administrative review process and provided a responsive solution to 

this grant’s requirements in the short term. Respondents, however, shared that the software was not 

intuitive and did not always meet their needs. Respondents noted that a long-term solution would 

require additional funding to completely update and integrate all State nutrition systems with software 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌75

that could evolve over time to meet requirements and allow for more sophisticated resources 

management. 

Lessons Learned

The project challenges detailed above highlighted the need for a designated project manager with both 

technical and program expertise who could liaise between the software vendor and program staff to lead 

testing, tracking, and documenting issues to ensure the final product delivers the needed functionality. 

Additionally, the limitations of the new AR module underscore the need for a comprehensive, integrated 

CNP system that can evolve as processes and guidelines change.

The State did successfully complete the scope of work detailed in its grant application and came in on 

time and under budget. However, as mentioned above, the software funded by the ART grant did not 

completely meet the needs of SA staff. Respondents noted that changes to the system would require 

additional technical assistance, funding, and planning and that these steps should be recommended for 

future grant applications. Staff conducting the administrative reviews should be heavily involved in the 

planning and development process in the future to ensure the new technology is functional and 

sustainable.

Rhode Island is investigating advancements made by other states and exploring solutions that may more 

comprehensively address its needs in the short term and the ability to adapt to its needs in the long term. 

State staff mentioned the possibility in the future of using a system that was already developed and tested 

by another SA.
Abt Associates CN ART Grants | Final Study Report Final ▌76

Overview

Grant Start Year: 2011 Grant End Year: 2014

Purpose: Implementation of a technology system and 

training

Extension(s): Yes

Project Name: ICAN Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $843,158.00 Grant Funds Returned: $315,580.22

In 2011, South Dakota received an Administrative Review and Training (ART) grant to implement a new 

technology system and conduct both in-person and online training. The goals of the interventions were to 

(1) decrease administrative error; (2) streamline data entry, data analysis, and accountability reporting 

processes; and (3) train State Agency (SA) and Local Education Agency (LEA) staff in the application, 

verification, meal counting, and meal claiming procedures. The program had not been fully implemented 

when the grant period closed. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Add a new technology system that streamlines data 

entry, data analysis, and accountability reporting 

processes to decrease administrative errors.

• Provide face-to-face and online training for Child & Adult 

Nutrition Services (CANS) and LEA staff on application, 

verification, and meal counting and claiming procedures 

using the new software.

Goals & Objectives

• The goals and objectives for this project did not change 

from the application to the implementation phase.

The following sections include information from grant applications, final grant reports, and discussions 

with SA and LEA staff members about their experiences related to the ART grant and its associated 

activities.

Highlights 

South Dakota’s ART grant project aimed to reduce overall administrative errors by implementing new 

technology and delivering SA and LEA staff training. 

• South Dakota designed a three-pronged technology project to upgrade the State’s system: Phase 1, 

Food Distribution Programs; Phase 2, School Nutrition Programs; and Phase 3, Summer Food Service 

and the Child and Adult Care Food Program. The ART grant funded Phase 2. 

• The State purchased an off-the-shelf, customizable software application from a software vendor for 

its school nutrition program for applications and claims. 

• SA staff and the software vendor collaborated to create training modules and deliver technical 

assistance to end users (SA and LEA staff). 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌77

Planning and Implementation

Management

The State’s office administrator and a project director from the South Dakota Technology and Innovation 

in Education (TIE) office managed the grant project. The South Dakota Bureau of Information and 

Technology (BIT) also assisted managing with State-mandated terminology, server information, and 

ensuring the new system maintained the look and feel of other State IT systems. The FNS technical 

assistance consultants) assisted with the management of the entire system upgrade (all three phases) and 

was involved in some aspects of Phase 2. 

SA team members would meet once or twice a month to evaluate the progress of the project. In addition 

to the State director, the SA team members included the office administrator, the TIE program manager, 

the BIT staff member, and one or two school food program representatives. At times, the FNS’ technical 

assistance consultants and the software vendor would also participate in meetings, depending on the topic 

or if there were specific issues where outside input was required.

Funding

In 2011, South Dakota won an ART grant to create and implement a new technology system. The award 

was for $843,158.00, and the State used 63% of the total allocated ART funds. South Dakota also used 

reallocated State Administrative Expense funds to assist with the school nutrition system installation, as 

well as approximately $100,000 annually from the South Dakota State Technology Funding to support the 

software vendors system.

 Initial Grant Amount: $843,158.00.

 Activities Funded by the Grant: Creating a new software system and training SA and LEA 

staff on its use.

 Additional Funding Sources: State Administrative Expenses, South Dakota State Technology 

Funding.

 Funds Returned: $315,580.22.

Successes

The grant provided the SA with the opportunity to create and implement a new software system for its 

school nutrition programs and provide training, tutorials, and technical assistance on the use of the new 

system.

• Administrative Costs: The new system saves significant time over the previous process. For 

example, staff are able to enter data as they becomes available (rather than during a small, two-month 

window). The certification process takes less than a week, whereas it took more than two weeks to 

complete previously. 

• Error Rates in Application Processing: The SA was not able to quantify the reduction in error rates, 

but did comment that typical errors seen in the past, such as checking the “offer versus serve” option, 

have been eliminated with the new system. Also, the system immediately catches potential errors, and 

staff are required to verify that the data entered are correct before submitting. The edit checks in the 

new system allow the SA to review what has been submitted by the LEA and ensure accuracy.

• Training and Technical Assistance: The SA conducted in-person trainings for LEA staff to ensure 

one staff member fulfilled the “business manager” role and one fulfilled the “school food service” 

role. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌78

• Data Systems: The SA and LEAs are pleased overall with the new system, particularly with the 

flexibility and centralized accessibility to information it affords. The previous system was rigid and 

allowed a short window of time during which to enter data. The new system allows flexibility to 

enter, and correct, data as needed. Also, state-level program specialists have easier access to the 

information needed to complete annual agreements with each LEA.

• Administration of School Nutrition Programs: The new system allows continuity at both the State 

and local levels, despite staff turnover and, unfortunately, natural disasters. 

Challenges

South Dakota achieved the goals and objectives for the ART project; however, it required a no-cost 

extension and experienced the following challenges with the project: 

• Administration: The SA did not realize the amount of time the entire process would take. During 

implementation, the SA requested an additional FTE from the South Dakota BIT department. To 

maintain the software vendor’s system (which includes programs other than school food), some 

$100,000 from South Dakota State Technology Funding is needed. 

• Error Rates in Application Processing: Not all of the LEAs came online at the same time. At the 

end of the grant period, the SA was still implementing the new system with LEAs.

• Training and Professional Standards: LEA staff had varying levels of competency with 

technology. The SA did not anticipate the need to offer remedial-level training for LEA staff with 

extremely limited computer experience. During the grant project implementation, the SA discovered 

it needed to train LEA staff on permissible use of newly installed computers and Internet access. 

• Data Systems: A significant number of LEAs lacked access to computer systems within their 

respective offices. The SA had to work with State and local staff to identify funding for equipment 

and Internet access, needs not anticipated when the grant application was written.

Lessons Learned

South Dakota shared the following lessons learned:

• Project Management: A dedicated project manager (from grant application phase to project 

completion) with in-depth computer language and processes background is integral to an IT project’s 

implementation success and efficiency. Many of the delays, unanticipated costs, and need for 

extensive staff time could have been avoided if a project manager with the IT skills, knowledge, and 

background had been involved.

• Software Testing: Request testing protocols from the software vendor to be able to see what the 

vendor has done before beginning internal testing. Involve LEAs and other end users in the testing 

phase to get a clearer picture of what they may experience when the system goes live. Make sure the 

IT vendor understands the State’s security requirements, and anticipate significant time for testing for 

security.

• Training Materials: The RFP and subsequent contract with the software vendor need to specifically 

address who is responsible for the development of end-user manuals and training materials—both 

upon initial implementation and to cover any updates that occur after the system is online. Step-bystep user manuals and training materials and recorded webinars/training sessions will help prevent 

issues with future training upon staff turnover. 

• Staff Training: It is difficult for SA staff to learn a new system and then provide training on it. Initial 

training for SA and LEA staff should be included in the RFP/contract with the software vendor.

• Software Licensing: Be prepared for the cost of maintaining the annual software license after the 

grant period has ended.
Abt Associates CN ART Grants | Final Study Report Final ▌79

Overview

Grant Start Year: 2011 Grant End Year: 2016

Purpose: Replace CNP 2000 with WINS Extension(s): Yes

Project Name: Washington Integrated Nutrition System 

(WINS)

Vendor: In-house IT department 

(began with Outside, for profit software vendor and then 

moved in-house)

Grant Funds Awarded: $1,478,700 Grant Funds Returned: $0 

In 2011, the Washington Office of Superintendent of Public Instruction (OSPI) was awarded an 

Administrative Review and Training (ART) grant to replace an existing (and aging) Child Nutrition 

application and claiming system (CNP 2000). The new platform, the Washington Integrated Nutrition 

System (WINS) integrates data from separate programs, provides error checks and alerts, eliminates 

duplicate data entry, and allows for site-based claiming. Local Educational Agencies (LEA) can directly 

upload fillable spreadsheets for their administrative reviews, reducing the amount of paper that passes 

between the State and LEAs. Importantly, the system pre-populates information where possible, 

streamlining data entry for sites that run multiple nutrition programs and providing error checks in real 

time. 

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Replace the claiming and application system with a new 

solution, so that the data entry process is streamlined, LEAs 

with multiple CN programs can consolidate data, and 

increase accuracy of claims.

• Increase data-sharing by creating a system that will interface 

with other Office of Superintendent of Public Instruction 

(OSPI) software, including the Comprehensive Education 

Data and Research System (CEDARS), the list of directly 

certified students from the Department of Social and Health 

Service, students who receive Temporary Assistance for 

Needy Families (TANF) or other food benefits, and students 

who are eligible based on foster care status.

• Improve program integrity and administrative accuracy 

through the new system by tracking and analyzing CRE 

results, analyzing nutrition of school meals, identifying at-risk 

LEAs that need additional training, and improving accuracy in 

USDA reports.

Goals & Objectives

• Did not meet the objective to develop capabilities 

for electronic verification due to insufficient grant 

funds and schedule delays. The SA plans to 

develop electronic verification (including nutrient 

analysis for meal plans) as a part of a follow-on 

maintenance and operations contract.

• Removed the goal of integration with CEDARS and 

the Department of Social and Health Service (upon 

more in-depth analysis, the agency learned that 

integration would not be beneficial because of 

differences in the types, timing, and requirements 

for data across the three agencies). 

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and LEA staff members about their experiences related to the ART grant and its 

associated activities.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌80

Highlights

Overcoming challenges with an IT vendor and a large team with multiple stakeholders across multiple 

agencies and contractors, the SA used ART grant funds to develop a unified application and site-based 

claiming system across multiple Child Nutrition Programs. The activities funded by the grant achieved 

the following:

• The system eliminates duplicate data entry for sites that run multiple programs, provides edit checks, 

and provides ad-hoc reports that allow the SA to identify areas for targeting training and correction. 

• WINS ensures revisions, adjustments, and financial transactions are always assigned the correct 

general ledger coding (the previous system did not accurately track and report these without manual 

intervention). 

• WINS consolidates claim data for all Child Nutrition Programs, enhancing reporting accuracy and 

ensuring School Food Authority (SFA) demographic characteristics are consistently reported. 

Planning and Implementation

Management

The State director, supervisors, fiscal staff, project manager, and child nutrition business analyst were all 

involved in writing the grant application; SFAs were not. An Executive Steering Committee and OSPI’s 

budget analyst were also involved in grant management. 

The State worked with a vendor for all but the final year of this project. Unfortunately, the vendor could 

not meet the needs of the complex system. In the final year of the contract, OSPI negotiated an end to that 

contract and the work was taken over by an internal IT team. Another vendor has been contracted to 

manage the maintenance and operations of WINS. 

Funding

Washington was awarded $1,478,700.00 through the ART grant and used 100% of the funds to complete 

its project. A significant portion of the grant funds were used for project management. In addition, it used 

State Administrative Expenses (SAE) funds to expand this system to cover the Child and Adult Care 

Food Program. The SAE funds continue to be used to cover maintenance and operations. 

 Initial Grant Amount: $1,478,700.

 Activities Funded by the Grant: This grant funded the replacement of CNP 2000 with WINS. 

In addition to funding for OSPI project and budget management, and the vendor services, 

the SA also hired a business analyst to build a list of specifications for the new data system 

and an external project manager to manage grant progress. The grant also funded internal 

trainings on the new system for LEAs and technical assistance as needed when the new 

system was launched.

 Additional Funding Sources: State Administrative Expenses (SAE).

 Funds Returned: $0.

Successes

The grant achieved the goal of creating a unified application and claims system that reduces data entry 

and errors for LEAs undergoing administrative reviews. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌81

• Administrative Costs: State staff time spent working on administrative reviews has been reduced 

and their tasks have been simplified. This cost is difficult to capture accurately, as this grant was 

implemented at the same time as a number of other changes, such as the move from the former 

Coordinated Review Effort (CRE) to the new administrative review.

• Error Rates in Application Processing: WINS processes applications at the site level rather than at 

the program level, and it retains information that would otherwise be entered multiple times for sites 

that run multiple programs. The SA and LEAs reported fewer errors as a result of streamlining data 

entry.

• Training and Professional Standards: The SA took a proactive approach in providing in-person 

training on the new system for LEAs. It developed documentation on how to use the new system. The 

SA also provides ongoing technical assistance.

• Data Systems: In addition to processing site-based applications and claims, WINS compares ongoing 

claims with October building data to flag potential errors in real time and allows for ad hoc reporting 

to allow State staff to analyze problem areas and errors. LEAs report that the application and claims 

process is easier in WINS than in the previous system and is very straightforward. 

Challenges

Washington encountered challenges with a vendor that did not have the resources to keep up with the 

demands of the project, resulting in four no-cost extensions, higher project management costs than 

anticipated, and the State ultimately bringing the system in-house. 

• Data Systems: Once the development of WINS was underway, the SA learned that it would not be 

beneficial to integrate data across multiple systems, for a variety of reasons. In some cases, data were 

collected at different times and for different purposes; in other cases, the LEAs were required to 

collect and report certain data even if they could be pulled from other systems. There are several 

enhancements (other than integration) that the SA has pursued after the implementation of the ART 

grant, including some that were originally planned for the ART grant period but the project schedule 

and funds ran out before completion. These include adding electronic verification and the ability of 

LEAs to upload claims.

• Working with a Vendor: The vendor that was originally contracted to create WINS did not have the 

resources to keep up with testing and development at the same time. Despite several efforts to add 

resources and get the development back on course, the SA negotiated an end to the contract before the 

project was completed. 

• Cost of Project Manager: The cost of hiring a project manager to work on this project was very

high, in part due to local economic factors and in part due to project delays from the vendor. In the 

end, the SA spent more of the grant budget on project management than initially planned. 

• Communications: There were communication challenges among the many stakeholders involved 

(including the vendor, the project manager, and multiple State agencies). Sometimes decisions were 

made without involving all parties, and sometimes communication was difficult because IT and 

business staff did not always know what questions to ask each other. Midway through the project, a 

business team with subject matter experts was assembled to improve communication and help 

everyone get on the same page.

Lessons Learned

The State learned that including representatives from both the IT and business sides in project planning, 

evaluation, and decision-making can help prevent miscommunications that can turn into bigger issues 

over time. Project management is key, as is ensuring staff time commitment especially on a large IT 

project where there is a need for collaboration between technical and business staff. The State also 

learned that having all the players in place early in the planning and development process (including LEA 

staff to help pilot new system developments) is very beneficial.
Abt Associates CN ART Grants | Final Study Report Final ▌82

Overview

Grant Start Year: 2010 Grant End Year: 2015

Purpose: Customization of nutrition program 

management software

Extension(s): Yes

Project Name: School Nutrition Accountability System 

(SNACS)

Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $1,960,526.00 Grant Funds Returned: $207,806.36

The Wisconsin Department of Public Instruction (DPI) received an Administrative Review and Training 

(ART) grant in 2010 to purchase and implement the School Nutrition Accountability System (SNACS). 

SNACS is an accountability system designed to help schools reach full Child Nutrition Program (CNP) 

compliance. Integration with existing DPI systems and applications improves accuracy of reimbursement 

claims and allows DPI to address administrative errors by School Food Authorities (SFA) as they occur, 

rather than errors compounding between review cycles. SNACS also provides all SFAs with the ability to 

use online applications. 

Goals and Objectives

Grant Application Post-Grant Implementation

• Provide technical assistance and training to schools at risk of 

administrative errors and help LEAs reach full CNP program 

compliance. 

• Purchase and host accountability software. This software will 

allow SA staff to identify and resolve errors as they occur 

and will target the reduction of errors in small LEAs. The 

update will improve the accuracy of claims by integrating the 

existing DPI systems and applications, and will provide 

online applications for all LEAs.

• The goals and objectives did not change from the 

grant application to implementation phase.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Wisconsin’s ART grant was awarded to foster school compliance with CNP regulations. The State 

customized an off-the-shelf system for online applications, claiming, verification, and administrative 

reviews. At the time of data collection for this report, Wisconsin had not conducted administrative 

reviews using SNACS due to the delays with the system. Activities funded by the ART grant achieved the 

following: 

• Development of an online CNP system that interfaces with existing DPI systems. 
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌83

• Development of an online application and verification system. The parent application is available at 

no cost 24 hours a day, 7 days a week for all SFAs. 

• Development of an online claim submission system. The system creates reminders, does claim 

comparisons, and suggests eligibility.

• Development of an automated administrative review process that allows error identification in real 

time. 

• Improved reporting; reports created for each module within the new system to monitor and 

benchmark the SFAs.

Planning and Implementation

Management

In addition to the DPI director, Wisconsin hired a contracted IT employee as project manager, and 

subsequently hired a project lead to manage the project. The State contracted with the software vendor to 

develop and implement SNACS. It also engaged nutrition program consultants as subject matter experts. 

Funding

In 2010, Wisconsin received ART grant funding to customize a technology solution and offer technical 

assistance to error-prone LEAs. The State was initially awarded $1,960,526 to carry out these activities, 

and it returned 11% of its awarded funds at the end of the grant period. The project required three no-cost 

extensions and used State Administrative Expense dollars to fund the project manager position. 

Successes

The grant activities were successful overall, as they developed a system that complies with USDA 

regulations and integrates with existing DPI systems. The system created an interface between the online 

contract and claiming system and SNACS accurate claims submission. SNACS automates the 

administrative review process and helps identify errors as they occur. 

• Administration of School Nutrition Programs: The free online application tool is now available 

through SNACS. Many small SFAs that previously could not afford an online application have taken 

advantage of it and have found the verification process to be less time-consuming. 

• Error Rates in Application Processing: The availability of the free online application tool has 

reduced application errors and expedited application processing. Error rates have declined, but staff 

cautioned that some schools have implemented the Community Eligibility Provision, which itself 

likely reduces the State’s administrative error rates. 

• Data Systems: Wisconsin created a system that complies with USDA regulations, interfaces with the 

State’s existing online system, and creates reports allowing the State to monitor SFAs. At the time of 

data collection for this report, approximately 40 of the State’s SFAs were using SNACS. 

Challenges

Though some the goals of the project were achieved, there was miscommunication with the vendor before 

and during the implementation of the grant, which significantly delayed the completion of grant activities. 

 Initial Grant Amount: $1,960,526.00.

 Activities Funded by the Grant: Customization of CNP management software (SNACS).

 Additional Funding Sources: State Administrative Expense funds.

 Funds Returned: $207,806.36.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌84

• Project Management: The State fired the first project manager, who was replaced by a project lead 

who was responsible for managing the project. There were personality conflicts between the State 

management and subject matter experts and the vendor, leading to project delays and staff turnover. 

The State had little leverage over the software vendor’s staff and schedule, noting there were long 

delays in system development that the State could not control. 

• Data Systems: The State did not clearly define requirements during the procurement process. 

Consequently, its expectations for the final product were not aligned with the vendor’s understanding. 

The State continued to address bugs in the administrative review module after the grant ended. The 

State owns the source code, but it has been challenging for staff to understand the programming logic, 

making it difficult to modify and further develop the system to meet the State’s needs. SFAs 

identified glitches in the application and verification modules that continued to be issues at the time 

of data collection for this report. 

• Communication: Poorly defined requirements during the procurement phase led to tension between 

the State and vendor, contributing to communication challenges throughout the contract. The State 

believes the vendor was not sufficiently responsive to the State’s timetable and needs. This belief was 

exacerbated by insufficient direct contact with the vendor staff developing the product. State staff 

would have preferred face-to-face communication with the software vendor’s developers rather than 

with the corporate representatives who came on-site.

• Administration of School Nutrition Programs: SFAs using PowerSchool or other point-of-sale 

systems must double enter student data, as the two systems do not communicate. 

Lessons Learned

Wisconsin learned that during the procurement process, it is critical for the RFP to specify all system 

functionality and testing requirements to ensure all expectations are clear and then later to minimize 

misunderstandings during system development and rollout. The State also learned that when hiring 

project staff, it is important to assess that both technical and interpersonal skills align with project needs. 

Wisconsin’s experience also highlighted the importance of including in-house IT programmers in early 

phases of planning and development, so they gain an understanding of the coding logic and can provide 

support after the contract with the software developer expires. 
Abt Associates CN ART Grants | Final Study Report Final ▌85

Overview

Grant Start Year: 2009 Grant End Year: 2013

Purpose: Customize technology for the AR process Extension(s): Yes

Project Name: Wyoming ART Vendor: Outside, for-profit software vendor

Grant Funds Awarded: $482,500 Grant Funds Returned: $160,314.37

In 2009, Wyoming received an Administrative Review and Training (ART) grant to develop and 

implement a new technology system to identify, reduce, and prevent administrative error for the following 

programs: National School Lunch Program, School Breakfast Program, Afterschool Care Program, Fresh 

Fruit and Vegetable Program, USDA Food Distribution Program, Special Milk Program for Schools, and 

the Summer Food Service Program. The new system allows the State Agency (SA) to identify at-risk 

Local Educational Agencies (LEAs) through real-time data analysis. SAs are able to use monitoring 

information to ensure corrective actions are implemented upon completion of administrative reviews. The 

new technology also allows the SA to provide appropriate training for LEA staff to prevent repeated 

administrative errors.

Goals and Objectives

Grant Application Post-Grant Implementation

Goals & Objectives

• Customize a new technology system to identify errorprone/at-risk LEAs, by comparing at-risk LEAs that have not 

met benchmarks to peer districts.

• Provide corrective action and training to LEA employees to 

prevent future errors. 

Goals & Objectives

• The goals and objectives did not change from those 

stated in the application.

The following sections include information from grant applications, final grant reports, and discussions 

with State Agency (SA) and Local Educational Agency (LEA) staff members about their experiences 

related to the ART grant and its associated activities.

Highlights

Wyoming’s ART grant was awarded to develop review modules within a new technology system for 

Child Nutrition Programs to (1) Identify at-risk districts through real-time data analysis; (2) add the 

ability to modify forms, reports, interfaces, and outputs to improve program integrity and accuracy; (3) 

use benchmarks as a tool to assess program performance and implement corrective action where 

warranted; (4) establish a Child Nutrition database to compare at-risk districts with peer districts and 

implement standards for evaluation; (5) provide training for LEAs to prevent repeated administrative 

errors; and (6) reduce risk levels for administrative errors. The activities funded by the ART grant 

achieved the following:

• Developed and implemented new technology and software for accessing and reviewing LEA data to 

assist with identifying LEAs that have a high level of (or are at risk for) administrative error. The 

system allows the SA to support and monitor corrective actions as errors are addressed.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌86

• Implemented targeted monitoring within the new system to identify training needs for LEAs, 

specifically tailored to prevent repeated administrative errors. 

Planning and Implementation

Management

The SA director and finance specialist managed and coordinated the project tasks. The SA director 

worked with the finance specialist to develop the RFP for the technology vendor. The Wyoming SA staff 

team is relatively small, so the director took on most of the project management activities, including 

establishing a timeline, progress reporting, and managing the software vendor. At times, other SA staff 

would assist, but their participation with the project tasks was limited. 

Funding

An ART grant was awarded to Wyoming for $482,500 to implement a technology solution with the 

objective of using benchmarks as a tool to assess program performance, establish a database to compare 

at-risk districts with peer districts, and provide targeted training and technical assistance to LEA 

employees to prevent repeated errors. The State also relied on State Administrative Expense funds. 

 Initial Grant Amount: $482,500.

 Activities Funded by the Grant: Customized software and trainings.

 Additional Funding Sources: State Administrative Expense funds. 

 Funds Returned: $160,314.37.

Successes

Wyoming’s ART grant allowed the SA to implement a system to reduce administrative errors with LEAs, 

particularly those at higher risk for producing such errors. Additionally, the system immediately flags 

trends that suggest technical assistance and training are needed for specific LEAs.

• Administrative Costs: The new system identifies at-risk districts with a module automatically 

producing red flags in each area indicated as a potential issue. Administrative reviewers are able to 

use time on-site with LEAs to target specific problem areas and implement corrective actions needed.

• Error Rates: The review module creates all required USDA reports, and the claiming module 

interfaces directly with the Wyoming State Government Payment System. Human error has decreased 

significantly with the automation available with the new technology. 

• Training and Professional Standards: Training is customized based on the errors reviewed within 

the system. SA staff are able to quickly identify training needed and monitor progress through the 

new technology.

• Administration of School Nutrition Programs: A single database for Child Nutrition programs 

allows the SA to identify at-risk districts and the strengths and weaknesses of the programs on a 

statewide basis. As a result, the SA has experienced an increase in data integrity and has been able to 

establish statewide standards.
APPENDIX A: GRANTEE PROFILES

Abt Associates CN ART Grants | Final Study Report Final ▌87

Challenges

Wyoming achieved the goals and objectives for the ART project; however, to do so required a no-cost 

extension and the State experienced the following challenges with the project:

• Administration: The SA director was, at times, overwhelmed. The ART grant project required 

consistent and detailed oversight, and the SA director was managing the ART project in addition to 

the day-to-day duties required of the SA director. 

• Training and Professional Standards: The training was developed as online modules. The SA did 

not anticipate the lack of technology skills of LEA staff. Basic computer tutorials were conducted 

with LEA staff to position them to take advantage of the online training sessions.

• Data Systems: The testing for the new system was onerous and took more time than originally 

anticipated. Due to the small number of SA staff, the SA director conducted most of the testing and 

provided feedback to the technology vendor. 

Lessons Learned 

The Wyoming ART grant project would have greatly benefitted from having a dedicated project manager. 

The project required detailed organization and oversight to ensure tasks were completed accurately and 

were timely. The SA director was often pulled off overall department functions to focus on the project. 

This caused a significant amount of stress, and project activities were delayed because the director had 

competing duties (other than the ART grant project) to perform. 

The amount of time and work it takes to apply for a grant can be overwhelming for a small State team. If 

Wyoming were to apply for a grant in the future, it would look to partner with a local university to help it 

develop its application. 
APPENDIX B: RESEARCH QUESTIONS

Abt Associates CN ART Grants | Final Study Report Final ▌88

Appendix B. Research Questions

1. Describe the interventions funded by the ART grants. 

a. Why were these interventions chosen?

b. How did the interventions and activities as implemented track with the original grant 

proposal? 

2. For those ART Grantees whose projects are related to administrative review, how did the State 

collect, aggregate, and report data to FNS for administrative reviews and for NSLP and SBP prior 

to the intervention(s)? 

a. What data systems and software were used?

b. How did interventions change the administrative review process and data systems at the 

State level? At the LEA level?

3. What were the intended measurable outcome(s) of the interventions?

a. How do States perceive the outcomes of the grant?

b. How successful were the interventions funded by ART grants in terms of their intended 

outcomes? 

c. If possible, what impact did the intervention have on error rates? 

4. How did ART Grants impact the perceived efficiency of administrative staff?

a. Did ART Grant interventions result in a net change to staff time on certain activities?

b. What impact did the interventions have on State administrative costs?

5. If ART Grants were used to improve the Administrative Review (AR) process, what impact did 

the intervention have on the State conducting the AR process? 

a. Did State and LEA staff spend more or less time on administrative review processes and 

other activities related to the AR process during and after the grant period than they did 

on these activities prior to ART Grant funding? 

6. If ART Grants were used to improve direct certification processes, what impact did the 

intervention have on State direct certification rates?

7. If ART Grants were used for training on program requirements and professional standards, what 

impact did the intervention have on the State meeting these requirements?

8. If ART Grants were used to improve direct certification processes, what impact did this have on 

claiming and eligibility under the Community Eligibility Provision (CEP)?

9. If grant funding was used to improve data systems or general administrative processes with wide 

applicability, how did interventions funded by ART Grants also impact administration of other 

programs beyond SBP and NSLP, such as the Summer Food Service Program (SFSP) and the 

Child and Adult Care Food Program (CACFP)? 

10. What challenges did grantees face in the implementation of their interventions?

a. How did they overcome these challenges?

11. Did intervention proceed according to schedule? In the case of delays, what were the causes? 

How many no-cost extensions were necessary to fully implement the intervention? Why were 

these extensions necessary?
APPENDIX B: RESEARCH QUESTIONS

Abt Associates CN ART Grants | Final Study Report Final ▌89

12. Do any challenges from the ART Grant interventions linger? If so, what steps are States taking to 

overcome these?

13. What facilitators aided grantees in implementation of their interventions? How did they identify 

these facilitators?

14. How were grant funds utilized? What other funding sources were used for the ART Grant 

interventions? 

15. How did States sustain their ART grant interventions after grant funding ended? How were State 

Administrative Expense (SAE) funds used to maintain projects, if at all?

16. How can USDA administrative review guidance be improved to take into account lessons learned 

from ART Grant Interventions? 
APPENDIX C: EXTANT DATA REVIEW PROTOCOL

Abt Associates CN ART Grants | Final Study Report Final ▌90

Appendix C. Extant Data Review Protocol

These reviews began with the FNS ART Grant Summary of Best Practices, the FNS request for 

application, grant application reviews, and then included a review of the final progress report, if it existed. 

The protocol is organized in four sections: (1) Preliminary Questions; (2) Grant Applications; (3) Final 

Progress Report; and (4) Challenges and Outcomes.
APPENDIX C: EXTANT DATA REVIEW PROTOCOL

Abt Associates CN ART Grants | Final Study Report Final ▌91

C1. Preliminary Questions
APPENDIX C: EXTANT DATA REVIEW PROTOCOL

Abt Associates CN ART Grants | Final Study Report Final ▌92

C2. Grant Applications
APPENDIX C: EXTANT DATA REVIEW PROTOCOL

Abt Associates CN ART Grants | Final Study Report Final ▌93

C3. Final Progress Report
APPENDIX C: EXTANT DATA REVIEW PROTOCOL

Abt Associates CN ART Grants | Final Study Report Final ▌94

C4. Challenges & Outcomes
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌95

Appendix D. Interview Protocols

There are three interview protocols: (1) State Agency; (2) Local Education Agency; and (3) State Agency 

Administrative Reviewers. 
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌96

D1. State Agency Interview Protocol

My name is [name], and I’m a researcher at [Abt Associates/Insight Policy Research]. My 

colleague, [name], is also present to take notes throughout the interview. As you may know, we 

are conducting a study of the Administrative Review and Training (ART) Grants for the Food 

and Nutrition Service (FNS) of the U.S. Department of Agriculture. The primary purpose of this 

study is to better understand the ART grant interventions and to gather information about 

perceived results and implementation challenges. Additionally, we are interested in finding out 

how States sustain the grant-funded activities once the grant has ended. Your candid responses 

can help identify opportunities for improving the effectiveness of the ART grants and inform 

FNS about the ways these grants have influenced the administration and delivery of child 

nutrition programs among grantees.

We are conducting interviews with a total of 20 States for this study. When we complete the 

interviews, we will summarize your responses with those provided by the other States in a final 

report for FNS. We will not use any names in that report or identify any individual respondents. 

We expect our conversation will take between 45 and 60 minutes. Do you have any questions for 

me about the project in general or what we will be discussing today? 

With your permission, we would like to record the conversation to ensure our notes accurately 

reflect your responses. Do I have your permission to record our conversation? 

Confirm permission once recording starts. Note the State Agency name and date and time of 

the call.

o Yes

o No

State Agency

Interview Date

Time Start

Time End
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌97

Let’s begin with introductions. 

Please provide your name, job title, role in the Administrative Review process generally, and 

role on ART grant funded activities in particular, including how long you have been involved in 

that role.

Name Title

Role in 

Administrative 

Review Process

Role with 

ART Grant

Time in Role

(Years/Months)

Planning and Implementation

Extant Data Summary Note Changes Given by 

Interviewee

State 

Year of Grant Award

Year of Grant Completion

Intervention Strategy(ies)

Goals/Objectives

Performance Measures

Grant Activity Summary

NOTE TO INTERVIEWER: The table above will be populated prior to the interview with data 

from the application, progress reports, final report, and the ART Grant Summary of Best 

Practices.

1. As we begin, we are referring to the ART Grant funded in year (verify from table above) 

and completed in (refer to table above). 

Based on my review of your ART grant application, I understand your State Agency 

applied for this grant for the following reasons (briefly summarize reasons/targeted issues 

discussed in grant application). Would you say my understanding is correct (as far as you 

remember)?

a. If not, can you tell me in your own words why your State Agency applied for the 

ART grant in the first place?
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌98

2. Let’s talk about the project that was implemented with ART grant funds and the issues the 

project was intended to address.

a. Prior to applying for the ART grant, who was involved in the decision-making to 

apply for the ART grant? Who was involved in the grant application process? 

(PROBE: If SFAs were not mentioned, ask:) Were SFAs involved in the grant 

planning or application process? What role did they play? 

b. After the ART grant was awarded what was the process for managing the ART 

grant project? How did you monitor the project tasks, schedule, and budget? 

3. We are interested in how States and their SFAs dealt with issues or challenges during 

implementation. (RQ4)

a. What issues did you encounter during implementation? 

b. How were these issues addressed?

c. Were the steps that were taken to address those issues effective? In other words, 

did they resolve the problems being targeted? Did they make it easier to implement 

the project afterwards?

d. What issues did your SFAs encounter during implementation? 

e. How were these issues addressed?

4. Oftentimes, project plans require some tweaking during implementation. How did your 

project plans or activities change between planning and implementation, if at all? (RQ1a)

5. Did the project progress according to the original schedule? If not, what caused the delay? 

(RQ4a)

a. Were any extensions needed? If so, how many? For what reasons? Did you receive 

any additional funding or were these no-cost extensions? (RQ4b) 

6. How did you evaluate the project’s successes? Who was involved in that process and what 

was their job title and role within the organization? 

7. We are also interested in getting a better understanding of how grant funds were utilized 

and whether you used any other funding sources to implement or sustain grant activities. 

a. Can you provide an overview of how the grant funds were allocated? (NOTE TO 

INTERVIEWER: If reports include this information, verify that the information 

in the reports is complete and up to date.) (RQ6)
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌99

b. What additional funding sources, if any, did you use to support grant 

implementation? (NOTE TO INTERVIEWER: Ask the questions below if 

additional funding sources were used.)

i. How were these additional funds used? 

ii. How long were these funds used (for example, duration of grant, when 

grant monies ran out, a specified number of months, etc.)? 

iii. Are these funds still available? Are you still eligible to apply for these 

funds in the future? (RQ6A)

Short- and Long-Term Effects

8. Now I want to ask a few questions about the specific procedures or activities your grant 

targeted, and the outcomes. 

a. (FOR GRANTS PROJECTS RELATED TO ADMINISTRATIVE REVIEW:)

Would you please briefly describe the process for Administrative Reviews prior to 

the intervention? We are interested in processes directly addressed by the 

Administrative Review such as the off-site and on-site review process, the 

corrective action process, and any fiscal action process, if applicable. (RQ2)

What tool was used to collect the Administrative Review information from SFAs 

(for example, Excel spreadsheets provided by USDA, paper forms, etc.)? 

i. How did your project change the Administrative Review process at the 

State level? (RQ2a, RQ3e)

ii. How did your project change the Administrative Review process at the 

SFA level?

b. Do you think State staff spent more or less time on the Administrative Review 

process and other activities related to the Administrative Review process during 

and after the grant than they did on these activities prior to the grant? What about 

SFA staff? (RQ3e)

i. (PROBE: If the new process requires additional time:) Do you think the 

new Administrative Review process will go faster for the State staff or SFA 

as they become more familiar with it?)

c. In general, what impact did the project have on State administrative error rates? 

(RQ3a)
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌100

d. How do you think the intervention affected program staff, at the State Agency 

level and the SFA level? (RQ3b) 

i. Did it simplify the work of State Agency staff or make it more 

complicated?

ii. How about SFA staff? 

iii. Do you think State Agency staff spend more or less time on administrative 

tasks targeted by the project than they did before funding was received? 

iv. How about SFA staff? 

e. (FOR GRANTS INTENDED TO REDUCE ADMINISTRATIVE COSTS:) What 

savings or increased administrative costs have you seen as a result of the ART 

grant intervention? (RQ3c)

i. Was there a specific area or process you were targeting to reduce costs? 

(PROBE: If targeted area was mentioned with an observed decrease in 

costs:) Did you see costs increase in any other area despite decreased costs 

in the targeted area as a result of ART grant activities?

ii. At what point would you say you started to see those savings/increased 

costs (e.g., how long after implementation of the project)? (NOTE TO 

INTERVIEWER: Probe from beginning of grant period.)

iii. What specific factors contributed to those costs or savings? 

iv. Do you have any evidence, such as the time necessary to complete the 

Administrative Reviews or changes in the number staff needed to perform 

Administrative Reviews that demonstrate these effects or is this based on 

your estimation?

f. (FOR GRANTS INTENDED TO IMPROVE DIRECT CERTIFICATION 

PROCESSES:) What impact, if any, did the project funded with the ART grant 

have on direct certification rates? (RQ3f)

i. What about the direct certification procedures? 

ii. What changes, if any, have you seen since the grant was implemented? 
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌101

iii. What effects, if any, did the grant have on claiming and eligibility under 

the Community Eligibility Provision (CEP)? (RQ3h)

g. (FOR GRANTS USED FOR TRAINING ON PROGRAM REQUIREMENTS 

AND PROFESSIONAL STANDARDS:) What outcomes were you hoping to see 

as a result of the training funded by your ART grant? (RQ3g)

i. Which of those outcomes were successfully accomplished by the end of the 

grant?

ii. How about since the grant ended? (PROBES: What changes on SFA error 

rates, if any, have you seen; for example, meal counting and claiming, 

DC matching, verification, etc.)?

iii. What are some other outcomes of the training you think we should know 

about? These may include positive or negative outcomes that were not 

specifically targeted.

iv. What do you consider the most important outcomes of the training? What 

outcomes did you see at the SFA level? At the State level?

9. The ART grants targeted the School Breakfast Program and the National School Lunch 

Program; however, we are interested in exploring impacts the grants may have had on the 

administration of other child nutrition programs. Do you administer either the Child and 

Adult Care Food Program or the Summer Food Service Program? (RQ3i)

a. Please describe any impact on your school food service program administration of 

the Child and Adult Care Food Program or the Summer Food Service Program due 

to the implementation of the [local project name]. (NOTE TO INTERVIEWER: 

Probe for unintended consequences.)

Outcomes and Challenges

10. From your perspective, what were the most important outcomes of the ART grant-funded 

intervention (s), and how do these compare to the anticipated outcomes you discussed 

earlier? (RQ3) (NOTE TO INTERVIEWER: These may be positive or negative and you 

may probe accordingly.)

a. Did you see any unanticipated outcomes? 

11. How long after implementation did you first begin to see results from the project funded 

with the ART grant? (RQ3)
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌102

12. From your perspective, what were the top three challenges your State Agency experienced 

with grant activities before and during the implementation of the ART grant? How did you 

address these? (NOTE TO INTERVIEWER: Probe for specifics as it relates to what has 

been discussed about the State Agency’s objectives for the grant activities.)

a. What are some lingering challenges following implementation of ART grant 

funding? What plans, if any, are in place to address those challenges? Have you 

applied or are you currently considering applying for another ART grant in order 

to address these lingering challenges? (RQ4c)

Lessons Learned

13. If you were to apply for another ART grant what would you do differently? (NOTE FOR 

INTERVIEWER: Probe for during the planning phase, implementation phase and 

closeout phase.)

Wrap-up

14. Is there anything we did not ask that you think is important for us to know, or you were 

hoping we would discuss? 

15. I realize that SFAs may have multiple ongoing projects at any given time, but we want to 

make sure the people we speak with focus specifically on ART grant-funded activities 

when answering our questions. How would you suggest we ask our questions so that 

respondents understand we are asking specifically about activities funded by the ART 

grant? 

Local Project Name

Thank you for answering my questions. 

That completes today’s interview. We may select State Agencies to further discuss details about 

challenges, successes and lessons learned for future grants. Would you be the best person to 

identify staff who may have expertise/experience with specific ART grant activities, such as 

training development, IT solutions, etc. for a future discussion? 

o Yes.

o No. Contact_______________________________________________ 

Thank you again. Have a nice day.

Stop recording and note time interview concluded.
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌103

D2. Local Educational Agency Interview Protocol

NOTE TO INTERVIEWER: The SFA may know the intervention/project by a name other 

than “ART Grant.” Prior to the interview(s) with SFA staff, verify data collected from State 

Agency.

Local Project Name

My name is [name], and I’m a researcher at [Abt Associates]. My colleague, [name], is also 

present to take notes throughout the interview. As you may know, we are conducting a study of 

projects funded by Administrative Review and Training grants, or ART grants. Your State 

received an ART grant in (insert year) to help fund the (insert project name).

The purpose of this study is to gather additional information about these projects, including their 

results and implementation challenges. Your candid responses can identify opportunities for 

improving the effectiveness of these projects and help FNS better understand their effects on the 

administration and delivery of child nutrition programs. 

We are conducting interviews with a total of 20 States and 40 SFAs for this study. When we 

complete the interviews, we will summarize your responses with those provided by others in a 

final report for FNS. We will not use any names in that report or identify any individual 

respondents. 

We expect our conversation will take between 45 and 60 minutes. Do you have any questions for 

me about the study or what we will be discussing today? 

With your permission, we would like to record the conversation to ensure our notes accurately 

reflect your responses. Do I have your permission to record our conversation? 

Confirm permission once recording starts. Note the SFA name and date and time of the call.

o Yes

o No

Local Education Agency 

Interview Date

Time Start

Time End
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌104

Let’s begin with introductions. 

I would like to confirm your names and job titles and ask your role in the Administrative Review 

process.

Name Title Role in Administrative 

Review Process

Length of Time with 

the AR Process

Planning and Implementation

Extant Data Summary

State

Year of Grant Award

Year of Grant Completion

Intervention Strategy(ies)

Goals/Objectives

Performance Measures

Grant Activity Summary

NOTE TO INTERVIEWER: Prior to the interview, the table above will be populated with data 

from the application, progress reports, and final report. Questions preceded by [For SFA] are 

best answered by the SFA/Food Service Director (or their designee). If we speak with someone 

other than the SFA/Food Service Director, we should ask the designated interviewee to gather 

the requested information from the SFA/Food Service Director prior to the interview.

1. [For SFA] I’d like to start with a few questions about your school food service program’s 

level of engagement in the [local project name]. 

a. [For SFA] How, if at all, were you involved in the grant planning or grant application 

process?

b. [For SFA] How did the State Agency involve you in other phases of the project, such 

as testing, reviewing training materials, etc.?

c. [For SFA] What information did the State Agency share with you about the goals of 

the [local project name] and the plans for reaching those goals? 

d. [For SFA] How did the State Agency share information about the [local project 

name] with you? Probe: For example, via newsletter, email, or through training? 
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌105

e. How could the State Agency have communicated more effectively with your school 

food service program about this project? 

Short- and Long-Term Effects

Now I want to ask a few questions about the [local project name] and how it affected your 

school food service program. 

2. (NOTE TO INTERVIEWER: If applicable to intervention(s):) [For SFA] Have you 

been through an Administrative Review since the [local project name] was implemented? 

(If yes:) Please briefly describe the Administrative Review process prior to, and 

following, the implementation of the [local project name]. We are interested in processes 

directly addressed by the Administrative Review such as the off-site and on-site review 

process, the corrective action process, and any fiscal action process, if applicable. (RQ2)

a. Do you think these changes were helpful or an improvement over the previous 

process?

3. Did your SFA staff receive any training on the new Administrative Review process as a 

result of the ART grant intervention? What did the training cover? How did it help? 

(RQ3g)

a. What topics have you or staff participated in? In what format was the training 

provided (e.g., in-person training, State conference, webinar, on-line training)?

b. What effect has the training had on errors in the administrative process for SFAs (for 

example, menu pattern compliance, meal counting and claiming, certification)? 

(RQ3a)

c. What feedback have you received on the training and its effectiveness?

d. What recommendations, if any, do you have for improving the training?

4. [For SFA] From your perspective, how has the [local project name] affected your 

program operations? (NOTE TO INTERVIEWER: These may be positive or negative, 

probe accordingly. Also probe specifically for staff time if not mentioned by 

interviewee.)

a. [For SFA] What tasks are simpler or less time-consuming for staff as a result of the 

[local project name]? (RQ3b)

b. [For SFA] How did you determine it was more or less time consuming?

c. (If not already addressed:) Have you experienced any changes in the administrative 

level of effort since the (local project name) was implemented? (RQ3c)
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌106

5. (IF THE ART GRANT WAS USED FOR TECHNOLOGY:) [For SFA] How did staff 

respond to the new system implemented under the grant? 

a. [For SFA] How has the use of new technology affected the level of effort? Do you 

think it saves time? Did it help to streamline procedures? (PROBE: What activities 

increased or decreased the level of effort (e.g., meal claims) through the ART 

project?)

b. What type of training and/or guidance was provided to the SFA from the State 

Agency? Was it helpful? Are there improvements that could have been made to how 

the training was provided? 

6. [For SFA] What was the initial opinion by staff of the new technology? Did their opinion 

change over time? (IF THE GRANT WAS USED TO IMPROVE DIRECT 

CERTIFICATION:) Do you think the direct certification process became easier as a 

result? How so? (RQ3f)

7. Does your school food service program participate in the Community Eligibility Provision 

(CEP)? If so, what effect did project activities have on claiming and eligibility? (RQ3h)

Outcomes and Lessons Learned

We want to ask about the positive and negative results of the [local project name]. 

9. From your perspective, what were the major positive results of the [local project name]? 

What were some negative results of the [local project name]? (RQ3)

a. About how long did it take to see those results? 

Challenges and Facilitators of Success

We are particularly interested in learning about challenges related to the [local project name] as 

well as any facilitators of success. (RQ4)

10. From your perspective, what were the top challenges your school food service program 

experienced during or after implementation of the [local project name]? 

11. (NOTE FOR INTERVIEWER: Relate to specific grant focus:) How did you address 

those challenges?

12. How did your school food service program get program staff buy-in for the [local project 

name] and any associated changes or activities?

13. What type of support did the State Agency offer during implementation of the [local 

project name] (e.g., technical assistance, training, manuals, etc.)? At what point in the 

[local project name] was assistance made available to your school food service program? 
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌107

Did your school food service program have to request the assistance or did the State 

make it available as part of the project?

14. How effective was the assistance provided by the State? What recommendations, if any, 

do you have for improving the assistance offered? 

15. [For SFA] How has the [local project name] assisted you in better managing your school 

nutrition program? 

a. [For SFA] What, if anything, was most helpful? 

b. [For SFA] What challenges were not addressed by the [local project name]? 

Wrap-up

16. Is there anything we did not ask you think is important for us to know, or you were 

hoping we would discuss? 

Thank you for answering my questions. Have a nice day. 

Stop recording and note time interview concluded.
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌108

D3. State Agency Administrative Reviewers Interview Protocol

My name is [name], and I’m a researcher at Abt Associates. My colleague, [name], is also 

present to take notes throughout the interview. As you may know, we are conducting a study of 

the Administrative Review and Training (ART) Grants for the Food and Nutrition Service (FNS) 

of the U.S. Department of Agriculture. The primary purpose of this study is to better understand 

the ART grant interventions and to gather information about perceived results and 

implementation challenges. Additionally, we are interested in finding out how States sustain the 

grant-funded activities once the grant has ended. Your candid responses about how the 

Administrative Review (AR) Process works in your State, both before and after the 

implementation of the ART grant, can help identify opportunities for improving the 

effectiveness of the ART grants and inform FNS about the ways these grants have influenced the 

administration and delivery of child nutrition programs among grantees. 

We are conducting interviews with a total of 20 States for this study; 10 of those were selected 

for interviews about their AR process. When we complete the interviews, we will summarize 

your responses with those provided by the other States in a final report for FNS. We will not use 

any names in that report or identify any individual respondents.

We expect our conversation will take 30 – 45 minutes. Do you have any questions for me about 

the project in general or what we will be discussing today? 

With your permission, we would like to record the conversation to ensure our notes accurately 

reflect your responses. Do I have your permission to record our conversation? 

Confirm permission once recording starts. Note the State Agency name and date and time of 

the call.

o Yes

o No

State Agency

Interview Date

Time Start

Time End
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌109

Let’s begin with introductions. 

Please provide your name, job title, role in the Administrative Review (AR) process and how 

long you have been involved with AR.

Name Title

Role in 

Administrative 

Review Process

Role with 

ART Grant

Time in Role

(Years/Months)

We have already spoken to [INSERT NAME OF SA INTERVEWEE] and learned about 

[INSERT NAME OF INTERVENTION], funded by the grant awarded in [20XX]. We want to 

learn more about how ARs are conducted in your State. 

1. Approximately how many ARs have you conducted in this State since [INSERT NAME OF 

INTERVENTION], funded by the grant awarded in [20XX] was implemented? 

2. Would you please briefly describe the process for ARs prior to the implementation of 

[INSERT NAME OF INTERVENTION] for [Grant awarded in 20XX]? We are interested in 

processes directly addressed by the AR such as, the off-site and on-site review process, the 

corrective action process, and any fiscal action process, if applicable. 

a. What tool was used to collect the AR information from LEAs (for example, Excel 

spreadsheets provided by USDA, paper forms, etc.)? 

b. Who at the State Agency (SA) was involved in the AR? What were their roles? How 

many reviewers did the State have? Were they SA employees or contractors?

3. Now, we will discuss the current AR process. We are interested in processes directly 

addressed by the AR such as the off-site and on-site review process, the corrective action 

process, and any fiscal action process, if applicable. 

a. What tool(s) is NOW used to collect the AR information from LEAs (for example, 

Excel spreadsheets provided by USDA, online forms, etc.)? 

b. Who at the SA is involved in the AR NOW? What are their roles? How many 

reviewers does the State have? Are they SA employees or contractors?

c. What specific parts of the AR were affected (i.e., mode of data collection, data 

tracking and reporting capabilities, LEA training, SA training?) PROBE for details.

d. How has the offsite process changed for the SA reviewer/LEA, since the ART grant 

project was implemented? PROBE for details.
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌110 

e. How has the onsite process changed for the SA reviewer/LEA, since the ART grant 

project was implemented? PROBE for details.

f. How has the corrective action or fiscal action process changed for the SA 

reviewer/LEA, since the ART grant project was implemented? PROBE for details.

g. What do you see as the biggest changes to the AR process at the State level? PROBE 

for off-site and on-site.

h. From your perspective, what do you think has been the biggest change for LEAs or 

at the LEA level?

i. How many of these changes are related to the [INSERT NAME OF 

INTERVENTION] versus USDA policy changes?

j. Do you now spend more or less time on the AR process and related activities than 

you did before the changes? Probe for why or why not.

i. (PROBE: If the new process requires additional time) How much of this 

additional time is related to the [INSERT NAME OF INTERVENTION] 

versus USDA policy changes? 

ii. (PROBE: If the new process requires additional time) Do you think the new 

AR process will go faster for you as you become more familiar with it?

k. Did [INSERT NAME OF INTERVENTION] simplify your work or make it more 

complicated? PROBE, why or why not?

Successes, Challenges and Lessons Learned

4. Does the new AR process result in improved performance? How? (i.e., lower error rate, 

better meal pattern compliance, fewer corrective actions, etc.)

5. From your perspective, what are the top three successes related to the new AR process? If not 

mentioned: Ask if there were successes at the State level that differed from the LEA 

level.

6. From your perspective, what were/are the top three challenges related to the new AR 

process? How did you address these? 

7. If you could change anything about the new AR process, what would you do differently? 
APPENDIX D: INTERVIEW PROTOCOLS

Abt Associates CN ART Grants | Final Study Report Final ▌111

Wrap-up

8. Is there anything we did not ask that you think is important for us to know, or you were 

hoping we would discuss? 

Thank you for answering my questions. That completes today’s interview. 

Have a nice day.

Stop recording and note time interview concluded.
APPENDIX E: DATA SUMMARY TABLE

Abt Associates CN ART Grants | Final Study Report Final ▌112

Appendix E. Data Summary Table

The following appendix includes a summary of the data presented in the above graphics. The information provided is drawn from both the extant 

data and qualitative interviews.

Grantee Intervention Type

Intended 

Duration 

(Years)

Number of 

Extensions

Extension 

Duration 

(Years)

Total 

Duration 

(Years)

Grant

Amount

Amount

Returned

AL Technology Intervention Only 2.6 1 1.0 3.6 $494,769.00 $217,491.00

AK Technology Intervention Only 2.1 1 1.0 3.1 $1,058,915.00 $3,751.98

CA Training Intervention Only 2.6 0 N/A 2.6 $500,000.00 $159,942.09

GU Technology Intervention Only 3.6 1 1.0 4.6 $1,700,000.00 $53,482.61

HI Training Intervention Only 2.1 2 4.0 6.1 $1,499,385.00 $46,613.71

ID Training Intervention Only 2.6 0 N/A 2.6 $251,655.00 $0.00

IN Technology Intervention Only 2.4 1 1.0 3.4 $1,017,920.00 $45,901.65

IA Technology Intervention Only 2.6 2 2.7 5.3 $1,700,000.00 $0.00

KS Both Intervention Types 3.0 1 0.8 3.8 $1,263,223.00 $0.00

MA '12 Both Intervention Types 2.1 1 0.3 2.4 $1,243,647.88 $9,894.19

MA '14 Training Intervention Only 3.0 0 N/A 3.0 $1,495,840.00 $4,220.80

MI Technology Intervention Only 3.0 1 0.8 3.8 $966,618.00 $0.00

MO Technology Intervention Only 1.9 4 3.5 5.4 $1,276,900.00 $143,392.56

NH Both Intervention Types 3.1 2 1.5 4.6 $1,409,532.85 $753,202.42

ND Technology Intervention Only 2.3 2 1.2 3.5 $389,973.00 $0.00

PA '13 Technology Intervention Only 3.0 0 N/A 3.0 $1,499,977.00 $0.00

PA '14 Training Intervention Only 3.0 0 N/A 3.0 $464,801.00 $0.00

RI Technology Intervention Only 2.0 0 N/A 2.0 $312,183.00 $5,680.15

SD Technology Intervention Only 3.1 1 0.2 3.4 $843,158.00 $315,580.22

WA Technology Intervention Only 2.4 4 2.8 5.1 $1,478,700.00 $0.00

WI Both Intervention Types 2.6 4 3.0 5.6 $1,960,526.00 $207,806.36

WY Technology Intervention Only 2.6 1 1.0 3.6 $482,500.00 $160,314.37