Child and Adult Care 

Food Program 

(CACFP) Family Day 

Care Homes Meal 

Claims Feasibility Study

Final Report 

September 2020 
USDA is an equal opportunity provider, employer, and lender
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

i 

Contents 

Executive Summary........................................................................................................................ 1

Methods to Estimate IP .............................................................................................................. 2

Study Findings ............................................................................................................................ 3

Conclusions ................................................................................................................................ 3

Recommendations ...................................................................................................................... 3

1. Overview of the Feasibility Study.............................................................................................. 5

2. Development of Data Collection Instruments............................................................................ 7

2.1 Application Development Process ...................................................................................... 8

2.1.1 Meal Service Reporting System (MSRS) for Providers ......................................... 9

2.1.2 Child Attendance Reporting System (CARS) for Parents .................................... 10

2.2 Confidentiality, Data Quality, and Security ...................................................................... 11

2.3 Pilot Test ......................................................................................................................... 12

3. Feasibility Test ......................................................................................................................... 13

3.1 Sample Design .................................................................................................................. 13

3.2 Recruitment for the Feasibility Study ............................................................................. 14

3.3 Study Sample .................................................................................................................... 15

3.3.1 Sponsors ................................................................................................................ 15

3.3.2 Providers ............................................................................................................... 16

3.3.3 Children................................................................................................................. 17

3.3.4 Providers with Participating Parents ..................................................................... 17

3.3.5 Lessons Learned from Recruitment ...................................................................... 18

3.4 Data and Data Sources ...................................................................................................... 20

3.4.1 Meal-Serving Time and Meal Claims ................................................................... 20

3.4.2 Child Attendance .................................................................................................. 20

3.4.3 Provider/Sponsor Administrative Data ................................................................. 20

3.5 Data Collection ................................................................................................................. 21

3.5.1 Distributing Materials and Information Prior to the Study Month ....................... 21

3.5.2 Launching the Study Month.................................................................................. 22

3.5.3 Monitoring Data Reporting ................................................................................... 24

3.5.4 Parent Follow-up ................................................................................................... 25

3.5.5 Closing the Study Month ...................................................................................... 25

3.5.6 Collecting Data from Sponsors After the Study Month........................................ 26

4. Data Processing ........................................................................................................................ 28
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

ii

4.1 Data from Sponsors........................................................................................................... 30

4.2 Data from CARS ............................................................................................................... 33

4.3 Data from MSRS............................................................................................................... 36

4.4 Lessons Learned from Data Collection and Processing ................................................... 37

5. Validating the Data Collection Methods .................................................................................. 38

5.1 Triangulation ..................................................................................................................... 38

5.2 Difference-in-Differences Method .................................................................................... 49

5.2.1 Data and Descriptive Statistics ............................................................................. 53

5.2.2 Difference-in-Differences Modeling Results ........................................................ 55

5.3 Estimation of Improper Payments .................................................................................... 57

5.3.1 Estimation of Improper Payments Using Triangulated Data ................................ 57

5.3.2 Error Estimates Based on Difference-in-Differences Model ................................ 59

6. Recommendations .................................................................................................................... 62

Appendices .................................................................................................................................... 65

Appendix 1. Reported Child Attendance in the Study Month Collected via CARS (Child 

n=215, days=21 ) and Imputed Child Attendance for Days Not Reported ...................... 65

Appendix 2. Percentage of Providers with Claim Errors, by Meal Type ................................. 66

Appendix 3. Meal Reimbursement Rates ................................................................................. 67

Appendix 4. Descriptive Statistics: Average Number of Attendance Days, Eligible 

Meals/Snacks, and Percentages of Children by Tiering Status and Enrollment Expiration

........................................................................................................................................... 68

Appendix 5. Meals Claimed but Not Served ............................................................................ 70
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

iii

This report was prepared for the U.S. Department of Agriculture (USDA), Food and Nutrition 

Service (FNS) by Manhattan Strategy Group, under contract number AG-3198-C-14-0026. 

Suggested citation: Zhang, Y., Geller, D., Huang, G., Walters, T., Simpson, A., Young, A., & 

Belyea, M. (2020). Child and Adult Care Food Program (CACFP) Family Day Care Homes 

Meal Claims Feasibility Study. Alexandria, VA: Food and Nutrition Service. 

The views expressed are those of the authors and should not be attributed to FNS, nor does 

mention of trade names, commercial products, or organizations imply endorsement of same 

by the U.S. Government. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

1 

Executive Summary

Under the Improper Payments Elimination and Recovery Improvement Act (IPERIA) of 2012, 

agencies must periodically review and identify programs that may be susceptible to significant 

improper payments.1 The Child and Adult Care Food Program (CACFP), which reimburses 

program providers for nutritious meals and snacks served to eligible enrollees for care in child 

care centers, day care homes, and adult day care centers, is subject to this review. This feasibility 

study focuses on meal claims made by CACFP family day care homes (FDCHs). According to the 

United States Department of Agriculture (USDA), Food and Nutrition Service (FNS), FDCHs 

served more than 458 million meals and snacks in FY 2018 to approximately 4.63 million 

children.2 

FNS administers CACFP through grants to States. Independent centers and sponsors sign 

agreements with States to assume administrative and financial responsibility for CACFP 

operations. Sponsors ensure that FDCHs meet the CACFP eligibility criteria, comply with 

applicable Federal and State regulations, and offer training and support to providers. Sponsors 

receive and verify meal reimbursement claims, submit claims to State CACFP offices, receive 

reimbursement payments, and distribute payments to FDCHs. 

Providers at FDCHs offer a safe, home-based day care setting for families and provide CACFPeligible children with access to nutritious food meeting USDA dietary guidelines. To participate 

in the CACFP, FDCHs must be supported by a sponsor; abide by the policies and procedures of 

USDA, the State agency, and the sponsor; attend sponsor and State agency training; and 

participate in monitoring visits. 

The purpose of this feasibility study is to design and test a data collection method that enables 

FNS to estimate improper payments (IP) due to meals claimed improperly by FDCHs 

participating in CACFP. Specifically, the study focuses on accurately measuring meals that are 

claimed but not served. If feasible, the data collection method needs to be viable for potential 

national-level implementation.

To fulfill the study purpose, FNS contracted with Manhattan Strategy Group to design, develop, 

and test the use of digital instruments to facilitate the verification of meal claims. These

instruments, the Meal Service Reporting System (MSRS) and the Child Attendance Reporting 

System (CARS), collect accurate meal-serving and child-attendance information from FDCH 

providers and parents without intruding into the routine practices of FDCH homes. MSRS allows 

providers to report meal-serving times for each child in attendance through a user-friendly 

smartphone mobile application or a website alternative. It does not replace current procedures that 

providers use to file their monthly meal claims. CARS allows parents to report daily drop-off and 

pick-up times of their children through a simple real-time text messaging system. We used data 

collected from MSRS and CARS, in combination with secondary data collected from sponsors, to 

approximate IP through data triangulation and an experimental study. We tested our data 

collection methods on a random sample of FDCHs in two States to determine the validity of the 

1 Public Law 111–204, Sec. 2. (a) (1), retrieved from https://www.govinfo.gov/content/pkg/PLAW111publ204/pdf/PLAW-111publ204.pdf on May 13, 2019.

2 Retrieved from https://fns-prod.azureedge.net/sites/default/files/pd/ccsummar.pdf on June 24, 2018.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

2 

method. We analyzed data collected from this field test to estimate the magnitude of improper 

claims that may result in IP and calculated the expected dollar amount of IP. 

This report describes the design and development of MSRS and CARS, the implementation of the 

two instruments in a field test, and the estimates of IP based on data collected from the field test. 

We discussed the prospective applicability of the methods and recommendations for a national 

scale-up study. 

Methods to Estimate IP

This study relied on two approaches to estimate improper payments: a triangulation-based 

approach from sample-specific data and a difference-in-differences model that used inferential 

statistics for the estimates. Similar conclusions from the two approaches prove` the validity of the 

IP estimates. 

Triangulation. Triangulation refers to the process of using multiple data sources to verify a meal 

claim. The approach assumes that improper claims exist even after sponsors edit the claims, 

avoiding some potential improper payments, and State agencies review the sponsor-edited claims 

before issuing reimbursement. The triangulation-based approach to identifying improper claims, 

however, does not take into consideration that meal claims may fluctuate across months in the 

business-as-usual condition. 

To control for this natural fluctuation over time and differentiate it from any change in meal 

claims when MSRS and CARS are implemented, we used a rigorous pre-post experimental design 

for testing the effectiveness of the treatment to improve the accuracy of meal claims. Specifically, 

we used an experimental design with a treatment condition and control condition. Under the 

treatment condition, the providers used MSRS to report meal-serving times, and parents used 

CARS to report absence, drop-off, and pick-up times of their children. Data reporting took place 

daily, and often in real time to ensure the accuracy of recall. We did not inform providers and 

parents in the control group of the study to preserve routine practices in a business-as-usual 

condition. 

Modeling. Difference-in-differences (DD) modeling is a statistical technique to compare

observed data from a treatment group—in this case, data reported during the study month using 

MSRS and CARs—with a comparison group that uses routine reporting procedures for meal 

serving and child attendance. Providers in the comparison group was in the business-as-usual 

condition throughout the study. Providers in the treatment group was in the business-as-usual 

condition in the pre-study months and used MSRS in the study month. We used the model to test 

whether the data collected would yield a proxy of overclaims based on the hypothesis that our 

intervention would result in more accurate data on meal reporting compared to the control group 

during the study month. 

The focus of the DD modeling was to determine if the two groups’ difference in their betweenperiod differences in meal claims was substantial and statistically significant. The DD modeling 

statistically controls for the time effects (i.e., changes in meal claims across the months due to 

factors other than the intervention) and the two groups’ background differences (enrollment size 

and rural-urban location), producing strong evidence validating the feasibility study, relative to 

many other analytic approaches. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

3 

Study Findings

For both approaches, we report improper claims for breakfast, lunch/supper (combined due to a 

common reimbursement rate), and snacks, and in total. The triangulation approach shows that 

overclaims (meals claimed but not served) were 19.4 percent for breakfast, 18.3 percent for 

lunch/supper, 23.5 percent for snacks and 20.6 percent across all meal types. The dollar value of 

overclaims was similar at 20.0 percent for all meal types. The DD modeling shows similar results, 

i.e., 18.3 percent of claims are overclaims, with an estimated range of 12.2 to 24.5 percent 

overclaims at the 95 percent confidence level. 

The data collection methods are viable as tested in this feasibility study. Without a nationally 

representative sample, however, the error rates generated from data in this feasibility study are

rather preliminary. While these estimates of error rates for overclaims may exceed the 10 percent 

threshold set in IPERIA as a compliance measure, they have two critical limitations. First, the 

rates are based on a study sample that is not representative of the population. Second, these rate 

estimates include overclaims due to failure to comply with allowable meal serving time or 

exceeding the maximum number of children in attendance, as set by their States. If we excluded

overclaims related to these two causes, the overclaims rates would be lower than 10 percent. FNS 

will need to clarify standards to apply State and local policies to count improper claims for federal 

reporting consistently. 

Conclusions 

The data collection methods designed for measuring IP are effective and feasible for large-scale 

implementation with a nationally representative sample of sponsors and providers. This 

feasibility study also establishes the validity of data collection using MSRS and CARS. These 

initial estimates serve to guide policies about program compliance and integrity to better serve the

CACFP community. In summary, we found that: 

• Providers were able to use the smartphone version of MSRS to make timely reports of mealserving times. Sufficient incentives and targeted technical assistance are essential to 

encourage the use of MSRS in addition to existing reporting systems. 

• Parents adapted readily to the use of the CARS text messages to report child drop-off and 

pick-up times. Reminders and sufficient incentives are necessary for their sustained use 

Recommendations

Based on the practices and lessons learned from the feasibility study, we make the following 

recommendations for a national study to measure improper meal claims in FDCHs.

Required participation: FNS should consider how to enforce required participation. Active State 

engagement of sponsors and providers may help, especially when combined with contractor 

technical assistance in onboarding their participation. 

Incentives: Behavioral science confirms that financial incentives are highly effective in gaining 

participation. The digital distribution of electronic gift cards was effective and reliable. 

• There was no provision for incentives to sponsors in the feasibility study. FNS should 

strongly consider implementing incentives for sponsor participation and requesting that 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

4 

they help in the recruitment of providers. Otherwise, many sponsors may not have the staff 

to take on additional responsibilities, even for a limited time. 

• The feasibility study provided only modest financial incentives for providers. FNS 

reduced the incentive from $100 proposed by the contractor to $60. Without an effective

incentive mechanism, it will be challenging to collect data from providers. 

• The modest financial incentive for parents, $25, was of limited value in gaining the 

participation of parents, who have no direct relationship with CACFP or FNS and cannot be 

required to participate. As such, sufficient incentives are essential for a successful national 

study. 

Time to prepare for study implementation: Providers and parents need at least one week to 

access and familiarize themselves with MSRS and CARS before they begin using them. 

Providers need additional time to communicate with parents about the study. 

Technical assistance: Offer robust technical assistance via instructional video, study hotline, 

email, text messaging, and social media to onboard providers and parents. MSG staff used these 

communication channels to assist providers and relied on text messaging and telephone to 

communicate with parents. Parents needed little assistance after their initial use of CARS. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

5 

1. Overview of the Feasibility Study

Under the Improper Payments Elimination and Recovery Act of 2010, agencies must 

“periodically review... and identify all programs and activities that may be susceptible to 

significant improper payments.” The Child and Adult Care Food Program (CACFP), which 

reimburses program providers for nutritious meals and snacks served to eligible enrollees for care 

in childcare centers, day care homes, and adult day care centers, is subject to this review.

 This feasibility study focuses on meal claims made by CACFP family day care homes (FDCHs). 

CACFP reimburses eligible FDCHs for the costs of providing nutritious meals and snacks to 

children from low-income families. According to the United States Department of Agriculture 

(USDA), Food and Nutrition Service (FNS), FDCHs served more than 485 million meals and 

snacks in FY 2017 to approximately 4.53 million children.3

The purpose of this feasibility study is to design and test a data collection method that enables 

FNS to estimate improper payments (IP) due to meals claimed improperly by FDCHs

participating in CACFP. Specifically, the study focuses on accurately measuring meals that are 

claimed but not served. If feasible, the data collection method needs to be viable for potential 

national-level implementation in the future.

FNS administers CACFP through grants to States. Independent centers and sponsors sign 

agreements with States to assume administrative and financial responsibility for CACFP 

operations. Sponsors are responsible for ensuring that homes meet the CACFP eligibility criteria, 

providing training and other support, and monitoring the homes to ensure compliance with 

applicable Federal and State regulations. Sponsors receive and verify meal reimbursement claims, 

submit claims to State CACFP offices, receive reimbursement payments, and distribute payments 

to FDCHs. 

Providers at FDCHs offer a safe, home-based day care setting for families and provide CACFPeligible children with nutritious food meeting USDA dietary guidelines. To participate in the 

CACFP, FDCHs must be supported by a sponsor; abide by the policies and procedures of USDA, 

the State agency, and the sponsor; attend sponsor and State agency training; and participate in 

monitoring visits. 

To fulfill the study purpose, we designed and developed digital instruments, the Meal Service

Reporting System (MSRS) and the Child Attendance Reporting System (CARS), to collect mealserving and child-attendance information from FDCH providers and parents, without intruding 

into the routine practices at the FDCH. We used data collected from MSRS and CARS, in 

combination with secondary data collected from sponsors, to approximate IP through a data 

triangulation approach and error estimates inferred from difference-in-differences models. We 

tested our data collection methods on a random sample of 283 FDCHs in two States to determine 

the validity of the method. We analyzed data collected from a field test to estimate the magnitude 

of improper claims that may result in IP and calculated the expected dollar amount of IP. 

3 Retrieved from https://fns-prod.azureedge.net/sites/default/files/pd/ccsummar.pdf on March 27, 2018.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

6 

In this report, we describe: 

• the approach to designing and developing the technology-assisted data collection 

instruments through pilot tests; 

• the design of the field test; 

• the operation of data collection during field testing; and 

• the procedure for calculating improper payments. 

The digital instruments are fundamental to the feasibility study as they provide a means for the

accurate estimation of IP in FDCH meal claims. We describe the design and development of the 

two technologies in Section 2.0. In Section 3.0, we describe the implementation of the feasibility 

test, including the recruitment of participants and the collection of administrative data and 

primary data using MSRS and CARS. In Section 4.0, we describe our approach to processing the 

data from multiple sources. We report the estimation of errors in meal claims and the payment for 

these errors in Section 5.0. In Section 6.0, we present the conclusion of the feasibility study and 

make recommendations for the approach in a prospective scale-up to assess IP in a nationally 

representative sample. 

 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

7 

2. Development of Data Collection Instruments

Improper payments are based on meal claim errors. To capture these errors, we designed two 

instruments to collection data from daycare home providers and parents whose children attend 

these day cares. In this chapter, we provide detailed descriptions of the features and development 

process for the MSRS and CARS data collection instruments. MSRS was designed for providers 

to report meal-serving times through a smartphone mobile application or a website. It does not 

replace the current procedures that providers use to file their monthly meal claims. CARS collects 

the child’s daily attendance from parents through a text messaging system or a website.

4 The

applications, available in English and Spanish, allow providers to report meal-serving time and 

parents to report child attendance, both in real time. The data collected through the user 

instruments are transmitted to a central database and stored there to allow for monitoring by 

system administrators. 

Exhibit 1 illustrates the architecture and components of the technology-assisted data collection 

process. The center portion of the diagram, in the black rounded rectangle, represents the server 

architecture for the website and database. Three data flows are represented in the remainder of the 

diagram. The left side of the diagram depicts how parents/guardians use CARS, a short message 

service (SMS) solution through a cloud service or a web browser, to communicate with the server. 

The lower right side of the diagram depicts how the provider can use either the mobile application 

or a browser to submit data to MSRS. The lower middle section of the diagram demonstrates how 

the administrator, Manhattan Strategy Group (MSG), uses the browser to view data in a webbased format. If interested, sponsors may view the meal-serving times reported by providers they 

supervise. Additionally, all mobile and web-based communication passed through Secure Sockets 

Layer/Transport Layer Security (SSL/TSL) encryption algorithms during data transmission. 

4 MSG considered an interactive voice response (IVR) option but believed it was not warranted and did not include 

it in the design. Based on available evidence in and shortly before 2014, access to a cell phone or computer was

high. About 88 percent of American adults had a cell phone, and both African Americans and English-speaking 

Latinos had adoption rates that were comparable to the national average for all Americans (Zickuhr & Smith, 2012). 

Among poverty-level households, most had only cell phones (no landlines), as did nearly half of near-poverty 

households (Blumberg & Luke, 2013). For populations with an annual household income below $30,000, nearly half 

had smartphones and Internet use and smartphone ownership rates for African Americans and Latinos were 

comparable to those of whites (Smith, 2014). By the time of data collection in November 2017, we found that cell 

phone access was universal among CAFCP parents recruited for the study.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

8 

Exhibit 1. Technology-Assisted Data Collection Process

2.1 Application Development Process

Users accessed these data collection tools through four interfaces: (1) CARS SMS, (2) CARS 

website, (3) MSRS mobile application, and (4) MSRS website. The underlying code and database 

structure use common services and aid each platform (web/SMS/mobile) in saving and retrieving 

information to and from the same central database. The development of this application consisted

of two unique phases described below: 

Design. The design phase encompassed the conceptual design, functional design, and graphical 

design tasks. The products of this phase were informal and internal conceptual design documents, 

formal functional specifications (System Design Document), and a set of graphical screen 

mockups showcasing how each main feature of each of the four applications would look. 

Development. The development phase consisted of multiple internal iterations to produce a 

complete, functional, and tested set of technologies for delivery as CARS and MSRS. The work 

products of this phase were the four unique technologies that meet the agreed-on graphical and 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

9

functional design specifications: CARS SMS, CARS website, MSRS mobile application, and 

MSRS website.

2.1.1 Meal Service Reporting System (MSRS) for Providers

The objective of MSRS is to capture meal-serving time as close as possible to the actual time 

when the meal is served by the provider, minimizing human error in recalling and reporting. 

MSRS collects meal-serving times with an advanced notification/alert system to send reminder 

messages to the providers if they have not provided any information (i.e., home closure or mealserving time) by 7:00PM. Providers are not able to log meal-serving times for a previous day. A 

web-based MSRS is also available for providers as an alternative method of reporting. The 

smartphone application and the website collect identical information and are stored centrally in 

the same database.

MSRS Functionality. MSRS allows providers to report in real time when meals are served to 

each child. It also allows providers to manage child enrollment status, report closures, and view 

the history of meals served. Exhibit 2 specifies additional information on the features of MSRS.

Exhibit 2. Features of MSRS

Feature Description

Enrollment MSG staff enrolled providers and sent them with a username and PIN via email 

along with instructions on how to download the mobile app and access the 

website.

Authentication Providers authenticate themselves into the system using a provider ID and PIN 

that MSG sends to them via email.

Log Meal-serving 

Times

Providers log meal times for each child and every meal served. Meal times can 

be logged by selecting multiple children and then clicking on a “Serving Now” 

button to log meals in a batch. Meal times can be changed before clicking 

“Serving Now,” in the event that the FDCH is late logging the meal. Meal times 

can be changed on a child-by-child basis to correct data entry errors.

View History The provider can view the meal log history for the duration of the field test by 

browsing the daily meal claims for each day.

Notification – Meal 

Logging

A notification is sent to the provider when no meals have been logged and the 

ending time of meal serving period is approaching.

Notification – No Data A notification is sent to the provider when no meals have been logged for the 

day. The goal is to continuously prompt the provider to log meal times in as 

close to real time as possible.

Day Care Closed An option is provided to allow the provider to inform MSRS that the day care 

was closed.

Help An option is provided for assistance through a toll-free phone line.

Time Zone The time zone reflects the providers’ time zone.

Child Management Providers can manage the child enrollment status by marking a child as 

“inactive” or reactivating a child who returns to the day care. When a child is 

inactive, no data are reported. When a child is reactivated, data collection 

resumes as normal.

Language Support Content is provided in English and Spanish. Providers choose their preferred 

language during enrollment, but the two language choices can also be toggled 

within the application.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

10 

2.1.2 Child Attendance Reporting System (CARS) for Parents 

CARS is an SMS- and web-based reporting solution that allows parents and guardians to report 

the drop-off and pick-up time of a child, even for a child who may leave the day care home and 

return later in the day.5 CARS is designed to collect attendance data from parents through a text 

messaging system or an online reporting website. The text messaging system is compatible with 

multiple technology platforms. It provides a reliable source of data that can be used to validate 

meal claims.

6 Parents report the time they drop off and pick up their children by simply sending 

text messages to CARS in the specified formats or submitting the information on the CARS 

website. 

All parents participating in the study had access to a mobile phone to receive and respond to text 

messages; a smartphone is not necessary for CARS as long as the mobile phone can receive and 

send text messages. The text messaging system for CARS sends two reminder text messages to a 

child’s parent each day, to inquire about the time the child is dropped off and picked up. The 

messaging system has preloaded information from the sponsors’ databases to ensure that text 

messages are delivered to parents. Parents were asked to confirm the name and current enrollment 

status of their child and their mobile phone number when they agreed to participate in the monthlong study. 

CARS Functionality. MSG built the SMS-based data collection system on SMS services 

provided by Twilio, which allows us to build custom processing webpages to handle information, 

such as saving information to a database and responding back to the user via SMS text. Exhibit 3 

summarizes the features of CARS.

Exhibit 3. Features of CARS

Feature Description

Enrollment in 

CARS System

Parents/guardians are pre-enrolled in the CARS system and prompted to confirm 

their enrollment via SMS text and/or email messages. During this enrollment, 

parents set up a password to access their web-based account.

Authentication Parents/guardians are authenticated through their enrollment in the SMS system, or 

by logging into the web-based system with their mobile number and password.

Multiple Accounts The system handles enrollment of multiple persons by associating different mobile 

numbers with the same children. Data provided by any of the parents/guardians are 

stored in the same central account and associated with the same child(ren).

Log Drop-off Time Parents provide drop-off times of children by either sending an SMS text or entering 

time in the web application.

Log Pick-up Time Parents provide pick-up times of children by either sending an SMS text or entering 

time in the web application.

5 Parents of children who leave for school and return after school by school bus are not expected to report bus times. 

Instead, they report them upon initial enrollment in CARS and on any days when school schedules change.

6 Depending on their plan with a cell phone carrier, some parents may incur additional text message charges. These 

charges tend to be modest and will more than be compensated by the $25 incentive offered to parents participating 

during the study month.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

11 

Feature Description

Send Notifications System notifies parents when any of the following triggers are instantiated:

• The parents/guardians have not logged a drop-off time AND the time is one hour 

past their normal drop-off time.

• The parents/guardians have not logged a pick-up time AND the time is one hour 

past their normal pick-up time.

Absences Parents report absences to CARS any time during the day, or in response to an SMS 

text sent to them from CARS.

Account 

Management

Parents/guardians can set up and maintain their profile information including name, 

mobile number, and email address.

Help Parents receive online instructional materials and a method of ad hoc assistance in 

the form of a toll-free support line.

Language Support CARS is available in English and Spanish. Parents can select their language 

preferences during enrollment. The languages can be toggled within the web 

application.

Security Data collected and stored in the web server are encrypted.

7 Personal identifying 

information is stored offline, and records are matched on the proprietary identifier 

created as part of the data collection process.

2.2 Confidentiality, Data Quality, and Security

Each web-based or mobile application includes progressive validation and logic checks as data 

are recorded by participants, with some exceptions. For CARS, the application provides logic 

checks that prohibit skipped data and confirm the times as AM or PM. It includes an 

administrative function that identifies problematic cases and specifies the resolution required and 

then documents the follow-up taken to resolve the issue.8 This administrative system supports 

data quality by minimizing both inaccurate data and parental nonresponse. For MSRS, the 

validation checks are limited to logic and typographical errors. Validation checks identify data 

entry issues in MSRS and administrative functions similar to those established for CARS, which 

facilitates the project team to monitor data collection. 

We used unique child identification numbers (ID) to link data in CARS and MSRS. The child’s 

name (i.e., first name and last initial) is used as a secondary validation measure to ensure the 

accuracy of record matching, the value of which we confirmed from the pilot test. We then

generated scrambled IDs using child-level information from the sponsoring organization to 

preload child names to the provider’s account. The provider could see each child’s first name and 

last name initial in their MSRS account to report meal-serving time. 

7 MSRS and CARS web services are encrypted with SSL by default. MSRS and CARS also use auth-token based 

authentication per provider. All content stored for MSRS and CARS is also encrypted at rest within SQL server.

8 We built the system with enough internal validity checks to identify and resolve errors as close to real time as 

possible, so we have a self-cleaning dataset produced by both MSRS and CARS. We would have conducted

additional data cleaning but would not have had the resources to do any type of additional contacts with providers or 

parents to resolve these issues. We would rely on our logic checks for resolving these cases. We will solicit 

information from sponsors and other resources to develop a full-fledged list of these logic checks.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

12 

Both CARS and MSRS use unique IDs for child, parent, and provider to link records, so no 

identifiable information about children at FDCHs is transported over the internet. This ensures 

that confidentiality is maintained. 

Data collected from MSRS and CARS are stored in a central database that records the day/time 

when the users submit data and when the data are saved into the database. Both the web-based 

application and the SQL server database are hosted on Amazon’s EC2 cloud servers. 

2.3 Pilot Test9

To test and finalize the instrument, we conducted cognitive interviews in Virginia and a monthlong pilot test in Texas in September 2015. During cognitive testing, providers and parents 

reviewed screenshots of MSRS and CARS and provided their interpretations of instrument 

design. Based on this feedback, MSG revised MSRS and CARS for the pilot test. During the pilot 

test month, providers and parents were given access to MSRS and CARS and used the technology 

for an entire month. MSG monitored their use daily and followed up with non-respondents as 

necessary. 

The pilot test demonstrated that MSRS and CARS are effective tools to collect information from 

providers and parents in a timely manner. During the pilot test month, MSG observed a 95 

percent response rate from providers using MSRS and a 100 percent response rate from parents 

using CARS. Providers and parents reported that the burden of using the technologies was 

minimal. Providers used MSRS primarily through smartphones, with limited use of the web-based

alternative. Some providers preferred reporting in real time; others preferred reporting all meals at 

the end of the day. Parents showed a strong preference for text messaging through their mobile 

phones and reported attendance drop-off and pick-up times mostly within five hours of the actual 

event.

After the pilot test, we conducted exit interviews with participants to understand their experience 

using the data collection instruments. Based on user feedback and an examination of the data, we 

made further improvements to MSRS and CARS to finalize the development of CARS and 

MSRS. 

9 More detail is available in the Pilot Test Memorandum MSG submitted in 2016. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

13 

3. Feasibility Test

After the pilot test in 2015 (as described above), we finalized the data collection instruments and 

applied for the Office of Management and Budget (OMB) clearance for a scaled-up field test. In 

April 2017, we obtained approval to conduct a field test to implement MSRS and CARS in two 

States, which is the main task of the feasibility study. In this section, we discuss the recruitment 

of study participants and the implementation of MSRS and CARS10. 

3.1 Sample Design

The goal of the study was to establish the internal validity of the proposed approach, not to 

generate nationally representative statistics of improperly claimed meals. Thus, our sample design 

serves two purposes: 

• to reflect the CACFP administrative structure; and 

• to ensure sufficient analytic power to detect improper meal claims at the lowest cost 

possible. 

For the feasibility study, we used a purposive sampling approach to identify States, sponsors, and 

FDCHs. An important consideration was the need to recruit a sufficient number of participating 

entities at each sampling level to ensure that the feasibility study could provide robust analytical 

findings. We conducted a power analysis to determine the needed sample size. Power refers to the 

probability (80 percent or 0.80 is a conventional level) at which we correctly reject the null 

hypothesis, which states that the treatment effect size is smaller than 0.35 standard deviations. We 

specified this moderate effect size based on a social science research convention.11 Using Optimal 

Design, a software program for advanced sample design,12 we conducted a power analysis to 

determine the number of providers needed to detect a group difference of 0.35 standard deviation. 

We assumed a conventional type I error, α=0.05; a minimum detectable effect size δ=0.35; and 

the proportion of the variance explained by the DD model, R squared=0.10. To achieve a power at 

or above the conventional level of 0.80, the analysis suggested that it is necessary to have 

approximately 230 providers. Exhibit 4 shows the results: the horizontal line for 0.80 power (yaxis) intersects with vertical lines for approximately 230 or more providers (x-axis). 

10 MSG also trained data collectors for the feasibility test. Detail of the training is available in the Memorandum on 

Data Collector Training submitted to FNS in December 2017.

11 The effect size in this analysis may be measured with β coefficient (standardized b coefficient) associated with the 

treatment status, D, in the regression analysis. It indicates the magnitude of the bi-serial correlation between the 

dichotomous treatment status and the continuous measure of meal claims. We refined the effect size definition, in 

the course of the study, as the group difference in between-period difference (DD) in meal claims (see Section 4.8).

12 Reviewed from https://sites.google.com/site/optimaldesignsoftware/home in November 2018.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

14 

Exhibit 4. Statistic Power as a Function of Number of Providers 

3.2 Recruitment for the Feasibility Study13

Since there is no national or regional database for CACFP that can provide detailed administrative 

information on sponsors and FDCHs/providers, we asked States to provide a list of sponsors and 

sponsors to provide a list FDCHs for sampling purposes. This approach reflected the 

administrative structure of CACFP and facilitated communication across agencies for recruitment. 

The study recruitment process includes three milestone activities: (1) issuing formal notification 

and invitation of sampled entities to the study; (2) obtaining data files needed for each phase of 

sampling; and (3) receiving the administrative data needed during the study’s data collection 

period. Below we describe the execution of recruitment activities at the State, sponsor, provider, 

and parent level. 

Contacting regional offices and States. We recommended four States—Minnesota, Texas, 

Virginia, and California—to obtain the cooperation of two. We selected these four States based 

on daily attendance in 2016, the amount of reimbursement payments in 2016, and geographical 

location. Two of the States are relatively small, with each serving less than three percent of the 

national total of children attending day care homes and receiving less than $60 million in Federal 

reimbursements. The two larger States each serve more than 12 percent of children attending day 

care homes and receive more than $300 million in reimbursements. The four States are in four 

different FNS Regional Offices (FNSROs): Midwest, Southwest, Mid-Atlantic, and West. 

Minnesota turned down the invitation; Texas and Virginia agreed to participate in the study. We 

did not contact California. 

Sponsor recruitment. Using demographic and location information provided by the State 

agency, MSG categorized sponsors by geographic region of the State, by rural and urban areas 

served, and by size (e.g., number of FDCHs). We first sent an invitation letter by email to the 

13 More detail is available in a summary report on recruitment MSG submitted in February 2018.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

15 

sponsor contact person in both States. We then prepared a memorandum of understanding (MOU) 

for sponsors and offered two webinars to further onboard sponsors after we emailed study 

invitations to them. During the webinar, we explained the purpose of the study, the study 

approach, and the expected levels of effort for sponsors and providers. Ten sponsors, four from 

Virginia and six from Texas, agreed to participate in the study. 

Provider recruitment. With information sponsors submitted about providers, we began sampling 

providers and requesting their participation. We developed an algorithm to randomly select up to 

35 providers14 from each sponsor that included providers with different enrollment sizes in both 

urban and rural areas across the State. Next, we randomly assigned selected providers to either the 

study or the control group. We then emailed providers in the treatment group to inform them of 

the study and invite them to participate. We sent reminder emails to providers if they did not open 

or did not respond to the initial email. We also made personal phone calls and sent text messages 

to providers who could not be reached by email. Once providers consented to the study, we 

requested parent contact information for parent recruitment. 

Parent recruitment. We sent parents a text message about the study once we received their 

contact information from the providers. Text messaging was the most viable means to recruit 

parents because they would later use it to report child attendance. We asked FDCH providers to 

inform parents about the study. Through the text messaging system, we offered parents the option 

to ask questions about the study via phone or email. 

3.3 Study Sample

We summarize in this section the recruitment results, including the number of entities that agreed 

to participate in the study and the characteristics of the final sample. Participants in the feasibility 

study did not participate in the pilot study as described in Section 2.3. 

3.3.1 Sponsors

In Virginia, four sponsors from a good cross-section of the State joined the study. The other three 

sponsors we contacted turned down the study because they had a conflict in adopting a new 

electronic system for meal claiming during the proposed study month. Adopting two new 

technologies at the same time would have created stress for busy day care providers. It might also 

negatively affect the quality of data for this study. Two of the Virginia sponsors also expressed 

concern about privacy but ultimately were excused because of the conflicting timing of our study 

and their adoption of a new electronic meal-claiming system. 

In Texas, four of 10 sponsors we contacted declined to participate; their reasons for declining 

varied. One sponsor was a very small agency with a small population of FDCH providers who 

were primarily older women and challenged by new technologies. One large sponsor declined 

because it was already involved in a multiyear CACFP study. Another sponsor started the process 

of joining and then stopped responding to our outreach. 

14 To ensure at least 230 providers in the final study (see Section 4.1), we oversampled providers to cope with 

potential issues of nonparticipation or nonresponse of the selected providers.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

16 

3.3.2 Providers 

Participating sponsors in the two States provided enrollment data for 363 providers—219 in 

Texas and 144 in Virginia (see Exhibit 5). These data served as the sampling frame for the study. 

We randomly selected 189 potential treatment group providers and 174 control group providers. 

Based on the provider ZIP code, we appended the rural/urban classification code as provided by 

USDA’s Economic Research Service.15 The treatment group’s mean enrollment (12.5) was 

similar to that of the control group (12.0), a difference that was not statistically significant. About 

61 percent of the FDCHs in the treatment group are in Texas, and 24 percent of all treatment 

FDCHs are located in rural areas. 

Exhibit 5. Allocation of Providers in Treatment and Control Groups

39%

61%

24%

41%

59%

22%

0%

10%

20%

30%

40%

50%

60%

70%

VA TX Rural

Treatment Group Control Group

Of the 189 randomly selected providers for the treatment group, we eliminated 47 for the 

following reasons: 

• No email

• No computer 

• No longer providing day care 

• Temporarily closed (five in Texas due to Hurricane Harvey).

The remaining 142 were eligible for inclusion in the treatment group, and we contacted them for 

participation.

15 The rural-urban commuting area code classifies U.S. Census tracts as urban or rural using measures of population 

density, urbanization, and daily commuting based on data from the decennial census and the American Community 

Survey.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

17 

Of the 142 providers we contacted, 101 agreed to participate. Of those, 72 providers (51 percent) 

supplied child roster data. Nine of the 72 providers had no parents consenting to the study; from 

the remaining 63 providers, we collected meal serving data that could be validated with child 

attendance reported by at least one parent. Exhibit 6 shows the number of providers who agreed to 

participate and submitted requested roster information, as well as the number of parents who 

consented. Only providers who submitted rosters and had at least one consenting parent could 

participate.

Exhibit 6. Provider Sample 

3.3.3 Children 

Once the provider agreed to participate, we obtained rosters of the children and their parents. 

These rosters contained fewer children than the enrollment data indicated. While the mean 

number of children cared for by the 72 providers who agreed to participate was 12.0, the rosters 

showed a mean of 8.1 because some students were enrolled but were no longer in attendance or 

were not receiving meals. This difference substantially impacted the number of children available 

for the study. Instead of 864 children for which providers could theoretically provide data, 

according to the rosters, 580 children remained with the possibility of some CARS data 

collection.

We solicited the participation of parents of all children cared for by the 72 providers to report 

drop-off and pick-up data using CARS. We provided an incentive to encourage participation. 

Parents representing 216 children agreed to participate, a participation rate of 37 percent based on 

the student rosters. 

3.3.4 Providers with Participating Parents 

Sixty-three providers submitted rosters and had at least one participating parent. Of these 63 

providers, the parental participation rate was 40 percent. Only five small providers had 100 

percent of their parents participating. The median participation rate among parents was 38 

percent. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

18 

These 63 providers cared for 538 children and had a mean enrollment of 12.8 children, which is 

slightly higher than the 12.3 mean enrollment of the 142 providers that did not agree to 

participate, did not submit rosters, or had no participating parents. 

Further examination of differences between the groups indicates that the providers with 

participating parents skewed toward rural areas and Texas (Exhibit 7), with the chi-squared test of 

the difference in rural percentage statistically significant (p < 0.05). The difference between 

States, however, was not statistically significant.

Exhibit 7. Allocation of Providers with and Without Participating Parents 

35%

65%

29%

43%

57%

15%

0%

10%

20%

30%

40%

50%

60%

70%

VA TX Rural (VA and TX)

Consented Providers Who Submitted Rosters and Had Participating Parents (n=63)

Non-participating Providers or Providers Without Participating Parents (n=79)

3.3.5 Lessons Learned from Recruitment

During the study month, the MSG team learned many lessons that have direct implications for 

any future efforts to undertake a meal claims study at CACFP FDCHs, either on a similar scale or 

a larger scale. 

Voluntary or Required Participation. To the extent possible, requiring participation in the study 

may help in achieving a robust level of participation among those sampled for the study. 

However, to the extent that sponsors and providers already have full-time program 

responsibilities, an additional one might be challenging for them to accommodate, even 

temporarily. Sponsors and providers asked whether participation was required. Initial efforts to 

recruit sponsors were challenging, as sponsor recruitment occurred before FNS decided that 

participation was required. 

While FNS agreed that we could state that provider participation was required, there were no 

consequences associated with the requirement. Still, many FDCH providers would not join the 

study. The $60 incentive for providers was helpful but not large enough on its own to entice 

providers to participate. We had originally suggested a $100 incentive but reduced it to $60 in 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

19 

keeping with OMB’s preference at the time for a strong justification for incentives based on 

actual participation costs. 

Despite required participation for providers, 31 of the 142 providers we contacted still chose not 

to participate. Some indicated they were willing to accept any consequences associated with their 

decision; however, there were no stated consequences.

We offered parents a $25 incentive to participate, based on mobile phone charges they might 

incur; we had initially proposed a $50 incentive for parents, which might well have increased 

participation rates.

Who Sends Recruitment Invitations. It is helpful if the recruiter is familiar with the sponsoring 

organizations and providers as was the case with our lead recruiter for Texas. In the future, it is 

worth considering asking State agencies to recruit sponsors, and sponsors to recruit the providers, 

and providers to contact the parents. This approach would save time and effort in getting the 

invitation noticed and getting each entity to respond. In this field test, the most successful and 

least time-consuming recruitment for FDCHs was with sponsors who were proactive in their 

engagement with CACFP and supported the study recruitment efforts. When the sponsor held 

back or did not fully support the study, it was much more difficult to recruit providers. Similarly, 

when the FDCHs contacted the parents of children in their care to let them know about the study, 

we saw more parents joining, presumably, to help their provider. 

Organizational Capacity of Sponsors. Some sponsors had difficulty with the secure file-sharing 

options (needed for provider lists and child enrollment forms) or even password-protecting 

documents. This required patience and staff support to explain the process. It is essential to 

provide sufficient time to work with sponsors, as their staff members have day-to-day internal 

organizational demands that may take priority in their schedules. 

Working with FDCH Providers. FDCH providers may have varying levels of familiarity and 

comfort with the skill sets needed to use MSRS. Older providers, in particular, may be challenged 

by technology. Providers in rural areas (especially in a large State with vast rural areas, such as 

Texas) cannot be counted on to have consistent, reliable connections to the Internet or cellular 

service, although this is likely to improve with the implementation of 5G, the fifth generation of 

cellular network technology. 

Understanding the day-to-day operations of the CACFP helps with the timing of communications 

at all levels as well as the strategies used. It was extremely beneficial that our recruitment staff 

included those who understood the program. This helped build rapport and keep participants 

invested in the study. 

Study Communications. It helped to use multiple platforms to reach providers (i.e., email, text, 

social media, video) and share information with them in multiple languages (i.e., English, 

Spanish). While we did not create a website for the study, doing so might be useful in keeping all 

the pertinent information in one area for participants to refer back to in case an email or text 

message is deleted. We created an invitation-only Facebook page, which had some of the same 

benefits.

Timing of Recruitment and Gaining Familiarity with Study Tools. It is critical to plan

sufficient time for recruitment and give study participants time to become familiar with the data 

reporting tools. We did not have providers or parents use the reporting tools in advance of the 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

20 

study month to minimize behavioral changes in the pre-study month. However, it would have 

been beneficial to release the tools a few days earlier before the study month. 

3.4 Data and Data Sources

The design of this study required data from three primary sources: (1) meal-serving records 

provided by FDCHs via MSRS, (2) daily child attendance records provided by parents via CARS, 

and (3) administrative data and meal claims provided by sponsor. 

3.4.1 Meal-Serving Time and Meal Claims 

We collected meal-related data from providers using MSRS and from sponsors who uploaded 

claims. Providers used MSRS to record times for all served meals in the study month, November 

2017. We then collected edited meal claims reimbursed from sponsors for the pre-study months, 

September and October 2017, and the study month to estimate IP. These meal claim records were 

disaggregated by child and meal type to determine specific kinds of errors. We used meal claims 

that sponsors edited and submitted for reimbursement. 

3.4.2 Child Attendance 

Using child enrollment forms obtained from Sponsors and providers, we confirmed each child’s 

name and enrollment status and asked parents to participate in the study for one month. 

Specifically, we asked parents to report daily arrival and departure times for their child(ren), 

including instances of multiple pick-up and drop-off times. For instance, a child might be dropped 

off at the day care in the morning, be picked up by a parent for a doctor appointment, and then 

return to the day care home for continued care. Parents also reported absence and daycare closure. 

This parental daily attendance record, in combination with provider-reported meal service times, 

provided a proxy of possible meal serving times for a child. 

3.4.3 Provider/Sponsor Administrative Data 

The final set of data that we collected during the feasibility study consisted of administrative data 

from both sponsors and providers. The main administrative records of interest for the study come 

from two primary sources: the sponsor-provider agreement and the child enrollment form. 

The sponsor-provider agreement documented the approved meal service pattern the provider 

could offer, hours of operation, hours of meal service for each meal, and meals for which the 

provider was approved to receive reimbursement. Additionally, it provided the tiering status of 

the home, in some cases, information on the licensed capacity of the home or the maximum 

number of children that could be in attendance at one time. The child enrollment form was 

essential for this feasibility study. It served as an additional source of validation for the improper 

meal claims analysis. The child enrollment form contained the date from which the child’s 

enrollment was valid, the days of the week the child was enrolled, the meals the child was eligible 

to receive, and the times the child was expected to be present in care each day. The form included 

the tiering status of the provider and, if not Tier I (based on provider geographic location or 

provider income eligibility), eligibility of the child based on household income. Tiering or 

household eligibility is needed to determine the reimbursement rate and the financial impact of 

any IP. The child enrollment form was also a main data source for contacting parents. It was used 

separately in the study for sampling and study recruitment purposes. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

21 

The sponsor-provider agreement, supporting licensing and capacity information, and the child 

enrollment form are typically paper forms or Excel spreadsheets. Sponsors provided these data for 

the sampled FDCHs. Since the data might not be in an electronic format (or a standardized one), 

the MSG team developed a standardized template for entering and abstracting this information to 

create the analytical dataset. 

Collectively, we used the administrative data to define allowable meals for a child and to validate 

the edited meal claims in the study month. (This process is described in further detail in Section 4, 

Data Processing).

3.5 Data Collection

In this section, we describe the implementation of MSRS and CARS in the study month, 

November 2017, including preparation prior to the study month, the launching of the study 

month, monitoring data reporting, parent follow-up, closing the study month, and data collection 

from sponsors after the study month. For each stage, we discuss issues we encountered and the 

solutions we implemented to address these issues. We also share lessons learned and 

afterthoughts that may inform the design of national implementation. 

3.5.1 Distributing Materials and Information Prior to the Study Month

We recruited providers for the study month, November 2017. Prior to the study month, we 

emailed providers a link to install the MSRS application on mobile devices along with their 

account access information (username and account PIN). 

The timing on when to distribute MSRS was challenging because the providers varied in the time 

and frequency with which they check email and their familiarity with technology. We decided to

send the information at 5:00AM on Monday, October 30, 2017, two days prior to the start of the 

study month, to allow time for providers to install the app and access their accounts while keeping 

the use of MSRS in October to a minimum, as we also used October as a pre-study month to 

compare the meal reporting patterns with the study month. 

We contacted providers by phone on October 30th, the same day the MSRS login information was 

delivered to the providers to make sure they received the email. We also posted an announcement 

about the email on the study’s Facebook page and called all the providers on October 30th and 

31st. If we were unable to speak to a provider directly over the phone, we left a voice message on 

the phone or emailed and made at least three phone attempts to reach them. 

Providers requested technical assistance through email and phone. Of 38 provider technical 

assistance requests during the study month, 15 concerned child enrollment (child missing or name 

entry issue), 14 needed the link to download the app resent, 5 had installation problems, and four 

needed to correct a reporting error they made. 

For providers who consented but could not be reached to confirm their access to MSRS, we 

continued daily follow-up until November 10th, a week and a half after the beginning of the study 

month. We assumed providers had dropped out of the study if we could not reach them and did 

not receive any reporting from them at that time. 

Parents received an invitation text message to the study on October 24, 2017, and some started 

responding on the same day. We started engaging with parents a week prior to the study month to 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

22 

ensure their consent and allow enough time for them to ask questions and get used to the daily 

reporting of drop-off and pick-up times. 

We used text messaging to communicate with parents about the study, including links to the 

consent documents and instructions to use CARS. Some parents did not receive the initial text 

after consenting. All parent communications were in both English and Spanish. The Spanish 

translation seemed necessary as some parents did not respond unless we sent the Spanish 

translation.

We sent the following reminder text message to all consenting parents on October 31st, one day 

prior to the beginning of the study month: 

Hello CACFP meal claims study parent! Start using CARS tomorrow, Nov. 1, to 

report child attendance (Here’s how: http://bit.ly/CARS_User). If you prefer to 

use CARS website, visit http://cars.manhattanstrategy.com/ and log in: Username 

= your cell phone number; Password = SRAC 

¡Hola padre del estudio CACFP! Comience a usar CARS mañana, 1 de 

noviembre, para informar sobre la asistencia de su hijo (haga clic aquí para ver 

cómo: http://bit.ly/CARS_ESP). Si prefiere usar el sitio web de CARS, visite 

http://cars.manhattanstrategy.com/ e inicie sesión: Nombre de usuario = su 

número de teléfono celular; Contraseña = SRAC

A few parents expressed interest in joining the study after the study month had started. We 

included all parents and their children in the participant list for CARS and MSRS if they informed 

us of their interest by November 3rd. 

3.5.2 Launching the Study Month 

We monitored responses from providers and parents on a daily basis and followed up with the 

ones whose responses were missing from the previous day. The follow-up required multiple 

efforts by phone, email, and/or text to reach the provider or parent. It was particularly intensive in 

the initial days of the study month. Below we summarize the challenges for collecting responses 

from providers and parents, respectively. 

Providers. Unless there are two caregivers in the home (or time of day when the provider has 

some help), the provider is unable to take time away from the children to respond to the treatment 

group. We found specific time windows during the day to be most favorable for contacting 

providers. These time windows often coincided with children’s nap time in the early afternoon 

and time when day cares were closing in the early evening. It was challenging to contact all 

providers during these time windows, especially in the early days of the study month. In some 

cases, we left multiple voicemails before reaching the provider. 

In our outreach efforts to obtain meal serving times, we prioritized providers who, if reached, 

would maximize the data reported. For example, if a provider missed reporting on November 1 

and we did not reach her on November 2nd, we prioritized this provider for a follow-up call on 

November 3rd if she/he did not report meal service on November 2nd as well. As the study month 

continued, we prioritized to follow up with providers who had more frequent reporting to 

minimize missing data from these providers. The data collection team kept an active log to 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

23 

document providers with whom they had spoken, issues discussed, and solutions or follow-up 

actions if needed. Providers reported the following issues: 

• Some providers entered incorrect meal serving times and did not know how to correct them. 

Providers could change their time reporting on the same day, but not retroactively. The user 

manual and training video instructed providers on how to do so. The study team walked 

through the steps with these providers to make sure they could edit meal times if needed. 

Providers who needed to report for a previous day or change reporting on previous days had 

to call the study hotline or email the study team to obtain assistance.

• Providers reported the burden of the study on their routine, mostly in the beginning of the 

study month, as they needed to report the same information twice, once to their sponsors for 

reimbursement and once to the study team. 

• Providers rarely used the “Summary and Submit as Final” function despite email reminders 

to use this function to confirm daily entry. We emailed the providers to emphasize the 

importance of using this function, but few used the function. 

• Many providers were not savvy with mobile apps, even though they had access to 

smartphones. Some also lacked fast and steady internet access. In Texas, one-third of the 

providers were unable or unwilling to report using MSRS. Of the other two-thirds, most, if 

not all, needed to be contacted at least once (and most more than that) to fill in missing data. 

About one-third of the Texas providers were manual claimers with their sponsors, but these 

providers were not always the same providers and were unable or unwilling to submit data 

electronically to the treatment group. We spot-checked and called providers to address any 

problems with submissions. 

In addition, when we reached out to a provider who had some reporting challenges, we 

occasionally reached someone who needed assistance in Spanish. We had a Spanish speaker on 

the study team follow up with these providers to help them access and use MSRS as intended. 

Parents. Just as for providers, we followed up with parents daily if they did not report drop-off 

and/or pick-up times for the previous business day. CARS automatically sent parents two 

reminders on their mobile phones if they had not reported drop-off time or absence by 10:00AM 

and pick-up time by 7:00PM. In the actual study or field test, we received 3,282 entries in CARS, 

which is 72.7 percent of entries we expected to receive based on the number of children, their 

attendance pattern, and the occasional multiple daily drop-offs and pick-ups reported by parents. 

Thirteen parents did not send any information in CARS though they consented to the study. This 

response pattern was in contrast to that of the pilot test during which all participating parents used 

CARS daily to report. This suggests that the difference in data quality may have been due to the 

stronger rapport with parents developed during the pilot test recruitment and the offer of a more 

attractive incentive. 

Additionally, a technical bug caused CARS to send multiple reminders to parents on November 7, 

2018, which may have caused confusion among parents in the field test. We stopped the reminder 

function on CARS temporarily and sent reminders via our back-up text messaging system while 

the MSG IT team fixed the CARS reminder function. On November 10th, the CARS reminder was 

repaired and working as intended. 

During the pilot test, parents expressed in interviews a willingness to participate as a service to 

their providers, and providers encouraged parents to participate. The overall setting of the pilot 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

24 

test was more personal; the pilot participants had also agreed to take part in a cognitive interview 

about their participation, so they were more motivated to work with us. It was challenging to 

instill the same sentiment during the field test due to its larger scale. Moreover, the goal of the 

pilot study was to develop an instrument while the field test was to test the instrument to identify 

errors. With reduced incentives, providers were less likely to encourage parents to participate as a 

favor to the providers themselves. 

During the field test, the amount of time available for onboarding parents was shorter than for the 

pilot test because we needed to use pre-study month data as a baseline and chose to minimize the 

interruption of the business routine at day cares; this may have affected the results. The shorter 

onboarding window likely contributed to fewer responses during the first week of the field test. 

There were also uncontrollable external factors that affected participants in Texas, including the 

aftereffects of Hurricane Harvey and the Sutherland Springs church shooting on November 5th. 

These events directly impacted providers, causing some to close their doors for part of the field 

test period. 

CARS required parents to strictly follow the text format to process the message. The user manual 

specified the accepted formats and scenarios for multiple children and multiple drop-off; 

however, it still took parents some trial and error to send their messages successfully. In another 

example, the parents’ phone number had to be in the CARS data system to enable them to text 

CARS. If a parent texted CARS from a number different from the one we had obtained from the 

provider, CARS was not able to process the message from that parent. In some of these cases, 

parents had not yet consented to the study but thought they had. As their phone numbers were not 

in the CARS database, they received error messages. When parents received error messages, they 

sometimes gave up on trying a second time or did not contact the study team to troubleshoot. The 

problem was solved once we reached out to them. 

Additional channels of communication, besides parents’ cell phone numbers, could help improve 

response rates. In some instances, we were never able to connect with parents, leaving voice 

messages each time we called. In other cases, the phone number was no longer in service. Having 

their email addresses to send data requests or inviting them to join a Facebook group, as we did 

with providers, might improve future response rates. 

In summary, the most significant challenge with parents was to obtain consent from them, as their 

participation was voluntary. Compensation level differences might have impacted the parent 

response rate. The parents received $50 as an incentive during the pilot test, which was reduced to

$25 during the field test, as previously noted. The multiple reminders parents received on 

November 7th could have turned off some parents. However, it seems unlikely as the response rate 

on November 8th was higher than on any of the previous days since the study month started. 

3.5.3 Monitoring Data Reporting 

The data collection team relied on the monitoring dashboard the IT team created to keep track of 

daily reporting from providers and parents. Data collectors recorded follow-up actions in 

SmartSheets (software as a service application well suited for dashboard management), one for 

providers and one for parents. 

The monitoring dashboard is a built-in function of the MSRS system. It automatically logs 

information providers entered to their accounts, along with metadata, such as the time of data 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

25 

entry and reporting methods (mobile device or MSRS website). The data collection team used the 

dashboard to identify providers and parents who had not reported information on the previous day 

so they could follow up. 

The biggest challenge of the dashboard during the study month was the time difference between 

the server location and the time zone the providers and parents were in, which we did not 

anticipate. As a result, data from providers could show up on the wrong date in MSRS and on 

reports. This made it challenging for the study team to determine if a provider needed to be 

contacted for data collection. As an alternative, the IT team sent data received from providers and 

parents on the previous day to the data collection team every morning at about 9:00AM. The data 

collection team then examined the data to identify non-respondents, missing data, or other data 

reporting issues before following up with providers and parents. 

Unexpectedly, as the study month progressed, rapport grew between the MSG study team and the 

providers. This happened mainly with the providers who needed to be contacted frequently to 

update their data, but not exclusively. Phone calls with the providers became more sociable and 

conversational. Once we established connections with providers, their reporting became more 

frequent, and follow-ups became easier.

3.5.4 Parent Follow-up

Using the monitoring dashboard, we compiled a list of parents who needed to be followed up each 

day. Parents receiving follow-ups were those who had not reported the previous day’s drop-off 

and pick-up times, those who contacted the helpline after hours, and those who had yet to respond 

to any attempts at contact. The study team used cell phone numbers on record to connect with 

parents. To minimize disturbance to parents and maximize the chance to reach them, we made 

follow-up calls during lunch hours (12:00−1:30PM) and towards the end of the work day 

(4:00−6:00PM). We left voice messages if we could not reach parents directly, detailing the 

reason for the call and the information requested from parents. Once we spoke to a parent, we 

collected the missing drop-off and/or pick-up information on the phone so we could manually 

enter it into the database. We also requested parent email addresses if we needed to send or 

collect further information to address technical issues.

Some parents forgot to report child attendance information despite the two daily reminders. For 

some parents, CARS reporting became part of their daily routine after one or two days of 

reporting. For others, it took longer to get used to the specific text format that CARS required. 

While providing high-quality data, this data collection process was labor-intensive. If the process 

were replicated for large-scale study, the data collection team would need to be expanded to 

handle the amount of work in a relatively short but intensive time period. Additionally, it could be 

helpful to build more flexible text formats so that parents would not be discouraged by frequent 

error messages. For parents who appreciate self-monitoring, it could be beneficial to build a datatracking feature into the app that records the number of times a parent successfully enters the data 

so they can see their daily success.

3.5.5 Closing the Study Month

On the day after the study month ended, we sent an email to providers and parents thanking them 

for their participation in the study, which concluded on November 30, 2017. We also indicated 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

26 

that we might contact them about any recent missing data and that we would contact them soon 

about sending their gift cards by email. 

In mid-December, we emailed the sponsors a thank you note to confirm the conclusion of the 

study month and remind them that we would be asking for additional data once the meal claims 

for the study month were final. 

After the completion of the study month, the CACFP study team distributed incentives as Visa 

gift cards to participating providers ($60) and parents ($25). The study team used a third-party 

service—www.giftcards.com—to deliver the incentives via email. Compared with the incentive 

delivery method used during the pilot study—delivering physical gift cards by U.S. Postal Service 

certified mail—electronic delivery reduced mailing costs and increased delivery success. 

Through the recruitment process and direct engagement with the providers prior to and during the 

study month, the study team was confident that the email addresses associated with participating 

providers were accurate. Yet, to be certain, that the providers’ email addresses were still valid for 

the distribution of incentives after the study month ended, the study team used Constant Contact 

to send providers a verification email that (1) indicated incentives would arrive via email and (2) 

instructed providers to send a reply email to cacfp@manhattanstrategy.com that included the 

word “confirm” in the email body. By doing so, providers confirmed the validity of their email 

accounts and indicated that they understood how incentives would be delivered. To maximize the 

number of verified provider email accounts, the study team re-sent the verification email to nonrespondents. The study team then input its list of confirmed provider email accounts into the 

www.giftcards.com user interface, provided payment, and executed delivery. The processes for 

delivering incentives to parents took place concurrently. We sent parents text messages to obtain 

email addresses, which we authenticated, and distributed incentives to their email address as with 

providers. 

3.5.6 Collecting Data from Sponsors After the Study Month 

Sponsors, who had no active role during the study month, were subsequently asked to submit data 

on actual reimbursed claims once available, several months later. We created a detailed document 

with instructions for creating the MinuteMenu16 export reports that would contain all the data 

elements needed for the analysis. 

We started collecting claims data from sponsors in February 2018. We emailed sponsors in 

January 2018 and followed up until April to collect information from all the sponsors. To assist 

sponsors in extracting data from their system, we drafted a step-by-step instruction for sponsors’ 

use of the MinuteMenu software program. Most sponsors using MinuteMenu followed the 

instructions and sent us the requested data without questions. We also walked through the steps 

with sponsors over the phone to help them extract the requested data. 

Sponsors followed the same steps to upload data to MSG’s secure cloud-based data file-sharing 

system as they did during recruitment. We emailed sponsors a link and step-by-step instructions 

to upload data. Sponsors uploaded all requested data by the end of April 2018. 

16 MinuteMenu is widely used software system used by sponsors and providers to manage CACFP claims and 

reimbursements. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

27 

Some of the sponsors entered their FDCH meal claims data into MinuteMenu and were able to 

generate and upload the requested data quickly in a spreadsheet format. These data were easily 

transferred to our data processing system. Based on anecdotal reports, these sponsors may have 

had more providers who used the electronic claiming system, or they may have had staff to enter 

the data once the manual claims were received. 

Some sponsors used MinuteMenu or another computer program to submit their claims to the State 

Agency but still had manual claim data for some providers. For these sponsors, we offered 

options to scan or take photos of the manual claim sheets, and our staff entered the data into our 

data system. We did not collect data on how many providers within each sponsoring agency used 

electronic versus manual claiming methods or the systems sponsors used for submitting their 

claims to the State Agency. 

Some issues we encountered with creating the reports were solved by altering the instructions. 

These instructions were developed for the feasibility study, and updates were made based on 

sponsor feedback. For the sponsors with providers using manual claims, several weeks passed 

before we received the data because of the challenges they had with scanning and uploading the 

files. We ended up having photographs of the data sent to us instead of computer scans, which 

worked well. 

We then scanned the photographs into our file-sharing system so the data collection team could 

follow up with sponsors quickly regarding any high-level questions, such as missing information. 

Some trailing issues emerged in the middle of data processing, and the data collection team 

followed up with sponsors on a case-by-case basis. For data, we entered manually from scanned 

images or photos, and used the meal summary to check the accuracy of daily data by meal type. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

28 

4. Data Processing

As previously stated, the feasibility study uses data from three principal sources: CARS, MSRS, 

and sponsors. Exhibit 8 shows all the data processing steps. 

The left panel shows the steps we took to process the administrative data files from the sponsors. 

These steps are explained in detail in Section 4.1, Data from Sponsors. Sponsors provided four 

types of information: (1) provider-level meal counts by type, tier, and total reimbursement for the 

study and pre-study months; (2) meals providers in the treatment group submitted to sponsors for 

reimbursement in the study and pre-study months; (3) child information for the treatment group 

(e.g., child’s birthdate, normal attendance patterns, enrollment status, relationship, tier, and 

participation to CACFP); and (4) study-group provider information (e.g., provider’s eligible 

service days, open time, close time, standard serving times, license type, tier, and capacity).

The right panel in Exhibit 8 shows the steps to process the primary data collected from CARS and 

MSRS, detailed in Section 4.2, CARS, and Section 4.3, MSRS. Parents reported drop-off and 

pick-up times for their children in CARS. They also reported child absences and provider 

closures. Parents received a reminder to submit their information if it was unreported as of one 

hour after the expected drop-off or pick-up time. Providers used MSRS to report meal-serving 

times and the number of children who were present for each meal. Additionally, child absences 

and the providers’ closure were submitted. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

29 

Exhibit 8. Overview of Data Processing Steps
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

30 

The following sections show the steps for processing data from each source, i.e., CARS, MSRS, 

and sponsors. 

4.1 Data from Sponsors 

Sponsors supplied four distinct types of administrative data, each with its own format: (1) child 

attendance data, (2) sponsor-edited provider-level meal claims, (3) child information, and (4) 

provider information. This section discusses each type and its processing. 

Child Attendance Data. The most detailed data providers submitted to sponsors to show the 

meals for days on which the provider reported that the child was present. Sponsors sent data for 

all providers in the treatment group for the study and pre-study months. These data are the 

building block of claims submitted for provider reimbursement and are the processing focus. 

Most provider data stem from MinuteMenu system downloads to Excel workbooks. These 

workbooks are semi-structured; MSG developed a SAS program to read and format them. 

Fourteen providers submitted at least some of their data on paper copies only, which were faxed 

or emailed to MSG. This total includes one Virginia sponsor whose providers all submitted paper 

copies. A data entry specialist entered these into formatted Excel workbooks for use. Since the 

paper forms display children’s names, some processing to standardize names was required.

Processing of sponsor-reported data occurred as follows, and Exhibit 9 shows summary statistics 

of meals in the child attendance data file.

• Read each MinuteMenu sponsor workbook (nine participating sponsors with three months of

data) and converted the semi-structured data into structured data.

• Added each sponsor workbook to create a MinuteMenu master file.

• Read each manually entered workbook. No shift was assumed for these meals.

• Added each manually entered workbook to create a manual input master file.

• Added the manual input master file to the MinuteMenu master file to create a child

attendance master file.

• Merged this master file with MSRS data to see if attendance records were missing for any

MSRS children.17

Exhibit 9. Average Number of Meals Served by Providers in the Treatment Group

Month Meal Type # of 

Providers Mean Standard Deviation Min Max

September 2017 Breakfast 68 103.1 47.3 1 222

(Pre-month 1) Lunch and Supper 68 119.5 62.8 0 346

Snack 68 150.1 93.5 25 407

October 2017 Breakfast 68 117.3 54.4 20 257

(Pre-month 2) Lunch and Supper 68 132.9 65.8 0 318

Snack 68 158.1 96.1 34 444

17 Thirteen MSRS children did not have any attendance records and were dropped from further processing.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

31 

Month Meal Type # of 

Providers Mean Standard Deviation Min Max

November 2017 Breakfast 70 99.9 53.5 1 242

(Study month) Lunch and Supper 70 115.4 59.2 12 314

Snack 70 139.6 85.5 29 430

Source: Attendance data collected from sponsors (provider N =70 [varied by month])

Sponsor-edited Provider-level Meal Claims. The meal claim data represent the most 

consolidated records of meal claims based on which providers received reimbursement. The meal 

claims in this data file are shown at the provider level, displaying the number of approved meals 

by meal type, tier, and the total approved payment in a month. Sponsors provided data for all 

providers in the treatment and comparison groups for the study and pre-study months. Sponsors in 

the study extracted data from MinuteMenu in Excel workbooks, with one workbook per sponsor. 

MSG then merged the approved meal claim file with provider information we had collected and 

recorded previously at random assignment. The assignment information for FDCHs included 

enrollment size, language preference, and zip code. MSG appended Rural-Urban Commuting 

Area (RUCA) codes from USDA’s Economic Research Service based on the zip code. These data 

are maintained in an Excel workbook with the treatment/control indicator. 

Claims data processing consists of reading and appending the MinuteMenu Excel workbooks, 

reading the MSG provider workbook, and merging them. Exhibit 10 shows the summary statistics 

of information included in the claim file. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

32

Exhibit 10. Average Attendance Days and Meals/Snacks Claimed by Providers

Meal 

Claim 

Month

Attendance/ 

Claimed Meal 

Type

# of

Providers Mean Standard Deviation Min Max

Comparison September # days 182 20.1 3.4 5 30

2017 Breakfast 182 99.5 75.1 0 356

(pre-study Lunch and supper 182 163.6 109.5 0 546

month 1) Total snacks 182 163.7 108.2 5 786

October # days 181 22 2.8 11 31

2017 Breakfast 181 110.3 81.2 0 465

(pre-study Lunch & supper 181 179.6 117 0 613

month 2) Total snacks 181 177.9 107.9 18 638

November # days 180 20 3.1 8 30

2017 Breakfast 180 101.9 72.9 0 357

(study Lunch & supper 180 167 107.7 0 563

month) Total snacks 180 163.4 105 0 621

Treatment September # days 63 19.9 2.3 15 30

2017 Breakfast 63 106.5 57.9 0 228

(pre-study Lunch & supper 63 131.1 84.4 2 489

month 1) Total snack 63 140.9 68.3 25 328

October # days 63 21.6 2.1 16 31

2017 Breakfast 63 118.4 61.4 0 260

(pre-study Lunch & supper 63 144.7 89.5 12 550

month 2) Total snacks 63 154.6 69.5 37 320

November # days 63 19.2 2.6 14 29

2017 Breakfast 63 103.8 54.7 0 236

(study Lunch & supper 63 126.1 74 12 454

month) Total snacks 63 130.3 59.8 23 270

Note: The meal/snack statistics are combined numbers from all the different tiers.

Source: Claims information collected from sponsors at the provider level (provider N=244)

Child Information.18 Sponsors provided child information Excel workbooks from MinuteMenu 

for the treatment group at the child level, i.e., one row per child. For each child, the data 

included birthdate, expected attendance patterns, enrollment status, relationship to the provider 

(e.g., provider’s child), tier, and the status of CACFP participation, among others. Data used in 

this study were limited to relationship to provider, participation, and enrollment date for all 

providers; and tier status for the one provider with mixed tier claims. Data processing of the child 

information files consisted of reading the ten sponsor workbooks, appending them together, and 

joining them with the MSRS and CARS data by child ID. 

18 Note: sponsors supplied information on children from providers outside the treatment group also.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

33

Provider Information.

19 MSG requested data for all treatment group providers. All participating 

sponsors extracted provider information from MinuteMenu. This data file contained many 

variables we used in the triangulation process for the treatment group. The workbooks were 

organized with one row per provider showing their eligible service days, opening time, closing 

time, standard meal serving times, license type, tier, and capacity. The files were processed by 

reading each of the ten sponsor workbooks, adding each to create a master file, and merging it 

with the MSRS data for triangulation use. Exhibit 11 shows the summary statistics of variables in 

the provider information data. 

Exhibit 11. Allowed Meal Types, Meal Combinations Offered, and Tier Status for Providers in 

Control and Treatment Groups

Control (n=136) Treatment (n=63) Total 

(n=199)

Breakfast No 32 23.53% 7 11.11% 39

Allowed Yes 104 76.47% 56 88.89% 160

Lunch and None 27 19.85% 5 7.94% 32

Supper Lunch or supper 60 44.12% 36 57.14% 96

Allowed Both 49 36.03% 22 34.92% 71

Snacks 0 27 19.85% 5 7.94% 32

Allowed 1 63 46.32% 32 50.79% 95

2 33 24.26% 22 34.92% 55

3 13 9.56% 4 6.35% 17

Meal Other combinations 71 52.21% 31 49.21% 102

Combination 1 meal and 2+ snacks 3 2.21% 0 0.00% 3

2+ meals and 1 snack 62 45.59% 32 50.79% 94

Tier Status 1 117 86.03% 55 87.30% 172

2 14 10.29% 6 9.52% 20

Mixed 5 3.68% 2 3.17% 7

Source: Provider information collected from sponsors (provider N =199)

4.2 Data from CARS

The CARS system output consists of one row per child per day, regardless of whether the activity 

is recorded. The CARS data include the drop-off time, pick-up time, absence indicator, closure 

indicator, various timestamps, and the actual text messages from parents. The CARS system 

parses out pick-up and drop-off time into different columns and records whether its source is from 

a text message or the CARS website.

The system generates daily Excel workbooks for the data collectors to contact parents who did not 

report child attendance time or status in the previous day. When data were collected via follow-up 

calls, the data collectors entered the child attendance information in the data file and indicated the 

data collection method as Manual Phone. Parents could also call an 800 number and leave a 

message or communicate through TextMagic, an SMS system we used as a backup to Twilio. 

19 Note: sponsors supplied information regarding providers outside the treatment group.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

34

These were captured in additional workbooks and entered by the data collector with the method 

designated as either Phone Message or TextMagic. 

CARS processing followed ten steps. 

1. Read the CARS system output file.

2. Read and consolidate the non-system files. This includes the daily workbooks, phone message 

workbook, and TextMagic workbook.

3. Update the CARS system output file with the consolidated non-system files.

4. Output an Excel workbook with potential data that were incomplete or needed further 

processing. For example, some SMS messages were not in the format CARS could 

automatically process or included drop-off or pick-up status without a time.

5. Manually review these data in the Excel workbook. 

6. Update the CARS data with the manually reviewed data for the first pick-up and drop-off for 

each child.

7. Append the records for any instances of second drop-offs and pick-ups.

8. Sum the number of times each child is dropped off and picked up with a valid time value.

9. Compute the mean drop-off and pick-up times of children at a day care.20 We used the mean 

times for a given child to replace missing drop-off and pick-up times.

10. For non-holiday weekdays with missing information, substitute with the mean values 

generated in the previous step. Consider the record valid if the child is reported as absent, the 

day care reported as closed, or a drop-off and pick-up time exists. Output the final dataset.

Exhibit 12 shows that children attended participating day care homes for an average of 12 days in 

November 2017. The average attendance was about 11 days in Texas and 13 days in Virginia.

The attendance of children in the five participating sponsors in Texas ranged from 10 to 14 days 

and in Virginia from six to 15 days. On average, children in the day care homes of the nine 

participating sponsors had approximately one absence and the homes closed for about one day in 

November 2017. The average attendance time of a child was about five hours and 14 minutes.

Exhibit 12. Child Attendance in the Study Month Reported in CARS (Child n=215)

Mean (Standard Deviation)

Level Days Attended Days 

Absent

Days 

Closed

Daily 

Attendance 

(hours:minutes)

Total 12.0 (8.5) 1.2 (2.2) 1.2 (1.5) 5:14 (4:02)

State

Texas 11.3 (8.5) 1.1 (1.9) 1.3 (1.7) 5:00 (4:13)

Virginia 13.3 (8.3) 1.4 (2.5) 1.0 (1.1) 5:40 (3:41)

20 When calculating the mean times, we included only children at the day care home with five or more drop-offs or 

pick-ups.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

35 

Exhibit 13 shows the number of parents reporting via different means, including SMS (CARS 

mobile application), CARS website, follow-up phone calls, study hotline messages, and 

TextMagic. The use of CARS mobile application was the most widely used method of reporting; 

we also collected a considerable number of drop-off and pick-up times from the CARS website. 

Exhibit 13. Number of Absent, Drop-Off, Pick-Up Occurrences Reported in CARS by 

Reporting Method 

Since CARS primarily utilizes free-form patterned messages through SMS, TextMagic, and the 

like, many have errors needing correction. The most common errors were around the drop-off and 

pick-up dates and times, particularly correcting for blatant errors 12 hours off. For example, 

6:00PM was entered for the drop-off time when it should have been 6:00AM, so a correction was 

made. Exhibit 14 shows the reasons that led us to correct the CARS responses. 

Exhibit 14. Counts of Manual Correction in CARS

Corrections in CARS Counts of Occurrence

Fixed drop-off date/time 21

Fixed misreporting of am and pm for drop-off time 20

Fixed misreporting of am and pm for pick-up time 12

Added drop-off and pick-up date/time 10

Removed duplicated reporting 9

Fixed pick-up date/ time 8

Deleted reporting of closure 6

Fixed drop-off and pick-up date/time 4

Added drop-off date/time 4

Added pick-up date/time 3

Added Absence 3

231

8 8 0 10

1765

61 56 16 25

1517

65 56 21 27

0

200

400

600

800

1000

1200

1400

1600

1800

2000

SMS Message Web Phone Call Voice Mail TextMagic

Counts of Occurrences

Reporting Methods

Absent Drop-Off Pick-Up
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

36 

Corrections in CARS Counts of Occurrence

Added Closure 2

Fixed misreporting of closure 1

Deleted reporting of absence 1

4.3 Data from MSRS

The MSRS system output consists of one row per child per day, regardless of activity recorded. In 

addition to child and provider identifiers, the data file includes absence and closure indicators and 

timestamps, confirmed flag, and a four-column array indicating the FDCH operating status (open, 

closed, or partial closure), time meal served, stamp for the time of reporting, and reporting 

method for each of the six possible serving types. We converted these data to one row per child 

per meal type for each day to simplify updating with manual data. 

As with CARS, daily Excel workbooks were also output for two individual data collectors to 

contact nonreporting providers. These data were consolidated and used to update the MSRS data. 

MSRS processing followed ten steps: 

1. Read the MSRS system output file.

2. Convert the file from one row per child per day to one row per child per meal type per day. 

3. Read and consolidate the manually collected input files. 

4. Update the MSRS system output file with the consolidated manual input files. Flag valid 

records as those with a serving time, an absence indicator, a closed indicator, a Not Served

status, or a confirmation of meal reporting for the day. 

5. Sort by date, provider, and assumed meal sequence of breakfast, AM snack, lunch, PM snack, 

supper, and evening snack. 

6. Assign invalid sequence flag if serving time for subsequent meal type before serving time of 

prior meal type. Assign valid flag to all provider records for the day if any record for the day 

is in proper order. 

7. Specify last participation date as a maximum date with a valid record.

8. Make AM/PM corrections. If serving time after 6:00PM for breakfast, AM snack, or lunch, 

subtract 12 hours. If serving time before 6:00AM for PM snack, supper, or evening snack, add 

12 hours. 

9. Compute the mean serving time for each provider and meal type for providers whose last 

participation date is November 8, 2017, or greater. These means will be used for the 

imputation of missing data. 

10. For the meal-serving time that was not reported on days a day care was open, substitute the 

mean meal-serving time of the provider for the missing data. Output the final dataset. 

Exhibit 15 shows that, during November 2017, the 63 participating providers served 5,558 

breakfasts, 45.3 percent of the total possible breakfasts that could be served. We combined lunch 

and supper because their reimbursement rates were the same. Similarly, we combined the 

reporting of snacks in the morning, afternoon, and evening. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

37 

Exhibit 15. Meal Service Status in MSRS by Meal Types 

Meal Type Served Not Served

Breakfast, n (%) 5,558 (45.3%) 6,706 (54.7%)

Lunch/Supper, n (%) 6,045 (24.6%) 18,483 (75.4%)

Snacks, n (%) 7,752 (21.1%) 29,040 (78.9%)

4.4 Lessons Learned from Data Collection and Processing

We make the following suggestions to improve data collection and processing based on lessons 

learned from the feasibility study. 

• Capture child and provider MinuteMenu or other ID at the beginning of the study for 

consistent use. This ID could be used in CARS and MSRS or in a cross-reference table 

linking all tables. 

• Contemporaneously edit times to ensure AM and PM designations are properly 

entered. Although it would be ideal to enter all times in a 24-hour clock standard, it is

unlikely that parents and providers can readily relate to this. The CARS and MSRS 

applications could be modified to automatically verify AM and PM designations at the time 

of entry or perhaps the following day. Simple checks for CARS could be drop-off time 

before pick-up time. For MSRS, simple checks could be breakfast and AM snack in the 

morning, and supper, PM snack, and evening snack in the evening. Lunch checks for AM or 

PM would not work much of the time, but lunch checks for after breakfast or AM snack and 

before PM snack would be reasonable. 

• Improve text processing of the CARS application. Most of the CARS entries are freeform texts requiring decomposition or interpretation. Because of the number of errors 

corrected manually, CARS might be improved to capture and validate more entries 

automatically.

• Obtain each State’s rules for valid meals at the beginning of the study. We applied each 

State’s licensing rules regarding the number of children allowed by age or number of 

caregivers, and elapsed time between meals, if any. These rules should be obtained early to 

help decide what information needs to be obtained. 

• Modify MSRS to capture two additional data points. MSRS should be modified to (1) 

allow input of the number of caregivers present for each meal and (2) if the meal counts as a 

shift-one or a shift-two meal. The number of caregivers impacts the number of children 

allowable for a meal, and the shift number impacts the validity of the service times.

• Obtain office error reports (OERs) from sponsors from MinuteMenu. In addition to the 

four reports we obtained from sponsors from MinuteMenu, the OER report showing why 

claims were denied in the system would be beneficial. These could be compared with our 

estimates to further estimate a proxy of the truth. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

38 

5. Validating the Data Collection Methods

The goal of the feasibility study is to develop and validate a measurement of improper meal 

claims (i.e., meals claimed but not served) and the related improper payments under the 

“business-as-usual” condition, as directed by the Performance Work Statement provided by FNS. 

In this study, we relied on two approaches to estimate improper payments: (1) a triangulationbased approach that verifies a claim from multiple sources of data and (2) a difference-indifferences approach that estimates error from in an inferential statistical model in an 

experimental design. 

The triangulation approach assumes that improper claims may exist even after sponsors edit the 

claims. Sponsors may not have had the data needed to determine the validity of every claim. For 

example, they did not have detailed attendance information, such as when a child is absent for a 

meal on a day for which the child is enrolled. Compared with the raw claims, the edited claims 

better reflect accurate claims, although State agencies review the sponsor-edited claims and may

make further corrections to claims before issuing reimbursement. As sponsors identify and correct 

improper meal claims during their review, potential improper payments are avoided. Claims that 

sponsors correct, therefore, are excluded from the feasibility study. The triangulation-based 

approach to identifying improper claims, however, does not take into consideration that meal 

claims may fluctuate across months in the business-as-usual condition. 

To control for this natural fluctuation over time and differentiate it from the change in meal 

claims when MSRS and CARS are implemented, we used a two-group pre-post experimental 

design, the most rigorous method for testing the effectiveness of an intervention, to improve the 

accuracy of meal claims. Specifically, we used an experimental design with a treatment condition 

and a control condition. Under the treatment condition, the providers used MSRS to report mealserving times. Parents whose children were attending FDCHs included in the treatment condition 

used CARS to report daily the drop-off and pick-up times of their children, thus enabling us to 

determine exactly when the children were in attendance. We did not inform providers and parents 

in the control group of the study to preserve routine practices in a business-as-usual condition. 

5.1 Triangulation 

The purpose of triangulation is to derive the most accurate records of meals served by providers 

in the treatment group during the study month (November 2017). Triangulation refers to the 

process of using multiple data sources to verify a meal claim, which is a provider report of a meal 

served to a child. Since the study focus was on accurately measuring meals that were claimed but 

not served, the triangulation process began with data from the child attendance file, which is the 

basis of meals claimed. If a child was not in attendance for the meal, the meal should not be 

claimed. 

As shown in the left panel in Exhibit 8, we started with the administrative child attendance 

records to apply rules about meals that should not be claimed based on administrative data. We 

then compared data parent-reported on child attendance for a given day from CARS with 

provider-reported data from MSRS to flag meals reported as served when the child was not in 

attendance. Finally, we compared MSRS data with administrative records to flag meals that were 

ineligible for reimbursement. We compared the triangulated meal records with the actual meal 

claims sponsors reimbursed for providers to generate estimates of improper meals. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

39 

Sponsors may have had data outside the scope of this study, such as specific menu information, 

and may have disallowed meals for reasons we do not consider. For this reason, we considered 

the minimum of our estimated meals served and the sponsor-approved meals to be the most 

accurate records of actual meals served. These data for the treatment providers in the study month 

were then combined with the claims in the pre-study months for the difference-in-differences 

(DD) modeling. A DD model uses a statistical technique to simulate an experimental research 

design by comparing observed data from a treatment group—in the case data reported during the 

study month using MSRS and CARS—with a comparison group that uses routine procedures for 

meal serving and child attendance.

Administrative Record Errors. We used administrative data to ascertain improper meal claims 

that could have been caught by a sponsor. We identified eight possible such conditions that 

resulted in an error: 

• Serving meal types that were not specified in the approved application with the sponsor, 

e.g., a provider claimed supper even if they were not approved to serve supper (from the 

provider information file per Part 226 Child and Adult Care Food Program § 226.18). 

• Serving a meal on a day unauthorized for the provider (from the provider information file).

• Serving a meal to a child who was a nonparticipant (from the child information file). 

• Not having a child enrollment form on file, not updating the form within the past 13 months, 

or having an expired form (from the child information file per Part 226 Child and Adult 

Care Food Program § 226.18).

• Claiming meals for only one child—the provider’s own child (per Part 226 Child and Adult 

Care Food Program § 226.18). 

• Claiming a meal that exceeded the maximum number of meals allowed in a day, e.g., a 

combination of two meals and a snack or two snacks and a meal (per Part 226 Child and 

Adult Care Food Program § 226.18). 

• Claiming meals for a day that exceeded the FDCH’s maximum approved capacity, i.e., the 

maximum number of meals that can be served, as specified in agreement with the sponsor 

(per Part 226 Child and Adult Care Food Program § 226.18). 

• In Texas, the mix of children receiving a meal based on exceeding the authorized capacity

depending upon the children’s ages. Both Texas and Virginia have rules regarding the 

number of caregivers necessary to be present based on the mix of children’s ages, although 

we did not have information on how many caregivers were licensed for each provider in 

Virginia as we did for Texas. 

These errors should have been caught by the sponsor before approving payment. The 

MinuteMenu system, if used by a provider, can be configured to uncover many of these errors. 

This part of the triangulation identifies errors that sponsors should have caught. Errors associated 

with the meal pattern, such as the actual content of meals served, fall outside the scope for this 

feasibility study. 

Administrative records error processing proceeded as follows: 

• Read three administrative files: child attendance, child information, and provider 

information. 

• Read the MSRS processed database file.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

40 

• Summarized elapsed times for meals and between the opening and breakfast and last meal 

served and closing reported by the providers. 

• Merged the files. 

• Applied the authorization, participation, and enrollment rules. We applied these rules for 

each meal. These corresponded to the first four of the eight conditions listed above. 

• Grouped by date and child. This was necessary to apply the maximum number of meals per 

day (error 6 in list above). 

• Grouped by provider, date, and meal type. This was necessary to evaluate “own child” and 

capacity errors (errors 5, 7, and 8 in list above). 

• Sorted the merged data by provider, date, meal type, and descending child birthdate (from 

the child attendance file).

• Applied two-and-one, own child, and capacity rules. 

– The two-and-one logic was applied to specific meals sequentially. If the meal count 

equaled three, breakfast was designated as the improper meal because its 

reimbursement value was lower than the other two meals. If the snack count equals 

three, AM snack is defined as the improper snack. For errors over two meals and one 

snack, and one of the snacks was AM snack, the AM snack became an improper snack. 

Proceeding through the logic, the PM snack was designated as improper, then the 

evening snack. 

– If the only child at a meal is the provider’s own child, the meal was marked as 

improper. 

– In applying the maximum capacity rule, meals served over the capacity were flagged as 

improper to the oldest children first. MinuteMenu uses this logic. Only the incremental 

meals over capacity were flagged as improper. For instance, if the maximum capacity is 

12 and the provider showed 14 children received the meal, the meals served to the two 

oldest children were improper. Capacity errors were computed separately for the first 

serving time (the vast majority of cases) and second serving time. 

– The Texas, child mix capacity rules were applied in a manner similar to the maximum 

capacity rule. The oldest children who received meals over capacity were improper. 

– Since multiple errors may have existed for any given meal, a flag was established for 

any errors, so the improper meal was only counted once. 

• Outputted the Detailed Child Attendance/Admin Errors intermediate dataset. This was used 

later in the final triangulation process. 

• Summed the served meals from the child attendance records.

• Summed the improper meals.

• Read the claims data.

• Merged the summarized data together. Because the reimbursement rates were based on tiers, 

we had to estimate the meals served by tier. We summed the tier 1 and tier 2 meals claimed 

by meal type (from the claims file) for each provider. We estimated the meals served after 

adjusting administrative records for the meal type as the sum of the served meals minus the 

improper payment flag. We allocated to tiers based on the providers’ tier-approved claims as 

a percentage of their total claims for that meal type. If a provider only submitted tier 1 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

41 

claims, 100 percent of their estimated meals were tier 1. One provider had mixed claims, 

and so their meals were allocated to the tiers based on their percentage.

• Output the Summarized Child Attendance/Admin Errors intermediate dataset. This was used 

later in the final triangulation process.

• Took the minimum of the estimated and the sponsor-approved meals. Since sponsors may 

have had data that we did not have, we prepared a Final Claims Meal Errors datafile for 

modeling that used the minimum of the sponsor-approved meals and our estimated meal 

count. 

• Output the resulting Claims Meals/Admin Errors for Modeling intermediate dataset.

Exhibit 16 shows the steps in the triangulation process. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

42 

Exhibit 16. Triangulation—Administrative Record Errors
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

43 

Exhibit 17 shows the number of errors in meal claims after we applied the administrative rules to 

meal claims providers submitted to sponsors. Errors in Exhibit 17 may well be identified by 

sponsors in the review process; if not, they would result in meal claim errors. The greatest number 

of errors stemmed from exceeding the maximum number of meals allowed for a day (a two-andone error). These account for about half the total. Capacity-related issues, both from exceeding 

the maximum capacity in both states or from the child mix specifically for Texas, represent the 

next highest category. A number of errors also occurred because a meal was claimed for a child 

not participating in CACFP. Child enrollment problems and unauthorized serving types were rare. 

A total count of 1,322 errors occurred. 

Exhibit 17. Errors of Meals Served by Type at the Child Level Identified After the 

Application of Administrative Rules 

Meal 

Type

Only Own 

Child 

PresentServing Unauthorized for Provide Serving Type Unauthorized for ProviderChild not 

Participate in 

CACFPChild 

Enrollment 

ExpiredTwo-and-One problem1Serving Over Max Capacity2Serving Child Mix Over Meal Capacity - TX2Total3

Breakfast 30 16 0 40 0 101 16 70 263 

Lunch 8 15 0 47 0 0 0 84 154 

Supper 0 0 0 17 0 0 24 0 41 

AM Snack 0 3 0 13 0 493 0 45 541 

PM Snack 44 22 0 48 0 81 55 73 320 

Eve. Snack 0 0 1 0 0 3 0 0 3 

Total 82 56 1 165 0 678 95 272 1,322 

1 Meals were eliminated in this order: Breakfast, AM Snack, PM Snack, Evening Snack.

2 Meals were corrected for only those over capacity with the oldest child first. For example, if six present and four is 

capacity, the oldest two are considered improper. 3 Totals may not sum because multiple reasons were eliminated in the total, such that each meal for a child could 

only have one error.

Source: Child Attendance File

CARS Errors. Since parents have no vested interest in inflating meal counts, parental data from 

CARS should provide an unbiased report on child attendance time, which serves as the validation 

of providers’ reporting of meals served. The CARS data shows the drop-off and pick-up times of 

each child and if they were absent or the provider was closed. Sponsors would not have this data 

to adjust the provider claims. 

We first merged the CARS data (one row per child per date) with the MSRS data (one row per 

child per date per meal type) that the providers entered daily by date and child ID. We then 

merged these data with the detailed child attendance error data file, as described in the section 

above. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

44 

One issue with the three major data sources (CARS, MSRS, and child attendance) is conflicting 

data on closures. Within CARS itself, conflicting reports of closures may come from different 

parents. One parent may have shown the provider as closed, while another may have reported

they dropped off and picked up their child on the same date. The provider may have also shown 

as being open that day in MSRS, but there might have been no attendance records submitted. We 

summed the data from the various sources by provider to determine a closure probability.21

Similarly, an absence probability was also computed. For absences, however, there was no 

possible conflict with CARS data. This is because if a parent communicated an absence through 

CARS the child was designated as absent.

To determine meals claimed when a valid drop-off or pick-up record existed, we allowed a 30-

minute window before the meal was considered improper. If the meal serving time from MSRS 

was more than 30 minutes prior to the drop-off time, the meal was improper because it was served 

before the drop-off time. If the meal was served after the pick-up time plus 30 minutes, the meal 

was improper for being served after the pick-up time. 

The number of CARS adjustments of the MSRS-claimed meals were summed across reasons. The 

majority have only one reason. 

MSRS Errors. For meals that were adjusted based on CARS attendance data, we evaluated the 

meals for conflicts with the provider-entered data into MSRS. We assessed only the meals 

supplied on non-holiday weekdays (note that the November study month included the 

Thanksgiving holiday and, for some providers, the day after Thanksgiving), and determined that a 

payment was improper, if claimed, and one of the following conditions was met: 

21 For CARS errors:

Closures (by day): We summed the number of parents reporting a closure for the day in CARS. If some parents 

reported a closure and the number reporting a closure is greater than the number reporting a pick-up, drop-off, or 

absence, we give the reporting a value of one to CARS closure. If a provider reported “Closed Today,” we assigned 

one to MSRS closure. If the provider reported no meals served for the day, assign 1 to No Attendance Data. Sum 

CARS closure, MSRS closure, and No Attendance Data. This ranges from zero to three. Other data was collected to 

see if there were any CARS, MSRS, or Attendance Data for the day. These are all binary variables. We sum them to 

get the number of closure sources, which ranges from one to three. Compute the closure probability by dividing the 

sum of the sources indicating a closure by the number of reporting sources for the day. For any meals served that 

day, use this probability for an expected value that the meal is served in error. The range is zero to .6667.

Absences (by specific meal): We summed CARS absent, MSRS absent, and No Attendance Data (for the day). This 

ranges from zero to two and we then divided the sum by the closure sources stated above. If there’s conflicting data 

from CARS (e.g., an absence and pickup or drop off), we set the absence probability to zero. If they are reported 

absent in CARS, set the absence probability to 1. We used this probability for the expected value that the meal is 

served in error, which ranges from zero to one. We summed all the possible CARS issues (meal served when closed, 

meal served when absent, meal served before drop-off, meal served after pickup). The range is zero to one.

For MSRS Errors (if questionable CARS is not flagged as 1):

If a provider reported in MSRS “Closed Today,” we assigned one to MSRS closure, which indicates meal served in 

error. The value of this indicator is either zero or one. If MSRS shows a child was absent, any meal served to that 

child was in error. It’s either zero or one. We summed all possible MSRS issues. The range is zero to three.

“Disallow” a meal if questionable CARS total or MSRS total is equal or greater than one. The net result is that none 

of the CARS closures result in a disallowed meal while 155 reported absences do. All the MSRS closures and 

absences result in disallowed meals. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

45 

• Provider was closed for that day or relevant part of the day. 

• Provider entered the child as absent for that meal.

• Provider specifically indicated that the meal was “Not Served.” 

• Provider entered a serving time before the provider’s opening time (from the provider 

information file).

• Provider entered a serving time after the provider’s closing time (from the provider 

information file).

• The serving time for the specific meal type was before or after the provider’s meal time 

(from the provider information file). 

• Specific policies regarding meal service times for Texas, including:

– Insufficient elapsed time between meals and snacks; 

– Too much time to serve a meal or snack; and

– Supper time too early, too late, or too close to closing time. 

Total Errors. We then summed the improper meals for MSRS conditions and flagged improper 

meals based on MSRS or CARS data. We created a flag to keep track of meals after adjustments 

based on data from sponsors and CARS. 

We then summed the data by provider and meal type and computed error percentages. Because 

not all of a provider’s parents participated in CARS, and of those participating, they did not 

always submit complete data for a day, we applied the CARS error percentage to all records for a

provider based on CARS reporting from parents who provided complete data. Similarly, we 

applied the MSRS error percentage to all records for the provider. As required to determine the 

dollar amount of improper payments, we assigned improper payments to tiers based on the 

provider’s tier code, or in the case of the mixed-tier provider, the child’s tier code. This was 

necessary because the reimbursement amount varied by tier. 

We then summed CARS and MSRS errors by meal and tier for each provider and merged with the 

summarized child attendance error data file. Total errors were calculated as the sum of the 

administrative records errors, CARS errors, and MSRS errors. Reimbursement rates were used to 

obtain the improper payments dollar amount and the estimated provider original submission 

amount based on the child attendance records. We computed improper payment percentages for 

the number of meals and dollar amounts and generated a data file with all the observed data with 

flags for all errors we discussed above. 

To prepare a data file for the difference-in-differences modeling, we merged the minimum meal 

counts by provider from the administrative records error process with the CARS and MSRS 

adjustment data process and claims file. We subtracted the CARS and MSRS adjustments from 

the administrative records error process meals to get to the final best records of meals served for 

the treatment group in the study month. These data were then used for the modeling process. 

Exhibit 18 depicts the steps used to identify errors by triangulating CARS and MSRS data. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

46 

Exhibit 18. Steps to Identify Errors in Meal Claims Using CARS and MSRS Data
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

47

Applying the CARS and MSRS portions of the triangulation process uncovered 3,513.5 estimated 

errors, as shown in Exhibit 19. The first six columns display the errors discovered by utilizing 

parental reporting. If we had missing data on CARS regarding a child’s drop-off time, we filled it 

in with the typical drop off time of that child (ref. footnote in Appendix 1); similarly, if we had 

missing data on meal serving time on MSRS, we used the typical time the provider served that 

meal. These project values were then used to calculate meals served before a child was dropped 

off or after a child was picked up, as well as MSRS errors in Exhibit 19. The major error factor is 

meals served before the child is reported as being dropped off. As expected, this is a problem with 

breakfast and somewhat with AM snacks. Parents also reported their child as absent a number of 

times when the provider claimed the child in attendance for a meal. In the case of attendance, the 

parent’s report was always considered correct, thus making the provider claim improper. Overall 

about one-third of the total errors were discovered using CARS data and two-thirds from MSRS.

Exhibit 19. Claim Errors Identified through Triangulation of CARS, MSRS, and 

Administrative Data, by Error Type

Meal 

Type

Child 

Absent

Provider 

Closed

Meal Served Before 

Drop-off

Meal Served After 

Pick-up

MSRS 

Error

Total

Error 1

Actual Actual Actual Projected Actual Projected Projected Projected

Breakfast 58.5 42.2 110.0 347.9 0.0 0.0 566.0 1,046.3

Lunch 67.0 40.8 19.0 115.2 2.0 3.3 830.4 1,085.2

Supper 6.7 3.0 0.0 0.0 12.0 21.4 126.8 156.1

AM Snack 31.8 7.3 8.0 142.9 0.0 0.0 156.0 389.8

PM Snack 45.5 37.2 5.0 8.9 35.0 67.9 615.8 761.9

Eve. Snack 1.0 0.0 0.0 0.0 2.0 7.0 67.3 74.3

Total 210.5 130.5 142.0 614.9 51.0 99.6 2,362.3 3,513.5

1 Totals may not sum because of projections not shown and duplicate elimination.

Sources: CARS, MSRS, and Adjusted Child Attendance

Exhibit 20 breaks out the MSRS errors into various causes. The major error type was serving a 

meal before the standard time, which is more than half the total. By meal, this was especially a 

problem for breakfast, lunch, and AM snack. Meals served after the standard time was also a 

problem. We also found a large number of errors when the Texas providers did not allow enough 

time between meals. Virginia may have similar requirements, but the state agency could not 

provide us with official documents for such requirements. In total, we found over 5,000 improper

meals from using MSRS data. Since our order of precedence in error reporting began with CARS

and we avoided double-counting, the projected number of improper meals attributed to MSRS fell 

to 2,362.3. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

48 

Exhibit 20. Meal Claim Errors Identified through MSRS, by Error Type 

Meal Type

Child 

Absent

Provider 

Closed

Meal 

Reported as 

"Not Served"Meal 

Serving 

Time 

Outside 

Operating 

HoursMeal 

Served 

Before 

Standard 

Time

Meal 

Served 

After 

Standard 

Time

TX Only

– Insufficient 

Elapsed 

Time 

Between 

MealsTX Only

– Overreporting of Meal 

Serving TimeTX Only – Supper Time Too Early, Too Late or Close to 

Closing

Total MSRS

Actual1 Projected2

Breakfast 64 100 20 3 695 510 0 108 0 1,377 566.0 

Lunch 63 98 5 17 761 38 469 126 0 1,532 830.4 

Supper 7 9 7 1 172 6 17 0 17 209 126.8 

AM snack 18 22 2 0 479 140 235 0 0 835 156.0 

PM snack 64 81 15 9 642 401 85 144 0 1248 615.8 

Evening snack 2 0 1 40 96 0 2 0 0 138 67.3 

Total 218 310 50 70 2,845 1,095 808 378 17 5,339 2,362.3 

1 Totals may not sum because of duplicate elimination.

2 Projected numbers lower than actual because MSRS correction only applied to records without a CARS adjustment to avoid over-counting.

Sources: CARS, MSRS, and Adjusted Child Attendance
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

49 

The average provider incurred 16.2 improper meal claims over the study month, uncovered 

through the CARS and MSRS triangulation process (see Exhibit 21). Of the providers claiming 

each specific meal, breakfast, lunch, AM snack, and evening snack were about equally 

problematic. Because so many more providers claimed breakfast and lunch meals, they account 

for the most errors. 

Exhibit 21. Total Errors in Study Month Identified Through Triangulation of CARS, MSRS, 

and Administrative Data, by Meal Type

Meal Type Number of 

Providers

Average Number of Improper

Meals per Provider1 Total Error 

Breakfast 59 17.7 1,046.3

Lunch 60 18.1 1,085.2

Supper 15 10.4 156.1

AM snack 22 18.6 389.8

PM snack 58 13.1 761.9

Eve. snack 4 18.6 74.3

Total 16.2 3,513.5

1 Of the 63 providers who initially agreed to participate, 61 used MSRS beyond the first week. The mean is of 

providers claiming that specific meal type. For example, only four providers had any claims for evening snacks, so 

the mean is for those four providers.

Sources: CARS, MSRS, and Adjusted Child Attendance

In looking at the CARS and MSRS error percentages by meal type, evening snack—the lowest 

claimed meal by four providers—had the highest percentage of errors at 40.6% (see Exhibit 22). 

AM snacks had a 27.3 percent error rate, and all three types of meals had similar error rates. The 

overall error rate was 17.5 percent (of the net meals after correcting for administrative errors). 

Appendix 5 shows the percentage of errors due to meals claimed but not served. 

Exhibit 22. Error Rates by Meal Type

Meal Type Error Rate

Breakfast 17.90%

Lunch 17.90%

Supper 19.10%

AM snack 27.30%

PM snack 13.20%

Evening snack 40.60%

5.2 Difference-in-Differences Method

We applied the two-group experimental design in two study periods—a study month and two prestudy months—to test the assumption that MSRS and CARS allow us to identify errors in meal 

claims and therefore allow us to improve the accuracy of the meal claims reported by the 

treatment group during the study month. Exhibit 23 illustrates the conditions of the control and 

treatment groups across the two study periods. The control group was in the business-as-usual 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

50 

condition throughout the study. The treatment group was in the business-as-usual condition in the 

pre-study months and used MSRS or CARS in the study month. 

Exhibit 23. Overview of the Experimental Design

It is worth emphasizing that the treatment is intended to produce proxy measures of the actual

meals served. We expected the meal claims from providers in the treatment group to be more 

accurate or truthful than claims made by providers under the business-as-usual condition, 

assuming that providers in the treatment condition were aware of the study and thus more vigilant 

about claiming meals during the study month. However, the resulting report of meals served can 

still differ from the actual meals served, as our approach could not identify meals claimed but not 

served in some possible scenarios, for example, if a child brought lunch on a particular day. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

51

Random Assignment. To minimize selection bias, we randomly selected providers affiliated with 

the same sponsors and assigned them to the treatment and the control groups.22 We asked the 

selected ten sponsors to provide records for all active providers. Within each sponsor, we 

stratified by urbanicity and FDCH size (i.e., the number of children enrolled) and randomly 

selected providers within each stratum and assigned them to the treatment or the control group. 

The power analysis showed that, with each sponsor, we would need ten providers in the treatment 

group; we oversampled providers to make up for the ineligibles and refusals (i.e., providers who 

did not have access to mobile devices or the Internet, or selected providers who refused to 

participate in the study). We did not contact the control group providers and only collected their 

administrative records and meal claim data from the sponsors. 

Difference-in-Differences (DD) Modeling. DD modeling is conventionally used to establish 

causal effects of a program intervention. In this study, we use the DD model to test whether the 

data collection would yield an estimate that served as a proxy of the truthfulness of meal claims, 

for which it also is well suited. In this section, we present a basic DD model with ordinary least 

squares regression (Equation 1). Because some parents in participating day care homes gave their 

consent to participate in the study (by reporting children’s pick-up and drop-off times), but not all, 

we would not be able to aggregate claim numbers from the child level to the provider level. As a 

result, we simplified the model from a two-level nested structure to a single-level regression 

analysis. 

Exhibit 24 shows that, for a given meal type, we calculated: (1) the aggregate number of meals for 

each of the two groups in the pre-study and study months (ŷ); (2) the change over time between 

pre-study months and study month for each group (ΔΔŷ); and (3) the group difference in each 

group’s change over time (ΔΔŷa − ΔΔŷb or DD), which indicates the effect of our proposed approach 

in reducing improper claims for the treatment group, net of unmeasured time effects (changes in 

meal claims across the months due to factors other than the intervention). If the resulting DD 

estimate is substantially large and statistically significant, we would consider it as a valid measure 

of improper claims. 

Exhibit 24. DD Analysis

Pre-study Months Study Month

Between-month 

Difference

Treatment ŷat1 ŷat2 ŷat1 − ŷat2 =ΔΔŷa

Control ŷbt1 ŷbt2 ŷbt1 − ŷbt2 =ΔΔŷb

Group Difference Between Pre-study and Study Month ΔΔŷa − ΔΔŷb

22 We have considered and dismissed alternatives to group assignment, for example, selection of samples by 

matching the groups. Sample matching between the two groups is not desirable or feasible because sufficient 

information on provider background is essential for sample matching, yet only limited data on provider 

characteristics (e.g., location and meal claim numbers) were available. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

52

The relation among the quantities in Table 13 can be expressed as:

yi = b0 + b1Xi + b2 Ti+b3 XiTi+ ei, (1)

where Xi is an indicator for a provider’s status in the treatment or control group, Ti denotes time 

(pre-study months vs. study month), and b1 and b2 are estimates of, respectively, the treatment and 

the time effects on meal claims (i.e., (ŷat1 − ŷbt1) and ΔΔŷb). The parameter b3 is associated with the 

interaction term, XiTi,, which is the DD estimate, or ΔΔŷa − ΔΔŷb. The DD estimate in relation to 

other regression coefficients is shown in Exhibit 25. 

Exhibit 25. DD Estimates in Regression Modeling: A Simplified Explanation

Pre-study Months Study Month Difference

Treatment b0+b1 b0+b1+b2+b3 b2+b3

Control b0 b0+b2 b2

Difference-in-Differences Estimate b3

We used DD modeling to determine if the treatment effect in reducing overclaims is statistically 

significant, net of the time effects and other effects of covariates, if any. We then added provider 

variables available for both control and treatment groups into the equation as covariates in an 

attempt to increase model fit and estimate precision. Such covariates must be conceptually 

relevant to meal claims and empirically predictive of meal claims to be included. We used two 

such covariates: providers’ meal claiming method23 (e.g., electronic filing vs. paper filing) and the 

number of children enrolled. 

To test if the between-month discrepancy in meal claims was statistically significant between the 

control and the treatment group, we estimated an equation to test the treatment-time interaction 

effect b3.

• If b3 is positive and large in magnitude, it indicates that the reduction in meal claims from 

the pre-study month(s) to the study month is substantially greater for the treatment group 

than for the control group, controlling for the time effects. 

• If b3 is also statistically significant at p < .05 level, it suggests that the group difference in 

cross-month claim reduction is unlikely to occur by chance. 

Together, this statistical evidence would imply that the treatment effectively reduced improper

claims.

It is plausible to consider the number of claims of the treatment group in the study month as a 

proxy for the “truthful” measure of meals served. We then compare it with the claims the 

treatment group filed in the pre-study months to identify overclaims. We used a generic threshold 

of 0.35 effect size for the hypothesis test. 

We used the DD model to validate the data collection method, assuming that our intervention 

should help encourage accurate meal claiming and reduce overclaiming, thus resulting in fewer 

claims for the treatment group during the study month. The DD modeling controls for 

23 While substantial numbers of providers now file claims electronically, we learned that many still file claims in 

hardcopy. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

53 

confounding factors that may contribute to the declining (or whatever variation of) meal claims 

for the group in the study month. 

The focus of the DD modeling was to determine if the two groups’ difference in their betweenperiod differences in meal claims was substantial and statistically significant. The DD modeling 

statistically controls for the time effects (i.e., changes in meal claims across the months due to 

factors other than the intervention) and the two groups’ background differences (enrollment size 

and rural-urban location), producing strong evidence validating the feasibility study, relative to 

many other analytic approaches. If, as hypothesized, the treatment group shows a between-period 

reduction in meal claims that is larger than that for the control group, we may conclude that the 

feasibility study has indeed generated more truthful measures of meal service.24

5.2.1 Data and Descriptive Statistics 

We created the following meal claim measures for the modeling: 

• For providers in the treatment group, business-as-usual meal claim counts in the two prestudy months and the triangulated meal counts in the study month 

• For the control providers, all three months meal claims collected through business-as-usual 

• For both groups, rural-urban location and enrollment size collected from the administrative 

records

Exhibit 26 displays the numbers of providers whose monthly meal claims records were available 

for analysis. The dataset, produced after numerous iterations of cleaning, editing, imputation, and 

cross-source reconciliation, contains a total of 849 cases. A case is a provider-month record with 

meal claim information. Of the 849 cases, 666 are of the control group and 183 the treatment 

group.25 In the SAS analytic dataset, records were sorted by provider ID so that each provider 

(222 of the control and 61 of the treatment) had three-month records with varying meal claims 

and the same background information. We removed cases that did not have the full three-month 

meal counts. 

Exhibit 26. Description of Provider Sample (Providers with 3-Month Meal Claims) 

Group Study Month Pre-study 

Month 2

Pre-study 

Month 1 Total 

Control 222 222 222 666

(78.45%)

Treatment 61 61 61 183

(21.55%)

Total 283

(33.33%)

283

(33.33%)

283

(33.33%)

849

(100.00%)

Data source: Modifiedc_claims_20180910

24 The “truthfulness” of meal claims is extremely difficult, if not impossible, to measure. We obtained relatively 

truthful measurement based on reasoned comparisons with the DD analysis.

25 We kept all providers that were not randomly selected to the treatment group in the control group instead of 

randomly selecting a subsample to these providers for the control condition. This leads to an unbalanced sample 

with a study-control ratio slightly higher than 1:4.Research shows the precision of estimation erodes when the ratio 

is lower than 1:4 (Bloom, 2006).
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

54 

To ensure sufficient power of analysis, we modified the treatment and control group identification 

by including in the control group some providers that were sampled in the treatment and failed to 

participate, but that had the three-month meal claims available. 

The records in the source dataset for modeling have detailed claim counts for each of the six meal 

types at the two tiers, with different reimbursement costs. As meals were in three cost categories, 

breakfast, lunch and supper, and snacks, we thus combined the six meal types into three. Given 

the need to analyze meal data separately by tiers, we tested models of meal counts separately by 

tiers, and found too few tier 2 meal counts to obtain meaningful estimates (approximately 10–20 

percent of meal counts were in tier 2). Therefore, we consolidated meal counts of breakfast, lunch 

and supper, and snacks without regard to tier. Exhibit 27 describes the meal claim patterns of the 

two groups across the three months, showing the mean and 95 percent confidence interval for 

each group in each month. 

We combined lunch and supper when reporting results based on the difference-in-differences 

models. This is mostly because of the low counts of supper, which will not allow us to build a 

robust model to produce reliable results for supper alone. Since lunch and supper have the same 

reimbursement rate, we combined the two meal types when running the models so we can use the 

model coefficients to calculate IP.

Exhibit 27. Descriptive Statistics: Meal Claims, by Group Status, Study Time Period, and Month

Confidence Level for Mean

Group Period Month Meal Type Mean Lower 95% Upper 95%

Control, n=666 Pre-study Pre-month 2 Breakfast 108.7 98.3 119

All snacks 166.5 152.7 180.4

Lunch and 

Supper 164.9 149.5 180.4 

Pre-month 1 Breakfast 99.1 89.6 108.7

All snacks 154.3 140.6 167.9

Lunch and 

Supper 150.8 136.4 165.2 

Study Study month Breakfast 99.7 90.3 109.1

All snacks 151 137.7 164.4

Lunch and 

Supper 153.2 139 167.4 

Treatment, n =61 Pre-study Pre-month 2 Breakfast 119 103.8 134.3

All snacks 154.3 136.4 172.2

Lunch and 

Supper 143.2 120 166.4 

Pre-month 1 Breakfast 106.3 92.2 120.4

All snacks 140.1 122.7 157.6

Lunch and 

Supper 129 107.3 150.7 

Study Study month Breakfast 89 75.9 102.1

All snacks 112.2 96.3 128.1

Lunch and 

Supper 105.9 86.8 124.9 

Data source: Modifiedc_claims_20180910
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

55 

Furthermore, as depicted in Exhibit 28, we calculated descriptive statistics to show the group 

difference in between-period differences in average meal claims (without statistical control). It 

seems clear that (1) both groups’ average meal claims across all three meal types declined from 

the pre-study months to the study month, and (2) the treatment group’s between-period declines 

were greater than the control group’s declines, again largely consistently for all three meal types. 

Exhibit 28. Descriptive Statistics: Group Differences in Meal Claim Change between PreStudy and Study Months

Mean Meal Claims

Meal Type Study Period Treatment 

Group

Control 

Group

Group Difference 

Over Time

Breakfast Study month 89.0 99.7

Pre-study months 112.7 103.9

Between-period change −23.7 −4.2 −19.5

Lunch and Supper Study month 105.9 153.2

Pre-study months 136.1 157.9

Between-period change −30.2 −4.7 −25.6

Snacks Study month 112.2 151.0

Pre-study months 147.2 160.4

Between-period change −35.0 −9.4 −25.6

Data source: Modifiedc_claims_20180910

5.2.2 Difference-in-Differences Modeling Results 

The DD equation is written as:

yi = b0 + b1Xi + b2 Ti + b3 XiTi + ei, (1)

Where yi is the expected meal claim count for provider i; 

b0 denotes intercept of the equation; 

b1 is group effect and Xi is group indicator; 

b2 is time effect and Ti indicates the pre-study (including 2 months) vs. study periods 

(1 month); 

b3 is the focal estimate in this study, representing the group difference in between-period 

difference in meal claims (DD); 

XiTi denotes the cross-product or interaction of the group and time; and 

ei is the error term, assuming normally distributed with a mean of zero. 

Three equations, respectively for meal counts of breakfast, lunch and supper, and snacks (on both 

tiers), were estimated using a SAS mixed procedure. Note that we do not show the two covariates, 

rural-urban location and enrollment size, from the above equation, although they were specified in 

the model. The point of the omission is to help readers focus on the DD estimation; the reason for 

this is that while the two background variables were statistically controlled for, their effects on 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

56 

meal claims are not major features of interest to the study. We also do not present the coefficient 

estimates of the two covariates in the following tables. 

The three models fit the data reasonably well, with chi square tests that are large and statistically 

significant (Exhibit 29), implying that, with specified effects, the models fit the data better than 

the null model with intercept only. Model diagnostics with residual analysis and predicted scores 

plotting also suggest an acceptable model fit. 

With the model for breakfast, the intercept, b0, estimated 83.57, indicates the average claims by 

the treatment group in the study month (Exhibit 44). The group effect, b1, estimated 11.78, not 

statistically significant, gives no evidence that breakfast claims by the two groups were different, 

net of time effects and other factors (location and enrollment size). The time effect, b2, estimated 

23.64 with p < 0.0001, indicates that, on average, the pre-study period’s meal claims were almost 

24 times larger than the study period’s, again controlling for the effects of group difference and 

location and enrollment size. Estimates from models of lunch/supper and snacks followed similar 

patterns. 

The DD effect, b3, was considerably large and statistically significant, as estimated from the three 

models (Exhibit 29). For breakfast, the coefficient was estimated −19.4 and statistically 

significant at p < 0.0001 level. It suggests that while both groups experienced decreased breakfast 

claims from the pre-study period to the study period, the treatment group’s decrease was 19.4 on 

average more than the control group’s decrease, net of time effects and the effects of enrollment 

and rural-urban locations. 

Exhibit 29. Difference-in-Differences Regression Model of Meals Claimable: Coefficients and 

Model Fit Indices

Meal type Parameter Estimate Standard

Error

Breakfast Intercept, b0 83.57* 8.35

Group effect, b1 11.78 9.06

Time effect, b2 23.64* 2.87

Group X Time effect, b3 −19.40* 3.24

Null model likelihood test Chi square (df=1) 1,086.99*

Lunch and Supper Intercept, b0 106.35* 12.60

Group effect, b1 45.90† 13.67

Time effect, b2 30.24* 4.13

Group X Time effect, b3 −25.61* 4.66

Null model likelihood test Chi square (df=1) 1,138.13*

Snacks Intercept, b0 109.60* 10.85

Group effect, b1 38.39† 11.77

Time effect, b2 35.01* 3.63

Group X Time effect, b3 −25.64* 4.10

Null model likelihood test Chi square (df=1) 1,116.51

N=849 (provider-month records); *p < 0.0001, †p < 0.001
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

57 

With models of lunch/supper and snack, the DD effects were also substantial and significant, 

respectively estimated −25.61 and −25.64, both at p < 0.0001 level. The consistent findings allow 

us to say that the intervention executed in the study effectively reduced the treatment group’s 

meal claims in the study month, and hence the resulting measurements were likely close to the 

reality of meal services. 

In short, we consider that the feasibility study was validated by the analysis and that we can

proceed to estimate the improper claims and payment based on these findings. 

5.3 Estimation of Improper Payments

The triangulation yielded a datafile with meals that, to our best knowledge, providers in the 

treatment group could be reimbursed for in the study month. Along with meal claim records in the 

pre-study month, the triangulation file allows us to compare the meal reporting over time and 

identify changes in reporting patterns that may attribute to reporting errors. 

We used two approaches to estimate the improper payment, one with observed data and the other 

with inferential estimates based on the DD models discussed previously. In this section, we 

describe the application of these two approaches and report the estimates of improper payments. 

While the DD models yield higher improper payment rates, both methods suggest an error rate 

that requires a program-wide inspection, as the error rate is greater than 10%. 

5.3.1 Estimation of Improper Payments Using Triangulated Data 

The observed data error estimation began with the child attendance data submitted by providers. 

The triangulation process flagged numerous child attendance meals as improper. We separated the 

improper and net meals into tiers to properly apply reimbursement rates. The reimbursement rates 

are shown in Appendix 3. 

The final results determined that 4,834.5 meals were improper for the 61 providers participating 

throughout the study month, as shown in Exhibit 30. The dollar amount of overpayments totals 

$6,316 at a weighted average cost per meal of $1.31. Lunch/supper accounts for over half the 

dollar amount of the errors. 

Exhibit 30. Observed Data Calculation of Meal/Snack Overclaims and Resulting Dollar 

Values of Overpayment 

Meal Type Total Improper

Claims

Cost per 

Meal1

Dollar Value of 

Overpayments

Breakfast 1,309.3 $1.23 $1,604.15 

Lunch and Supper 1,436.2 $2.30 $3,300.19

Snack 2,088.9 $0.68 $1,412.30

Total 4,834.5 $1.31 $6,316.64

1 Weighted Average of Tier 1 and Tier 2 Rates

Source: Triangulation, n=61

On a percentage basis, we estimated that 20.6% of the meals claimed were in error, as shown in 

Exhibit 31. Errors were evenly distributed across meal types, with snacks accounting for a slightly 

higher percentage than breakfast and lunch/supper. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

58 

Exhibit 31. Counts and Percentage of Overclaims Based on Child Attendance Counts

Meal Type Total Improper

Claims

Child 

Attendance Percentage

Breakfast 1,309.3 6,747 19.4%

Lunch and Supper 1,436.2 7,850 18.3%

Snack 2,088.9 8,906 23.5% 

Total 4,834.5 23,503 20.6%

Source: Triangulation, n=61

As Exhibit 32 shows, we estimated an overall error rate of 20 percent of payments. While the 

highest dollar amount was for lunch/supper, this is the lowest percentage because of the higher

dollar amount of submitted claims for these meals. We estimate that providers should have 

submitted reimbursement requests for $31,571.72 based on their child attendance records. 

Sponsors approved $31,953.44 in payments to these providers, so our estimate is close to what 

was actually paid. Please note that we believe providers are entitled to $25,255.08 in 

reimbursements ($31,571.72, the amount providers should have submitted reimbursement

requests for, minus $6,316.64. the amount of overpayments) after applying the results from the 

triangulation process. A rough breakdown of the errors is: five percent administrative records; 

five percent from parental reporting through CARS; and 10 percent from contemporaneous 

provider reporting of meal service times through MSRS (See Appendix 5 for information 

regarding the estimates of meals reported but not served). We also rigorously applied rules that 

are in place but might not have been completely enforced, primarily around valid meal service 

times, because of a lack of data or perceived need for flexibility.

Exhibit 32. Estimated Overpayments Based on Child Attendance Records

Meal Type Dollar Value of 

Overpayments

Estimated Payments Based 

on Child Attendance Percentage

Breakfast $1,604.15 $7,921.61 20.3% 

Lunch and Supper $3,300.19 $18,196.46 18.1%

Snack $1,412.30 $5,453.65 25.9% 

Total $6,316.64 $31,571.72 20.0%

Source: Triangulation, n=61

The advantage of this approach is that we could directly link errors with their sources because we 

can track the errors directly from observed data in the triangulation process. Exhibit 33 shows the 

cause of errors by meal type. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

59 

Exhibit 33. Percentage of Improper Meals Based on CARS and MSRS Reporting, by Meal

Type and Error Type 

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%

Breakfast

Lunch

Supper

AM Snack

PM Snack

Evening Snack

Total

Child Absence Provider Closing Meal Served before Drop-Off Meal Served After Pick-Up

Triangulating MSRS data with administrative data (Total MSRS) generates most of the errors. 

Triangulation between CARS and MSRS enabled us to identify four types of errors: (1) a meal 

was claimed when a child was absent (Child absent as shown in Exhibit 49),(2) a meal was 

claimed when an FDCH was closed (Provider closed), (3) a meal was reported33s served before 

the child was dropped off, and (4) a meal was reported as served after the child was picked up. A 

meal served before a child’s drop-off is a common reason for errors in claims for breakfast and 

AM snack. The results based on the observed data, however, are not generalizable because the 

sample for this feasibility study is small, not nationally representative, and tends to be affected by 

factors unrelated to the study, such as the hurricane that hit Austin, Texas in 2017. 

5.3.2 Error Estimates Based on Difference-in-Differences Model

We calculated the improper claims based on DD regression model inferential statistics (with a 95 

percent confidence interval) to yield a more reliable measure of the error. The advantages of 

model-generated inferential statistics include (1) representing the net estimates of improper claims 

generated by the intervention (controlling for the confounding factors that may have contributed 

to the meal claim variation); and (2) offering the range of the estimated overclaims in addition to 

a point estimate. However, with a small and unrepresentative study sample, the statistics would 

not be generalizable to any population. 

We focused on overclaims in this analysis. The calculation entailed inferring the total overclaims 

from the model-generated mean and standard error of the DD effect. The inferred statistics 

included a point estimate of the total overclaims and its range derived from the 95 percent

confidence interval of the mean. The range suggests the low and high range of the total 

overclaims that may occur with a 95 percent chance in repeated sampling. 

For each meal type, we (1) multiplied the DD estimate, b3 , by the treatment group size (61 

providers), to obtain the total overclaims point estimate, (2) converted the model-generated 

standard error into the 95 percent interval of the DD effect, and (3) multiplied the resulting low 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

60

and high values with the treatment group size to obtain the low and high total overclaims. The 

resulting overclaim estimates are presented in Exhibit 34.

Exhibit 34. Estimated Overclaims: Mean and Total Meal Counts for the Treatment Group in 

the Study Month (n=61)

Meal 

Type

Average

Overclaim

95% 

Confidence 

Level–

Higher 

Bound

95% 

Confidence 

Level–

Lower 

Bound

Total 

Overclaim

Total 

Overclaim 

Higher

Bound

Total 

Overclaim 

Lower 

Bound

Breakfast 19.4 25.8 13.0 1,183.4 1,572.1 794.9

Lunch and 

Supper

25.61 34.8 16.4 1,562.21 2,121.9 1,002.9

Snacks 25.64 33.7 17.6 1,564.04 2,055.8 1,072.8

Next, we calculated dollar amounts of the overclaims by multiplying the total overclaims with the 

meal costs. For each meal type, we developed an average cost that was weighted by the proportion of 

the two tiers claimed. Applying these tier-weighted meal costs allowed relatively accurate calculation 

of the total dollar amounts due to overclaims. The results are shown in Exhibit 35.

Exhibit 35. Estimated Meal Overpayment: Mean and Total Dollar Amounts the Treatment 

Group in the Study Month (n=61)

Meal Type

Tier-weighted 

Average Meal 

Cost

Total 

Overpayment

Total 

Overpayment 

Higher Bound

Total 

Overpayment 

Lower Bound

Breakfast $1.17 $1,384.58 $1,839.37 $930.08

Lunch and Supper $2.32 $3,624.33 $4,922.74 $2,326.76

Snacks $0.62 $969.71 $1,274.61 $665.16

Finally, we computed the overclaim rates and overpayment rates for three types (Exhibits 36 and 

37). The denominators were total meal served counts and total meal expenditure extracted from 

the administrative files of the sampled providers. Again, we presented the high and low range of 

both overclaim and overpayment rates. Note that the overclaim rates and the overpayment rates 

are almost identical, as a function of the meal costs. 

Exhibit 36. Overclaim Rates: Overclaimed Meal Counts as Percentages of the Total Number 

of Meals Served (n=61 providers)

Meal Type Overclaim

Total Meals 

Served Based on 

Child Attendance

Overclaim 

Rate

Overclaim 

Rate –

Higher 

Bound

Overclaim 

Rate –

Lower 

Bound

Breakfast 1,183.4 6,747 17.5% 23.3% 11.8%

Lunch and Supper 1,562.2 7,850 19.9% 27.0% 12.8%

Snacks 1,564.0 8,906 17.6% 23.1% 12.0%

Total 4,309.65 23,503 18.3% 24.5% 12.2%
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

61 

Exhibit 37. Overpayment Rates: Overpayment Dollar Amount as Percentage of the Total Meal 

Expenditure (n=61 providers) 

Meal Type Overpayment

Total Meal 

Expenditure

Overpayment

Rate

Overpayment

Rate – Higher 

Bound

Overpayment

Rate – Lower 

Bound

Breakfast $1,384.58 $7,945 17.4% 23.2% 11.7%

Lunch and Supper $3,624.33 $18,233 19.9% 27.0% 12.8%

Snacks $969.70 $5,499 17.6% 23.2% 12.1%

Total $5,978.61 $31,677 18.9% 25.4% 12.4%

In summary, estimates based on both observed data and DD modeling suggest potential error rates 

for breakfast, snacks, or lunch and supper combined are greater than 10 percent, a threshold set in 

IPERA as a compliance measure. The observed data show error rates that are on the high end of 

the error range predicted by the inferential DD model for all four meal types. 

As discussed earlier, we used two approaches to estimate errors to compare results as there is no 

earlier work to chart our course. It helps establish the validity of the approach and the reliability 

of the results when both approaches produced similar error rate estimates. The specific error rates 

are not as informative as the directions these estimates suggest, as the purpose of this feasibility is 

to test the viability of data collection methods and, therefore, the sample is not nationally 

representative. These initial estimates are particularly useful for guiding conversations about 

program compliance within the CACFP community to improve program delivery and data quality 

in the future. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

62

6. Recommendations

The Improper Payments Information Act of 2002 and the Improper Payments Elimination and 

Recovery Improvement Act of 2012 requires agencies to estimate the annual amount of improper 

payments in programs and report on actions to reduce improper payments that exceed $10 

million. The first step is to find a method to measure improper payments that can be implemented 

for a limited period of time for measurement purposes without necessarily changing the behavior 

of making regular claims for reimbursement. This can be challenging in CACFP FDCHs, which, 

by their nature, are the private homes of providers and do not readily permit intervention or 

observation without disruption or invasion of privacy.

This feasibility study represents the third attempt by FNS to design and test a data collection 

method that enables FNS to estimate IP due to meals claimed improperly by FDCHs. Specifically, 

the study focuses on accurately measuring meals that are claimed but not served. If valid, the data 

collection method needs to be viable for potential national-level implementation in the future.

We designed and developed digital instruments to collect meal-serving and child attendance 

information from FDCH providers and parents without invasive intrusion into the routine 

practices at FDCHs. We used these data, in combination with secondary data already collected 

from sponsors, to estimate IP through an approach that yielded accurate records of meals served 

by providers and the children to which they were served for any given meal and day. We 

collected and analyzed the data in a quasi-experimental design comparing meal claims in 

treatment FDHCs with a control group to yield an estimate of IP. 

Here, we summarize the practices and lessons learned during the feasibility study and present 

recommendations that could be used in a large-scale national study to measure improper meal 

claims in CACFP FDCHs.

The major takeaway is that the data collection methods designed for measuring IP by providers 

and parents are effective and feasible for large-scale implementation in State or nationally 

representative samples of sponsors and providers. In summary, we found that:

• Parents adapted readily to the use of the CARS text messages to report child drop-off and 

pick-up times. 

– Incentives are necessary to obtain their willingness to use the system.

– Reminders helped parents remember to report their child’s attendance. 

– The need for technical assistance was limited and occurred only in the initial use of the 

system.

• Providers were able to use MSRS to report meal-serving times and chose the smartphone 

version over the web version. 

– Incentives are essential to encourage the use of MSRS in addition to whatever existing 

reporting systems they are already using.

– MSRS was effective in encouraging the timely reporting of meals served.

– Ongoing provision of technical assistance is necessary.

In addition, it is necessary for FNS to decide how to consistently apply local and State policies 

and their impact on whether a claim should be considered in error or not. In the course of the 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

63

feasibility study, we found that one of the two States had more stringent requirements for the 

maximum number of children in care for a licensed FDCH than the other. (In some States—none 

in this feasibility study—local jurisdictions such as cities or counties may have licensing 

requirements more stringent than the State.) After consulting with FNS, we decided to treat the 

small number of claims affected by this provision as improper for the feasibility study. 

Based on the experiences of the feasibility study, we make the following five recommendations 

on the process of implementation if FNS decides to conduct a nationally representative study to 

measure IP in FDCH meal claims.

Required v. voluntary participation: FNS decided that participation in the feasibility study 

would be required for sponsors and providers and stated so in recruitment materials. However, in 

the absence of stated consequences for lack of participation, the requirement had little impact and 

was treated as if it were voluntary. This may be, in part, because this was a feasibility study. In an 

actual large-scale or national study with a nationally representative sample, FNS should consider 

how it would enforce provisions for required participation. We recommend sponsors 

communicate this requirement to participate in FNS studies to selected providers. A contractor 

can provide technical assistance to sponsors to alleviate burden on sponsor’s staff. 

Incentives: While stating that participation is important as a message to prospective participants, 

behavioral science confirms that financial incentives are highly effective in gaining participation. 

The digital distribution of incentives in the form of an electronic gift card was effective and 

reliable; the contractor used dual-factor authentication to ensure secure distribution. 

• There was no provision for incentives to sponsors in the feasibility study. FNS should 

strongly consider implementing incentives for sponsor participation, especially if sponsors 

are asked to help in the recruitment of providers. Many smaller sponsors operate with 

limited staff and may have challenges taking on additional responsibilities, even for a 

limited time. Furthermore, incentives would encourage sponsors to help in recruiting 

providers.

• The feasibility study provided modest incentives for providers—$60 for one month, which 

was reduced from $100 initially proposed by the contractor. The incentive amount was, at 

best, only modestly helpful in engaging provider participation as only 71.1 percent of 

providers recruited agreed to participate. A few providers even stated that they would rather 

quit the program than participate in the feasibility study. In the absence of enforceable 

consequences for their participation, no one needed to quit; there was nothing the contractor 

could do to gain the participation of providers who did not want to participate, and there

were no consequences to providers for failure to participate.

• The feasibility study provided a $25 incentive for parents. The amount was modest and 

was probably not sufficient to gain the participation of some parents. The incentive had been 

reduced from $50, initially proposed by the contractor.

Onboarding: Once the contractor selects the geographic areas for an IP study and identifies a 

sample of providers from sponsor-provided lists, it is desirable for sponsors, along with the 

contractor, to be engaged in the onboarding of providers. Sponsors have an existing relationship 

with providers that can encourage providers to participate in the study. Similarly, providers can be 

engaged to facilitate the onboarding of parents along with the contractor, simply by asking parents 

to help them by participating in the study. 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

64

Time to prepare for study implementation: Providers and parents need time to access and 

familiarize themselves with using MSRS and CARS sufficiently in advance of the actual study 

period to begin using them. We recommend that at least one week for the majority of the 

providers. Consideration should be given for more than one week of advance time for providers to 

communicate with parents about participation.

Technical assistance: It is crucial to provide robust technical assistance during the feasibility 

study by offering technical assistance via telephone, email, and social media. The contractor in 

the feasibility study used all of these modes to communicate with providers and relied on text and 

telephone for parents. Parents did not need technical assistance after their initial use of CARS, but 

it should be available in the event of a system problem.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

65 

Appendices

Appendix 1. Reported Child Attendance in the Study Month Collected via 

CARS (Child n=215, days=2126) and Imputed Child Attendance for Days 

Not Reported

Reported Imputed1

Reported and 

imputed 

combined

Drop-offs, n (%) 1,935 (42.9%) 801 (17.7%) 2,736 (60.6%)

Pick-ups, n (%) 1,697 (37.6%) 937 (20.8%) 2,634 (58.3%)

Drop-off and pick-up, n (%) 1,632 (36.1%) 942 (20.9%) 2,574 (57.0%)

2nd Drop-offs, n (%)2 13 (0.3%) 0 (0.0%) 13 (0.3%) 

2nd Pick-ups, n (%) 9 (0.2%) 4 (0.1%) 13 (0.3%) 

2nd Drop-off and pick-up, n (%) 9 (0.2%) 4 (0.1%) 13 (0.3%) 

Absences, n (%) 259 (5.7%) 259 (5.7%)

Closures, n (%) 252 (5.6%) 252 (5.6%)

Any action for the day, n (%)3 3,282 (72.7%) 

Days with complete valid data, n (%)4 3,071 (68.0%) 

1 Drop-off and pick-up imputation applied only to children with five or more reported times for each. For both, 

only data for non-holiday weekdays included to compute the mean drop-off and pick-up times, respectively. Where 

the drop-off or pick-up time is missing, the child is not reported as absent, the provider is not reported as closed by 

the parent, and the day is a non-holiday weekday, the mean drop-off or pick-up for the child is used. 2 Children may be dropped off and picked up multiple times per day. In the data, 13 children-days had a second 

drop-off and pick-up times. 3 Action occurs if any drop-offs, pick-ups, absences, or closures included for the day, either reported or imputed.

4 Data is complete and if drop-off and pick-up or absence or closure included for the day, either reported or imputed.

26 When calculating the mean times, we included only children at the day care home with five or more drop-offs or 

pick-ups. N shows the total number of drop-offs and pick-ups these children had during the study month.
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

66 

Appendix 2. Percentage of Providers with Claim Errors, by Meal Type

32

37

20

41

33

0

19

20

13

9

19

0

39

10

0

18

5

0 0

2

20

0

10

25

0

5

10

15

20

25

30

35

40

45

Breakfast Lunch Supper AM snack PM snack Evening snack

Percentage of Providers

Meal Type

Child absent Provider closed Meal served before drop-off Meal served after pick-up
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

67 

Appendix 3. Meal Reimbursement Rates

Meal Type Tier 1 Tier 2

Breakfast $1.31 $0.48

Lunch and Supper $2.46 $1.48

Snack $0.73 $0.20
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

68 

Appendix 4. Descriptive Statistics: Average Number of Attendance Days, 

Eligible Meals/Snacks, and Percentages of Children by Tiering Status and 

Enrollment Expiration

Average attendance days of week

N Mean Std. Deviation Min Max

644 5 0.5 1 7 

Number and percentage of children with specific attendance days of week

N of attendance 

days in a week N of children Percent CumulativeFrequency

Cumulative

Percent

Blank 2 0.31 2 0.31 

1 1 0.15 3 0.46 

2 1 0.15 4 0.62

3 13 2.01 17 2.63 

4 7 1.08 24 3.72 

5 583 90.25 607 93.96 

6 20 3.10 627 97.06 

7 19 2.94 646 100

Number and percentage of children eligible for breakfast

Eligible for 

breakfast Frequency Percent CumulativeFrequency

Cumulative

Percent

No 18 2.79 18 2.79

Yes 576 89.16 594 91.95 

Blank 52 8.05 646 100

Number and percentage of children eligible for lunch and/or supper

Eligible for lunch 

and supper Frequency Percent CumulativeFrequency

Cumulative

Percent

None 3 0.46 3 0.46 

Either lunch or supper 424 65.63 427 66.09 

Both 171 26.47 598 92.56 

Blank 48 7.43 646 100 
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

69 

Number and percentage of children eligible for snacks

Eligible for 

snacks Frequency Percent CumulativeFrequency

Cumulative

Percent

One 315 48.76 315 48.76

Two 252 39.01 567 87.77 

Three 33 5.11 600 92.88 

Blank 46 7.12 646 100 

Number and percentage of children's tier status

Child Tier Frequency Percent Cumulative 

Frequency

Cumulative 

Percent

1 11 1.70 11 1.70 

2 595 92.11 606 93.81

Blank 40 6.19 646 100 

Number and percentage of children's enrollment expiration 

Expiration Frequency Percent Cumulative

Frequency

Cumulative

Percent

Not expired 598 92.57 598 92.57 

Blank 48 7.43 646 100 

Note: Enrollment expiration is identified as children whose enrollment expire before 10/01/2017.

Source: Child information collected from sponsors for treatment group only (Provider N=63, child N =646)
Feasibility Study of Meal Claims at CACFP Family Day Care Homes

70 

Appendix 5. Meals Claimed but Not Served

# of Meals That Are Reported but Not Served # of Meals Claimed and 

Reimbursed

CARS MSRS

Meal 

Type

Child 

Absent

FDCH 

Closed

Meal 

Served 

Before 

DropOff

Meal 

Served 

After 

PickUpChild 

Absent

FDCH 

Closed

Meal 

Reported 

as "Not 

Served"

Total 

Meals 

Not 

Served

Child 

Attendance 

Served 

Meals

Admin 

Records 

Errors 

Corrected 

by 

Sponsors

Estimated 

Net Meals 

Reimbursed

% Not 

Served but 

Reimbursed

Breakfast 59 42 348 0 64 100 20 633 6,747 263 6,484 9.8%

Lunch or

Supper 74 44 115 25 70 107 12 447 7,850 195 7,655 5.8% 

Snack 78 45 152 75 84 103 18 555 8,906 864 8,042 6.9%

Total 211 131 615 100 218 310 50 1635 23,503 1,322 22,181 7.4%