DOI: 10.3886/icpsr38949.v1
title: Assessing the Implementation and Cost of High Quality Early Care and Education Project (ICHQ) Field Test, United States, 2021
title: Version 1

description: Mathematica conducted a field test for the Assessing the Implementation and Cost of High Quality Early Care and Education (ICHQ) Project. The goals of the field test were to:

Refine implementation measures to further improve and establish their psychometric properties
Refine cost measures to identify costs associated with each key function and across ages of children
Test the associations between implementation, cost, and quality measures.
The research team recruited a purposive sample of 80 centers in Arizona, Arkansas, Colorado, and Pennsylvania to participate in the field test and conducted data collection from April to December 2021. Implementation interviews (DS2) were conducted by phone with either one or a few key staff who knew the most about the center's operations and educational program. Cost data (DS5) were collected in an electronic spreadsheet by staff who knew the most about each center's finances. A web-based time-use survey collected information about how all teaching staff and select center administrators spend their time in activities related to providing early care and education services (DS3). The research team also conducted a center engagement interview to gather information about center characteristics (DS1), and an additional survey of all teaching staff (the same respondents identified for the time-use survey) to assess their perceptions of center practices and the work environment (DS4).
description: The purpose of this study included refining the implementation measures to improve and establish their psychometric properties, refine the cost measures to identify costs associated with each key function and across ages of children, and test the associations between implementation, cost, and quality measures that are reflected in the conceptual framework.
description: This study included five data collection activities:

<b>Center engagement interview</b>: After a center agreed to participate in the project, data collectors conducted the center engagement interview with a staff person who oversaw and managed center activities (often the center director) to gather information about center characteristics. As part of the 30-minute phone interview, information was collected about the center's quality rating and improvement system (QRIS) rating and accreditation status, which the research team used as proxies for quality so they could begin to explore associations between implementation, cost, and quality.
<b>Implementation interview</b>: Implementation data collectors conducted the implementation interview by phone with either one or a few key staff who knew the most about the center's operations and educational program. The implementation data collectors did not ask center staff to prepare for the interviews or provide any materials in advance, although they did suggest that respondents have a few materials readily available, such as a list of center staff, an organizational chart, information on the number of children in the center with special needs, and information on funding sources. Implementation data collectors used a structured interview protocol in a fillable PDF that covered the five key functions in the ICHQ conceptual model. Implementation interviews were completed in two or three sessions to avoid respondent fatigue.
Each section of the interview included questions about (1) the structural features and adopted practices a center has in place to carry out each of the five key functions of its operations, and (2) the presence of specific activities (for example, approach to selecting and hiring staff, use of evidence-based tools, and type and frequency of training and coaching) that are needed to ensure intended features and practices are carried out effectively to achieve positive outcomes. The interviewer also collected information about the impacts of the COVID-19 pandemic on operations and services. The reference period for implementation data collection was August 2020 to March 2021. The research team selected this period to capture services provided during a typical program year, because some centers adjust their programming in the summer months.
<b>Time use survey</b>: The research team collected information about how a subset of center administrators and teaching staff spend their time in activities related to providing child care and early education services through a time use survey. The research team used this information to produce cost estimates by function by allocating salaries and fringe costs to the five key functions. The research team included teaching staff who provided direct instruction or care to children from birth through age 5 (not yet in kindergarten) in the classroom. This encompassed lead and head teachers and co-teachers and assistant teachers, but not substitutes or "floaters." The research team also included the administrators who had oversight of the center's operations and the educational program. After the research team had a list of all the staff whom the research team wanted to respond to the survey, data collectors obtained their contact information. All potential survey respondents received log-in information for the web survey. Administrators only received the 15-minute time use survey, whereas teaching staff received both the time use survey and a 30-minute teaching staff survey--the Supportive Environmental Quality Underlying Adult Learning (SEQUAL) survey. Respondents received a gift card as a token of appreciation for completing a survey.
<b>Teaching staff survey (SEQUAL)</b>: The research team conducted an additional 30-minute, web-based survey of teaching staff (the same respondents identified for the time use survey) to assess their perceptions of center practices and the work environment through an external, validated teaching staff survey--the Supportive Environmental Quality Underlying Adult Learning survey (SEQUAL; Whitebook et al. 2016).
<b>Cost</b>: Cost data was collected remotely with the staff member (often the center director) who knew the most about the center's finances. Data collection began with a 30-minute orientation call. The cost data collector went over the cost workbook with the respondent(s), discussed the file sharing process, and answered any questions the respondent(s) had about the workbook and process for providing cost information. During this call, the data collector scheduled a short follow-up call before the workbook was due to answer questions and review the workbook. Respondents then filled in the workbook on their own within two weeks of the orientation call and sent the completed workbook back to the cost data collector.
The workbook included instructions on how to complete it and space for responses or drop-down lists with prespecified options, depending on the question. The workbook had a series of tabs to enter costs for different categories of resources a center uses (for example, salaries and fringe benefits, facilities, equipment, and contracted services). Drop-down lists embedded throughout the workbook allowed respondents to select the key function associated with specific costs. Respondents also entered information about the number of children served in each age group and the number of hours of care provided during the specified reference period. The research team also collected costs of specific resources centers needed to operate and serve children during the COVID-19 pandemic and asked open-ended questions to capture how resource use changed in response to the pandemic. The reference period for cost data collection was planned to be the most recently completed calendar year (2020) or fiscal year; however, again because of the pandemic, the research team asked centers to select a recent three-month period when the center was open and serving some or all children in person.
The cost team members used a common protocol to review the workbooks and sent follow-up questions. The cost team members worked with respondents by telephone and email to address follow-up questions. They prioritized information on resource categories likely to make up the largest proportion of costs, such as salaries, fringe benefits, and facilities.
description: The center characteristics data (DS1) includes data from the center engagement interview: item-level variables and constructed variables about the size of the center (licensed capacity), ages of children served in the center, profit status, affiliation with larger organization or chain, funding mix, QRIS rating level, and accreditation status. The file has one record for each center.
The implementation data (DS2) includes data from the implementation interview, both item-level variables and constructed variables. The file also contains the scores for each of the five implementation measures (one for each center function). The file has one record for each center.
The time use survey data, staff level (DS3) contains data from the time use survey. It includes item-level variables for each staff respondent in a center to gather information on staff time use. There is one record for each staff member that completed the time use survey. Center IDs are in the file to identify staff in the same center and facilitate merging the file with center-level data.
The time use and teaching staff survey (SEQUAL) data (DS4), center level contains data from the time use survey and the teaching staff survey (SEQUAL). It includes constructed variables that aggregate time use by job category (such as administrator or teaching staff). The file also contains center-level average scores for each of the four domains of the teaching staff survey (SEQUAL). The file has one record for each center whose staff completed a time use or teaching staff survey (SEQUAL).
The cost data (DS5) captures data from the cost workbook: some item-level variables and some constructed variables, including child care hours, and costs by resource category (such as salaries, facilities, etc). The file also includes the estimates for each of the cost measures: (1) total cost per child care hour, (2) total cost per child care hour by the age of children served (infants, toddlers, and preschool-age children); (3) cost per child care hour by resource category, (4) cost allocation by resource category, (5) cost per child care hour for each of the five center functions, and (6) cost allocation for each of the five center functions. This file has one record for each center that completed a cost workbook.
description: Response Rates: The ICHQ field test data collection included the 80 centers who were eligible and recruited for the project. Some of these centers had participated in Phase 1 of the ICHQ data collection. However, not all 80 centers completed all data collection tools. As data collection extended into fall 2021, some centers encountered challenges in completing data collection tools due to ongoing service disruptions related to the COVID-19 pandemic and staff turnover.

<b>Center engagement interview</b>: 80 centers completed this phase.
<b>Implementation interview</b>: 80 centers completed this phase.
<b>Time use survey</b>: 581 staff from 57 centers completed this phase.
<b>Teaching staff survey (SEQUAL)</b>: 430 teaching staff from 57 centers completed this phase.
<b>Cost workbook</b>: 51 centers completed this phase.
description: The research team chose four states (Arizona, Arkansas, Colorado, and Pennsylvania) to select centers for the field test. The research team limited the number of states to include more centers from each state and ensure sufficient variation on key center characteristics within each state. Each of the four states met the following three criteria:

<b>An existing quality rating and improvement system (QRIS) and a large number of centers with QRIS ratings in the top and bottom levels:</b> This gave the research team the ability to recruit centers with varying levels of quality.
<b>Variation in the stringency of state licensing requirements:</b> The research team measured this using Child Care Aware rankings of licensing stringency. Child Care Aware ranked states based on the relative strength of their licensing requirements for staff credentialing and training, child-to-adult ratios, group size, health and safety practices, and other criteria (Child Care Aware of America 2013).
<b>Variation in geographic region:</b> The research team based this on the four census-defined regions of the country.
The research then selected a purposive sample of centers across the four states to recruit 80 centers for the field test. The research team first recruited centers that participated in an earlier phase of ICHQ data collection (Phase 1) and recruited new centers to get to a total of 80 CCEE centers. Centers in the field test needed to operate full-day, full-year programs and not be embedded within a public school system test. The research team also worked to maximize variation on three center characteristics:

<b>Ages of children served:</b> All centers recruited for the field test needed to serve preschoolers (children ages 3 through 5, not yet in kindergarten), but could also serve infants and toddlers or school-age children. The research team aimed to include about 25 percent of centers in the sample that served infants and toddlers in order to develop measures of cost by age group of children served.
<b>Funding mix:</b> The research team aimed to recruit centers that varied in the types of funding received: child care subsidies, private tuition, Head Start or pre-K, or a mix of these sources. Findings from Phase 2 suggested that costs and implementation, as captured by the ICHQ measures, vary by funding source, and the research team wanted to be able to explore these patterns in more depth with approximately equal numbers of centers in funding subgroups.
<b>QRIS rating:</b> The research team sought variation in QRIS ratings to test associations between the ICHQ measures and quality, a key goal of the field test. The research team categorized centers as having high or low QRIS ratings based on the requirements for the different levels in each state. In Arizona and Colorado, where the QRIS have five rating levels, the research team recruited centers at the top two rating levels. In Arkansas and Pennsylvania, with three and four rating levels, the research team recruited centers at the highest QRIS level. To recruit centers with low QRIS ratings, the research team determined the QRIS rating level that most closely reflects low quality in each state. Three of the states (Colorado, Arizona, and Pennsylvania) set the first level as an entry level at which centers are not officially rated. In these states, the research team did not recruit centers at the lowest QRIS rating level.
description: Datasets: 
DS0: Study-Level Files
DS1: Center Characteristics Data
DS2: Implementation Data
DS3: Staff-Level Time Use Survey Data
DS4: Center-Level Time Use and Teaching Staff Survey (SEQUAL) Data
DS5: Cost Data
description: Center-based child care and early education (CCEE) centers in Arizona, Arkansas, Colorado, and Pennsylvania.
description: telephone interview; web-based survey<b>Project Director:</b>
Pia Caronongan (Mathematica)
<b>Principal Investigators:</b>
Gretchen Kirby (Mathematica), Pia Caronongan (Mathematica)To protect confidentiality, the data producers did not include geographic information in the data.