Appendix A:

National Evaluation of i3 (NEi3) 

Data Collection Survey

This work is supported by the U.S. Department of Education under 

contract number ED-IES-10-C-0064, ED-IES-13-C-0005 and/or EDIES-14-C-0007, Abt Associates, Prime Contractor.
National Evaluation of i3 (NEi3) Data Collection Survey Page 2

Welcome [User_Name]!

The National Evaluation of Investing in Innovation (NEi3) is being conducted by Abt Associates Inc. on 

behalf of the U.S. Department of Education (ED) Institute of Education Sciences (IES). A primary goal of 

the NEi3 is to assess the strength of evidence generated by independent local evaluations of i3 grants and 

to provide summaries of evaluation findings. The purpose of this data collection is to collect information 

from evaluators like you regarding the findings of your i3 independent evaluation. We look forward to 

working with you on this important endeavor!

Abt Associates and its subcontractors follow the confidentiality and data protection requirements of IES 

(The Education Sciences Reform Act of 2002, Title I, Part E, Section 183), which require that all 

collection, maintenance, use and wide dissemination of data conform to the requirements of the Privacy 

Act of 1974 (5 U.S.C. 552a), the Family Educational Rights and Privacy Act of 1974 (20 U.S.C. 1232g), 

and the Protection of Pupil Rights Amendment (20 U.S.C. 1232h). The study team will not be collecting 

any individually identifiable information. All of the data requested from independent evaluators will be in 

the form of aggregated reports of the methods, measures, plans, and results in their independent 

evaluations.

Responses to this data collection will be used to disseminate information about your study findings. Individual 

i3 grants will be identified in this study. The characteristics, results, and findings reported by independent 

evaluators, as well as assessments of the quality of the independent evaluations and i3 projects, may potentially 

be reported. This study cannot, however, associate responses with a specific school or individual participating in 

i3 independent evaluations, since we are not collecting that information.

Grantees and their independent evaluators are required to comply with the requirements of the NEi3. 

Participation in at least one of the NEi3’s data collection periods is required. Your response is critical for 

producing valid and reliable data. You may skip any questions you do not wish to answer; however, we hope 

that you answer as many questions as you can. Your answers to questions will not affect your grant—now or in 

the future. Participation in the NEi3 Findings Survey will not impose any risks to you as a respondent. If you 

have any questions about your rights as a research participant, you can contact Teresa Doksum, the Chairperson 

of Abt’s IRB at 617-349-2896 or by email at irb@abtassoc.com. 

This survey is authorized by law:American Recovery and Reinvestment Act of 2009 (ARRA), Section 

14007, Title XIV (Public Law (P.L.) 111-5.

We estimate that the survey will take you approximately 15 hours to complete.

OMB Clearance Number: xxxx-xxxx Expiration Date: xx/xx/xxxx

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it

displays a valid OMB control number. The valid OMB control number for this information collection is xxxx-xxxx. The time required 

to complete this information collection is estimated to average approximately 15 hours per respondent, including the time to review 

instructions, gather the data needed, and complete and review the information collected. If you have any comments concerning the 

accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, 

DC 20202. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: U.S. 

Department of Education, Institute of Education Sciences, 555 New Jersey Avenue, N.W., Washington, DC 20208.
National Evaluation of i3 (NEi3) Data Collection Survey Page 3

Overview of the NEi3 Data Collection Survey

Throughout the data collection survey, the NEi3 study team has pre-populated information available from 

the following sources:

 your analysis plan

 your contrast tool

 reports or other documents on interim or final results

Please review this information carefully to ensure its accuracy (you’ll be asked to actively confirm the 

accuracy of all pre-populated information).

We realize that it will likely take you a few visits to this website to complete entering your findings into 

this data collection survey. Therefore, you may log into and out of the survey as many times as necessary 

to complete the entire data collection survey by MM/DD/YYYY. The information you enter into this 

survey will be automatically saved once you navigate to a different survey section.

The data collection website is organized into three main sections:

1. Background Section

2. Fidelity of Implementation Section

3. Impact Study Section

After you enter the survey, you can access each of the above sections by clicking on the appropriate 

navigation button at the top of your screen. Below, we briefly describe the content of each of these 

sections.

The Background Section asks you about the parties involved in developing, implementing and evaluating 

the i3-funded intervention on which you worked. It also asks some general questions about the grant, 

intervention, and evaluation.

The Fidelity of Implementation Section asks you to describe the key components of the i3-funded 

intervention you evaluated, the measures constructed to evaluate fidelity of implementation for those 

components, and the findings from your implementation study.

The Impact Study Section asks you to describe each of your evaluation’s confirmatory contrasts. 

Specifically, the survey will ask you to provide information about the results from the analysis for each 

contrast, including the estimated effects of the intervention and the size of the analysis sample. In addition 

to your confirmatory contrasts, you are welcome to provide the details of any additional exploratory 

contrasts from this study that you would like the NEi3 to consider including in the NEi3 Final Report. 

Detailed guidance about how to enter information in each column is available by hovering your mouse on 

the pertinent column header. This section is divided up into the following nine subsections:

 Contrast Description: Key Features

 Contrast Description: Design

 Contrast Description: Outcome Measurement

 Contrast Description: Baseline Measures

 Confounds

 Outcomes

 Assessing Attrition

 Baseline Measurement

 Reported Findings

The same contrasts appear on each subsection, but different items appear on each subsection. So for each 

contrast, you’ll be asked to enter/update descriptive information, and information on confounds, 
National Evaluation of i3 (NEi3) Data Collection Survey Page 4

outcomes, attrition, baseline equivalence, and findings.

Monitoring Progress and Submitting Your Results

As you complete each section (and subsection) of the survey, you will check off the “Section Complete” 

or “Subsection Complete” checkboxes that appear in each section (and subsection). That will allow you to 

track your own progress. Once you have indicated your completion of each section, check off the “Survey 

Complete” checkbox at the top of the survey. That will automatically notify the NEi3 AR team that you 

have completed the survey and it is ready for their review.

Getting Help

For help with technical issues using this website, please call 617-520-2447. For help selecting appropriate 

data for entry and/or for further guidance on survey items, please call 301-347-5638.

Accessing the Survey

For each study there is a corresponding data collection survey. If your grant has only one study, then only 

one survey will appear on the survey selection page. If your grant has multiple studies, you’ll be able to 

open the desired survey on the survey selection page.
National Evaluation of i3 (NEi3) Data Collection Survey Page 5

Background Information

The following section asks you about the parties involved in developing, implementing and evaluating the 

i3-funded intervention on which you worked. It also asks some general questions about the grant, 

intervention, and evaluation. 

1. What is the name of the grantee for which you conducted/are conducting the evaluation?

___________________________________________________________

2. What is the name of the grant under which you conducted/are conducting the evaluation?

________________________________________________________

3. What is the i3 grant type of this grant? 

 Development  Scale-Up  Validation

4. How many impact studies/evaluations are funded under this grant? ______________________

5. If there are multiple impact studies/evaluations funded under this grant, what is the name of 

the study reported in this survey? 

____________________________________________________________________________

6. What is/are the name(s) of the organization(s) that conducted/is conducting the impact study? 

____________________________________________________________________________

7. What is/are the name(s) of the principal investigator(s) that conducted/is conducting the study? 

____________________________________________________________________________

8. What is the name of the i3-funded intervention? 

____________________________________________________________________________
National Evaluation of i3 (NEi3) Data Collection Survey Page 6

9. Please review the following summary description of the i3-funded intervention, and indicate 

its accuracy by selecting the appropriate box below. If the description requires any revision, 

please insert a corrected description in the area provided. Please note that this description is 

intended for inclusion in the NEi3 Findings Report; however, the NEi3 may condense, or seek 

clarifications for, proposed revisions. 

NEi3 Description: This section will be pre-populated by the NEi3 team with a summary of the 

intervention in sufficient detail to help readers understand what makes this intervention similar to 

or different from other interventions. The level of detail will be similar to what would be provided 

in an introductory section of a typical impact evaluation report. It will include the length of the 

intervention and the dosage, as well as information about the content, delivery, and 

implementation of the intervention. 

 The intervention description is correct. (If this box is checked, the NEi3 Description that 

appears above will be used in the NEi3 Findings Report.) 

 The intervention description requires some revision. (If this box is checked, please enter a 

revision below.)

Evaluator Revision: 

10. What are the names of the individual(s) and organization(s) that developed the intervention? 

Individual Organizational affiliation

11. What are the name(s) of the individuals and organization(s) that implemented and/or are 

implementing the intervention being evaluated under this grant? 

Individual Organizational affiliation

Please insert a revised description here if needed.
National Evaluation of i3 (NEi3) Data Collection Survey Page 7

12. Did the following entities review the study findings prior to their release and/or submission to 

NEi3: 

Yes No

The individual/organization implementing the intervention (from 

question 11)  

The individual/organization developing the intervention (from 

question 10)  

Participating School District(s)  

Participating School Principal(s)  

Other, please specify: ____________________________  

If you answered “yes” to any question above, please explain: 

______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts 

are implicated if warranted based on open-ended response.

13. Did the following entities provide comments on the study findings prior to their release and/or 

submission to NEi3? 

Yes No

The individual/organization implementing the intervention (from 

question 11)  

The individual/organization developing the intervention (from 

question 10)  

Participating School District(s)  

Participating School Principal(s)  

Other, please specify: _________________________________  

If you answered “yes” to any question above, please explain how comments were addressed: 

______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts 

are implicated if warranted based on open-ended response.
National Evaluation of i3 (NEi3) Data Collection Survey Page 8

14. For each entity below, did the entity request changes to the analysis and reporting prior to their 

release and/or submission to NEi3? 

Yes No

The individual/organization implementing the intervention (from 

question 11)  

The individual/organization developing the intervention (from 

question 10)  

Participating School District(s)  

Participating School Principal(s)  

Other, please specify: _________________________________  

If you answered “yes” to any question above, please explain whether the requested changes were 

made, and why: 

______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts 

are implicated if warranted based on open-ended response.

15. For each of the following entities, did the entity conduct any of the statistical analysis for the 

findings submitted to NEi3? 

Yes No

The individual/organization implementing the intervention (from 

question 11)  

The individual/organization developing the intervention (from 

question 10)  

Participating School District(s)  

Participating School Principal(s)  

Other, please specify: _________________________________  

If you answered “yes” to any question above, please explain: 

______________________________________________________________________________

Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts 

are implicated if warranted based on open-ended response.
National Evaluation of i3 (NEi3) Data Collection Survey Page 9

16. Please review the following description of the population served by this grant’s i3-funded 

intervention and indicate its accuracy by selecting the appropriate box below. If the 

description requires any revision, please insert a corrected description in the area provided. 

NEi3 Description: This section will be prepopulated with a description of the population served 

by this grant’s i3-funded intervention(s). The description will include the target age/grade, or 

teacher type (for example, fifth grade teacher, elementary school counselor, high school science 

teacher, etc.), of the intervention. It will also include the location of the intervention and a 

description of the type and number of schools served. 

 The intervention description is correct. (If this box is checked, the NEi3 Description that 

appears above will be used in the NEi3 Findings Report.) 

 The intervention description requires some revision. (If this box is checked, please enter a 

revision below.)

Evaluator Revision: 

17. Please review the following description of the design of this study’s impact evaluation. If the 

description requires any revision, please insert a corrected description in the area provided.

NEi3 Description: This section will be prepopulated with a brief narrative description of key 

features of the study design. This description will include the design and level of assessment.

 This description is correct. (If this box is checked, the NEi3 Description that appears above 

will be used in the NEi3 Findings Report.)

 This description requires some revision. (If this is checked, please enter a revision below.)

Evaluator Revision: 

18. Please review the following description of how the study planned to select or assign treatment 

and comparison groups for the impact evaluation. Please update the description to reflect the 

selection or assignment process as implemented and to add details as needed. Information that 

needs to be added is indicated in capital letters.

NEi3 Description of Plans: This section will be prepopulated with the description from the 

Analysis Plan of the process by which treatment and comparison groups are assigned and/or 

identified. For RCTs, this description will include: how the sample for random assignment was 

selected (articulate all inclusion and exclusion criteria); the process for random assignment, 

and blocks or strata, and its timing; any changes to the analysis sample after the point of 

random assignment and reasons for sample loss or gain (e.g., identification of ineligible 

students; joiners); when clusters and individuals were first notified of random assignment; 

when consent was obtained. For non-experimental designs, this description will include: how 

Please insert a revised description here if needed.

Please insert a revised description here if needed.
National Evaluation of i3 (NEi3) Data Collection Survey Page 10

the treatment and comparison samples were selected (articulate all inclusion and exclusion 

criteria); if matching was used, the dimensions on which the groups will be matched and the 

procedures for matching; whether the treatment sample is comprised of individuals who 

volunteered for the intervention; whether the comparison sample is comprised of individuals 

who were not eligible for or actively declined to participate in the intervention; and any 

differences in eligibility criteria between the treatment and comparison groups. If key 

information is not available in the Analysis Plan, the prepopulation will indicate this in capital 

letters.

Evaluator Description of Implementation:

19. Please review the following description of the sample eligibility and inclusion for this study’s 

impact evaluation. If the description requires any revision, please insert a corrected 

description in the area provided. 

NEi3 Description: [Describe how the districts/schools were chosen to be in the study (e.g. 

universe, random sample from a defined population, convenience sample , etc...). Please 

articulate all inclusion and exclusion criteria. For example, “The five largest school districts in 

XYZ County, Texas, were recruited to participate in this study because of their size, and because 

of the high percentage of ELL students in XYZ County. Within each of the five school districts, two 

to four elementary schools will be recruited and half will be randomly assigned to treatment and 

half will be assigned to control. The intervention is a school-wide intervention. All recruited 

schools will be included in the impact evaluation. Outcomes will be assessed for students in grades 

three through five. The primary research question concerns the impact on school average student 

test scores after the intervention has been in schools for three years.” Finally, please ensure it is 

clear in the summary to whom the sample was designed to generalize.]

 The intervention description is correct. (If this box is checked, the NEi3 Description that 

appears above will be used in the NEi3 Findings Report.) 

 The intervention description requires some revision. (If this box is checked, please enter a 

revision below.)

Evaluator Revision: 

20. For both the full population served by this grant’s i3-funded intervention, and the sample of 

the population included in this study’s impact estimates, please provide the following statistics 

at the student-level if available and at the school-level if the student-level is not available:

Please insert a revised description here.

Please insert a revised description here if needed.
National Evaluation of i3 (NEi3) Data Collection Survey Page 11

Full Population Served by this 

Grant’s i3-Funded 

Intervention

Sample of the Population 

Included in this Study’s 

Impact Estimates

Location 

City n_____ (___%) n_____ (___%)

Suburban n_____ (___%) n_____ (___%)

Town n_____ (___%) n_____ (___%)

Rural n_____ (___%) n_____ (___%)

Demographics

American Ind/Alaskan n_____ (___%) n_____ (___%)

Asian/Pacific Islander n_____ (___%) n_____ (___%)

Black n_____ (___%) n_____ (___%)

Hispanic n_____ (___%) n_____ (___%)

White n_____ (___%) n_____ (___%)

Two or more races n_____ (___%) n_____ (___%)

Region 

Northeast n_____ (___%) n_____ (___%)

Southeast n_____ (___%) n_____ (___%)

Midwest n_____ (___%) n_____ (___%)

Southwest n_____ (___%) n_____ (___%)

West n_____ (___%) n_____ (___%)

Grade Level 

Before Kindergarten n_____ (___%) n_____ (___%)

K-5 n_____ (___%) n_____ (___%)

6-8 n_____ (___%) n_____ (___%)

9-12 n_____ (___%) n_____ (___%)

School Ranking 

Student attending Priority 

Schools/ Priority Schools

n_____ (___%) n_____ (___%)

Students attending Focus 

Schools/Focus Schools

n_____ (___%) n_____ (___%)

Other i3 Priority Groups

English Language Learners n_____ (___%) n_____ (___%)

Free/Reduced-Price Lunch n_____ (___%) n_____ (___%)

Overall Number n_____ n_____

Please indicate the level at which the above data are reported: 

 student-level 

 school-level 

School Characteristics

Average school size n_____/school n_____/school

Title I eligible school-wide ___ schools ___ schools

Title I eligible targeted ___ schools ___ schools

Overall Number ___ schools ___ schools
National Evaluation of i3 (NEi3) Data Collection Survey Page 12

Please complete a row for each of the key components of the i3 intervention that is included in your logic model 

and fidelity measurement system. The following instructions define the data that should be entered in each of 

the columns. 

Column Column Header Instructions for completing corresponding cells 

A Key Component

Please enter the name of the key component. 

Key components are the activities and inputs that are under the direct control of the individual or organization 

responsible for program implementation (e.g., program developer, grant recipient), and are considered by the 

developer to be essential for implementing the intervention. Key components may include financial resources, 

professional development for teachers, curricular materials, or technology products.

B Indicators Characterizing Key Component

Using your study’s logic model, design summary, and fidelity matrix as a guide (whenever applicable), please 

provide the definition of each key component, i.e., a list of the indicators that represent each key component. 

Indicators should be quantified where possible. Depending upon how you defined each key component, 

indicators might encompass a combination of activities at the teacher, classroom, school, or sample level. 

Example: Fidelity of implementation for the key component “professional development” is based on four 

professional development indicators: a one-week summer institute, five webinars, two coaching sessions, and 

ongoing PD.

C

Year 1: Component Level Threshold for 

Fidelity of Implementation for the Unit that 

is the Basis for the Sample-Level 

Representation of Fidelity (i.e. definition of 

implementation with fidelity)

Please provide the definition of implementation with fidelity for the unit that is the basis for the sample-level 

representation of fidelity in Year 1 for each key component; i.e., the component level threshold that delineates 

implementation with fidelity from implementation without fidelity for a particular sample unit. 

Example of unit-level: If the sample-level fidelity threshold is defined as the percent of schools that have 

received an adequate score on a key component, the unit that is the basis for the sample-level representation of 

fidelity is “school”. The definition of implementation with fidelity could be: “a single school will be considered to 

have implemented the key component with fidelity if at least 90% of teachers at the school: attended the oneweek summer institute, attended four of five webinars, attended two of two coaching sessions, and received 

ongoing PD as needed (as self-reported in teacher survey)."
National Evaluation of i3 (NEi3) Data Collection Survey Page 13

Column Column Header Instructions for completing corresponding cells 

D

Year 1: Sample Size (of Units Measured at 

the Sample Level) 

Please enter the sample size (N) of units associated with the sample-level threshold and results. For example, if 

sample-level threshold is 80% or more schools are “high” implementers, please enter the number of schools.

E

Year 1: Representativeness of sample: 

Measured on all (A), some (S), or none (N) 

of the members representing the 

intervention group in the impact 

evaluation

Please enter “A”, “S”, or “N” for this key component.

Example: If Kindergarten (K) teachers are included in the measurement of this key component and the 

intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for 

some. 

Example: If all school districts in the evaluation group were also included in the implementation measurement of 

this key component, the evaluator would enter “A” for all.

F

Year 1: Component Level Threshold for 

Fidelity of Implementation at the Sample 

Level (i.e. definition of implementation 

with fidelity)

Please provide the definition of implementation with fidelity at the sample-level in Year 1 for each key component; 

i.e., the component level threshold that delineates implementation with fidelity from implementation without 

fidelity across sample units. 

Examples of sample-level: “At least 80% of schools implement with fidelity” or “60% or more of districts 

implement with fidelity”.

The sample-level threshold (e.g., across schools) should be a summary measure of fidelity score ranges and 

thresholds established at level of the unit that is the basis for the sample-level representation of fidelity (e.g., 

teacher, classroom, school).

G

Year 1: Component Level Fidelity Score for 

the Entire Sample

Please report the component level fidelity scores achieved in Year 1 of the intervention for each key component. 

Examples: “85% of schools implemented with fidelity” or “70% of districts implemented with fidelity”

H

Year 1: Component Level Implementation 

with Fidelity at the Sample Level? (Yes/No)

Note: This field is automatically calculated based on the two prior fields.

I

Year 2: Component Level Threshold for 

Fidelity of Implementation for the Unit that 

is the Basis for the Sample-Level 

Representation of Fidelity (i.e. definition of 

implementation with fidelity)

Same instructions as in column C will appear in survey instrument, replacing year one with year two.

J

Year 2: Sample Size (of Units Measured at 

the Sample Level

Same instructions as in column D will appear in survey instrument.

K

Year 2: Representativeness of sample: 

Measured on all (A), some (S), or none (N) 

of the members representing the 

intervention group in the impact 

evaluation

Please enter “A”, “S”, or “N” for this key component.

Example: If Kindergarten (K) teachers are included in the measurement of this key component and the 

intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for 

some. 

Example: If all school districts in the evaluation group were also included in the implementation measurement of 

this key component, the evaluator would enter “A” for all. 
National Evaluation of i3 (NEi3) Data Collection Survey Page 14

Column Column Header Instructions for completing corresponding cells 

L

Year 2: Component Level Threshold for 

Fidelity of Implementation at the Sample 

Level (i.e. definition of implementation 

with fidelity)

Same instructions as in column E will appear in survey instrument, replacing year one with year two.

M

Year 2: Component Level Fidelity Score for 

the Entire Sample

Same instructions as in column F will appear in survey instrument, replacing year one with year two.

N

Year 2: Component Level Implementation 

with Fidelity at the Sample Level? (Yes/No)

Same instructions as in column G will appear in survey instrument.

O

Year 3: Component Level Threshold for 

Fidelity of Implementation for the Unit that 

is the Basis for the Sample-Level 

Representation of Fidelity (i.e. definition of 

implementation with fidelity)

Same instructions as in column C will appear in survey instrument, replacing year one with year three.

P

Year 3: Sample Size (of Units Measured at 

the Sample Level

Same instructions as in column D will appear in survey instrument.

Q

Year 3:Representativeness of sample: 

Measured on all (A), some (S), or none (N) 

of the members representing the 

intervention group in the impact 

evaluation

Please enter “A”, “S”, or “N” for this key component.

Example: If Kindergarten (K) teachers are included in the measurement of this key component and the 

intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for 

some. 

Example: If all school districts in the evaluation group were also included in the implementation measurement of 

this key component, the evaluator would enter “A” for all. 

R

Year 3: Component Level Threshold for 

Fidelity of Implementation at the Sample 

Level (i.e. definition of implementation 

with fidelity)

Same instructions as in column E will appear in survey instrument, replacing year one with year three.

S

Year 3: Component Level Fidelity Score for 

the Entire Sample

Same instructions as in column F will appear in survey instrument, replacing year one with year three.

T

Year 3: Component Level Implementation 

with Fidelity at the Sample Level? (Yes/No)

Same instructions as in column G will appear in survey instrument.

U

Year 4: Component Level Threshold for 

Fidelity of Implementation for the Unit that 

is the Basis for the Sample-Level 

Representation of Fidelity (i.e. definition of 

implementation with fidelity)

Same instructions as in column C will appear in survey instrument, replacing year one with year four.

V

Year 4: Sample Size (of Units Measured at 

the Sample Level

Same instructions as in column D will appear in survey instrument.
National Evaluation of i3 (NEi3) Data Collection Survey Page 15

Column Column Header Instructions for completing corresponding cells 

W

Year 4: Representativeness of sample: 

Measured on all (A), some (S), or none (N) 

of the members representing the 

intervention group in the impact 

evaluation

Please enter “A”, “S”, or “N” for this key component.

Example: If Kindergarten (K) teachers are included in the measurement of this key component and the 

intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for 

some. 

Example: If all school districts in the evaluation group were also included in the implementation measurement of 

this key component, the evaluator would enter “A” for all. 

X

Year 4: Component Level Threshold for 

Fidelity of Implementation at the Sample 

Level (i.e. definition of implementation 

with fidelity)

Same instructions as in column E will appear in survey instrument, replacing year one with year four.

Y

Year 4: Component Level Fidelity Score for 

the Entire Sample

Same instructions as in column F will appear in survey instrument, replacing year one with year four.

Z

Year 4: Component Level Implementation 

with Fidelity at the Sample Level? (Yes/No)

Same instructions as in column G will appear in survey instrument.

AA

Year 5: Component Level Threshold for 

Fidelity of Implementation for the Unit that 

is the Basis for the Sample-Level 

Representation of Fidelity (i.e. definition of 

implementation with fidelity)

Same instructions as in column C will appear in survey instrument, replacing year one with year five.

AB Year 5: Sample Size (of Units Measured at 

the Sample Level

Same instructions as in column D will appear in survey instrument.

AC

Year 5: Representativeness of sample: 

Measured on all (A), some (S), or none (N) 

of the members representing the 

intervention group in the impact 

evaluation

Please enter “A”, “S”, or “N” for this key component.

Example: If Kindergarten (K) teachers are included in the measurement of this key component and the 

intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for 

some. 

Example: If all school districts in the evaluation group were also included in the implementation measurement of 

this key component, the evaluator would enter “A” for all. 

AD

Year 5: Component Level Threshold for 

Fidelity of Implementation at the Sample 

Level (i.e. definition of implementation 

with fidelity)

Same instructions as in column E will appear in survey instrument, replacing year one with year five.

AE Year 5: Component Level Fidelity Score for 

the Entire Sample

Same instructions as in column F will appear in survey instrument, replacing year one with year five.

AF Year 5: Component Level Implementation 

with Fidelity at the Sample Level? (Yes/No)

Same instructions as in column G will appear in survey instrument.
National Evaluation of i3 (NEi3) Data Collection Survey Page 16

Column Column Header Instructions for completing corresponding cells 

Scale-Up Key Components (Scale-Up grants only)

AG Key component / activities related to 

scale-up

Please enter each key component necessary to support scale-up efforts separately.

Scale-Up Goals (Scale-Up grants only)

AH Goal Please enter scale-up goals separately. Scale-up goals should be quantified and measurable, e.g., 500 additional 

teachers trained.

AI Results at grant end Please enter the results of each scale-up goal achieved by the end of the grant period, e.g., If 523 additional 

teachers were trained, enter “523” in the field

AJ Goal met Please enter a final determination (yes/no) indicating whether each scale-up goal was met by grant end.

AK Results Year 1 Optional: if you choose, please enter scale-up results for each goal at the end of each or any year(s) of 

implementation. For example, if scale-up goals were measured in years 3 and 4, enter results for each component 

in those fields and enter N/A (or leave blank) fields associated with years 1, 2, and 5.

AL Results Year 2

AM Results Year 3

AN Results Year 4

AO Results Year 5
National Evaluation of i3 (NEi3) Data Collection Survey Page 17

Please complete a row for each contrast measured by your impact study. The following instructions define the 

data that should be entered in each of the columns. 

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

Contrast Description

A Contrast Name

Please enter the names of each of the evaluation’s confirmatory and exploratory 

contrasts. This field does not affect the assessment or reporting of the contrast; 

the contrast name simply facilities communication between the evaluator and the 

NEi3 AR Team, and facilitates referencing contrasts between the data-entry 

fields of this survey. The contrast name should uniquely identify the contrast and 

differentiate among similar contrasts: e.g. “Math Achievement-confirmatory” and 

“Math Achievement-cohort 1” only might be the names of two similar contrasts.

Field is required.

Contrast Description: Key features that define a contrast

B

Confirmatory or 

Exploratory?

Indicate whether this contrast is confirmatory or exploratory. Please note that 

any pre-specified contrasts that were registered as confirmatory will be prepopulated as such and cannot be changed to be exploratory. All other contrasts 

will be listed as exploratory.

Field is required.

Only AR team members can edit this 

field.

Default value for new contrasts is 

"exploratory".

C Contrast Status

For each contrast, please select from among the following options:

- Contrast Confirmed if the effectiveness findings from this contrast are reported 

in this survey

- Not Reported if the effectiveness findings from this contrast are not reported in 

this survey and will not be reported to the NEi3 in the future

- Pending Additional Data if the effectiveness findings from this contrast will be 

reported in future waves of data collection. If you select this option, please 

provide the date when you plan to report effectiveness findings from this contrast 

in the next field.

Field is always applicable.

Drop down menu:

- Contrast confirmed

- Pending Additional Data

- Not reported
National Evaluation of i3 (NEi3) Data Collection Survey Page 18

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

D Expected Reporting Date

If effectiveness findings for this contrast are not yet available, please provide the 

date when you plan to report effectiveness findings.

(Not applicable for contrasts for which data is provided in this survey or for 

contrasts for which findings will never be reported.)

If the contrast status is NOT "pending 

additional data", display "Not 

Applicable".

E Contrast Notes

This field is not required, but is available to provide contextual information or 

notes about each specific contrast as needed.

Field is always applicable.

F Outcome Domain

For confirmatory contrasts, this field is populated with the registered domain 

name of your pre-specified confirmatory contrast. As the domain is a defining 

feature of a contrast, it cannot be edited by the evaluator. Contact the helpdesk 

to correct typos or other minor errors.

For exploratory contrasts, please enter the name of the domains measured. If 

the exploratory contrast measures the same construct as one of the confirmatory 

contrasts, please use the same domain name.

Field is always applicable.

Field cannot be edited for confirmatory 

contrasts.

G Intervention Condition

For confirmatory contrasts, this field is populated with the registered description 

of the intervention tested by the pre-specified confirmatory contrast. As the 

intervention condition is a defining feature of a contrast, it cannot be edited.

For exploratory contrasts, please name the intervention(s) experienced by the 

treatment group in the study. Do not use acronyms. If the condition is the same 

as the one tested by a confirmatory contrast, please use the same name.

If the contrast involves comparing one treatment to another, please identify one 

treatment as the condition and another as the comparison condition for the 

purposes of responding to this survey. Use this identification throughout the 

survey when asked about the treatment/intervention group and the comparison 

group.

Field is always applicable.

Field cannot be edited for confirmatory 

contrasts.
National Evaluation of i3 (NEi3) Data Collection Survey Page 19

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

H Comparison Condition

For confirmatory contrasts, this field is populated with the registered description 

of the counterfactual condition in the pre-specified confirmatory contrast. As the 

intervention condition is a defining feature of a contrast, it cannot be edited.

For exploratory contrasts, please name the intervention(s) experienced by the 

comparison group in the study. This may be the name of another intervention or 

Business as Usual, if no specific intervention. Do not use acronyms. If the 

condition is the same as the one tested by a confirmatory contrast, please use 

the same name.

If the contrast involves comparing one treatment to another, please identify one 

treatment as the condition and another as the comparison condition for the 

purposes of responding to this survey. Use this identification throughout the 

survey when asked about the treatment/intervention group and the comparison 

group.

Field is always applicable.

Field cannot be edited for confirmatory 

contrasts.

I Educational Level

For confirmatory contrasts, this field is populated with the educational level that 

was registered in the pre-specified confirmatory contrast. As the educational level 

is a defining feature of a contrast, it cannot be edited.

For exploratory contrasts, please report the educational level (e.g. elementary, 

middle or high school) of the students that experience the intervention. For 

teacher outcomes, report the educational level of the students in the teachers’ 

classrooms.

Field is always applicable.

Field cannot be edited for confirmatory 

contrasts.
National Evaluation of i3 (NEi3) Data Collection Survey Page 20

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

Contrast Description: Approach to addressing contrast

J Design

Please select one of the following designs: RCT, RDD, QED, Comparison Group 

Design Without a Pretest, ITS, Pre-post.

For confirmatory contrasts, this field is populated with the registered design for 

the contrast. If you edit the pre-populated contents of this field – which suggests 

a change to the study plan registered with the NEi3 – please explain why this 

change was made using the pop-up box that will appear.

Definitions:

Randomized controlled trial (RCT). Under this design, an evaluation uses a 

random process to assign study units to an intervention groups and to a control 

group, and measures the effect of the intervention by comparing the outcomes of 

units assigned to the two groups. 

Regression discontinuity design (RDD). Under this design, an evaluation would 

use a continuous “scoring” rule to assign study units to the intervention. Units 

with scores above a pre-set cutoff value are assigned to the treatment group and

units with scores below the cutoff value are assigned to the comparison group, or 

vice versa. The evaluation then compares the outcomes of the two groups close 

to the cutoff to measure the effect of the intervention. 

Quasi-Experimental Design (QED). A QED compares outcomes for intervention 

participants with outcomes for a comparison group chosen through methods 

other than randomization (RCT) or a scoring rule with a well-defined threshold 

(RDD) and includes one or more pre-intervention measures. Comparative 

Interrupted Time Series designs are treated as QEDs by the NEi3.

Comparison Group Design Without a Pretest. A comparison group design 

without a pre-test compares outcomes for intervention participants with outcomes 

for a comparison group chosen through methods other than randomization (RCT) 

or a scoring rule with a well-defined threshold (RDD) and does not include preintervention observations.

Interrupted Time Series Design (ITS). An ITS design uses observations at 

multiple points in time both before and after the intervention was implemented, 

Field is always applicable.

Drop down menu:

-RCT

-RDD

-QED

-Comparison Group Design without a 

Pretest

-ITS

-Pre-post

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 21

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

and compares the outcomes after the intervention with the trend established prior 

to the intervention. These designs do not use a comparison group.

Pre-post Design. A pre-post design compares outcomes for intervention 

participants after receiving the intervention to their outcomes before receiving the 

intervention. These designs do not use a comparison group.

K Unit of Assignment

For each contrast, please report the unit of assignment or the level at which 

assignment to the intervention and comparison conditions occur (e.g., 

student, teacher, classroom, or school). If you edit the pre-populated contents 

of this field – which suggests a change to the study plan registered with the NEi3 

– please explain why this change was made using the pop-up box that will 

appear.

Field is always applicable.

Drop down menu:

- District

- School

- Teacher

- Classroom

- Student

- Other (contact helpdesk to enter)

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

L

Intervention Group 

[alternately, “Treatment 

Group”]: Description

For confirmatory contrasts, this cell is prepopulated with the registered 

description of the intervention group for the contrast. If you edit the pre-populated 

contents of this field – which suggests a change to the study plan registered with 

the NEi3 – please explain why this change was made using the pop-up box that 

will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please describe which schools, teachers and/or students will be included in 

the intervention (“treatment”) group for this analysis.

Example 1 (Candy Valley): “2012 first-graders in all Candy Valley District 

Elementary Schools reading below grade level who do not have an IEP and were 

randomly assigned to the intervention.” 

Example 2 (Apple Valley): “This contrast pools data across all treated cohorts 

and schools. All students in treated grades in Fuji Elementary (K-2), Granny 

Smith Elementary (K-1) and Red Delicious Elementary (K-2) who were present 

for data collection in Spring 2014 are included in the treatment group.”

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 22

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

M

Age/Grade during the 

Intervention

For confirmatory contrasts, this cell is prepopulated with the registered 

description of the age/grade of the students in the intervention group for the 

contrast. If you edit the pre-populated contents of this field – which suggests a 

change to the study plan registered with the NEi3 – please explain why this 

change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please describe the age or grade-level of students in the intervention group 

during their exposure to the intervention. For teacher outcomes, please enter 

the grade(s) taught by the teachers.

Example 1 (Candy Valley): “The intervention was fielded to first grade students.” 

Example 2 (Apple Valley): “Both Fuji and Red Delicious Elementary Schools 

introduced the intervention in the 2011-2012 school year to all kindergarten 

students, offered the intervention to all kindergarten and first graders in 2012-

2013 and to kindergarten through second graders in 2013-2014. Granny Smith 

Elementary introduced the intervention in the 2012-2013 school year to all 

kindergarten students and offered the intervention to all kindergarten and first 

graders in 2013-2014.”

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

N

Length of Student 

Exposure to Intervention

For confirmatory contrasts, this cell is prepopulated with the registered 

description of the age/grade of the students in the intervention group for the 

contrast. If you edit the pre-populated contents of this field – which suggests a 

change to the study plan registered with the NEi3 – please explain why this 

change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please describe how long (no. of semesters/years) the students in the 

intervention group were engaged in the intervention at the time the 

outcome was measured for this contrast. If you edit the pre-populated 

contents of this field – which suggests a change to the study plan registered with 

the NEi3 – please explain why this change was made using the pop-up box that 

will appear.

Example 1 (Candy Valley): “Students received one year of treatment” 

Example 2 (Apple Valley): “When outcomes were measured in Spring 2014, 

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 23

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

both Fuji and Red Delicious Elementary Schools students had the following 

pattern of exposure to treatment: second grade students had three years of 

treatment, first grade students had two years of treatment and kindergarten 

students had one year of treatment. When outcomes were measured in Spring 

2014, Granny Smith Elementary first graders had two years of treatment and 

kindergarten students had one year of treatment.”

O

Comparison Group: 

Description

For confirmatory contrasts, this cell is prepopulated with the registered 

description of the comparison group for the contrast. If you edit the pre-populated 

contents of this field – which suggests a change to the study plan registered with 

the NEi3 – please explain why this change was made using the pop-up box that 

will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please describe which schools, teachers and/or students will be included in 

the comparison group for this analysis.

Example 1 (Candy Valley): “2012 first-graders in all Candy Valley District 

Elementary Schools reading below grade level who do not have an IEP and were 

randomly assigned to the control group.”

Example 2 (Apple Valley): “This contrast pools data across all cohorts and 

schools matched to treated cohorts and schools. K-2 students from Honey Crisp 

Elementary (comparison for Fuji Elementary), K-1 students from Ginger Gold 

Elementary serves (comparison for Granny Smith Elementary), and K-2 students 

Jonagold Elementary (comparison for Red Delicious Elementary) who were 

present for data collection in Spring 2014 are included in the treatment group.”

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

Contrast Description: Outcome Measurement

P

Outcome or Instrument 

Name (full names, not 

acronyms)

For confirmatory contrasts, this cell is prepopulated with the registered outcome 

measure for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report the full name of the outcome measure, including the name of the 

subtest or subscale if applicable. If the instrument for the outcome measure is not 

used in one of the confirmatory contrasts, please describe (1) what the 

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 24

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

instrument is designed to measure and (2) how the outcome measure is 

constructed. 

Q

Construction of Outcome 

Measure (binary, ordinal, 

or continuous)

For confirmatory contrasts, this cell is prepopulated with the registered scale of 

the outcome measure for the contrast. If you edit the pre-populated contents of 

this field – which suggests a change to the study plan registered with the NEi3 –

please explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please indicate whether the outcome measure is binary (dichotomous), ordinal 

(ordered, e.g., “failing, provisionally passing, proficient, advanced”), or 

continuous.

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

R

Unit of Observation for 

Outcome Measure

For confirmatory contrasts, this cell is prepopulated with the registered unit of 

observation for the outcome measure—i.e. the units for which outcomes are 

observed in the analysis data. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear. For

exploratory contrasts and for confirmatory contrasts you wish to change, please 

report the unit for which outcomes are observed in your data (e.g. student, 

teacher, classroom, school, district).

Example: If your data are school-level averages of student test scores, enter 

“School”.

Field is always applicable.

Drop down menu:

- District

- School

- Teacher

- Classroom

- Student

- Other (contact helpdesk to enter)

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

S

Timing of Outcome

Measure (year, Fall/Spring) 

for the Intervention Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

measurement for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

please report when during the calendar year the outcome was measured.

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

T

Timing of Outcome

Measure (age/grade) for 

the Intervention Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

measurement for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 25

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

please report the age or grade level for which the outcome was measured. 

U

Timing of Outcome

Measure (year, Fall/Spring) 

for the Comparison Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

measurement for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

please report when during the calendar year the outcome was measured. 

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

V

Timing of Outcome

Measure (age/grade) for 

the Comparison Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

measurement for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

please report the age or grade level for which the outcome was measured. 

Field is always applicable.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

Contrast Description: Baseline Measures

W Key Baseline Measures

For confirmatory contrasts, this cell is prepopulated with the registered baseline 

measure for the contrast. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

please report the baseline variable(s) on which you are establishing baseline 

equivalence.

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

X

Construction of Baseline 

Measure (binary, ordinal, 

or continuous)

For confirmatory contrasts, this cell is prepopulated with the registered scale of 

the baseline measure for the contrast. If you edit the pre-populated contents of 

this field – which suggests a change to the study plan registered with the NEi3 –

please explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change,

please indicate whether the baseline measure is binary (dichotomous), ordinal 

(ordered, e.g., “failing, provisionally passing, proficient, advanced”), or 

continuous. 

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 26

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

Y

Unit of Observation for 

Baseline Measure

For confirmatory contrasts, this cell is prepopulated with the registered unit of 

observation for the baseline measure—i.e. the units for which outcomes are 

observed in the analysis data. If you edit the pre-populated contents of this field –

which suggests a change to the study plan registered with the NEi3 – please 

explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report the unit for which baseline measures are observed in your data 

(e.g. student, teacher, classroom, school, district).

Example: If your data are school-level averages of student test scores, enter 

“School”.

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

Z

Timing of Baseline 

Measure (year, 

Fall/Spring,) for the 

Intervention Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

baseline measurement for the contrast.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report when during the calendar year the data you will use to test 

baseline equivalence was measured. 

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

AA

Timing of Baseline 

Measure (age/grade) for 

the Intervention Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

baseline measurement for the contrast. If you edit the pre-populated contents of 

this field – which suggests a change to the study plan registered with the NEi3 –

please explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report the age or grade level for which the data you will use to test 

baseline equivalence was measured. 

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.
National Evaluation of i3 (NEi3) Data Collection Survey Page 27

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

AB

Timing of Baseline 

Measure (year, Fall/Spring) 

for the Comparison Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

baseline measurement for the contrast. If you edit the pre-populated contents of 

this field – which suggests a change to the study plan registered with the NEi3 –

please explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report when during the calendar year the data you will use to test 

baseline equivalence was measured. 

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

AC

Timing of Baseline 

Measure (age/grade) for 

the Comparison Group

For confirmatory contrasts, this cell is prepopulated with the registered timing of 

baseline measurement for the contrast. If you edit the pre-populated contents of 

this field – which suggests a change to the study plan registered with the NEi3 –

please explain why this change was made using the pop-up box that will appear.

For exploratory contrasts and for confirmatory contrasts you wish to change, 

please report when the data you will use to test baseline equivalence was 

measured. 

(Field is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

For confirmatory contrasts, changes to 

this field after as planned pre-population 

should bring up a pop-up box asking for 

an explanation.

Confounds

AD Intervention Condition:

Other Activities

To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered 

to the intervention group but not the comparison group? If so, please describe the 

other intervention.

Field is always applicable.

AE

Fundamental 

Confound at the District 

Level: Intervention

Condition

Does the intervention condition include schools/teachers/students from a 

single school district which does not also include comparison group members?

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

AF Fundamental 

Confound at the School 

Does the intervention condition include teachers/students from a single 

school which does not also include comparison group members?

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 
National Evaluation of i3 (NEi3) Data Collection Survey Page 28

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

Level: Intervention

Condition

display “Not Applicable”.

AG Comparison Condition:

Other Activities

To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered 

to the comparison group but not the intervention group? If so, please describe the 

other intervention.

Field is always applicable.

AH Fundamental Confound at 

the District Level: 

Comparison Condition

Does the comparison condition include schools/teachers/students from a 

single school district which does not also include intervention group members?

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

AI

Fundamental Confound at 

the School Level: 

Comparison Condition

Does the comparison condition include teachers/students from a single 

school which does not also include intervention group members?

If the Design is Comparison Group 

without a Pretest, ITS or Pre-post, 

display “Not Applicable”.

Outcomes

AJ

Outcome or Instrument 

Name (full names, not 

acronyms)

This field displays the outcome measure you entered previously. This field is not 

editable. If you wish to edit this entry, go to the Contrast Description: Outcome 

Measurement tab.

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Outcome 

Measurement"

AK

Who Provided the 

Outcome Data?

For each contrast, please report the name of the entity that provided the 

outcome data. 

Field is always applicable.

AL

Value and Type of 

Reliability Measure of 

Instrument for Outcome 

Data

Please provide a narrative description of the outcome and discuss the reliability 

and validity of the measure, including the value and type of reliability measure. 

For example, you may report an internal consistency, test-retest reliability, or 

inter-rater reliability measure.

For outcomes for which reliability statistics cannot be obtained, please describe 

why reliability statistics are not available.

Example 1: “The Orange Gulch District writing assessment was scored by 

trained reviewers according to a holistic rubric. A random subset of essays was 

double scored and demonstrated an inter-rater reliability of 89%.”

Example 2: “Attendance is the proportion of school days for which an individual 

student was present for at least one course at Orange Gulch High School. 

Reliability statistics cannot be calculated for attendance. However, it is a face 

Field is always applicable.
National Evaluation of i3 (NEi3) Data Collection Survey Page 29

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

valid measure of the pre-specified domain: High School Attendance.” 

AM Consistency of Collection

Please describe any differences in the process for collecting data on the outcome 

measure between the groups.

To be consistently collected, an outcome measure must be collected using the 

same rules or procedures in both groups. For example, data would not be 

consistently collected if intervention students take the post-test in January, but 

comparison students take the test in March – two months later.

Field is always applicable.

AN Treatment of Missing Data Describe how the study addressed missing values for the outcome measure. Field is always applicable.

Assessing Attrition

AO Unit of Assignment

This field displays the unit of assignment you entered previously. This field is not 

editable. If you wish to edit this entry, go to the Contrast Description: Design tab.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Design"

AP Unit of Observation for 

Outcome Measure

This field displays the unit of observation for outcome entered previously. This 

field is not editable. If you wish to edit this entry, go to the Contrast Description: 

Outcome Measurement tab.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Outcome 

Measurement"

AQ Cluster Randomized 

Design?

Select “Yes” if the contrast is from a cluster randomized design.

Select “No”, otherwise 

For NEi3, all cluster randomized designs are those in which clusters (of 

individuals) were assigned by a process that was functionally random. These 

clusters may be associated with, for instance, particular teachers, classrooms, 

schools, or districts. 

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Drop down menu:

- Yes

- No

- Not Applicable

AR Analytic Sample Includes 

Joiners?

Select “not applicable” if your design is not a cluster randomized design.

Select “yes” if your analytic sample includes individuals who were not present in 

the cluster at the time of random assignment.

Select “no” if your analytic sample is restricted to individuals who were present at 

If the contrast is NOT a cluster 

randomized design, display "Not 

Applicable".

Drop down menu:

- Yes
National Evaluation of i3 (NEi3) Data Collection Survey Page 30

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

the time of random assignment. 

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

- No

- Not Applicable

AS Description of Joiners

Provide a narrative description of the joiners in your analytic sample. Include 

information on when they joined the sample relative to a) the point of random 

assignment and b) the point at which clusters were notified of their assignment 

status.

Example 1: The analysis sample includes all students in study schools in Year 3. 

Some students may have been transferred into study schools after the point of 

random assignment (Year 1) and may not have two full years of exposure.

Example 2: The student sample is comprised of all students in study teachers’ 

classrooms in Year 2 of the study. Because teacher rosters were not set at the 

point of random assignment (Year 1), all students in Year 2 are considered 

joiners.

(Not applicable if your sample does not include joiners.)

If the contrast does NOT include joiners, 

display "Not Applicable".

AT

Intervention Group: Units 

of Assignment 

Randomized

Enter the number of clusters randomly assigned to the intervention group.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of analysis.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

AU

Intervention Group: 

Reasons for Loss of Units 

of Assignment

List reasons for sample loss separated by a carriage return.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of observation.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".
National Evaluation of i3 (NEi3) Data Collection Survey Page 31

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

AV

Intervention Group: Units 

of Assignment Lost (for 

each reason)

For each reason for sample loss, list the number of units lost.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of observation.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

AW

Intervention Group: Units 

of Assignment in Analytic 

Sample

Enter the number of clusters for the intervention group for the analytic sample.

Complete this field if the contrast is from a cluster randomized design AND the

unit of assignment differs from the unit of analysis

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

AX

Intervention Group: Units 

of Observation 

Randomized or in 

Randomized Cluster

Enter the sample size for the intervention group at the time of random 

assignment

Complete this field the contrast is from:

(a)a cluster randomized design AND the unit of assignment differs from the unit 

of analysis AND the contrast does not include joiners

(b)any randomized design where the unit of analysis is the same as the unit of 

assignment

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

AY

Intervention Group: Units 

of Observation in 

Remaining Clusters

Enter the sample size for the intervention group at the time of random 

assignment in clusters that appear in the analytic sample. This excludes units of 

observation for which the entire cluster left the sample.

E.g. For designs where students are the unit of observation and schools are the 

unit of assignment, enter the number of eligible students at the time of random 

assignment in schools that did not attrit

This field is not applicable if the contrast is from a cluster randomized design 

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".
National Evaluation of i3 (NEi3) Data Collection Survey Page 32

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

AZ

Intervention Group: 

Reasons for Loss of Units 

of Observation

List reasons for sample loss separated by a carriage return.

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

BA

Intervention Group: Units 

of Observation Lost (for 

each reason)

For each reason for sample loss, list the number of units lost.

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

BB

Intervention Group: Units 

of Observation in Analytic 

Sample

Enter the sample size for the intervention group for the analytic sample

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

BC

Comparison Group: Units 

of Assignment

Randomized

Enter the number of clusters randomly assigned to the comparison group.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of analysis

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 
National Evaluation of i3 (NEi3) Data Collection Survey Page 33

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

design, 

OR (c) the unit of observation for 

outcome measure is the same as the

unit of assignment, then display "Not 

Applicable".

BD

Comparison Group: 

Reasons for Loss of Units 

of Assignment

List reasons for sample loss separated by a carriage return.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of observation.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

BE

Comparison Group: Units 

of Assignment Lost (for 

each reason)

For each reason for sample loss, list the number of units lost.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of observation.

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

BF

Comparison Group: Units 

of Assignment in Analytic 

Sample

Enter the number of clusters for the comparison group for the analytic sample.

Complete this field if the contrast is from a cluster randomized design AND the 

unit of assignment differs from the unit of analysis

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is 

(a) NOT an RCT design, 

OR (b) NOT a cluster randomized 

design, 

OR (c) the unit of observation for 

outcome measure is the same as the 

unit of assignment, then display "Not 

Applicable".

BG

Comparison Group: Units 

of Observation 

Randomized or in 

Randomized Cluster

Enter the sample size for the comparison group at the time of random 

assignment.

E.g. For designs where students are the unit of observation and schools are the 

unit of assignment, enter the number of eligible students in all schools at the time 

of random assignment

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 
National Evaluation of i3 (NEi3) Data Collection Survey Page 34

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

the unit of assignment, then display "Not 

Applicable".

BH

Comparison Group: Units 

of Observation in 

Remaining Clusters

Enter the sample size for the comparison group at the time of random 

assignment in clusters that appear in the analytic sample. This excludes units of 

observation for which the entire cluster left the sample.

E.g. For designs where students are the unit of observation and schools are the 

unit of assignment, enter the number of eligible students at the time of random 

assignment in schools that did not attrit

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

BI

Comparison Group: 

Reasons for Loss of Units 

of Observation

List reasons for sample loss separated by a carriage return.

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

BJ

Comparison Group: Units 

of Observation Lost (for 

each reason)

For each reason for sample loss, list the number of units lost.

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 
National Evaluation of i3 (NEi3) Data Collection Survey Page 35

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

designs without a pretest, ITS designs and pre-post designs.) the unit of assignment, then display "Not 

Applicable".

BK

Comparison Group: Units 

of Observation in Analytic 

Sample

Enter the sample size for the comparison group for the analytic sample

This field is not applicable if the contrast is from a cluster randomized design 

AND the unit of assignment differs from the unit of observation AND the contrast 

includes joiners

(Section is not applicable to the following designs: QEDs, comparison group 

designs without a pretest, ITS designs and pre-post designs.)

If the contrast is NOT an RCT design, 

then display "Not Applicable".

If the contrast is (a) a cluster 

randomized design, 

AND (b) includes joiners, 

AND (c) the unit of observation for 

outcome measure is NOT the same as 

the unit of assignment, then display "Not 

Applicable".

Baseline Measurement

BL Key Baseline Measures

This field displays the key baseline measure(s) you entered previously. This field 

is not editable. If you wish to edit this entry, go to the Contrast Description: 

Baseline Measures tab.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Baseline 

Measures"

BM

Construction of Baseline 

Measure (binary, ordinal, 

or continuous)

This field displays the description of the construction of the baseline measure you 

entered previously. This field is not editable. If you wish to edit this entry, go to 

the Contrast Description: Baseline Measures tab.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Baseline 

Measures"

BN Unit of Observation for

Baseline Measure

This field displays the level of observation for the baseline measure you entered 

previously. This field is not editable. If you wish to edit this entry, go to the 

Contrast Description: Baseline Measures tab.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

Field is always applicable.

Field can only be edited in previous tab: 

"Contrast Description: Baseline 

Measures"

BO

Treatment of Missing Data 

for Baseline Balance 

Testing

Describe how the study addressed missing values for the baseline measure for 

baseline balance testing.

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".
National Evaluation of i3 (NEi3) Data Collection Survey Page 36

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

BP

Narrative Description of 

Sample for Baseline 

Balance Testing

Describe the eligibility criteria for inclusion in the sample for baseline balance 

testing. If balance is established using cluster-level averages (e.g. school-level 

average student test score), describe which individual units are included in the 

average.

BQ

Type of Information Used 

to Assess Baseline 

Equivalence

Select one option that best represents the information generated by your 

baseline balance testing:

Unadjusted Mean and Standard Deviation

T-statistic

Dichotomous

T/C Difference

This field determines which of the following fields you need to complete. Fields 

that do not need to be completed are populated with “Not applicable”

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BR

Narrative Description of 

Analysis for Baseline

Balance Testing

Provide a brief narrative description of the method used to assess baseline 

balance.

Example 1: Baseline summary statistics for both groups in the analytic sample 

were calculated. Data are from unadjusted means and standard deviations.

Example 2: Differences in baseline means were calculated using a statistical 

model with the same structural components as the statistical model that was 

used to estimate impacts on the outcome variable for the same contrast. This is a 

two level model with students nested in schools. The pretest variable is the 

independent variable. The only covariates included in the model are an intercept 

term and a school level treatment indicator.

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BS

Baseline Balance Testing 

Sample Size: Intervention 

Group

Enter the sample size for the intervention group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".
National Evaluation of i3 (NEi3) Data Collection Survey Page 37

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

BT

Mean of 

Baseline Measure: 

Intervention Group

Enter the pre-intervention mean for the intervention group. For binary measures, 

enter a decimal value.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BU

Standard Deviation of 

Baseline Measure: 

Intervention Group

Enter the pre-intervention standard deviation for the intervention group.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BV

Baseline Balance Testing 

Sample Size: Comparison 

Group

Enter the sample size for the comparison group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BW

Mean of 

Baseline Measure: 

Comparison Group

Enter the pre-intervention mean for the comparison group. For binary measures, 

enter a decimal value.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BX

Standard Deviation of 

Baseline Measure: 

Comparison Group

Enter the pre-intervention standard deviation for the comparison group.

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

BY Baseline InterventionComparison Difference

Enter the difference between the intervention and comparison group means. If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".
National Evaluation of i3 (NEi3) Data Collection Survey Page 38

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

BZ T-Statistic for Baseline 

Balance Test

Enter the t-statistic for the pre-intervention difference in means between the 

treatment and the comparison

(Section is not applicable to the following designs: comparison group designs 

without a pretest, ITS designs and pre-post designs.)

If the contrast design is 

(a) Design is Comparison Group without 

a Pretest,

OR (b) ITS, 

OR (c) Pre-post, then display "Not 

Applicable".

Reported Findings

CA Type of Findings Reported

Select one of the following types of data to use to report findings:

Unadjusted means 

Adjusted means, e.g. ANOVA

T-test

Coefficient from a linear regression model, e.g. OLS, HLM, etc. 

Coefficient from a non-linear regression model, e.g. logit or probit 

This field determines which of the following fields you need to complete. Fields 

that do not need to be completed are populated with “Not applicable”

Field is always applicable.

Drop down menu:

- Adjusted means, e.g. ANOVA

- T-test

- Coefficient from a linear regression 

model, e.g. OLS, HLM, etc. 

- Coefficient from a non-linear 

regression model, e.g. logit or probit

CB

Narrative Description of 

Analysis for Impact 

Evaluation

Provide a brief narrative description of the method used to estimate impact. 

Include information that describes: 

- the nested or longitudinal structure of the model

- any adjustments for clustering of standard errors

- if the outcome is a binary or categorical measure and, if so, what kind of 

estimation is performed

- whether the pre-test variable is included and the level of inclusion

- the level at which the treatment variable enters the model

Example 1: Impact estimates come from a two-level regression model with 

students nested in schools. The pretest variable is included as a student-level 

covariate. Treatment enters the model at the school level. 

Example 2: We use logistic regression to estimate the impact of the intervention 

on attendance: a binary measure defined for each student. Estimates are logodds-ratios. Standard errors are clustered at the school-level.

Field is always applicable.

CC Intervention Group: 

Sample Size

Please report the size of the intervention group sample for the analysis. Field is always applicable.

CD Intervention Group: 

Number of Clusters in 

Please report the number of clusters assigned to the intervention group that 

appear in the analysis sample. 

If the unit of observation for outcome 

measure is the same as the unit of 
National Evaluation of i3 (NEi3) Data Collection Survey Page 39

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

Analytic Sample

Although your analysis may account for clustering at multiple levels, this field only 

asks about clusters at the level of assignment: e.g. number of schools, if schools 

were assigned.

assignment, display "Not Required".

CE Comparison Group: 

Sample Size

Please report the size of the comparison group sample for the analysis. Field is always applicable.

CF Comparison Group: 

Number of Clusters in 

Analytic Sample

Please report the number of clusters assigned to the comparison group that 

appear in the analysis sample. 

Although your analysis may account for clustering at multiple levels, this field only 

asks about clusters at the level of assignment: e.g. number of schools, if schools 

were assigned.

If the unit of observation for outcome 

measure is the same as the unit of 

assignment, display "Not Required".

CG Intraclass Correlation 

Coefficient

Please report the intraclass correlation coefficient to describe the similarity of 

units at the level of observation within the cluster at the level of assignment 

Although your analysis may account for clustering at multiple levels, this field only 

asks about clusters at the level of assignment, e.g. the correlation between 

outcomes of students within the same school, if schools were assigned.

If the unit of observation for outcome 

measure is the same as the unit of 

assignment, display "Not Required".

CH

Unadjusted Outcome 

Mean for the Intervention 

Group

Please provide the unadjusted mean of the outcome measure in the intervention 

group.

(You do not need to complete this field if it is automatically populated with “Not 

applicable”. However, you can choose to provide this information if you wish.)

If the type of findings is NOT 

unadjusted means, display "Not 

Required".

CI Adjusted Outcome Mean 

for the Intervention Group

Please provide the adjusted mean of the outcome measure in the intervention 

group. 

(You do not need to complete this field if it is automatically populated with “Not 

applicable”. However, you can choose to provide this information if you wish.)

If the type of findings is NOT adjusted 

means, display "Not Required".

CJ

Unadjusted Standard 

Deviation in the Outcome 

Measure of the 

Intervention Group

Please provide the unadjusted standard deviation of the outcome measure in the 

intervention group.

Field is always applicable.

CK

Unadjusted Outcome 

Mean for the Comparison 

Group

Please provide the unadjusted mean of the outcome measure in the comparison 

group. 

(You do not need to complete this field if it is automatically populated with “Not 

If the type of findings is NOT 

unadjusted means, display "Not 

Required".
National Evaluation of i3 (NEi3) Data Collection Survey Page 40

Column Column Header

Instructions for completing corresponding cells (for each contrast)

INTERNAL USE ONLY: 

Programming notes about skip 

patterns

applicable”. However, you can choose to provide this information if you wish.)

CL Adjusted Outcome Mean 

for the Comparison Group

Please provide the unadjusted mean of the outcome measure in the comparison 

group. 

(You do not need to complete this field if it is automatically populated with “Not 

applicable”. However, you can choose to provide this information if you wish.)

If the type of findings is NOT adjusted 

means, display "Not Required".

CM

Unadjusted Standard 

Deviation in the Outcome 

Measure of the 

Comparison Group

Please provide the unadjusted standard deviation of the outcome measure in the 

comparison group.

Field is always applicable.

CN

Impact Estimate (e.g., 

coefficient from OLS or 

HLM)

Please report the regression coefficient that estimates of the effect of the 

intervention. If your impact estimate is the sum of multiple regression 

coefficients, please enter this sum here. 

(You do not need to complete this field if it is automatically populated with “Not 

applicable”.)

If the type of findings is NOT a 

coefficient from a linear or non-linear 

regression model, display "Not 

Required".

CO Standard Error of Impact 

Estimate

Please insert the estimated standard error of the regression coefficient here. 

If your impact estimate is the sum of multiple coefficients, please enter the 

standard deviation of the summed variable. 

(You do not need to complete this field if it is automatically populated with “Not 

applicable”.)

If the type of findings is unadjusted 

means, display "Not Required".

CP Two-tailed p-value

Please insert the associated p-value that has not been adjusted for multiple 

comparisons here. If your impact estimate is the sum of multiple coefficients, 

please enter the associated p-value here.

(You do not need to complete this field if it is automatically populated with “Not 

applicable”.)

If the type of findings is unadjusted 

means, display "Not Required".

CQ Degrees of Freedom

Please enter the degrees of freedom of your statistical analysis. 

(You do not need to complete this field if it is automatically populated with “Not 

applicable”.)

If the type of findings is unadjusted 

means, display "Not Required".