# Appendix A: **National Evaluation of i3 (NEi3) Data Collection Survey** 

**Partners:** Abt Associates Inc. Dillon -Goodson Research Associates Chesapeake Research Associates ANALYTICA, Inc. Westat Century Analytics American Institutes for Reserach

This work is supported by the U.S. Department of Education under contract number ED- IES- 10-C-0064, ED- IES- 13-C-0005 and/or EDIES- 14-C-0007, Abt Associates, Prime Contractor.National Evaluation of i3 (NEi3) Data Collection Survey Page 2



### Welcome [ **First_Name Last_Name ]!**  The National Evaluation of Investing in Innovation (NEi3) is being conducted by Abt Associates Inc. on behalf of the U.S. Department of Education (ED) Institute of Education Sciences (IES). A primary goal of the NEi3 is to assess the strength of evidence generated by independent local evaluations of i3 grants and to provide summaries of evaluation findings. The purpose of this data collection is to collect information from evaluators like you regarding the findings of your i3 independent evaluation. We look forward to working with you on this important endeavor! 

### We would like y**ou to know that...** Your response is critical for producing valid and reliable data. You may skip any questions you do not wish to answer; however, we hope that you answer as many questions as you can. Your answers to qu estions will not affect your grant

—

now or in future. Participation in the evaluator survey will not impose any risks to you as a respondent. If you have any questions about your rights as a research participant, you can contact Teresa Doksum, the Chairpers on of Abt’s IRB at 617 -349 -2896 or by email at irb@abtassoc.com. Responses to this data collection will be used to disseminate information about your study findings . This survey is authorized by law: 

### American Recovery and Reinvestment Act of 2009 (ARRA), Section 14007, Title XIV (Public Law (P.L.) 111 - 5 . **OM B Clearance Number:**  **xxxx -** **xxxx Expiration Date:**  **xx/xx/xxxx** According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is **xxxx -** **xxxx** . The time required to complete this information collection is estimated to average approximately 3 hours per respondent, in cluding the time to review instructions, gather the data needed, and complete and review the information collected. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: U.S. Department of Education, Institute of Education Sciences, 555 New Jersey Avenue, N.W., Washington, DC 20208.National Evaluation of i3 (NEi3) Data Collection Survey Page 3



## Overview**of the NEi3 Data Collection Surve y**  Throughout the data collection survey, the NEi3 study team has pre-populated information available from the following sources:  Documents you produced to fulfill requirements of ED’s cooperative agreements (your analysis plan and your contrast tool )  Reports or other documents on interim or final results you’ve provided us Please review this information carefully to ensure its accuracy (you’ll be asked to actively confirm the accuracy of all pre-populated information). We realize that it may take you a few visits to this website to complete entering your findings into this data collection survey. Therefore, you may log into and out of the survey as many times as necessary to complete the entire data collection survey by MM/DD/YYYY. The information you enter into this survey will be automatically saved once you navigate to a different survey section. The data collection website is organized into three main sections: 1. Background Section 2. Fidelity of Implementation Section 3. Impact Study Section After you enter the survey, you can access each of the above sections by clicking on the appropriate navigation button at the top of your screen. Below, we briefly describe the content of each of these sections. The **Background Section** asks you about the parties involved in developing, implementing and evaluating the i3 -funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation. The **Fidelity of Implementation Section** asks you to describe the key components of the i3 -funded intervention you evaluated, the measures constructed to evaluate fidelity of implementation for those components, and the findings from your implementation study. The **Impact Study Section** asks you to describe each of your evaluation’s contrasts. Specifically, the survey will ask you to provide information about the results from the analysis for each contrast, including the estimated effects of the intervention and the size of the analysis sample. 

## Getting Help For Help with technical issues using this website, please call **617 -** **520 -** **2447** For Help with selecting appropriate data for entry and/or for further guidance on survey items, please call **301 -** **347 -** **5638** .National Evaluation of i3 (NEi3) Data Collection Survey Page 4



## Background Information The following section asks you about the parties involved in developing, implementing and evaluating the i3 -funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation. 1. What is the name of the grantee for which you conducted/are conducting the evaluation?

___________________________________________________________ 2. What is the name of the **grant** under which you conducted/are conducting the evaluation?



### ________________________________________________________ 3. What is the i3 grant type of this grant?

Development Scale-Up Validation 4. How many impact studies/evaluations are funded under this grant? ______________________ 5. If there are multiple impact studies/evaluations funded under this grant, what is the name of the study reported in this survey?

____________________________________________________________________________ 6. What is/are the name(s) of the organization(s) that conducted/is conducting the impact study?

____________________________________________________________________________ 7. What is/are the name(s) of the principal investigator(s) that conducted/is conducting the study?

____________________________________________________________________________ 8. What is the name of the i3 -funded intervention?

____________________________________________________________________________National Evaluation of i3 (NEi3) Data Collection Survey Page 5



### 9. Please review the following summary description of the i3 -funded intervention, and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided. Please note that this description is intended for inclusion in the NEi3 Findings Report; however, the NEi3 may condense, or seek clarifications for, proposed revisions. **NEi3 Description:** This section will be pre
- populated by the NEi3 team with a summary of the intervention in sufficient detail to help readers understand what makes this intervention similar to or different from other interventions. The level of detail will be similar to what would be provided in an introductory section of a typical impact evaluation report. It will include the length of the intervention and the dosage, as well as information about the content, delivery, and implementation of the intervention.  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:**

10. What are the names of the individual(s) and organization(s) that developed the intervention? **Individual Organizational affiliation** 

11. What are the name(s) of the individuals and organization(s) that implemented and/or are implementing the intervention being evaluated under this grant? **Individual Organizational affiliation**  

### Please insert a revised description here if needed.National Evaluation of i3 (NEi3) Data Collection Survey Page 6



### 12. Did the following entities review the study findings prior to their release and/or submission to NEi3: **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: ____________________________ 

#   If you answered “yes” to any question above, please explain:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response. 13. Did the following entities provide comments on the study findings prior to their release and/or submission to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain how comments were addressed:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response.National Evaluation of i3 (NEi3) Data Collection Survey Page 7



### 14. For each entity below, did the entity request changes to the analysis and reporting prior to their release and/or submission to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain whether the requested changes were made, and why:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response. 15. For each of the following entities, did the entity conduct any of the statistical analysis for the findings submitted to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open-ended response.National Evaluation of i3 (NEi3) Data Collection Survey Page 8



### 16. Please review the following description of the population served by this grant’s i3 -funded intervention and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided. **NEi3 Description** : This section will be prepopulated with a description of the population served by this grant’s i3
-

funded intervention(s). The description will include the target age/grade, or teacher type (for example, fifth grade teacher, elementary school counselor, high school science teacher, etc.), of the intervention. It will also include the location of the intervention and a description of the type and number of schools served.  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:**

17. Please review the following description of the sample eligibility and inclusion for this study’s impact evaluation. If the description requires any revision, please insert a corrected description in the area provided. **NEi3 Description** : [Describe how the districts/schools were chosen to be in the study (e.g. universe, random sample from a defined population, convenience sample , etc...). Please articulate all inclusion and exclusion criteria. For example, “The five largest school districts in XYZ County, T

exas, were recruited to participate in this study because of their size, and because of the high percentage of ELL students in XYZ County. Within each of the five school districts, two to four elementary schools will be recruited and half will be randomly assigned to treatment and half will be assigned to control. The intervention is a school
-wide intervention. All recruited schools will be included in the impact evaluation. Outcomes will be assessed for students in grades three through five. The primary re search question concerns the impact on school average student test scores after the intervention has been in schools for three years.” Finally, please ensure it is clear in the summary to whom the sample was designed to generalize.]  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:** 

### Please insert a revised description here if needed.National Evaluation of i3 (NEi3) Data Collection Survey Page 9



### 18. For both the full population served by this grant’s i3 -funded intervention, and the sample of the population included in this study’s impact estimates, please provide the following statistics at the student-level if available and at the school-level if the student-level is not available: **Full Population Served**  **by this Grant’s i3**  **- Funded Intervention**  **Sample of the Population Included**  **in this Study’s Impact Estimates**  **Location** City n_____ (___%) n_____ (___%) Suburban n_____ (___%) n_____ (___%) Town n_____ (___%) n_____ (___%) Rural n_____ (___%) n_____ (___%) **Demographics** Am erican Ind/Alaskan n_____ (___%) n_____ (___%) Asian/Pacific Islander n_____ (___%) n_____ (___%) Black n_____ (___%) n_____ (___%) Hispanic n_____ (___%) n_____ (___%) Wh ite n_____ (___%) n_____ (___%) Two or more races n_____ (___%) n_____ (___%) **Region** Northeast n_____ (___%) n_____ (___%) Southeast n_____ (___%) n_____ (___%) Midwest n_____ (___%) n_____ (___%) Southwest n_____ (___%) n_____ (___%) West n_____ (___%) n_____ (___%) **Grade Level** Before Kindergarten n_____ (___%) n_____ (___%) K-5 n_____ (___%) n_____ (___%) 6-8 n_____ (___%) n_____ (___%) 9- 12 n_____ (___%) n_____ (___%) **School Ranking** Student attending Priority Schools/ Priority Schools n_____ (___%) n_____ (___%) Students attending Focus Schools/Focus Schools n_____ (___%) n_____ (___%) **Other i3 Priority Groups** English Language Learner n_____ (___%) n_____ (___%) Free/Reduced - Price Lunch n_____ (___%) n_____ (___%) **Overall Number n_____**  **n_____**

Please indicate the level at which the above data are reported: 

### Please insert a revised description here if needed.National Evaluation of i3 (NEi3) Data Collection Survey Page 10 **Full Population Served**  **by this Grant’s i3**  **- Funded Intervention**  **Sample of the Population Included**  **in this Study’s Impact Estimates**  student- level school - levelNational Evaluation of i3 (NEi3) Data Collection Survey Page 11



## Please **com**

**plete a row for each of the key components of the i3 intervention that is included in your logic model**  **and fidelity**  **me asurem**  **ent system** **. The**  **following instructions**  **define the**  **data that**  **should be**  **entered in**  **each of**  **the colu mn** **s. Column**  **Column Header Instructions for completing corresponding cells**  A **Key Component** Please enter the name of the key component
. Key components are the activities and inputs that are under the direct control of the individual or organization responsible for program implementation (e.g., program developer, grant recipient), and are considered by the developer to be essential for implementing the intervention. Key components may include financial resources, professional development for teachers, curricular materials, or technology products.
B **Indicators Characterizing Key Component** Using your study’s logic model, design summary, and fidelity matrix as a guide (whenever applicable), please provide the definition of each key component, i.e., a list of the indicators that represent each key component. Indicators should be quantified where possible. Depending upon how you defined each key component, indicators might encompass a combination of activities at the teacher, classroom, school, or sample level. **Example:** Fidelity of implementation for the key component “professional development” is based on four professional development indicators: a one -week summer institute, five webinars, two coaching sessions, and ongoing PD.

C **Year 1: Component Level Threshold for F**  **idelity of Implementation for the Unit that is the Basis for the Sample** **- Level**  **Representation of Fidelity (i.e. definition of implementation with fidelity)**  Please provide the definition of implementation with fidelity for the unit that is the basis for the sample-level representation of fidelity in Year 1 for each key component; i.e., the component level threshold that delineates implementation with fidelity from implementation without fidelity for a particular sample unit. **Example of**  **unit**

**- level** : If the sample-level fidelity threshold is defined as the percent of schools that have received an adequate score on a key component, the unit that is the basis for the sample -level representation of fidelity is “school”. The definition of implementation with fidelity could be: “a single school will be considered to have implemented the key component with fidelity if at least 90% of teachers at the school: attended the oneweek summer institute, attended four of five webinars, attended two of two coaching sessions, and received ongoing PD as needed (as self -reported in teacher survey)."National Evaluation of i3 (NEi3) Data Collection Survey Page 12

**Column Column Header**  **Instructions for completing corresponding cells**

D **Year 1: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation with**

**fidelity)** Please provide the definition of implementation with fidelity at the sample-level in Year 1 for each key component; i.e., the component level **threshold** that delineates implementation with fidelity from implementation without fidelity across sample units. **Examples of**  **sample -** **level:** “At least 80%of schools implement with fidelity” or “60%or more of districts implement with fidelity”. The sample-level threshold (e.g., across schools) should be a summary measure of fidelity score ranges and thresholds established at level of the unit that is the basis for the sample-level representation of fidelity (e.g., teacher, classroom, school). E **Year 1: Component Level Fidelity Score for**

**the Entire Sample** Please report the component level fidelity scores achieved in Year 1 of the intervention for each key component. **Examples:** “85%of schools implemented with fidelity” or “70% of districts implemented with fidelity” F **Year 1: Component Level Implementati on**  **with Fidelity at the Sample Level? (Yes/No)** Note: This field is automatically calculated based on the two prior fields.
G **Year 2: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity) Same instructions as in column C will appear in survey instrument, replacing year one with year two.** 

H **Year 2: Component Level Threshold for Fidelity of Implementation at the Sam**  **ple Level (i.e. definition of implementation with** 

**fidelity) Same instructions as in column D will appear in survey instrument, replacing year one with year two.** 

I **Year 2: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column**  **E will appear in survey instrument, replacing year one with year two.** J **Year 2: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column F will appear in survey instrument, replacing year one with year two.**
K **Year 3: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity) Same instructions as in column C will appear in survey**  **instrument, replacing year one with year three.**

L **Year 3: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation with**

**fidelity) Same instructions as in column D will appear in survey instrument, rep** **lacing year one with year three.**National Evaluation of i3 (NEi3) Data Collection Survey Page 13

**Column Column Header**  **Instructions for completing corresponding cells** M **Year 3: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column E will appear in survey instrument, replacing year one with year three.**  N **Year 3: Component Level Implementation with Fidelity**  **at the Sample Level? (Yes/No) Same instructions as in column F will appear in survey instrument, replacing year one with year three.** 
O **Year 4: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity) Same instructions as in column C will appear in survey instrument, replacing year one with year four.** 

P **Year 4: Component Level Threshold for Fidelity of Implementation at the Sa** **mple Level (i.e. definition of implementation with** 

**fidelity) Same instructions as in column D will appear in survey instrument, replacing year one with year four.** 
Q **Year 4: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in colum** **n E will appear in survey instrument, replacing year one with year four.** R **Year 4: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column F will appear in survey instrument, replacing year one with year four.** 
S **Year 5: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity) Same instructions as in column C will appear in surve** **y instrument, replacing year one with year five.**

T **Year 5: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation with**

**fidelity) Same instructions as in column D will appear in survey instrument,**  **replacing year one with year five.**
U **Year 5: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column E will appear in survey instrument, replacing year one with year five.**  V **Year 5: Component Level Implementation with Fidelity**  **at the Sample Level? (Yes/No) Same instructions as in column F will appear in survey instrument, replacing year one with year five.** National Evaluation of i3 (NEi3) Data Collection Survey Page 14



## Please **com**

**plete a**  **row for**  **ea ch**  **contrast me** **asured by**  **your impact**  **study .**  **The following**  **instructions define**  **the data that should be entered in each of the colum**  **ns. Column**  **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms Contrast Description** 
A **Contrast Name** Please enter the names of each of the evaluation’s confirmatory and exploratory contrasts. This field does not affect the assessment or reporting of the contrast; the contrast name simply facilities communication between the evaluator and the NEi3 AR Team, and facilitates referencing contrasts between the data -entry fields of this survey. The contrast name should uniquely identify the contrast and differentiate among similar contrasts: e.g. “Math Achievement-confirmatory” and “Math Achievement-cohort 1” only might be the names of two similar contrasts. B **Confirmatory or** 

**Exploratory?** Indicate whether this contrast is confirmatory or exploratory. P lease note that

any pre-specified contrasts that were registered as confirmatory will be prepopulated as such and cannot be changed to be exploratory. All other contrasts will be listed as exploratory. **Contrast Description: Key features that define a contrast**
C **Outcome Domain** For confirmatory contrasts, this field is populated with the registered domain name of your pre-specified confirmatory contrast. As the domain is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please enter the name of the domains measured. If the exploratory contrast measures the same construct as one of the confirmatory contrasts, please use the same domain name.National Evaluation of i3 (NEi3) Data Collection Survey Page 15



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

D **Treatment Condition**  For confirmatory contrasts, this field is populated with the registered description of the intervention tested by the pre-specified confirmatory contrast. As the treatment condition is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please name the intervention(s) experienced by the treatment group in the study. Do not use acronyms. If the condition is the same as the one tested by a confirmatory contrast, please use the same name. If the contrast involves comparing one treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group and the comparison group.

E **Comparison Condition** For confirmatory contrasts, this field is populated with the registered description of the counterfactual condition in the pre-specified confirmatory contrast. As the treatment condition is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please name the intervention(s) experienced by the comparison group in the study. This may be the name of another intervention or Business as Usual, if no specific intervention. Do not use acronyms. If the condition is the same as the one tested by a confirmatory contrast, please use the same name. If the contrast involves comparing one treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group and the comparison group
F **Educational Level** For confirmatory contrasts, this field is populated with the educational level that was registered in the pre-specified confirmatory contrast. As the educational level is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please report the educational level (e.g. elementary, middle or high school) of the students that experience the intervention. For teacher outcomes, report the educational level of the students in the teachers’ classrooms.National Evaluation of i3 (NEi3) Data Collection Survey Page 16



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms Contrast Description: Approach to addressing contrast** 

G **Design** Please select one of the following designs: RCT, RDD, QED, Comparison Group Design Without a Pretest, ITS, Pre-post. For confirmatory contrasts, this field is populated with the registered design for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop- up box that will appear. Definitions: Randomized controlled trial (RCT). U nder this design, an evaluation uses a random process to assign study units to an intervention groups and to a control group, and measures the effect of the intervention by comparing the outcomes of units assigned to th e two groups. Regression discontinuity design (RDD). U nder this design, an evaluation would use a continuous “scoring” rule to assign study units to the intervention. Units with scores above a pre-set cutoff value are assigned to the treatment group and units with scores below the cutoff value are assigned to the comparison group, or vice versa. The evaluation then compares the outcomes of the two groups close to the cutoff to measure the effect of the intervention. Quasi- Experimental Design (QED). A QED compares outcomes for intervention participants with outcomes for a comparison group chosen through methods other than randomization (RCT) or a scoring rule with a well-defined threshold (RDD) and includes one or more pre-intervention measures. Compa rative Interrupted Time Series designs are treated as QEDs by the NEi3. Comparison Group Design Without a Pretest. A comparison group design without a pre-test compares outcomes for intervention participants with outcomes for a comparison group chosen th rough methods other than randomization (RCT) or a scoring rule with a well-defined threshold (RDD) and does not include preintervention observations. Interrupted Time Series Design (ITS). An ITS design uses observations at multiple points in time both before and after the intervention was implemented,National Evaluation of i3 (NEi3) Data Collection Survey Page 17



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** and compares the outcomes after the intervention with the trend established prior to the intervention. These designs do not use a comparison group. Pre-post Design. A

p re-post design compares outcomes for intervention participants after receiving the intervention to their outcomes before receiving the intervention. These designs do not use a comparison group. H **Unit of Assignment** For each contrast, please report **the unit**  **of assignment**  **or the**  **level at**  **which assignment**  **to the**  **treatment and**  **comparison conditions**  **occur (e.g.,**  **student, teacher, classroom, or school).** If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3

– please explain why this change was made using the pop-up box that will appear.

I **Intervention Group**

**[alternately, “Treatment Group”]: Description**  For confirmatory contrasts, this cell is prepopulated with the registered description of the intervention group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **which schools,**  **teachers and/or**  **students will**  **be included**  **in the intervention (“treatm** **ent”) group** for this analysis. **Example 1**  **(Candy Valley):**  “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the intervention.” **Example 2**  **(Apple Valley):**  “This contrast pools data across all treated cohorts and schools. All students in treated grades in Fuji Elementary (K -2), Granny Smith Elementary (K

- 1) and Red Delicious Elementary (K -2) who were present for data collection in Spring 2014 are included in the treatment group.” J **Age/Grade during the**

**Intervention** For confirmatory contrasts, this cell is prepopulated with the registered

description of the age/grade of the students in the treatment group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop- up box that will appear.National Evaluation of i3 (NEi3) Data Collection Survey Page 18



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **the age**  **or grade** **- level**  **of students**  **in the**  **intervention group**  **during their exposure to the intervention** . For teacher outcomes, please enter the grade(s) taught by the teachers. **Example 1 (Candy Valley):** “The intervention was fielded to first grade students.” **Example 2**  **(Apple Valley):**  “Both Fuji and Red Delicious Elementary Schools introduced the intervention in the 2011 -2012 school year to all kindergarten students, offered the intervention to all kindergarten and first graders in 2012- 2013 and to kindergarten through second graders in 2013-2014. Granny Smith Elementary introduced the intervention in the 2012-2013 school year to all kindergarten students and offered the intervention to all kindergarten and first graders in 2013-2014.”

K **Length of Student**

**Exposure to Intervention** For confirmatory contrasts, this cell is prepopulated with the registered description of the age/grade of the students in the treatment group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop- up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **how long**  **(no. of**  **semesters/years) the**  **students in**  **the i**  **ntervention group**  **were engaged**  **in the**  **intervention at**  **the time**  **the outcome**  **was measured**  **for this**  **contrast** . If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. **Example 1 (Candy Valley):** “Students received one year of treatment” **Example 2**  **(Apple Valley):**  “When outcomes were measured in Spring 2014, both Fuji and Red Delicious Elementary Schools students had the following pattern of exposure to treatment: second grade students had three years of treatment, first grade students had two years of treatment and kindergarten students had one year of treatment. When outcomes were measured in Spring 2014, Granny Smith Elementary first graders had two years of treatment and kindergarten students had one year of treatment.”National Evaluation of i3 (NEi3) Data Collection Survey Page 19



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

L **Comparison Group:**

**Description** For confirmatory contrasts, this cell is prepopulated with the registered

description of the comparison group for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **which schools,**  **teachers and/or**  **students will**  **be included**  **in the comparison group**  for this analysis. **Example 1**  **(Candy Valley):**  “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the control group.” **Example 2**  **(Apple Valley):**  “This contrast pools data across all cohorts and schools matched to treated cohorts and schools. K -2 students from Honey Crisp Elementary (comparison for Fuji Elementary), K - 1 students from Ginger Gold Elementary serves (comparison for Granny Smith Elementary), and K -2 students Jonagold Elementary (comparison for Red Delicious Elementary) who were present for data collection in Spring 2014 are included in the treatment group.”

M **Outcome or Instrument**

**Name (full names, not acronyms)**  For confirmatory contrasts, this cell is prepopulated with the registered outcome

measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the full name of the outcome measure, including the name of the subtest or subscale if applicable. If the instrument for the outcome measure is not used in one of the confirmatory contrasts, please describe (1) what the instrument is designed to measure and (2) how the outcome measure is constructed.National Evaluation of i3 (NEi3) Data Collection Survey Page 20



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

N **Construction of Outcome**

**Measure (binary, ordinal, or**  **continuous)** For confirmatory contrasts, this cell is prepopulated with the registered scale of

the outcome measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please indicate whether the outcome measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous.

O **Level at which Outcomes**

**are Observed** For confirmatory contrasts, this cell is prepopulated with the registered level at

which outcomes are observed, called level of analysis in some NEi3 correspondence. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the level at which outcomes are observed in your d ata (e.g. student, teacher, classroom, school, district). **Example:** If your data are school-level averages of student test scores, enter “School ”.
P **Timing of Outcome** 

**Measure (year, Fall/Spring) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report when during the calendar year the **outcome** was **measured**
. Q **Timing of Outcome** 

**Measure (age/grade) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the age or grade level for which the **outcome** was **measur ed** 
.National Evaluation of i3 (NEi3) Data Collection Survey Page 21



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**
R **Timing of Outcome** 

**Measure (year, Fall/Spring) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report when during the calendar year the **outcome** was **measured**
. S **Timing of Outcome** 

**Measure (age/grade) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the age or grade level for which the **outcome** was **measured**
.

T **Key Baseline**  **Measures** For confirmatory contrasts, this cell is prepopulated with the registered baseline measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the baseline variable(s) on which you are establishing baseline equivalence. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes:If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”

U **Construction of Baseline**

**Measure (binary, ordinal, or continuous)**  For confirmatory contrasts, this cell is prepopulated with the registered scale of

the baseline measure for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please indicate whether the baseline measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes: If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”National Evaluation of i3 (NEi3) Data Collection Survey Page 22



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

V **Level of at which the** 

**Baseline Measure is**  **Observed** For confirmatory contrasts, this cell is prepopulated with the registered level of

observation for the baseline measure for the contrast. If you edit the prepopulated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the level at which baseline measures are observed in your data (e.g. student, teacher, classroom, school, district). **Example:** If your data are school-level averages of student test scores, enter “School ”. (Field is not appl icable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes: If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”

W **Timing of Baseline**

**Measure (year,**  **Fall/Spring,) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when during**  **the calendar**  **year the**  **data you**  **will use**  **to test**  **baseline equivalence was measured**
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes: If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”

X **Timing of Baseline**

**Measure (age/grade) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the age or grade level **for which**  **the data**  **you will**  **use to**  **test baseline equivalence was**  **measured**
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes: If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”National Evaluation of i3 (NEi3) Data Collection Survey Page 23



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

Y **Timing of Baseline**

**Measure (year, Fall/Spring) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when during**  **the calendar**  **year the**  **data you**  **will use**  **to test**  **baseline equivalence was measured**
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes: If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”

Z **Timing of Baseline**

**Measure (age/grade) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field – which suggests a change to the study plan registered with the NEi3 – please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when the**  **data you**  **will use**  **to test**  **baseline equivalence**  **was measured** 
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming notes:. If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”

**Confounds** AA **Treatment Condition**  **-**

**Other Activities** To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered to the treatment group but not the comparison group? If so, please describe the other intervention.
AB **Comparison Condition -** 

**Other Activities** To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered to the comparison group but not the treatment group? If so, please describe the other intervention. AC **Fundamental Confound at**

**the District Level: Treatment Condition**  Does the **treatment condition**  include schools/teachers/students from a **single school district**  which does not also include comparison group members?National Evaluation of i3 (NEi3) Data Collection Survey Page 24



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** AD **Fundamental Confound at**

**the District Level:**  **Comparison Condition** Does the **comparison condition**  include schools/teachers/students from a **single school district** which does not also include treatment group members? AE **Fundamental Confound at**

**the School Level: Treatment Condition**  Does the **treatment condition**  include teachers/students from a **single school**  which does not also include comparison group members? AF **Fundamental Confound at**

**the School Level: Comparison Condition**  Does the **comparison condition**  include teachers/students from a **single school**  which does not also include treatment group members? **Outcomes** AG **Outcome or Instrument**

**Name (full names, not acronyms)**  This field displays the outcome measure you entered previously. This field is not

editable. If you wish to edit this entry, go to <reference location>. AH **Who Provided the**

**Outcome Data?** For each contrast, please report the **name of**  **the entity**  **that provided**  **the**

**outcome data.**

AI **Value and Type of**

**Reliability Measure of Instrument**  **for Outcome Data**  Please provide a narrative description of the outcome and discuss the reliability

and validity of the measure, including **the value and type of reliability measure**
. For example, you may report an internal consistency, test-retest reliability, or inter-rater reliability measure. For outcomes for which reliability statistics cannot be obtained, please describe why reliability statistics are not available. **Example 1:**  “The Orange Gulch District writing assessment was scored by trained reviewers according to a holistic rubric. A random subset of essays was double scored and demonstrated an inter -rater reliability of 89%.” **Example 2:**  “Attendance is the proportion of school days for which an individual student was present for at least one course at Orange Gulch High School. Reliability statistics cannot be calculated for attendance. However, it is a face valid measure of the pre-specified domain: High School Attendance.”National Evaluation of i3 (NEi3) Data Collection Survey Page 25



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

AJ **Consistency of Definition** Please describe any differences in the definition of the outcome measure between the groups. To be consistently defined, an outcome measure must be defined using the same rules or procedures in both groups. Standardized tests (e.g., the Terra Nova or Stanford 10) and state-required tests are assumed to be consistently defined. For other outcomes (e.g., measures of grades or GPA, course credits, and graduation), the evaluator should provide an assessment of whether the outcome is consistently defined in all study groups.
AK **Consistency of Collection** Please describe any differences in the process for collecting data on the outcome measure between the groups. To be consistently collected, an outcome measure must be collected using the same rules or procedures in both groups. For example, data would not be consistently collected if treatment students take the post-test in January, but control students take the test in March – two months later. AL **Treatment of Missing Data** Describe how the study addressed missing values for the outcome measure. **Assessing Attrition** AM **Level of Assignment** This field displays the unit of assignment you entered previously. This field is not editable. If you wish to edit this entry, go to <reference location>. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not RCT.

AN **Level at which Outcomes**

**are Observed** This field displays the level at which outcomes are observed entered previously.

This field is not editable. If you wish to edit this entry, go to <reference location>. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not RCT.

AO **Cluster Randomized**

**Design?** Select “Yes” if the contrast is from a cluster randomized design.

Select “No”, otherwise For NEi3, all **cluster randomized**  **designs** are those in which clusters (of individuals) were assigned by a process that was functionally random. These clusters may be associated with, for instance, particular teachers, classrooms, schools, or districts. Programming note: Display “Not Applicable” if the Design is not RCT.National Evaluation of i3 (NEi3) Data Collection Survey Page 26



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** (Section is not applicable to the followi ng designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)

AP **Analytic Sample Includes**

**Joiners?** Select “not applicable” if your design is not a cluster randomized design.

Select “yes” if your analytic sample includes individuals who were not present in the cluster at the time of random assignment. Select “no” if your analytic sample is restricted to individuals who were present at the time of random assignment. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is NOT a cluster randomized design.

AQ **Unit of Assignment**

**Baseline Sample: Intervention Group**  Enter the number of clusters randomly assigned to the intervention group. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is (a) NOT a cluster randomized design OR (b) the level of observation is the same as the level of ass ignment

AR **Unit of Assignment**

**Baseline Sample: Comparison Group**  Enter the number of clusters randomly assigned to the comparison group. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT.D isplay “Not applicable” if Design is not RCT. Display “Not applicable” if the contrast is (a) NOT a cluster randomized design OR (b) the level of observation is the same as the level of assignment
AS **Unit of Assignment**

**Analytic Sample: Intervention Group**  Enter the number of clusters for the intervention group for the analytic sample. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is (a) NOT a cluster randomized design OR (b) the level of observation is the same as the level of assignmentNational Evaluation of i3 (NEi3) Data Collection Survey Page 27



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms**

AT **Unit of Assignment**

**Analytic Sample: Comparison Group**  Enter the number of clusters for the comparison group for the analytic sample. Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if Design is not RCT. Display “Not applicable” if the contrast is (a) NOT a cluster randomized design OR (b) the level of observation is the same as the level of assignment

AU **Unit of Analysis Baseline**

**Sample: Intervention Group**  Enter the sample size for the intervention group at the time of random

assignment Complete this field the contrast is from: (a)a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners (b)any randomized design where the unit of analysis is the same as the unit of assignment (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is a cluster randomized design AND includes joiners AND the level of observation does not equal the level of assignment

AV **Unit of Analysis Baseline**

**Sample: Comparison**  **Group** Enter the sample size for the comparison group at the time of random

assignment Complete this field the contrast is from: (c)a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners (d)any randomized design where the unit of analysis is the same as the unit of assignment (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is a cluster randomized design AND includes joiners AND the level of observation does not equal the level of assignment

AW **Unit of Analysis Analytic**

**Sample: Intervention**  **Group** Enter the sample size for the intervention group for the analytic sample

Complete this field the contrast is from: (e)a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners (f)any randomized design where the unit of analysis is the same as the unit of assignment Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is a cluster randomized design AND includes joiners AND the level of observation does not equal the level of assignmentNational Evaluation of i3 (NEi3) Data Collection Survey Page 28



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)

AX **Unit of Analysis Analytic**

**Sample: Comparison**  **Group** Enter the sample size for the comparison group for the analytic sample

Complete this field the contrast is from: (g)a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners (h)any randomized design where the unit of analysis is the same as the unit of assignment (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Programming Note: Display “Not Applicable” if the Design is not RCT. Display “Not applicable” if the contrast is a cluster randomized design AND includes joiners AND the level of observation does not equal the level of assignment
**Baseline Measurement**  AY **Key Baseline**  **Measures** This field displays the key baseline measure(s) you entered previously. This field is not editable. If you wish to edit this entry, go to <reference location>. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.

AZ **Construction of Baseline**

**Measure (binary, ordinal, or continuous)**  This field displays the description of the construction of the baseline measure you

entered previously. This field is not editable. If you wish to edit this entry, go to <reference location>. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.

BA **Level of Observation for**

**Baseline Equivalence**  This field displays the level of observation for the baseline measure you entered previously. This field is not editable. If you wish to edit this entry, go to <reference location>. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.

BB **Treatment of Missing Data**

**for Baseline Balance Testing**  Describe how the study addressed missing values for the baseline measure for

baseline balance testing. Programming note: Display “Not Applicable” if the Design is not an RCT or QED.National Evaluation of i3 (NEi3) Data Collection Survey Page 29



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** BC **Narrative Description of**

**Sample for Baseline Balance Testing**  Describe the eligibility criteria for inclusion in the sample for baseline balance

testing. If balance is established using cluster-level averages (e.g. school-level average student test score), describe which individual units are included in the average. Programming note: Display “Not Applicable” if the Design is not an RCT or QED.

BD **Type of Information Used**

**to Assess Baseline Equivalence**  Select one option that best represe nts the information generated by your

baseline balance testing:  Unadjusted Mean and Standard Deviation T-statistic  Dichotomous This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable” (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.

BE **Baseline Balance Testing** 

**Sample Size: Intervention Group**  Enter the sample size for the intervention group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.
BF **Mean of**

**Baseline Measure: Intervention Group**  Enter the pre-intervention mean for the intervention group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED. Display “Not applicable” if the type of information used to assess baseline equivalence is “T-statistic” BG **Standard Deviation of**

**Baseline Measure: Intervention Group**  Enter the pre-intervention standard deviation for the intervention group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED. D isplay “Not applicable” if the type of information used to assess baseline equivalence is “T-statistic” BH **Baseline Balance Testing**

**Sample Size: Comparison Group**  Enter the sample size for the comparison group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED.National Evaluation of i3 (NEi3) Data Collection Survey Page 30



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** BI **Mean of**

**Baseline Measure: Comparison Group**  Enter the pre-intervention mean for the comparison group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED. D isplay “Not applicable” if the type of information used to assess baseline equivalence is “T-statistic” BJ **Standard Deviation of**

**Baseline Measure: Comparison Group**  Enter the pre-intervention standard deviation for the comparison group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED. Display “Not applicable” if the type of information used to assess baseline equivalence is “T-statistic” BK **T -** **Statistic for Baseline** 

**Balance Test** Enter the t-statistic for the pre-intervention difference in means between the

treatment and the comparison (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Programming note: Display “Not Applicable” if the Design is not an RCT or QED. Display “Not applicable” if the type of information used to assess baseline equivalence is “T-statistic” **Reported Findings**

BL **Type of Findings Reported** Select one of the following types of data to use to report findings:  Unadjusted means Adjusted means, e.g. ANOVA T-test Coefficient from a linear regression model, e.g. OLS, HLM, etc. Coefficient from a non-linear regression model, e.g. logit or probit This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable” BM **Adjusted for Baseline/Pre******

**Test Measure? (If applicable)**  Please describe **whether and how your analysis accounted for pre** **- test** 

**measures** . Programming Note: Display “Not Applicable” if type of findings is “Unadjusted means” BN **Adjustment for Clustering**

**Used in Analysis?** Please indicate whether or not the analysis accounted for the clustering of

sample units in larger units, and if applicable describe the type of method used to adjust for this clustering (e.g., multi-level modeling or robust standard errors). Programming Note: Display “Not Applicable” if type of findings is “Unadjusted means” BO **Intervention Group:**

**Sample Size** Please report the size of the intervention group sample for the analysis. BP **Intervention Group:**

**Number of Clusters in Analytic Sample**  Please report the **number of clusters assigned to the intervention group that**

**appear in the analysis sample**
.National Evaluation of i3 (NEi3) Data Collection Survey Page 31



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned. BQ **Comparison Group:**

**Sample Size** Please report the size of the comparison group sample for the analysis.

BR **Comparison Group:**

**Number of Clusters in Analytic Sample**  Please report the number of clusters assigned to the comparison group that

appear in the analysis sample. Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned. Programming Note: Display “Not applicable” if the level of observation is the same as the level of assignment

BS **Intraclass Correlation**

**Coefficient** Please report the intraclass correlation coefficient to describe the similarity of

units at the level of observation within the cluster at the level of assignment Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment, e.g. the correlation between outcomes of students within the same school, if schools were assigned. Programming Note: Display “Not applicable” if the level of observation is the same as the level of assignment

BT **Unadjusted Outcome**

**Mean for the Intervention**  **Group** Please provide the unadjusted mean of the outcome measure in the intervention

group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) Programming Note: Display “Not Applicable” if type of findings is NOT “Unadjusted means”

BU **Adjusted Outcome Mean**

**for the Intervention Group** Please provide the adjusted mean of the outcome measure in the intervention group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) Programming Note: Display “Not Applicable” if type of findings is NOT “Adjusted means”
BV **Unadjusted Standard**

**Deviation in the Outcome Measure of the**  **Intervention Group** Please provide the unadjusted standard deviation of the outcome measure in the intervention group.

BW **Unadjusted Outcome**

**Mean for the Comparison Group**  Please provide the unadjusted mean of the outcome measure in the comparison

group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) Programming Note: Display “Not Applicable” if type of findings is NOT “Unadjusted means”National Evaluation of i3 (NEi3) Data Collection Survey Page 32



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming**  **notes about skip** 

**patterms** BX **Adjusted Outcome Mean**

**for the Comparison Group** Please provide the unadjusted mean of the outcome measure in the comparison group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) Programming Note: Display “Not Applicable” if type of findings is NOT “Adjusted means”

BY **Unadjusted Standard**

**Deviation in the Outcome Measure of the**  **Comparison Group** Please provide the unadjusted standard deviation of the outcome measure in the comparison group.

BZ **Impact Estimate (e.g.,**

**coefficient from OLS or HLM)**  Please report the **regression coefficient** that estimates of the effect of the

intervention . If your impact estimate is the sum of multiple regression coefficients, please enter this sum here. (You do not need to complete this field if it is automatically populated with “Not applicable”.) Programming Note: Display “Not applicable” if the type of findings is NOT a coefficient from a linear or non -linear regression model.

CA **Standard Error of Impact**

**Estimate** Please insert the **estimated standard error of the regression coefficient** here.

If your impact estimate is the sum of multiple coefficients, please enter the standard deviation of the summed variable. (You do not need to complete this field if it is automatically populated with “Not applicable”.) Programming Note: Display “Not applicable” if the type of findings is NOT a coefficient from a linear or non -linear regression model.

CB **Two -** **tailed p -** **value** Please insert the associated **p -** **value that has not been adjusted for multiple**  **comparisons** here. If your impact estimate is the sum of multiple coefficients, please enter the associated p-value here. (You do not need to complete this field if it is automatically populated with “Not applicable”.) Programming Note: Display “Not applicable” if the type of findings is NOT a coeffi cient from a linear or non-linear regression model.
CC **Degrees of Freedom** Please enter the degrees of freedom of your statistical analysis. (You do not need to complete this field if it is automatically populated with “Not applicable”.) Programming Note: Display “Not applicable” if the type of findings is “unadjusted means”.
CD **Number of Time Periods in** 

**the Model (ITS or CITS)** For each contrast, please indicate the number of time periods in the model. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you may report this information if it is available and you wish to do so) Programming Note: Display “Not applicable” if the design is NOT “ITS”