# Appendix A: **National Evaluation of i3 (NEi3) Data Collection Survey** 

This work is supported by the U.S. Department of Education under contract number ED- IES - 10 -C-0064, ED- IES - 13-C-0005 and/or EDIES - 14 -C-0007, Abt Associates, Prime Contractor.National Evaluation of i3 (NEi3) Data Collection Survey Page 2



### Welcome **[ User_Name** **] !**  The National Evaluation of Investing in Innovation (NEi3) is being conducted by Abt Associates Inc. on behalf of the U.S. Department of Education (ED) Institute of Education Sciences (IES). A primary goal of the NEi3 is to assess the strength of evidence generated by independent local evaluations of i3 grants and to provide summaries of evaluation findings. The purpose of this data collection is to collect information from evaluators like you regarding the findings of your i3 independent evaluation. We look forward to working with you on this important endeavor! Abt Associates and its subcontractors follow the confidentiality and data protection requirements of IES (The Education Sciences Refo rm Act of 2002, Title I,Part E, Section 183), which require that all collection, maintenance, use and wide dissemination of data conform to the requirements of the Privacy Act of 1974 (5 U.S.C. 552a), the Family Educational Rights and Privacy Act of 1974 (20 U.S.C. 1232g), and the Protection of Pupil Rights Amendment (20 U.S.C. 1232h). The study team will not be collecting any individually identifiable information. All of the data requested from independent evaluators will be in the form of aggregated repo rts of the methods, measures, plans, and results in their independent evaluations. Responses to this data collection will be used to disseminate information about your study findings. Individual i3 grants will be identified in this study. The characteristics, results, and findings reported by independent evaluators, as well as assessments of the quality of the independent evaluations and i3 projects, may potentially be reported. This study cannot, however, associate responses with a specific sc hool or individual participating in i3 independent evaluations, since we are not collecting that information. Grantees and their independent evaluators are required to comply with the requirements of the NEi3. Participation in at least one of the NEi3’s data collection periods is required. Your response is critical for producing valid and reliable data. You may skip any questions you do not wish to answer; however, we hope that you answer as many questions as you can. Your answers to questions wil l not affect your grant

— now or in the future. Participation in the NEi3 Findings Survey will not impose any risks to you as a respondent. If you have any questions about your rights as a research participant, you can contact Teresa Doksum, the Chairperson of Abt’s IRB at 617
-349
-2896 or by email at irb@abtassoc.com
. This survey is authorized by law: American Recovery and Reinvestment Act of 2009 (ARRA), Section 14007, Title XIV (Public Law (P.L.) 111
-5
. We estimate that the survey will take you approximately 1 5 hours to complete
. **OM B Clearance Number:**  **xxxx -** **xxxx Expiration Date:**  **xx/xx/xxxx** According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is **xxxx -** **xxxx** . The time required to complete this information collection is estimated to average approximately 1 5 hours per respondent, in cluding the time to review instructions, gather the data needed, and complete and review the information collected. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202. If you have comments or concerns regarding the status of your individual submission of this form, write direc tly to: U.S. Department of Education, Institute of Education Sciences, 555 New Jersey Avenue, N.W., Washington, DC 20208.National Evaluation of i3 (NEi3) Data Collection Survey Page 3



### Overview**of the NEi3 Data Collection Survey** Throughout the data collection survey, the NEi3 study team has pre
-populated information available from the following sources:  your analysis plan  your contrast tool  reports or other documents on interim or final results Please review this information carefully to ensure its accuracy (you’ll be asked to actively confirm the accuracy of all pre-populated information). We realize that it will likely take you a few visits to this website to complete entering your findings into this data collection survey. Therefore, you may log into and out of the survey as many times as necessary to complete the entire data collection survey by MM/DD/YYYY. The information you enter into this survey will be automatically saved once you navigate to a different survey section. The data collection website is organized into three main sections: 1. Background Section 2. Fidelity of Implementation Section 3. Impact Study Section After you enter the survey, you can access each of the above sections by clicking on the appropriate navigation button at the top of your screen. Below, we briefly describe the content of each of these sections. The **Background Section**  asks you about the parties involved in developing, implementing and evaluating the i3
-funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation. The **Fidelity of** **Implementation Section**  asks you to describe the key components of the i3
-funded intervention you evaluated, the measures constructed to evaluate fidelity of implementation for those components, and the findings from your implementation study. The **Impact Study**  **Section** asks you to describe each of your evaluation’s confirmatory contrasts. Specifically, the survey will ask you to provide information about the results from the analysis for each contrast, including the estimated effects of the intervention and the size of the analysis sample. In addition to your confirmatory contrasts, you are welcome to provide the details of any additional exploratory contrasts from this study that you would like the NEi3 to consider including in the NEi3 Final Report. Detailed guidance about how to enter information in each column is available by hovering your mouse on the pertinent column header. This section is divided up into the following nine subsections:  Contrast Description: Key Features  Contrast Description: Design  Contrast Description: Outcome Measurement  Contrast Description: Baseline Measures  Confounds  Outcomes  Assessing Attrition  Baseline Measurement  Reported Findings The same contrasts appear on each subsection, but different items appear on each subsection. So for each contrast, you’ll be asked to enter/update descriptive information, and information on confounds,National Evaluation of i3 (NEi3) Data Collection Survey Page 4



### outcomes, attrition, baseline equivalence, and findings. 

### Monitoring Progress and Submitting Your Results As you complete each section (and subsection) of the survey, you will check off the “Section Complete” or “Subsection Complete” checkboxes that appear in each section (and subsection). That will allow you to track your own progress. Once you have indicated your completion of each section, check off the “Survey Complete” checkbox at the top of the survey. That will automatically notify the NEi3 AR team that you have completed the survey and it is ready f or their review. 

### Getting Help For help with technical issues using this website, please call **617 -** **520 -** **2447**
. For help selecting appropriate data for entry and/or for further guidance on survey items, please call **301 -** **347 -** **5638**
. 

### Ac**cessing the Survey** For each study there is a corresponding data collection survey. If your grant has only one study, then only one survey will appear on the survey selection page. If your grant has multiple studies, you’ll be able to open the desired survey on the survey selection page.National Evaluation of i3 (NEi3) Data Collection Survey Page 5



## Background Information The following section asks you about the parties involved in developing, implementing and evaluating the i3
-funded intervention on which you worked. It also asks some general questions about the grant, intervention, and evaluation. 1. What is the name of the **grantee** for which you conducted/are conducting the evaluation?

___________________________________________________________ 2. What is the name of the **grant** under which you conducted/are conducting the evaluation?



### ________________________________________________________ 3. What is the i3 grant type of this grant?

Development Scale
-Up Validation 4. How many impact studies/evaluations are funded under this grant? ______________________
5. If there are multiple impact studies/evaluations funded under this grant, what is the name of the study reported in this survey?

____________________________________________________________________________ 6. What is/are the name(s) of the organization(s) that conducted/is conducting the impact study?

____________________________________________________________________________ 7. What is/are the name(s) of the principal investigator(s) that conducted/is conducting the study?

____________________________________________________________________________ 8. What is the name of the i3
-funded intervention?

____________________________________________________________________________National Evaluation of i3 (NEi3) Data Collection Survey Page 6



### 9. Please review the following summary description of the i3
-funded intervention, and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided. Please note that this description is intended for inclusion in the NEi3 Findings Report; however, the NEi3 may condense, or seek clarifications for, proposed revisions. **NEi3 Description:** This section will be pre
- populated by the NEi3 team with a summary of the intervention in sufficient detail to help readers understand what makes this intervention similar to or different from other interventions. The level of detail will be similar to what would be provided in an introductory section of a typical impact evaluation report. It will include the length of the intervention and the dosage, as well as information about the content, delivery, and implementation of the intervention.  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:**

10. What are the names of the individual(s) and organization(s) that developed the intervention? **Individual Organizational**  **affiliation**

11. What are the name(s) of the individuals and organization(s) that implemented and/or are implementing the intervention being evaluated under this grant? **Individual Organizational affiliation**  

### Please insert a revised description here if needed.National Evaluation of i3 (NEi3) Data Collection Survey Page 7



### 12. Did the following entities review the study findings prior to their release and/or submission to NEi3: **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: ____________________________ 

#   If you answered “yes” to any question above, please explain:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open
-ended response. 13. Did the following entities provide comments on the study findings prior to their release and/or submission to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain how comments were addressed:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open
-ended response.National Evaluation of i3 (NEi3) Data Collection Survey Page 8



### 14. For each entity below, did the entity request changes to the analysis and reporting prior to their release and/or submission to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain whether the requested changes were made, and why:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open
-ended response. 15. For each of the following entities, did the entity conduct any of the statistical analysis for the findings submitted to NEi3? **Yes No**  The individual/organization implementing the intervention (from question 11) 

#   The individual/organization developing the intervention (from question 10) 

#   Participating School District(s) 

#   Participating School Principal(s) 

#   Other, please specify: _________________________________ 

#   If you answered “yes” to any question above, please explain:

______________________________________________________________________________ Programming note: if “yes” is checked above, flag for NEi3 reviewer to discuss which contrasts are implicated if warranted based on open
-ended response.National Evaluation of i3 (NEi3) Data Collection Survey Page 9



### 16. Please review the following description of the population served by this grant’s i3
-funded intervention and indicate its accuracy by selecting the appropriate box below. If the description requires any revision, please insert a corrected description in the area provided. **NEi3 Description** : This section will be prepopulated with a description of the population served by this grant’s i3
-

funded intervention(s). The description will include the target age/grade, or teacher type (for example, fifth grade teacher, elementary school counselor, high school science teacher, etc.), of the intervention. It will also include the location of the intervention and a description of the type and number of schools served.  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:**

17. Please review the following description of the design of this study’s impact evaluation. If the description requires any revision, please insert a corrected description in the area provided
. **NEi3 Description** : This section will be prepopulated with a brief narrative description of key features of the study design. This description will include the design and level of assessment.  This description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  This description requires some revision. (If this is checked, please enter a revision below.) **Evaluator Revision:**

18. Please review the following description of how the study planned to select or assign treatment and comparison groups for the impact evaluation. Please update the description to reflect the selection or assignment process as implemented and to add details as needed. Information that needs to be added is indicated in capital letters. **NEi3 Description of** **Plans**: This section will be prepopulated with the descriptionfrom the Analysis Plan of the process by which treatment and comparison groups are assigned and/or identified. For RCTs, this description will include: how the sample for random assignment was selected (articulate all inclusion and exclusion criteria); the process for random assignment, and blocks or strata, and its timing; any changes to the analysis sample after the point of random assignment and reasons for sample loss or gain (e.g., identification of ineligible students; joiners); when clusters and individuals were first noti fied of random assignment; when consent was obtained. For non
-experimental designs, this description will include: how 

### Please insert a revised description here if needed.

**Please insert a revised descriptio n here if needed.** National Evaluation of i3 (NEi3) Data Collection Survey Page 10



### the treatment and comparison samples were selected (articulate all inclusion and exclusion criteria); if matching was used, the dimensions on which the groups will be matched and the procedures for matching; whether the treatment sample is comprised of individuals who volunteered for the intervention; whether the comparison sample is comprised of individuals who were not eligible for or actively declined to participate in the intervention; and any differences in eligibility criteria between the treatment and comparison groups. Ifkey information is not available in the Analysis Plan, the prepopulation will indicate this in capital letters. **Evaluator Description of Implementation**  **:**

19. Please review the following description of the sample eligibility and inclusion for this study’s impact evaluation. If the description requires any revision, please insert a corrected description in the area provided. **NEi3 Description** : [Describe how the districts/schools were chosen to be in the study (e.g. universe, random sample from a defined population, convenience sample , etc...). Please articulate all inclusion and exclusion criteria. For example, “The five largest school districts in XYZ County, Texas, were recruited to participate in this study because of their size, and because of the high percentage of ELL students in XYZ County. Within each of the five school districts, two to four elementary schools will be recruited and half will be randomly assigned to treatment and half will be assigned to control. The intervention is a school
-wide intervention. All recruited schools will be included in the impact evaluation. Outcomes will be assessed for students in grades three through five. The primary research question concerns the impact on school average student test scores after the intervention has been in schools for three years.” Finally, please ensure it is clear in the summary to whom the sample was designed to generalize.]  The intervention description is correct. (If this box is checked, the NEi3 Description that appears above will be used in the NEi3 Findings Report.)  The intervention description requires some revision. (If this box is checked, please enter a revision below.) **Evaluator Revision:**

20. For both the full population served by this grant’s i3
-funded intervention, and the sample of the population included in this study’s impact estimates, please provide the following statistics at the student
-level if available and at the school
-level if the student
-level is not available: 

### Please insert a revised description here.

**Please insert a revised description here if needed.**National Evaluation of i3 (NEi3) Data Collection Survey Page 11 **Full Population Served**  **by this Grant’s i3**  **- Funded**  **Intervention Sample of the Population**  **Included in this Study’s**  **Impact Estimates Location**  City n_____ (___%) n_____ (___%) Suburban n_____ (___%) n_____ (___%) Town n_____ (___%) n_____ (___%) Rural n_____ (___%) n_____ (___%) **Demographics** Am erican Ind/Alaskan n_____ (___%) n_____ (___%) Asian/Pacific Islander n_____ (___%) n_____ (___%) Black n_____ (___%) n_____ (___%) Hispanic n_____ (___%) n_____ (___%) Wh ite n_____ (___%) n_____ (___%) Two or more races n_____ (___%) n_____ (___%) **Region** Northeast n_____ (___%) n_____ (___%) Southeast n_____ (___%) n_____ (___%) Midwest n_____ (___%) n_____ (___%) Southwest n_____ (___%) n_____ (___%) West n_____ (___%) n_____ (___%) **Grade Level** Before Kindergarten n_____ (___%) n_____ (___%) K-5 n_____ (___%) n_____ (___%) 6-8 n_____ (___%) n_____ (___%) 9- 12 n_____ (___%) n_____ (___%) **School Ranking** Student attending Priority Schools/ Priority Schools n_____ (___%) n_____ (___%) Students attending Focus Schools/Focus Schools n_____ (___%) n_____ (___%) **Other i3 Priority Groups** English Language Learners n_____ (___%) n_____ (___%) Free/Reduced - Price Lunch n_____ (___%) n_____ (___%) **Overall Number n_____**  **n_____**

Please indicate the level at which the above data are reported: student- level school - level **School Characteristic** **s** Average school size n_____ /school n_____ /school Title I eligible school -wide ___ schools ___ schools Title I eligible targeted ___ schools ___ schools **Overall Number ___ schools**  **___ schools**National Evaluation of i3 (NEi3) Data Collection Survey Page 12



## Please **com**

**plete a row for each of the key components of the i3 intervention that is included in your logic model**  **and fidelity**  **me asurem**  **ent system** **. The**  **following instructions**  **define the**  **data that**  **should be**  **entered in**  **each of**  **the colu mn** **s. Column**  **Column Header Instructions for completing corresponding cells** 
A **Key Component** Please enter the name of the key component
. Key components are the activities and inputs that are under the direct control of the individual or organization responsible for program implementation (e.g., program developer, grant recipient), and are considered by the developer to be essential for implementing the intervention. Key components may include financial resources, professional development for teachers, curricular materials, or technology products.

B **Indicators Characterizing Key Component** Using your study’s logic model, design summary, and fidelity matrix as a guide (whenever applicable), please provide the definition of each key component, i.e., a list of the indicators that represent each key component. Indicators should be quantified where possible. Depending upon how you defined each key component, indicators might encompass a combination of activities at the teacher, classroom, school, or sample level. **Example:** Fidelity of implementation for the key component “professional developm ent” is based on four professional development indicators: a one -week summer institute, five webinars, two coaching sessions, and ongoing PD.

C **Year 1: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity)** Please provide the definition of implementation with fidelity for the unit that is the basis for the sample -level representation of fidelity in Year 1 for each key component ; i.e., the component level threshold that delineates implementation with fidelity from implementation without fidelity for a particular sample unit. **Example of**  **unit**

**- level** : If the sample-level fidelity threshold is defined as the percent of schools that have received an adequate score on a key component, the unit that is the basis for the sample -level representation of fidelity is “school”. The definition of implementation with fidelity could be: “a single school will be considered to have implemented the key component with fidelity if at least 90% of teachers at the school: attended the oneweek summer institute, attended four of five webinars, attended two of two coaching sessions, and received ongoing PD as needed (as self -reported in teacher survey)."National Evaluation of i3 (NEi3) Data Collection Survey Page 13

**Column Column Header**  **Instructions for completing corresponding cells** D **Year 1: Sample Size (of Units Measured at**

**the Sample Level)** Please enter the sample size (N) of units associated with the sample-level threshold and results. For example, if sample-level threshold is 80%or more schools are “high” implementers, please enter the number of schools.

E **Year 1: Representativeness of sample: Measured on all (A), some (S), or none (N)**  **of the members representing the intervention group in the impact** 

**evaluation** Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.

F **Year 1: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation** 

**with fidelity)** Please provide the definition of implementation with fidelity at the sample-level in Year 1 for each key component; i.e., the component level **threshold** that delineates implementation with fidelity from implementation without fidelity across sample units. **Examples of**  **sample -** **level:** “At least 80%of schools implement with fidelity” or “60%or more of districts implement with fidelity”. The sample-level threshold (e.g., across schools) should be a summary measure of fidelity score ranges and thresholds established at level of the unit that is the basis for the sample-level representation of fidelity (e.g., teacher, classroom, school). G **Year 1: Component Level Fidelity Score for**

**the Entire Sample** Please report the component level fidelity scores achieved in Year 1 of the intervention for each key component. **Examples:** “85%of schools implemented with fidelity” or “70% of districts implemented with fidelity” H **Year 1: Component Level Implementati on**  **with Fidelity at the Sample Level? (Yes/No)** Note: This field is automatically calculated based on the two prior fields.
I **Year 2: Component Level Threshold for Fidelity of Implementation for the Unit that**  **is the Basis for the Sample -**  **Level Representation of Fidelity (i.e. definition of**  **implementation with fidelity) Same instructions as in column C will appear in survey instrument, replacing year one with year two.** 

J **Year 2: Sample Size (of Units Measured at**

**the Sample Level Same**  **instructions as in column D will appear in survey instrument .** 

K **Year 2: Representativeness of sample: Measured on all (A), some (S), or none (N)**  **of the members representing the intervention group in the impact** 

**evaluation** Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all.National Evaluation of i3 (NEi3) Data Collection Survey Page 14

**Column Column Header**  **Instructions for completing corresponding cells** L **Year 2: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation** 

**with fidelity) Same instructions as in column**  **E will appear in survey instrument, replacing year one with year two.** 

M **Year 2: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column**  **F will appear in survey instrument, replacing year one with year two.**  N **Year 2: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column G**  **will appear in survey instrument.**
O **Year 3: Component Level Threshold for**  **Fidelity of Implementation for the Unit that is the Basis for the Sample** **- Level**  **Representation of Fidelity (i.e. definition of implementation with fidelity)**  **Same instructions as in column C will appear in survey instrument, replacing year one with year three.** 

P **Year 3: Sample Size (of Units Measured at**

**the Sample Level Same instructions as in column D will appear in survey instrument**  **.**

Q **Year 3:Representativeness of sample: Measured on all (A), some (S), or none (N)**  **of the members representing the intervention group in the impact** 

**evaluation** Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all. R **Year 3: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation** 

**with fidelity) Same instructions as in column**  **E will appear in survey instrument, replacing year one with year three.** 

S **Year 3: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column**  **F will appear in survey instrument, replacing year one with year three.**  T **Year 3: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column G**  **will appear in survey instrument.**
U **Year 4: Component Level Threshold for**  **Fidelity of Implementation for the Unit that is the Basis for the Sample** **- Level**  **Representation of Fidelity (i.e. definition of implementation with fidelity)**  **Same instructions as in column C will appear in survey instrument, replacing year one with year four.** 

V **Year 4: Sample Size (of Units Measured at**

**the Sample Level Same instructions as in column D will appear in survey instrument**  **.**National Evaluation of i3 (NEi3) Data Collection Survey Page 15

**Column Column Header**  **Instructions for completing corresponding cells**

W **Year 4: Representativeness of sample:**  **Measured on all (A), some (S), or none (N) of the members representing the**  **intervention group in the impact**

**evaluation** Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are included i n the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all. X **Year 4: Component Level Threshold for Fidelity of Implementation at the Sample**  **Level (i.e. definition of implementation**

**with fidelity) Same instructions as in column**  **E will appear in survey instrument, replacing year one with year four.** 

Y **Year 4: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column**  **F will appear in survey instrument, replacing year one with year four.**  Z **Year 4: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column G**  **will appear in survey instrument.**
AA **Year 5: Component Level Threshold for**  **Fidelity of Implementation for the Unit that is the Basis for the Sample** **- Level**  **Representation of Fidelity (i.e. definition of implementation with fidelity)**  **Same instructions as in column C will appear in survey instrument, replacing year one with year five.** 

AB **Year 5: Sample Size (of Units Measured at**

**the Sample Level Same instructions as in column D will appear in survey instrument**  **.**

AC **Year 5: Representativeness of sample: Measured on all (A), some (S), or none (N)**  **of the members representing the intervention group in the impact** 

**evaluation** Please enter “A”, “S”, or “N” for this key component. Example: If Kindergarten (K) teachers are in cluded in the measurement of this key component and the intervention group consists of those K teachers and teachers from grades 1 to 3, the evaluator would enter “S” for some. Example: If all school districts in the evaluation group were also included in the implementation measurement of this key component, the evaluator would enter “A” for all. AD **Year 5: Component Level Threshold for**

**Fidelity of Implementation at the Sample Level (i.e. definition of implementation** 

**with fidelity) Same instructions as in column**  **E will appear in survey instrument, replacing year one with year five.** 

AE **Year 5: Component Level Fidelity Score for**

**the Entire Sample Same instructions as in column**  **F will appear in survey instrument, replacing year one wit**  **h year five.** AF **Year 5: Component Level Implementation with Fidelity at the Sample Level? (Yes/No)**  **Same instructions as in column G**  **will appear in survey instrument.**National Evaluation of i3 (NEi3) Data Collection Survey Page 16

**Column Column Header**  **Instructions for completing corresponding cells S** **cale -**  **Up Key Components (Scale -**  **Up grants only)** AG **Key c** **omponent / activities related to**  **scale -**  **up** Please enter each key component necessary to support scale-up efforts separately. **Scale -**  **Up Goals (Scale** **- Up grants only)**  AH **Goal** Please enter scale-up goals separately. Scale-up goals should be quantified and measurable, e.g., 500 additional teachers trained. AI **Results at grant end** Please enter the results of each scale-up goal achieved by the end of the grant period, e.g., If 523 additional teachers were trained, enter “523” in the field AJ **Goal met** Please enter a final determination (yes/no) indicating whether each scale-up goal was met by grant end. AK **Results Year 1** Optional: if you choose, please enter scale-up results for each goal at the end of each or any year(s) of implementation. For example, if scale-up goals were measured in years 3 and 4, enter results for each component in those fields and enter N/A (or leave blank) fields associated with years 1, 2, and 5. AL **Results Year 2** AM **Results Year 3** AN **Results Year 4** AO **Results Year 5**National Evaluation of i3 (NEi3) Data Collection Survey Page 17



## Please **com**

**plete a**  **row for**  **each contrast**  **me asured**  **by your**  **impact study**  **. The**  **following instructions**  **define the**  **data that should be entered in each of the colum ns.** 
**Column Column Header**  **Instructions for completing corresponding cells (for each contrast) INTERNAL USE ONLY:**  **Programming notes about skip** 

**patterns Contrast Description** 
A **Contrast Name** Please enter the names of each of the evaluation’s confirmatory and exploratory contrasts. This field does not affect the assessment or reporting of the contrast; the contrast name simply facilities communication between the evaluator and the NEi3 AR Team, and facilitates referencing contrasts between the data -entry fields of this survey. The contrast name should uniquely identify the contrast and differentiate among similar contrasts: e.g. “Math Achievement -confirmatory” and “Math Achievement-cohort 1” only might be the names of two similar contrasts. Field is required.

**Contrast Description: Key features that define a contrast** B **Confirmatory or** 

**Exploratory?** Indicate whether this contrast is confirmatory or exploratory. P lease note that

any pre-specified contrasts that were registered as confirmatory will be prepopulated as such and cannot be changed to be exploratory. All other contrasts will be listed as exploratory. Field is required. Only AR team members can edit this field. Default value for new contrasts is "exploratory".

C **Contrast Status** For each contrast, please select from among the following options:

- Contrast Confirmed if the effectiveness findings from this contrast are reported in this survey

- Not Reported if the effectiveness findings from this contrast are not reported in this survey and will not be reported to the NEi3 in the future

- Pending Additional Data if the effectiveness findings from this c ontrast will be reported in future waves of data collection. If you select this option, please provide the date when you plan to report effectiveness findings from this contrast in the next field. Field is always applicable. Drop down menu:

- Contrast confirmed

- Pending Additional Data

- Not reportedNational Evaluation of i3 (NEi3) Data Collection Survey Page 18



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**

D **Expected Reporting Date**  If effectiveness findings for this contrast are not yet available, please provide the date when you plan to report effectiveness findings. (Not applicable for contrasts for which data is provided in this survey or for contrasts for which findings will never be reported.) If the contrast status is NOT "pending additional data", display "Not Applicable".

E **Contrast Notes** This field is not required, but is available to provide contextual information or notes about each specific contrast as needed. Field is always applicable.

F **Outcome Domain** For confirmatory contrasts, this field is populated with the registered domain name of your pre-specified confirmatory contrast. As the domain is a defining feature of a contrast, it cannot be edited by the evaluator. Contact the helpdesk to correct typos or other minor errors. For exploratory contrasts, please enter the name of the domains measured. If the exploratory contrast measures the same construct as one of the confirmatory contrasts, please use the same domain name. Field is always applicable. Field cannot be edited for confirmatory contrasts.

G **Intervention Condition**  For confirmatory contrasts, this field is populated with the registered description of the intervention tested by the pre-specified confirmatory contrast. As the intervention condition is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please name the intervention(s) experienced by the treatment group in the study. Do not use acronyms. If the condition is the same as the one tested by a confirmatory contrast, please use the same name. If the contrast involves comparing o ne treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group a nd the comparison group. Field is always applicable. Field cannot be edited for confirmatory contrasts.National Evaluation of i3 (NEi3) Data Collection Survey Page 19



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**

H **Comparison Condition** For confirmatory contrasts, this field is populated with the registered description of the counterfactual condition in the pre-specified confirmatory contrast. As the intervention condition is a defining feature of a contrast, it cannot be edited. For exp loratory contrasts, please name the intervention(s) experienced by the comparison group in the study. This may be the name of another intervention or Business as Usual, if no specific intervention. Do not use acronyms. If the condition is the same as the o ne tested by a confirmatory contrast, please use the same name. If the contrast involves comparing one treatment to another, please identify one treatment as the condition and another as the comparison condition for the purposes of responding to this survey. Use this identification throughout the survey when asked about the treatment/intervention group and the comparison group . Field is always applicable. Field cannot be edited for confirmatory contrasts.

I **Educational Level** For confirmatory contrasts, this field is populated with the educational level that was registered in the pre-specified confirmatory contrast. As the educational level is a defining feature of a contrast, it cannot be edited. For exploratory contrasts, please report the educational level (e.g. elementary, middle or high school) of the students that experience the intervention. For teacher outcomes, report the educational level of the students in the teachers’ classrooms. Field is always applicable. Field cannot be edited for confirmatory contrasts.National Evaluation of i3 (NEi3) Data Collection Survey Page 20



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns Contrast Description: Approach to addressing contrast** 

J **Design** Please select one of the following designs: RCT, RDD, QED, Comparison Group Design Without a Pretest, ITS, Pre-post. For confirmatory contrasts, this field is populated with the registered design for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop- up box that will appear. Definitions: Randomized controlled trial (RCT). U nder this design, an evaluation uses a random process to assign study units to an intervention groups and to a control group, and measures the effect of the intervention by comparing the outcomes of units assigned to th e two groups. Regression discontinuity design (RDD). U nder this design, an evaluation would use a continuous “scoring” rule to assign study units to the intervention. Units with scores above a pre-set cutoff value are assigned to the treatment group and units with scores below the cutoff value are assigned to the comparison group, or vice versa. The evaluation then compares the outcomes of the two groups close to the cutoff to measure the effect of the intervention. Quasi- Experimental Design (QED). A QED compares outcomes for intervention participants with outcomes for a comparison group chosen through methods other than randomization (RCT) or a scoring rule with a well -defined threshold (RDD) and includes one or more pre-intervention measures. Compa rative Interrupted Time Series designs are treated as QEDs by the NEi3. Comparison Group Design Without a Pretest. A comparison group design without a pre-test compares outcomes for intervention participants with outcomes for a comparison group chosen th rough methods other than randomization (RCT) or a scoring rule with a well -defined threshold (RDD) and does not include preintervention observations. Interrupted Time Series Design (ITS). An ITS design uses observations at multiple points in time both before and after the intervention was implemented, Field is always applicable. Drop down menu:

- RCT

- RDD

-QED

-Comparison Group Design without a Pretest

- ITS

- Pre-post For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 21



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** and compares the outcomes after the intervention with the trend established prior to the intervention. These designs do not use a comparison group. Pre-post Design. A

p re-post design compares outcomes for intervention participants after receiving the intervention to their outcomes before receiving the intervention. These designs do not use a comparison group.

K **Unit of Assignment** For each contrast, please report **the unit**  **of assignment**  **or the**  **level at**  **which assignment**  **to the**  **intervention and**  **comparison conditions**  **occur (e.g.,**  **student, teacher, classroom, or school).** If you edit the pre-populated contents of this field
– which suggests a change to the study plan reg istered with the NEi3

– please explain why this change was made using the pop-up box that will appear. Field is always applicable. Drop down menu:

- District

- School

- Teacher

- Classroom

- Student

- Other (contact helpdesk to enter) For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

L **Intervention Group**

**[alternately, “Treatment Group”]: Description**  For confirmatory contrasts, this cell is prepopulated with the registered description of the intervention group for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– pl ease explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **which schools,**  **teachers and/or**  **students will**  **be included**  **in the intervention (“treatm**  **ent”) group** for this analysis. **Example 1**  **(Candy Valley):**  “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the intervention.” **Example 2**  **(Apple Valley):**  “This contrast pools data across all treated cohorts and schools. All students in treated grades in Fuji Elementary (K -2), Granny Smith Elementary (K

- 1) and Red Delicious Elementary (K -2) who were present for data collection in Spring 2014 are included in the treatment group.” Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 22



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**

M **Age/Grade during the**

**Intervention** For confirmatory contrasts, this cell is prepopulated with the registered

description of the age/grade of the students in the intervention group for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study p lan registered with the NEi3
– please explain why this change was made using the pop- up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **the age**  **or grade** **- level**  **of students**  **in the**  **intervention group**  **during their exposure to the intervention** . For teacher outcomes, please enter the grade(s) taught by the teachers. **Example 1 (Candy Valley):** “The intervention was fielded to first grade students.” **Example 2**  **(Apple Valley):**  “Both Fuji and Red Delicious Elementary Schools introduced the intervention in the 2011 -2012 school year to all kindergarten students, offered the intervention to all kindergarten and first graders in 2012- 2013 and to kindergarten through second graders in 2013-2014. Granny Smith Elementary introduced the intervention in the 2012-2013 school year to all kindergarten students and offered the intervention to all kindergarten and first graders in 2013-2014.” Field is always applicable. For confirmatory contrasts, ch anges to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

N **Length of Student**

**Exposure to Intervention** For confirmatory contrasts, this cell is prepopulated with the registered description of the age/grade of the students in the intervention group for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study p lan registered with the NEi3
– please explain why this change was made using the pop- up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **how long**  **(no. of**  **semesters/years) the**  **students in**  **th e**  **intervention group**  **were engaged**  **in the**  **intervention at**  **the time**  **the outcome**  **was measured**  **for this**  **contrast**
. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why th is change was made using the pop-up box that will appear. **Example 1 (Candy Valley):** “Students received one year of treatment” **Example 2**  **(Apple Valley):**  “When outcomes were measured in Spring 2014, Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 23



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** both Fuji and Red Delicious Elementary Schools student s had the following pattern of exposure to treatment: second grade students had three years of treatment, first grade students had two years of treatment and kindergarten students had one year of treatment. When outcomes were measured in Spring 2014, Granny Smith Elementary first graders had two years of treatment and kindergarten students had one year of treatment.”

O **Comparison Group:**

**Description** For confirmatory contrasts, this cell is prepopulated with the registered

description of the comparison group for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please describe **which schools,**  **teachers and/or**  **students will**  **be included**  **in the comparison group**  for this analysis. **Example 1**  **(Candy Valley):**  “2012 first-graders in all Candy Valley District Elementary Schools reading below grade level who do not have an IEP and were randomly assigned to the control group.” **Example 2**  **(Apple Valley):**  “This contrast pools data across all cohorts and schools matched to treated cohorts and schools. K -2 students from Honey Crisp Elementary (comparison for Fuji Elementary), K - 1 students from Ginger Gold Elementary serves (

comparison for Granny Smith Elementary), and K -2 students Jonagold Elementary (comparison for Red Delicious Elementary) who were present for data collection in Spring 2014 are included in the treatment group.” Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

**Contrast Description: Outcome Measurement**

P **Outcome or Instrument**

**Name (full names, not acronyms)**  For confirmatory contrasts, this cell is prepopulated with the registered outcome

measure for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change, please report the full name of the outcome measure, including the name of the subtest or subscale if applicable. If the instrument for the outcome measure is not used in one of th e confirmatory contrasts, please describe (1) what the Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should b ring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 24



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** instrument is designed to measure and (2) how the outcome measure is constructed.

Q **Construction of Outcome**

**Measure (binary, ordinal, or continuous)**  For confirmatory contrasts, this cell is prepopulated with the registered scale of

the outcome measure for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please indicate whether the outcome measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous. Field is always applicable. For confirmatory contrasts, changes to this field afte r as planned pre-population should bring up a pop-up box asking for an explanation.

R **Unit of Observation for**

**Outcome Measure** For confirmatory contrasts, this cell is prepopulated with the registered unit of

observation for the outcome measure—i.e. the units for which outcomes are observed in the analysis data. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will a ppear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the unit for which outcomes are observed in your data (e.g. student, teacher, classroom, school, district). **Example:** If your data are school -level averages of student test scores, enter “School ”. Field is always applicable. Drop down menu:

- District

- School

- Teacher

- Classroom

- Student

- Other (contact helpdesk to enter) For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.
S **Timing of Outcome** 

**Measure (year, Fall/Spring) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report when during the calendar year the **outcome** was **measured**
. Field is always applicable. For confirmatory co ntrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

T **Timing of Outcome** 

**Measure (age/grade) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 25



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** please report the age or grade level for which the **outcome** was **measured**
.
U **Timing of Outcome** 

**Measure (year, Fall/Spring) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report when during the calendar year the **outcome** was **measured**
. Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

V **Timing of Outcome** 

**Measure (age/grade) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the age or grade level for which the **outcome** was **measured**
. Field is always applicable. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation. **Contrast Description: Baseline Measures**

W **Key Baseline**  **Measures** For confirmatory contrasts, this cell is prepopulated with the registered baseline measure for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the baseline variable(s) on which you are establishing baseline equivalence. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

X **Construction of Baseline**

**Measure (binary, ordinal, or continuous)**  For confirmatory contrasts, this cell is prepopulated with the registered scale of

the baseline measure for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please exp lain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please indicate whether the baseline measure is binary (dichotomous), ordinal (ordered, e.g., “failing, provisionally passing, proficient, advanced”), or continuous. If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 26



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)

Y **Unit of Observation for**

**Baseline Measure** For confirmatory contrasts, this cell is prepopulated with the registered unit of

observation for the baseline measure—i.e. the units for which outcomes are observed in the analysis data. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the unit for which baseline measures are observed in your data (e.g. student, teacher, classroom, school, district). **Example:** If your data are school -level averages of student test scores, enter “School ”. (Field is not applicable to the following designs: comparison grou p designs without a pretest, ITS designs and pre-post designs.) If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring u p a pop-up box asking for an explanation.

Z **Timing of Baseline**

**Measure (year, Fall/Spring,) for the**  **Intervention Group** For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when during**  **the calendar**  **year the**  **data you**  **will use**  **to test**  **baseline equivalence was measured**
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a po p-up box asking for an explanation.

AA **Timing of Baseline**

**Measure (age/grade) for the Intervention Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report the age or grade level **for which**  **the data**  **you will**  **use to**  **test baseline equivalence was measured** 
. If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.National Evaluation of i3 (NEi3) Data Collection Survey Page 27



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.)

AB **Timing of Baseline**

**Measure (year, Fall/Spring) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when during**  **the calendar**  **year the**  **data you**  **will use**  **to test**  **baseline equivalence was measured**
. (Field is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field after as planned pre-population should bring up a pop-up box asking for an explanation.

AC **Timing of Baseline**

**Measure (age/grade) for the Comparison Group**  For confirmatory contrasts, this cell is prepopulated with the registered timing of baseline measurement for the contrast. If you edit the pre-populated contents of this field
– which suggests a change to the study plan registered with the NEi3
– please explain why this change was made using the pop-up box that will appear. For exploratory contrasts and for confirmatory contrasts you wish to change , please report **when the**  **data you**  **will use**  **to test**  **baseline equivalence**  **was measured** 
. (Field is not applicab le to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. For confirmatory contrasts, changes to this field af

ter as planned pre-population should bring up a pop-up box asking for an explanation.

**Confounds** AD **Intervention Condition**  **:**

**Other Activities** To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered to the intervention group but not the comparison group? If so, please describe the other intervention. Field is always applicable.

AE **Fundamental**

**Confound at**  **the District**  **Level: Intervention**  **Condition** Does the **intervention condition**  include schools/teachers/students from a **single school district** which does not also include comparison group members? If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. AF **Fundamental**

**Confound at**  **the School**  Does the **intervention condition**  include teachers/students from a **single school**  which does not also include comparison group members? If the Design is Comparison Group without a Pretest, ITS o r Pre-post,National Evaluation of i3 (NEi3) Data Collection Survey Page 28



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns Level:**  **Intervention Condition**  display “Not Applicable”.
AG **Comparison Condition :** 

**Other Activities** To your knowledge, were any other interventions (other than your project’s i3-

funded intervention) targeting the outcome domain listed in [C] that were offered to the comparison group but not the intervention group? If so, please describe the other intervention. Field is always applicable.

AH **Fundamental Confound at**

**the District Level: Comparison Condition**  Does the **comparison condition**  include schools/teachers/students from a **single school district** which does not also include intervention group members? If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. AI **Fundamental Confound at**

**the School Level:**  **Comparison Condition** Does the **comparison condition**  include teachers/students from a **single school**  which does not also include intervention group members? If the Design is Comparison Group without a Pretest, ITS or Pre-post, display “Not Applicable”. **Outcomes** AJ **Outcome or Instrument**

**Name (full names, not acronyms)**  This field displays the outcome measure you entered previously. This field is not

editable. If you wish to edit this entry, go to the Contrast Description: Outcome Measurement tab. Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Outcome Measurement" AK **Who Provided the**

**Outcome Data?** For each contrast, please report the **name of**  **the entity**  **that provided**  **the**

**outcome data.** Field is always applicable.

AL **Value and Type of**

**Reliability Measure of Instrument for Outcome**  **Data** Please provide a narrative description of the outcome and discuss the reliability

and validity of the measure, including **the value and type of reliability measure**
. For example, you may report an internal consistency, test-retest reliability, or inter-rater reliability measure. For outcomes for which reliability statistics cannot be obtained, please describe why reliability statistics are not available. **Example 1:**  “The Orange Gulch District writing assessment was scored by trained reviewers according to a holistic rubric. A random subset of essays was double scored and demonstrated an inter -rater reliability of 89%.” **Example 2:**  “Attendance is the proportion of school days for which an individual student was present for at least one course at Orange Gulch High School. Reliability statistics cannot be calculated for attendance. However, it is a face Field is always applicable.National Evaluation of i3 (NEi3) Data Collection Survey Page 29



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** valid measure of the pre-specified domain: High School Attendance.”
AM **Consistency of Collection** Please describe any differences in the process for collecting data on the outcome measure between the groups. To be consistently collected, an outcome measure must be collected using the same rules or procedures in both groups. For example, data would not be consistently collected if intervention students take the post-test in January, but comparison students take the test in March
– two months later. Field is always applicable.

AN **Treatment of Missing Data** Describe how the study addressed missing values for the outcome measure. Field is always applicable. **Assessing Attrition** AO **Unit of Assignment**  This field displays the unit of assignment you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Design tab. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Design"
AP **Unit of Observation for** 

**Outcome Measure** This field displays the unit of observation for outcome entered previously. This

field is not editable. If you wish to edit this entry, go to the Contrast Description: Outcome Measurement tab. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Outcome Measurement"

AQ **Cluster Randomized**

**Design?** Select “Yes” if the contrast is from a cluster randomized design.

Select “No”, otherwise For NEi3, all **cluster randomized**  **designs** are those in which clusters (of individuals) were assigned by a process that was functionally random. These clusters may be associated with, for instance, particular teachers, classrooms, schools, or districts. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Drop down menu:

- Yes

- No

- Not Applicable

AR **Analytic Sample Includes**

**Joiners?** Select “not applicable” if your design is not a cluster randomized design.

Select “yes” if your analytic sample includes individuals who were not present in the cluster at the time of random assignment. Select “no” if your analytic sample is restricted to individuals who were present at If the contrast is NOT a cluster randomized design, display "Not Applicable". Drop down menu:

- YesNational Evaluation of i3 (NEi3) Data Collection Survey Page 30



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** the time of random assignment. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) - No

- Not Applicable

AS **Description of Joiners** Provide a narrative description of the joiners in your analytic sample. I nclude information on when they joined the sample relative to a) the point of random assignment and b) the point at which clusters were notified of their assignment status. Example 1: The analysis sample includes all students in study schools in Year 3. Some students may have been transferred into study schools after the point of random assignment (Year 1) and may not have two full years of exposure. Example 2: The student sample is comprised of all students in study teachers’ classrooms in Year 2 of the study. Because teacher rosters were not set at the point of random assignment (Year 1), all students in Year 2 are considered joiners. (Not applicable if your sample does not include joiners.) If the contrast does NOT include joiners, display "Not Applicable".

AT **Intervention Group: Units** 

**of Assignment Randomized**  Enter the number of clusters randomly assigned to the intervention group.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcom e measure is the same as the unit of assignment, then display "Not Applicable".

AU **Intervention Group:**

**Reasons for Loss of Units of Assignment**  List reasons for sample loss separated by a carriage return.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".National Evaluation of i3 (NEi3) Data Collection Survey Page 31



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**

AV **Intervention Group: Units**

**of Assignment Lost (for each reason)**  For each reason for sample loss, list the number of units lost.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".

AW **Intervention Group: Units**

**of Assignment in Analytic Sample**  Enter the number of clusters for the intervention group for the analytic sample.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".

AX **Intervention Group: Units**

**of Observation Randomized or in**  **Randomized Cluster** Enter the sample size for the intervention group at the time of random assignment Complete this field the contrast is from: (a)a cluster randomized design AND the unit of assignment differs from the unit of analysis AND the contrast does not include joiners (b)any randomized design where the unit of analysis is the same as the unit of assignment (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".

AY **Intervention Group: Units**

**of Observation in Remaining Clusters**  Enter the sample size for the intervention group at the time of random assignment in clusters that appear in the analytic sample. This excludes units of observation for which the entire cluster left the sample. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students at the time of random assignment in schools that did not attrit This field is not applicable if the contrast is from a cluster randomized design If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome m easure is NOT the same as the unit of assignment, then display "Not Applicable".National Evaluation of i3 (NEi3) Data Collection Survey Page 32



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.)

AZ **Intervention Group:**

**Reasons for Loss of Units of Observation**  List reasons for sample loss separated by a carriage return.

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".

BA **Intervention Group: Units**

**of Observation Lost (for each reason)**  For each reason for sample loss, list the number of units lost.

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".

BB **Intervention Group: Units**

**of Observation in Analytic Sample**  Enter the sample size for the intervention group for the analytic sample

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Appl icable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable". BC **Comparison Group: Units**

**of Assignment Randomized**  Enter the number of clusters randomly assigned to the comparison group.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomizedNational Evaluation of i3 (NEi3) Data Collection Survey Page 33



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** (Section is not applicable to the fol lowing designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".

BD **Comparison Group:**

**Reasons for Loss of Units of Assignment**  List reasons for sample loss separated by a carriage return.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".

BE **Comparison Group: Units**

**of Assignment Lost (for each reason)**  For each reason for sample loss, list the number of units lost.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation. (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Appli cable".

BF **Comparison Group: Units**

**of Assignment in Analytic Sample**  Enter the number of clusters for the comparison group for the analytic sample.

Complete this field if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of analysis (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is (a) NOT an RCT design, OR (b) NOT a cluster randomized design, OR (c) the unit of observation for outcome measure is the same as the unit of assignment, then display "Not Applicable".
BG **Comparison Group: Units**

**of Observation Randomized or in**  **Randomized Cluster** Enter the sample size for the comparison group at the time of random assignment. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students in all schools at the time of random assignment If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same asNational Evaluation of i3 (NEi3) Data Collection Survey Page 34



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) the unit of assignment, then display "Not Applicable".

BH **Comparison Group: Units**

**of Observation in Remaining Clusters**  Enter the sample size for the comparison group at the time of random assignment in clusters that appear in the analytic sample. This excludes units of observation for which the entire cluster left the sample. E.g. For designs where students are the unit of observation and schools are the unit of assignment, enter the number of eligible students at the time of random assignment in schools that did not attrit This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable".

BI **Comparison Group:**

**Reasons for Loss of Units of Observation**  List reasons for sample loss separated by a carriage return.

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and p re-post designs.) If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assign ment, then display "Not Applicable".
BJ **Comparison Group: Units**

**of Observation Lost (for each reason)**  For each reason for sample loss, list the number of units lost.

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the unit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group If the contrast is NOT an RCT design, then display "Not Applicable". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same asNational Evaluation of i3 (NEi3) Data Collection Survey Page 35



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** designs without a pretest, ITS designs and pre-post designs.) the unit of assignment, then display "Not Applicable".

BK **Comparison Group: Units**

**of Observation in Analytic Sample**  Enter the sample size for the comparison group for the analytic sample

This field is not applicable if the contrast is from a cluster randomized design AND the unit of assignment differs from the u nit of observation AND the contrast includes joiners (Section is not applicable to the following designs: QEDs, comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast is NOT an RCT design, then display "Not Applicabl e". If the contrast is (a) a cluster randomized design, AND (b) includes joiners, AND (c) the unit of observation for outcome measure is NOT the same as the unit of assignment, then display "Not Applicable". **Baseline Measurement** 
BL **Key Baseline**  **Measures** This field displays the key baseline measure(s) you entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"

BM **Construction of Baseline**

**Measure (binary, ordinal, or continuous)**  This field displays the description of the construction of the baseline measure you

entered previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the fol lowing designs: comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"

BN **Unit of Observation for** 

**Baseline Measure**  This field displays the level of observation for the baseline measure you entered

previously. This field is not editable. If you wish to edit this entry, go to the Contrast Description: Baseline Measures tab. (Section is not applicable to the following des igns: comparison group designs without a pretest, ITS designs and pre-post designs.) Field is always applicable. Field can only be edited in previous tab: "Contrast Description: Baseline Measures"

BO **Treatment of Missing Data**

**for Baseline Balance Testing**  Describe how the study addressed missing values for the baseline measure for

baseline balance testing. If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".National Evaluation of i3 (NEi3) Data Collection Survey Page 36



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** BP **Narrative Description of** 

**Sample for Baseline Balance Testing**  Describe the eligibility criteria for inclusion in the sample for baseline balance

testing. If balance is established using cluster-level averages (e.g. school -level average student test score), describe which individual units are included in the average.

BQ **Type of Information Used**

**to Assess Baseline Equivalence**  Select one option that best represents the information generated by your

baseline balance testing:  Unadjusted Mean and Standard Deviation T-statistic  Dichotomous T/C Difference This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable” (Section is not applicable to the following designs: comparison g roup designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".

BR **Narrative Description of**

**Analysis for Baseline Balance Testing**  Provide a brief narrative description of the method used to assess baseline

balance. Example 1: Baseline summary statistics for both groups in the analytic sample were calculated. Data are from unadjusted means and standard deviations. Example 2: Differences in baseline means were calculated using a statistical model with the same structural components as the statistical model that was used to estimate impacts on the outcome variable for the same contrast. This is a two level model with students nested in schools. The pretest variable is the independent variable. The only covariates included in the model are an intercept term and a school level treatment indicator. If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".

BS **Baseline Balance Testing**

**Sample Size: Intervention Group**  Enter the sample size for the intervention group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".National Evaluation of i3 (NEi3) Data Collection Survey Page 37



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**
BT **Mean of**

**Baseline Measure: Intervention Group**  Enter the pre-intervention mean for the intervention group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs. ) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BU **Standard Deviation of**

**Baseline Measure: Intervention Group**  Enter the pre-intervention standard deviation for the intervention group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BV **Baseline Balance Testing**

**Sample Size: Comparison Group**  Enter the sample size for the comparison group for baseline balance testing

(Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BW **Mean of**

**Baseline Measure: Comparison Group**  Enter the pre-intervention mean for the comparison group. For binary measures, enter a decimal value. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BX **Standard Deviation of**

**Baseline Measure: Comparison Group**  Enter the pre-intervention standard deviation for the comparison group. (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".
BY **Baseline Intervention**  

** Comparison Difference** Enter the difference between the intervention and comparison group means. If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable".National Evaluation of i3 (NEi3) Data Collection Survey Page 38



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns**
BZ **T -** **Statistic for Baseline**

**Balance Test** Enter the t-statistic for the pre-intervention difference in means between the

treatment and the comparison (Section is not applicable to the following designs: comparison group designs without a pretest, ITS designs and pre-post designs.) If the contrast design is (a) Design is Comparison Group without a Pretest, OR (b) ITS, OR (c) Pre-post, then display "Not Applicable". **Reported Findings**

CA **Type of Findings Reported** Select one of the following types of data to use to report findings:  Unadjusted means Adjusted means, e.g. ANOVA T-test  Coefficient from a linear regression model, e.g. OLS, HLM, etc.  Coefficient from a non -linear regression model, e.g. logit or probit This field determines which of the following fields you need to complete. Fields that do not need to be completed are populated with “Not applicable” Field is always applicable. Drop down menu:

- Adjusted means, e.g. ANOVA

- T-test

- Coefficient from a linear regress ion model, e.g. OLS, HLM, etc.

- Coefficient from a non -linear regression model, e.g. logit or probit

CB **Narrative Description of**

**Analysis for Impact**  **Evaluation** Provide a brief narrative description of the method used to estimate impact.

Include information that describes:

- the nested or longitudinal structure of the model

- any adjustments for clustering of standard errors

- if the outcome is a binary or categorical measure and, if so, what kind of estimation is performed

- whether the pre-test variable is included and the level of inclusion

- the level at which the treatment variable enters the model Example 1: Impact estimates come from a two-level regression model with students nested in schools. The pretest variable is included as a student-level covariate. Treatment enters the model at the school level. Example 2: We use logistic regression to estimate the impact of the intervention on attendance: a binary measure defined for each student. Estimates are log odds-ratios. Standard errors are clustered at the school -level. Field is always applicable.

CC **Intervention Group:**

**Sample Size** Please report the size of the intervention group sample for the analysis. Field is always applicable. CD **Intervention Group:**

**Number of Clusters in** Please report the **number of clusters assigned to the intervention group that appear in the analysis sample** 
. If the unit of observation for outcome measure is the same as the unit ofNational Evaluation of i3 (NEi3) Data Collection Survey Page 39



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns Analytic Sample**  Although your analysis may account for clustering at multiple levels, this field only

asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned. assignment, display "Not Required".
CE **Comparison Group:**

**Sample Size** Please report the size of the comparison group sample for the analysis. Field is always applicable.
CF **Comparison Group:**

**Number of Clusters in Analytic Sample**  Please report the number of clusters assigned to the comparison group that

appear in the analysis sample. Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment: e.g. number of schools, if schools were assigned. If the unit of observation for outcome measure is the same as the unit of assignment, display "Not Required".

CG **Intraclass Correlation**

**Coefficient** Please report the intraclass correlation coefficient to describe the similarity of

units at the level of observation within the cluster at the level of assignment Although your analysis may account for clustering at multiple levels, this field only asks about clusters at the level of assignment, e.g. the correlation between outcomes of students within the same school, if schools were assigned. If the unit of observation for outcome measure is the same as the unit of assignment, display "Not Required".

CH **Unadjusted Outcome**

**Mean for the Intervention Group**  Please provide the unadjusted mean of the outcome measure in the intervention

group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) If the type of findings is NOT unadjusted means, display "Not Required".

CI **Adjusted Outcome Mean**

**for the Intervention Group** Please provide the adjusted mean of the outcome measure in the intervention group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) If the type of findings is NOT adjusted means, display "Not Required".

CJ **Unadjusted Standard**

**Deviation in the Outcome Measure of the**  **Intervention Group** Please provide the unadjusted standard deviation of the outcome measure in the intervention group. Field is always applicable.

CK **Unadjusted Outcome**

**Mean for the Comparison Group**  Please provide the unadjusted mean of the outcome measure in the comparison

group. (You do not need to complete this field if it is automatically populated with “Not If the type of findings is NOT unadjusted means, display "Not Required".National Evaluation of i3 (NEi3) Data Collection Survey Page 40



### Column **Column Header Instructions for completing corresponding cells (for each contrast)**  **INTERNAL USE ONLY: Programming notes**  **about skip**

**patterns** applicable”. However, you can choose to provide this information if you wish.) CL **Adjusted Outcome Mean**

**for the Comparison Group** Please provide the unadjusted mean of the outcome measure in the comparison group. (You do not need to complete this field if it is automatically populated with “Not applicable”. However, you can choose to provide this information if you wish.) If the type of findings is NOT adjusted means, display "Not Required".

CM **Unadjusted Standard**

**Deviation in the Outcome Measure of the**  **Comparison Group** Please provide the unadjusted standard deviation of the outcome measure in the comparison group. Field is always applicable.

CN **Impact Estimate (e.g.,**

**coefficient from OLS or HLM)**  Please report the **regression coefficient** that estimates of the effect of the

intervention . If your impact estimate is the sum of multiple regression coefficients, please enter this sum here. (You do not need to complete this field if it is automatically populated with “Not applicable”.) If the type of findings is NOT a coefficient from a linear or non -linear regression model, display "Not Required".

CO **Standard Error of Impact**

**Estimate** Please insert the **estimated standard error of the regression coefficient** here.

If your impact estimate is the sum of multiple coefficients, please enter the standard deviation of the summed variable. (You do not need to complete this field if it is automatically populated with “Not applicable”.) If the type of findings is unadjusted means, display "Not Required".

CP **Two -** **tailed p -** **value** Please insert the associated **p -** **value that has not been adjusted for multiple**  **comparisons** here. If your impact estimate is the sum of multiple coefficients, please enter the associated p-value here. (You do not need to complete this field if it is automatically populated with “Not applicable”.) If the type of findings is unadjusted means, display "Not Required".

CQ **Degrees of Freedom** Please enter the degrees of freedom of your statistical analysis. (You do not need to complete this field if it is automatically populated with “Not applicable”.) If the type of findings is unadjusted means, display "Not Required".